Image processing apparatus, image processing method, and program

Abstract

To solve a problem such that intention of operation relative to a certain position image cannot be diverted to other position image.SOLUTION: An image processing device 1 includes: a response management unit 113 for storing two or more pieces of response information indicating correspondence between an operation sequence definition for defining a set of operation identifiers relative to two or more each operation and an intention identifier for identifying intention of the operation; a first output unit 141 for outputting a first image associated with position information; an operation sequence receiving unit 121 for receiving an operation sequence that is a set of operation identifiers for identifying each of two or more operation for the first image; an intention sequence acquisition unit 131 for acquiring an intention sequence corresponding to the operation sequence received by the operation sequence receiving unit 121 from the response management unit 113; a second output unit 142 for outputting a second image; and an operation unit 136 for performing operation on the intention sequence acquired by the intention sequence acquisition unit 131 on the second image. By the image processing device, intention of operation on the first image can be diverted to the second image, and as a result, the second image can be automatically operated according to intention of a user.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to, for example, an image processing apparatus that performs processing on a position image that is an image associated with position information such as Street View.

Background Art

[0002] Conventionally, there has been a technique for improving the operability of Street View. More specifically, conventionally, in consideration of the form of using a landscape image to be displayed and a mobile terminal device, the operability has been improved, and it has been possible to view landscape information at a desired viewpoint height when going out into the city for a social studies field trip or the like with a Street View viewer system (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, the intention of an operation on a position image that is an image associated with position information could not be diverted to other position images.

Means for Solving the Problems

[0005] The image processing apparatus of the first invention includes a correspondence management unit that stores two or more pieces of correspondence information indicating the correspondence between an operation sequence definition that defines a set of operation identifiers for each of two or more operations and an intention identifier that identifies the intention of the operation, a first output unit that outputs a first image that is an image associated with position information, an operation sequence reception unit that receives an operation sequence that is a set of operation identifiers for each of two or more operations on the first image, an intention sequence acquisition unit that acquires from the correspondence management unit an intention sequence that is an intention identifier corresponding to one or more operation sequence definitions that match the operation sequence received by the operation sequence reception unit, a second output unit that outputs a second image, and an operation unit that performs one or more operations on the second image that satisfy the operation sequence definition for the intention sequence acquired by the intention sequence acquisition unit.

[0006] With such a configuration, the intention of the operation on the first image can be diverted to the second image, and as a result, the second image can be automatically operated according to the user's intention.

[0007] Further, the image processing apparatus of the second invention, compared with the first invention, has the correspondence management unit storing two or more pieces of first correspondence information indicating the correspondence between an operation sequence definition and a chunk identifier that identifies a chunk which is a block of operations, and two or more pieces of second correspondence information indicating the correspondence between a chunk sequence definition that is a definition of a chunk sequence which is a set of one or more chunk identifiers and an intention identifier. The intention sequence acquisition unit acquires from the two or more pieces of first correspondence information one or more chunk sequences that are a set of one or more chunk identifiers corresponding to one or more operation sequence definitions that match the operation sequence received by the operation sequence reception unit, and acquires from the two or more pieces of second correspondence information one or more intention identifiers corresponding to the chunk sequence definition that matches the one or more chunk sequences.

[0008] With such a configuration, the intention of the operation on the first image can be diverted to the second image, and as a result, the second image can be automatically operated according to the user's intention.

[0009] Further, the image processing apparatus of the third invention, with respect to the second invention, refers to a feature management unit in which two or more pieces of feature information having a feature identifier for identifying a feature, and two or more feature attribute values including the category of the feature and the position information of the feature are stored, and uses the position information associated with the first image to determine a target feature that is a feature in the first image and meets the attention conditions, a category acquisition unit that acquires the category of the target feature from the feature management unit, and a target feature determination unit that determines a target feature that is a feature in the second image and has the same category as the category acquired by the category acquisition unit. The operation unit is an image processing apparatus that performs one or more operations that satisfy the operation sequence definition for each of the one or more intention identifiers acquired by the intention sequence acquisition unit on the target feature.

[0010] With such a configuration, the intention of the operation on the first image can be diverted to the second image, and as a result, the second image can be automatically operated according to the user's intention.

[0011] Further, the image processing apparatus of the fourth invention, with respect to the third invention, the operation unit is an image processing apparatus that uses the position information of the target feature to perform an operation on the target feature so as to maintain the positional relationship corresponding to the intention identified by the intention identifier.

[0012] With such a configuration, the intention of the operation on the first image can be diverted to the second image, and as a result, the second image can be automatically operated according to the user's intention.

[0013] Further, the image processing apparatus of the fifth invention, with respect to the fourth invention, the operation sequence definition is one or more operation identifiers using a regular expression, the chunk sequence definition is one or more chunk identifiers using a regular expression, and the operation unit uses the position information of the target feature to change the number of repetitions of the chunk identifier, or change the selection of the chunk identifier, or change the number of repetitions of the operation identifier, or change the selection of the operation identifier so as to maintain the positional relationship corresponding to the intention identified by the intention identifier, and performs an operation on the target feature.

[0014] With such a configuration, the intention of the operation on the first image can be diverted to the second image, and as a result, the second image can be automatically operated according to the user's intention.

[0015] Further, the image processing apparatus according to the sixth invention further includes an image determination unit that acquires a position image including a feature of the same category as the category acquired by the category acquisition unit from an image storage unit in which two or more position images are stored, and the second image is a position image acquired by the image determination unit.

[0016] With such a configuration, the second image can be automatically determined.

[0017] Further, the image processing apparatus according to the seventh invention, with respect to the sixth invention, the image determination unit is one or more second images including features of the same category as the category acquired by the category acquisition unit, and determines one or more second images associated with position information that satisfies a position condition with respect to the position information associated with the first image, and the operation unit performs one or more operations that satisfy the operation sequence definition for each of the one or more intention identifiers acquired by the intention sequence acquisition unit on each of the one or more second images determined by the image determination unit.

[0018] With such a configuration, the second image can be automatically determined.

[0019] Further, the image processing apparatus according to the eighth invention includes a correspondence management unit that stores two or more pieces of correspondence information indicating the correspondence between the operation sequence definition and the intention identifier, an intention reception unit that receives intention information including an intention sequence acquired according to an operation on the first image, a second output unit that outputs the second image, and an operation unit that acquires an operation sequence definition for each of the one or more intention identifiers included in the intention sequence included in the intention information received by the intention reception unit, and performs one or more operations that satisfy each of the one or more operation sequence definitions on the second image.

[0020] With such a configuration, the intention information for specifying the intention of the operation on the first image can be reused.

[0021] Further, the image processing apparatus of the ninth invention, with respect to the eighth invention, the intention information includes the category of the target feature, and is a feature of the same category as the category included in the intention information, and further includes a target feature determination unit that determines a target feature that is a feature in the second image, and the operation unit acquires an operation sequence definition for the intention sequence included in the intention information, and performs one or more operations that satisfy each of the one or more operation sequence definitions on the target feature.

[0022] With such a configuration, the intention information for specifying the intention of the operation on the first image can be reused.

[0023] Further, the image processing apparatus of the tenth invention, with respect to any one of the first to ninth inventions, the first image and the second image are image processing apparatuses that are different types of position images.

[0024] With such a configuration, the intention of the operation on the first type of position image can be diverted to the second type of position image.

[0025] Further, the image processing apparatus of the eleventh invention, with respect to any one of the first to eighth inventions, the first image and the second image are image processing apparatuses that are a street view, a three-dimensional image, an indoor image, or a drive recorder image.

[0026] With such a configuration, as a result of being able to divert the intention of the operation on the first image to the second image, the second image can be automatically operated according to the user's intention.

Advantages of the Invention

[0027] According to the image processing apparatus of the present invention, as a result of being able to divert the intention of the operation on the first image to the second image, the second image can be automatically operated according to the user's intention.

Brief Description of the Drawings

[0028]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Embodiments for Carrying Out the Invention

[0029] Hereinafter, embodiments of an image processing apparatus and the like will be described with reference to the drawings. In the embodiments, components denoted by the same reference numerals perform the same operations, and thus the description may be omitted again.

[0030] (Embodiment 1) In the present embodiment, an operation sequence that is one or more operations on a first image that is an image associated with position information is acquired, an intention sequence indicating a set of intentions of the operations based on the operation sequence is acquired, and an image processing apparatus that performs an operation according to the intention sequence on a second image will be described. Note that the image processing apparatus usually does not directly use the operation sequence on the first image for the operation on the second image.

[0031] Also, in the present embodiment, a chunk sequence indicating a set of one or more chunks is acquired from the operation sequence on the first image, an intention sequence is acquired from the chunk sequence, and an image processing apparatus that performs an operation according to the intention sequence on a second image will be described.

[0032] Also, in the present embodiment, a feature of interest in the first image is acquired, and an image processing apparatus that performs an operation on the second image on a feature having the same category as the category of the feature of interest will be described.

[0033] Also, in the present embodiment, an image processing apparatus that does not directly use the operation sequence on the first image to operate the second image, but performs an adjustment according to the attribute value of the target feature of the second image and operates the second image will be described.

[0034] Also, in the present embodiment, an image processing apparatus that selects one or more second images and operates each of the one or more second images will be described. In such a case, an image processing apparatus that selects one or more second images that satisfy a position condition with respect to the first image will be described.

[0035] Also, in the present embodiment, an image processing apparatus that selects one or more second images and compares each of the one or more second images with a first image, and operates each using an operation sequence for the first image will be described.

[0036] Furthermore, in the present embodiment, a case where the first image and the second image are different types of images will also be described.

[0037] Note that in this specification, the fact that information X is associated with information Y means that information Y can be obtained from information X, or information X can be obtained from information Y, and the method of the association is not limited. Information X and information Y may be linked, may exist in the same buffer, information X may be included in information Y, information Y may be included in information X, or the like.

[0038] Also, in this specification, determining information Z or selecting information Z means obtaining information Z, obtaining a pointer to information Z, obtaining an ID of information Z, setting a flag for information Z, or the like, as long as information Z can be accessed.

[0039] FIG. 1 is a block diagram of the image processing apparatus 1 in the present embodiment. The image processing apparatus 1 may be a stand-alone apparatus, or may be a server that receives instructions and information from a terminal apparatus (not shown) and transmits a position image to the terminal apparatus. When the image processing apparatus 1 is a stand-alone apparatus, the image processing apparatus 1 is, for example, a so-called personal computer, smartphone, or tablet terminal, but its type is not limited. When the image processing apparatus 1 is a server, the image processing apparatus 1 is, for example, a cloud server or an ASP server, but its type is not limited.

[0040] The image processing apparatus 1 includes a storage unit 11, a reception unit 12, a processing unit 13, and an output unit 14. The storage unit 11 includes a ground object management unit 111, an image storage unit 112, and a correspondence management unit 113. The reception unit 12 includes an operation sequence reception unit 121. The processing unit 13 includes an intention sequence acquisition unit 131, a target ground object determination unit 132, a category acquisition unit 133, an image determination unit 134, a target object determination unit 135, and an operation unit 136. The output unit 14 includes a first output unit 141 and a second output unit 142.

[0041] Various types of information are stored in the storage unit 11 that constitutes the image processing apparatus 1. The various types of information are, for example, the ground object information described later, the position image described later, and the correspondence information described later. Note that the ground object management unit 111, the image storage unit 112, and the correspondence management unit 113 may exist in an external server (not shown).

[0042] Two or more pieces of ground object information are stored in the ground object management unit 111. The ground object information is information regarding a ground object. A ground object is usually an object existing on the ground, but may include an object existing indoors. A ground object may also be referred to as an object. Ground objects are, for example, buildings, parks, bridges, rivers, and mountains.

[0043] The ground object information has a ground object identifier and one or more ground object attribute values. The ground object identifier is information for identifying a ground object. The ground object identifier is, for example, a ground object ID or a ground object name. The ground object attribute value is an attribute value of a ground object. It is preferable that the one or more ground object attribute values include a category and position information. The category is the type of a ground object. Categories are, for example, "izakaya", "school", "university", "rotating sushi restaurant", "park", and "museum". It is preferable that the one or more ground object attribute values include two or more pieces of position information. It is preferable that the one or more ground object attribute values included in the ground object information have region information for specifying the region of the ground object. Region information usually includes two or more pieces of position information. The position information is information for specifying a position. It is preferable that the position information is information for specifying a position in a three-dimensional space, but it may also be information for specifying a position in two dimensions. The position information is, for example, (latitude, longitude, altitude), but may also be (latitude, longitude).

[0044] The image storage unit 112 stores two or more position images. A position image is an image with which one or more pieces of position information are associated. A position image is usually a photographed image. A position image is an image to be an operation target. Note that the content of the operation is not limited.

[0045] The image with which position information is associated may be an image including the position information. The position information is, for example, a street view, a three-dimensional image, an indoor image, or a drive recorder image. The three-dimensional image is, for example, an image of google (registered trademark) earth. The indoor image is an image obtained by photographing the inside of a room. The indoor image is, for example, an image acquired by LiDAR. The drive recorder image is an image photographed by a drive recorder installed in a moving body.

[0046] As described above, the image storage unit 112 may exist not in the image processing apparatus 1 but in an external server. The external server is, for example, a server that manages street views or a google earth server.

[0047] The correspondence management unit 113 stores two or more pieces of correspondence information. The correspondence information is information indicating the correspondence between an operation sequence and an intention identifier. The operation sequence is a set of one or more operation identifiers. The set of operation identifiers is a sequence of operation identifiers in time series.

[0048] The correspondence information has, for example, an operation sequence definition and an intention identifier. The operation sequence definition is information for defining an operation sequence. The operation sequence definition is information for defining a set of two or more operation identifiers. It is preferable that the operation sequence definition be a description of one or more operation identifiers using a regular expression. The intention identifier is information for identifying an intention. The intention identifier is, for example, an intention ID or an intention name. An intention is an intention of an operation. It is preferable that two or more pieces of correspondence information be associated with an image type identifier. The image type identifier is information for identifying the type of an image. The image type identifier is, for example, "street view", "three-dimensional image", "indoor image", or "drive recorder image".

[0049] The correspondence information has, for example, first correspondence information and second correspondence information. The first correspondence information is information indicating the correspondence between the operation sequence definition and the chunk identifier. The chunk identifier is information for identifying a chunk. A chunk is a block of operations. A chunk is a set of one or more operations. The chunk identifier is the ID of the chunk or the chunk name. The second correspondence information is information indicating the correspondence between the chunk sequence definition and the intention identifier. The chunk sequence definition is the definition of a chunk sequence that is a set of one or more chunk identifiers. It is preferable that the chunk sequence definition is a description of one or more chunk identifiers using a regular expression. Note that it is preferable that the first correspondence information and the second correspondence information are associated with an image type identifier.

[0050] The reception unit 12 receives various instructions and information. The various instructions and information are, for example, an operation sequence for a first image, an output instruction, an operation end instruction, and correspondence information. The operation sequence is a set of operation identifiers for identifying two or more respective operations. The first image is a position image that is an operation target by the user.

[0051] The output instruction is an instruction to output the first image. The output instruction usually has an image identifier. The image identifier is an identifier of the position image. The image identifier is, for example, a URL or a file name.

[0052] The operation end instruction is an instruction to end the operation on the first image. The operation end instruction is, for example, an instruction to make the first image non-displayed.

[0053] Here, reception means, for example, reception of information transmitted via a wired or wireless communication line from a user's terminal device (not shown), but may also be a concept including reception of information input from an input device such as a keyboard, a mouse, or a touch panel, reception of information read from a recording medium such as an optical disk, a magnetic disk, or a semiconductor memory, and the like.

[0054] The input means for various instructions and information can be anything, such as a touch panel, keyboard, mouse, menu screen, etc.

[0055] The operation sequence reception unit 121 receives an operation sequence that is a set of operation identifiers for identifying two or more operations on the first image. Note that the operation on the first image is a user operation. The operation sequence reception unit 121, for example, receives an operation sequence from the user or receives an operation sequence from a terminal device (not shown) of the user. Also, the operation is, for example, any one of "move", "face left", "face right", "face up", "face down", "enlarge", "reduce". However, the operation is not limited.

[0056] The processing unit 13 performs various processes. The various processes are, for example, processes performed by the intention sequence acquisition unit 131, the target feature determination unit 132, the category acquisition unit 133, the image determination unit 134, the target object determination unit 135, or the operation unit 136.

[0057] The processing unit 13 acquires the first image corresponding to the output instruction. The processing unit 13, for example, acquires the first image from an image server (not shown). The processing unit 13, for example, acquires the first image from the image storage unit 112.

[0058] The processing unit 13 stores the correspondence information received by the reception unit 12 in the correspondence management unit 113.

[0059] The intention sequence acquisition unit 131 acquires, from the correspondence management unit 113, an intention sequence that is one or more intention identifiers corresponding to one or more operation sequence definitions that match the operation sequence received by the operation sequence reception unit 121. The intention sequence acquisition unit 131, for example, acquires an image type identifier of the image to be operated on and uses the correspondence information corresponding to the image type identifier to acquire the intention sequence.

[0060] The intention sequence acquisition unit 131 acquires, from two or more pieces of first correspondence information, a chunk sequence that is a set of one or more chunk identifiers corresponding to one or more operation sequence definitions that match the operation sequence received by the operation sequence reception unit 121, for example. Next, the intention sequence acquisition unit 131 acquires, from two or more pieces of second correspondence information, one or more intention identifiers corresponding to a chunk sequence definition that matches the chunk sequence, for example.

[0061] The target feature determination unit 132 refers to the feature management unit 111 and determines a target feature that is a feature in the first image and matches the target condition, using the position information associated with the first image. The target feature determination unit 132 usually determines one target feature, but may determine two or more target features.

[0062] The target condition is a condition for determining a target feature. The target condition is usually a condition related to the position information of the feature. The target condition is, for example, that the object has been present in the direction the user was facing for the longest time. The target condition is, for example, that the object has been present in the central region within a threshold range from the center point of the first image for the longest time. The target condition is, for example, that the object is present in the last first image in which all operations have been performed and is present in the central region within a threshold range from the center point of the first image. The target condition is, for example, that the object is present in the last first image in which all operations have been performed and is present in the direction the user was facing. Note that the direction the user was facing is, for example, the direction in which the focus point (e.g., the position of the cursor, the center point) in the first image is moving.

[0063] The category acquisition unit 133 acquires, from the feature management unit 111, the category that is the attribute value of the target feature determined by the target feature determination unit 132.

[0064] The image determination unit 134 determines one or more second images. For example, the image determination unit 134 acquires, from the image storage unit 112, one or more position images including ground features of the same category as the category acquired by the category acquisition unit 133. The ground features included in the position image are ground features in which the position specified by the position information of the ground feature is included in the region specified by the region information corresponding to the position image. Note that the position image acquired by the image determination unit 134 is a second image.

[0065] When the first image and the second image are different types of position images, the image determination unit 134 determines, for example, one or more second images including the same ground feature as the target ground feature in the first image.

[0066] The image determination unit 134 determines, for example, one or more second images including ground features of the same category as the category acquired by the category acquisition unit 133, and one or more second images associated with position information that satisfies the position condition with respect to the position information associated with the first image. The image determination unit 134 determines, for example, position information of ground features of the same category as the category acquired by the category acquisition unit 133, and one or more second images associated with position information that satisfies the position condition with respect to the position information associated with the first image. The position condition is, for example, that the proximity of the distance is in the top N or less (N is a natural number of 1 or more), or the distance is less than or equal to the threshold value or smaller than the threshold value.

[0067] The target ground feature determination unit 135 determines the target ground feature in each of the one or more second images determined by the image determination unit 134. The target ground feature is a ground feature of the same category as the category acquired by the category acquisition unit 133.

[0068] The operation unit 136 performs one or more operations that satisfy the operation sequence definition for the intention sequence acquired by the intention sequence acquisition unit 131 on each of the one or more second images determined by the image determination unit 134. For example, for each of the one or more second images, the operation unit 136 acquires an image type identifier, uses two or more pieces of corresponding information (which may also be referred to as a correspondence table) corresponding to the image type identifier to acquire an operation sequence definition, and performs one or more operations that satisfy the operation sequence definition on each of the one or more second images determined by the image determination unit 134.

[0069] The operation unit 136 performs, for example, one or more operations that satisfy the operation sequence definition for each of the one or more intention identifiers acquired by the intention sequence acquisition unit 131, on the target object. Performing an operation on the target object means performing an operation on the second image having the target object. The operation unit 136 uses, for example, the position information of the target object determined by the target object determination unit 135 to perform an operation on the target object while maintaining a positional relationship corresponding to the intention identified by the intention identifier. The operation unit 136 uses, for example, the position information of the target object to change the number of repetitions of the chunk identifier, or change the selection of the chunk identifier, or change the number of repetitions of the operation identifier, or change the selection of the operation identifier, while maintaining a positional relationship corresponding to the intention identified by the intention identifier, and performs an operation on the target object. Here, the position information of the target object is usually two or more pieces of position information, which is region information for specifying the region where the target object exists.

[0070] The operation unit 136 performs, for each of the one or more second images determined by the image determination unit 134, one or more operations that satisfy the operation sequence definition for each of the one or more intention identifiers acquired by the intention sequence acquisition unit 131.

[0071] When the operation unit 136 applies the intention to the target feature, there are cases where the operation sequence for specifying the user's operation on the first image cannot be directly applied due to differences in terrain between the first image and the second image. Therefore, the operation unit 136 needs to make adjustments based on the feature. For example, to avoid the situation where the operation unit 136 continues to perform operations such as "proceed" even when the road in the second image reaches a dead end, when the operation cannot be further performed in the second image, the operation unit 136 does not perform operations beyond the limit. Also, for example, in the case of the intention identifier "appearance" described later, since the distance traveled and the angle of turning at intersections may be different between the first image and the second image, the operation unit 136 modifies the operation sequence for the first image as a result and performs operations on the second image using the modified operation sequence. For example, when the intention identifier is "appearance" and the distances traveled are different, the operation unit 136 adds or deletes the operation identifier "proceed" so as to move across both ends of the target feature. Also, when the turning angles are different between the road in the first image and the road in the second image, the operation unit 136 adjusts the number and angle of the operations of "turn left" or "turn right" in the second image to be parallel to the road at the turning destination.

[0072] The output unit 14 outputs various types of information. The various types of information are, for example, the first image, one or more second images.

[0073] Here, output means, for example, transmission to the user's terminal device (not shown), but it is a concept including display on a display, projection using a projector, printing by a printer, transmission to other external devices, storage in a recording medium, delivery of the processing result to other processing devices and other programs, etc.

[0074] The first output unit 141 outputs the first image associated with the position information. The trigger for the first output unit 141 to output the first image is not limited. The first output unit 141 outputs the first image, for example, when an output instruction is received.

[0075] The second output unit 142 outputs a second image. The second output unit 142 outputs, for example, each of one or more second images determined by the image determination unit 134. Further, the second output unit 142 normally outputs a second image that changes according to an operation performed by the operation unit 136. Note that it may be considered that the output of the second image is performed by the operation unit 136.

[0076] The storage unit 11, the ground feature management unit 111, the image storage unit 112, and the correspondence management unit 113 are preferably a non-volatile recording medium, but can also be realized with a volatile recording medium.

[0077] The process of storing information in the storage unit 11 or the like is not limited. For example, information may be stored in the storage unit 11 or the like via a recording medium, information transmitted via a communication line or the like may be stored in the storage unit 11 or the like, or information input via an input device may be stored in the storage unit 11 or the like.

[0078] The reception unit 12 and the operation sequence reception unit 121 are preferably realized by wireless or wired communication means, but may also be realized by means for receiving broadcasts, device drivers of input means such as touch panels and keyboards, control software for menu screens, or the like.

[0079] The processing unit 13, the intention sequence acquisition unit 131, the target ground feature determination unit 132, the category acquisition unit 133, the image determination unit 134, the target object determination unit 135, and the operation unit 136 can usually be realized from a processor, a memory, or the like. The processing procedures of the processing unit 13 or the like are usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (a dedicated circuit). Note that the processor may be a CPU, an MPU, a GPU, or the like, and its type is not limited.

[0080] The output unit 14, the first output unit 141, and the second output unit 142 are usually realized by wireless or wired communication means, but may also be realized by driver software for output devices such as displays and speakers, or driver software for output devices and output devices.

[0081] Next, a first operation example of the image processing apparatus 1 will be described with reference to the flowchart of FIG. 2.

[0082] (S201) The reception unit 12 determines whether an output instruction has been received. If an output instruction has been received, the process proceeds to S202; if not, the process proceeds to S216.

[0083] (S202) The processing unit 13 acquires a first image corresponding to the output instruction received in S201.

[0084] (S203) The first output unit 141 outputs the first image acquired in S202.

[0085] (S204) The operation sequence reception unit 121 determines whether an operation on the output first image has been received. If an operation has been received, the process proceeds to S205; if not, the process proceeds to S208.

[0086] (S205) The operation unit 136 acquires the first image that has been changed according to the operation received in S204. The second output unit 142 outputs the first image.

[0087] (S206) The operation unit 136 acquires an operation identifier that identifies the operation received in S204, and temporarily stores the operation identifier in a buffer (not shown). For example, the operation identifiers for the operations of "move", "turn left", "turn right", "turn up", "turn down", "enlarge", and "reduce" are "m", "l", "r", "u", "d", "i", and "o", respectively.

[0088] (S207) The processing unit 13 temporarily stores the frame, which is the first image that has been changed after the operation, in a buffer (not shown). The process returns to S204. The frame is associated with one or more pieces of position information.

[0089] (S208) The reception unit 11 determines whether an operation end instruction has been received. If an operation end instruction has been received, the process proceeds to S209; if not, the process returns to S204.

[0090] (S209) The intention sequence acquisition unit 131 acquires an operation sequence, which is a set of one or more operation identifiers, from a buffer (not shown).

[0091] (S210) The intention sequence acquisition unit 131 acquires an intention sequence corresponding to the operation sequence acquired in S209. An example of such intention sequence acquisition processing will be described using the flowchart of FIG. 3 or FIG. 4.

[0092] (S211) The target object determination unit 132 determines a target object in the first image. An example of such target object determination processing will be described using the flowchart of FIG. 5.

[0093] (S212) The category acquisition unit 133 acquires the category from the ground object management unit 111 among the ground object information of the target object determined in S211.

[0094] (S213) The intention sequence acquisition unit 131 stores the intention sequence acquired in S209 and the category acquired in S212 in a buffer (not shown).

[0095] (S214) The image determination unit 134 determines one or two or more second images. An example of such image determination processing will be described using the flowchart of FIG. 6.

[0096] (S215) The operation unit 136 performs an operation according to the intention sequence stored in S213 on two or more images including one or more second images determined in S214. An example of such image operation processing will be described using the flowchart of FIG. 7.

[0097] (S216) The reception unit 12 determines whether corresponding information has been received. If the corresponding information has been received, it proceeds to S217; if not, it returns to S201.

[0098] (S217) The processing unit 13 stores the corresponding information received in S216 in the correspondence management unit 113. It returns to S201.

[0099] In the flowchart of FIG. 2, the process ends due to a power-off or an interrupt at the end of the process.

[0100] Next, a first example of the intention list acquisition process in S210 will be described using the flowchart of FIG. 3.

[0101] (S301) The intention list acquisition unit 131 assigns 1 to the counter i and the counter j.

[0102] (S302) The intention list acquisition unit 131 determines whether the i-th first correspondence information exists in the correspondence management unit 113. If the i-th first correspondence information exists, it proceeds to S303; if it does not exist, it proceeds to S311.

[0103] (S303) The intention list acquisition unit 131 acquires the operation sequence definition included in the i-th first correspondence information from the correspondence management unit 113.

[0104] (S304) The intention list acquisition unit 131 determines whether there is a partial operation sequence that matches the operation sequence definition acquired in S303 among the partial operation sequences that are a set of one or more consecutive operation identifiers starting from the j-th operation identifier. If such a partial operation sequence exists, it proceeds to S305; if it does not exist, it proceeds to S310.

[0105] (S305) The intention list acquisition unit 131 acquires the chunk identifier corresponding to the operation sequence definition acquired in S303 from the i-th first correspondence information, and temporarily stores the chunk identifier in a buffer (not shown).

[0106] (S306) The intention list acquisition unit 131 advances j to the next operation identifier after the partial operation sequence with the maximum length that matches the operation sequence definition in S304.

[0107] (S307) The intention list acquisition unit 131 determines whether j advanced in S306 is the position next to the last operation identifier of the operation sequence acquired in S209 (whether the operation sequence has ended). If the operation sequence has ended, it proceeds to S308; if it has not ended, it proceeds to S309.

[0108] (S308) The intention list acquisition unit 131 acquires a chunk sequence, which is one or more chunk identifiers, from a buffer (not shown).

[0109] (S309) The intention list acquisition unit 131 assigns 1 to the counter i. Return to S302.

[0110] (S310) The intention list acquisition unit 131 increments the counter i by 1. Return to S302.

[0111] (S311) The intention list acquisition unit 131 increments the counter j by 1. Return to S302.

[0112] (S312) The intention list acquisition unit 131 assigns 1 to the counter i and the counter j.

[0113] (S313) The intention list acquisition unit 131 determines whether the i-th second correspondence information exists in the correspondence management unit 113. If the i-th second correspondence information exists, go to S314; if it does not exist, go to S322.

[0114] (S314) The intention list acquisition unit 131 acquires the chunk sequence definition included in the i-th second correspondence information from the correspondence management unit 113.

[0115] (S315) The intention list acquisition unit 131 determines whether there is a partial chunk sequence that matches the chunk sequence definition acquired in S314 in a partial chunk sequence, which is a set of one or more consecutive chunk identifiers after the j-th chunk identifier. If the partial chunk sequence exists, go to S316; if it does not exist, go to S321.

[0116] (S316) The intention list acquisition unit 131 acquires the intention identifier paired with the chunk sequence definition acquired in S315 from the i-th second correspondence information, and temporarily stores the intention identifier in a buffer (not shown).

[0117] (S317) The intention list acquisition unit 131 advances j up to the next chunk identifier of the maximum-length partial chunk list that matches the chunk list definition in S315.

[0118] (S318) The intention list acquisition unit 131 determines whether the j advanced in S317 is at the position next to the last chunk identifier of the chunk list acquired in S308 (whether the chunk list has ended). If the chunk list has ended, it proceeds to S319; if not, it proceeds to S320.

[0119] (S319) The intention list acquisition unit 131 acquires an intention list that is one or more intention identifiers from a buffer (not shown). It returns to the upper-level process.

[0120] (S320) The intention list acquisition unit 131 assigns 1 to the counter i. It returns to S313.

[0121] (S321) The intention list acquisition unit 131 increments the counter i by 1. It returns to S313.

[0122] (S322) The intention list acquisition unit 131 increments the counter j by 1. It returns to S313.

[0123] Next, regarding the second example of the intention list acquisition process in S210, it will be described using the flowchart in FIG. 4.

[0124] (S401) The intention list acquisition unit 131 assigns 1 to the counter i and the counter j.

[0125] (S402) The intention list acquisition unit 131 determines whether the i-th corresponding information exists in the correspondence management unit 113. If the i-th corresponding information exists, it proceeds to S403; if not, it proceeds to S411.

[0126] (S403) The intention list acquisition unit 131 acquires the operation list definition that the i-th corresponding information has from the correspondence management unit 113.

[0127] (S404) The intention sequence acquisition unit 131 determines whether there is a partial operation sequence that matches the operation sequence definition acquired in S403 in a partial operation sequence that is a set of one or more consecutive operation identifiers after the j-th operation identifier. If the partial operation sequence exists, it proceeds to S405; if it does not exist, it proceeds to S410.

[0128] (S405) The intention sequence acquisition unit 131 acquires an intention identifier that pairs with the operation sequence definition acquired in S403 from the i-th corresponding information, and temporarily stores the intention identifier in a buffer (not shown).

[0129] (S406) The intention sequence acquisition unit 131 advances j to the operation identifier next to the partial operation sequence with the maximum length that matches the operation sequence definition in S404.

[0130] (S407) The intention sequence acquisition unit 131 determines whether j advanced in S406 is the position next to the last operation identifier of the operation sequence acquired in S209 (whether the operation sequence has ended). If the operation sequence has ended, it proceeds to S408; if it has not ended, it proceeds to S409.

[0131] (S408) The intention sequence acquisition unit 131 acquires an intention sequence that is one or more intention identifiers from the buffer (not shown). Return to the upper-level process.

[0132] (S409) The intention sequence acquisition unit 131 assigns 1 to the counter i. Return to S402.

[0133] (S410) The intention sequence acquisition unit 131 increments the counter i by 1. Return to S402.

[0134] (S411) The intention sequence acquisition unit 131 increments the counter j by 1. Return to S402.

[0135] Next, an example of the target feature determination process in S211 will be described using the flowchart of FIG. 5.

[0136] (S501) The target feature determination unit 132 assigns 1 to the counter i.

[0137] (S502) The target object determination unit 132 determines whether the i-th frame exists in the frames stored in a buffer (not shown in FIG. S207). If the i-th frame exists, it proceeds to S503; if not, it proceeds to S514.

[0138] (S503) The target object determination unit 132 acquires range information for specifying the range of the area photographed by the i-th frame. The target object determination unit 132 acquires the range information, for example, from the position information (position information of the representative point) associated with the i-th frame and the resolution of the frame. The target object determination unit 132 acquires the range information associated with the i-th frame, for example. The range information is information indicating the range of a three-dimensional space, for example, "(x1, y1, h1)(x2, y2, h2)".

[0139] (S504) The target object determination unit 132 acquires the target position information associated with the i-th frame. The target position information is information for specifying the position of the target point in the frame or the position information of the focus point, and is, for example, (x, y, h).

[0140] (S505) The target object determination unit 132 acquires the direction information corresponding to the i-th frame. The direction information is information indicating a direction, for example, an angle from true north and is a numerical value between 0 and 360. The direction information is information indicating a direction that can be obtained from the difference in the target position information of two consecutive frames.

[0141] (S506) The target object determination unit 132 substitutes 1 into the counter j.

[0142] (S507) The target object determination unit 132 determines whether the j-th object information exists in the object management unit 111. If the j-th object information exists, it proceeds to S508; if not, it proceeds to S513.

[0143] (S508) The target object determination unit 132 acquires the position information included in the j-th object information from the object management unit 111.

[0144] (S509) The target object determination unit 132 determines whether the j-th object exists in the area captured in the i-th frame by using the area information acquired in S503 and the position information acquired in S508. If the j-th object exists in the area of the i-th frame, the process proceeds to S510; if not, the process proceeds to S512.

[0145] (S510) The target object determination unit 132 determines whether the position information of the j-th object acquired in S508 meets the target conditions. If it meets the target conditions, the process proceeds to S511; if not, the process proceeds to S512.

[0146] (S511) The target object determination unit 132 increments by 1 the counter associated with the j-th object information. Note that the initial value of the counter associated with each object information is "0".

[0147] (S512) The target object determination unit 132 increments the counter j by 1. The process returns to S507.

[0148] (S513) The target object determination unit 132 increments the counter i by 1. The process returns to S502.

[0149] (S514) The target object determination unit 132 acquires the object information associated with the counter having the maximum value. The process returns to the upper-level process.

[0150] In the flowchart of FIG. 5, only one piece of object information is acquired, but two or more pieces of object information may be acquired. In such a case, for example, in S514, the target object determination unit 132 acquires the object information for which the value of the counter is equal to or greater than the threshold value or the top N (N is a natural number of 1 or more) counters.

[0151] Next, an example of the image determination process of S214 will be described with reference to the flowchart of FIG. 6.

[0152] (S601) The image determination unit 134 acquires the position information of the first image.

[0153] (S602) The image determination unit 134 assigns 1 to the counter i.

[0154] (S603) The image determination unit 134 determines whether there is an i-th candidate image that can be the second image. If the i-th candidate image exists, it proceeds to S604; if not, it proceeds to S613. Note that the i-th candidate image is, for example, a position image existing in the image storage unit 112 or an image server (not shown).

[0155] (S604) The image determination unit 134 acquires the region information for specifying the region photographed by the i-th candidate image.

[0156] (S605) The image determination unit 134 assigns 1 to the counter j.

[0157] (S606) The image determination unit 134 determines whether the j-th ground object information exists in the ground object management unit 111. If the j-th ground object information exists, it proceeds to S607; if not, it proceeds to S611.

[0158] (S607) The image determination unit 134 acquires the category of the j-th ground object information from the ground object information.

[0159] (S608) The image determination unit 134 determines whether the category acquired in S607 is the same as the category of the target ground object in the first image. If the two categories are the same, it proceeds to S609; if not, it proceeds to S612.

[0160] (S609) The image determination unit 134 uses the position information of the j-th ground object information and the position information of the target ground object to acquire the distance between the two ground objects, and associates the distance with the j-th ground object information.

[0161] (S610) The image determination unit 134 temporarily stores the identifier of the i-th candidate image in a buffer (not shown) in a pair with the distance.

[0162] (S611) The image determination unit 134 increments the counter i by 1. Return to S603.

[0163] (S612) The image determination unit 134 increments the counter j1. Return to S606.

[0164] (S613) The image determination unit 134 determines one or more candidate images whose distances paired with each of the one or more candidate images stored in S610 match the distance condition, and acquires one or more second images that are the one or more candidate images. Return to the upper-level process. Note that the distance condition is, for example, "the top three from the smaller distance" or "the distance is within or smaller than the threshold value".

[0165] Note that in the flowchart of FIG. 6, when the first image and the second image are different types of position images, the image determination unit 134 may acquire one or more second images including the same object as the object of interest in the first image.

[0166] Next, an example of the image operation process of S215 will be described using the flowchart of FIG. 7.

[0167] (S701) The operation unit 136 acquires the number of images for which operations using the acquired intention sequence are to be performed. Note that the images for which operations are to be performed are usually the one or more second images and the first image acquired in S613. That is, the number of images is usually "the number of images acquired in S613 + 1". However, the images for which operations are to be performed may be only the one or more second images acquired in S613.

[0168] (S702) The operation unit 136 generates windows for the number of images acquired in S701.

[0169] (S703) The operation unit 136 substitutes 1 for the counter i.

[0170] (S704) The operation unit 136 determines whether the i-th image of the operation target exists. If the i-th image exists, it proceeds to S705; if it does not exist, it proceeds to S707.

[0171] (S705) The operation unit 136 acquires the i-th image of the operation target and outputs the image to the i-th window.

[0172] (S706) The operation unit 136 increments the counter i by 1. It returns to S704.

[0173] (S707) The operation unit 136 assigns 1 to the counter j.

[0174] (S708) The operation unit 136 determines whether the j-th intention identifier exists in the acquired intention sequence. If the j-th intention identifier exists, it proceeds to S709; if it does not exist, it returns to the upper-level process.

[0175] (S709) The operation unit 136 refers to the correspondence management unit 113 and acquires the operation sequence definition corresponding to the j-th intention identifier.

[0176] (S710) The operation unit 136 assigns 1 to the counter k.

[0177] (S711) The operation unit 136 determines whether the k-th image of the operation target exists. If the k-th image exists, it proceeds to S712; if it does not exist, it proceeds to S714.

[0178] (S712) For the k-th image, the operation unit 136 performs the operation corresponding to the j-th intention identifier using the operation sequence definition acquired in S709. An example of such intention execution processing will be described using the flowchart of FIG. 8.

[0179] (S713) The operation unit 136 increments the counter k by 1. It returns to S711.

[0180] (S714) The operation unit 136 increments the counter j by 1. Return to S708.

[0181] Next, an example of the intention execution process of S712 will be described using the flowchart of FIG. 8.

[0182] (S801) The operation unit 136 acquires the region information of the target feature of the second image to be operated on.

[0183] (S802) The operation unit 136 assigns 1 to the counter i.

[0184] (S803) The operation unit 136 determines whether the i-th operation identifier in the operation sequence definition exists. If the i-th operation identifier exists, go to S804; if it does not exist, return to the upper-level process.

[0185] (S804) The operation unit 136 determines whether the i-th operation identifier is "move". If it is "move", go to S805; if it is not "move", go to S811.

[0186] (S805) The operation unit 136 places the focus point on the road in the image to be operated on. Note that the operation unit 136, for example, sets the focus point to the point on the road closest to the center position of the image to be operated on.

[0187] (S806) The operation unit 136 acquires the distance between the target feature and the focus point.

[0188] (S807) The operation unit 136 determines whether the distance acquired in S806 satisfies the condition. If it satisfies the condition, go to S808; if it does not satisfy the condition, go to S810. Here, the condition is, for example, "first threshold <= distance <= second threshold". That is, when moving, if the focus point gets too close to or too far from the target feature, it is judged inappropriate and the movement is stopped.

[0189] (S808) The operation unit 136 acquires an image with the focus point moved. Note that the amount of movement is, for example, a predetermined unit amount. The image with the focus point moved is the image advanced by the amount of movement.

[0190] (S809) The operation unit 136 increments the counter i by 1. Return to S803.

[0191] (S810) In the operation sequence definition, the operation unit 136 advances i until the operation identifier indicated by i becomes the operation identifier next to the "move" operation identifier. Return to S803.

[0192] (S811) The operation unit 136 determines whether the i-th operation identifier is "enlarge". If it is "enlarge", go to S812; if it is not "enlarge", go to S816.

[0193] (S812) The operation unit 136 acquires the position information of the target object and the position information of the focus point, and uses the two pieces of position information to acquire the distance between the target object and the focus point.

[0194] (S813) The operation unit 136 determines whether the distance acquired in S812 satisfies the condition. If it satisfies the condition, go to S814; if it does not satisfy the condition, go to S815. Note that the condition is, for example, "third threshold <= distance <= fourth threshold". That is, when enlarging (zooming in), if the focus point gets too close to or too far from the target object, it is determined to be inappropriate and the enlargement is stopped.

[0195] (S814) The operation unit 136 enlarges the image to be operated on. Go to S809. Note that the degree of enlargement is usually a predetermined unit amount.

[0196] (S815) In the operation sequence definition, the operation unit 136 advances i until the operation identifier indicated by i becomes the position of the operation identifier next to the "enlarge" operation identifier. Return to S803.

[0197] (S816) The operation unit 136 determines whether the i-th operation identifier is "zoom out". If it is "zoom out", it proceeds to S817; if it is not "zoom out", it proceeds to S821.

[0198] (S817) The operation unit 136 acquires the position information of the target feature and the position information of the focus point, and uses these two pieces of position information to acquire the distance between the target feature and the focus point.

[0199] (S818) The operation unit 136 determines whether the distance acquired in S817 satisfies the condition. If it satisfies the condition, it proceeds to S819; if it does not satisfy the condition, it proceeds to S820. The condition is, for example, "the fifth threshold <= distance <= the sixth threshold". That is, when zooming out, if the focus point gets too close to or too far from the target feature, it is determined to be inappropriate, and the zooming out is stopped.

[0200] (S819) The operation unit 136 reduces the image to be operated on. It proceeds to S809. The degree of reduction is usually a predetermined unit amount.

[0201] (S820) In the operation sequence definition, the operation unit 136 advances i until the position of the operation identifier indicated by i becomes the position of the operation identifier next to the "zoom out" operation identifier. It returns to S803.

[0202] (S821) The operation unit 136 executes the operation identified by the operation identifier. It returns to S809. Such operations are, for example, "turn left", "turn right", "turn up", or "turn down".

[0203] Next, a second operation example of the image processing apparatus 1 will be described with reference to the flowchart of FIG. 9. In the flowchart of FIG. 9, the description of the same steps as in the flowchart of FIG. 9 is omitted.

[0204] (S901) The intention list acquisition unit 131 determines whether an intention identifier can be acquired in S210. If an intention identifier can be acquired, it proceeds to S211; if not, it returns to S204. Here, in S210, usually, using an operation sequence that is one or two or more operation identifiers, a maximum of one intention identifier is acquired.

[0205] (S902) The operation unit 136 performs an operation corresponding to the intention identifier accumulated in S210 on one or more second images determined in S214. It returns to S201. An example of such image operation processing will be described using the flowchart of FIG. 10.

[0206] Note that in the flowchart of FIG. 9, while the user is operating the first image, one or more second images are automatically output, and according to the user's operation intention, each of the one or more second images is automatically operated.

[0207] Also, in the flowchart of FIG. 9, the process ends due to a power-off or a processing end interrupt.

[0208] Next, an example of the image operation processing in S902 will be described using the flowchart of FIG. 10.

[0209] (S1001) The second output unit 142 determines whether a window for the second image exists. If a window for the second image exists, it proceeds to S1004; if not, it proceeds to S1002.

[0210] (S1002) The second output unit 142 acquires the number of the acquired second images. The second output unit 142 generates windows for the number of the second images.

[0211] (S1003) The second output unit 142 outputs each of the one or more acquired second images to each window.

[0212] (S1004) The operation unit 136 refers to the correspondence management unit 113 and acquires an operation sequence definition corresponding to the intention identifier acquired in S210.

[0213] (S1005) The operation unit 136 substitutes 1 for the counter i.

[0214] (S1006) The operation unit 136 determines whether or not the i-th second image of the operation target exists. If the i-th second image exists, it proceeds to S1007; if it does not exist, it returns to the upper-level process.

[0215] (S1007) For the i-th second image, the operation unit 136 performs an operation corresponding to the intention identifier acquired in S210 using the operation sequence definition acquired in S1004. An example of such intention execution processing has been described using the flowchart of FIG. 8.

[0216] (S1008) The operation unit 136 increments the counter i by 1. It returns to S1006.

[0217] Hereinafter, a specific operation example of the image processing apparatus 1 in the present embodiment will be described.

[0218] FIG. 11 is a diagram for explaining an example of the concept of the image processing apparatus 1. In the image processing apparatus 1, one or more chunks are determined from one or more operations on the user's first image, an intention is determined from the chunk sequence, and an operation for reflecting the intention on the second image is performed. As described above, in the image processing apparatus 1, an intention may be determined from one or more operations on the user's first image, and an operation for reflecting the intention on the second image may be performed. That is, in the image processing apparatus 1, it is not essential to consider chunks. Hereinafter, two specific examples will be described. Specific example 1 is the case where the first image and the second image are the same type of position image. In specific example 1, the first image and the second image are street views. Specific example 2 is the case where the first image and the second image are different types of position images. In specific example 2, the first image is a street view and the second image is a Google earth image.

[0219] (Specific Example 1) The correspondence management unit 113 of the image processing apparatus 1 stores the first correspondence table in Fig. 12(b) and the second correspondence table in Fig. 12(c). Fig. 12(a) is the operation identifier in the street view.

[0220] The first correspondence table (Fig. 12(b)) has a plurality of records having "chunk", "chunk identifier", and "operation sequence definition". Here, the chunks are "move", "look left and right", "look up and down", and "zoom", and the respective chunk identifiers are "M", "P", "T", and "Z". Also, the operation sequence definition corresponding to each chunk identifier is a regular expression using one or more operation identifiers among the operation identifiers "m", "l", "r", "u", "d", "i", and "o". In the regular expression, [] indicates that any one is adopted, + indicates a positive closure (one or more times), and * indicates a closure (zero or more times).

[0221] The screen image in such a case is shown in Fig. 13. First, assume that the user inputs an output instruction for the street view including store A, which is an izakaya, to the image processing apparatus 1. Then, the image processing apparatus 1 outputs the street view 1302 including store A.

[0222] Next, assume that the user operates as "i", "o", "o", "u", "u", "u", "d", "m", "m", "r", "l", "l", "r", "m", "m", "m", "i", "i", "i", "o" as shown in Fig. 14 using the screen operation button 1301 in Fig. 13.

[0223] Then, the operation sequence reception unit 121 sequentially receives such operations. Next, the intention sequence acquisition unit 131 acquires the chunk identifier "Z" from the partial operation sequences "i", "o", "o", acquires the chunk identifier "T" from the partial operation sequences "u", "u", "u", "d", acquires the chunk identifier "M" from the partial operation sequences "m", "m", acquires the chunk identifier "P" from the partial operation sequences "r", "l", "l", "r", acquires the chunk identifier "M" from the partial operation sequences "m", "m", "m", and acquires the chunk identifier "Z" from the partial operation sequences "i", "i", "i", "o". That is, the intention sequence acquisition unit 131 refers to the first correspondence table (Fig. 12(b)) from the operation sequence "i", "o", "o", "u", "u", "u", "d", "m", "m", "r", "l", "l", "r", "m", "m", "m", "i", "i", "i", "o" and acquires the chunk sequence "Z", "T", "M", "P", "M", "Z".

[0224] Next, the intention sequence acquisition unit 131 acquires the intention identifier "detail" from the chunk sequence "Z", "T". The intention sequence acquisition unit 131 acquires the intention identifier "periphery" from the chunk sequence "M", "P", "M". That is, the intention sequence acquisition unit 131 refers to the second correspondence table (Fig. 12(c)) from the acquired chunk sequence and acquires the intention sequence "detail", "periphery".

[0225] Next, the target feature determination unit 132 determines the target feature "Store A" in the first image 1302 according to the above-described algorithm.

[0226] Next, the category acquisition unit 133 acquires the category "Izakaya" from the feature information of the target feature "Store A" from the feature management unit 111. Then, the intention sequence acquisition unit 131 stores the acquired intention sequence "detail", "periphery" and the category "Izakaya" in a buffer (not shown).

[0227] Next, the image determination unit 134 determines a street view that includes an izakaya and is a target feature of the category "izakaya" corresponding to the position information close to the position information of "Store A", and includes the features of the top three stores ("Store B", "Store C", "Store D") in terms of proximity to "Store A", and acquires the three street views from an image server (not shown). Such second images are 1303, 1304, and 1305 in FIG. 13.

[0228] Next, for each of the three street views (1303 to 1305), the operation unit 136 performs operations corresponding to the intention columns "Details" and "Surroundings" using the above-described algorithm.

[0229] As a result, the user can view an image in which the three street views are automatically operated, and can check the "Details" and "Surroundings" of "Store B", "Store C", and "Store D".

[0230] (Specific Example 2) A correspondence table shown in FIG. 15 is stored in the correspondence management unit 113 of the image processing apparatus 1. The correspondence table has a plurality of records having "intention identifier" and "operation column definition". This correspondence table is a correspondence table for the operation of Google earth.

[0231] In Specific Example 2, as a result of the user performing the same operation as in Specific Example 1, the image processing apparatus 1 outputs a street view 1302 including Store A, and the intention column acquisition unit 131 acquires the intention columns "Details" and "Surroundings". Further, the target feature determination unit 132 determines the target feature "Store A" in the first image 1302 by the above-described algorithm. The processing up to this point is the same as in Specific Example 1.

[0232] Next, the intention column acquisition unit 131 accumulates the acquired intention columns "Details" and "Surroundings" and the target feature "Store A" in a buffer (not shown).

[0233] Next, the image determination unit 134 inputs the target feature "Store A" to the Google earth image and outputs a Google earth image including "Store A". Such an output image is 1601 in FIG. 16.

[0234] Next, the operation unit 136 obtains an operation sequence definition "zoom in" that pairs with the intention identifier "detail" + from the correspondence table (FIG. 15). Further, the operation unit 136 obtains an operation sequence definition [zoom in|zoom out|left|right|up|down] that pairs with the intention identifier "periphery" + from the correspondence table (FIG. 15).

[0235] Then, the operation unit 136 repeats zooming in on the Google earth image 1601 according to the operation sequence definition "zoom in" + until the position information of the focus point in the Google earth image 1601 reaches a predetermined distance with respect to the position information of "Store A". Next, the operation unit 136 randomly selects one of the operations of zoom in, zoom out, left, right, up, or down according to the operation sequence definition [zoom in|zoom out|left|right|up|down] + based on the intention identifier "periphery", and performs the selected operation, presenting the "periphery" of "Store A" to the user using the Google earth image.

[0236] In a specific example, the correspondence table corresponding to various types of position images (first image, second image) may be one correspondence table as shown in FIG. 15, or two correspondence tables as shown in FIGS. 12(b) and (c). That is, the correspondence table may be composed of correspondence information, or may be composed of first correspondence information and second correspondence information.

[0237] As described above, according to the present embodiment, the intention of the operation on the first image can be applied to one or more second images, and as a result, one or more second images can be automatically operated according to the user's intention.

[0238] Also, according to the present embodiment, the intention of the operation on the first image can be applied to one or more second images of a type different from the first image, and as a result, one or more second images can be automatically operated according to the user's intention.

[0239] Also, according to this embodiment, the second image can be automatically determined.

[0240] Furthermore, according to this embodiment, the intention information for specifying the intention of the operation on the first image can be reused.

[0241] Note that the processing in this embodiment may be realized by software. And this software may be distributed by software download or the like. Also, this software may be recorded on a recording medium such as a CD-ROM and distributed. Note that this also applies to other embodiments in this specification. The software for realizing the image processing apparatus 1 in this embodiment is the following program. That is, this program is a computer that can access a correspondence management unit that stores two or more pieces of correspondence information indicating the correspondence between an operation sequence definition that defines a set of operation identifiers for each of two or more operations and an intention identifier that identifies the intention of the operation, a first output unit that outputs a first image that is an image associated with position information, an operation sequence reception unit that receives an operation sequence that is a set of operation identifiers for each of two or more operations on the first image, an intention sequence acquisition unit that acquires from the correspondence management unit an intention sequence that is an intention identifier corresponding to one or more operation sequence definitions that match the operation sequence received by the operation sequence reception unit, a second output unit that outputs a second image, and a program for causing an operation unit that performs one or more operations that satisfy the operation sequence definition for the intention sequence acquired by the intention sequence acquisition unit on the second image to function.

[0242] (Embodiment 2) In this embodiment, an image processing apparatus 2 that distributes the intention sequence obtained by operating on the first image and uses the intention sequence for operating on the second image will be described.

[0243] FIG. 17 is a block diagram of the image processing apparatus 2 in the present embodiment. The image processing apparatus 2 may be a stand-alone device or a server that receives instructions and information from a terminal device (not shown) and transmits a position image to the terminal device. When the image processing apparatus 2 is a stand-alone device, the image processing apparatus 2 is, for example, a so-called personal computer, a smartphone, or a tablet terminal, but its type is not limited. When the image processing apparatus 2 is a server, the image processing apparatus 2 is, for example, a cloud server or an ASP server, but its type is not limited.

[0244] The image processing apparatus 2 includes a storage unit 11, a reception unit 22, a processing unit 23, and an output unit 24. The reception unit 22 includes an intention reception unit 221. The processing unit 23 includes a target feature determination unit 132, a category acquisition unit 133, an image acquisition unit 234, a target object determination unit 135, and an operation unit 136. The output unit 24 includes a second output unit 142.

[0245] The reception unit 22 receives various instructions and information. The various instructions and information are, for example, intention information, automatic operation instructions, intention lists, etc., which will be described later.

[0246] The automatic operation instruction is an instruction to automatically operate the second image. The automatic operation instruction includes, for example, an identifier of the position image of the operation target. The position image of the operation target is the second image. The automatic operation instruction includes, for example, information for identifying an intention list.

[0247] The intention list, etc. may be only an intention list, or an intention list and a category, or an intention list and a target object identifier. The category is the category of the target object. The target object identifier is information for identifying the target object. The target object identifier is, for example, the name of the object to be operated or the ID of the object.

[0248] The intention reception unit 221 receives intention information. The intention information is information for specifying the user's intention with respect to the position image. The intention information includes an intention list. Such an intention list is information acquired in response to an operation on the first image. It is preferable that the intention information also includes an intention list and a category. The intention information may have an intention list and a target feature identifier.

[0249] The processing unit 23 performs various processes. The various processes are, for example, processes performed by the target feature determination unit 132, the category acquisition unit 133, the image acquisition unit 234, the target feature determination unit 135, and the operation unit 136.

[0250] The image acquisition unit 234 acquires a second image of the object to be automatically operated. The image acquisition unit 234 acquires the second image, for example, using the identifier of the position image included in the automatic operation instruction. The image acquisition unit 234 acquires the second image from, for example, the image storage unit 112 or an image server (not shown). Note that the types of the second image and the first image may be the same or different.

[0251] The output unit 24 outputs various information. The various information is, for example, the second image acquired by the image acquisition unit 234.

[0252] Here, the output is, for example, transmission to the user's terminal device (not shown), but includes concepts such as display on a display, projection using a projector, printing by a printer, transmission to other external devices, storage in a recording medium, and delivery of processing results to other processing devices and other programs.

[0253] The reception unit 22 and the intention reception unit 221 are preferably realized by wireless or wired communication means, but may also be realized by means for receiving broadcasts, device drivers of input means such as touch panels and keyboards, control software of a menu screen, and the like.

[0254] The processing unit 23 and the image acquisition unit 234 can usually be realized from a processor, a memory, or the like. The processing procedures of the processing unit 23 and the like are usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (a dedicated circuit). Note that the processor may be a CPU, an MPU, a GPU, or the like, and its type does not matter.

[0255] The output unit 24 is usually realized by wireless or wired communication means, but may also be realized by driver software for an output device such as a display or a speaker, or by driver software for an output device and an output device.

[0256] Next, an operation example of the image processing apparatus 2 will be described using the flowchart of FIG. 18.

[0257] (S1801) The reception unit 22 determines whether or not an automatic operation instruction has been received. If an automatic operation instruction has been received, the process proceeds to S1802, and if not, the process proceeds to S1806.

[0258] (S1802) The image acquisition unit 234 acquires a second image using the identifier of the position image included in the automatic operation instruction. Such a second image is a position image of the automatic operation target.

[0259] (S1803) The second output unit 142 outputs the second image acquired in S1802.

[0260] (S1804) The intention reception unit 221 acquires intention information. The intention reception unit 221 acquires, for example, the intention information stored in the storage unit 11. The intention reception unit 221 acquires, for example, the intention information included in the automatic operation instruction. The intention reception unit 221 acquires, for example, the intention information identified by the identifier of the intention information included in the automatic operation instruction.

[0261] (S1805) The operation unit 136 operates the second image output in S1803 using the intention information acquired in S1804. An example of such an image operation will be described using the flowchart of FIG. 19.

[0262] (S1806) The intention reception unit 221 determines whether it has received an intention column or the like. If it has received an intention column or the like, it proceeds to S1807; if not, it returns to S1801.

[0263] (S1807) The processing unit 23 stores the intention column or the like received in S1806 in the storage unit 11. It returns to S1801. Note that the intention column or the like is intention information. The intention column or the like is, for example, an intention column and a category, an intention column and a ground object identifier, or only an intention column.

[0264] Note that in the flowchart of FIG. 18, the process ends due to a power-off or a processing end interrupt.

[0265] Next, an example of the image operation in S1805 will be described using the flowchart of FIG. 19.

[0266] (S1901) The operation unit 136 assigns 1 to the counter i.

[0267] (S1902) The operation unit 136 determines whether the i-th intention identifier exists in the acquired intention column. If the i-th intention identifier exists, it proceeds to S1903; if not, it returns to the upper-level process.

[0268] (S1903) The operation unit 136 refers to the correspondence management unit 113 and acquires the operation column definition corresponding to the i-th intention identifier.

[0269] (S1904) The operation unit 136 performs an operation corresponding to the i-th intention identifier on the output second image using the operation column definition acquired in S1903. An example of such an intention execution process was described using the flowchart of FIG. 8.

[0270] (S1905) The operation unit 136 increments the counter i by 1. It returns to S1902.

[0271] Hereinafter, a specific operation example of the image processing apparatus 2 in the present embodiment will be described. Now, it is assumed that in the storage unit 11 of the image processing apparatus 2, an intention column "detail" "periphery" paired with the intention column identifier "intention X" is stored.

[0272] In such a situation, it is assumed that the user inputs an automatic operation instruction including the target feature "store A" and the intention column identifier "intention X" to the image processing apparatus 2.

[0273] Then, the reception unit 22 receives the automatic operation instruction "<target feature> store A <intention column identifier> intention X".

[0274] Next, the image acquisition unit 234 uses "<target feature> store A" included in the automatic operation instruction to acquire a street view including "store A". Also, the category acquisition unit 133 acquires the category "izakaya" of "store A". Next, the target feature determination unit 135 determines a predetermined number (here, three) of features "store B", "store C", and "store D" that are features of the category "izakaya" and are close to "store A" in distance. Next, the image acquisition unit 234 acquires three street views including the features "store B", "store C", and "store D" from an image server (not shown).

[0275] Next, the output unit 24 outputs four street views each including "store A", "store B", "store C", and "store D".

[0276] Next, the operation unit 136 applies the intention columns "detail" "periphery" paired with the intention column identifier "intention X" to the four street views with "store A", "store B", "store C", and "store D" as target features respectively, and automatically operates the four street views. Such an operation was described in the specific example of Embodiment 1.

[0277] Then, the output unit 24 outputs street views to each of the windows 1302 to 1305 in FIG. 13, for example.

[0278] As described above, according to the present embodiment, the intention information for specifying the intention of the operation on the first image can be reused. That is, according to the present embodiment, the intention of the operation on the first image can be diverted to each of one or more second images, and as a result, one or more second images can be automatically operated according to the user's intention.

[0279] Furthermore, according to the present embodiment, the second image can be automatically determined.

[0280] Note that the software for realizing the image processing apparatus 2 in the present embodiment is the following program. That is, this program is a computer that can access a correspondence management unit that stores two or more pieces of correspondence information indicating the correspondence between operation sequence definitions and intention identifiers, and receives intention information including an intention sequence acquired according to an operation on a first image. An intention reception unit, a second output unit that outputs a second image, and an operation sequence definition for each of one or more intention identifiers included in the intention sequence included in the intention information received by the intention reception unit is obtained, and one or more operations that satisfy the one or more operation sequence definitions are performed on the second image. It is a program for functioning as an operation unit.

[0281] Also, FIG. 20 shows the appearance of a computer that executes the program described in this specification and realizes the image processing apparatus 1 or the image processing apparatus 2 of the various embodiments described above. The above-described embodiments can be realized by computer hardware and a computer program executed thereon. FIG. 20 is an overview diagram of this computer system 300, and FIG. 21 is a block diagram of the system 300.

[0282] In FIG. 20, the computer system 300 includes a computer 301 including a CD-ROM drive, a keyboard 302, a mouse 303, and a monitor 304.

[0283] In FIG. 21, in addition to the CD-ROM drive 3012, the computer 301 includes an MPU 3013, a bus 3014 connected to the CD-ROM drive 3012 and the like, a ROM 3015 for storing programs such as a boot-up program, a RAM 3016 connected to the MPU 3013 for temporarily storing instructions of an application program and providing a temporary storage space, and a hard disk 3017 for storing an application program, a system program, and data. Here, although not shown, the computer 301 may further include a network card for providing connection to a LAN.

[0284] A program for causing the computer system 300 to execute functions such as the image processing apparatus 1 of the above-described embodiment may be stored in a CD-ROM 3101, inserted into the CD-ROM drive 3012, and further transferred to the hard disk 3017. Alternatively, the program may be transmitted to the computer 301 via a network (not shown) and stored in the hard disk 3017. The program is loaded into the RAM 3016 during execution. The program may be loaded directly from the CD-ROM 3101 or the network.

[0285] The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 301 to execute functions such as the image processing apparatus 1 of the above-described embodiment. The program only needs to include a portion of instructions that call appropriate functions (modules) in a controlled manner so as to obtain a desired result. How the computer system 300 operates is well known, and a detailed description thereof is omitted.

[0286] Note that in the above program, in steps such as the step of transmitting information and the step of receiving information, processing performed by hardware, for example, processing performed by a modem or an interface card in the transmission step (processing that can only be performed by hardware) is not included.

[0287] Also, the computer that executes the above program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

[0288] Also, in each of the above embodiments, it goes without saying that two or more communication means existing in one device may be physically realized by one medium.

[0289] Also, in each of the above embodiments, each process may be realized by being centrally processed by a single device, or may be realized by being distributedly processed by a plurality of devices.

[0290] Needless to say, the present invention is not limited to the above embodiments, and various modifications are possible and are also included within the scope of the present invention.

Industrial Applicability

[0291] As described above, the image processing apparatus 1 according to the present invention has an effect that the intention of the operation on the first image can be diverted to the second image, and as a result, the second image can be automatically operated according to the user's intention, and is useful as a viewer of a position image or the like.

Explanation of Signs

[0292] 1, 2 Image processing apparatus 11 Storage unit 12, 22 Reception unit 13, 23 Processing unit 14, 24 Output unit 111 Feature management unit 112 Image storage unit 113 Correspondence management unit 121 Operation sequence reception unit 131 Intention sequence acquisition unit 132 Target feature determination unit 133 Category acquisition unit 134 Image determination unit 135 Target object determination unit 136 Operation unit 141 First output unit 142 Second output unit 221 Intention reception unit 234 Image acquisition unit

Claims

1. A correspondence management unit that stores two or more pieces of correspondence information indicating the correspondence between an operation sequence definition that defines a set of operation identifiers for each of two or more operations and an intention identifier that identifies the intention of the operation; A first output unit that outputs a first image that is an image associated with position information; An operation sequence reception unit that receives an operation sequence that is a set of operation identifiers for each of two or more operations on the first image; An intention sequence acquisition unit that acquires, from the correspondence management unit, an intention sequence that is an intention identifier corresponding to one or more operation sequence definitions that match the operation sequence received by the operation sequence reception unit; A second output unit that outputs a second image; An image processing apparatus comprising: an operation unit that performs one or more operations that satisfy an operation sequence definition on the second image with respect to the intention sequence acquired by the intention sequence acquisition unit.

2. In the correspondence management unit, Two or more pieces of first correspondence information indicating the correspondence between an operation sequence definition and a chunk identifier that identifies a chunk that is a block of operations, and two or more pieces of second correspondence information indicating the correspondence between a chunk sequence definition that is a definition of a chunk sequence that is a set of one or more chunk identifiers and an intention identifier are stored, The intention sequence acquisition unit, Acquires, from the two or more pieces of first correspondence information, one or more chunk sequences that are a set of one or more chunk identifiers corresponding to one or more operation sequence definitions that match the operation sequence received by the operation sequence reception unit, and acquires, from the two or more pieces of second correspondence information, one or more intention identifiers corresponding to a chunk sequence definition that matches the one or more chunk sequences. The image processing apparatus according to claim 1.

3. Referring to a feature management unit that stores two or more pieces of feature information having a feature identifier that identifies a feature, and two or more feature attribute values including a category of the feature and position information of the feature, and using the position information associated with the first image, a feature of interest determination unit that determines a feature of interest that is a feature in the first image and that matches a condition of interest; A category acquisition unit that acquires the category of the feature of interest from the feature management unit; A target object determination unit that determines a target object, which is an object in the same category as the category acquired by the category acquisition unit and is an object in the second image. The operation unit The image processing apparatus according to claim 2, wherein one or more operations that satisfy the operation sequence definition for each of the one or more intention identifiers acquired by the intention sequence acquisition unit are performed on the target object.

4. The operation unit The image processing apparatus according to claim 3, wherein the operation unit uses the position information of the target object to perform an operation on the target object so as to maintain a positional relationship corresponding to the intention identified by the intention identifier.

5. The operation sequence definition is one or more operation identifiers using a regular expression. The chunk sequence definition is one or more chunk identifiers using a regular expression. The operation unit The image processing apparatus according to claim 4, wherein the operation unit uses the position information of the target object to change the number of repetitions of the chunk identifier, or change the selection of the chunk identifier, or change the number of repetitions of the operation identifier, or change the selection of the operation identifier so as to maintain a positional relationship corresponding to the intention identified by the intention identifier, and performs an operation on the target object.

6. An image determination unit that acquires a position image including an object in the same category as the category acquired by the category acquisition unit from an image storage unit in which two or more position images associated with one or more pieces of position information are stored. The second image is the position image acquired by the image determination unit, and the image processing apparatus according to claim 3.

7. The image determination unit One or more second images including an object in the same category as the category acquired by the category acquisition unit, and determining one or more second images associated with position information that satisfies a position condition with respect to the position information associated with the first image. The operation unit The image processing apparatus according to claim 6, wherein one or more operations that satisfy the operation sequence definition for each of the one or more intention identifiers acquired by the intention sequence acquisition unit are performed on each of the one or more second images determined by the image determination unit.

8. A correspondence management unit that stores two or more pieces of correspondence information indicating the correspondence between the operation sequence definition and the intention identifier; An intention reception unit that receives intention information including an intention sequence acquired in response to an operation on a first image; A second output unit that outputs a second image; An operation unit that acquires, from the correspondence management unit, an operation sequence definition corresponding to each of the one or more intention identifiers included in the intention sequence included in the intention information received by the intention reception unit, and performs one or more operations that satisfy each of the one or more operation sequence definitions on the second image. An image processing apparatus comprising:

9. The intention information further includes a category of a target feature, The image processing apparatus further comprises a target feature determination unit that determines a target feature that is a feature of the same category as the category included in the intention information and is a feature in the second image. The operation unit: The image processing apparatus according to claim 8, wherein an operation sequence definition for the intention sequence included in the intention information is acquired, and one or more operations that satisfy each of the one or more operation sequence definitions are performed on the target feature.

10. The image processing apparatus according to any one of claims 1 to 9, wherein the first image and the second image are different types of position images.

11. The image processing apparatus according to claim 1, wherein the first image and the second image are a street view, a three-dimensional image, an indoor image, or a drive recorder image.

12. An image processing method realized by a correspondence management unit that stores two or more pieces of correspondence information indicating the correspondence between an operation sequence definition that defines a set of operation identifiers for each of two or more operations and an intention identifier that identifies the intention of the operation, a first output unit, an operation sequence reception unit, an intention sequence acquisition unit, a second output unit, and an operation unit, A first output step in which the first output unit outputs a first image that is an image associated with position information; An operation sequence reception step in which the operation sequence reception unit receives an operation sequence that is a set of operation identifiers for identifying two or more operations on the first image; An intention sequence acquisition step in which the intention sequence acquisition unit acquires, from the correspondence management unit, an intention sequence that is a set of intention identifiers corresponding to one or more operation sequence definitions that match the operation sequence received by the operation sequence reception unit; A second output step in which the second output unit outputs a second image; An image processing method comprising: an operation step in which the operation unit performs one or more operations on the second image that satisfy an operation sequence definition for the intention sequence acquired by the intention sequence acquisition unit.

13. An image processing method realized by a correspondence management unit that stores two or more pieces of correspondence information indicating the correspondence between an operation sequence definition and an intention identifier, an intention reception unit, a second output unit, and an operation unit, comprising: An intention reception step in which the intention reception unit receives intention information including an intention sequence acquired in response to an operation on a first image; A second output step in which the second output unit outputs a second image; An operation step in which the operation unit acquires, from the correspondence management unit, an operation sequence definition corresponding to one or more intention identifiers included in the intention sequence included in the intention information received by the intention reception unit, and performs one or more operations on the second image that satisfy the one or more operation sequence definitions.

14. A computer accessible to a correspondence management unit that stores two or more pieces of correspondence information indicating the correspondence between an operation sequence definition that defines a set of operation identifiers for two or more operations and an intention identifier that identifies the intention of an operation, A first output unit that outputs a first image that is an image associated with position information; An operation sequence reception unit that receives an operation sequence that is a set of operation identifiers for identifying two or more operations on the first image; An intent list acquisition unit that acquires, from the correspondence management unit, an intent list that is one or more intent identifiers corresponding to each operation list definition that matches the operation list received by the operation list reception unit; A second output unit that outputs a second image; A program for causing an operation unit to function to perform one or more operations that satisfy the operation list definition for the intent list acquired by the intent list acquisition unit on the second image.

15. A computer that can access a correspondence management unit that stores two or more pieces of correspondence information indicating the correspondence between an operation list definition and an intent identifier; An intent reception unit that receives intent information including an intent list acquired according to an operation on a first image; A second output unit that outputs a second image; A program for acquiring, from the correspondence management unit, an operation list definition corresponding to one or more intent identifiers included in the intent list included in the intent information received by the intent reception unit, and causing an operation unit to function to perform one or more operations that satisfy the one or more operation list definitions on the second image.

Images