Image processing device and image processing method

Abstract

To provide an image processing device and an image processing method capable of retrieving images on an object captured from a specific angle even when the object to be retrieved has periodic image features.SOLUTION: The image processing device acquires information on an imaging apparatus when capturing an image with respect to individual image data constituting a group of image data to be retrieved. The information to be acquired includes at least information on a posture. On the basis of the information on the posture, the image processing device causes a display device to display a selection screen that allows a user to select a posture of an imaging apparatus. Further, the image processing device retrieves the image data on the basis of the posture selected on the selection screen from the group of image data to be retrieved.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method, and more particularly to an image search technique. [Background technology]

[0002] For example, when the same scene or the same object is photographed from multiple viewpoints, a large number of similar still images may be captured, or a video composed of similar frame images may be captured.

[0003] Patent Document 1 discloses a person search function for a surveillance system that handles images taken at various locations. In Patent Document 1, search results for images of the person being searched for are presented in a three-dimensional layout on coordinates determined by three criteria: the difference (distance) between the shooting locations, the similarity of the images, and the difference in shooting time. This allows the user to visually grasp the relationship between the image used in the search and each individual image obtained by the search on multiple criteria. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-160733 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the method described in Patent Document 1, for example, when the object to be searched has periodic image features, it is difficult to search for an image of the object photographed from a specific angle.

[0006] The present invention has been made in view of the problems of the conventional art. In one aspect, the present invention provides an image processing device and an image processing method that are capable of searching for images of an object photographed from a specific angle, even if the object to be searched has periodic image features. [Means for solving the problem]

[0007] The object of the present invention is to provide an image processing system that acquires information about an image capturing device at the time of capturing an image of each piece of image data that constitutes a group of image data to be searched, and the information includes at least information about the posture of the image capturing device at the time of capturing the image. and location information posture information, including and location information Based on this, the user is position a display control means for displaying a selection screen for selecting a posture on a display device; and a display control means for selecting a posture from the image data group to be searched, the image data selected on the selection screen. position A search means for searching image data based on posture. The selection screen includes a plurality of selectable icons superimposed on a background image obtained based on image data other than the image data constituting the image data group, at positions corresponding to the position and orientation of the imaging device. The object is achieved by an image processing device characterized by the above. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device and an image processing method that are capable of searching for images of an object photographed from a specific angle, even if the object to be searched has periodic image features. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera 100 as an example of an image processing apparatus according to an embodiment. [Figure 2] Flowchart regarding the operation of the digital camera 100 [Figure 3] Flowchart regarding the operation of the digital camera 100 [Figure 4] FIG. 1 shows an example of a GUI presented by the digital camera 100. [Figure 5] Flowchart regarding the operation of the digital camera 100 [Figure 6] FIG. 1 shows an example of a GUI presented by the digital camera 100. [Figure 7] FIG. 1 is a diagram for explaining a position and orientation calculation method according to an embodiment; [Figure 8] Flowchart regarding the operation of the digital camera 100 [Figure 9] 1A is a diagram showing an example of a GUI presented by the digital camera 100, and FIG. 1B is a flowchart showing the operation of the digital camera 100. [Figure 10] FIG. 1 is a diagram for explaining a method for generating a background image in an embodiment. [Figure 11] 1A is a diagram showing an example of a GUI presented by the digital camera 100, and FIG. 1B is a diagram for explaining camera icons. [Figure 12] FIG. 10 is a diagram for explaining a position and orientation selection process according to an embodiment. [Figure 13] FIG. 1 shows an example of a GUI presented by the digital camera 100. [Figure 14] FIG. 1 shows an example of a GUI presented by the digital camera 100. [Figure 15] FIG. 1 shows an example of a GUI presented by the digital camera 100. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following embodiments, the present invention will be described with reference to a digital camera. However, the present invention can be implemented in any electronic device. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] ●(First embodiment) Fig. 1(a) is a block diagram showing an example of the configuration of a digital camera 100 as an example of an image processing device according to the present invention. Fig. 1(b) and (c) are diagrams showing examples of the external appearance of the digital camera 100 as seen from the front and back, respectively.

[0013] The digital camera 100 includes an imaging unit 101, a calculation unit 102, a storage unit 103, a shutter button 104, an operation unit 105, a display unit 106, and a control unit 107. The imaging unit 101 includes a lens 101a and an image sensor 101b.

[0014] The lens 101a forms an optical image of a subject on the imaging plane of the image sensor 101b. The image sensor 101b has a pixel array in which pixels each having a photoelectric conversion element are arranged two-dimensionally. The image sensor 101b photoelectrically converts the optical image using the pixel array and outputs a pixel signal group (analog image signal) to the calculation unit 102.

[0015] When the digital camera 100 detects that the shutter button 104 has been pressed, it captures and records a still image. The calculation unit 102 applies image processing, also known as A / D conversion processing and development processing, to the analog image signal output by the image sensor 101b to generate a digital image signal (image data). The calculation unit 102 also calculates position and orientation information of the digital camera 100 at the time of image capture, for example, from the outputs of a position sensor and orientation sensor (not shown) that the digital camera 100 has.

[0016] The position of the digital camera 100 can be calculated using known technology, such as GPS, WiFi, beacons, visible light, ultrasonic waves, BLE, RFID, IMES, UWB, and Quuppa. The digital camera 100 is assumed to have a built-in or attached receiver according to the positioning method. The orientation sensor may be, for example, a gyro sensor. Note that the position and orientation information of the digital camera 100 may be acquired from an external device communicably connected to the digital camera 100.

[0017] The calculation unit 102 applies processing to the image data according to its intended use, such as recording or display. The image data to be recorded and the position and orientation information are stored in a data file in a predetermined format and stored in the storage unit 103. The storage unit 103 may be a removable device such as an SD card, or may be provided inside the digital camera 100.

[0018] The operation unit 105 is a general term for input devices (buttons, switches, dials, touch panel, etc.) provided in the digital camera 100 that can be operated by the user. The shutter button 104 also constitutes the operation unit 105, but for convenience it is shown as a separate component. The operation unit 105 has directional keys, a decision button, a menu button, etc. Search conditions for recorded images can also be entered using the operation unit.

[0019] The display unit 106 is provided on the back surface of the digital camera 100 (the surface facing the user when taking a picture). The display unit 106 is, for example, an LCD, and displays live view images, recorded images, menu screens, etc. If the display unit 106 is a touch display having a touch panel, the touch panel can be used as the shutter button 104 and the operation unit 105.

[0020] Although the digital camera 100 has been described in terms of still image capture, the digital camera 100 is also capable of video capture. Video capture is performed in response to the operation of a video capture button provided separately from the shutter button 104. Video is captured and recorded through operations similar to those for still image capture, except that capture is repeated at a predetermined frame rate, the resolution (number of pixels) of each frame is generally lower than that of a single still image frame, and the encoding method may differ from that of a still image.

[0021] The control unit 107 has one or more processors (CPUs) capable of executing programs, a ROM, and a RAM. The control unit 107, for example, loads programs stored in the ROM into the RAM, executes the programs, and controls the operation of each functional block, thereby realizing the functions of the digital camera 100. The ROM is, for example, a rewritable non-volatile memory, and stores programs executable by the CPU of the control unit 107, setting values, GUI data, and the like. The RAM is used to load programs executed by the CPU of the control unit 107 and to store values ​​required during program execution. The RAM may also be used as a buffer memory for image data or as a video memory for the display unit 106.

[0022] FIG. 2 is a flowchart showing the operation of the digital camera 100 to take a photograph and to search for an image obtained by taking a photograph. In S201, the control unit 107 acquires image data of the same scene or the same object photographed from different viewpoints. Here, the images may be multiple still images or a video including multiple frame images photographed from different viewpoints. The image data may be acquired by photographing with the digital camera 100 when S201 is executed, or may be acquired by reading image data obtained in a previous photograph from the storage unit 103. As described above, each still image data and frame image is assumed to be associated with position and orientation information of the digital camera 100 at the time of photographing.

[0023] In S202, the control unit 107 switches the operation mode to the search mode. The search mode is one of the operation modes of the digital camera 100. The user can switch the operation mode by, for example, using a mode switching dial included in the operation unit 105 or by operating the GUI displayed on the display unit 106 via the operation unit 105.

[0024] FIG. 3 is a flowchart showing an example of the operation of the control unit 107 in S202. In S2021, the control unit 107 detects that an instruction to switch to search mode has been given via the operation unit 105. Fig. 4(a) shows an example of a setting GUI 401 for instructing the display unit 106 to start and end the search mode. The setting GUI 401 is displayed on the display unit 106 through operation of a hierarchical menu screen via the operation unit 105, for example.

[0025] The setting GUI 401 allows the selection of start and end. The selection operation may be a combination of a focus movement operation and a confirmation operation via the operation unit 105, or a touch operation such as tapping. When the selection operation of "start" is detected, the control unit 107 switches the operation mode of the digital camera 100 to search mode. Furthermore, when the selection operation of "end" is detected, the control unit 107 switches the operation mode of the digital camera 100 to the operation mode before transitioning to search mode (for example, shooting mode or playback mode).

[0026] In S2022, the control unit 107 prompts the user to select an image data file to be searched from among the image data files stored in the storage unit 103. For example, as shown in FIG. 4(b), the control unit 107 causes the display unit 106 to display a selection screen that displays a list of selectable thumbnail images of image data files (still image data, images with a lower resolution (fewer pixels) than one frame of video data). The selection screen may display a message 502 that prompts the user to select an image (scene) to be selected. The user scrolls the selection screen as necessary and selects an image that captures the scene to be searched. FIG. 4(b) shows that the user has selected image 503 as the scene to be searched by touching the display unit 106.

[0027] The control unit 107 may generate thumbnail images of the video data from a representative frame (for example, the first frame), or may generate thumbnail images for each scene if there is a scene change.

[0028] In S2023, when a selection operation on the selection screen is detected, the control unit 107 determines whether or not a thumbnail image of video data has been selected on the selection screen. If it is determined that a thumbnail image of video data has been selected, the control unit 107 executes S2024, and if not, executes S2025.

[0029] In S2024, the control unit 107 breaks down the video data corresponding to the selected thumbnail image into image data for each frame (frame image data). If the video data contains multiple scenes, it is sufficient to break down only the scene corresponding to the selected thumbnail image frame by frame. The control unit 107 handles the obtained group of frame image data as a group of image data to be searched.

[0030] In S2025, the control unit 107 identifies still image data from the still image data stored in the storage unit 103 that captures the same scene as the thumbnail image selected on the selection screen. There are no particular limitations on the method for identifying still image data that captures the same scene, and any known method can be used. For example, still image data that captures the same scene can be identified based on one or more of the following conditions: the difference in the shooting dates and times is within a predetermined time, the similarity of the images is equal to or greater than a threshold, and the difference in location information recorded in the image data is within a predetermined distance. The control unit 107 treats the identified still image data group as an image data group to be searched.

[0031] Returning to Fig. 2, in S203, the control unit 107 as an acquisition unit acquires the position and orientation of the digital camera 100 at the time of shooting for the group of image data to be searched identified in S202. Here, a case where both the position and orientation of the digital camera 100 are acquired will be described, but in cases where the subject distance is essentially infinite, such as in astronomical photography, the position has little meaning and so only the orientation may be acquired. In this way, the control unit 107 acquires at least the information related to the orientation of the information related to the position and orientation of the digital camera 100.

[0032] FIG. 5 is a flowchart showing an example of the operation of the control unit 107 in S203. In S2031, the control unit 107 displays a setting GUI for allowing the user to select a method for obtaining the position and orientation on the display unit 106. An example of the setting GUI is shown in FIG.

[0033] The setting GUI 701 allows a selection operation of "read" (acquisition by reading from image data) and "calculation" (acquisition by calculating on the spot). When the control unit 107 detects a selection operation of "read", it executes S2032, and when it detects a selection operation of "calculation", it executes S2033. Note that here, the user is allowed to select the method of acquiring the position and orientation when executing S203, but it is not essential that the user be allowed to make a selection. For example, if position and orientation information is recorded in the image data group to be searched, the position and orientation information may be read from the image data, and if position and orientation information is not recorded in the image data group to be searched, the position and orientation may be calculated.

[0034] In S2032, the control unit 107 reads out the position and orientation information from each data file storing the image data to be searched, and stores it in the RAM.

[0035] In S2033, the control unit 107 calculates the position and orientation of each piece of image data to be searched and stores the calculated position and orientation in RAM. Calculating the position and orientation of the device that captured the images from multiple images can be achieved using known position and orientation estimation techniques such as SfM (Structure from Motion) and SLAM (Simultaneous Localization and Mapping).

[0036] When using SfM or SLAM, the position and orientation of the digital camera 100 is calculated from the relationship between the subject and the composition of multiple images. Therefore, even if the main subject does not have clear image features or has periodic image features, the position and orientation can be calculated with high accuracy using image features of other subjects that are common to multiple images. Furthermore, because the position and orientation relative to the main subject can be calculated, a more precise position can be calculated than with GPS.

[0037] 7 schematically shows the principle of position and orientation calculation using multiple images. Assume that the same main subject 800 is photographed with digital camera 100 at different positions and orientations 801 and 802. Assume that control unit 107 detects feature point 805 of main subject 800 as feature point 806 in image 803 photographed at position and orientation 801, and as feature point 807 in image 804 photographed at position and orientation 802.

[0038] In this case, the control unit 107 calculates positions and orientations 801 and 802 such that the three-dimensional coordinates of feature point 805 become feature point 806 in image 803 and feature point 807 in image 804. The optical parameters of the digital camera 100 used in this calculation (such as the focal length of the lens 101a and information regarding distortion) can be stored in advance in the storage unit 103 or the ROM of the control unit 107, for example.

[0039] Returning to FIG. 2, in S205, the control unit 107 creates a background image for superimposing a camera icon. A camera icon is a two-dimensional image that represents the position and orientation of the digital camera 100 (details will be described later). If one of the image data sets to be searched is used as the background image for a camera icon, the camera icons that can be superimposed are limited. For example, a camera icon related to the image data used as the background image cannot be superimposed. Therefore, a background image is generated so that the camera icons can be superimposed without any deficiencies.

[0040] FIG. 8 is a flowchart showing an example of the operation of the control unit 107 in S204. In S2041, the control unit 107 displays a setting GUI for allowing the user to select whether or not to automatically generate a background image on the display unit 106. An example of the setting GUI is shown in Fig. 9(a).

[0041] The setting GUI 1001 allows the selection of "auto-generation" (generating a background image without taking a photo) and "photography" (taking a photo of a background image). The control unit 107 executes S2042 when it detects the selection of "auto-generation," and executes S2043 when it detects the selection of "photography." Note that, although the user is allowed to select the method for generating the background image here, it is not essential that the user be allowed to make a selection; for example, the background image may always be generated automatically.

[0042] In S2024, the control unit 107 generates a background image using, for example, one piece of still image data from the image data group to be searched. FIG. 9(b) schematically shows an example of a method for automatically generating a background image. The control unit 107 adds a margin 1102 around an image 1101 corresponding to one piece of still image data from the image data group to be searched, and generates a background image 1103 with a pseudo-widened angle of view. This increases the number of camera icons that can be superimposed, and also allows the camera icon corresponding to the image 1101 itself to be superimposed. Note that adding a margin around the image is one example of a method for generating a background image, and other methods may be used to generate a background image.

[0043] 10 is a diagram showing the generation of a background image with a pseudo-enlarged angle of view by generating a panoramic image from multiple still image data. Images 1201 and 1202 are two images that contain a common main subject from the image data group to be searched. By stitching these two images using the corresponding positions of the common subject, a panoramic image 1203 with a pseudo-enlarged angle of view in the horizontal direction can be generated.

[0044] For example, based on the position and orientation acquired in S203, images 1201 and 1202 can be selected as two images that contain a common subject from multiple images captured with little change in position and at different horizontal angles. Note that a panoramic image may also be generated from three or more images.

[0045] Alternatively, a 3D image may be generated as the background image. The 3D image can use the 3D points of the subject acquired when calculating the position and orientation in S2033. On the other hand, if the position and orientation is acquired from a data file, the 3D points of the subject are not calculated. In this case, the process of acquiring the 3D points of the subject, which is part of the process of S2033, can be executed within the process of S205 for generating the background image. A 3D image has the advantage that the angle of view and composition can be freely changed. Note that the generation of the background image may be executed by the calculation unit 102 rather than the control unit 107.

[0046] In S2043, the control unit 107 prompts the user to capture a background image. FIG. 11A shows an example of the display on the screen 106 when the user is prompted to capture a background image. It is considered that capturing a background image is selected when a subject similar to the main subject included in the image data group to be searched can be captured. Therefore, the control unit 107 temporarily changes the operating mode of the digital camera 100 to a capture mode and causes the display unit 106 to display a live view image.

[0047] The control unit 107 then uses the calculation unit 102 to detect the subject area in the live view image and instructs the user to adjust the shooting position so that the subject area is an appropriate size for the background image. For example, if the subject area is too large, a message 1301 is superimposed on the live view image to prompt the user to distance themselves from the subject. When an appropriate background image can be obtained in this manner, the control unit 107 displays a message instructing the user to start shooting. When the control unit 107 detects that the shutter button 104 has been pressed, the control unit 107 executes still image shooting using the imaging unit 101 as described above and causes the calculation unit 102 to generate a background image. Note that the resolution of the background image (here, pixel density) may be matched to the resolution of the display unit 106.

[0048] Here, the camera icon will be described using Fig. 11(b). The camera icon 1401 may be, for example, a simple three-dimensional wireframe representing a camera, and the display position 1402 may be, for example, the center of gravity of the three-dimensional figure. To accurately indicate the position of the camera, a mark (for example, a dot) indicating the display position 1402 can be displayed together with the camera icon 1401.

[0049] The orientation of the camera icon 1401 indicates the shooting direction (the optical axis direction of the lens 101a), and is indicated in Fig. 11(b) by an arrow 1403. If the aspect ratio of the image is not 1:1, it is possible to use a camera icon 1401 with a shape that allows the portrait mode (vertical) and landscape mode (horizontal) to be distinguished.

[0050] The camera icon 1401 is not limited to the shape and display form shown in Fig. 11(b) as an example, as long as it can represent the position and orientation of the digital camera 100 at the time of shooting. However, it is desirable to use a form that does not reduce visibility even when multiple camera icons are displayed overlapping each other.

[0051] Returning to FIG. 2, in S205, the control unit 107 selects the superimposition position and orientation of the camera icon.

[0052] 15 shows a schematic diagram of the selection process for the superimposition position and orientation of the camera icon. The selection process for the superimposition position and orientation is a process for selecting the position and orientation at which the camera icon is displayed from among the positions and orientations corresponding to the image data group to be searched. FIG. 12(a) shows a state in which the position and orientation of the digital camera 100 at the time of image capture is displayed by a camera icon for all of the image data group to be searched. In contrast, FIG. 12(b) shows a state in which the position and orientation after the selection process for the superimposition position and orientation has been applied is displayed by a camera icon.

[0053] There are no particular limitations on the method for selecting the position and orientation, but for example, the selection can be made so that adjacent camera icons do not overlap. Furthermore, to prevent multiple camera icons from being selected by a touch operation or an icon next to an intended icon from being selected, the spacing between adjacent icons may be selected so that it is wider than the average size of a fingertip. Note that if the conditions that the position and orientation after the selection process must satisfy are already satisfied, such as when the camera icons do not overlap even when all the positions and orientations of the image data group to be searched are indicated by camera icons, the position and orientation selection process does not need to be performed.

[0054] 2, in S206, the control unit 107 changes the association between the position and orientation at which the camera icon is displayed and the still image data, as necessary. When the selection process is executed in S205 and the position and orientation at which the camera icon is displayed are reduced, it becomes impossible to select still image data corresponding to the position and orientation at which the camera icon is not displayed.

[0055] Therefore, in S206, the control unit 107 associates the still image data corresponding to the position and orientation thinned out in S205 with the position and orientation of the camera icon to be displayed. If the selection process was not performed in S205, the process of S206 does not need to be performed. For example, in FIG. 12(a), images 1504 to 1506 are associated with camera icons 1501 to 1503, one each. Here, it is assumed that the positions and orientations of camera icons 1502 and 1503 are deleted by the selection process of S205. In this case, in S206, the control unit 107 changes the association between the positions and orientations and the still image data so that images 1505 and 1506 correspond to the position and orientation of camera icon 1501, as shown in FIG. 12(b).

[0056] The still image data corresponding to the position and orientation deleted (not selected) in S205 may be associated with the position and orientation that has the smallest difference from the original position and orientation among the selected positions and orientations. For example, if the position and orientation corresponding to the unselected camera icon 1503 is closer to the position and orientation corresponding to the camera icon 1507 than the position and orientation corresponding to the camera icon 1501, the image 1506 is associated with the camera icon 1507. The closeness of the position and orientation can be calculated, for example, by Δ i, j It can be evaluated by the value of

number

[0057] In FIG. 12, the positions and orientations corresponding to the camera icons 1501, 1503, and 1507 are assumed to be the 1501st, 1503rd, and 1507th positions and orientations, respectively. In this case, Δ 1503, 1501 <Δ 1503, 1507 If so, the control unit 107 associates the image 1506 that originally corresponded to the position and orientation of the camera icon 1503 with the position and orientation of the camera icon 1501 .

[0058] In S207, the control unit 107, taking into account the results of S205 and S206, which are executed as necessary, causes the calculation unit 102 to generate image data for display in which a camera icon is superimposed on a background image. The calculation unit 102, which serves as a display control means, stores the generated image data in a video memory area in the RAM of the control unit 107. As a result, an image in which a camera icon 1401 is superimposed on a background image 1103 is displayed on the display unit 106, as shown in FIG. 13(a).

[0059] In S208, the control unit 107 detects the selection of a camera icon. FIG. 13(b) schematically shows the selection of a camera icon by a touch operation on the display unit 106. The camera icon may be selected through operation of the operation unit 105. For example, the control unit 107 displays the currently selected camera icon in a color different from the other camera icons, and moves the selected camera icon in response to operation of the direction keys of the operation unit 105. The control unit 107 considers that the camera icon that was selected when the enter button of the operation unit 105 was pressed has been selected.

[0060] In S209, the control unit 107 causes the display unit 106 to display an image associated with the position and orientation of the camera icon selected in S208. Here, it is assumed that a camera icon corresponding to the camera icon 1501 shown in Fig. 12 has been selected. In this case, as shown in Fig. 14(a), an image list 1801 displaying a list of thumbnails of images 1504 to 1506 in a selectable manner is displayed on the display unit 106.

[0061] The user can select an image that the user wants to check from the image list 1801 and enlarge it. Fig. 14(b) shows a state in which image 1504 is selected in Fig. 14(a) and image 1504 is enlarged and displayed. As with the selection of the camera icon, an image can be selected from the image list 1801 by a touch operation on the display unit 106 or by operating the operation unit 105. When selecting by operating the selection unit 105, the display color of the frame of the thumbnail of the selected image can be changed.

[0062] If there is only one image associated with the position and orientation of the camera icon selected in S208, the image list 1801 may be skipped and an enlarged display may be performed. Image identification information (for example, a file name for a still image, or at least a time code among the file name and time code for a video frame) may be displayed in at least one of the image list 1801 and the enlarged display.

[0063] FIG. 15(a) shows an example of the display of an image list when a frame image of a video is associated with the position and orientation of the camera icon selected in S208. A time code 2001 is displayed along with the image. When a frame image of a video is selected from the image list, instead of simply enlarging it, the video containing the selected frame image may be played. When playing back a video, for example, as shown in FIG. 15(b), playback may be paused at the selected frame image. When the play icon 2101 is operated, the control unit 107 starts playback from the selected frame image onwards.

[0064] As described above, according to this embodiment, it is possible to search for images based on the position and orientation of the image capture device at the time of capture. Therefore, even if the object to be searched has periodic image features, it is possible to search for images of the object captured from a specific angle.

[0065] (Other embodiments) In the above-described embodiment, a configuration has been described in which position and orientation estimation techniques such as SfM and SLAM are used during search, rather than during shooting (recording). However, these position and orientation estimation techniques may also be applied to images obtained during shooting. For example, if position and orientation estimation cannot be performed correctly because the distance between adjacent shooting positions is too long, the user can be prompted to reshoot. This makes it possible to prevent situations in which position estimation cannot be performed correctly during search, for example. Furthermore, even if the user reduces the amount of image data between recording and searching, the position and orientation of the remaining image data can be reliably obtained.

[0066] Furthermore, by recording video images for live view display captured while still images are not being taken, they can be used for position estimation during searches. For example, if adjacent still images are taken at different positions and do not contain common feature points, position and orientation estimation cannot be performed correctly. However, by using frame images from video images captured while still images are not being taken instead of still images, position and orientation estimation becomes possible.

[0067] In the above-described embodiment, a configuration has been described in which a background image for superimposing a camera icon is generated or captured. However, a camera icon may also be superimposed on an image displayed as a live view (live view image) in the shooting standby mode of digital camera 100. In this case, only a camera icon that can be displayed so as to fit within the field of view of the live view image, among positions and orientations at which images have been captured in the past, is superimposed and displayed. By continuously calculating the position and orientation for video shooting for live view display for each frame or for every predetermined number of frames and superimposing a camera icon, the user can be guided to a position and orientation that corresponds to the composition they wish to reshoot.

[0068] In the above embodiment, a configuration for executing an image search using the digital camera 100 has been described. However, an imaging function is not essential to the present invention, and the above-described image search processing may be performed by executing an application program on a computing device (such as a personal computer) that generally has higher processing power than an imaging device. By performing an image search using a computing device, it becomes possible to calculate the position and orientation with higher accuracy and to handle large amounts of image data than when performing the image search using an imaging device.

[0069] In addition to checking the results of shooting, it can also be used for image search for image editing. For example, by searching for images that were shot at a roughly constant position but differed only in horizontal orientation, it is possible to easily extract images that can be used to create a panoramic image.

[0070] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0071] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0072] 100... digital camera, 101a... lens, 101b... image pickup element, 102... calculation unit, 105... operation unit, 107... control unit

Claims

1. an acquisition means for acquiring information about the image capturing device at the time of capturing each piece of image data constituting the image data group to be searched, said information including at least information about the posture and information about the position; a display control means for displaying on a display device a selection screen that allows a user to select a position and orientation of the imaging device based on the information about the orientation and the information about the position; a search unit that searches for image data from the image data group to be searched based on the position and orientation selected on the selection screen; and The image processing device is characterized in that the selection screen includes a background image obtained based on image data other than the image data that constitutes the image data group, and includes a plurality of icons that are superimposed and displayed in a selectable manner at positions corresponding to the position and attitude of the imaging device.

2. An acquisition means for acquiring information about an imaging device at the time of photographing each piece of image data constituting a group of image data to be searched, wherein said information includes at least information about the posture and information about the position; a display control means for displaying on a display device a selection screen that allows a user to select a position and orientation of the imaging device based on the information about the orientation and the information about the position; a search unit that searches for image data from the image data group to be searched based on the position and orientation selected on the selection screen; and The image processing device is characterized in that the selection screen is an image including a background image obtained by pseudo-enlarging the angle of view of one of the image data constituting the image data group, and a plurality of icons superimposed and displayed selectably at positions corresponding to the position and attitude of the imaging device.

3. 3. The image processing apparatus according to claim 1, wherein the search means searches for image data associated with information corresponding to an icon selected on the selection screen.

4. 4. The image processing device according to claim 1, wherein the information acquired by the acquisition means is selected and used to display the plurality of icons so that the plurality of icons are displayed without overlapping each other.

5. 5. The image processing device according to claim 4, wherein image data corresponding to information not used to display the plurality of icons is associated with any of the information used to display the plurality of icons.

6. 6. The image processing apparatus according to claim 1, wherein the acquiring unit acquires the information from a data file of the image data group to be searched.

7. 7. The image processing device according to claim 1, wherein the acquisition means acquires the information from the group of image data to be searched using a position and orientation estimation technique based on positions of feature points common to a plurality of images.

8. The image processing device according to claim 7, characterized in that the acquisition means acquires the information from the image data group to be searched using other image data in addition to the image data group to be searched, and the other image data is video data taken for live view display.

9. 8. The image processing apparatus according to claim 1, wherein the image data group to be searched includes at least one of a still image data group and an image data group obtained by breaking down a moving image into frame units.

10. An image processing method executed by an image processing device, an acquisition step of acquiring information about the image capturing device at the time of capturing each piece of image data constituting the image data group to be searched, wherein the information includes at least information about the posture and information about the position; a display step of displaying on a display device a selection screen that allows a user to select a position and orientation of the imaging device based on the information about the orientation and the information about the position; a search step of searching for image data from the image data group to be searched based on the position and orientation selected on the selection screen; and An image processing method characterized in that the selection screen includes a background image obtained based on image data other than the image data that constitutes the image data group, and includes a plurality of icons superimposed and displayed in a selectable manner at positions corresponding to the position and attitude of the imaging device.

11. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 9.

Images