Information processing device, information processing method, and program

Abstract

An information processing device (10) comprises a circuit (11) and a memory (12) that is connected to the circuit (11). In an operation, the circuit (11) acquires a plurality of consecutive first frames, stores a plurality of key frames among the plurality of first frames in the memory (12), acquires position information indicating a position at which a second frame which is not consecutive to the plurality of first frames is imaged, and selects a third frame corresponding to the second frame from among the plurality of key frames on the basis of the position information.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] Conventionally, a method is known in which the position and orientation of an image capturing device are detected from a plurality of frames (images) and a three-dimensional model is generated.

[0003] For example, Patent Document 1 discloses a model forming device that forms a three-dimensional model of an object using previously acquired three-dimensional model data of the object as a starting point. The model forming device disclosed in Patent Document 1 includes a recognition unit that recognizes an unmodeled portion of the object, and an imaging instruction information unit that obtains imaging instruction information related to imaging of the unmodeled portion, and the imaging unit images the object in accordance with the imaging instruction information obtained by the imaging instruction information unit.

[0004] Japanese Patent Application Laid-Open No. 2006-24161

[0005] In such a device, it is desirable that a three-dimensional model be generated with high accuracy. Therefore, the present disclosure provides an information processing device and the like that can improve the accuracy of a three-dimensional model generated from multiple frames.

[0006] An information processing device according to one aspect of the present disclosure includes a circuit and a memory connected to the circuit, and in operation, the circuit acquires a plurality of consecutive first frames, stores a plurality of key frames from among the plurality of first frames in the memory, acquires positional information indicating a position at which a second frame that is not consecutive to the plurality of first frames is captured, and selects a third frame corresponding to the second frame from among the plurality of key frames based on the positional information.

[0007] An information processing method according to one aspect of the present disclosure acquires a plurality of consecutive first frames, stores a plurality of key frames from among the plurality of first frames in a memory, acquires positional information indicating a position at which a second frame that is not consecutive to the plurality of first frames is captured, and selects a third frame corresponding to the second frame from among the plurality of key frames based on the positional information.

[0008] A program according to one aspect of the present disclosure is a program for causing a computer to execute the information processing method described above.

[0009] The present disclosure can provide an information processing device and the like that can improve the accuracy of a three-dimensional model.

[0010] FIG. 1 is a block diagram showing the configuration of a 3D model generation system according to an embodiment. FIG. 2 is a sequence diagram showing the processing steps of the 3D model generation system according to an embodiment. FIG. 3 is a flowchart showing a first example of the processing steps of a candidate frame selection device according to an embodiment. FIG. 4 is a diagram showing a first example of a user interface displayed when the candidate frame selection device according to an embodiment accepts a position specification. FIG. 5 is a diagram showing a second example of a user interface displayed when the candidate frame selection device according to an embodiment accepts a position specification. FIG. 6 is a diagram showing a third example of a user interface displayed when the candidate frame selection device according to an embodiment accepts a position specification. FIG. 7 is a diagram showing a fourth example of a user interface displayed when the candidate frame selection device according to an embodiment accepts a position specification. FIG. 8 is a diagram showing a fifth example of a user interface displayed when the candidate frame selection device according to an embodiment accepts a position specification. FIG. 9 is a diagram showing a sixth example of a user interface displayed when the candidate frame selection device according to an embodiment accepts a position specification. FIG. 10 is a diagram showing a seventh example of a user interface displayed when the candidate frame selection device according to an embodiment accepts a position specification. Fig. 11 is a diagram showing a first example of a user interface displayed when a candidate frame selection device according to an embodiment accepts the designation of a frame. Fig. 12 is a flowchart showing a second example of a processing procedure of a candidate frame selection device according to an embodiment. Fig. 13 is a flowchart showing a third example of a processing procedure of a candidate frame selection device according to an embodiment. Fig. 14 is a diagram for explaining frames displayed when a candidate frame selection device according to an embodiment accepts the selection of a base frame. Fig. 15 is a diagram showing a second example of a user interface displayed when a candidate frame selection device according to an embodiment accepts the designation of a frame. Fig. 16 is a block diagram showing the configuration of an information processing device according to an embodiment. Fig. 17 is a flowchart showing an information processing method according to an embodiment.

[0011] (Summary of the Present Disclosure) Conventionally, there is a technology called Visual-SLAM that performs Simultaneous Localization and Mapping (SLAM) using frames as input. For example, in processing using Visual-SLAM, tracking, local mapping, loop closing, and mapping (specifically, real-time point cloud mapping) are performed in a multi-threaded manner simultaneously with image capture by a depth sensor.

[0012] In tracking, the position and orientation of the image capture device are estimated based on each input frame, and feature points detected in each frame are matched.

[0013] In local mapping, the positions of key frames among multiple input frames and the position of a map (3D map) are optimized (also called bundle adjustment). For example, if all input frames were used for this optimization, the amount of processing would increase. Therefore, for example, only multiple key frames among the multiple input frames are optimized.

[0014] In loop closing, when an imaging device repeatedly captures images while moving and then returns to the same location, a process is performed to eliminate errors. For example, this process determines whether the captured images are from the same location for each key frame. For example, this process involves matching feature points between frames. For example, if two frames are determined to have been captured at the same location, the error between the two frames is used to optimize the entire system based on the map (three-dimensional map) and the positions of the key frames.

[0015] During mapping, for each keyframe, the depth values ​​are checked for duplication between the previous and next keyframes. This process removes non-duplicate depth values ​​as noise. This process also integrates the 3D point information (point cloud information) for all keyframes. This allows the captured location to be represented as a 3D map, or in other words, a 3D model (e.g., a 3D mesh).

[0016] In this way, for example, Visual-SLAM technology is used to generate a three-dimensional model.

[0017] Here, after continuous imaging, the imaging may be interrupted once and then restarted.

[0018] However, conventionally, there is no mechanism for updating a generated three-dimensional model, that is, a result of previous processing, by taking additional images.

[0019] For example, a method can be considered in which a 3D model is generated each time imaging is interrupted and restarted, and the generated multiple 3D models are integrated. However, this method has the problem of requiring a large amount of processing. Also, there is a possibility that the multiple 3D models may not be properly aligned, resulting in the integrated 3D models being misaligned from their intended positions.

[0020] Furthermore, when automatically aligning a newly captured frame (also referred to as a new frame) with an existing 3D model, a frame that is similar to the new frame is selected from among the multiple frames used to generate the 3D model, and the selected frame is aligned with the new frame. In this case, an incorrect frame may be selected, resulting in reduced alignment accuracy. Furthermore, because frame selection is based solely on frame similarity, a frame that appears similar to the new frame may be selected that was captured at a location far from where the new frame was captured.

[0021] Therefore, an object of the present disclosure is to provide an information processing device or the like that can improve the accuracy of the generated three-dimensional model even when a process is performed in which the captured frame is used to regenerate the three-dimensional model (i.e., the existing three-dimensional model is updated) after the frame at the position where image capture has resumed is aligned with the existing three-dimensional model using Visual-SLAM or the like.

[0022] The information processing device of Example 1 includes a circuit and a memory connected to the circuit, and in operation, the circuit acquires a plurality of consecutive first frames, stores a plurality of key frames from among the plurality of first frames in the memory, acquires positional information indicating a position at which a second frame that is not consecutive to the plurality of first frames is captured, and selects a third frame corresponding to the second frame from among the plurality of key frames based on the positional information.

[0023] As a result, for example, when image capture of frames used to generate a three-dimensional model is resumed, a key frame (third frame) suitable for alignment with the frame (second frame) generated when image capture is resumed can be selected from among the multiple key frames used to generate the already generated three-dimensional model based on position information indicating the position when image capture was resumed. This improves the accuracy of SLAM after image capture is resumed. In this way, it is possible to accurately align the already generated three-dimensional model with a three-dimensional model generated using frames captured after the three-dimensional model was generated. Therefore, the accuracy of the three-dimensional model can be improved using the frames generated when image capture is resumed.

[0024] The information processing device of Example 2 may be the information processing device of Example 1, and may extract a plurality of fourth frames from the plurality of key frames based on the position information, and select the third frame from the plurality of fourth frames.

[0025] This allows, for example, multiple fourth frames that are close to the position indicated by the position information to be extracted from multiple key frames, making it easier to select key frames that are suitable for alignment with the generated frames when image capture is resumed.

[0026] The information processing device of Example 3 may be the information processing device of Example 2, which acquires the second frame, extracts one or more fourth frames from the plurality of fourth frames whose similarity to the second frame is higher than a predetermined threshold, and selects the third frame from the one or more fourth frames.

[0027] High frame similarity is considered more suitable for frame alignment than low frame similarity, which makes it easier to select key frames suitable for alignment with the generated frames when imaging resumes.

[0028] The information processing device of Example 4 may be the information processing device of Example 2, and may select the third frame by presenting the plurality of fourth frames and accepting designation of the third frame from among the plurality of fourth frames.

[0029] This makes it easier for the user to select a key frame that is suitable for alignment with the generated frame when image capture is resumed.

[0030] The information processing device of Example 5 is the information processing device of Example 2, and may present recommended values ​​for each of the plurality of fourth frames when the plurality of fourth frames are used for alignment with the second frame.

[0031] This makes it easier for the user to select a key frame that is suitable for alignment with the generated frame when image capture is resumed.

[0032] An information processing device of Example 6 is the information processing device of Example 5, wherein the recommended value may be calculated by comparing values ​​related to the plurality of fourth frames with a value related to the second frame.

[0033] From this, a recommended value can be calculated.

[0034] The information processing device of Example 7 is the information processing device of Example 5 or Example 6, and the recommended value may be calculated based on at least one of the frame similarity between each of the plurality of fourth frames and the second frame, the degree of blur of each of the plurality of fourth frames, and the acceleration of the imaging device when the plurality of fourth frames were captured.

[0035] From this, a recommended value can be calculated.

[0036] The information processing device of Example 8 may be any of the information processing devices of Examples 5 to 7, and may receive the recommended value and present the received recommended value.

[0037] This makes it easy to get recommendations.

[0038] The information processing device of Example 9 may be any of the information processing devices of Examples 5 to 7, and may calculate the recommended value and present the calculated recommended value.

[0039] This allows the recommended value to be calculated without using an external device such as a server.

[0040] The information processing device of Example 10 may be the information processing device of Examples 1 to 9, and may acquire the position information by presenting a three-dimensional model generated based on the plurality of first frames and accepting a specification of the position.

[0041] This allows the user to easily specify the position at which to resume imaging.

[0042] The information processing device of Example 11 is the information processing device of Examples 1 to 10, and the position information may indicate a position at which the second frame, which is the first of a plurality of consecutive fifth frames including the second frame, is captured.

[0043] This allows, for example, when the capture of frames used to generate a three-dimensional model is resumed, to select a key frame (third frame) suitable for alignment with the frame (second frame) generated when capture is resumed from among the multiple key frames used to generate the three-dimensional model that has already been generated, based on position information indicating the position at which capture began among the multiple frames captured after the resumption.

[0044] The information processing device of Example 12 is the information processing device of Examples 1 to 10, and may store a plurality of consecutive fifth frames including the second frame in the memory, and select the second frame from among the plurality of fifth frames.

[0045] This makes it possible to accurately align an already generated three-dimensional model with a three-dimensional model generated using a frame group captured after the three-dimensional model was generated, even when there are multiple frame groups consisting of consecutive frames.

[0046] The information processing device of Example 13 may be the information processing device of Example 12, and may select the second frame by presenting the plurality of fifth frames and accepting designation of the second frame from among the plurality of fifth frames.

[0047] This allows the user to specify the second frame to be used for alignment with the key frame.

[0048] The information processing method of Example 14 acquires a plurality of consecutive first frames, stores a plurality of key frames from among the plurality of first frames in a memory, acquires positional information indicating a position at which a second frame that is not consecutive to the plurality of first frames is captured, and selects a third frame corresponding to the second frame from among the plurality of key frames based on the positional information.

[0049] This makes it possible to obtain the same effects as those of the information processing device of Example 1.

[0050] The program of Example 15 is a program for causing a computer to execute the information processing method described in Example 14.

[0051] This provides the same effects as the information processing device of Example 1.

[0052] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. Therefore, the numerical values, shapes, materials, components, component arrangements and connection forms, steps, and step sequences shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Therefore, among the components in the following embodiments, components that are not recited in the independent claims of the present disclosure will be described as optional components.

[0053] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and redundant explanations are omitted or simplified.

[0054] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not refer to the number or order of components unless otherwise specified, but are used for the purpose of avoiding confusion and distinguishing between components of the same type.

[0055] (Embodiment) [Configuration] First, the configuration of a three-dimensional model generation system according to an embodiment will be described.

[0056] FIG. 1 is a block diagram showing the configuration of a three-dimensional model generation system 400 according to an embodiment.

[0057] The 3D model generation system 400 is a system that generates a 3D model from a plurality of frames (images). The 3D model generation system 400 includes an imaging device 300, an estimation device 200, and a candidate frame selection device 100.

[0058] The imaging device 300 is a camera that generates multiple frames by capturing an image of a subject and outputs the generated multiple frames to the estimation device 200. For example, the imaging device 300 captures multiple frames obtained by capturing images of the subject (object) from different viewpoints (image capturing positions and image capturing directions), and outputs the multiple frames to the estimation device 200. For example, the imaging device 300 captures images while moving, thereby capturing multiple frames. The multiple frames captured consecutively by the imaging device 300 are an example of multiple consecutive first frames.

[0059] The imaging device 300 includes a storage unit 310 , a control unit 320 , an optical system 330 , and an image sensor 340 .

[0060] The storage unit 310 is a storage device that stores various types of information. For example, the storage unit 310 stores frames captured by the imaging device 300. The storage unit 310 is realized by, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0061] The control unit 320 is a processing unit that executes various processes in the imaging device 300. For example, the control unit 320 controls the image sensor 340 to cause the image sensor 340 to capture an image, and stores frames captured by the image sensor 340 in the storage unit 310. The control unit 320 is realized by, for example, a memory and a processor such as a CPU (Central Processing Unit) that executes a control program stored in the memory.

[0062] The optical system 330 is an optical member for controlling light input to the image sensor 340. The optical system 330 is realized by, for example, a lens.

[0063] The image sensor 340 is a sensor for generating frames by capturing an image of a subject. For example, the image sensor 340 is a sensor capable of detecting R (Red), G (Green), and B (Blue). The image sensor 340 is realized by an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). That is, in this embodiment, the imaging device 300 captures RGB images as frames.

[0064] The imaging device 300 may include various sensors, such as a distance sensor (depth sensor), a laser sensor such as a LiDAR (Light Detection and Ranging), or an IMU (Inertial Measurement Unit). The IMU is a sensor that includes at least one of an acceleration sensor, a rotational angular acceleration sensor, and a gyro sensor.

[0065] The imaging device 300 outputs sensor information including multiple frames to the estimation device 200, for example, via a communication interface (not shown) that the imaging device 300 is provided with. Note that the sensor information may include, for example, a depth image obtained by a distance sensor, point cloud data obtained by a laser sensor, and acceleration information obtained by an IMU.

[0066] The estimation device 200 is a computer that estimates the position and orientation of the imaging device 300 (hereinafter simply referred to as the position and orientation of the imaging device 300) using multiple frames acquired from the imaging device 300. The estimation device 200 also generates a three-dimensional model using the multiple frames and the position and orientation of the imaging device 300. For example, the estimation device 200 generates a three-dimensional map (hereinafter simply referred to as the map) as the three-dimensional model. The estimation device 200 displays the map on a display device such as a display (not shown), by outputting the generated map (specifically, map information indicating the map). The estimation device 200 also outputs, for example, sensor information including the frames acquired from the imaging device 300, the estimated position and orientation of the imaging device 300 (specifically, position and orientation information indicating the position and orientation of the imaging device 300), and the generated map (specifically, map information indicating the map) to the candidate frame selection device 100.

[0067] For example, the estimation apparatus 200 generates position and orientation information indicating the position and orientation of the image capturing device 300 at the time of capturing each frame, using multiple frames captured by the image capturing device 300. Here, the orientation of the image capturing device 300 indicates at least one of the image capturing direction of the image capturing device 300 and the tilt of the image capturing device 300. The image capturing direction of the image capturing device 300 is the direction of the optical axis of the image capturing device 300. The tilt of the image capturing device 300 is the rotation angle around the optical axis of the image capturing device 300 from the reference orientation.

[0068] The estimation apparatus 200 also generates map information indicating a three-dimensional model (three-dimensional map) composed of a three-dimensional point cloud indicating three-dimensional positions together with the position and orientation information.

[0069] For example, the estimation device 200 performs feature point matching processing on multiple frames captured by the image capture device 300. That is, the estimation device 200 extracts feature points from each of the multiple frames and extracts, from the extracted multiple feature points, a set of similar points (corresponding points) that are similar between the multiple frames. Next, the estimation device 200 uses the extracted set of similar points to estimate the position and orientation of the image capture device 300 for each frame.

[0070] The map (map information) indicates a plurality of map points, each of which is a three-dimensional point (specifically, a point indicating the position of a three-dimensional point). The estimation device 200 generates the map by performing triangulation using the result of the feature point matching process and the position and orientation of the image capture device 300. Note that the generated map (map information) may include, in addition to information indicating the position of the map points, information indicating the visibility of each map point, information indicating the color of each map point, and information indicating the surface shape around each map point (e.g., information indicating normals). The map may also be modified to high-precision information by undergoing an optimization process together with the position and orientation of the image capture device 300.

[0071] For example, the estimation device 200 generates a mesh model represented by a plane with map points as vertices as the three-dimensional model. The three-dimensional model is not limited to a mesh model and may be any three-dimensional model. Furthermore, for example, the three-dimensional model may be generated using point cloud data obtained by LiDAR or a depth image obtained by a range sensor.

[0072] The estimation method used by the estimation apparatus 200 to estimate the position and orientation of the image capture apparatus 300 is, for example, Visual-SLAM, but is not particularly limited to this.

[0073] The estimation device 200 is realized by, for example, a communication interface for communicating with the imaging device 300 and the candidate frame selection device 100, etc., a non-volatile memory in which a program is stored, a volatile memory that is a temporary storage area for executing the program, an input / output port for transmitting and receiving signals, a processor that executes the program, etc. The communication interface may be realized by, for example, an antenna and a wireless communication circuit or the like to enable wireless communication, or by a connector to which a communication line is connected to enable wired communication.

[0074] The candidate frame selection device 100 is a computer that selects a candidate frame from among a plurality of frames captured by the imaging device 300. Specifically, the candidate frame selection device 100 acquires a plurality of consecutive frames captured by the imaging device 300, and saves (stores) a plurality of key frames from among the acquired plurality of frames in a storage unit 110. The candidate frame selection device 100 also acquires position information indicating a position at which a predetermined frame that is not consecutive to the plurality of frames captured by the imaging device 300 is captured, and selects a candidate frame corresponding to the predetermined frame from among the plurality of key frames based on the acquired position information.

[0075] The position information indicates, for example, the position at which the first frame of a series of consecutive frames, including a predetermined frame, is captured. That is, for example, the candidate frame selection device 100 acquires position information indicating the position at which the capture of a current frame, which is an example of a predetermined frame not consecutive to the first frame captured by the imaging device 300, begins, and selects a candidate frame corresponding to the current frame from among multiple key frames based on the acquired position information. For example, the candidate frame selection device 100 selects a candidate frame from among multiple frames captured consecutively by the imaging device 300 in the past, based on the position of the imaging device 300 that captured the current frame or the position of the imaging device 300 that is about to start capturing. The selected candidate frame is output to the estimation device 200. The estimation device 200 aligns the candidate frame with the current frame that has already been captured or the current frame that will be captured, and resumes map generation based on the alignment results.

[0076] Alternatively, the candidate frame selection device 100 stores, for example, a plurality of consecutive frames including a predetermined frame in the storage unit 110, and selects the predetermined frame from the plurality of frames stored in the storage unit 110. For example, suppose that a plurality of frame groups each consisting of a plurality of consecutive frames are stored in the storage unit 110. Also, suppose that the plurality of frame groups are not consecutive. For example, suppose that frame group B is captured a predetermined time after frame group A is captured. In such a case, for example, the candidate frame selection device 100 selects a predetermined frame from frame group B. The predetermined frame may be the first frame captured in frame group B (for example, the current frame), or may be a frame other than the first frame. The predetermined frame is an example of a second frame.

[0077] The candidate frame selection device 100 is realized by, for example, a communication interface for communicating with the estimation device 200 and the imaging device 300, etc., a non-volatile memory in which a program is stored, a volatile memory that is a temporary storage area for executing the program, an input / output port for transmitting and receiving signals, a processor that executes the program, etc. The communication interface may be realized by, for example, an antenna and a wireless communication circuit or the like to enable wireless communication, or by a connector to which a communication line is connected to enable wired communication.

[0078] The current frame is an example of a second frame, and the candidate frame is an example of a third frame.

[0079] Furthermore, the multiple frames may be multiple still images captured by the imaging device 300 while it is moving, or multiple images included in a moving image captured by the imaging device 300 while it is moving. Furthermore, when a moving image is used as the multiple frames, the multiple frames do not need to be all images included in the moving image, but may be images that are a portion of the multiple images included in the moving image.

[0080] A key frame is, for example, a key image extracted from a plurality of images included in a moving image at a predetermined time interval. Alternatively, a key frame may be a key image extracted from a plurality of still images at a predetermined time interval. The predetermined time interval may be determined arbitrarily and is not particularly limited.

[0081] The candidate frame selection device 100 includes a storage unit 110 , a data input unit 120 , an output unit 130 , a recommended value calculation unit 140 , a search range calculation unit 150 , a UI unit 160 , and a control unit 170 .

[0082] The storage unit 110 is a storage device that stores various types of information. For example, the storage unit 110 stores a plurality of key frames among a plurality of consecutive frames captured by the imaging device 300.

[0083] The storage unit 110 may store all of the consecutive frames, or may store only the key frames. The storage unit 110 stores, for example, sensor information, position and orientation information, and maps acquired from the estimation device 200. The storage unit 110 is realized by, for example, a solid state drive (SSD) or a hard disk drive (HDD). The storage unit 110 is an example of a memory.

[0084] The data input unit 120 is a processing unit that acquires various information (various data) from the estimation device 200. Specifically, the data input unit 120 acquires various information from the estimation device 200 via a communication interface (not shown) that the candidate frame selection device 100 is equipped with. The data input unit 120 stores, for example, sensor information, position and orientation information, maps, and the like acquired from the estimation device 200 in the storage unit 110. In other words, the storage unit 110 stores the various information acquired by the data input unit 120.

[0085] For example, the data input unit 120 acquires multiple consecutive frames from the estimation device 200 and stores multiple key frames from the acquired multiple frames in the storage unit 110. Furthermore, for example, the data input unit 120 acquires position information indicating the position at which a current frame that is not consecutive to the multiple frames is captured. For example, the data input unit 120 acquires position information indicating the position at which capturing of the current frame that is not consecutive to the multiple frames begins. Alternatively, for example, the data input unit 120 acquires multiple new frames different from the acquired multiple frames and acquires position information indicating the position at which one frame (also referred to as a reference frame) of the newly acquired multiple frames was captured. In this example, the data input unit 120 acquires position information indicating the position at which capturing of the current frame began.

[0086] The position information may be acquired from the estimation device 200, from the imaging device 300, or from a user via an operation device such as a touch panel, a mouse, or a keyboard. The operation device may be a microphone or the like that accepts voice operations. The display device may also include a speaker that outputs voice and a vibrator that transmits vibrations to the user as means for presenting information to the user. The information acquired via the operation device may be acquired by the data input unit 120 or by the UI unit 160. In this embodiment, it is described that the UI unit 160 acquires input such as position information from a user via the operation device (in other words, accepts input).

[0087] The data input unit 120 may acquire the sensor information directly from the imaging device 300 without going through the estimation device 200 .

[0088] The output unit 130 is a processing unit that outputs the candidate frames to the estimation device 200. For example, the output unit 130 transmits the candidate frames to the estimation device 200 via a communication interface (not shown) that the candidate frame selection device 100 has.

[0089] The recommended value calculation unit 140 is a processing unit that calculates recommended values ​​for frames. Specifically, when multiple candidate frames (multiple recommended frames) among multiple key frames are used for alignment with the current frame, the recommended value calculation unit 140 calculates the recommended value for each of the multiple candidate frames. The multiple candidate frames (multiple recommended frames) are an example of multiple fourth frames.

[0090] The recommendation value is a numerical value that indicates how suitable each candidate frame is for registration when multiple candidate frames are used to align with the current frame. For example, a frame with a higher recommendation value indicates that the estimation apparatus 200 can align the frame with the current frame better.

[0091] For example, the recommended value is calculated by comparing values ​​for the multiple candidate frames with values ​​for the current frame. For example, the recommended value is calculated based on at least one of the frame similarity between each of the multiple candidate frames and the current frame, the degree of blur of each of the multiple candidate frames, and the acceleration of the imaging device 300 when the multiple candidate frames were captured. In other words, the recommended value calculation unit 140 calculates the recommended value based on at least one of the frame similarity between each of the multiple candidate frames and the current frame, the degree of blur (amount of blur) of each of the multiple candidate frames, and the acceleration of the imaging device 300 when the multiple candidate frames were captured.

[0092] For example, the recommendation value calculation unit 140 calculates the recommendation value such that the higher the similarity between frames, the higher the recommendation value of the candidate frame. Also, for example, the smaller the degree of blur of a frame, the higher the recommendation value of the candidate frame. Also, for example, the recommendation value calculation unit 140 calculates the recommendation value such that the smaller the acceleration of the imaging device 300, the higher the recommendation value of the candidate frame. The acceleration of the imaging device 300 may be acquired from the imaging device 300, may be acquired from the estimation device 200, or may be acquired from the user via an operation device.

[0093] The frame similarity and the degree of blurring of the frames may be calculated by any method. For example, the frame similarity may be calculated using Bag of Visual Words (BoVW). For example, the frame blurring degree may be calculated using Fourier transform.

[0094] The search range calculation unit 150 is a processing unit that extracts (selects) multiple candidate frames from multiple key frames based on location information. Specifically, the search range calculation unit 150 calculates (determines) a search range on a map based on the location information. For example, the search range calculation unit 150 determines a predetermined range on the map that includes a position indicated by the location information as the search range. The search range calculation unit 150 extracts frames captured within the determined search range from the multiple key frames as candidate frames. The recommended value calculation unit 140, for example, calculates a recommended value for each of the multiple candidate frames extracted by the search range calculation unit 150.

[0095] The UI unit 160 is a processing unit that displays a user interface (UI) on a display or other display device connected to the candidate frame selection device 100. The UI unit 160 also accepts input (also referred to as designation or selection) from the user via an operation device or the like connected to the candidate frame selection device 100. For example, the UI unit 160 acquires position information from the user via the operation device.

[0096] For example, the UI unit 160 may use a display device to present the user with recommended values ​​for each of multiple candidate frames when multiple candidate frames are used to align the current frame. For example, the display device may display the frames and recommended values ​​sent from the candidate frame selection device 100 under the control of the UI unit 160. The user may input location information by operating an operation device, check the information displayed on the display device, such as the multiple candidate frames determined based on the location information and the recommended values ​​for each candidate frame, and select one candidate frame (one alignment frame) from among the multiple candidate frames by operating the operation device. The output unit 130 outputs the one candidate frame selected by the user via the operation device to the estimation device 200. This one candidate frame (one alignment frame) is an example of a third frame. For example, the UI unit 160 may use the display device to present multiple candidate frames to the user and accept the user's designation of one candidate frame from among the multiple candidate frames via the operation device. As a result, for example, the candidate frame selection device 100 selects one candidate frame.

[0097] In this way, the candidate frame selection device 100 selects a candidate frame corresponding to the current frame from among a plurality of key frames, for example, based on position information. For example, the candidate frame selection device 100 extracts a plurality of candidate frames from among a plurality of key frames based on position information, and selects one candidate frame from among the plurality of candidate frames. For example, information indicating the selected candidate frame is output to the estimation device 200, and the estimation device 200 aligns the selected candidate frame with the current frame.

[0098] Furthermore, for example, the UI unit 160 may extract and display one or more candidate frames from among a plurality of candidate frames, the candidate frames having a similarity to the current frame higher than a predetermined threshold. As a result, for example, the output unit 130 may select one or more candidate frames from among the one or more candidate frames based on a user input, and output the selected candidate frames to the estimation device 200.

[0099] The predetermined threshold may be arbitrarily determined in advance and is not particularly limited.

[0100] Alternatively, the selected candidate frame itself may be output to the estimation device 200, or information such as an identifier indicating the selected candidate frame may be output to the estimation device 200.

[0101] The UI unit 160 may also be realized to include the functions of a display device and an operation device.

[0102] Furthermore, for example, the UI unit 160 may display a plurality of key frames on a display device, or may present a three-dimensional model (a map in this embodiment) generated based on a plurality of frames captured by the imaging device 300. For example, the UI unit 160 may acquire location information by presenting a map and accepting a location specification from the user.

[0103] Furthermore, in this embodiment, the recommended value calculation unit 140 calculates the recommended value, and the UI unit 160 presents the calculated recommended value. For example, the candidate frame selection device 100 may receive recommended values ​​from an external server or the like and present the received recommended values. In this case, the candidate frame selection device 100 does not need to be equipped with the recommended value calculation unit 140.

[0104] The control unit 170 is a processing unit that controls the storage unit 110, the data input unit 120, the output unit 130, the recommended value calculation unit 140, the search range calculation unit 150, and the UI unit 160. For example, the control unit 170 controls each unit in response to an operation of an operating device by a user.

[0105] Each processing unit, such as the data input unit 120, the output unit 130, the recommended value calculation unit 140, the search range calculation unit 150, the UI unit 160, and the control unit 170, is realized, for example, by a processor such as a CPU (Central Processing Unit) and a memory in which a control program executed by the processor is stored.

[0106] The candidate frame selection device 100, the estimation device 200, and the imaging device 300 may be realized as a single device. Alternatively, multiple processing units included in each device may be distributed across multiple devices.

[0107] For example, in a use case in which a user captures an image of a moving subject using a terminal (such as a smartphone, tablet terminal, or personal computer) carried by the user, the candidate frame selection device 100, the estimation device 200, and the imaging device 300 may be included in the terminal. In another use case, the imaging device 300 may be included in a moving object (such as a vehicle, robot, or drone) operated by the user, and the candidate frame selection device 100 and the estimation device 200 may be included in the terminal carried by the user. In either case, some of the processing units included in each device may be included in another device, such as a server, connected to the terminal operated by the user via a communication network. For example, some or all of the processing units included in the candidate frame selection device 100 may be included in the server.

[0108] 1 may be transmitted and received by any method, such as wired or wireless communication. This communication may be performed directly between the devices, or indirectly via another communication device or a server. This transmission and reception may involve data transfer within a single device.

[0109] [Processing Procedure] Next, a description will be given of the processing procedure of the three-dimensional model generation system 400. For example, in the three-dimensional model generation system 400, the estimation apparatus 200 creates (generates) a map, which is an example of a three-dimensional model, in real time based on frames captured by the imaging device 300.

[0110] 2 is a sequence diagram showing the processing procedures of the 3D model generation system 400 according to the embodiment. Specifically, FIG. 2 is a sequence diagram for explaining the processing procedures of the imaging device 300, the estimation device 200, and the candidate frame selection device 100 included in the 3D model generation system 400.

[0111] First, the image capturing device 300 captures an image of a subject (S101). Specifically, the image capturing device 300 captures an image of the subject to generate a frame.

[0112] Next, the imaging device 300 transmits sensor information including the captured frame (specifically, the frame generated by capturing an image of the subject) to the estimation device 200 (S102).

[0113] Furthermore, the imaging device 300 captures an image of the subject (S103) and transmits sensor information including the generated frame to the estimation device 200 (S104).

[0114] In this way, the imaging device 300 repeatedly captures images of a subject while, for example, moving, and transmits a plurality of consecutive frames to the estimation device 200 .

[0115] The estimation apparatus 200 estimates the position and orientation of the image capturing apparatus 300 based on the sensor information acquired from the image capturing apparatus 300, and creates a map based on the estimated position and orientation of the image capturing apparatus 300 and the sensor information (S105).

[0116] Next, the estimation device 200 transmits a frame group, which is a plurality of frames included in the sensor information, the position and orientation of the imaging device 300 (specifically, position and orientation information indicating the position and orientation), and the created map (specifically, map information indicating the map) to the candidate frame selection device 100 (S106).

[0117] The candidate frame selection device 100 displays the frame group acquired from the estimation device 200 on a display device (S107). The user checks the frame group displayed on the display device and uses the operation device to specify (input) the position at which to start capturing images using the imaging device 300. The candidate frame selection device 100 acquires information indicating the position accepted by the operation device as position information. For example, the UI unit 160 receives the start position for additional image capturing from the user by clicking.

[0118] Next, the candidate frame selection device 100 calculates a search range based on the input from the user, in other words, based on the acquired position information (S108).

[0119] The search range is information used to select multiple candidate frames from multiple key frames in the acquired frame group. For example, the search range indicates a partial area of ​​a map created by the estimation device 200.

[0120] Here, for example, the imaging device 300 resumes imaging to capture an image of the subject (S109), and transmits sensor information including the generated frame (current frame) to the estimation device 200 (S110).

[0121] The estimation apparatus 200 estimates (calculates) the current position and orientation of the image capture device 300 based on the acquired sensor information (S111). Note that the term "current" here refers to the position and orientation at the time when the image capture device 300 captured the current frame.

[0122] Next, the estimation device 200 transmits the current frame included in the sensor information and the current position and orientation of the imaging device 300 (specifically, position and orientation information indicating the current position and orientation) to the candidate frame selection device 100 (S112).

[0123] The candidate frame selection device 100 calculates a recommended value for each frame (S113). For example, the candidate frame selection device 100 calculates the similarity between the acquired current frame and each frame included in the search range (specifically, multiple candidate frames), and calculates a recommended value for each frame based on the calculated similarity. Note that each frame included in the search range is a frame whose image position is included in the search range.

[0124] Next, the candidate frame selection device 100 displays on the display device the search range (specifically, information indicating the search range), each frame in the search range (specifically, multiple candidate frames included in the search range), the current frame, and each recommended value for each frame in the search range (S114). The user checks the information displayed on the display device and uses the operation device to specify (input) the candidate frame to be used for alignment. The candidate frame selection device 100 selects the frame accepted by the operation device as the candidate frame to be used for alignment.

[0125] Next, the candidate frame selection device 100 transmits the selected candidate frames to the estimation device 200 (S115). If the estimation device 200 stores frames acquired from the imaging device 300, the candidate frame selection device 100 may transmit information such as identifiers indicating the candidate frames to the estimation device 200.

[0126] The estimation apparatus 200 aligns the acquired candidate frame with the current frame (S116), thereby restarting calculation of the position and orientation of the image capture device 300 and map creation.

[0127] The processing of steps S107 to S108 and the processing of steps S109 to S112 may be performed in any order. For example, the processing of steps S107 to S108 may be followed by the processing of steps S109 to S112, or the processing of steps S107 to S108 may be performed after the processing of steps S109 to S112, or the processing of steps S107 to S108 and the processing of steps S109 to S112 may be performed in parallel.

[0128] FIG. 3 is a flowchart showing a first example of a processing procedure of the candidate frame selection device 100 according to the embodiment.

[0129] First, the data input unit 120 acquires a frame group, position and orientation information indicating the position and orientation of the image capture device 300, and a map (map information) from the estimation device 200, and stores them in the storage unit 110 (S201). Note that the data input unit 120 may store all of the acquired frame group in the storage unit 110, or may store only a plurality of key frames from the acquired frame group in the storage unit 110.

[0130] Next, the UI unit 160 displays the frame group acquired from the estimation device 200 on a display device (S202). The UI unit 160 may display the acquired frame group, i.e., multiple frames, or may display a map (three-dimensional model) created based on the frame group. That is, the UI unit 160 may display the frame group in two dimensions or three dimensions.

[0131] Next, the search range calculation unit 150 calculates a search range based on the input from the user, in other words, based on the acquired position information (S203).

[0132] Next, the data input unit 120 acquires the current frame and the current position and orientation of the image capturing device 300 (specifically, position and orientation information indicating the current position and orientation) (S204).

[0133] Next, the recommended value calculation unit 140 calculates the recommended value for each frame (S205).

[0134] Next, the UI unit 160 displays on the display device the search range (specifically, information indicating the search range), each frame of the search range (specifically, multiple candidate frames included in the search range), the current frame, and each recommended value for each frame of the search range (S206).

[0135] Next, the output unit 130 outputs the candidate frames to the estimation device 200 based on the input from the user (S207). Specifically, the output unit 130 transmits the candidate frames selected by the user to the estimation device 200.

[0136] [User Interface] Next, specific examples of user interfaces (display images) displayed on UI unit 160 when accepting a designation (input) from the user will be described with reference to Figures 4 to 11. Figures 4 to 10 show specific examples of user interfaces displayed in steps S107 and S202, and Figure 11 shows a specific example of a user interface displayed in steps S114 and S206.

[0137] FIG. 4 is a diagram showing a first example of a user interface displayed when the candidate frame selection device 100 according to the embodiment accepts a position specification.

[0138] The user interface 501 shown in this example includes an overhead view that schematically shows the map as seen from above, that is, an image of the map displayed in two dimensions. The user interface 501 also includes information indicating the position and orientation of the image capture device 300. For example, the display (information) indicating the position and orientation of the image capture device 300 is represented by a triangle.

[0139] The user checks the user interface 501 and inputs a position. In this example, the position input by the user is shown as selected position A1. The search range calculation unit 150 calculates the search range using this input position. For example, the UI unit 160 changes the user interface 501 so that the display indicating the positions and orientations of the image capture devices 300 included in the calculated search range is displayed with a solid-line triangle, and the display indicating the positions and orientations of the image capture devices 300 not included in the calculated search range is displayed with a dashed-line triangle.

[0140] In this way, for example, in this example, the user selects a restart position from the bird's-eye view of the map, and the candidate frame selection device 100 sets the search range to the frame related to the frame closest to the selected position.

[0141] This makes it easier for the user to distinguish between areas that are included in the search range and areas that are not included in the search range.

[0142] FIG. 5 is a diagram showing a second example of a user interface displayed when the candidate frame selection device 100 according to the embodiment accepts a position specification.

[0143] The user interface 502 shown in this example includes a bird's-eye view that schematically shows the map as seen from above. The user interface 502 also includes information indicating the position and orientation of the image capture device 300. For example, the display indicating the position and orientation of the image capture device 300 is displayed as a triangle.

[0144] The user checks the user interface 502 and inputs a position. In this example, the user inputs a position as a range rather than a point, and the position (range) input by the user is indicated by a dotted line as a selected range A2. The search range calculation unit 150 calculates a search range using this input range. In this example, the search range calculation unit 150 uses this input range as the search range as is. For example, the UI unit 160 changes the user interface 502 so that a solid-line triangle indicates the position and orientation of the image capture device 300 that is included in the calculated search range, and a dashed-line triangle indicates the position and orientation of the image capture device 300 that is not included in the calculated search range.

[0145] In this way, in this example, the user specifies the search range including the resumption position from the bird's-eye view of the map, for example, by dragging.

[0146] FIG. 6 is a diagram showing a third example of a user interface displayed when the candidate frame selection device 100 according to the embodiment accepts a position specification.

[0147] The user interface 503 shown in this example includes a bird's-eye view that schematically shows the map as seen from above. The user interface 503 also includes information indicating the position and orientation of the image capture device 300. For example, the display indicating the position and orientation of the image capture device 300 is displayed as a triangle.

[0148] The user checks the user interface 503 and inputs a position. In this example, the user inputs a position as a range rather than a point, and the position (range) input by the user is indicated by a dotted line as a selection range A2. The search range calculation unit 150 calculates a search range using this input range. In this example, the search range calculation unit 150 calculates a search range that is slightly larger than the input range. In this manner, the search range calculation unit 150 may use the input range as the search range as is, or may arbitrarily change the size and / or shape of the input range to set the search range. For example, the UI unit 160 changes the user interface 503 so that a solid-line triangle indicates the position and orientation of the image capture device 300 that is included in the calculated search range, and a dashed-line triangle indicates the position and orientation of the image capture device 300 that is not included in the calculated search range.

[0149] In this example, the user specifies a range, for example by dragging, and the candidate frame selection device 100 sets the search range to frames located within the specified range and frames related to that frame.

[0150] FIG. 7 is a diagram showing a fourth example of a user interface displayed when the candidate frame selection device 100 according to the embodiment accepts a position specification.

[0151] The user interface 504 shown in this example includes a bird's-eye view that schematically shows the map as seen from above. The user interface 504 also includes information indicating the position and orientation of the image capture device 300. For example, the display indicating the position and orientation of the image capture device 300 is displayed as a triangle.

[0152] The user checks the user interface 504 and inputs a position. In this example, the user inputs the position not by a point or an area but by selecting an indication indicating the position and orientation of the image capturing device 300. For example, when the UI unit 160 receives a position from the user, specifically, a designation of one of a plurality of indications indicating the position and orientation of the image capturing device 300, the UI unit 160 displays a frame corresponding to the display indicating the accepted position and orientation of the image capturing device 300. The UI unit 160 also displays a display for accepting a designation from the user, such as "Candidate? Y / N." For example, the user selects Y or N. For example, when the search range calculation unit 150 receives a selection of Y from the user, the search range calculation unit 150 determines the position of the indication indicating the position and orientation of the image capturing device 300, which has been accepted as the position designated by the user. On the other hand, when the search range calculation unit 150 receives a selection of N from the user, the search range calculation unit 150 determines the position of the indication indicating the position and orientation of the image capturing device 300, which has been accepted as the position not designated by the user. By repeating this process, the UI unit 160 accepts position designation from the user, and the search range calculation unit 150 calculates the search range. For example, the UI unit 160 changes the user interface 504 so that the display indicating the position and orientation of the image capture device 300 included in the calculated search range is displayed with a solid triangle, and the display indicating the position and orientation of the image capture device 300 not included in the calculated search range is displayed with a dashed triangle.

[0153] In this way, in this example, the user checks and selects each frame from the bird's-eye view of the map, and the candidate frame selection device 100 sets the multiple frames selected by the user as the search range.

[0154] FIG. 8 is a diagram showing a fifth example of a user interface displayed when the candidate frame selection device 100 according to the embodiment accepts a position specification.

[0155] The user interface 505 shown in this example includes a bird's-eye view that schematically shows the map as seen from above. The user interface 505 also includes information indicating the position and orientation of the image capture device 300. For example, the display indicating the position and orientation of the image capture device 300 is displayed as a triangle.

[0156] The user checks the user interface 505 and inputs a position. In this example, the position input by the user is shown as a selected position A1. Furthermore, in this example, the user selects an object included in the user interface 505. For example, in this example, it is assumed that the user selects a selected object A3 included in the user interface 505. The search range calculation unit 150 calculates a search range using the input position and the selected object A3. Specifically, the search range calculation unit 150 calculates a search range based on the input position, and further calculates, as the final search range, only the display of the position and orientation of the image capturing device 300 closest to the input position, and the display of the position and orientation of the image capturing device 300 in which the selected object A3 appears in a frame corresponding to the position and orientation of the image capturing device 300. For example, the UI unit 160 changes the user interface 505 so that the display indicating the position and orientation of the imaging device 300 included in the calculated search range is displayed with a solid triangle, and the display indicating the position and orientation of the imaging device 300 not included in the calculated search range is displayed with a dashed triangle.

[0157] Thus, in this example, the user selects a restart position and a characteristic object, and the candidate frame selection device 100 sets the search range to, for example, the frame closest to the selected restart position and the frame in which the selected object appears.

[0158] FIG. 9 is a diagram showing a sixth example of a user interface displayed when the candidate frame selection device 100 according to the embodiment accepts a position specification.

[0159] The user interface 506 shown in this example includes an image of the map displayed as a three-dimensional model A4. For example, the UI unit 160 first displays the user interface 506 as shown in FIG. 9A. The user checks the three-dimensional model A4 generated during the previous image capture using a three-dimensional viewer, for example. The three-dimensional model A4 may be a point cloud or a mesh.

[0160] The user operates an operating device, for example, to move the display of the three-dimensional model A4 included in the user interface 506 so that the three-dimensional model A4 can be viewed from a position desired by the user, as shown in the user interface 507 illustrated in Fig. 9B. For example, the UI unit 160 determines the position of the viewpoint when the three-dimensional model A4 displayed on the user interface 507 is viewed as illustrated in Fig. 9B as a position designated by the user, and accepts this position as input from the user. The search range calculation unit 150 calculates the search range using this input position.

[0161] In this way, the user may operate the viewer to move the viewpoint to the current position of the photographer using the imaging device 300, and the candidate frame selection device 100 may set the frame closest to this viewpoint and the frame associated with it as the search range.

[0162] 9C, the user interface 508 may include an image (icon) such as a selection position A5 indicating the position of the viewpoint. For example, the user may move the three-dimensional model A4 by manipulating the camera image indicated as the selection position A5. In other words, the UI unit 160 may change the display of the three-dimensional model A4 based on the operation of the camera image indicated as the selection position A5 received from the user.

[0163] In this way, the user may operate the camera on the viewer and move the camera to the current position of the photographer using the imaging device 300, and the candidate frame selection device 100 may use the frame closest to this camera and related frames as its search range.

[0164] 10 is a diagram showing a seventh example of a user interface displayed when the candidate frame selection device 100 according to the embodiment accepts a position specification. Fig. 10 shows a modified example of the user interface 508 shown in Fig. 9(c).

[0165] As shown in the user interface 509 of FIG. 10, in this example, in addition to the display included in the user interface 508, the UI unit 160 further displays frame A6, which is closest to the frame that would be generated if the camera shown schematically as the selected position A5 captured an image, from among the multiple frames acquired as a frame group.

[0166] In this way, the candidate frame selection device 100 may display the frame closest to the current position of the photographer while the user is operating the viewer.

[0167] FIG. 11 is a diagram showing a first example of a user interface displayed when the candidate frame selection device 100 according to the embodiment accepts the designation of a frame.

[0168] 11A, the user interface 510 shown in this example includes an overhead view that schematically shows the map as seen from above. The user interface 510 also includes information indicating the position and orientation of the image capture device 300. For example, the display (information) indicating the position and orientation of the image capture device 300 is indicated by a triangle. Specifically, in this example, the UI unit 160 displays the user interface 510 so that the display indicating the position and orientation of the image capture device 300 included in the calculated search range is indicated by a solid triangle, the display indicating the position and orientation of the image capture device 300 not included in the calculated search range is indicated by a dashed triangle, and the display indicating the position and orientation of the image capture device 300 when the current frame was captured is indicated by a dotted triangle.

[0169] The display of the position and orientation of the image capturing device 300 when the current frame was captured is the current position and orientation of the image capturing device 300 estimated by the estimation device 200 .

[0170] 11B, the UI unit 160 displays a plurality of candidate frames and a current frame together with the user interface 510. For example, the UI unit 160 displays frames corresponding to the position and orientation of the image capture device 300 included in the search range as candidate frames. For example, the UI unit 160 also displays recommended values ​​for each candidate frame.

[0171] The UI unit 160 may display all frames corresponding to the position and orientation of the image capture device 300 included in the search range as candidate frames, or may display a predetermined number of frames, such as three frames, in descending order of recommended value as candidate frames. Alternatively, frames with a value less than the predetermined recommended value may not be displayed. The predetermined recommended value may be determined arbitrarily in advance and is not particularly limited.

[0172] For example, the user selects one candidate frame from the displayed plurality of candidate frames.

[0173] For example, when the UI unit 160 receives the selection of one candidate frame, it may display the current frame, the selected candidate frame (the selected frame shown in FIG. 11(c)), and frames similar to the selected frame in order of decreasing similarity from all frames acquired by the data input unit 120, as shown in FIG. 11(c). In this case, for example, the candidate frame selection device 100 calculates the similarity between the selected frame and each of all frames acquired by the data input unit 120. For example, the user may reselect a frame from the multiple displayed frames. In this way, the candidate frame selection device 100 may display frames similar to the frame selected by the user in order of decreasing similarity from all frames, and then allow the user to select a frame.

[0174] In this way, the candidate frame selection device 100 visualizes the positional relationship between the frame and the map, and assists the user in selecting a candidate frame. Note that, for example, if the candidate frame selection device 100 receives information from the user indicating that the search range is incorrect, it may return to the process of selecting a search range. Alternatively, the user may select (reselect) a candidate frame from all frames.

[0175] [Variation 1] In the above example, the candidate frame selection device 100 selects candidate frames by accepting designation of candidate frames from the user. The candidate frame selection device 100 may select candidate frames by itself without accepting designation of candidate frames from the user.

[0176] 12 is a flowchart showing a second example of the processing procedure of the candidate frame selection device 100 according to the embodiment. In the explanation of FIG. 11, the same steps as those in FIG. 3 are denoted by the same reference numerals, and the explanation may be simplified or omitted.

[0177] First, the data input unit 120 acquires a frame group, position and orientation information indicating the position and orientation of the image capturing device 300, and a map (map information) from the estimation device 200, and stores them in the storage unit 110 (S201).

[0178] Next, the UI unit 160 causes the display device to display the frame group acquired from the estimation device 200 (S202).

[0179] Next, the search range calculation unit 150 calculates a search range based on the input from the user, in other words, based on the acquired position information (S203).

[0180] Next, the recommended value calculation unit 140 calculates a recommended value for each frame within the search range (S205). Specifically, the recommended value calculation unit 140 calculates a recommended value for each frame corresponding to the position and orientation of the image capture device 300 included in the search range.

[0181] In this way, in this example, the process of step S204 in which the data input unit 120 acquires the current frame and the current position and orientation of the image capture device 300 is not performed.

[0182] Next, the output unit 130 selects a frame based on the calculated recommendation value for each frame, and outputs the selected frame as a candidate frame to the estimation device 200 (S207A). For example, the output unit 130 outputs the frame with the highest recommendation value among the calculated multiple recommendation values ​​to the estimation device 200 as a candidate frame.

[0183] In this example, the UI unit 160 does not perform the process of step S206 in which it displays each frame in the search range and each frame's recommended values ​​on the display device, and does not accept the selection of candidate frames from the user.

[0184] 12 is completed, the image capturing device 300 resumes capturing images and transmits the current frame to the estimation device 200. The estimation device 200 aligns the current frame received from the image capturing device 300 with the candidate frame received from the candidate frame selection device 100, and resumes calculating the position and orientation of the image capturing device 300 and creating a map.

[0185] As described above, in this example, the candidate frame selection device 100 selects one candidate frame from among a plurality of candidate frames based on the recommendation value, and outputs the selected candidate frame to the estimation device 200. For example, the candidate frame selection device 100 first calculates a frame range (search range) based on input from the user, and then selects the frame with the highest recommendation value within that frame range as the candidate frame.

[0186] This allows the candidate frame selection device 100 to automatically select candidate frames without receiving a selection from the user.

[0187] [Variation 2] In the above example, the candidate frame selection device 100 acquires position information indicating the position where imaging of the current frame begins, and selects a candidate frame corresponding to the current frame. The candidate frame selection device 100 may select a base frame from one of multiple non-consecutive frame groups stored in the storage unit 110, and select a candidate frame corresponding to the base frame from the other frame group. The base frame is an example of a second frame.

[0188] For example, the data input unit 120 acquires a plurality of new frames different from the acquired plurality of frames, and acquires position information indicating the position at which a reference frame among the newly acquired plurality of frames was captured.

[0189] FIG. 13 is a flowchart illustrating a third example of the processing procedure of the candidate frame selection device 100 according to the embodiment.

[0190] First, the data input unit 120 acquires a frame group, position and orientation information indicating the position and orientation of the imaging device 300, and a map from the estimation device 200, and stores the acquired data as first data in the storage unit 110 (S301). Note that the data input unit 120 may store all of the acquired frame group in the storage unit 110, or may store only a plurality of key frames from the acquired frame group in the storage unit 110.

[0191] Next, the data input unit 120 again acquires data from the estimation device 200 that is different from the first data acquired in step S301, such as a frame group, position and orientation information indicating the position and orientation of the imaging device 300, and a map, and stores the acquired new data in the storage unit 110 as second data (S302).

[0192] The group of frames in the first data is an example of a plurality of consecutive first frames, and the frame in the second data is an example of a plurality of consecutive fifth frames.

[0193] Next, the UI unit 160 displays the frame group of the first data acquired from the estimation device 200 on the display device (S303). The UI unit 160 may display the acquired frame group of the first data, i.e., multiple frames of the first data, or may display a map (three-dimensional model) of the first data created based on the frame group of the first data. That is, the UI unit 160 may display the frame group of the first data in two dimensions or three dimensions. This allows the UI unit 160 to accept input from the user (e.g., specification of a position, specification of a search range, or specification of a frame).

[0194] Next, the search range calculation unit 150 calculates a search range based on the input from the user, in other words, based on the acquired position information (S304).

[0195] Next, the UI unit 160 displays on the display device the frames of the search range in the frame group of the first data acquired from the estimation device 200 and the frame group of the second data (S305), thereby accepting input (designation of the reference frame) from the user.

[0196] Next, the data input unit 120 acquires the position and orientation of the image capturing device 300 corresponding to the reference frame (specifically, position and orientation information indicating the position and orientation of the image capturing device 300 when the reference frame was captured) (S306).

[0197] Next, the recommendation value calculation unit 140 calculates a recommendation value for each frame in the search range (S307). For example, the recommendation value calculation unit 140 calculates the similarity between the reference frame and each frame (specifically, a plurality of candidate frames) included in the search range, and calculates a recommendation value for each frame based on the calculated similarity.

[0198] Next, the UI unit 160 displays the search range (specifically, information indicating the search range), each frame of the search range (specifically, multiple candidate frames included in the search range), the reference frame, and each recommended value for each frame of the search range on the display device (S308), thereby accepting input (designation of candidate frames) from the user.

[0199] Next, the output unit 130 selects a candidate frame based on the input from the user, and outputs the selected candidate frame, the first data, and the second data to the estimation device 200 (S309). Specifically, the output unit 130 transmits information indicating the candidate frame selected by the user to the estimation device 200.

[0200] As a result, the estimation device 200 uses a candidate frame (specifically, a candidate frame selected by the user) to align the map in the first data with the map in the second data, combine these maps, and optimize the entire combined map.

[0201] 14A and 14B are diagrams illustrating frames displayed when the candidate frame selection device 100 according to an embodiment accepts the selection of a base frame. (a) of Fig. 14 is a diagram schematically illustrating frames in a search range in a frame group of the first data. (b) of Fig. 14 is a diagram schematically illustrating a frame group of the second data.

[0202] For example, in step S305, the UI unit 160 causes the display device to display frames in the search range in the frame group of the first data and some or all of the frames in the frame group of the second data.

[0203] The user selects a frame captured at the same location as the frame in the search range as a reference frame from the group of frames in the second data by operating a user interface displayed on the display device, for example. The UI unit 160 then selects the selected frame as the reference frame.

[0204] This allows the user to select a frame from the second data that is suitable for combining the map of the first data with the map of the second data while checking the frames in the search range of the first data.

[0205] If the search range is incorrect, the display device may return to the search range selection screen. For example, when UI unit 160 acquires information from the user indicating that the search range is incorrect, UI unit 160 may change the image displayed on the display device from the image displayed in step S305 to the image displayed in step S303.

[0206] FIG. 15 is a diagram showing a second example of a user interface displayed when the candidate frame selection device 100 according to the embodiment accepts the designation of a frame.

[0207] 15A , the user interface 511 shown in this example includes an overhead view that schematically shows the map in the first data as viewed from above. The user interface 511 also includes information indicating the position and orientation of the image capture device 300. For example, the display (information) indicating the position and orientation of the image capture device 300 is indicated by a triangle. Specifically, in this example, the UI unit 160 displays the user interface 511 so that the display indicating the position and orientation of the image capture device 300 included in the calculated search range is indicated by a solid triangle, the display indicating the position and orientation of the image capture device 300 not included in the calculated search range is indicated by a dashed triangle, and the display indicating the position and orientation of the image capture device 300 when the reference frame was captured is indicated by a dotted triangle.

[0208] Note that the display of the position and orientation of the imaging device 300 when the reference frame was captured displayed here is the display of the position and orientation of the imaging device 300 when the imaging device 300 captured the reference frame, estimated by the estimation device 200.

[0209] 15B, the UI unit 160 displays a plurality of candidate frames and a reference frame together with the user interface 511. For example, the UI unit 160 displays frames corresponding to the position and orientation of the image capture device 300 included in the search range as candidate frames. For example, the UI unit 160 also displays the recommended value for each candidate frame.

[0210] The UI unit 160 may display all frames corresponding to the position and orientation of the image capture device 300 included in the search range as candidate frames, or may display a predetermined number of frames, such as three frames, in descending order of recommended value as candidate frames. Alternatively, frames with a value less than the predetermined recommended value may not be displayed. The predetermined recommended value may be determined arbitrarily in advance and is not particularly limited.

[0211] For example, the user selects one candidate frame from the displayed plurality of candidate frames, and the selected candidate frame is output to the estimation device 200, for example.

[0212] For example, when the UI unit 160 receives the selection of one candidate frame, it may display the reference frame, the selected candidate frame (the selected frame shown in FIG. 15(c)), and frames similar to the selected frame in order of decreasing similarity from all frames acquired by the data input unit 120, as shown in FIG. 15(c). In this case, for example, the candidate frame selection device 100 calculates the similarity between the selected frame and each of all frames acquired by the data input unit 120. For example, the user may reselect a frame from the multiple displayed frames. In this way, the candidate frame selection device 100 may display frames similar to the frame selected by the user in order of decreasing similarity from all frames, and then allow the user to select a frame.

[0213] In this way, the candidate frame selection device 100 visualizes the positional relationship between the frame and the map, and assists the user in selecting a candidate frame. The frame selected by the user is output to the estimation device 200 as the final candidate frame. Note that, for example, if the candidate frame selection device 100 receives information from the user indicating that the search range is incorrect, it may return to the process of selecting a search range. Alternatively, the user may select (reselect) a candidate frame from all frames.

[0214] [Summary] Fig. 16 is a block diagram showing the configuration of an information processing device 10 according to an embodiment. Fig. 17 is a flowchart showing an information processing method according to an embodiment.

[0215] The candidate frame selection device 100 described above is a specific example of the information processing device 10. For example, the information processing device 10 includes a circuit 11 and a memory 12. The circuit 11 is connected to the memory 12, and in operation, executes the information processing method shown in Fig. 17 using the memory 12. That is, for example, the candidate frame selection device 100 executes the information processing method shown in Fig. 17.

[0216] First, the information processing device 10 acquires a plurality of consecutive first frames (S10). The plurality of consecutive first frames are, for example, frames captured consecutively in time by the imaging device 300 at a predetermined time interval. For example, the imaging device 300 generates a plurality of first frames by capturing images every 10 fps (frames per second). The information processing device 10 acquires the plurality of consecutive first frames thus generated.

[0217] Next, the information processing device 10 stores a plurality of key frames from the plurality of first frames in the memory 12 (S20). The method for selecting the plurality of key frames may be determined arbitrarily and is not particularly limited. For example, a predetermined number of first frames may be selected as key frames from the plurality of first frames. For example, the plurality of key frames may be selected from the plurality of first frames such that the number of key frames is less than the number of the plurality of first frames.

[0218] Next, the information processing device 10 acquires position information indicating a position at which a second frame not consecutive to the plurality of first frames will be captured (S30). A second frame not consecutive to the plurality of first frames is, for example, a frame captured not temporally consecutive to the plurality of first frames. Specifically, when the first frames are captured at a predetermined time interval, a second frame not consecutive to the plurality of first frames is a frame captured at a time interval longer than the predetermined time interval from the last captured first frame among the plurality of first frames. The position information is acquired, for example, by receiving, from the user, a designation of a position at which capturing of the second frame will begin. The second frame is, for example, the current frame or the reference frame.

[0219] Next, the information processing device 10 selects a third frame corresponding to the second frame from among the multiple key frames based on the acquired position information (S30). That is, the information processing device 10 selects a third frame to be aligned with the second frame from among the multiple key frames. For example, the information processing device 10 outputs the selected third frame to the estimating device 200. For example, the estimating device 200 aligns the acquired third frame with the second frame, and generates a further 3D model based on the second frame and frames acquired after the second frame, so as to update the 3D model generated based on the multiple first frames.

[0220] As a result, when, for example, image capture of frames used to generate a three-dimensional model is resumed, the information processing device 10 can select, from among multiple key frames used to generate the already-generated three-dimensional model, a key frame (third frame) suitable for alignment with the frame (second frame) generated when image capture is resumed, based on position information indicating the position at which image capture was resumed. For example, for alignment with the second frame, selecting a frame captured at a position close to the position at which the second frame was captured can achieve accurate alignment. Therefore, by using the position information, it is possible to prevent a frame that is not suitable for alignment, such as one captured at a position far from the position at which the second frame was captured, from being selected as the frame to be aligned with the second frame. For example, by using a user interface that allows the user to specify the start position of additional image capture, the information processing device can limit the search to the position specified by the user, thereby preventing the selection of an incorrect frame. This can improve the accuracy of SLAM after image capture is resumed. In this way, it is possible to accurately align an already-generated three-dimensional model with a three-dimensional model generated using frames captured after the three-dimensional model was generated. Therefore, the accuracy of the three-dimensional model can be improved using frames generated when imaging is resumed.

[0221] Furthermore, for example, the information processing device 10 extracts a plurality of fourth frames from among a plurality of key frames based on the position information, and selects a third frame from among the plurality of fourth frames.

[0222] This allows, for example, multiple fourth frames that are close to the position indicated by the position information to be extracted from multiple key frames, making it easier to select key frames that are suitable for alignment with the generated frames when image capture is resumed.

[0223] Also, for example, the information processing device 10 acquires a second frame, extracts one or more fourth frames from a plurality of fourth frames whose similarity to the second frame is higher than a predetermined threshold, and selects a third frame from the one or more fourth frames.

[0224] High frame similarity is considered more suitable for frame alignment than low frame similarity, which makes it easier to select key frames suitable for alignment with the generated frames when imaging resumes.

[0225] Furthermore, for example, the information processing device 10 presents a plurality of fourth frames and selects the third frame by accepting designation of the third frame from among the plurality of fourth frames.

[0226] For example, the information processing device 10 presents a plurality of fourth frames to the user by controlling a display device such as a monitor, and also receives a selection of a third frame from the user via an input device such as a mouse, a keyboard, or a touch panel.

[0227] This makes it easier for the user to select a key frame that is suitable for alignment with the generated frame when image capture is resumed. Note that the information processing device 10 may be provided with a display device and an input device.

[0228] Furthermore, for example, the information processing device 10 presents recommended values ​​for each of the multiple fourth frames when the multiple fourth frames are used for alignment with the second frame.

[0229] This makes it easier for the user to select a key frame that is suitable for alignment with the generated frame when image capture is resumed.

[0230] Also, for example, the recommended value is calculated by comparing values ​​relating to a plurality of fourth frames with a value relating to the second frame.

[0231] This allows a recommended value indicating the degree of suitability for alignment to be calculated.

[0232] Also, for example, the recommended value is calculated based on at least one of the frame similarity between each of the multiple fourth frames and the second frame, the degree of blur of each of the multiple fourth frames, and the acceleration of the imaging device when the multiple fourth frames were captured.

[0233] From this, a recommended value can be calculated.

[0234] Also, for example, the recommended value is received and the received recommended value is presented.

[0235] In this case, for example, the information processing device 10 receives information indicating recommended values ​​from an external server with which the information processing device 10 can communicate. For example, the imaging device 300 transmits captured frames to the server. The server calculates recommended values ​​for each frame based on the received frames and transmits information indicating the calculated recommended values ​​to the information processing device 10.

[0236] This allows the information processing device 10 to easily obtain the recommended value.

[0237] Furthermore, for example, the information processing device 10 calculates a recommended value and presents the calculated recommended value.

[0238] This allows the recommended value to be calculated without using an external device such as a server.

[0239] Furthermore, for example, the information processing device 10 may present a three-dimensional model generated based on a plurality of first frames and receive a position specification, thereby acquiring position information.

[0240] This allows the user to easily specify the position at which to resume imaging.

[0241] Furthermore, for example, the position information indicates the position at which the second frame, which is the first frame to be captured among a plurality of consecutive fifth frames including the second frame, is captured.

[0242] The second frame here is, for example, the current frame mentioned above.

[0243] This allows, for example, when the capture of frames used to generate a three-dimensional model is resumed, to select a key frame (third frame) suitable for alignment with the frame (second frame) generated when capture is resumed from among the multiple key frames used to generate the three-dimensional model that has already been generated, based on position information indicating the position at which capture began among the multiple frames captured after the resumption.

[0244] Furthermore, for example, the information processing device 10 stores a plurality of consecutive fifth frames, including the second frame, in the memory 12, and selects the second frame from among the plurality of fifth frames.

[0245] The first frames may be, for example, a group of frames in the first data, the fifth frames may be, for example, a group of frames in the second data, and the second frames may be, for example, the reference frames.

[0246] This makes it possible to accurately align an already generated three-dimensional model with a three-dimensional model generated using a frame group captured after the three-dimensional model was generated, even when there are multiple frame groups consisting of consecutive frames.

[0247] Furthermore, for example, the information processing device 10 presents a plurality of fifth frames and selects the second frame by accepting designation of the second frame from among the plurality of fifth frames.

[0248] For example, the information processing device 10 presents a plurality of fifth frames to a user by controlling a display device such as a monitor, and also receives a selection of a second frame from a user via an input device such as a mouse, a keyboard, or a touch panel.

[0249] This allows the user to specify the second frame to be used for alignment with the key frame.

[0250] Furthermore, an information processing method according to one aspect of the present disclosure is an information processing method executed by a computer such as information processing device 10, which acquires a plurality of consecutive first frames (S10), stores a plurality of key frames from among the plurality of first frames in memory 12 (S20), acquires positional information indicating a position at which a second frame that is not consecutive to the plurality of first frames is captured (S30), and selects a third frame corresponding to the second frame from among the plurality of key frames based on the positional information (S40).

[0251] This allows the same effects as those of the information processing device 10 to be obtained.

[0252] Furthermore, for example, a program according to one aspect of the present disclosure is a program for causing a computer to execute an information processing method according to one aspect of the present disclosure.

[0253] This provides the same effects as those of the information processing device 10.

[0254] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0255] Other Embodiments Although the three-dimensional model generation system 400 and the information processing device 10 according to the embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments.

[0256] For example, the information processing device 10 may have only the configuration of the candidate frame selection device 100, or may have the configuration of the imaging device 300 in addition to the configuration of the candidate frame selection device 100, or may have the configuration of the estimation device 200 in addition to the configuration of the candidate frame selection device 100.

[0257] Furthermore, for example, each processing unit included in the information processing device 10 according to the above embodiment is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0258] Furthermore, the integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor, for example. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI may also be used.

[0259] Furthermore, for example, in each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0260] Furthermore, for example, the present disclosure may be realized as an information processing method executed by the three-dimensional model generation system 400 or the information processing device 10, etc.

[0261] Furthermore, for example, the division of functional blocks in the block diagram is merely an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0262] Furthermore, for example, the order in which each step is performed in the flowchart is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. Furthermore, some of the steps may be performed simultaneously (in parallel) with other steps.

[0263] The three-dimensional model generation system 400 and the information processing device 10 according to one or more aspects have been described above based on the embodiments, but the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by a person skilled in the art to the embodiments and forms constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0264] The present disclosure is applicable to a system for generating a three-dimensional model.

[0265] REFERENCE SIGNS LIST 10 Information processing device 11 Circuit 12 Memory 100 Candidate frame selection device 110, 310 Storage unit 120 Data input unit 130 Output unit 140 Recommended value calculation unit 150 Search range calculation unit 160 UI unit 170, 320 Control unit 200 Estimation device 300 Imaging device 330 Optical system 340 Image sensor 400 Three-dimensional model generation system 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511 User interface A1, A5 Selection position A2 Selection range A3 Selected object A4 Three-dimensional model A6 Frame

Claims

1. An information processing device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, acquires a plurality of consecutive first frames; stores a plurality of key frames from the plurality of first frames in the memory; acquires position information indicating a position at which a second frame not consecutive to the plurality of first frames is captured; and selects a third frame corresponding to the second frame from the plurality of key frames based on the position information.

2. The information processing device according to claim 1, wherein a plurality of fourth frames are extracted from the plurality of key frames based on the position information, and the third frame is selected from the plurality of fourth frames.

3. An information processing device as described in claim 2, which acquires the second frame, extracts one or more fourth frames from the plurality of fourth frames whose similarity to the second frame is higher than a predetermined threshold, and selects the third frame from the one or more fourth frames.

4. The information processing device according to claim 2, wherein the information processing device presents the plurality of fourth frames, and selects the third frame by accepting designation of the third frame from among the plurality of fourth frames.

5. The information processing device according to claim 2, wherein recommended values ​​for each of the plurality of fourth frames when the plurality of fourth frames are used for alignment with the second frame are presented.

6. The information processing device according to claim 5, wherein the recommended value is calculated by comparing values ​​relating to the plurality of fourth frames with a value relating to the second frame.

7. The information processing device of claim 5, wherein the recommended value is calculated based on at least one of the frame similarity between each of the plurality of fourth frames and the second frame, the degree of blur of each of the plurality of fourth frames, and the acceleration of the imaging device when the plurality of fourth frames were captured.

8. The information processing device according to claim 5, further comprising: receiving the recommended value; and presenting the received recommended value.

9. The information processing device according to claim 5, wherein the recommended value is calculated and the calculated recommended value is presented.

10. The information processing device according to claim 1, wherein a three-dimensional model generated based on the plurality of first frames is presented, and the position information is acquired by accepting the position specification.

11. The information processing device according to claim 1, wherein the position information indicates a position at which the second frame, which is the first of a plurality of consecutive fifth frames including the second frame, is captured.

12. The information processing device according to claim 1, wherein a plurality of consecutive fifth frames including the second frame are stored in the memory, and the second frame is selected from the plurality of fifth frames.

13. The information processing device according to claim 12, wherein the second frame is selected by presenting the plurality of fifth frames and accepting designation of the second frame from among the plurality of fifth frames.

14. An information processing method comprising: acquiring a plurality of consecutive first frames; storing a plurality of key frames from among the plurality of first frames in a memory; acquiring position information indicating a position at which a second frame not consecutive to the plurality of first frames is captured; and selecting a third frame corresponding to the second frame from among the plurality of key frames based on the position information.

15. A program for causing a computer to execute the information processing method according to claim 14.

Images