Three-dimensional model generation device, three-dimensional model generation method, and program

Abstract

To provide a three-dimensional model generation device capable of achieving further improvements.SOLUTION: A three-dimensional model generation device 102 includes a memory 12 and a circuit 11 connected to the memory 12. The circuit 11 is configured to perform the following steps: acquiring multiple consecutive images using the memory 12 (S201); generating a three-dimensional model including multiple three-dimensional points, each having position information by using multiple images (S202); determining whether to correct the accumulated error of the three-dimensional model (S203); and when determining to perform the correction (Yes in S203), displaying a piece of first information to instruct a first position posture for imaging a first image for correction, on a display part (S204).SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present disclosure relates to a three-dimensional model generation device, a three-dimensional model generation method, and a program.

Background Art

[0002] A method is known in which the position and orientation of a camera are detected from a plurality of images using SLAM (Simultaneous Localization And Mapping), and a three-dimensional model is generated. At this time, a method of reducing the accumulated error using loop closing (also called a loop closer) is known (see, for example, Patent Document 1 and Patent Document 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In such a three-dimensional model generation device and three-dimensional model generation method, further improvement is desired. Therefore, the present disclosure provides a three-dimensional model generation device or a three-dimensional model generation method capable of achieving further improvement.

Means for Solving the Problems

[0005] A three-dimensional model generation device according to one aspect of the present disclosure includes a memory and a circuit connected to the memory. The circuit uses the memory to acquire a plurality of consecutive images, and uses the plurality of images to generate a three-dimensional model including a plurality of three-dimensional points each having position information, determines whether to correct the accumulation error of the three-dimensional model, and when it is determined to perform the correction, causes a display unit to display first information for instructing a first position and orientation for taking a first image for performing the correction.

Advantages of the Invention

[0006] The present disclosure can provide a three-dimensional model generation device or a three-dimensional model generation method capable of achieving further improvement.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

DETAILED DESCRIPTION OF THE INVENTION

[0008] A three-dimensional model generation apparatus according to one aspect of the present disclosure includes a memory and a circuit connected to the memory. The circuit uses the memory to acquire a plurality of consecutive images, and uses the plurality of images to generate a three-dimensional model including a plurality of three-dimensional points each having position information, determines whether to correct the accumulation error of the three-dimensional model, and when it is determined to perform the correction, causes a display unit to display first information for instructing a first position and orientation for taking a first image for performing the correction.

[0009] According to this, the three-dimensional model generation apparatus can support taking a first image for correcting the accumulation error. Therefore, by appropriately performing the correction process, the accuracy of the generated three-dimensional model can be improved. In this way, the three-dimensional model generation apparatus can achieve further improvement.

[0010] For example, the first information may be a second image included in the plurality of images and taken in the first position and orientation.

[0011] According to this, the user can determine the first position and orientation while referring to the second image.

[0012] For example, the circuit may display the second image on the display unit by superimposing it on the currently captured image.

[0013] According to this, the user can capture the first image by capturing in a state where the currently captured image matches the second image.

[0014] For example, the circuit may further display second information indicating the accumulation error on the display unit.

[0015] According to this, the user can determine whether to correct the accumulation error by referring to the second information.

[0016] For example, as the second information, the circuit may display the three-dimensional model on the display unit by superimposing a third image seen from the current position and orientation on the currently captured image.

[0017] According to this, the user can recognize the difference between the currently captured image and the third image. Also, the user can determine the necessity of correcting the accumulation error based on the difference.

[0018] For example, as the second information, the circuit may further display on the display unit an area where there is a difference between the third image and the currently captured image.

[0019] According to this, the user can determine the necessity of correcting the accumulation error based on the third information.

[0020] For example, the circuit may further calculate the amount of the accumulation error based on the difference between the third image and the currently captured image, and display third information indicating the calculated amount on the display unit.

[0021] According to this, the user can recognize the magnitude (degree) of the difference between the currently captured image and the third image. Further, the user can determine the necessity of correcting the accumulated error based on the magnitude of the difference.

[0022] For example, the circuit may further display, on the display unit, a user interface for the user to instruct whether to perform the correction together with the second information, and when an instruction to perform the correction is received, may display the first information on the display unit.

[0023] According to this, correction can be performed based on the determination of the user who referred to the second information.

[0024] For example, the circuit may further display, on the display unit, a user interface for the user to select the second image from a plurality of fourth images included in the plurality of images.

[0025] According to this, the user can select the second image.

[0026] For example, the circuit may further select the plurality of fourth images from the plurality of images based on feature points included in each of the plurality of images.

[0027] According to this, the accuracy of feature point matching for the second image can be improved, so the accuracy of correcting the accumulated error can be improved.

[0028] For example, the circuit may further select the second image from the plurality of images based on feature points included in each of the plurality of images.

[0029] According to this, the accuracy of feature point matching for the second image can be improved, so the accuracy of correcting the accumulated error can be improved.

[0030] For example, when the first information is displayed on the display unit, (1) when the first image captured from the first position and orientation is acquired, the correction is performed using the acquired first image, and (2) when the sixth image captured at the position and orientation of the fifth image captured other than the first position and orientation among the plurality of images is acquired, the correction of the accumulation error may not be performed using the acquired sixth image.

[0031] According to this, it is possible to suppress performing correction using an inappropriate image.

[0032] For example, the circuit estimates an estimated position and orientation, which is the position and orientation at which each of the plurality of images was captured, using the plurality of images, generates the three-dimensional model using the estimated estimated position and orientation, and in the correction of the accumulation error, the accumulation error may be corrected based on the error between the estimated position and orientation of the second image captured at the first position and orientation included in the plurality of images and the estimated position and orientation of the first image.

[0033] For example, the circuit causes the display unit to display the current three-dimensional model and the past three-dimensional model as the second information, the circuit further receives an operation for aligning the positions of the current three-dimensional model and the past three-dimensional model, and in the correction, the correction may be performed using information obtained by the operation for aligning the positions as auxiliary information.

[0034] According to this, it is possible to reduce the processing amount of the correction of the accumulation error and improve the accuracy.

[0035] A three-dimensional model generation method according to an aspect of the present disclosure acquires a plurality of consecutive images, generates a three-dimensional model including a plurality of three-dimensional points each having position information using the plurality of images, determines whether to correct an accumulation error of the three-dimensional model, and when it is determined to perform the correction, causes the display unit to display first information for instructing a first position and orientation at which a first image for performing the correction is captured.

[0036] According to this, the three-dimensional model generation method can support the shooting of images for correcting the accumulation error. Therefore, by appropriately performing the correction process, the accuracy of the generated three-dimensional model can be improved. In this way, the three-dimensional model generation method can achieve further improvement.

[0037] Also, a program according to an aspect of the present disclosure is a program for causing a computer to execute the three-dimensional model generation method.

[0038] Note that these general or specific aspects may be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0039] Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, a more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and duplicate descriptions of substantially the same configurations may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art.

[0040] Note that the inventors provide the accompanying drawings and the following description for those skilled in the art to fully understand the present disclosure, and do not intend to limit the subject matter described in the claims thereby.

[0041] (Embodiment) [1. Configuration] First, the configuration of the three-dimensional model generation system according to the present embodiment will be described. FIG. 1 is a block diagram showing the configuration of the three-dimensional model generation system according to the present embodiment. The three-dimensional model generation system is a system that generates a three-dimensional model from a plurality of images, and includes an imaging device 101, a three-dimensional model generation device 102, and a display device 103.

[0042] The imaging device 101 generates a plurality of images and a plurality of sensor information, and outputs the generated plurality of images and the plurality of sensor information to the three-dimensional model generation device 102. This imaging device 101 includes a camera 111 and a sensor 112. The camera 111 captures a subject (object) from different viewpoints (shooting positions and shooting directions), and outputs the obtained plurality of images (frames) to the three-dimensional model generation device 102. For example, by shooting while the imaging device 101 moves, a plurality of images are captured.

[0043] The sensor 112 includes, for example, at least one of a distance sensor (depth sensor), a laser sensor (e.g., LiDAR), and an IMU (Inertial Measurement Unit). The IMU includes, for example, at least one of an acceleration sensor, a rotational angular acceleration sensor, and a gyro sensor. The sensor 112 outputs sensor information corresponding to each of the plurality of images to the three-dimensional model generation device 102. Here, the sensor information includes, for example, at least a part of a depth image obtained by a distance sensor, point cloud data obtained by a laser sensor, and acceleration information obtained by an IMU. Note that the imaging device 101 may not include the sensor 112.

[0044] The three-dimensional model generation device 102 generates a three-dimensional model using a plurality of images and a plurality of sensor information. This three-dimensional model generation device 102 includes an acquisition unit 121, a storage unit 122, a camera position and orientation estimation unit 123, a three-dimensional model generation unit 124, an accumulation error detection unit 125, a user guide unit 126, an output unit 127, and a control unit 128.

[0045] The acquisition unit 121 acquires a plurality of images and a plurality of sensor information from the imaging device 101. The storage unit 122 stores the plurality of images and the plurality of sensor information acquired by the acquisition unit 121. The storage unit 122 also stores camera position and orientation information, map information, a three-dimensional model, etc., which will be described later.

[0046] The camera position and orientation estimation unit 123 generates camera position and orientation information indicating the position and orientation (direction) of the imaging device 101 at the time of shooting each image, using a plurality of images. Here, the orientation of the imaging device 101 indicates at least one of the shooting direction of the imaging device 101 and the inclination of the imaging device 101. The shooting direction of the imaging device 101 is the direction of the optical axis of the imaging device 101. The inclination of the imaging device 101 is the rotation angle around the optical axis of the imaging device 101 from the reference orientation.

[0047] Also, the camera position and orientation estimation unit 123 generates map information indicating a three-dimensional point cloud indicating a three-dimensional position together with the camera position and orientation information.

[0048] The three-dimensional model generation unit 124 generates a three-dimensional model of the subject based on a plurality of images, the camera position and orientation information, the map information, and the sensor information.

[0049] The accumulated error detection unit 125 detects the accumulated error (also called the cumulative error) of the three-dimensional model and generates accumulated error information indicating the accumulated error. The user guide unit 126 generates guide information for supporting the movement of the user to the shooting position of the correction image used for the loop closing process.

[0050] The output unit 127 outputs the current image, the accumulated error information, the guide information, etc. to the display device 103. The control unit 128 controls each processing unit. For example, the control unit 128 controls each processing unit according to a user operation or the like.

[0051] The display device 103 displays the current image, accumulated error information, guide information, etc. sent from the three-dimensional model generation device 102. The display device 103 includes a display unit 131 and an operation reception unit 132. The display unit 131 displays the current image, accumulated error information, guide information, etc. The operation reception unit 132 receives the user's operation. For example, the operation reception unit 132 is a touch panel, or a keyboard and a mouse, etc. Note that the operation reception unit 132 may be a microphone that receives voice operations. Also, the display device 103 may include a speaker that performs voice output, a vibrator that conveys vibration to the user, etc. as means for presenting information to the user.

[0052] Note that the imaging device 101, the three-dimensional model generation device 102, and the display device 103 shown in FIG. 1 may be realized as a single device. Also, a plurality of processing units included in each device may be distributed among a plurality of devices.

[0053] For example, in a use case where the user takes a picture while moving the subject with a terminal (such as a smartphone, tablet terminal, or personal computer) they possess, the imaging device 101, the three-dimensional model generation device 102, and the display device 103 may be included in the terminal. Also, in another use case, the imaging device 101 may be included in a moving body (such as a vehicle, robot, or drone) operated by the user, and the three-dimensional model generation device 102 and the display device 103 may be included in the terminal possessed by the user. Also, in any case, some of the processing units included in each device may be included in other devices connected via a communication network, etc. to the terminal operated by the user, such as a server. For example, some or all of the processing units included in the three-dimensional model generation device 102 may be included in a server.

[0054] Also, the transmission and reception of data between the devices shown in FIG. 1 may be performed by any method such as wired communication or wireless communication. Also, these communications may be performed directly between devices, or may be performed indirectly via other communication devices or a server, etc. Also, these transmissions and receptions may be the transfer of data within a single device.

[0055] [2. Operation] Next, the operation of the three-dimensional model generation device 102 will be described. For example, the three-dimensional model generation system generates a three-dimensional model in real time based on the images captured by the imaging device 101.

[0056] FIG. 2 is a flowchart showing the operation of the three-dimensional model generation device 102. Here, the processing shown in FIG. 2 is repeated, for example, in units of one frame. Note that the processing shown in FIG. 2 may be repeated in units of a plurality of frames.

[0057] Also, the plurality of images may be a plurality of still images captured while the imaging device 101 is moving, or may be a plurality of images included in a moving image captured while the imaging device 101 is moving. When a moving image is used, the plurality of images do not have to be all the images included in the moving image, and may be main images (key frames) that are a part of the plurality of images included in the moving image. Here, the main image is, for example, an image extracted from a plurality of images included in the moving image at a predetermined time interval.

[0058] [2-1. Estimation of Camera Position and Orientation] First, the acquisition unit 121 acquires the current image (frame) and a plurality of sensor information from the imaging device 101 (S101). Next, the camera position and orientation estimation unit 123 generates camera position and orientation information and map information using a plurality of images including the current image (S102).

[0059] Note that the estimation method by which the camera position and orientation estimation unit 123 estimates the position and orientation of the imaging device 101 is not particularly limited. For example, the camera position and orientation estimation unit 123 may estimate the position and orientation of the plurality of imaging devices 101 using, for example, Visual-SLAM (Simultaneous Localization and Mapping), Structure-From-Motion, or ICP (Iterative Closest Point).

[0060] FIG. 3 is a diagram for explaining the camera position and orientation estimation process. In this figure, an example of performing the camera position and orientation estimation process using three frames F0, F1, and F2 is shown.

[0061] As shown in FIG. 3, the camera position and orientation estimation unit 123 performs feature point matching processing on a plurality of frames F0 to F2 captured by the imaging device 101. That is, the camera position and orientation estimation unit 123 extracts feature points of each of the plurality of frames F0 to F2, and extracts a set of similar points (corresponding points) that are similar among the plurality of extracted feature points between the plurality of frames. Next, the camera position and orientation estimation unit 123 estimates the position and orientation of the imaging device 101 of each frame using the set of extracted similar points.

[0062] Also, the map information indicates a plurality of map points that are three-dimensional points each indicating a three-dimensional position. The camera position and orientation estimation unit 123 generates map information by performing triangulation using the result of the feature point matching process and the camera position and orientation information. Note that the map information may include, in addition to the position information, information indicating the color of each map point and the surface shape around the map point (for example, information indicating a normal line), etc. The map information may be further corrected to high-precision information by performing an optimization process together with the camera position and orientation.

[0063] [2-2. Generation of 3D Model] Next, the three-dimensional model generation unit 124 generates a three-dimensional model of the subject based on a plurality of images including the current image and the camera position and orientation information (S103). For example, the three-dimensional model generation unit 124 generates a three-dimensional model using the map information. FIG. 4 is a diagram for explaining the three-dimensional model generation process in this case. The three-dimensional model generation unit 124 generates, for example, a mesh model represented by a plane with map points as vertices as the three-dimensional model as shown in FIG. 4.

[0064] Note that the three-dimensional model is not limited to the mesh model, and any three-dimensional model may be used. For example, the map information may be used as the three-dimensional model as it is. Further, when the imaging device 101 includes the sensor 112, the three-dimensional model may be generated using the sensor information obtained by the sensor 112 and the camera position and orientation information. For example, the three-dimensional model may be generated using the point cloud data obtained by LiDAR or the depth image obtained by the distance sensor. Further, a plurality of these map information, point cloud data, and depth information may be used. For example, the depth information may be estimated from the image using AI (artificial intelligence).

[0065] Further, the three-dimensional model may include, in addition to the position information, attribute information (information indicating color or normal), etc.

[0066] [2-3. Detection of Accumulated Error] Next, the accumulated error detection unit 125 detects the accumulated error of the current frame and displays the accumulated error information indicating the accumulated error on the display unit 131 (S104). Here, the accumulated error is the difference (error) between the three-dimensional model generated using the past frames and the three-dimensional model generated using the current frame. In other words, the accumulated error is the difference (error) between the estimated camera position and orientation of the current frame and the actual camera position and orientation.

[0067] FIG. 5 is a diagram for explaining this accumulation error. FIG. 5 shows an example in which the imaging device 101 captures a plurality of objects while moving. Also, in this example, the imaging device 101 starts imaging and moving from time t0, and at time t = x, it has returned to the same position as at time t = 0. Here, in the estimation of the camera position and orientation and the generation of the three-dimensional model, the camera position and orientation of a newly captured image are estimated using the previously estimated camera position and orientation of the image, and the three-dimensional model is updated (added). Therefore, as the number of images used increases (the imaging time becomes longer), there is a possibility that the error between the estimated camera position and orientation and three-dimensional model and the actual camera position and orientation and the position of the object increases. For example, as shown in FIG. 5, as time progresses, the difference (accumulation error) between the actual camera position and orientation (actual trajectory) and the estimated actual camera position and orientation (estimated trajectory) becomes larger.

[0068] The loop closing process is a process for correcting this accumulation error. By performing the loop closing process, the estimated camera position and orientation (estimated trajectory) are corrected to approach the actual camera position and orientation (actual trajectory). Also, based on the corrected camera position and orientation, the position information of the three-dimensional model is updated. For example, using the estimated camera position and orientation of the first image (image at t = 0) captured in the past and the second image (image at t = x) captured at the same position as the first image, the camera position and orientation of all images included in the range from t = 0 to t = x are corrected so that the camera position and orientation of the second image becomes the camera position and orientation of the first image. That is, in order to perform the loop closing process, an image captured at the same camera position and orientation as any of a plurality of images captured in the past is required.

[0069] Hereinafter, an example of the accumulation error information will be described. FIG. 6 is a diagram for explaining the generation of the accumulation error information. For example, as shown in FIG. 6, the accumulation error information is generated by projecting the past three-dimensional model onto the current image. Here, the past three-dimensional model is, for example, a three-dimensional model generated using a plurality of images from t = 0 to x - 1.

[0070] FIG. 7 is a diagram showing an example of displaying accumulated error information. For example, as shown in FIG. 7, as the accumulated error information, an image obtained by projecting a past three-dimensional model onto the current image is displayed. Thereby, the user can determine the necessity of loop closing processing based on the presence or absence and the magnitude of the difference between the current image shown in the image and the image obtained by projecting the past three-dimensional model. Note that the projected three-dimensional model may be displayed semi-transparently, for example. Alternatively, feature portions such as the outline of the subject may be extracted and only the feature portions may be displayed.

[0071] Further, the accumulated error detection unit 125 may calculate the accumulated error. For example, the accumulated error detection unit 125 calculates the difference in pixel values of each pixel between a first image viewed from the current camera position and orientation based on the current image or the current three-dimensional model, and a second image viewed from the current camera position and orientation of the past three-dimensional model, using the first image and the second image. Here, the pixel value is, for example, a depth value, a color value, a normal vector, or the like. The depth value of the first image may be obtained from the current three-dimensional model, for example, or when the depth image is included in the sensor information, the current depth image may be used as the first image. Alternatively, a depth image generated using a plurality of images including the current image may be used as the first image. Note that the current three-dimensional model is a three-dimensional model generated from a plurality of images including the current image (for example, images from t = x - 2 to t = x). Further, the depth value of the second image is generated from the past three-dimensional model.

[0072] When comparing color values, the first image is the current image, and the second image is, for example, generated from the past three-dimensional model.

[0073] The accumulated error information may include information indicating the difference calculated in this way. FIG. 8 is a diagram showing an example of the display of the accumulated error information in this case. For example, as shown in FIG. 8, the region with a difference may be highlighted. Also, in the example shown in FIG. 8, the magnitude of the difference is indicated by shading. Note that the highlighting method is not limited to this, and any method such as blinking may be used. Also, it is not always necessary to perform highlighting, and any method that allows the user to recognize the region with a difference is acceptable. Also, the number of gradations indicating the magnitude of the difference may be arbitrary. Also, instead of gradation display, the presence or absence of a difference (whether the difference is equal to or greater than a threshold value) may be indicated.

[0074] Also, in FIG. 8, although the region with a difference is being displayed, the object with a difference may be highlighted. Also, the calculation of the difference is not limited to the pixel unit, and may be in units of regions including a plurality of pixels. In this case, the total value, average value, median value, or maximum value of the differences (or absolute values of the differences) of the plurality of pixels included in the region may be calculated as the difference of the region.

[0075] Also, the difference of the entire current image may be calculated as the accumulated error. For example, as the difference of the entire image, the total value, average value, median value, or maximum value of the differences of a plurality of pixels may be calculated as the difference of the image. Alternatively, a weighted addition value, a weighted average value, etc. may be used. In this case, for example, a larger weight may be set for the pixels at the center of the image.

[0076] Also, weights may be set according to the accuracy of the three-dimensional model. For example, the higher the accuracy of a region, the greater the weight may be set. This accuracy may be determined based on, for example, the distance between the camera position of the image used to generate the three-dimensional model and the subject. For example, it may be determined that the closer the distance, the higher the accuracy. Alternatively, the accuracy may be determined based on the distance from the detected feature points, vertices, or edges. For example, it may be determined that the closer the distance, the higher the accuracy. Further, the accuracy may be determined based on whether or not there is a texture. For example, when there is a texture, it may be determined that the accuracy is high. Also, when depth images or point cloud data obtained by a laser sensor are used in the generation of the three-dimensional model, the accuracy of the three-dimensional model may be determined based on information indicating the accuracy of the sensor included in the additional information of the sensor information or the accuracy of the sensor result. Also, the accuracy may be determined by combining a plurality of these elements.

[0077] Also, in order to facilitate the above determination, when generating the three-dimensional model, information indicating the accuracy of each point of the three-dimensional model may be added to each point of the three-dimensional model. Also, information for identifying the frame used for generating each point may be added to each point of the three-dimensional model.

[0078] The accumulated error information may include information indicating the accumulated error of the entire image calculated as described above. FIG. 9 is a diagram showing an example of display of the accumulated error information in this case. For example, as shown in FIG. 9, information 201 indicating the necessity of loop closing processing may be displayed as information indicating the accumulated error of the entire image. Note that the information 201 may indicate the magnitude of the accumulated error instead of the necessity of loop closing processing. Also, here, the information 201 indicates the degree of necessity of loop closing processing (magnitude of accumulated error), but may indicate whether or not loop closing processing is necessary (whether or not there is an accumulated error (or greater than a threshold)).

[0079] In addition, the necessity for loop closing processing may be determined based on, in addition to the accumulation error, the shooting time during which loop closing processing has not been performed (the number of images used for generating the three-dimensional model). For example, it is determined that the necessity for loop closing processing is higher as the shooting time is longer.

[0080] Moreover, the method for notifying (presenting) to the user whether loop closing processing is necessary (or the degree of necessity) is not limited to this, and any display method such as changing or blinking the color of the screen frame may be used. Also, not limited to display, other notification methods such as sound or vibration may be used. Further, a combination of a plurality of these methods may be used.

[0081] Note that in the above, as an example of accumulation error information, a past three-dimensional model is projected onto the current image. However, as accumulation error information, the current three-dimensional model and the past three-dimensional model may be displayed. FIG. 10 is a diagram showing an example of display of accumulation error information in this case. For example, the current three-dimensional model and the past three-dimensional model may be displayed in different colors. Also, in FIG. 10, only the past three-dimensional model corresponding to the current three-dimensional model is displayed, but the entire three-dimensional model including the three-dimensional models of subjects not included in the current image may be displayed as the past three-dimensional model. In this case, the color or the like of the past three-dimensional model may be changed according to the distance from the current three-dimensional model.

[0082] Also, the viewpoint of the displayed three-dimensional model may be changeable. For example, as shown in FIG. 10, a user interface 202 for changing (translating, rotating, and zooming in and out) the viewpoints of the displayed current three-dimensional model and past three-dimensional model may be displayed. Note that the user interface 202 does not necessarily have to be displayed, and the viewpoint may be changed by a predetermined operation (such as drag, flick, pinch-in, and pinch-out) on the touch panel.

[0083] Further, the accumulated error information shown in FIG. 10 may be displayed instead of the accumulated error information shown in FIG. 7 or the like, may be displayed simultaneously by screen splitting or the like, or either one may be selectively displayed according to the user's operation. Also, differences may be displayed as in the example shown in FIG. 8, or information indicating the necessity of loop closing processing (the magnitude of the accumulated error) or the like may be displayed as in FIG. 9.

[0084] Also, in FIG. 10, manual alignment may be performed between the current three-dimensional model and the past three-dimensional model. That is, an operation for aligning the positions of the current three-dimensional model and the past three-dimensional model may be input by the user. Further, information based on this manually input operation may be used as auxiliary information for loop closing processing. For example, by this operation, the direction and amount of deviation between the current three-dimensional model and the past three-dimensional model can be obtained. By using at least one of the direction and amount of this deviation for setting the initial value of the search for the matching process in the loop closing process, the processing amount can be reduced and the accuracy can be improved.

[0085] Also, in the above, as a method for calculating the accumulated error, a method of comparing the pixel values of each pixel between a first image viewed from the current camera position and orientation based on the current image or the current three-dimensional model and a second image viewed from the current camera position and orientation of the past three-dimensional model has been described, but other methods may be used. For example, the accumulated error detection unit 125 may detect corresponding points that are corresponding feature points between the current three-dimensional model and the past three-dimensional model, and calculate the accumulated error based on the distance between the corresponding points. In this case, the accumulated error may be calculated in any of point units, region units including a plurality of points, and image units. For example, as the accumulated error in region or image units, the total value, average value, median value, maximum value, weighted addition value, or weighted average value of a plurality of differences of a plurality of corresponding points included in the region or image may be used. Note that the weights can be set in the same manner as above, for example.

[0086] Further, instead of detecting corresponding points between three-dimensional models, corresponding points may be detected between the current image and an image of the past three-dimensional model viewed from the current camera position and orientation.

[0087] [2-4. Determination of Execution of Loop Closing Process] As shown in FIG. 2, next, the accumulated error detection unit 125 determines whether to perform the loop closing process (S105). Note that the determination here is not a determination of whether to immediately perform the loop closing process, but a determination of whether to display guide information for taking a correction image used for the loop closing process described later.

[0088] For example, the accumulated error detection unit 125 displays a user interface for the user to instruct the execution of the loop closing process together with the accumulated error information. FIG. 11 is a diagram showing a display example of this user interface 203. When the user instructs the execution of the loop closing process via the user interface 203, the accumulated error detection unit 125 determines to perform the loop closing process (Yes in S105), and otherwise determines not to perform the loop closing process (No in S105).

[0089] Note that the method of instructing the user to execute the loop closing process is not limited to the above and any method may be used.

[0090] Further, the user interface 203 is not always displayed, and may be displayed when the calculated accumulated error (necessity of loop closing process) is equal to or greater than a predetermined threshold, and may not be displayed otherwise.

[0091] If it is determined not to perform the loop closing process (No in S105), the processes after step S101 are performed on the next image.

[0092] [2-5. Generation of Guide Information] On the other hand, when it is determined that loop closing processing is to be performed (Yes in S105), the user guide unit 126 selects a target image (S106). Here, the target image is an image included in past images and is an image for determining the camera position and orientation of the correction image to be captured hereafter. For example, the camera position and orientation of the target image are determined as the camera position and orientation of the correction image. Note that the camera position and orientation of the correction image do not necessarily have to be exactly the same as those of the target image. For example, it is sufficient if the difference between the camera position and orientation of the correction image and that of the target image is within a predetermined range.

[0093] First, the user guide unit 126 selects a plurality of candidate images from a plurality of past images. For example, the user guide unit 126 selects a plurality of candidate images from the plurality of images used for generating a past three-dimensional model (that is, a three-dimensional model corresponding to the subject in the current image) projected onto the current image during the calculation of the accumulation error. Note that the plurality of candidate images may be selected from a plurality of images related to the plurality of images. Here, related images are, for example, images with a camera position and orientation close to the camera position and orientation of the plurality of images.

[0094] Specifically, the user guide unit 126 preferentially selects an image with a camera position and orientation close to the estimated current camera position and orientation. Alternatively, the user guide unit 126 preferentially selects an image with a large number of feature points. Note that in addition to the number of feature points, the user guide unit 126 may determine the degree of dispersion (degree of variance) of the feature points and preferentially select an image with a high degree of dispersion. Alternatively, the user guide unit 126 may preferentially select an image in which the projected past three-dimensional model is centered. Note that the user guide unit 126 may use any one of these plurality of determination criteria or may use a combination of two or more determination criteria.

[0095] Note that the above-mentioned feature points are, for example, points detected by feature amount analysis from a three-dimensional model. Alternatively, the above-mentioned feature points are points detected by feature amount analysis from a two-dimensional image. Alternatively, the above-mentioned feature points are points in a three-dimensional model corresponding to points detected by feature amount analysis from a two-dimensional image.

[0096] Next, the user guide unit 126 displays a user interface for the user to select a target image from the selected plurality of candidate images. FIG. 12 is a diagram showing a display example of this user interface 205. When the user selects a target image from the plurality of displayed candidate images 206, the target image is determined. Note that the method by which the user selects the target image is not limited to this.

[0097] FIG. 13 is a diagram showing another display example of a user interface for the user to select a target image from a plurality of candidate images. The user interface 207 shown in FIG. 13 displays a past three-dimensional model, the camera position and orientation of the current image, and the camera position and orientation of the plurality of candidate images. In this case, when the user selects any one of the camera position and orientation of the plurality of candidate images, the candidate image is determined. Note that instead of the user interface 205 shown in FIG. 12, the user interface 207 shown in FIG. 13 may be used, or both of them may be displayed simultaneously.

[0098] Alternatively, instead of the user selecting the target image, the user guide unit 126 may select the target image from the plurality of images. In this case, for example, the user guide unit 126 selects the image with the highest priority as the target image in the above-described method for selecting candidate images.

[0099] Next, the user guide unit 126 generates guide information for the user to capture the target image and displays the generated guide information (S107). Here, the guide information is information for assisting the user in capturing a correction image, and is information for guiding the camera position and orientation of the camera 111 to the camera position and orientation of the correction image (that is, the camera position and orientation of the target image).

[0100] FIG. 14 is a diagram showing an example of guide information. For example, as shown in FIG. 14, as guide information, a target image is superimposed on the current image. For example, a semi-transparent target image is superimposed on the current image. Thereby, the user can capture a correction image with the same camera position and orientation as the target image by capturing the image with the current image overlapping the target image. Alternatively, only the feature portions such as the contour of the subject in the target image may be extracted and only the feature portions may be superimposed on the current image.

[0101] Further, the guide information displays information 208 for instructing the user to move to the camera position of the target image (the camera position of the correction image). Note that the guide information may include an instruction to adjust the camera orientation to the camera orientation of the target image (for example, "Please turn the camera upward", etc.). Note that only one of superimposing the target image on the current image and displaying the information 208 may be performed.

[0102] Here, an example of superimposing the target image on the current image has been described, but the current image and the target image may be displayed separately. Also, in this case, a plurality of target images may be displayed. For example, the plurality of target images may include a plurality of candidate images or a part thereof.

[0103] FIG. 15 is a diagram showing another example of guide information. The guide information 209 shown in FIG. 15 includes a three-dimensional model, the camera position and orientation of the current image, the camera position and orientation of the target image (the camera position and orientation of the correction image), and information 210 for instructing the user to move to the camera position of the target image. Note that instead of the guide information shown in FIG. 14, the guide information 209 shown in FIG. 15 may be used, or both of these may be displayed simultaneously.

[0104] When a correction image is captured (Yes in S108), loop closing processing is performed (S109). Specifically, the camera position and orientation estimation unit 123 corrects the camera position and orientation information and map information of a series of images (for example, the images from time t = 0 to t = x shown in FIG. 5) using the correction image. The three-dimensional model generation unit 124 corrects the three-dimensional model by regenerating the three-dimensional model using the corrected camera position and orientation information and map information. Further, the corrected camera position and orientation information, map information, and three-dimensional model are stored in the storage unit 122 as information generated from a series of images.

[0105] Note that during the period until a correction image is captured (No in S108), although not shown in FIG. 2, at least part of the processing of steps S101 to S104 is performed as necessary, and the guide information is updated to the content based on the current image (S107).

[0106] Also, in the above, the three-dimensional model generation device 102 determines whether a correction image has been captured, and performs loop closing processing when a correction image has been captured. However, it may be determined whether an image with the same (or similar) camera position and orientation as any of a plurality of images captured in the past has been captured, and loop closing processing may be performed when such an image has been captured.

[0107] Alternatively, the three-dimensional model generation device 102 performs loop closing processing when a correction image has been captured, and may not perform loop closing processing when an image with the same (or similar) camera position and orientation as other than the target image among a plurality of images captured in the past has been captured. That is, the three-dimensional model generation device 102 may determine whether to perform loop closing processing only for the target image among a plurality of images captured in the past. Thereby, the processing amount can be reduced, and the occurrence of unnecessary loop closing processing can be reduced.

[0108] [3. Summary] As described above, the three-dimensional model generation device according to the present embodiment (for example, the three-dimensional model generation device 102) performs the processing shown in FIG. 16. FIG. 16 is a flowchart of the three-dimensional model generation process by the three-dimensional model generation device according to the present embodiment. FIG. 17 is a block diagram showing the hardware configuration of the three-dimensional model generation device 102. For example, the three-dimensional model generation device 102 includes a circuit 11 and a memory 12. The circuit 11 is connected to the memory 12 and performs the following processing using the memory 12.

[0109] First, the three-dimensional model generation device acquires a plurality of consecutive images (S201), generates a three-dimensional model including a plurality of three-dimensional points each having position information using the plurality of images (S202), determines whether to correct the accumulation error of the three-dimensional model (for example, loop closing processing) (S203), and if it is determined to perform the correction (Yes in S203), causes the display unit (for example, the display unit 131) to display first information (guide information) for instructing a first position and orientation for taking a first image (for example, a correction image) for performing the correction (S204).

[0110] According to this, the three-dimensional model generation device can support taking a first image for correcting the accumulation error. Therefore, by appropriately performing the correction process, the accuracy of the generated three-dimensional model can be improved. In this way, the three-dimensional model generation device can achieve further improvement.

[0111] For example, the first information is a second image (for example, a target image) taken at the first position and orientation included in the plurality of images. According to this, the user can determine the first position and orientation while referring to the second image.

[0112] For example, the circuit 11 superimposes the second image on the currently captured image and displays it on the display unit (for example, FIG. 14). According to this, the user can take the first image by taking a picture in a state where the currently captured image matches the second image.

[0113] For example, the circuit 11 further causes the display unit to display second information indicating the accumulation error (for example, accumulation error information). According to this, the user can refer to the second information and determine whether to correct the accumulation error.

[0114] For example, as the second information, the circuit 11 causes the display unit to display a three-dimensional model and a third image viewed from the current position and orientation, superimposed on the currently captured image (for example, FIG. 7). According to this, the user can recognize the difference between the currently captured image and the third image. Further, the user can determine the necessity of correcting the accumulation error based on the difference.

[0115] For example, as the second information, the circuit 11 further causes the display unit to display an area where there is a difference between the third image and the currently captured image (for example, FIG. 8). According to this, the user can determine the necessity of correcting the accumulation error based on the third information.

[0116] For example, the circuit 11 further calculates the amount of the accumulation error based on the difference between the third image and the currently captured image, and causes the display unit to display third information indicating the calculated amount (for example, FIG. 9). According to this, the user can recognize the magnitude (degree) of the difference between the currently captured image and the third image. Further, the user can determine the necessity of correcting the accumulation error based on the magnitude of the difference.

[0117] For example, the circuit 11 further causes the display unit to display a user interface for the user to indicate whether to perform correction together with the second information (for example, FIG. 11). When an instruction to perform correction is received, the circuit 11 causes the display unit to display the first information. According to this, correction can be performed based on the determination of the user who has referred to the second information.

[0118] For example, the circuit 11 further causes the display unit to display a user interface for the user to select the second image from a plurality of fourth images included in a plurality of images (for example, FIG. 12). According to this, the user can select the second image.

[0119] For example, circuit 11 further selects a plurality of fourth images from the plurality of images based on the feature points included in each of the plurality of images. According to this, the accuracy of feature point matching for the second image can be improved, so that the accuracy of correction of the accumulated error can be improved.

[0120] For example, circuit 11 further selects a second image from the plurality of images based on the feature points included in each of the plurality of images. According to this, the accuracy of feature point matching for the second image can be improved, so that the accuracy of correction of the accumulated error can be improved.

[0121] For example, when the first information is displayed on the display unit, (1) when the first image taken from the first position and orientation is acquired, correction is performed using the acquired first image, and (2) among the plurality of images, when the sixth image taken at the position and orientation of the fifth image taken other than the first position and orientation is acquired, correction of the accumulated error is not performed using the acquired sixth image. According to this, it is possible to suppress the correction being performed using an inappropriate image.

[0122] For example, circuit 11 estimates the estimated position and orientation, which is the position and orientation at which each of the plurality of images was taken, using the plurality of images, generates a three-dimensional model using the estimated estimated position and orientation, and in the correction of the accumulated error, corrects the accumulated error based on the error between the estimated position and orientation of the second image taken at the first position and orientation and the estimated position and orientation of the first image, which are included in the plurality of images.

[0123] For example, circuit 11 causes the display unit to display the current three-dimensional model and the past three-dimensional model as the second information (for example, FIG. 10). Circuit 11 further receives an operation for aligning the positions of the current three-dimensional model and the past three-dimensional model, and in the correction of the accumulated error, performs correction using the information obtained by the operation for aligning the positions as auxiliary information. According to this, the processing amount of correction of the accumulated error can be reduced and the accuracy can be improved.

[0124] For example, a three-dimensional model generation device includes a memory and a circuit connected to the memory. The circuit uses the memory to acquire a plurality of consecutive images, generates a three-dimensional model including a plurality of three-dimensional points each having position information using the plurality of images, selects, from the plurality of images, a first image that is one of two images taken in the same position and orientation for correcting the accumulation error of the three-dimensional model, and selects the first image from the plurality of images based on feature points included in each of the plurality of images in the selection of the first image.

[0125] For example, the circuit selects a plurality of second images including the first image from the plurality of images based on the feature points.

[0126] For example, the circuit selects the first image from the plurality of images based on the number of the feature points. For example, the circuit preferentially selects, as the first image, an image with a large number of the feature points. For example, the circuit selects the first image from the plurality of images based on the degree of dispersion (degree of variance) of the feature points. For example, the circuit preferentially selects, as the first image, an image with a high degree of dispersion of the feature points.

[0127] For example, the feature points are points detected by feature amount analysis from the first three-dimensional model. For example, the feature points are points detected by feature amount analysis from the plurality of images. Alternatively, the above feature points are points in the three-dimensional model corresponding to the points detected by feature amount analysis from the plurality of images.

[0128] As described above, the three-dimensional model generation system and the three-dimensional model generation device according to the embodiments of the present disclosure have been described, but the present disclosure is not limited to these embodiments.

[0129] In addition, each processing unit included in the three-dimensional model generation device and the like according to the above embodiments is typically realized as an LSI which is an integrated circuit. These may be individually formed into one chip, or may be formed into one chip so as to include some or all of them.

[0130] Also, the integration into an integrated circuit is not limited to LSI, and it may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor capable of reconfiguring the connection and setting of circuit cells inside the LSI may also be used.

[0131] Also, in each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0132] Also, the present disclosure may be realized as a three-dimensional model generation method or the like executed by a three-dimensional model generation system or a three-dimensional model generation device.

[0133] Also, the division of functional blocks in the block diagram is an example, and a plurality of functional blocks may be realized as one functional block, one functional block may be divided into a plurality, or a part of the function may be transferred to another functional block. Also, the functions of a plurality of functional blocks having similar functions may be processed by a single piece of hardware or software in parallel or in time division.

[0134] Also, the order in which each step in the flowchart is executed is for illustration in order to specifically describe the present disclosure, and may be an order other than the above. Also, a part of the above steps may be executed simultaneously (in parallel) with other steps.

[0135] The three-dimensional model generation system, three-dimensional model generation device, etc. according to one or more aspects have been described based on the embodiments. However, the present disclosure is not limited to these embodiments. Without departing from the spirit of the present disclosure, various modifications conceived by those skilled in the art applied to these embodiments, or forms constructed by combining components in different embodiments may also be included within the scope of one or more aspects.

Industrial Applicability

[0136] The present disclosure can be applied to a three-dimensional model generation system, a three-dimensional model generation device, etc.

Explanation of Signs

[0137] 11 Circuit 12 Memory 101 Imaging device 102 Three-dimensional model generation device 103 Display device 111 Camera 112 Sensor 121 Acquisition unit 122 Storage unit 123 Camera position and orientation estimation unit 124 Three-dimensional model generation unit 125 Accumulated error detection unit 126 User guide unit 127 Output unit 128 Control unit 131 Display unit 132 Operation reception unit 201, 208, 210 Information 202, 203, 205, 207 User interface 206 Candidate image 209 Guide information

Claims

1. A memory, and a circuit connected to the memory, wherein the circuit uses the memory to acquire a plurality of consecutive images, generate a three-dimensional model including a plurality of three-dimensional points each having position information using the plurality of images, determine whether to correct an accumulation error of the three-dimensional model, and when it is determined to perform the correction, cause a display unit to display first information for instructing a first position and orientation for taking a first image for performing the correction. A three-dimensional model generation device.

2. The first information is a second image captured in the first position and orientation, which is included in the plurality of images. The three-dimensional model generation device according to claim 1.

3. The circuit causes the display unit to display the second image by superimposing it on an image being currently captured. The three-dimensional model generation device according to claim 2.

4. The circuit further causes the display unit to display second information indicating the accumulation error. The three-dimensional model generation device according to claim 1.

5. As the second information, the circuit causes the display unit to display a third image of the three-dimensional model as viewed from the current position and orientation by superimposing it on an image being currently captured. The three-dimensional model generation device according to claim 4.

6. As the second information, the circuit further causes the display unit to display an area where there is a difference between the third image and the image being currently captured. The three-dimensional model generation device according to claim 5.

7. The circuit further calculates an amount of the accumulation error based on a difference between the third image and the image being currently captured, and causes the display unit to display third information indicating the calculated amount. The three-dimensional model generation device according to claim 5.

8. The circuit further displays, on the display unit, a user interface for a user to instruct whether to perform the correction together with the second information, and when an instruction to perform the correction is received, causes the display unit to display the first information. The three-dimensional model generation device according to claim 4.

9. The circuit further displays, on the display unit, a user interface for a user to select the second image from a plurality of fourth images included in the plurality of images. The three-dimensional model generation device according to claim 2.

10. The circuit further selects the plurality of fourth images from the plurality of images based on feature points included in each of the plurality of images. The three-dimensional model generation device according to claim 9.

11. The circuit further Selecting the second image from the plurality of images based on feature points included in each of the plurality of images The three-dimensional model generation device according to claim 2

12. When the first information is displayed on the display unit (1) When the first image taken from the first position and orientation is acquired, perform the correction using the acquired first image (2) When the sixth image taken in the position and orientation of the fifth image taken other than the first position and orientation among the plurality of images is acquired, do not perform the correction of the accumulated error using the acquired sixth image The three-dimensional model generation device according to claim 1

13. The circuit Using the plurality of images, estimate the estimated position and orientation which is the position and orientation at which each of the plurality of images was taken, and generate the three-dimensional model using the estimated estimated position and orientation In the correction of the accumulated error, correct the accumulated error based on the error between the estimated position and orientation of the second image taken in the first position and orientation included in the plurality of images and the estimated position and orientation of the first image The three-dimensional model generation device according to claim 1

14. The circuit causes the display unit to display the current three-dimensional model and the past three-dimensional model as the second information The circuit further Receives an operation for aligning the positions of the current three-dimensional model and the past three-dimensional model In the correction, perform the correction using the information obtained by the operation for aligning the positions as auxiliary information The three-dimensional model generation device according to claim 4

15. Acquire a plurality of consecutive images Using the plurality of images, generate a three-dimensional model including a plurality of three-dimensional points each having position information Determine whether to correct the accumulated error of the three-dimensional model When it is determined to perform the correction, cause the display unit to display first information for instructing a first position and orientation for taking a first image for performing the correction Three-dimensional model generation method

16. A program for causing a computer to execute the three-dimensional model generation method according to claim 15

Images