Imaging device, imaging operation support method, and imaging operation support program

Abstract

To present the necessity of a loop closing process to a user during image capturing, in an easily understood manner, to prompt the user to perform appropriate imaging operation, thereby improving the accuracy of three-dimensional measurement using SLAM.SOLUTION: An imaging apparatus images spots of places to be measured, for performing three-dimensional measurement processing to generate three-dimensional spatial information of the places to be measured on the basis of captured images of the spots of the places to be measured. An imaging screen 101 displayed on a display unit displays support information on an imaging operation in which a visible camera sequentially images the spots of places to be measured while a user holding the body of the apparatus moves to the places to be measured. Specifically, a main window 102 and a sub window 103 display, as the support information, a support image obtained by visualizing a range in which a loop closing process has been executed in the three-dimensional measurement processing.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a photographing device that photographs each point of a measurement target location, and a photographing work support method and a photographing work support program that support a user's photographing work using the photographing device, in order to perform three-dimensional measurement processing for generating three-dimensional space information of the measurement target location based on the photographed images of each point of the measurement target location.

Background Art

[0002] A technique of three-dimensional measurement for generating three-dimensional space information (map data) regarding a measurement target location based on a photographed image of the measurement target location is known. In this three-dimensional measurement, in recent years, the SLAM (Simultaneous Localization And Mapping) method has attracted attention. In the SLAM method, it is possible to generate three-dimensional space information as a measurement result and the position information of the own vehicle based on photographed images of each point of the measurement target location by holding a photographing device on a moving body. In particular, when a user performs a photographing operation of moving within a measurement target location while photographing using a hand-held and portable photographing device to obtain photographed images of each point of the measurement target location, three-dimensional measurement can be performed simply.

[0003] On the other hand, in three-dimensional measurement using SLAM, since relative self-position estimation is repeated, accumulation of errors cannot be avoided. Such accumulation of errors in self-position estimation causes a significant decrease in the accuracy of the three-dimensional measurement result.

[0004] Therefore, as a method for improving the accuracy degradation due to such cumulative errors, conventionally, in the middle of creating three-dimensional space information (map data), when an error occurs, it is possible to restart creating three-dimensional space information from the middle without starting over from the beginning (see Patent Document 1). Further, in this technique, by detecting an event (loss) that causes an error and presenting the period during which the event occurred to the user, the user can grasp the position at which to restart creating the three-dimensional space information.

[0005] Also, as a method for improving the accuracy degradation due to cumulative error, loop closing technology has been conventionally known (see Non-Patent Document 1). In this technology, when it is detected that the moving path of the own vehicle forms a loop, that is, the own vehicle has returned to a point where it was previously photographed, the self-position estimation result (past position) obtained at the time of the previous photographing at that point is regarded as the correct position, and the positions in the path from the current position to the past position are corrected, thereby eliminating the cumulative error. According to this technology, like the technology disclosed in Patent Document 1, the trouble of retaking photographs midway can be saved.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Non-Patent Documents

[0007]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] In three-dimensional measurement using SLAM, to improve the accuracy degradation due to cumulative error, loop closing technology is effective as disclosed in Patent Document 2. On the other hand, the loop closing process can be carried out when the moving path forms a loop, that is, when the vehicle returns to a point where it was previously photographed. Therefore, it is important to deliberately create a state where the loop closing process can be carried out. Thus, it is desirable to clearly present the necessity of the loop closing process to the user during the photographing operation. However, in the conventional technology, no consideration has been given to such a demand.

[0009] Therefore, the main object of the present invention is to provide a photographing apparatus, a photographing work support method, and a photographing work support program that can improve the accuracy of three-dimensional measurement using SLAM by clearly presenting the necessity of loop closing processing to the user during the photographing work and prompting the user to perform appropriate photographing work.

Means for Solving the Problems

[0010] The photographing apparatus of the present invention is a photographing apparatus that photographs each point of a measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location, and includes a device body held by a user, a photographing unit provided on the device body for photographing the measurement target location, a display unit for displaying support information related to a photographing work for sequentially photographing each point of the measurement target location by the photographing unit while the user holds the device body and moves within the measurement target location, and a processor for controlling the photographing unit and the display unit. The processor is configured to display, on the display unit, a support image visualizing a range where loop closing processing has been performed in the three-dimensional measurement processing as the support information.

[0011] Further, the photographing work support method of the present invention is a photographing work support method in which a processor performs processing for supporting a user's photographing work of sequentially photographing each point of a measurement target location by a photographing apparatus while holding the photographing apparatus and moving within the measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location. As support information related to the photographing work, a support image visualizing a range where loop closing processing has been performed in the three-dimensional measurement processing is configured to be displayed on a display unit.

[0012] In addition, the photographing work support program of the present invention is a photographing work support program that causes a processor to execute a process of supporting a user's photographing work of holding a photographing device and sequentially photographing each point of a measurement target location while moving within the measurement target location in order to perform a three-dimensional measurement process of generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location. As support information regarding the photographing work, a support image obtained by visualizing a range where loop closing processing has been performed in the three-dimensional measurement process is configured to be displayed on a display unit.

Effect of the Invention

[0013] According to the present invention, it is possible to clearly present the necessity of loop closing processing to a user (operator) during photographing work by means of a support image, and prompt the user to perform appropriate photographing work. As a result, by the user performing appropriate photographing work according to the support image, the accuracy of the three-dimensional measurement process using SLAM can be improved.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0015] A first invention made to solve the above problems is a photographing device that photographs each point of a measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location, and includes a device body held by a user, a photographing unit provided in the device body for photographing the measurement target location, a display unit for displaying support information regarding a photographing operation for sequentially photographing each point of the measurement target location by the photographing unit while the user holds the device body and moves the measurement target location, and a processor for controlling the photographing unit and the display unit. The processor is configured to display, on the display unit, a support image that visualizes a range where loop closing processing has been performed in the three-dimensional measurement processing as the support information.

[0016] According to this, for the user (operator) during the photographing operation, the necessity of loop closing processing can be clearly presented to the user by the support image, and the user can be prompted to perform an appropriate photographing operation. As a result, by the user performing an appropriate photographing operation according to the support image, the accuracy of three-dimensional measurement processing using SLAM can be improved. Note that the range where loop closing processing has been performed may be visualized by an area or by a path.

[0017] Further, in a second invention, the processor is configured to display, on the display unit, the support image that visualizes the range where loop closing processing has been performed in at least one of an overhead image generated from the measurement result of the three-dimensional measurement processing and the current photographed image.

[0018] According to this, the range where the loop closing process has been executed can be presented to the user in an easy-to-understand manner. In this case, for example, the range where the loop closing process has been executed may be colored and visualized. Also, the bird's-eye view image may be a bird's-eye view point cloud image, a bird's-eye view point cloud path image, or a bird's-eye view path image.

[0019] Further, in the third invention, the processor is configured to visualize the range where the loop closing process has been executed in the support image in a display mode different from the range where the execution of the loop closing process is recommended.

[0020] According to this, the range where the loop closing process has been executed can be presented to the user in an easy-to-understand manner. In this case, for example, the range where the loop closing process has been executed may be visualized in a color different from the range where the execution of the loop closing process is recommended.

[0021] Further, in the fourth invention, the processor is configured to acquire the number of executions in the range where the loop closing process has been executed.

[0022] According to this, the number of executions in the range where the loop closing process has been executed can be presented to the user. In this case, the number of executions for each shooting location (shooting image) can be acquired. Also, the number of executions for each point of the point cloud data as three-dimensional space information can be acquired.

[0023] Further, in the fifth invention, the processor is configured to change the display mode of the range where the loop closing process has been executed in the support image according to the number of executions.

[0024] According to this, the number of executions in the range where the loop closing process has been executed can be presented to the user in an easy-to-understand manner. In this case, for example, the color may be changed as the display mode is changed. For example, the range with one execution is drawn in green, the range with two executions is drawn in blue, and the range with zero executions is drawn in red.

[0025] Also, a sixth invention is a photographing work support method in which a processor performs a process of supporting a user's photographing work of sequentially photographing each point of a measurement target location with the photographing device while holding the photographing device and moving within the measurement target location in order to perform a three-dimensional measurement process of generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location. As support information regarding the photographing work, a support image visualizing a range where loop closing processing has been performed in the three-dimensional measurement process is configured to be displayed on a display unit.

[0026] According to this, similar to the first invention, by clearly presenting the necessity of loop closing processing to the user during the photographing work and prompting the user to perform appropriate photographing work, the accuracy of three-dimensional measurement using SLAM can be improved.

[0027] Also, a seventh invention is a photographing work support program that causes a processor to execute a process of supporting a user's photographing work of sequentially photographing each point of a measurement target location with the photographing device while holding the photographing device and moving within the measurement target location in order to perform a three-dimensional measurement process of generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location. As support information regarding the photographing work, a support image visualizing a range where loop closing processing has been performed in the three-dimensional measurement process is configured to be displayed on a display unit.

[0028] According to this, similar to the first invention, by clearly presenting the necessity of loop closing processing to the user during the photographing work and prompting the user to perform appropriate photographing work, the accuracy of three-dimensional measurement using SLAM can be improved.

[0029] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0030] FIG. 1 is an explanatory diagram showing a situation of a photographing work performed by a user using the photographing device 1 according to the present embodiment. FIG. 2 is a plan view showing a measurement target location.

[0031] The imaging device 1 includes a device main body 11 and a sensor unit 12. The sensor unit 12 includes a visible camera 21 (imaging unit). The visible camera 21 is a monocular camera that detects visible light and captures an object, and outputs a captured image, for example, a color image in the RGB format. Note that the imaging device 1 can be configured as a tablet terminal or a notebook PC.

[0032] The user (operator) walks through the measurement target location while holding the device main body 11 of the imaging device 1. At this time, in the imaging device 1, the measurement target location is captured by the visible camera 21, and captured images of each point in the measurement target location are sequentially acquired.

[0033] Also, in this embodiment, the imaging device 1 serves as a three-dimensional measurement device. That is, in the imaging device 1, three-dimensional measurement processing is performed based on the captured images of each point sequentially acquired by the visible camera 21, and three-dimensional space information regarding the measurement target location is generated. In the three-dimensional measurement processing, point cloud data (environmental map) as three-dimensional space information regarding the measurement target location is generated using the SLAM method. At this time, together with the generation of the point cloud data, self-position estimation processing is performed, and the position of each self-position, that is, the imaging location, is estimated.

[0034] Also, in the imaging device 1, when it is detected that the movement path of the own device forms a loop, that is, the own device has returned to a location previously captured, loop closing processing is performed. In the loop closing processing, regarding the location previously captured, the self-position estimation result (past position) obtained at the time of that previous capture is regarded as the correct position, and the positions of each imaging location in the path from the current position to the past position are corrected.

[0035] In the example shown in FIG. 2, imaging is started from the imaging start point, and after traversing a part of the measurement target location, it returns to the imaging start point. Further, imaging is continued, and another area of the measurement target location is imaged. In this case, loop closing processing is performed when returning to the imaging start point. As a result, the accuracy of the point cloud data and the self-position estimation result regarding each imaging point included in the loop is improved.

[0036] Next, an assistance image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud image will be described. FIG. 3 is an explanatory diagram showing an assistance image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud image.

[0037] In the present embodiment, an assistance image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud image is generated, and the assistance image is presented to the user during the imaging operation. The bird's-eye view point cloud image is an image (rendering) obtained by imaging each point of the point cloud data from an aerial viewpoint. The completed range represents the range in which the loop closing process has been performed. The recommended implementation range represents the range in which the loop closing process is recommended, that is, the range in which the loop closing process has not been performed, or the range in which the accuracy of the three-dimensional measurement result is low.

[0038] The completed area (completed range) and the recommended implementation area (recommended implementation range) are drawn with color separation. For example, the completed area is drawn in blue, and the recommended implementation area is drawn in red. Specifically, in the bird's-eye view point cloud image, each point included in the completed area is drawn in blue, and each point included in the recommended implementation area is drawn in red.

[0039] At this time, each point drawn in a predetermined size in the bird's-eye view point cloud image is colored with a predetermined color. Also, a sphere with a predetermined diameter centered on each point may be drawn in a predetermined color. Further, the target space may be divided into cubic voxels, and the voxels including each point may be drawn in a predetermined color.

[0040] By viewing the support image, the user grasps the range where shooting work for loop closing processing is necessary and performs the shooting work. Specifically, the user performs the shooting work on the recommended implementation area drawn in red. That is, the shooting work is performed so as to pass through the recommended implementation area. Note that the specific content of the shooting work, that is, the movement route and the like, may be appropriately determined by the user according to the situation at the site.

[0041] As described above, in this embodiment, the support image can clearly present to the user the necessity of loop closing processing. Furthermore, the range where shooting work for loop closing processing is necessary can be clearly presented to the user. Thereby, the user can be prompted to perform appropriate shooting work, and since the loop closing processing is surely carried out, the accuracy of the three-dimensional measurement processing can be improved.

[0042] Note that both the implemented area and the recommended implementation area represent the range where shooting has been performed within the measurement target location. That is, the area that is not included in either the implemented area or the recommended implementation area in the measurement target location represents the range where shooting has not been performed. Therefore, the user can also confirm the range where shooting has not been performed by viewing the support image.

[0043] Also, in this embodiment, the number of times the loop closing processing is performed is counted, and visualization (color-coding) of the area corresponding to the number of times the loop closing processing is performed is carried out. For example, the area where the number of times is 1 is drawn in green, the area where the number of times is 2 is drawn in blue, and the area where the number of times is 0 is drawn in red.

[0044] Next, a support image in which the implemented range and the recommended range are visualized on the overhead point cloud path image will be described. FIG. 4 is an explanatory diagram showing a support image in which the implemented range and the recommended range are visualized on the overhead point cloud path image.

[0045] In this embodiment, a support image in which the completed range and the recommended execution range are visualized on the overhead point cloud path image is generated, and the support image is presented to the user during the shooting operation. The overhead point cloud path image is one in which a path (movement trajectory) is superimposed on the overhead point cloud image. The overhead point cloud image is an image (rendering) of each point of the point cloud data as seen from an overhead viewpoint. The path is drawn by connecting the self-position estimation results based on each captured image, that is, the shooting points of each time, with a line on the overhead point cloud image.

[0046] The completed path (completed range) and the recommended execution path (recommended execution range) are drawn with color separation. For example, the completed path is drawn in blue, and the recommended execution path is drawn in red.

[0047] In addition, a shooting point mark is drawn on the support image. The shooting point mark represents the shooting point and the shooting direction. Specifically, the shooting point mark is drawn as an isosceles triangle, the vertex of the isosceles triangle represents the shooting point, and the direction from the vertex to the base represents the shooting direction.

[0048] By viewing the support image, the user grasps the range where the shooting operation is required for the loop closing process and performs the shooting operation. Specifically, the user performs the shooting operation on the recommended execution path drawn in red. That is, the shooting operation is performed so that a loop including the recommended execution path is formed.

[0049] In addition, in this embodiment, the number of times the loop closing process is performed is counted, and the visualization (color separation) of the path is performed according to the number of times the loop closing process is performed. For example, the path with the execution count of 1 time is drawn in green, the path with the execution count of 2 times is drawn in blue, and the path with the execution count of 0 times is drawn in red.

[0050] Next, a support image in which the completed range and the recommended execution range are visualized on the overhead path image will be described. FIG. 5 is an explanatory diagram showing a support image in which the completed range and the recommended execution range are visualized on the overhead path image.

[0051] In this embodiment, a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view route image is generated, and the support image is presented to the user during the shooting operation. The bird's-eye view route image is in a state where the bird's-eye view point cloud image is removed from the bird's-eye view point cloud route image (see FIG. 4), and the self-position estimation result based on each captured image, that is, the line connecting the shooting points of each time, is drawn as seen from an aerial viewpoint.

[0052] Next, a support image in which the completed range and the recommended implementation range are visualized on the captured image will be described. FIG. 6 is an explanatory diagram showing a support image in which the completed range and the recommended implementation range are visualized on the captured image.

[0053] In this embodiment, a support image in which the completed range and the recommended implementation range are visualized on the captured image is generated, and the support image is presented to the user during the shooting operation. The captured image is the current captured image, that is, an image captured in real time output from the visible camera 21.

[0054] The completed area (completed range) and the recommended implementation area (recommended implementation range) are drawn with color separation. For example, the completed area is drawn in blue, and the recommended implementation area is drawn in red. Specifically, an image representing the completed area drawn in a predetermined color and an image representing the recommended implementation area drawn in a predetermined color are superimposed on the captured image in a semi-transparent state.

[0055] Thus, in this embodiment, the completed range and the recommended implementation range (unimplemented range) are visualized on the captured image. Thereby, the user can easily determine whether the place visible in front of the eyes is the completed range or the recommended implementation range. And the user performs the shooting operation so as to include the recommended implementation range.

[0056] Next, the schematic configuration of the imaging device 1 will be described. FIG. 7 is a block diagram showing the schematic configuration of the imaging device 1.

[0057] In addition to the sensor unit 12, the imaging device 1 includes a display 13 (display unit), an input device 14, a memory 15, and a processor 16 (CPU).

[0058] In addition to the visible camera 21, the sensor unit 12 includes a depth camera 22 and an IMU 23 (Inertial Measurement Unit). The depth camera 22 is a stereo camera that detects infrared light and images a subject, and outputs depth information (distance image) as a detection result. Based on the detection result of the depth camera 22, the distance to the subject can be measured. The IMU 23 detects three-dimensional angular velocity and acceleration. Based on the detection result of the IMU 23, the amount of movement and rotation of the imaging device 1 can be measured. Note that the visible camera 21, the depth camera 22, and the IMU 23 may not be integrated as a sensor unit. Also, a configuration in which the depth camera 22 and the IMU 23 are omitted and only the visible camera 21 is provided may be used.

[0059] The display 13 presents various types of information related to the imaging operation to the user, and displays an imaging screen 101 (see FIGS. 8 and 9), etc. The input device 14 is for the user to perform input operations. The input device 14 may be a keyboard, a mouse, a touch pad, a touch panel, etc. Note that when the imaging device 1 is configured as a tablet terminal, a touch panel display in which the touch panel as the input device 14 and the display panel as the display 13 are integrated is provided.

[0060] The memory 15 stores programs executed by the processor 16, etc. The memory 15 also stores the captured image of the visible camera 21, and the detection results of the depth camera 22 and the IMU 23. The memory 15 also stores the measurement results generated by the processor 16.

[0061] The processor 16 performs various processes by executing the programs stored in the memory 15. In the present embodiment, the processor 16 performs a detection information acquisition process, a point cloud generation process, a first assistance image generation process, a second assistance image generation process, a screen control process, a message notification process, and the like.

[0062] In the detection information acquisition process, the processor 16 acquires the captured image of the visible camera 21. Further, the processor 16 acquires the detection results of the depth camera 22 and the IMU 23.

[0063] In the point cloud generation process (3D measurement process), the processor 16 generates point cloud data (environmental map) as three-dimensional space information regarding the measurement target location using the SLAM method based on the captured images of each point sequentially acquired by the visible camera 21. In the point cloud generation process, self-position estimation is performed in conjunction with the generation of the point cloud data, and the self-position at each time, that is, the position of the shooting location is acquired.

[0064] Further, the point cloud generation process includes a loop detection process. In the loop detection process, the processor 16 detects that the path of the own vehicle forms a loop, that is, the own vehicle has returned to a point that was previously photographed. At this time, in the detection of the previously photographed point, it is not compared with a preset specified captured image, but when the current captured image is similar to any of the past captured images, it is determined that the own vehicle has returned to a point that was previously photographed. Note that the similarity of the captured images may be determined based on the feature amounts extracted from the captured images.

[0065] Further, the point cloud generation process includes a loop closing process. When a loop is detected in the loop detection process, the loop closing process is executed. In the loop closing process, the processor 16 regards the self-position estimation result (past position) acquired at the time of the previous photographing as the correct position for the previously photographed point, and corrects the positions of each photographing point in the path from the current position to the past position.

[0066] In addition, the point cloud generation process includes an execution count acquisition process. In the execution count acquisition process, the processor 16 counts the number of times the loop closing process has been executed for each shooting location. The execution count information is added to the captured image (frame) at each shooting location. Further, the execution count information is added to each point in the point cloud data. At this time, for example, the execution count information is added to each point included in the point cloud data generated from the captured image (frame) at the shooting location where the loop closing process has been executed. Note that the execution count information also includes information regarding the execution of the loop closing process. When the execution count is 0 times, it means not executed, and when the execution count is 1 or more times, it means executed.

[0067] In addition, the point cloud generation process includes an accuracy acquisition process. In the accuracy acquisition process, the processor 16 acquires the accuracy of the three-dimensional measurement results (point cloud data and self-position estimation results) at each shooting location. Specifically, the accuracy of each shooting location is acquired based on the separation amount from the location where the loop closing process has been executed. Specifically, the separation amount is, for example, the moving distance (length of the movement trajectory) from the location where the loop closing process was executed, the elapsed time from the time when the loop closing process was executed, the number of captured images (number of frames) taken after the loop closing process was executed, and so on. Note that the accuracy may be determined by combining the moving distance, elapsed time, and number of frames as the separation amount. Further, the accuracy may be determined by combining the moving speed with the separation amount. The accuracy information is added to the captured image (frame) at each shooting location. Further, the accuracy information may be added to each point in the point cloud data.

[0068] In addition, regardless of the execution status of the loop closing process, the accuracy information of each point in the point cloud data can be acquired. For example, when the measurement results (point cloud data) based on the captured images at each shooting location do not match the detection results of the depth camera 22 or the IMU 23, it can be determined that the accuracy is low. Further, the accuracy may be determined by comparing the measurement results based on the front and rear captured images (frames). In this case, the accuracy information is added to each point in the point cloud data.

[0069] In the point cloud generation process, in addition to the captured image of the visible camera 21, the detection results of the depth camera 22 and the IMU 23 may also be used. Specifically, based on the detection result of the depth camera 22, the relative position information (distance from the imaging device 1) of each feature point extracted from the captured image of the visible camera 21 with respect to the imaging device 1 can be obtained, and the self-position estimation result based on the captured image of the visible camera 21 can be corrected. Also, based on the detection result of the IMU 23, the relative position information of each imaging location can be obtained, and based on the relative position information of that imaging location, the self-position estimation result based on the captured image of the visible camera 21 can be corrected.

[0070] In the first support image generation process, the processor 16 generates a support image (first support image) in which the completed range and the recommended implementation range are visualized on the bird's-eye view image. Specifically, a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud image (see FIG. 3), a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud path image (see FIG. 4), and a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view path image (see FIG. 5) are generated.

[0071] In the visualization of the completed range and the recommended implementation range on the bird's-eye view point cloud image, based on the implementation count information added to each point in the point cloud data, the information on the completed range and the recommended implementation range is obtained, and each point on the point cloud image included in each of the completed range and the recommended implementation range is colored (point cloud coloring process).

[0072] Also, in the visualization of the completed range and the recommended implementation range on the bird's-eye view point cloud path image or the bird's-eye view path image, based on the implementation count information added to the captured image (frame) of each imaging location, the image of the path (movement trajectory) connecting the positions of each imaging location with a line is colored (path coloring process).

[0073] In addition, in the present embodiment, in visualizing the recommended implementation range on the bird's-eye view point cloud image, a range with low accuracy of the three-dimensional measurement result is used as the recommended implementation range, and based on the accuracy information added to each point in the point cloud data, points with low accuracy are colored in the color of the recommended implementation range. Note that an area with many points of low accuracy in the point cloud data may be extracted as the recommended implementation range.

[0074] In addition, in the present embodiment, in visualizing the recommended implementation range on the bird's-eye view point cloud path image or the bird's-eye view path image, a range with low accuracy among the paths connecting each shooting point is used as the recommended implementation range, and based on the accuracy information added to the captured image (frame) of each shooting point, information representing the recommended implementation range is acquired, and the path image is colored.

[0075] In the second support image generation process, the processor 16 generates a support image (second support image) (see FIG. 6) in which the implemented range and the recommended implementation range are visualized on the captured image. Specifically, an image representing the implemented area drawn in a predetermined color and an image representing the recommended implementation area drawn in a predetermined color are generated (superimposed image generation process), and the images representing the implemented area and the recommended implementation area are superimposed on the captured image in a semi-transparent state. Note that in the superimposed image generation process, information representing the ranges of the implemented area and the recommended implementation area may be acquired based on the implementation frequency information of each point in the point cloud data viewed from the same viewpoint as the captured image.

[0076] In the screen control process, the processor 16 controls the screen to be displayed on the display 13. In the present embodiment, a shooting screen 101 (see FIGS. 8 and 9) is generated and displayed on the display 13. On the shooting screen 101, the first support image (see FIGS. 3, 4, and 5) generated in the first support image generation process and the second support image (see FIG. 6) generated in the second support image generation process are displayed.

[0077] In the message notification process, the processor 16 performs a process of notifying the user of messages regarding guidance for shooting operations and various warnings. In this embodiment, a message prompting the user to perform a shooting operation for loop closing processing is displayed on the shooting screen 101 (see FIGS. 8 and 9).

[0078] At this time, when a predetermined notification condition is satisfied, a message is displayed. For example, when the amount of movement from the point where the loop closing process is performed is equal to or greater than a predetermined threshold value, a message is displayed. Also, when the elapsed time since the loop closing process was performed is equal to or greater than a predetermined threshold value, a message is displayed. Here, the amount of movement and the elapsed time are indicators for evaluating the deterioration of the accuracy of the three-dimensional measurement result. When the amount of movement and the elapsed time are each equal to or greater than their respective threshold values, it is determined that the accuracy deterioration has exceeded the allowable limit, and a message is displayed for accuracy improvement. Note that the message may be always displayed.

[0079] Note that in this embodiment, the imaging device 1 performs three-dimensional measurement processing (point cloud generation processing), but the three-dimensional measurement processing may be performed by a server device (not shown) that can communicate with the imaging device 1.

[0080] Next, the shooting screen 101 displayed on the display 13 will be described. FIGS. 8 and 9 are explanatory diagrams showing the shooting screen 101.

[0081] The shooting screen 101 is provided with a main window 102 (main image display frame) and a sub-window 103 (sub-image display frame). The display magnifications are different between the main window 102 and the sub-window 103. In the main window 102, the image is displayed enlarged, and in the sub-window 103, the image is displayed reduced.

[0082] In addition, the first support image 121 and the second support image 122 are each displayed on the main window 102 and the sub-window 103. In the shooting screen 101 shown in FIG. 8, the second support image 122 is enlarged and displayed on the main window 102, and the first support image 121 is reduced and displayed on the sub-window 103. In the shooting screen 101 shown in FIG. 9, the first support image 121 is enlarged and displayed on the main window 102, and the second support image 122 is reduced and displayed on the sub-window 103.

[0083] The first support image 121 includes a support image in which the completed range and the recommended implementation range are visualized on the overhead point cloud image (see FIG. 3), a support image in which the completed range and the recommended implementation range are visualized on the overhead point cloud path image (see FIG. 4), and a support image in which the completed range and the recommended implementation range are visualized on the overhead path image (see FIG. 5), and any of these images is displayed according to the setting. In the examples shown in FIGS. 8 and 9, as the first support image 121, a support image in which the completed range and the recommended implementation range are visualized on the overhead point cloud image (see FIG. 3) is displayed.

[0084] The second support image 122 is a support image in which the completed range and the recommended implementation range are visualized on the captured image (see FIG. 6). The captured image is a real-time captured image output from the visible camera 21.

[0085] Note that either the first support image 121 or the second support image 122 displayed on the main window 102 and the sub-window 103 may be displayed as an overhead image (overhead point cloud image, overhead point cloud path image, overhead path image) or a captured image without visualizing the completed range and the recommended implementation range. Whether or not to visualize the completed range and the recommended implementation range in the main window 102 and the sub-window 103 may be set by the user in advance.

[0086] In addition, on the shooting screen 101, a "Shooting Start" button 105 and a "Recording Confirmation" button 106 are provided. When the user operates the "Shooting Start" button 105, shooting by the visible camera 21 starts, and the captured images are stored in the memory 15. When the user operates the "Recording Confirmation" button 106, the mode changes to play back the captured images stored in the memory 15. Thereby, the user can confirm whether the shooting was properly performed.

[0087] In addition, on the shooting screen 101, a "Screen Switch" button 107 and a checkbox 108 regarding the display of the sub-window 103 are provided. When the user operates the "Screen Switch" button 107, the state of the shooting screen 101 shown in FIG. 8 and the state of the shooting screen 101 shown in FIG. 9 can be switched. That is, the state where the captured image is displayed in the main window 102 and the overview map is displayed in the sub-window 103, and the state where the overview map is displayed in the main window 102 and the captured image is displayed in the sub-window 103 are switched. When the user inputs a check to the checkbox 108, the sub-window 103 transitions to a non-display state.

[0088] In addition, a message window 110 is provided on the shooting screen 101. Messages to notify the user are displayed in the message window 110. Specifically, messages regarding guidance on the shooting operation and various warnings are displayed.

[0089] In particular, during the shooting operation, when there is a place where loop closing processing is recommended, that is, a place where loop closing processing has not been performed, or a place where the accuracy of the three-dimensional measurement result is low, a support message is displayed to prompt the user to perform the shooting operation for the place where loop closing processing is recommended, that is, the recommended implementation range drawn in red. In the examples shown in FIGS. 8 and 9, a message "Please perform shooting so that loop closing is performed in the red area." is displayed.

[0090] In this way, in the present embodiment, a real-time captured image is displayed on the shooting screen 101. As a result, the user can check the current shooting status during the shooting operation. Further, since the completed area and the recommended implementation area (uncompleted area) are visualized on the captured image, the user can easily determine whether the visible place in front of the eyes is the completed area or the recommended implementation area.

[0091] By the way, in the present embodiment, there are a method of visualizing (color-coding) an area where the loop closing process has not been performed as a recommended implementation area, and a method of visualizing an area with low accuracy of the three-dimensional measurement result as a recommended implementation area. These two methods of visualizing the recommended implementation area can be switched appropriately. For example, the method of visualizing the recommended implementation area may be preset in a setting screen (not shown). Also, the user may be able to specify the method of visualizing the recommended implementation area on the shooting screen 101.

[0092] As described above, embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments with changes, replacements, additions, omissions, etc. Also, it is possible to form a new embodiment by combining the respective components described in the above embodiments.

Industrial Applicability

[0093] The imaging device, imaging operation support method, and imaging operation support program according to the present invention have the effect of improving the accuracy of three-dimensional measurement using SLAM by clearly presenting the necessity of loop closing processing to the user during the imaging operation and prompting the user to perform an appropriate imaging operation, and for performing a three-dimensional measurement process of generating three-dimensional spatial information of a measurement target location based on the captured images of each point of the measurement target location, it is useful as an imaging device that captures each point of the measurement target location, and an imaging operation support method and an imaging operation support program that support the user's imaging operation using the imaging device.

Explanation of Reference Numerals

[0094] 1: Imaging device 11: Device body 12: Sensor unit 13: Display (display unit) 14: Input device 15: Memory 16: Processor 21: Visible camera (imaging unit) 101: Imaging screen 102: Main window (main image display frame) 103: Sub window (sub image display frame) 110: Message window 121: First support image 122: Second support image

Claims

1. An imaging device that captures each point of a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on captured images of each point of the measurement target location, a device body held by a user, an imaging unit provided on the device body for imaging the measurement target location, a display unit that displays support information regarding an imaging operation for sequentially imaging each point of the measurement target location by the imaging unit while the user holds the device body and moves the measurement target location, and a processor that controls the imaging unit and the display unit, wherein the processor displays, on the display unit, a support image visualizing a range where loop closing processing has been performed in the three-dimensional measurement process as the support information. The imaging device is characterized by this.

2. The processor displays, on the display unit, the support image visualizing the range where loop closing processing has been performed in at least one of an aerial image generated from the measurement result of the three-dimensional measurement process and the current captured image. The imaging device according to claim 1 is characterized by this.

3. The processor visualizes, in the support image, the range where loop closing processing has been performed in a display mode different from the range where performing loop closing processing is recommended. The imaging device according to claim 1 is characterized by this.

4. The processor acquires the number of times of execution in the range where loop closing processing has been performed. The imaging device according to claim 1 is characterized by this.

5. The processor changes the display mode of the range where loop closing processing has been performed in the support image according to the number of times of execution. The imaging device according to claim 4 is characterized by this.

6. An imaging operation support method in which a processor performs a process for supporting an imaging operation of a user who holds an imaging device and sequentially images each point of a measurement target location while moving within the measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on captured images of each point of the measurement target location, wherein, as support information regarding the imaging operation, a support image visualizing a range where loop closing processing has been performed in the three-dimensional measurement process is displayed on a display unit. The imaging operation support method is characterized by this.

7. In order to perform a three-dimensional measurement process for generating three-dimensional spatial information of a measurement target location based on captured images of each point in the measurement target location, a shooting work support program that causes a processor to execute a process for supporting a user's shooting work of sequentially capturing each point in the measurement target location by the imaging device while moving the imaging device within the measurement target location while holding the imaging device, A shooting work support program characterized by displaying, on a display unit, a support image that visualizes a range where loop closing processing has been performed in the three-dimensional measurement process as support information regarding the shooting work.

Images