Imaging device, imaging work support method, and imaging work support program

Abstract

To present the necessity of a loop closing process to a user during image capturing, in an easily understood manner, to prompt the user to perform appropriate imaging operation, thereby improving the accuracy of three-dimensional measurement using SLAM.SOLUTION: An imaging apparatus images spots of places to be measured, for performing three-dimensional measurement processing to generate three-dimensional spatial information of the places to be measured on the basis of captured images of the spots of the places to be measured. An imaging screen 101 displayed on a display unit displays support information on an imaging operation in which a visible camera sequentially images the spots of places to be measured while a user holding the body of the apparatus moves to the places to be measured. Specifically, a main window 102 and a sub window 103 display, as the support information, a support image obtained by visualizing a range in which a loop closing process is recommended in the three-dimensional measurement processing.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a photographing device that photographs each point of a measurement target location, and a photographing work support method and a photographing work support program that support a user's photographing work using the photographing device, in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location.

Background Art

[0002] A technique of three-dimensional measurement for generating three-dimensional spatial information (map data) regarding a measurement target location based on a photographed image of the measurement target location is known. In this three-dimensional measurement, in recent years, the SLAM (Simultaneous Localization And Mapping) method has attracted attention. In the SLAM method, it is possible to generate three-dimensional spatial information as a measurement result and the position information of the own vehicle based on the photographed images of each point of the measurement target location by holding a photographing device on a moving body. In particular, when a user performs a photographing operation of moving within a measurement target location and photographing while using a hand-held and portable photographing device, and acquires photographed images of each point of the measurement target location, three-dimensional measurement can be performed easily.

[0003] On the other hand, in three-dimensional measurement using SLAM, since relative self-position estimation is repeated, accumulation of errors cannot be avoided. Such accumulation of errors in self-position estimation causes a significant decrease in the accuracy of the three-dimensional measurement result.

[0004] Therefore, as a method for improving the accuracy degradation due to such cumulative errors, conventionally, when an error occurs during the creation of three-dimensional spatial information (map data), it is possible to restart the creation of three-dimensional spatial information from the middle without starting over from the beginning (see Patent Document 1). Further, in this technique, an event (loss) that causes an error is detected, and the user can grasp the position where the creation of three-dimensional spatial information is restarted by presenting the period during which the event occurred to the user.

[0005] Also, as a method for improving the accuracy degradation due to cumulative error, a loop closing technique has been conventionally known (see Non-Patent Document 1). In this technique, when it is detected that the moving path of the own vehicle forms a loop, that is, the own vehicle has returned to a point where it was photographed previously, the self-position estimation result (past position) obtained at the time of the previous photographing at that point is regarded as the correct position, and the positions in the path from the current position to the past position are corrected, whereby the cumulative error can be eliminated. According to this technique, like the technique disclosed in Patent Document 1, it is possible to save the trouble of re-taking pictures midway.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Non-Patent Documents

[0007]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] In three-dimensional measurement using SLAM, in order to improve the accuracy degradation due to cumulative error, Non-Patent Document 1 as disclosed, the loop closing technique is effective. On the other hand, the loop closing process can be performed when the moving path forms a loop, that is, when the vehicle returns to a point where it was photographed previously. For this reason, it is important to deliberately create a state in which the loop closing process can be performed. Therefore, it is desirable to clearly present to the user during the photographing operation the necessity of the loop closing process. However, the conventional technology has not taken any consideration for such a demand.

[0009] Therefore, the main object of the present invention is to provide a photographing apparatus, a photographing work support method, and a photographing work support program that can improve the accuracy of three-dimensional measurement using SLAM by clearly presenting the necessity of loop closing processing to the user during the photographing work and prompting the user to perform appropriate photographing work.

Means for Solving the Problems

[0010] The photographing apparatus of the present invention is a photographing apparatus that photographs each point of a measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location, and includes a device body held by a user, a photographing unit provided on the device body for photographing the measurement target location, a display unit for displaying support information regarding a photographing work for sequentially photographing each point of the measurement target location by the photographing unit while the user holds the device body and moves within the measurement target location, and a processor for controlling the photographing unit and the display unit. The processor uses, as the support information, Display a path image representing the movement path of the own device on the display unit, and visualize the executed range and the recommended execution range of the loop closing process in the three-dimensional measurement process on the path image configured.

[0011] In addition, the photographing work support method of the present invention is for performing three-dimensional measurement processing for generating three-dimensional spatial information of a measurement target location based on the photographed images of each point of the measurement target location, Imaging device equipped with a display unit A photographing work support method in which a processor performs a process of supporting a user's photographing work of sequentially photographing each point of a measurement target location by the photographing apparatus while moving within the measurement target location while holding, and as support information regarding the photographing work, Display a path image representing the movement path of the imaging device on the display unit, and visualize the executed range and the recommended execution range of the loop closing process in the three-dimensional measurement process on the path image configured.

[0012] In addition, the photographing work support program of the present invention is for performing three-dimensional measurement processing for generating three-dimensional spatial information of a measurement target location based on the photographed images of each point of the measurement target location, Imaging device equipped with a display unit A photographing work support program that causes a processor to execute a process of supporting a user's photographing work of sequentially photographing each point of a measurement target location by the photographing apparatus while moving within the measurement target location while holding, and as support information regarding the photographing work, Display a path image representing the movement path of the imaging device on the display unit, and visualize the executed range and the recommended execution range of the loop closing process in the three-dimensional measurement process on the path imageIt is configured as follows.

Advantages of the Invention

[0013] According to the present invention, for a user (operator) during a shooting operation, Path image it is possible to clearly present the necessity of loop closing processing and prompt the user to perform an appropriate shooting operation. As a result, Path image by the user performing an appropriate shooting operation according to this, the accuracy of the three-dimensional measurement process using SLAM can be improved.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0015] The first invention made to solve the above problems is a photographing device that photographs each point of a measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location based on photographed images of each point of the measurement target location, including a device body held by a user, a photographing unit provided on the device body for photographing the measurement target location, a display unit for displaying support information regarding a photographing operation for sequentially photographing each point of the measurement target location by having the user hold the device body and move while photographing the measurement target location, and a processor for controlling the photographing unit and the display unit, wherein the processor is configured to use, as the support information, Display a path image representing the movement path of the own device on the display unit, and visualize the executed range and the recommended execution range of the loop closing process in the three-dimensional measurement process on the path image configured as.

[0016] According to this, for the user (operator) during the photographing operation, Path image it is possible to clearly present the necessity of loop closing processing and prompt the user to perform an appropriate photographing operation. As a result, Path image by having the user perform an appropriate photographing operation according to, it is possible to improve the accuracy of three-dimensional measurement processing using SLAM. Note that the range in which the implementation of loop closing processing is recommended may be visualized by area or by path.

[0017] Further, in the second invention, the processor is configured to display, on the display unit, The support information with the path image superimposed on the bird's-eye view point cloud image generated from the measurement result of the three-dimensional measurement processing.

[0018] According to this, It is possible to clearly present the necessity of the loop closing process on the bird's-eye view point cloud image

[0019] Further, in the third invention, in the Path image the unimplemented range of the loop closing processing is configured to be visualized as the Recommended execution range of the loop closing processing.

[0020] According to this, it is possible to appropriately present to the user the range in which the implementation of loop closing processing is recommended.

[0021] Further, in the fourth invention, the processor is thePath image In this case, a range where the accuracy of the measurement result of the three-dimensional measurement process is low is configured to be visualized as the Recommended execution range loop closing process.

[0022] According to this, the range in which the execution of the loop closing process is recommended can be appropriately presented to the user.

[0023] Further, in a fifth invention, the processor is configured to visualize the Path image loop closing process in the Color-code the executed range and the recommended execution range of the loop closing process above.

[0024] According to this, The executed range and the recommended execution range of the loop closing process can be clearly presented to the user

[0025] Further, a sixth invention is a photographing work support method in which a processor performs a process of supporting a user's photographing work of sequentially photographing each point of a measurement target location with the photographing device while moving within the measurement target location while holding the Imaging device equipped with a display unit for performing a three-dimensional measurement process of generating three-dimensional spatial information of the measurement target location based on photographed images of each point of the measurement target location, and as support information regarding the photographing work, it is configured to be Display a path image representing the movement path of the imaging device on the display unit, and visualize the executed range and the recommended execution range of the loop closing process in the three-dimensional measurement process on the path image as above.

[0026] According to this, similar to the first invention, by clearly presenting the necessity of the loop closing process to the user during the photographing work and prompting the user to perform an appropriate photographing work, the accuracy of three-dimensional measurement using SLAM can be improved.

[0027] Further, a seventh invention is a photographing work support program that causes a processor to execute a process of supporting a user's photographing work of sequentially photographing each point of a measurement target location with the photographing device while moving within the measurement target location while holding the Imaging device equipped with a display unit for performing a three-dimensional measurement process of generating three-dimensional spatial information of the measurement target location based on photographed images of each point of the measurement target location, and as support information regarding the photographing work, it is configured to be Display a path image representing the movement path of the imaging device on the display unit, and visualize the executed range and the recommended execution range of the loop closing process in the three-dimensional measurement process on the path image as above.

[0028] According to this, similar to the first invention, by clearly presenting the necessity of loop closing processing to the user during the shooting operation and prompting the user to perform appropriate shooting operations, the accuracy of three-dimensional measurement using SLAM can be improved.

[0029] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0030] FIG. 1 is an explanatory diagram showing a situation of a shooting operation performed by a user using the imaging device 1 according to the present embodiment. FIG. 2 is a plan view showing a measurement target location.

[0031] The imaging device 1 includes a device main body 11 and a sensor unit 12. The sensor unit 12 includes a visible camera 21 (imaging unit). The visible camera 21 is a monocular camera that detects visible light and captures a subject, and outputs a captured image, for example, a color image in the RGB format. Note that the imaging device 1 can be configured by a tablet terminal or a notebook PC.

[0032] The user (operator) walks while holding the device main body 11 of the imaging device 1. At this time, in the imaging device 1, the measurement target location is captured by the visible camera 21, and captured images of each point in the measurement target location are sequentially acquired.

[0033] In addition, in the present embodiment, the imaging device 1 serves as a three-dimensional measurement device. That is, in the imaging device 1, three-dimensional measurement processing is performed based on the captured images of each point sequentially acquired by the visible camera 21, and three-dimensional space information regarding the measurement target location is generated. In the three-dimensional measurement processing, point cloud data (environmental map) as three-dimensional space information regarding the measurement target location is generated using the SLAM method. At this time, together with the generation of the point cloud data, self-position estimation processing is performed, and the position of each self-position, that is, the shooting location, is estimated.

[0034] In addition, in the imaging device 1, when it is detected that the movement path of the own device forms a loop, that is, the own device has returned to a point where it has previously taken a picture, loop closing processing is performed. In the loop closing processing, regarding the point where the picture was previously taken, the self-position estimation result (past position) obtained at the time of that previous picture taking is regarded as the correct position, and the positions of each picture taking point in the path from the current position to the past position are corrected.

[0035] In the example shown in FIG. 2, imaging starts from the imaging start point, travels around a part of the measurement target area, and returns to the imaging start point. Further, imaging is continued, and another area of the measurement target area is imaged. In this case, loop closing processing is performed when returning to the imaging start point. As a result, the accuracy of the point cloud data and the self-position estimation result regarding each imaging point included in the loop is improved.

[0036] Next, a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud image will be described. FIG. 3 is an explanatory diagram showing a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud image.

[0037] In the present embodiment, a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud image is generated, and the support image is presented to the user during the imaging operation. The bird's-eye view point cloud image is an image (rendering) of each point of the point cloud data viewed from an aerial viewpoint. The completed range represents the range where the loop closing processing has been performed. The recommended implementation range represents the range where the loop closing processing is recommended, that is, the range where the loop closing processing has not been performed, or the range where the accuracy of the three-dimensional measurement result is low.

[0038] The completed area (completed range) and the recommended implementation area (recommended implementation range) are drawn with color separation. For example, the completed area is drawn in blue, and the recommended implementation area is drawn in red. Specifically, in the bird's-eye view point cloud image, each point included in the completed area is drawn in blue, and each point included in the recommended implementation area is drawn in red.

[0039] At this time, each point drawn in the overhead point cloud image with a predetermined size is colored with a predetermined color. Also, spheres with a predetermined diameter centered on each point may be drawn with a predetermined color. Further, the target space may be divided into cubic voxels, and the voxels including each point may be drawn with a predetermined color.

[0040] By viewing the support image, the user grasps the range where the shooting work for the loop closing process is necessary and performs the shooting work. Specifically, the user performs the shooting work on the recommended implementation area drawn in red. That is, the shooting work is performed so as to pass through the recommended implementation area. Note that the specific content of the shooting work, that is, the movement path and the like, may be appropriately determined by the user according to the site situation and the like.

[0041] As described above, in this embodiment, the support image can clearly present the necessity of the loop closing process to the user. Further, the range where the shooting work for the loop closing process is necessary can be clearly presented to the user. Thereby, the user can be prompted to perform appropriate shooting work, and since the loop closing process is surely carried out, the accuracy of the three-dimensional measurement process can be improved.

[0042] Note that both the implemented area and the recommended implementation area represent the range where shooting has been performed in the measurement target location. That is, the area not included in either the implemented area or the recommended implementation area in the measurement target location represents the range where shooting has not been performed. Therefore, the user can also confirm the range where shooting has not been performed by viewing the support image.

[0043] Also, in this embodiment, the number of times the loop closing process is performed is counted, and visualization (color coding) of the area according to the number of times the loop closing process is performed is performed. For example, the area where the number of times is 1 is drawn in green, the area where the number of times is 2 is drawn in blue, and the area where the number of times is 0 is drawn in red.

[0044] Next, an assistance image in which the completed range and the recommended implementation range are visualized on the overhead point cloud path image will be described. FIG. 4 is an explanatory diagram showing an assistance image in which the completed range and the recommended implementation range are visualized on the overhead point cloud path image.

[0045] In this embodiment, an assistance image in which the completed range and the recommended implementation range are visualized on the overhead point cloud path image is generated, and the assistance image is presented to the user during the shooting operation. The overhead point cloud path image is one in which a path (movement trajectory) is superimposed on the overhead point cloud image. The overhead point cloud image is an image (rendering) of each point of the point cloud data as viewed from an overhead viewpoint. The path is drawn by connecting the self-position estimation results based on each captured image, that is, the shooting points of each time, with a line on the overhead point cloud image.

[0046] The completed path (completed range) and the recommended path (recommended implementation range) are drawn with color separation. For example, the completed path is drawn in blue, and the recommended path is drawn in red.

[0047] In addition, a shooting point mark is drawn on the assistance image. The shooting point mark represents the shooting point and the shooting direction. Specifically, the shooting point mark is drawn as an isosceles triangle, the vertex of the isosceles triangle represents the shooting point, and the direction from the vertex to the base represents the shooting direction.

[0048] By viewing the assistance image, the user grasps the range where the shooting operation for the loop closing process is necessary and performs the shooting operation. Specifically, the user performs the shooting operation on the recommended path drawn in red. That is, the shooting operation is performed so that a loop including the recommended path is formed.

[0049] In addition, in this embodiment, the number of times the loop closing process is performed is counted, and the visualization (color separation) of the path is performed according to the number of times the loop closing process is performed. For example, a path with 1 execution is drawn in green, a path with 2 executions is drawn in blue, and a path with 0 executions is drawn in red.

[0050] Next, an assistance image in which the completed range and the recommended implementation range are visualized on the bird's-eye view path image will be described. FIG. 5 is an explanatory diagram showing the assistance image in which the completed range and the recommended implementation range are visualized on the bird's-eye view path image.

[0051] In the present embodiment, an assistance image in which the completed range and the recommended implementation range are visualized on the bird's-eye view path image is generated, and the assistance image is presented to the user during the shooting operation. The bird's-eye view path image is in a state where the bird's-eye view point cloud image is removed from the bird's-eye view point cloud path image (see FIG. 4), and the self-position estimation result based on each captured image, that is, the line connecting each shooting point, is drawn as viewed from an aerial viewpoint.

[0052] Next, an assistance image in which the completed range and the recommended implementation range are visualized on the captured image will be described. FIG. 6 is an explanatory diagram showing the assistance image in which the completed range and the recommended implementation range are visualized on the captured image.

[0053] In the present embodiment, an assistance image in which the completed range and the recommended implementation range are visualized on the captured image is generated, and the assistance image is presented to the user during the shooting operation. The captured image is the current captured image, that is, an image captured in real time output from the visible camera 21.

[0054] The completed area (completed range) and the recommended implementation area (recommended implementation range) are drawn with color separation. For example, the completed area is drawn in blue, and the recommended implementation area is drawn in red. Specifically, an image representing the completed area drawn in a predetermined color and an image representing the recommended implementation area drawn in a predetermined color are superimposed on the captured image in a semi-transparent state.

[0055] In this way, in the present embodiment, the completed range and the recommended implementation range (unimplemented range) are visualized on the captured image. Thereby, the user can easily determine whether the place visible in front of the eyes is the completed range or the recommended implementation range. Then, the user performs the shooting operation so as to include the recommended implementation range.

[0056] Next, the schematic configuration of the imaging device 1 will be described. FIG. 7 is a block diagram showing the schematic configuration of the imaging device 1.

[0057] In addition to the sensor unit 12, the imaging device 1 includes a display 13 (display unit), an input device 14, a memory 15, and a processor 16 (CPU).

[0058] In addition to the visible camera 21, the sensor unit 12 includes a depth camera 22 and an IMU 23 (Inertial Measurement Unit). The depth camera 22 is a stereo camera that detects infrared light and images a subject, and outputs depth information (distance image) as a detection result. Based on the detection result of the depth camera 22, the distance to the subject can be measured. The IMU 23 detects three-dimensional angular velocity and acceleration. Based on the detection result of the IMU 23, the movement amount and rotation amount of the imaging device 1 can be measured. Note that the visible camera 21, the depth camera 22, and the IMU 23 may not be integrated as a sensor unit. Also, a configuration in which the depth camera 22 and the IMU 23 are omitted and only the visible camera 21 is provided may be used.

[0059] The display 13 presents various types of information related to the imaging operation to the user, and displays an imaging screen 101 (see FIGS. 8 and 9), etc. The input device 14 is for the user to perform an input operation. The input device 14 may be a keyboard, a mouse, a touch pad, a touch panel, or the like. When the imaging device 1 is configured as a tablet terminal, a touch panel display in which the touch panel as the input device 14 and the display panel as the display 13 are integrated is provided.

[0060] The memory 15 stores programs executed by the processor 16 and the like. The memory 15 also stores the captured image of the visible camera 21, the detection results of the depth camera 22 and the IMU 23. The memory 15 also stores the measurement results generated by the processor 16.

[0061] The processor 16 performs various processes by executing the program stored in the memory 15. In the present embodiment, the processor 16 performs a detection information acquisition process, a point cloud generation process, a first support image generation process, a second support image generation process, a screen control process, a message notification process, and the like.

[0062] In the detection information acquisition process, the processor 16 acquires the captured image of the visible camera 21. Further, the processor 16 acquires the detection results of the depth camera 22 and the IMU 23.

[0063] In the point cloud generation process (3D measurement process), the processor 16 generates point cloud data (environmental map) as three-dimensional space information regarding the measurement target location using the SLAM method based on the captured images of each point sequentially acquired by the visible camera 21. Further, in the point cloud generation process, self-position estimation is performed in conjunction with the generation of the point cloud data, and the self-position at each time, that is, the position of the shooting location is acquired.

[0064] Further, the point cloud generation process includes a loop detection process. In the loop detection process, the processor 16 detects that the path of the own vehicle forms a loop, that is, the own vehicle has returned to a point that was previously photographed. At this time, in the detection of the previously photographed point, it is not compared with a preset specified captured image, but when the current captured image is similar to any of the past captured images, it is determined that the own vehicle has returned to a point that was previously photographed. Note that the similarity of the captured images may be determined based on the feature amounts extracted from the captured images.

[0065] Further, the point cloud generation process includes a loop closing process. When a loop is detected in the loop detection process, the loop closing process is executed. In the loop closing process, the processor 16 regards the self-position estimation result (past position) acquired at the time of the previous shooting regarding the previously photographed point as the correct position, and corrects the positions of each shooting point in the path from the current position to the past position.

[0066] In addition, the point cloud generation process includes an execution count acquisition process. In the execution count acquisition process, the processor 16 counts the number of times the loop closing process has been executed for each shooting location. The execution count information is added to the captured image (frame) of each shooting location. Also, the execution count information is added to each point in the point cloud data. At this time, for example, the execution count information is added to each point included in the point cloud data generated from the captured image (frame) of the shooting location where the loop closing process has been executed. Note that the execution count information also includes information regarding the execution or non-execution of the loop closing process. When the execution count is 0 times, it is not executed, and when the execution count is 1 time or more, it is considered executed.

[0067] In addition, the point cloud generation process includes an accuracy acquisition process. In the accuracy acquisition process, the processor 16 acquires the accuracy of the three-dimensional measurement results (point cloud data and self-position estimation results) at each shooting location. Specifically, the accuracy of each shooting location is acquired based on the separation amount from the location where the loop closing process has been executed. Specifically, the separation amount is, for example, the moving distance (length of the movement trajectory) from the location where the loop closing process was executed, the elapsed time since the time when the loop closing process was executed, the number of captured images (number of frames) taken after the loop closing process was executed, and the like. Note that the accuracy may be determined by combining the moving distance, elapsed time, and number of frames as the separation amount. Furthermore, the accuracy may be determined by combining the separation amount with the moving speed. The accuracy information is added to the captured image (frame) of each shooting location. Also, the accuracy information may be added to each point in the point cloud data.

[0068] Also, regardless of the execution status of the loop closing process, the accuracy information of each point in the point cloud data can be acquired. For example, when the measurement results (point cloud data) based on the captured image at each shooting location do not match the detection results of the depth camera 22 or the IMU 23, it can be determined that the accuracy is low. Also, the accuracy may be determined by comparing the measurement results based on the front and rear captured images (frames). In this case, the accuracy information is added to each point in the point cloud data.

[0069] In the point cloud generation process, in addition to the captured image of the visible camera 21, the detection results of the depth camera 22 and the IMU 23 may also be used. Specifically, based on the detection result of the depth camera 22, the relative position information (distance from the imaging device 1) of each feature point extracted from the captured image of the visible camera 21 with respect to the imaging device 1 can be obtained, and the self-position estimation result based on the captured image of the visible camera 21 can be corrected. Also, based on the detection result of the IMU 23, the relative position information of each imaging location can be obtained, and based on the relative position information of that imaging location, the self-position estimation result based on the captured image of the visible camera 21 can be corrected.

[0070] In the first support image generation process, the processor 16 generates a support image (first support image) in which the completed range and the recommended implementation range are visualized on the bird's-eye view image. Specifically, a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud image (see FIG. 3), a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud path image (see FIG. 4), and a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view path image (see FIG. 5) are generated.

[0071] In the visualization of the completed range and the recommended implementation range on the bird's-eye view point cloud image, based on the implementation count information added to each point in the point cloud data, the information on the completed range and the recommended implementation range is obtained, and each point on the point cloud image included in each of the completed range and the recommended implementation range is colored (point cloud coloring process).

[0072] Also, in the visualization of the completed range and the recommended implementation range on the bird's-eye view point cloud path image and the bird's-eye view path image, based on the implementation count information added to the captured image (frame) of each imaging location, the image of the path (movement trajectory) connecting the positions of each imaging location with a line is colored (path coloring process).

[0073] In addition, in the present embodiment, in the visualization of the recommended implementation range on the bird's-eye view point cloud image, a range with low accuracy of the three-dimensional measurement result is used as the recommended implementation range, and based on the accuracy information added to each point in the point cloud data, points with low accuracy are colored with the color of the recommended implementation range. Note that an area with many points with low accuracy in the point cloud data may be extracted as the recommended implementation range.

[0074] In addition, in the present embodiment, in the visualization of the recommended implementation range on the bird's-eye view point cloud path image or the bird's-eye view path image, a range with low accuracy among the paths connecting the respective shooting points is used as the recommended implementation range, and based on the accuracy information added to the shooting image (frame) of each shooting point, information representing the recommended implementation range is acquired, and the path image is colored.

[0075] In the second support image generation process, the processor 16 generates a support image (second support image) (see FIG. 6) in which the implemented range and the recommended implementation range are visualized on the captured image. Specifically, an image representing the implemented area drawn in a predetermined color and an image representing the recommended implementation area drawn in a predetermined color are generated (superimposed image generation process), and the images representing the implemented area and the recommended implementation area are superimposed on the captured image in a semi-transparent state. Note that in the superimposed image generation process, information representing the ranges of the implemented area and the recommended implementation area may be acquired based on the implementation frequency information of each point in the point cloud data viewed from the same viewpoint as the captured image.

[0076] In the screen control process, the processor 16 controls the screen to be displayed on the display 13. In the present embodiment, a shooting screen 101 (see FIGS. 8 and 9) is generated and displayed on the display 13. On the shooting screen 101, the first support image (see FIGS. 3, 4, and 5) generated in the first support image generation process and the second support image (see FIG. 6) generated in the second support image generation process are displayed.

[0077] In the message notification process, the processor 16 performs a process of notifying the user of messages regarding guidance for the shooting operation, various warnings, and the like. In the present embodiment, a message prompting the user to perform a shooting operation for loop closing processing is displayed on the shooting screen 101 (see FIGS. 8 and 9).

[0078] At this time, when a predetermined notification condition is satisfied, a message is displayed. For example, when the amount of movement from the point where the loop closing process is performed is equal to or greater than a predetermined threshold value, a message is displayed. Also, when the elapsed time since the loop closing process was performed is equal to or greater than a predetermined threshold value, a message is displayed. Here, the amount of movement and the elapsed time are indicators for evaluating the degradation of the accuracy of the three-dimensional measurement result. When the amount of movement and the elapsed time are each equal to or greater than their respective threshold values, it is determined that the accuracy degradation has exceeded the allowable limit, and a message is displayed for accuracy improvement. Note that the message may always be displayed.

[0079] Note that in the present embodiment, the imaging device 1 performs three-dimensional measurement processing (point cloud generation processing), but the three-dimensional measurement processing may be performed by a server device (not shown) that can communicate with the imaging device 1.

[0080] Next, the shooting screen 101 displayed on the display 13 will be described. FIGS. 8 and 9 are explanatory diagrams showing the shooting screen 101.

[0081] The shooting screen 101 is provided with a main window 102 (main image display frame) and a sub-window 103 (sub-image display frame). The display magnifications are different between the main window 102 and the sub-window 103. In the main window 102, the image is displayed enlarged, and in the sub-window 103, the image is displayed reduced.

[0082] In addition, the first support image 121 and the second support image 122 are each displayed on the main window 102 and the sub-window 103. In the shooting screen 101 shown in FIG. 8, the second support image 122 is enlarged and displayed on the main window 102, and the first support image 121 is reduced and displayed on the sub-window 103. In the shooting screen 101 shown in FIG. 9, the first support image 121 is enlarged and displayed on the main window 102, and the second support image 122 is reduced and displayed on the sub-window 103.

[0083] The first support image 121 includes a support image in which the completed range and the recommended implementation range are visualized on the overhead point cloud image (see FIG. 3), a support image in which the completed range and the recommended implementation range are visualized on the overhead point cloud path image (see FIG. 4), and a support image in which the completed range and the recommended implementation range are visualized on the overhead path image (see FIG. 5), and any one of the images is displayed according to the setting. In the examples shown in FIGS. 8 and 9, as the first support image 121, a support image in which the completed range and the recommended implementation range are visualized on the overhead point cloud image (see FIG. 3) is displayed.

[0084] The second support image 122 is a support image in which the completed range and the recommended implementation range are visualized on the captured image (see FIG. 6). The captured image is a real-time captured image output from the visible camera 21.

[0085] Note that either the first support image 121 or the second support image 122 displayed on the main window 102 and the sub-window 103 may be displayed as an overhead image (overhead point cloud image, overhead point cloud path image, overhead path image) or a captured image without visualizing the completed range and the recommended implementation range. Whether or not to visualize the completed range and the recommended implementation range in the main window 102 and the sub-window 103 may be set by the user in advance.

[0086] In addition, on the shooting screen 101, a "Shooting Start" button 105 and a "Recording Confirmation" button 106 are provided. When the user operates the "Shooting Start" button 105, shooting by the visible camera 21 starts, and the shot images are stored in the memory 15. When the user operates the "Recording Confirmation" button 106, the mode shifts to one for playing back the shot images stored in the memory 15. Thereby, the user can confirm whether the shooting has been appropriately performed.

[0087] In addition, on the shooting screen 101, a "Screen Switch" button 107 and a check box 108 regarding the display of the sub-window 103 are provided. When the user operates the "Screen Switch" button 107, the state of the shooting screen 101 shown in FIG. 8 and the state of the shooting screen 101 shown in FIG. 9 can be switched. That is, the state where a shot image is displayed on the main window 102 and an aerial map is displayed on the sub-window 103, and the state where an aerial map is displayed on the main window 102 and a shot image is displayed on the sub-window 103 can be switched. When the user inputs a check to the check box 108, the mode shifts to a state where the sub-window 103 is not displayed.

[0088] In addition, a message window 110 is provided on the shooting screen 101. Messages to notify the user are displayed in the message window 110. Specifically, messages regarding guidance on the shooting operation and various warnings are displayed.

[0089] In particular, during the shooting operation, when there is a place where loop closing processing is recommended, that is, a place where loop closing processing has not been performed, or a place where the accuracy of the three-dimensional measurement result is low, a support message is displayed to prompt the user to perform the shooting operation on the place where loop closing processing is recommended, that is, the recommended implementation range drawn in red. In the examples shown in FIGS. 8 and 9, a message "Please perform shooting so that loop closing is carried out in the red area." is displayed.

[0090] In this way, in the present embodiment, a real-time captured image is displayed on the shooting screen 101. As a result, the user can check the current shooting situation during the shooting operation. Further, since the completed range and the recommended implementation range (uncompleted range) are visualized on the captured image, the user can easily determine whether the visible place in front of him / her is the completed range or the recommended implementation range.

[0091] By the way, in the present embodiment, there are a method of visualizing (color-coding) an area where the loop closing process has not been performed as the recommended implementation range, and a method of visualizing an area with low accuracy of the three-dimensional measurement result as the recommended implementation range. These two methods of visualizing the recommended implementation range can be switched as appropriate. For example, the method of visualizing the recommended implementation range may be preset in a setting screen (not shown). Alternatively, the user may be able to specify the method of visualizing the recommended implementation range on the shooting screen 101.

[0092] As described above, embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments in which changes, replacements, additions, omissions, etc. are made. Further, it is also possible to combine the respective components described in the above embodiments to form a new embodiment.

Industrial Applicability

[0093] The imaging device, imaging operation support method, and imaging operation support program according to the present invention have an effect of improving the accuracy of three-dimensional measurement using SLAM by clearly presenting the necessity of loop closing processing to the user during the imaging operation and prompting the user to perform an appropriate imaging operation, and are useful as an imaging device that captures each point of a measurement target location, and an imaging operation support method and an imaging operation support program that support the user's imaging operation using the imaging device, for performing three-dimensional measurement processing for generating three-dimensional space information of the measurement target location based on the captured images of each point of the measurement target location.

Explanation of Reference Numerals

[0094] 1: Imaging device 11: Device body 12: Sensor unit 13: Display (display unit) 14: Input device 15: Memory 16: Processor 21: Visible camera (imaging unit) 101: Imaging screen 102: Main window (main image display frame) 103: Sub window (sub image display frame) 110: Message window 121: First support image 122: Second support image

Claims

1. An imaging device that captures each point of a measurement target location in order to perform 3D measurement processing for generating 3D spatial information of the measurement target location based on captured images of each point of the measurement target location, comprising: a device body held by a user; an imaging unit provided on the device body for imaging the measurement target location; a display unit that displays support information regarding an imaging operation for sequentially imaging each point of the measurement target location by having the user hold the device body and move while imaging each point of the measurement target location with the imaging unit; a processor that controls the imaging unit and the display unit, wherein the processor: displays, as the support information, a path image representing the movement path of the own device on the display unit, and visualizes a completed range and a recommended implementation range of loop closing processing in the 3D measurement processing on the path image. An imaging device characterized by this.

2. The processor: displays, on the display unit, the support information in which the path image is superimposed on an overhead point cloud image generated from the measurement result of the 3D measurement processing. The imaging device according to claim 1, characterized by this.

3. The processor: visualizes, in the path image, a range where the loop closing processing has not been performed as a recommended implementation range of the loop closing processing. The imaging device according to claim 1, characterized by this.

4. The processor: visualizes, in the path image, a range where the accuracy of the measurement result of the 3D measurement processing is low as a recommended implementation range of the loop closing processing. The imaging device according to claim 1, characterized by this.

5. The processor: visualizes, in the path image, by color-coding the completed range of the loop closing processing and the recommended implementation range of the loop closing processing. The imaging device according to claim 1, characterized by this.

6. An imaging operation support method in which a processor performs processing to support an imaging operation of a user who holds an imaging device provided with a display unit and sequentially images each point of a measurement target location while moving within the measurement target location in order to perform 3D measurement processing for generating 3D spatial information of the measurement target location based on captured images of each point of the measurement target location, wherein, as support information regarding the imaging operation, a path image representing the movement path of the imaging device is displayed on the display unit, and a completed range and a recommended implementation range of loop closing processing in the 3D measurement processing are visualized on the path image. An imaging operation support method characterized by this. Claim 7 A photographing work support program that causes a processor to execute a process for supporting a user's photographing work of holding a photographing device having a display unit and sequentially photographing each point of a measurement target location while moving within the measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on photographed images of each point of the measurement target location, The photographing work support program is characterized in that, as support information regarding the photographing work, a path image representing a movement path of the photographing device is displayed on the display unit, and a completed range and a recommended execution range of loop closing processing in the three-dimensional measurement process are visualized on the path image.

Images