Imaging device, imaging work support method, and imaging work support program

Abstract

To present the necessity of a loop closing process to a user during image capturing, in an easily understood manner, to prompt the user to perform appropriate imaging operation, thereby improving the accuracy of three-dimensional measurement using SLAM.SOLUTION: An imaging apparatus images spots of places to be measured, for performing three-dimensional measurement processing to generate three-dimensional spatial information of the places to be measured on the basis of captured images of the spots of the places to be measured. An imaging screen 101 displayed on a display unit displays support information on an imaging operation in which a visible camera sequentially images the spots of places to be measured while a user holding the body of the apparatus moves to the places to be measured. Specifically, a message window 110 displays, as the support information, a support message which prompts an imaging operation necessary for executing a loop closing process in the three-dimensional measurement processing.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a photographing device that photographs each point of a measurement target location, and a photographing work support method and a photographing work support program that support a user's photographing work using the photographing device, in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location based on photographed images of each point of the measurement target location.

Background Art

[0002] A three-dimensional measurement technique for generating three-dimensional spatial information (map data) regarding a measurement target location based on a photographed image of the measurement target location is known. In this three-dimensional measurement, in recent years, the SLAM (Simultaneous Localization And Mapping) method has attracted attention. In the SLAM method, three-dimensional spatial information as a measurement result and the position information of the own vehicle can be generated based on photographed images of each point of the measurement target location with a photographing device held by a moving object. In particular, when a user performs a photographing operation of moving within a measurement target location and photographing while using a hand-held and portable photographing device, photographed images of each point of the measurement target location can be acquired, and three-dimensional measurement can be performed easily.

[0003] On the other hand, in three-dimensional measurement using SLAM, since relative self-position estimation is repeated, accumulation of errors cannot be avoided. Such accumulation of errors in self-position estimation causes a significant decrease in the accuracy of the three-dimensional measurement result.

[0004] Therefore, as a method for improving the accuracy degradation due to such cumulative errors, conventionally, when an error occurs during the creation of three-dimensional spatial information (map data), it is possible to restart the creation of three-dimensional spatial information from the middle without starting over from the beginning (see Patent Document 1). Further, in this technique, an event (loss) that causes an error is detected, and the period during which the event has occurred is presented to the user, so that the user can grasp the position at which to restart the creation of three-dimensional spatial information.

[0005] Also, as a method for improving the accuracy degradation due to cumulative error, a loop closing technique has been conventionally known (see Non-Patent Document 1). In this technique, when it is detected that the movement path of the own vehicle forms a loop, that is, the own vehicle has returned to a point that was photographed previously, the self-position estimation result (past position) obtained at the time of the previous photographing at that point is regarded as the correct position, and the positions in the path from the current position to the past position are corrected, whereby the cumulative error can be eliminated. According to this technique, as in the technique disclosed in Patent Document 1, the trouble of re-taking photographs midway can be saved.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Non-Patent Documents

[0007]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] In three-dimensional measurement using SLAM, in order to improve the accuracy degradation due to cumulative error, Non-Patent Document 1 as disclosed in, the loop closing technique is effective. On the other hand, the loop closing process can be carried out when the movement path forms a loop, that is, when the own vehicle has returned to a point that was photographed previously. For this reason, it is important to deliberately create a state in which the loop closing process can be carried out. Therefore, it is desirable to clearly present the necessity of the loop closing process to the user during the photographing operation. However, in the conventional technique, no consideration has been given to such a demand.

[0009] Therefore, the main object of the present invention is to provide a photographing apparatus, a photographing work support method, and a photographing work support program that can improve the accuracy of three-dimensional measurement using SLAM by clearly presenting the necessity of loop closing processing to the user during photographing work and prompting the user to perform appropriate photographing work.

Means for Solving the Problems

[0010] The photographing apparatus of the present invention is a photographing apparatus that photographs each point of a measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location, and includes a device body held by a user, a photographing unit provided on the device body for photographing the measurement target location, a display unit for displaying support information regarding a photographing work for sequentially photographing each point of the measurement target location by the photographing unit while the user holds the device body and moves within the measurement target location, and a processor for controlling the photographing unit and the display unit. The processor is configured to display, as the support information, Support image visualizing the range where loop closing processing has been performed in the three-dimensional measurement processing and support message prompting the shooting work necessary for performing the loop closing processing on the display unit.

[0011] In addition, the photographing work support method of the present invention is a photographing work support method in which a processor performs a process of supporting a user's photographing work of sequentially photographing each point of a measurement target location by a photographing device while holding the photographing device and moving within the measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location. As support information regarding the photographing work, Support image visualizing the range where loop closing processing has been performed in the three-dimensional measurement processing and support message prompting the shooting work necessary for performing the loop closing processing is configured to be displayed on a display unit.

[0012] In addition, the photographing work support program of the present invention is a photographing work support program that causes a processor to execute a process of supporting a user's photographing work of sequentially photographing each point of a measurement target location by a photographing device while holding the photographing device and moving within the measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location. As support information regarding the photographing work, Support image visualizing the range where loop closing processing has been performed in the three-dimensional measurement processing and support message prompting the shooting work necessary for performing the loop closing processingIt is configured to be displayed on the display unit.

Advantages of the Invention

[0013] According to the present invention, for a user (operator) during a shooting operation, Support image and it is possible to clearly present the necessity of loop closing processing by a support message and prompt the user to perform an appropriate shooting operation. As a result, Support image and when the user performs an appropriate shooting operation according to the support message, the accuracy of the three-dimensional measurement processing using SLAM can be improved.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0015] A first invention made to solve the above problems is a photographing device that photographs each point of a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on a photographed image of each point of the measurement target location, including a device body held by a user, a photographing unit provided in the device body for photographing the measurement target location, a display unit for displaying support information regarding a photographing operation for sequentially photographing each point of the measurement target location by the photographing unit while the user holds the device body and moves the measurement target location, and a processor for controlling the photographing unit and the display unit. The processor is configured to display, on the display unit, as the support information, Support image visualizing the range where loop closing processing has been performed in the three-dimensional measurement processing and support message prompting the shooting work necessary for performing the loop closing processing in the above configuration.

[0016] According to this, for the user (operator) during the photographing operation, Support image and the necessity of loop closing processing can be clearly presented to the user by a support message, and the user can be prompted to perform an appropriate photographing operation. Thereby, Support image and by the user performing an appropriate photographing operation according to the support message, the accuracy of the three-dimensional measurement process using SLAM can be improved.

[0017] Further, in a second invention, the processor is configured to display, on the display unit, the support message for prompting the user to perform a photographing operation for a range where the implementation of the loop closing processing is recommended.

[0018] According to this, the range where the loop closing processing is necessary can be clearly presented to the user.

[0019] Further, in a third invention, at least one of an aerial image generated from the measurement result of the three-dimensional measurement process and the current photographed image is configured to be displayed on the display unit. The support image in the above configuration.

[0020] According to this, a range where loop closing processing is unnecessary is presented to the user in an easy-to-understand manner. In this case, for example, the range where loop closing processing has been performed may be displayed in color. Note that the range where loop closing processing has been performed may be visualized by an area or by a path. Also, the bird's-eye view image may be a bird's-eye view point cloud image, a bird's-eye view point cloud path image, or a bird's-eye view path image.

[0021] Also, in the fourth invention, the processor Visualize the range where the implementation of the loop closing processing is recommended in the support image is configured to

[0022] According to this, a range where loop closing processing is necessary is presented to the user in an easy-to-understand manner. In this case, for example, the range where loop closing processing is recommended may be displayed in color. Note that the range where loop closing processing is recommended may be visualized by an area or by a path. Also, the bird's-eye view image may be a bird's-eye view point cloud image, a bird's-eye view point cloud path image, or a bird's-eye view path image.

[0023] Also, in the fifth invention, the processor enlarges and arranges one of the bird's-eye view image generated from the measurement result of the three-dimensional measurement processing and the current captured image in the main image display frame, and reduces and arranges the other in the sub-image display frame, and according to a user's instruction operation, configures to swap the bird's-eye view image and the captured image arranged in the main image display frame and the sub-image display frame.

[0024] According to this, since the user can swap the image mainly to be browsed and the image to be browsed for reference according to the content to be confirmed, etc., convenience can be improved.

[0025] Also, in the sixth invention, the processor is configured to make the sub-image display frame non-displayable according to a user's instruction operation.

[0026] According to this, since the user can view only the main image display frame with the sub-image display frame non-displayed as needed, convenience can be improved.

[0027] Further, the seventh invention is a photographing work support method in which a processor performs a process of supporting a user's photographing work of sequentially photographing each point of a measurement target location with the photographing device while moving within the measurement target location while holding the photographing device in order to perform a three-dimensional measurement process for generating three-dimensional space information of the measurement target location based on the photographed images of each point of the measurement target location. As support information regarding the photographing work, Support image visualizing the range where loop closing processing has been performed in the three-dimensional measurement processing and support message prompting the shooting work necessary for performing the loop closing processing is configured to be displayed on a display unit.

[0028] According to this, similar to the first invention, by clearly presenting the necessity of loop closing processing to the user during the photographing work and prompting the user to perform an appropriate photographing work, the accuracy of three-dimensional measurement using SLAM can be improved.

[0029] Further, the eighth invention is a photographing work support program that causes a processor to execute a process of supporting a user's photographing work of sequentially photographing each point of a measurement target location with the photographing device while moving within the measurement target location while holding the photographing device in order to perform a three-dimensional measurement process for generating three-dimensional space information of the measurement target location based on the photographed images of each point of the measurement target location. As support information regarding the photographing work, Support image visualizing the range where loop closing processing has been performed in the three-dimensional measurement processing and support message prompting the shooting work necessary for performing the loop closing processing is configured to be displayed on a display unit.

[0030] According to this, similar to the first invention, by clearly presenting the necessity of loop closing processing to the user during the photographing work and prompting the user to perform an appropriate photographing work, the accuracy of three-dimensional measurement using SLAM can be improved.

[0031] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0032] FIG. 1 is an explanatory diagram showing a situation of a photographing work performed by a user using the photographing device 1 according to the present embodiment. FIG. 2 is a plan view showing a measurement target location.

[0033] The imaging device 1 includes a device main body 11 and a sensor unit 12. The sensor unit 12 includes a visible camera 21 (imaging unit). The visible camera 21 is a monocular camera that detects visible light and captures a subject, and outputs a captured image, for example, a color image in the RGB format. Note that the imaging device 1 can be configured as a tablet terminal or a notebook PC.

[0034] The user (operator) walks through the measurement target location while holding the device main body 11 of the imaging device 1. At this time, in the imaging device 1, the measurement target location is captured by the visible camera 21, and captured images of each point in the measurement target location are sequentially acquired.

[0035] Also, in this embodiment, the imaging device 1 serves as a three-dimensional measurement device. That is, in the imaging device 1, three-dimensional measurement processing is performed based on the captured images of each point sequentially acquired by the visible camera 21, and three-dimensional spatial information regarding the measurement target location is generated. In the three-dimensional measurement processing, point cloud data (environmental map) as three-dimensional spatial information regarding the measurement target location is generated using the SLAM method. At this time, together with the generation of the point cloud data, self-position estimation processing is performed, and the position of each self-position, that is, the imaging location, is estimated.

[0036] Also, in the imaging device 1, when it is detected that the movement path of the own device forms a loop, that is, the own device has returned to a point that was previously imaged, loop closing processing is performed. In the loop closing processing, regarding the point that was previously imaged, the self-position estimation result (past position) obtained at the time of that previous imaging is regarded as the correct position, and the positions of each imaging point in the path from the current position to the past position are corrected.

[0037] In the example shown in FIG. 2, imaging is started from the imaging start point, and after making a round of a part of the measurement target location, the imaging returns to the imaging start point. Further, imaging is continued, and another area of the measurement target location is imaged. In this case, loop closing processing is performed when returning to the imaging start point. As a result, the accuracy of the point cloud data and the self-position estimation result regarding each imaging point included in the loop is improved.

[0038] Next, an assistance image in which the completed area and the recommended implementation area are visualized on the bird's-eye view point cloud image will be described. FIG. 3 is an explanatory diagram showing an assistance image in which the completed area and the recommended implementation area are visualized on the bird's-eye view point cloud image.

[0039] In this embodiment, an assistance image in which the completed area and the recommended implementation area are visualized on the bird's-eye view point cloud image is generated, and the assistance image is presented to the user during the shooting operation. The bird's-eye view point cloud image is an image (rendering) of each point of the point cloud data as seen from an overhead viewpoint. The completed area represents the area where the loop closing process has been completed. The recommended implementation area represents the area where the implementation of the loop closing process is recommended, that is, the area where the loop closing process has not been implemented, or the area where the accuracy of the three-dimensional measurement result is low.

[0040] The completed area (completed range) and the recommended implementation area (recommended range) are drawn with color separation. For example, the completed area is drawn in blue, and the recommended implementation area is drawn in red. Specifically, in the bird's-eye view point cloud image, each point included in the completed area is drawn in blue, and each point included in the recommended implementation area is drawn in red.

[0041] At this time, each point drawn in a predetermined size in the bird's-eye view point cloud image is colored with a predetermined color. Also, a sphere with a predetermined diameter centered on each point may be drawn in a predetermined color. Also, the target space may be divided into cubic voxels, and the voxels including each point may be drawn in a predetermined color.

[0042] By viewing the assistance image, the user grasps the range where the shooting operation for the loop closing process is required and performs the shooting operation. Specifically, the user performs the shooting operation on the recommended implementation area drawn in red. That is, the shooting operation is performed so as to pass through the recommended implementation area. Note that the specific content of the shooting operation, that is, the movement path, etc., may be appropriately determined by the user according to the on-site situation.

[0043] Thus, in this embodiment, the support image can clearly present to the user the necessity of the loop closing process. Further, the range in which the shooting operation for the loop closing process is necessary can be clearly presented to the user. Thereby, the user can be prompted to perform an appropriate shooting operation, and since the loop closing process is surely carried out, the accuracy of the three-dimensional measurement process can be improved.

[0044] Note that both the completed area and the recommended implementation area represent the range where shooting has been performed within the measurement target location. That is, the area that is not included in either the completed area or the recommended implementation area in the measurement target location represents the range where shooting has not been performed. Therefore, the user can also confirm the range where shooting has not been performed by viewing the support image.

[0045] Also, in this embodiment, the number of times the loop closing process is carried out is counted, and visualization (color-coding) of the area corresponding to the number of times the loop closing process is carried out is performed. For example, the area where the number of times is 1 is drawn in green, the area where the number of times is 2 is drawn in blue, and the area where the number of times is 0 is drawn in red.

[0046] Next, an explanation will be given of the support image in which the completed range and the recommended implementation range are visualized on the overhead point cloud path image. FIG. 4 is an explanatory diagram showing the support image in which the completed range and the recommended implementation range are visualized on the overhead point cloud path image.

[0047] In this embodiment, a support image in which the completed range and the recommended implementation range are visualized on the overhead point cloud path image is generated, and the support image is presented to the user during the shooting operation. The overhead point cloud path image is one in which a path (movement trajectory) is superimposed on the overhead point cloud image. The overhead point cloud image is an image (rendering) of each point of the point cloud data viewed from an overhead viewpoint. The path is drawn by connecting the self-position estimation results based on each captured image, that is, the shooting locations of each time, with lines on the overhead point cloud image.

[0048] The completed path (completed range) and the recommended implementation path (recommended implementation range) are drawn with color separation. For example, the completed path is drawn in blue, and the recommended implementation path is drawn in red.

[0049] In addition, a shooting location mark is drawn on the support image. The shooting location mark represents the shooting location and the shooting direction. Specifically, the shooting location mark is drawn as an isosceles triangle, the vertex of the isosceles triangle represents the shooting location, and the direction looking from the vertex to the base represents the shooting direction.

[0050] By viewing the support image, the user grasps the range where shooting work is required for loop closing processing and performs the shooting work. Specifically, the user performs the shooting work targeting the recommended implementation path drawn in red. That is, the shooting work is performed so that a loop including the recommended implementation path is formed.

[0051] In addition, in this embodiment, the number of times of loop closing processing is counted, and the visualization (color separation) of the path according to the number of times of loop closing processing is performed. For example, the path with 1 execution is drawn in green, the path with 2 executions is drawn in blue, and the path with 0 executions is drawn in red.

[0052] Next, a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view path image will be described. FIG. 5 is an explanatory diagram showing a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view path image.

[0053] In this embodiment, a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view path image is generated, and the support image is presented to the user during the shooting work. The bird's-eye view path image is in a state where the bird's-eye view point cloud image is removed from the bird's-eye view point cloud path image (see FIG. 4), and the self-position estimation result based on each shooting image, that is, the line connecting each shooting location, is drawn in a state seen from an aerial viewpoint.

[0054] Next, an assistance image in which the completed range and the recommended implementation range are visualized on the captured image will be described. FIG. 6 is an explanatory diagram showing an assistance image in which the completed range and the recommended implementation range are visualized on the captured image.

[0055] In the present embodiment, an assistance image in which the completed range and the recommended implementation range are visualized on the captured image is generated, and the assistance image is presented to the user during the shooting operation. The captured image is the current captured image, that is, an image captured in real time output from the visible camera 21.

[0056] The completed area (completed range) and the recommended implementation area (recommended implementation range) are drawn with color separation. For example, the completed area is drawn in blue, and the recommended implementation area is drawn in red. Specifically, an image representing the completed area drawn in a predetermined color and an image representing the recommended implementation area drawn in a predetermined color are superimposed on the captured image in a semi-transparent state.

[0057] In this way, in the present embodiment, the completed range and the recommended implementation range (unimplemented range) are visualized on the captured image. As a result, the user can easily determine whether the area visible in front of the eyes is the completed range or the recommended implementation range. Then, the user performs the shooting operation so as to include the recommended implementation range.

[0058] Next, the schematic configuration of the imaging device 1 will be described. FIG. 7 is a block diagram showing the schematic configuration of the imaging device 1.

[0059] In addition to the sensor unit 12, the imaging device 1 includes a display 13 (display unit), an input device 14, a memory 15, and a processor 16 (CPU).

[0060] In addition to the visible camera 21, the sensor unit 12 includes a depth camera 22 and an IMU 23 (Inertial Measurement Unit). The depth camera 22 is a stereo camera that detects infrared light and captures an image of a subject, and outputs depth information (distance image) as a detection result. Based on the detection result of the depth camera 22, the distance to the subject can be measured. The IMU 23 detects three-dimensional angular velocity and acceleration. Based on the detection result of the IMU 23, the amount of movement and rotation of the imaging device 1 can be measured. Note that the visible camera 21, the depth camera 22, and the IMU 23 may not be integrated as a sensor unit. Also, a configuration in which the depth camera 22 and the IMU 23 are omitted and only the visible camera 21 is provided may be used.

[0061] The display 13 presents various types of information related to the imaging operation to the user, and displays an imaging screen 101 (see FIGS. 8 and 9), etc. The input device 14 is for the user to perform input operations. The input device 14 may be a keyboard, a mouse, a touch pad, a touch panel, or the like. When the imaging device 1 is configured as a tablet terminal, a touch panel display in which the touch panel as the input device 14 and the display panel as the display 13 are integrated is provided.

[0062] The memory 15 stores programs executed by the processor 16, etc. The memory 15 also stores the captured image of the visible camera 21, and the detection results of the depth camera 22 and the IMU 23. The memory 15 also stores the measurement results generated by the processor 16.

[0063] The processor 16 performs various processes by executing the programs stored in the memory 15. In the present embodiment, the processor 16 performs detection information acquisition processing, point cloud generation processing, first support image generation processing, second support image generation processing, screen control processing, message notification processing, and the like.

[0064] In the detection information acquisition process, the processor 16 acquires the captured image of the visible camera 21. Also, the processor 16 acquires the detection results of the depth camera 22 and the IMU 23.

[0065] In the point cloud generation process (3D measurement process), the processor 16 generates point cloud data (environmental map) as 3D spatial information regarding the measurement target location using the SLAM method based on the captured images of each point sequentially acquired by the visible camera 21. Also, in the point cloud generation process, self-position estimation is performed in conjunction with the generation of the point cloud data, and the self-position at each time, that is, the position of the shooting point, is acquired.

[0066] Also, the point cloud generation process includes a loop detection process. In the loop detection process, the processor 16 detects that the path of the own vehicle forms a loop, that is, the own vehicle has returned to a point that was previously photographed. At this time, in the detection of the previously photographed point, it is not compared with a preset specified captured image, but when the current captured image is similar to any of the past captured images, it is determined that the own vehicle has returned to a point that was previously photographed. Note that the similarity of the captured images may be determined based on the feature amounts extracted from the captured images.

[0067] Also, the point cloud generation process includes a loop closing process. When a loop is detected in the loop detection process, the loop closing process is executed. In the loop closing process, the processor 16 regards the self-position estimation result (past position) acquired at the time of the previous shooting regarding the previously photographed point as the correct position, and corrects the positions of each shooting point in the path from the current position to the past position.

[0068] In addition, the point cloud generation process includes an execution count acquisition process. In the execution count acquisition process, the processor 16 counts the number of times the loop closing process has been executed for each shooting location. The execution count information is added to the captured image (frame) of each shooting location. Also, the execution count information is added to each point in the point cloud data. At this time, for example, the execution count information is added to each point included in the point cloud data generated from the captured image (frame) of the shooting location where the loop closing process has been executed. Note that the execution count information also includes information regarding whether the loop closing process has been executed. When the execution count is 0 times, it means not executed, and when the execution count is 1 time or more, it means executed.

[0069] In addition, the point cloud generation process includes an accuracy acquisition process. In the accuracy acquisition process, the processor 16 acquires the accuracy of the three-dimensional measurement results (point cloud data and self-position estimation results) at each shooting location. Specifically, the accuracy of each shooting location is acquired based on the separation amount from the location where the loop closing process has been executed. Specifically, the separation amount is, for example, the moving distance (length of the movement trajectory) from the location where the loop closing process has been executed, the elapsed time from the time when the loop closing process has been executed, the number of captured images (number of frames) taken after the loop closing process has been executed, and so on. Note that the accuracy may be determined by combining the moving distance, elapsed time, and number of frames as the separation amount. Furthermore, the accuracy may be determined by combining the moving speed with the separation amount. The accuracy information is added to the captured image (frame) of each shooting location. Also, the accuracy information may be added to each point in the point cloud data.

[0070] Also, regardless of the execution status of the loop closing process, the accuracy information of each point in the point cloud data can be acquired. For example, when the measurement results (point cloud data) based on the captured images at each shooting location do not match the detection results of the depth camera 22 or the IMU 23, it can be determined that the accuracy is low. Also, the accuracy may be determined by comparing the measurement results based on the front and rear captured images (frames). In this case, the accuracy information is added to each point in the point cloud data.

[0071] In the point cloud generation process, in addition to the captured image of the visible camera 21, the detection results of the depth camera 22 and the IMU 23 may also be used. Specifically, based on the detection result of the depth camera 22, the relative position information (distance from the imaging device 1) of each feature point extracted from the captured image of the visible camera 21 with respect to the imaging device 1 can be obtained, and the self-position estimation result based on the captured image of the visible camera 21 can be corrected. Further, based on the detection result of the IMU 23, the relative position information of each imaging location can be obtained, and based on the relative position information of that imaging location, the self-position estimation result based on the captured image of the visible camera 21 can be corrected.

[0072] In the first support image generation process, the processor 16 generates a support image (first support image) in which the completed range and the recommended implementation range are visualized on the bird's-eye view image. Specifically, a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud image (see FIG. 3), a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view point cloud path image (see FIG. 4), and a support image in which the completed range and the recommended implementation range are visualized on the bird's-eye view path image (see FIG. 5) are generated.

[0073] In the visualization of the completed range and the recommended implementation range on the bird's-eye view point cloud image, based on the implementation count information added to each point in the point cloud data, the information on the completed range and the recommended implementation range is obtained, and each point on the point cloud image included in each of the completed range and the recommended implementation range is colored (point cloud coloring process).

[0074] Also, in the visualization of the completed range and the recommended implementation range on the bird's-eye view point cloud path image or the bird's-eye view path image, based on the implementation count information added to the captured image (frame) of each imaging location, the image of the path (movement trajectory) connecting the positions of each imaging location with a line is colored (path coloring process).

[0075] Further, in the present embodiment, in visualizing the recommended implementation range on the bird's-eye view point cloud image, a range with low accuracy of the three-dimensional measurement result is used as the recommended implementation range, and based on the accuracy information added to each point in the point cloud data, points with low accuracy are colored in the color of the recommended implementation range. Note that an area with many points of low accuracy in the point cloud data may be extracted as the recommended implementation range.

[0076] Further, in the present embodiment, in visualizing the recommended implementation range on the bird's-eye view point cloud path image or the bird's-eye view path image, a range with low accuracy among the paths connecting the respective photographing points is used as the recommended implementation range, and based on the accuracy information added to the photographed image (frame) of each photographing point, information representing the recommended implementation range is acquired, and the path image is colored.

[0077] In the second support image generation process, the processor 16 generates a support image (second support image) (see FIG. 6) in which the implemented range and the recommended implementation range are visualized on the photographed image. Specifically, an image representing the implemented area drawn in a predetermined color and an image representing the recommended implementation area drawn in a predetermined color are generated (superimposed image generation process), and the images representing the implemented area and the recommended implementation area are superimposed on the photographed image in a semi-transparent state. Note that in the superimposed image generation process, information representing the ranges of the implemented area and the recommended implementation area may be acquired based on the implementation count information of each point in the point cloud data viewed from the same viewpoint as the photographed image.

[0078] In the screen control process, the processor 16 controls the screen to be displayed on the display 13. In the present embodiment, a photographing screen 101 (see FIGS. 8 and 9) is generated and displayed on the display 13. On the photographing screen 101, the first support image (see FIGS. 3, 4, and 5) generated in the first support image generation process and the second support image (see FIG. 6) generated in the second support image generation process are displayed.

[0079] In the message notification process, the processor 16 performs a process of notifying the user of messages regarding guidance for the shooting operation and various warnings. In this embodiment, a message prompting the user to perform a shooting operation for loop closing processing is displayed on the shooting screen 101 (see FIGS. 8 and 9).

[0080] At this time, when a predetermined notification condition is satisfied, a message is displayed. For example, when the amount of movement from the point where the loop closing process is performed is equal to or greater than a predetermined threshold, a message is displayed. Also, when the elapsed time since the loop closing process is performed is equal to or greater than a predetermined threshold, a message is displayed. Here, the amount of movement and the elapsed time are indicators for evaluating the deterioration of the accuracy of the three-dimensional measurement result. When the amount of movement and the elapsed time are each equal to or greater than their respective thresholds, it is determined that the accuracy deterioration has exceeded the allowable limit, and a message is displayed for accuracy improvement. Note that the message may be always displayed.

[0081] Note that in this embodiment, the imaging device 1 performs three-dimensional measurement processing (point cloud generation processing), but the three-dimensional measurement processing may be performed by a server device (not shown) that can communicate with the imaging device 1.

[0082] Next, the shooting screen 101 displayed on the display 13 will be described. FIGS. 8 and 9 are explanatory diagrams showing the shooting screen 101.

[0083] The shooting screen 101 is provided with a main window 102 (main image display frame) and a sub-window 103 (sub-image display frame). The display magnifications are different between the main window 102 and the sub-window 103. In the main window 102, the image is displayed enlarged, and in the sub-window 103, the image is displayed reduced.

[0084] In addition, the first assistance image 121 and the second assistance image 122 are each displayed on the main window 102 and the sub-window 103. In the shooting screen 101 shown in FIG. 8, the second assistance image 122 is enlarged and displayed on the main window 102, and the first assistance image 121 is reduced and displayed on the sub-window 103. In the shooting screen 101 shown in FIG. 9, the first assistance image 121 is enlarged and displayed on the main window 102, and the second assistance image 122 is reduced and displayed on the sub-window 103.

[0085] The first assistance image 121 includes an assistance image in which the completed range and the recommended implementation range are visualized on the overhead point cloud image (see FIG. 3), an assistance image in which the completed range and the recommended implementation range are visualized on the overhead point cloud path image (see FIG. 4), and an assistance image in which the completed range and the recommended implementation range are visualized on the overhead path image (see FIG. 5), and any one of the images is displayed according to the setting. In the examples shown in FIGS. 8 and 9, as the first assistance image 121, an assistance image in which the completed range and the recommended implementation range are visualized on the overhead point cloud image (see FIG. 3) is displayed.

[0086] The second assistance image 122 is an assistance image in which the completed range and the recommended implementation range are visualized on the captured image (see FIG. 6). The captured image is an image captured in real time output from the visible camera 21.

[0087] Note that either the first assistance image 121 or the second assistance image 122 displayed on the main window 102 and the sub-window 103 may be displayed as it is, such as an overhead image (overhead point cloud image, overhead point cloud path image, overhead path image) or a captured image, without visualizing the completed range and the recommended implementation range. Whether or not to visualize the completed range and the recommended implementation range in the main window 102 and the sub-window 103 may be set by the user in advance.

[0088] In addition, on the shooting screen 101, a "Shooting Start" button 105 and a "Recording Confirmation" button 106 are provided. When the user operates the "Shooting Start" button 105, shooting by the visible camera 21 starts, and the captured images are stored in the memory 15. When the user operates the "Recording Confirmation" button 106, the mode changes to playing the captured images stored in the memory 15. Thus, the user can confirm whether the shooting was properly performed.

[0089] In addition, on the shooting screen 101, a "Screen Switching" button 107 and a checkbox 108 related to the display of the sub-window 103 are provided. When the user operates the "Screen Switching" button 107, the state of the shooting screen 101 shown in FIG. 8 and the state of the shooting screen 101 shown in FIG. 9 can be switched. That is, the state where the captured image is displayed in the main window 102 and the bird's-eye view map is displayed in the sub-window 103, and the state where the bird's-eye view map is displayed in the main window 102 and the captured image is displayed in the sub-window 103 can be switched. When the user checks the checkbox 108, the sub-window 103 transitions to a non-display state.

[0090] In addition, a message window 110 is provided on the shooting screen 101. Messages to notify the user are displayed in the message window 110. Specifically, messages regarding guidance on the shooting operation and various warnings are displayed.

[0091] In particular, during the shooting operation, if there is a place where loop closing processing is recommended, that is, a place where loop closing processing has not been performed, or a place where the accuracy of the three-dimensional measurement result is low, a support message is displayed to prompt the user to perform the shooting operation on the place where loop closing processing is recommended, that is, the recommended implementation range drawn in red. In the examples shown in FIGS. 8 and 9, a message "Please take a shot so that loop closing is performed in the red area." is displayed.

[0092] In this way, in the present embodiment, a real-time captured image is displayed on the shooting screen 101. As a result, the user can check the current shooting situation during the shooting operation. Further, since the completed range and the recommended implementation range (uncompleted range) are visualized on the captured image, the user can easily determine whether the visible place in front of him / her is the completed range or the recommended implementation range.

[0093] By the way, in the present embodiment, there are a method of visualizing (color-coding) an area where loop closing processing has not been performed as a recommended implementation range and a method of visualizing an area with low accuracy of 3D measurement results as a recommended implementation range. These two methods of visualizing the recommended implementation range can be switched as appropriate. For example, the method of visualizing the recommended implementation range may be preset in a setting screen (not shown). Also, the user may be able to specify the method of visualizing the recommended implementation range on the shooting screen 101.

[0094] As described above, embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments with changes, replacements, additions, omissions, etc. Also, it is possible to combine the respective components described in the above embodiments to form a new embodiment.

Industrial Applicability

[0095] The imaging device, imaging work support method, and imaging work support program according to the present invention have the effect of improving the accuracy of 3D measurement using SLAM by clearly presenting the necessity of loop closing processing to the user during the imaging work and prompting the user to perform appropriate imaging work. For performing 3D measurement processing for generating 3D spatial information of the measurement target location based on the captured images of each point of the measurement target location, it is useful as an imaging device for capturing each point of the measurement target location, and an imaging work support method and an imaging work support program for supporting the user's imaging work using the imaging device.

Explanation of Signs

[0096] 1: Imaging device 11: Device body 12: Sensor unit 13: Display (display unit) 14: Input device 15: Memory 16: Processor 21: Visible camera (imaging unit) 101: Imaging screen 102: Main window (main image display frame) 103: Sub window (sub image display frame) 110: Message window 121: First support image 122: Second support image

Claims

1. An imaging device for taking images of each point in a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on the captured images of each point in the measurement target location, a device body held by a user, an imaging unit provided on the device body for imaging the measurement target location, a display unit for displaying support information regarding an imaging operation for sequentially imaging each point in the measurement target location by having the user hold the device body and move while imaging the measurement target location, a processor for controlling the imaging unit and the display unit, and comprising: The processor as the support information, a support image visualizing a range where loop closing processing has been performed in the three-dimensional measurement process and a support message prompting an imaging operation necessary for performing the loop closing processing, is displayed on the display unit. The imaging device is characterized by this.

2. The processor displays, on the display unit, the support message for prompting the user to perform an imaging operation targeting a range where implementation of the loop closing processing is recommended. The imaging device according to claim 1 is characterized by this.

3. The processor displays the support image on the display unit in at least one of an aerial view image generated from the measurement result of the three-dimensional measurement process and the current captured image. The imaging device according to claim 1 is characterized by this.

4. The processor visualizes, in the support image, a range where implementation of the loop closing processing is recommended. The imaging device according to claim 3 is characterized by this.

5. The processor enlarges and arranges one of an aerial view image generated from the measurement result of the three-dimensional measurement process and the current captured image in a main image display frame, and reduces and arranges the other in a sub-image display frame, and switches the aerial view image and the captured image arranged in the main image display frame and the sub-image display frame according to a user's instruction operation. The imaging device according to claim 1 is characterized by this.

6. The processor hides the sub-image display frame according to a user's instruction operation. The imaging device according to claim 5 is characterized by this.

7. In order to perform a three-dimensional measurement process for generating three-dimensional spatial information of a measurement target location based on captured images of each point in the measurement target location, a shooting work support method in which a processor performs a process of assisting a user's shooting work of sequentially shooting each point in the measurement target location with the shooting device while holding the shooting device and moving within the measurement target location, As support information regarding the shooting work, a support image visualizing a range where loop closing processing has been performed in the three-dimensional measurement process and a support message prompting the shooting work necessary for performing the loop closing processing are displayed on a display unit, characterized by the shooting work support method.

8. A shooting work support program that causes a processor to execute a process of assisting a user's shooting work of sequentially shooting each point in a measurement target location with a shooting device while holding the shooting device and moving within the measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on captured images of each point in the measurement target location, As support information regarding the shooting work, a support image visualizing a range where loop closing processing has been performed in the three-dimensional measurement process and a support message prompting the shooting work necessary for performing the loop closing processing are displayed on a display unit, characterized by the shooting work support program.

Images