Imaging device, imaging operation support method, and imaging operation support program

Abstract

To enable a user to check, in advance, locations where various phenomena causing deterioration of accuracy of point cloud data will occur when actual imaging is performed, whereby highly accurate measurement results can be obtained by single actual imaging operation.SOLUTION: In an imaging device for imaging a measurement target location in order to perform 3D measurement processing for generating 3D spatial information of the measurement target location on the basis of imaging data of the measurement target location, display work support information related to imaging work for imaging a measurement target location where a user performs imaging while moving is displayed on a display, and in particular, a careful imaging place in the actual imaging is detected on the basis of the imaging data obtained by preliminary imaging, a confirmation map 31 in which the careful imaging place is drawn on a map image 32 representing the measurement target location is generated, and a screen including the confirmation map is displayed on the display.SELECTED DRAWING: Figure 30

Description

[Technical field]

[0001] The present invention relates to an imaging device that photographs a measurement target location in order to perform 3D measurement processing that generates 3D spatial information of the measurement target location based on imaging data of the measurement target location, and to a photography work support method and photography work support program that support a user's photography work using the imaging device. [Background technology]

[0002] There is known a 3D measurement technology that generates 3D spatial information (map data) about a measurement target location based on image data obtained by photographing the measurement target location. In recent years, the SLAM (Simultaneous Localization And Mapping) method has been attracting attention for this type of 3D measurement. In the SLAM method, a mobile object is held by a camera device and photographs the measurement target location, and based on the photograph data obtained, 3D spatial information and the position information of the mobile object can be generated as a measurement result. In particular, when a user uses a handheld camera device to perform a photographing task of photographing the measurement target location while moving within the measurement target location, thereby obtaining photograph data of the measurement target location, 3D measurement can be easily performed.

[0003] On the other hand, when a user photographs a measurement target location with a photographing device held in his / her hand while walking through the measurement target location, it is desirable to provide work support that prompts the user to perform appropriate photographing work during the photographing work so that highly accurate and stable 3D measurement can be performed.From this perspective, a technology has been known in the past that displays multiple marks such as arrows that indicate the moving direction and moving speed of the photographing device (sensor) held by the user to guide the user on how to proceed with the photographing work (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6489566 Summary of the Invention [Problem to be solved by the invention]

[0005] In photography work where a user holds a photography device and photographs a measurement target location while moving around the measurement target location, it may be difficult to obtain highly accurate measurement results with only one photograph, depending on the user's proficiency and the characteristics of the measurement target location, etc. In other words, various events that cause the accuracy of point cloud data to decrease may occur during photography, and a high level of skill is required to proceed with the photography work while avoiding the occurrence of such events.

[0006] For this reason, a technology is desired that can present to a user in advance locations where various events that may cause a decrease in the accuracy of point cloud data occur as locations that require caution when photographing the data. However, the conventional technology does not take such a request into consideration at all.

[0007] Therefore, the main object of the present invention is to provide an imaging device, a method for supporting imaging work, and a program for supporting imaging work that enable a user to check in advance the locations where various phenomena that can reduce the accuracy of point cloud data will occur when performing actual imaging, thereby enabling highly accurate measurement results to be obtained with a single actual imaging session. [Means for solving the problem]

[0008] The photographing device of the present invention is an imaging device that photographs a location to be measured in order to perform 3D measurement processing to generate 3D spatial information of the location to be measured based on photographing data of the location to be measured, and is equipped with a device main body held by a user, a photographing unit provided in the device main body for photographing the location to be measured, a display unit that displays work support information related to the photographing work in which the user moves around the location to be measured and has the photographing unit photograph the location to be measured, and a processor that controls the photographing unit and the display unit, and the processor is configured to detect areas that need to be photographed in the actual photographing based on the photographing data obtained in preliminary photographing, generate a confirmation map in which the areas to be photographed are drawn on a map image representing the location to be measured, and display a screen including the confirmation map on the display unit.

[0009] In addition, the photography support method of the present invention is a photography support method in which a processor performs processing to support the photography work of a user who holds a photography device and moves around the measurement target location while having the photography device photograph the measurement target location, in order to perform 3D measurement processing that generates 3D spatial information of the measurement target location based on photography data of the measurement target location.The method is configured to detect areas that need to be photographed in the actual photography based on the photography data acquired in preliminary photography, generate a confirmation map in which the areas that need to be photographed are drawn on a map image representing the measurement target location, and display a screen including the confirmation map on a display unit.

[0010] In addition, the photography work support program of the present invention is a photography work support program that causes a processor to execute a process to support the photography work of a user who holds a photography device and moves around the measurement target location while having the photography device photograph the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photography data of the measurement target location.The photography work support program is configured to detect areas that need to be photographed in the actual photography based on the photography data acquired in preliminary photography, generate a confirmation map in which the areas need to be photographed are drawn on a map image representing the measurement target location, and display a screen including the confirmation map on a display unit. Effect of the Invention

[0011] According to the present invention, when taking actual photographs, the user can check in advance the areas that require careful photography, i.e., the areas where various phenomena that may reduce the accuracy of the point cloud data may occur, thereby making it possible to obtain highly accurate measurement results with a single actual photographing run. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram showing a situation of a photographing operation using the photographing device according to the present embodiment; [Diagram 2] An explanatory diagram showing an example of a confirmation map including points to be careful of when taking photographs. [Diagram 3] FIG. 1 is an explanatory diagram showing an example of a confirmation map including a recommended route. [Figure 4] An explanatory diagram showing the shooting situation of a strong reflection area by a shooting device. [Diagram 5] FIG. 1 is a block diagram showing a schematic configuration of an imaging device. [Figure 6] Block diagram showing an overview of the processing performed by the processor [Figure 7] FIG. 1 is an explanatory diagram showing an overlapping region used in the point cloud density evaluation process and the feature point evaluation process. [Figure 8] A flow diagram showing the steps of processing performed by the processor during shooting. [Figure 9] FIG. 1 is a flow diagram showing the procedure of processing performed by a processor during preliminary photography. [Figure 10] A flow diagram showing the steps of processing performed by the processor during actual shooting. [Figure 11] FIG. 1 is a flow diagram showing the procedure of processing performed when the processor finishes preliminary photography. [Figure 12] FIG. 13 is an explanatory diagram showing a screen displayed before shooting in a preliminary shooting mode. [Figure 13] FIG. 13 is an explanatory diagram showing a screen displayed during shooting in the preliminary shooting mode. [Figure 14] FIG. 13 is an explanatory diagram showing a screen displayed during shooting in the preliminary shooting mode. [Figure 15] FIG. 13 is an explanatory diagram showing a screen displayed immediately after shooting in the preliminary shooting mode. [Figure 16]FIG. 13 is an explanatory diagram showing a screen displayed when confirming after shooting in the preliminary shooting mode. [Figure 17] FIG. 13 is an explanatory diagram showing a screen displayed when confirming after shooting in the preliminary shooting mode. [Figure 18] FIG. 13 is an explanatory diagram showing a screen displayed when confirming after shooting in the preliminary shooting mode. [Figure 19] FIG. 13 is an explanatory diagram showing a screen displayed when confirming after shooting in the preliminary shooting mode. [Figure 20] FIG. 13 is an explanatory diagram showing a screen related to progress setting displayed in the setting change mode; [Figure 21] FIG. 13 is an explanatory diagram showing a screen for setting the display of points to be noted when taking pictures, which is displayed in the setting change mode; [Figure 22] FIG. 13 is an explanatory diagram showing a screen related to recommended route settings displayed in the setting change mode; [Figure 23] FIG. 13 is an explanatory diagram showing a screen related to recommended route settings displayed in the setting change mode; [Figure 24] FIG. 13 is an explanatory diagram showing a screen displayed in the main shooting mode; [Diagram 25] FIG. 13 is an explanatory diagram showing a screen displayed in the main shooting mode; [Figure 26] FIG. 13 is an explanatory diagram showing a screen displayed in the main shooting mode; [Figure 27] FIG. 13 is an explanatory diagram showing a screen displayed in the main shooting mode; [Figure 28] FIG. 13 is an explanatory diagram showing a screen displayed in the main shooting mode; [Figure 29] FIG. 13 is an explanatory diagram showing a screen displayed in the main shooting mode; [Diagram 30] FIG. 13 is an explanatory diagram showing a screen displayed in the main shooting mode; [Diagram 31] FIG. 13 is an explanatory diagram showing a screen displayed in the main shooting mode; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The first invention made to solve the above problem is an imaging device that photographs a location to be measured in order to perform 3D measurement processing to generate 3D spatial information of the location to be measured based on imaging data of the location to be measured, the imaging device comprising: a device main body held by a user; an imaging unit provided in the device main body for photographing the location to be measured; a display unit that displays work support information related to the imaging work in which the user moves around the location to be measured and has the imaging unit photograph the location to be measured; and a processor that controls the imaging unit and the display unit, the processor detecting areas that need to be photographed in the actual imaging based on the imaging data acquired in preliminary imaging, generating a confirmation map in which the areas that need to be photographed are drawn on a map image representing the location to be measured, and displaying a screen including the confirmation map on the display unit.

[0014] This allows the user to check in advance the areas that require special attention when taking the actual photographs, i.e., the areas where various phenomena that can reduce the accuracy of the point cloud data can occur, thereby enabling highly accurate measurement results to be obtained with a single actual photograph.

[0015] In addition, in the second invention, the processor generates point cloud data as a preliminary shooting result based on the shooting data acquired in the preliminary shooting, generates an overhead point cloud image based on the point cloud data, and generates the confirmation map by superimposing an image representing the points to be careful about in shooting on the map image including the overhead point cloud image.

[0016] This makes it possible to easily generate an appropriate map image showing the measurement target location and present a confirmation map that is easy for the user to understand.

[0017] In addition, in a third invention, the processor is configured to switch between a first shooting mode in which preliminary shooting is performed and a second shooting mode in which main shooting is performed in response to a user operation to select a shooting mode.

[0018] This allows the user to appropriately select and execute preliminary photography and main photography. For example, the user can appropriately omit preliminary photography and execute main photography.

[0019] In addition, a fourth invention is configured such that the processor causes the display unit to display a screen including a real-time captured image output from the photographing unit during actual photographing and the confirmation map on which the areas to be careful about in photographing are depicted.

[0020] This allows the user to proceed with the photographing operation while checking points along the route that require attention for photographing, together with the photographing status of the photographing unit.

[0021] In addition, a fifth aspect of the present invention is configured such that the processor causes the display unit to display a screen including the confirmation map on which the areas to be careful of in photographing are depicted before the main photographing operation following the preliminary photographing operation.

[0022] This allows the user to check points that need to be photographed that exist in the measurement target location after the preliminary photographing is completed and before the main photographing is started.

[0023] In addition, a sixth aspect of the present invention is configured such that the processor generates simplified point cloud data as a preliminary photography result by processing that is simplified compared to that used in actual photography, and detects the areas to be careful about when photographing based on the simplified point cloud data.

[0024] This makes it possible to reduce the waiting time of the user when the point cloud data as the result of the preliminary photography is presented to the user after the preliminary photography is completed.

[0025] In addition, in a seventh aspect of the present invention, the processor is configured to generate the confirmation map in which at least one of strong reflection areas, areas lacking features, areas of sudden movement, and areas lacking point clouds is depicted as the areas to be careful of when photographing.

[0026] This makes it possible to present to the user appropriate points to be careful about when taking the actual photograph.

[0027] In addition, the eighth invention is a photography support method in which a processor performs processing to support a user's photography work by having the photography device photograph the location to be measured while moving within the location to be measured while holding the photography device, in order to perform 3D measurement processing to generate 3D spatial information of the location to be measured based on photography data of the location to be measured. The method is configured to detect areas that need to be photographed in the actual photography based on the photography data acquired in preliminary photography, generate a confirmation map in which the areas to be photographed are drawn on a map image representing the location to be measured, and display a screen including the confirmation map on a display unit.

[0028] Like the first invention, this allows the user to check in advance the areas that require careful photography when taking the actual photograph, i.e., the areas where various phenomena that may reduce the accuracy of the point cloud data may occur, and enables highly accurate measurement results to be obtained with a single actual photograph.

[0029] In addition, a ninth invention is a photography operation support program that causes a processor to execute a process to support a user's photography operation by having the photography device photograph the location to be measured while holding the photography device and moving around the location to be measured, in order to perform 3D measurement processing to generate 3D spatial information of the location to be measured based on photography data of the location to be measured. The program is configured to detect areas that need to be photographed in the actual photography based on the photography data acquired in preliminary photography, generate a confirmation map in which the areas to be photographed are drawn on a map image representing the location to be measured, and display a screen including the confirmation map on a display unit.

[0030] Like the first invention, this allows the user to check in advance the areas that require careful photography when taking the actual photograph, i.e., the areas where various phenomena that may reduce the accuracy of the point cloud data may occur, and enables highly accurate measurement results to be obtained with a single actual photograph.

[0031] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0032] FIG. 1 is an explanatory diagram showing the situation of a photographing operation performed by a user using a photographing device 1 according to this embodiment.

[0033] The photographing device 1 includes a device body 11 and a sensor unit 12. The sensor unit 12 includes a visible camera 21 (photographing unit). The visible camera 21 is a monocular camera that detects visible light to photograph a subject, and outputs a photographed image, for example, a color image in an RGB format. The photographing device 1 can be configured as a tablet terminal or a notebook PC.

[0034] A user (worker) walks through a location to be measured while holding the device body 11 of the photographing device 1. At this time, the location to be measured is photographed by the visible camera 21 of the photographing device 1, and photographing data of each point in the location to be measured is sequentially acquired. The photographing data includes data such as the photographed image, the sensor detection result, and the photographing time.

[0035] In this embodiment, the photographing device 1 is a three-dimensional measuring device. That is, the photographing device 1 performs three-dimensional measurement processing based on photographed images of each point sequentially acquired by the visible camera 21, and generates three-dimensional spatial information on the measurement target location. In the three-dimensional measurement processing, point cloud data (environmental map) is generated as three-dimensional spatial information on the measurement target location using the SLAM method.

[0036] 1 shows an example in which a user holding the device main body 11 walks through the measurement target location to photograph the measurement target location, but the measurement target location may also be photographed using a self-propelled robot equipped with the photographing device 1. In this embodiment, the user is assisted in the work, and when the user himself holds the device main body 11 and performs the photographing work, the work assistance is provided to the user. On the other hand, when a self-propelled robot photographs, the work assistance is provided to the user operating the robot.

[0037] In addition, in this embodiment, an imaging device 1 equipped with a sensor unit 12 such as a visible camera 21 performs 3D measurement processing, but the sensor unit 12 may be configured as an imaging device 1 independent of a PC (information processing device) that performs 3D measurement processing.

[0038] In addition, the user photographs the measurement target location with the imaging device 1 held in the hand while walking through the measurement target location. Therefore, the imaging work includes the action of walking (moving) while holding the imaging device 1, the action of pointing the imaging device 1 in an appropriate direction, etc.

[0039] Next, the points to be noted for photography and the confirmation map 31 will be described. Fig. 2 is an explanatory diagram showing an example of the confirmation map 31 including the points to be noted for photography. Fig. 3 is an explanatory diagram showing an example of the confirmation map 31 including the recommended route. Fig. 4 is an explanatory diagram showing the photography situation by the photography device 1 in a strongly reflective point.

[0040] In a photographing operation in which a user photographs a measurement target location while holding the photographing device 1 and moving around the measurement target location, it may be difficult to obtain highly accurate measurement results with only one photographing operation, depending on the user's proficiency and the characteristics of the measurement target location. In other words, various events that cause the accuracy of the point cloud data to decrease may occur during photographing, and a high level of skill is required to proceed with the photographing operation while avoiding the occurrence of such events. For this reason, an inexperienced user will have to repeat the photographing operation many times while using trial and error.

[0041] Therefore, in this embodiment, a preliminary photograph is taken in addition to the main photographing (regular photographing) for acquiring three-dimensional spatial information (point cloud data) as a measurement result, and based on the result of the preliminary photographing, points to be careful of in the main photographing, i.e., points to be careful of in the main photographing, are detected and presented to the user. This makes it possible to improve the accuracy of the main photographing result, i.e., the point cloud data acquired in the main photographing.

[0042] The areas to be careful of when taking pictures include areas with strong reflections, areas lacking features, areas with sudden movements, and areas lacking point clouds. The areas to be careful of when taking pictures include areas where an event that may cause the accuracy of the point cloud data to decrease actually occurred in the preliminary photography, and areas where an event that may cause the accuracy of the point cloud data to decrease is likely to occur.

[0043] Here, for example, assume that lighting is installed on the ceiling and the floor is made of a reflective material (see FIG. 4). In this case, the lighting light is reflected by the floor, and the reflected light enters the image capture device 1, causing the reflected light from the floor to appear in the captured image. At this time, the reflected light from the floor is extracted as a feature point, and position estimation is performed based on this feature point. However, the position of the reflected light from the floor also moves as the image capture device 1 moves. As a result, position estimation based on the feature points cannot be performed appropriately, and the accuracy of the point cloud data as the measurement result is significantly reduced.

[0044] Therefore, in this embodiment, a location where strong reflected light is estimated to have appeared in the image captured by the visible camera 21 or a location where strong reflected light has actually appeared in the captured image in the preliminary capture is detected as a strong reflection location and presented to the user. Furthermore, guidance (such as a suggestion of an improvement measure) regarding the strong reflection location is provided to the user. Specifically, when it is estimated that the reflected light is caused by the lighting in the room, the user is prompted to improve the shooting environment regarding the lighting. For example, the user is prompted to dim the lighting. Also, when it is estimated that the reflected light is caused by the external light entering through the window, the user is prompted to improve the shooting environment regarding the external light. For example, the user is prompted to close the curtains. Note that when there is a possibility that the reflected light is caused by both the lighting in the room and the external light entering through the window, the user may be prompted to dim the lighting or close the curtains. This allows the user to appropriately improve the shooting environment regarding the strong reflection location.

[0045] Furthermore, when an image is captured of an area that includes many areas with few distinctive patterns or objects, such as walls, floors, and ceilings, feature point extraction will fail and the number of extracted feature points will be reduced, preventing proper self-location estimation and reducing the accuracy of the point cloud data.

[0046] Therefore, in this embodiment, a portion where feature points could not be sufficiently extracted from the image captured by the visible camera 21 is detected as a feature-deficient portion and presented to the user. Furthermore, guidance regarding the feature-deficient portion (such as suggestions for improvement) is provided to the user. For example, if the feature-deficient portion is caused by the shooting environment of the measurement target location, the user is prompted to improve the shooting environment. For example, the user is prompted to place something with texture (e.g., a distinctive wallpaper). Furthermore, if the feature-deficient portion is caused by the user's shooting method, the user is prompted to improve the shooting method. For example, the user is prompted to shoot with an angle of view that includes a distinctive pattern or object.

[0047] Furthermore, if the user moves the image capturing device 1 too quickly while capturing an image, errors will occur in the position estimation and the point cloud will not be generated appropriately, resulting in reduced accuracy of the point cloud data.

[0048] Therefore, in this embodiment, a location where the user actually performed a sudden movement during preliminary shooting or a location where it is estimated that the user is likely to perform a sudden movement is detected as a sudden movement location and presented to the user. For example, a location where a sudden acceleration / deceleration state in which the acceleration generated in the photographing device 1 falls outside a predetermined range or a sudden rotation state in which the angular velocity generated in the photographing device 1 falls outside a predetermined range occurs is detected as a sudden movement location and presented to the user. In addition, a location where the space through which the user passes is narrow or a location where the route the user takes turns suddenly is detected as a sudden movement location and presented to the user because it is likely that the user will perform a sudden movement. Furthermore, guidance (such as a suggestion of a remedy) regarding the sudden movement location is provided to the user. For example, the user is urged not to perform a sudden movement.

[0049] In addition, places hidden by the shadow of objects, such as the back of a monitor on a desk, will not be photographed, and in places where photography is insufficient, the point cloud will not be generated with sufficient density, resulting in a decrease in the accuracy of the point cloud data. Also, if the user does not move the photographing device 1 appropriately while photographing, the point cloud will not be generated with sufficient density, resulting in a decrease in the accuracy of the point cloud data.

[0050] Therefore, in this embodiment, areas where a point cloud was not generated because they were not photographed in the preliminary shooting (missed areas), and areas where they were photographed but a point cloud was not generated with sufficient density are detected as areas with insufficient point clouds and presented to the user.

[0051] 2 and 3, in this embodiment, a confirmation map 31 is generated for the user to confirm points to be noted before or during actual shooting and is presented to the user. In the confirmation map 31, the preliminary shooting result, i.e., an overhead point cloud image 35 generated from point cloud data from the preliminary shooting, is used as a map image 32 representing the measurement target location, and an image representing points to be noted is superimposed on the map image 32. The overhead point cloud image 35 is an image (rendering) of each point of the point cloud data as viewed from an aerial viewpoint.

[0052] In the example shown in FIG. 2(A), strong reflection areas, feature-lacking areas, sudden movement areas, and point cloud-lacking areas are detected as areas to be careful of when photographing, and mark images 41, 42, and 43 representing the strong reflection areas, feature-lacking areas, and sudden movement areas, and an area image 44 representing the point cloud-lacking areas are drawn on the confirmation map 31. The mark images 41, 42, and 43 are drawn in different shapes depending on the type (strong reflection areas, feature-lacking areas, and sudden movement areas), but may be drawn in the same shape by changing the drawing manner (e.g., color) other than the shape. The area image 44 is drawn in a predetermined color (e.g., red). The area image 44 may be drawn in a pattern (design) or a semi-transparent color. Note that the mark images 41, 42, and 43 and the area image 44 are not limited to mark images and area images, respectively, and for example, the mark images 41, 42, and 43 may be drawn as area images indicating areas to be careful of when photographing in their respective predetermined colors different from the area image 44, or may be drawn in a pattern (design) or a semi-transparent color.

[0053] In this example, areas that are not captured because they are hidden by the shadow of an object, such as the back of a monitor on a desk, are detected as areas that are lacking in point cloud data. In addition, the area on the periphery of the measurement target location is detected as an area that is lacking in point cloud data. Although this area appears in the captured image, it is detected as an area that is lacking in point cloud data because the point cloud density is low.

[0054] In the example shown in Fig. 2(B), a point cloud insufficient area is detected as an area requiring careful photography, and an area image 44 representing the point cloud insufficient area is drawn on the confirmation map 31. In this example, similar to the example shown in Fig. 2(A), an area on the outer periphery of the measurement target location is detected as a point cloud insufficient area.

[0055] Here, as in the example shown in Fig. 2(A), in many cases the user does not need to pay special attention to the point cloud insufficient areas on the outer periphery of the measurement target location when taking the actual photograph. On the other hand, as in the example shown in Fig. 2(B), when the point cloud insufficient areas extend from the outer periphery of the measurement target location toward the inside, the user needs to pay special attention to the area when taking the actual photograph.

[0056] In this embodiment, strong reflection areas, areas lacking features, areas with sudden movements, and areas lacking point clouds are detected as areas to be careful of when photographing and presented to the user, but in addition, blurring areas, i.e. areas estimated to be highly likely to cause blurring (out-of-focus) in the image photographed by the visible camera 21, and areas where blurring actually occurred in the photographed image in preliminary photographing may be detected and presented to the user. If blurring occurs in the photographed image, point cloud generation is not performed appropriately, but by presenting the areas where blurring occurs to the user before actual photographing, it is possible to suppress blurring in the photographed image.

[0057] In this case, an evaluation value (blur evaluation value) that indicates the occurrence of blur in the captured image is obtained, and the evaluation value is compared with a predetermined threshold value to determine the location where blur has occurred. The blur evaluation value is obtained by image analysis of the captured image. The blur evaluation value is also obtained based on the exposure time of the visible camera 21 and the detection data (acceleration, angular velocity) of the IMU 23. The blur evaluation value is also obtained based on the shooting environment such as brightness.

[0058] In this embodiment, a target shooting range to be the target of main shooting is set as shown in Fig. 3. The target shooting range is set based on the preliminary shooting result, i.e., the point cloud data acquired in the preliminary shooting. For example, the range in which the point cloud is generated in the preliminary shooting is set as the target shooting range. At this time, if there is an unphotographed area (area in which a point cloud is not generated) surrounded by a photographed area (area in which a point cloud is generated), the unphotographed area may be included in the target shooting range as an unphotographed area.

[0059] In addition, in this embodiment, a recommended route is presented to encourage the user to take during actual photography. The recommended route is generated based on the target photography range. Specifically, a route that covers the entire target photography range and has the shortest possible travel distance is generated as the recommended route.

[0060] In the confirmation map 31, a route image 51 representing a recommended route is superimposed on a map image 32 consisting of an overhead point cloud image 35. A line representing the recommended route, an arrow representing the direction of travel, a mark representing the start position, a mark representing the current position, and a mark representing the end position are drawn on the route image 51 representing the recommended route.

[0061] Here, when the areas to be careful about when photographing are presented to the user, the user can be prompted to improve the photographing environment and photographing method related to the areas to be careful about when photographing, so that the user can take appropriate measures to improve the photographing environment and photographing method, thereby improving the accuracy of the measurement results (point cloud). However, there are cases where it is difficult to improve the photographing environment and photographing method related to the areas to be careful about when photographing.

[0062] In this case, in this embodiment, a recommended route is generated in response to a user's instruction to avoid the points that require shooting. This makes it impossible to obtain a point cloud generated by shooting at the points that require shooting, but it is possible to avoid a decrease in the accuracy of the measurement results (point cloud) caused by passing through the points that require shooting without improving the shooting environment or the shooting method.

[0063] The example shown in Fig. 3(A) is a case where the points to be careful of are not avoided, and a route that covers the entire target shooting range and has the shortest possible travel distance is generated as the recommended route, regardless of the points that lack features. On the other hand, the example shown in Fig. 3(B) is a case where strong reflection points are avoided, and a recommended route is generated to avoid the strong reflection points.

[0064] Next, a description will be given of a schematic configuration of the photographing device 1. Fig. 5 is a block diagram showing a schematic configuration of the photographing device 1. Fig. 6 is a block diagram showing an overview of the processing performed by the processor 16. Fig. 7 is an explanatory diagram showing an overlapping region used in the feature point evaluation process P22 and the point cloud density evaluation process P24.

[0065] As shown in FIG. 5, in addition to the sensor unit 12, the photographing device 1 includes a display 13 (display unit), an input device 14, a memory 15 (storage unit), and a processor 16 (CPU).

[0066] In addition to the visible camera 21, the sensor unit 12 includes a depth camera 22 and an IMU 23 (Inertial Measurement Unit).

[0067] The depth camera 22 is a stereo camera that detects infrared light to capture an image of a subject, and outputs depth information (distance image) as a detection result. The distance to the subject can be measured based on the detection result of the depth camera 22. Note that the depth camera 22 may be replaced by a sensor capable of acquiring depth information using another method, such as LiDAR, in addition to the stereo camera.

[0068] The IMU 23 (detection unit) detects the motion state of the own device, specifically, three-dimensional angular velocity and acceleration. Based on the detection result of the IMU 23, the movement amount and rotation amount of the image capturing device 1 are measured.

[0069] The visible camera 21, the depth camera 22, and the IMU 23 may not be integrated into a sensor unit. Alternatively, the depth camera 22 and the IMU 23 may be omitted, and only the visible camera 21 may be provided. Alternatively, the visible camera 21 may be provided together with either the depth camera 22 or the IMU 23.

[0070] The display 13 presents various information related to the shooting operation to the user, and displays screens 101, 201 of the preliminary shooting mode (see Figures 12, 16, etc.), screens 301, 401, 501 of the setting change mode (see Figures 20, 21, 22, etc.), and screen 601 of the main shooting mode (see Figure 24, etc.).

[0071] The input device 14 is used by the user to perform input operations. The input device 14 may be a keyboard, a mouse, a touch pad, a touch panel, etc. When the photographing device 1 is configured as a tablet terminal, a touch panel display in which the touch panel as the input device 14 and the display panel as the display 13 are integrated is provided.

[0072] The memory 15 stores programs and the like executed by the processor 16. The memory 15 also stores images captured by the visible camera 21, distance information from the depth camera 22, and detection data from the IMU 23. The memory 15 also stores measurement results (point cloud data) generated by the processor 16.

[0073] The processor 16 performs various processes by executing the programs stored in the memory 15. In this embodiment, the processor 16 performs a three-dimensional measurement process P1, a preliminary shooting evaluation process P2, a shooting attention area detection process P3, a support information generation process P4, a display information generation process P5, a display process P6, and the like.

[0074] In this embodiment, the photographing device 1 performs a 3D measurement process P1, a preliminary photographing evaluation process P2, a photographing attention area detection process P3, a support information generation process P4, and a display information generation process P5, but all or part of these processes may be performed by a server device (not shown) capable of communicating with the photographing device 1.

[0075] As shown in FIG. 6, the three-dimensional measurement process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0076] In the feature extraction process P11, the processor 16 extracts feature information (feature points, etc.) from the images (frames) captured by the visible camera 21.

[0077] In tracking process P12, processor 16 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition related to the position and orientation of the image capture device 1, and updates the position and orientation trajectory data based on the amount of transition. Note that the position and orientation trajectory data is obtained by tracking the position and orientation of the image capture device 1, and includes the tracking results of the position and orientation at the time of capturing each captured image (frame), i.e., information on the position and orientation of the image capture device 1 at each time of capturing.

[0078] In the position and orientation correction process P13, the processor 16 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, the position and orientation trajectory data is corrected so as to compensate for the measurement results of places with few features, such as walls and ceilings.

[0079] In the point cloud generation process P14, the processor 16 generates point cloud data based on the distance information of the depth camera 22 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13.

[0080] Here, when performing preliminary photography, a simplified process is performed compared to the main photography, and simple point cloud data is generated as a result of the preliminary photography. This makes it possible to reduce the user's waiting time when the point cloud data as a result of the preliminary photography is presented to the user after the preliminary photography is completed. In addition, since the processing load of the processor 16 is reduced, battery consumption is reduced and photography can be performed for a long time. In addition, in a configuration in which the point cloud generation process P14 and the like are performed by a server device separate from the photography device 1, the amount of data communicated is reduced, and therefore the communication load can be reduced.

[0081] In addition, the point cloud generation process P14 performed during preliminary photography may be an even simpler process depending on the purpose of the preliminary photography. For example, when the point cloud data generated during preliminary photography is used only for the process of acquiring the target photography range and the process of detecting points to be taken into consideration for photography or the process of generating a recommended route based on the point cloud data is not performed, a point cloud that can roughly grasp the positional relationship of objects is sufficient, so the process may be even simpler. In this case, for example, the photographed data is thinned out, that is, the frame rate is artificially lowered and the process is performed, so that the load on the processor 16 can be further reduced.

[0082] Furthermore, in the point cloud generation process P14 performed during preliminary photography, if only a small number of point clouds need to be generated, the load on the processor 16 is reduced. Therefore, the movement speed of the photographing device 1 can be made faster than that during main photography, thereby making it possible to shorten the work time for preliminary photography.

[0083] The preliminary photography evaluation process P2 includes a reflection evaluation process P21, a feature point evaluation process P22, a sudden movement evaluation process P23, and a point cloud density evaluation process P24.

[0084] In the reflection evaluation process P21, the processor 16 acquires an evaluation value (reflection evaluation value) representing the intensity of the reflected light appearing in the captured image based on the detection data of the IMU 23 and the transition amount regarding the position and posture of the imaging device 1 acquired in the tracking process P12.

[0085] Specifically, the processor 16 compares the amount of movement of the image capturing device 1 based on the detection data of the IMU 23 with the amount of movement (amount of transition) of the image capturing device 1 acquired in the tracking process P12 to acquire a reflection evaluation value that indicates the degree of inconsistency between the two. Here, the amount of movement of the image capturing device 1 based on the detection data of the IMU 23 is obtained in the position and orientation correction process P13. In addition, in the tracking process P12, the amount of movement (amount of transition) of the image capturing device 1 is obtained by comparing the feature points extracted from the current captured image with the feature points extracted from the previous captured image. In this case, the reflection evaluation value indicates the degree of abnormality in the position estimation based on the captured image, that is, the magnitude of the error occurring in the position estimation based on the feature points extracted from the captured image, and indirectly indicates the appearance of strong reflected light in the captured image.

[0086] In another embodiment, the processor 16 obtains a reflection evaluation value that represents the appearance of reflected light in the captured image. That is, the size of the part in the captured image where the luminance value is locally high, specifically, the number of pixels whose luminance value is different from the surrounding pixels by a predetermined threshold or more, is obtained as the reflection evaluation value. In this case, the reflection evaluation value directly represents the appearance of strong reflected light in the captured image. Note that the reflection evaluation value in this case can also be used as a criterion for detecting shine or blown-out highlights in the captured image.

[0087] In the feature point evaluation process P22, the processor 16 counts the feature points within the target range based on the extraction result of the feature extraction process P11, and obtains an evaluation value (the number of feature points) that indicates the extraction status of the feature points.

[0088] The number of feature points may be the number of feature points extracted in one captured image (one frame), or the number of feature points extracted in an overlapping area (see FIG. 7) between the capture range at the current capture time t and the capture range at the previous capture time t-1.

[0089] In the sudden movement evaluation process P23, the processor 16 acquires an evaluation value (sudden movement evaluation value) representing the degree of suddenness of the movement of the user moving the photographing device 1 based on the detection data (acceleration, angular velocity) of the IMU 23. Note that the degree of suddenness of the movement represents the degree of suddenness of the change in the position and orientation of the own device, and specifically represents the degree of acceleration (including deceleration) occurring in the photographing device 1, or the degree of angular velocity occurring in the photographing device 1.

[0090] As another aspect, in the sudden movement evaluation process P23, the processor 16 acquires a sudden movement evaluation value based on the amount of transition regarding the position and orientation of the photographing device 1 acquired in the tracking process P12. Here, the amount of transition represents the change in the position and orientation of the photographing device 1, and the suddenness of the movement can be evaluated from the magnitude of the amount of transition.

[0091] In another aspect, in the sudden movement evaluation process P23, the processor 16 acquires the movement route (traffic line) of the user (photography device 1) during preliminary photographing, and acquires a sudden movement evaluation value as an evaluation value representing the likelihood that the user will perform a sudden movement based on the movement route and the positional status of surrounding objects estimated from the generated point cloud.

[0092] In the point cloud density evaluation process P24, the processor 16 counts the number of point clouds in the target range based on the point cloud data generated in the point cloud generation process P14, and obtains an evaluation value (point cloud density evaluation value) that indicates the point cloud generation status. The point cloud density evaluation value indicates whether the real-time point cloud generation is being performed appropriately. The smaller the point cloud density evaluation value, the lower the point cloud density.

[0093] The point cloud density evaluation value may be calculated from the number of point clouds generated in a predetermined unit time (for example, one second) (the total number of points generated in the unit time) and the size of the shooting range of the visible camera 21, as shown in the following formula. In this case, if the number of point clouds generated in the entire shooting range of the visible camera 21 is small, the point cloud density evaluation value will be small. Point cloud density evaluation value = (number of point clouds generated per unit time) / (shooting range)

[0094] The point cloud density evaluation value may be the number of generated points (the total number of points included in the overlapping area) in the overlapping area (see FIG. 7) between the shooting range at the current shooting time t and the shooting range at the previous shooting time t-1. In this case, the smaller the numerical value of the point cloud density evaluation value, the lower the point cloud density. In addition to the fact that the number of generated points in the shooting range of the visible camera 21 is small, the point cloud density evaluation value becomes small when the overlapping area between the shooting range at the current shooting time t and the shooting range at the previous shooting time t-1 is narrow. It is considered that the density of the point cloud may be improved by repeatedly shooting the same place. For this reason, the point cloud density evaluation value may be calculated by taking into account the existing point cloud generation situation at the location. In this case, the number of generated points is not only the point cloud generated at the current shooting time t but also the total number of all point clouds included in the overlapping area.

[0095] In another aspect, in the point cloud density evaluation process P24, the processor 16 calculates normal vectors perpendicular to the object surface by surface approximation or meshing of the point cloud data, and detects areas where the normal vectors are not continuous as areas where the point cloud is lacking.

[0096] In another aspect, in the point cloud density evaluation process P24, the processor 16 calculates the density for each position of the point cloud data, and detects areas where a point cloud exists but the density of the point cloud is extremely low (for example, when the density is within the interval between two predetermined thresholds) as areas where the point cloud is lacking.

[0097] The processing for detecting regions requiring caution in photography P3 includes processing for detecting strong reflection regions P31, processing for detecting regions lacking features P32, processing for detecting sudden movement regions P33, and processing for detecting regions lacking points P34.

[0098] In the strong reflection area detection process P31, the processor 16 detects strong reflection areas, i.e., areas where significant reflected light appears in the image captured by the visible camera 21, based on the reflection evaluation value acquired in the reflection evaluation process P21. Specifically, the reflection evaluation value is defined so that the larger the value, the stronger the reflection, and an area is determined to be a strong reflection area when the reflection evaluation value is greater than a predetermined threshold value. When a strong reflection area is detected, the position of the image capture device 1 at that time is stored in the memory 15 as the position of the strong reflection area. The strong reflection area detection process P31 is performed during the preliminary photographing.

[0099] In the feature-lacking portion detection process P32, the processor 16 detects a feature-lacking portion, i.e., a portion where feature points necessary for generating point cloud data with sufficient accuracy could not be sufficiently extracted, based on the number of feature points acquired in the feature point evaluation process P22. Specifically, a feature-lacking portion is determined to be a portion where the number of feature points is less than a predetermined threshold. When a feature-lacking portion is detected, the position of the image capture device 1 at that time is stored in the memory 15 as the position of the feature-lacking portion. At this time, since the reliability of the position estimation result based on the feature points is low, the position estimation result based on the detection data of the IMU 23 is adopted. The feature-lacking portion detection process P32 is performed during the preliminary shooting.

[0100] In the sudden movement part detection process P33, the processor 16 detects a sudden movement part, that is, a part where the user actually made a sudden movement during preliminary shooting, or a part where it is estimated that the user is likely to make a sudden movement, based on the sudden movement evaluation value acquired in the sudden movement evaluation process P23. Specifically, the sudden movement evaluation value is defined so that the larger the value, the quicker the movement, and when the sudden movement evaluation value is greater than a predetermined threshold, it is determined to be a sudden movement part. When a sudden movement part is detected by a method in which the sudden movement evaluation value is acquired based on the detection data of the IMU 23 or the transition amount acquired in the tracking process P12, the position of the image capture device 1 at that time is stored in the memory 15 as the position of the sudden movement part. On the other hand, in a method in which the sudden movement evaluation value is acquired based on the movement route (traffic line) of the image capture device 1 in preliminary shooting, the position determined to be a sudden movement part is stored in the memory 15 as the position of the sudden movement part based on the movement route and the arrangement status of surrounding objects estimated from the generated point cloud.

[0101] In a method in which the sudden movement evaluation value is acquired based on the detection data of the IMU 23 or the transition amount acquired in the tracking process P12, the sudden movement part detection process P33 is performed during the preliminary shooting. On the other hand, in a method in which the sudden movement evaluation value is acquired based on the movement route (traffic line) of the image capturing device 1 in the preliminary shooting, the sudden movement part detection process P33 is performed when the preliminary shooting is completed.

[0102] In the point cloud insufficient portion detection process P34, the processor 16 detects a point cloud insufficient portion, that is, a portion where the point cloud was not sufficiently generated in the preliminary shooting, based on the point cloud density evaluation value acquired in the point cloud density evaluation process P24. Specifically, when the point cloud density evaluation value is less than a predetermined threshold, it is determined to be a point cloud insufficient portion. When a point cloud insufficient portion is detected by a method in which the point cloud density evaluation value is obtained by counting the number of point clouds generated in real time (the existing point cloud generation situation at the portion may be taken into account), the position of the photographing device 1 at that time is stored in the memory 15 as the position of the point cloud insufficient portion. On the other hand, in a method of detecting a portion where the normal vector is not continuous, or a method of detecting a portion where the point cloud exists but the density of the point cloud is extremely low (for example, when the density is within a section between two predetermined thresholds), the position determined to be the non-continuous portion or the extremely low density portion is stored in the memory 15 as the position of the point cloud insufficient portion.

[0103] In a method in which the point cloud density evaluation value is calculated by counting the number of generated point clouds in real time (the existing point cloud generation status at the location may be taken into consideration), the point cloud lacking portion detection process P34 is performed during the preliminary shooting. On the other hand, in a method in which a portion where normal vectors are not continuous is detected, or a portion where a point cloud exists but the density of the point cloud is extremely low (for example, when the density is within an interval between two predetermined thresholds), the point cloud lacking portion detection process P34 is performed when the preliminary shooting is completed.

[0104] The support information generation process P4 includes a target shooting range acquisition process P41, a progress rate acquisition process P42, a recommended route generation process P43, a progress status image generation process P44, a map image generation process P45, a route image generation process P46, and a confirmation map generation process P47.

[0105] In the target shooting range acquisition process P41, the processor 16 sets a target shooting range to be the target of main shooting based on the point cloud data generated in the point cloud generation process P14 during preliminary shooting. Also, the processor 16 sets a user-specified target shooting range in response to a user's operation to change the range from the initial target shooting range set based on the point cloud data.

[0106] In the progress rate acquisition process P42, the processor 16 acquires the photographed range in the main photography based on the point cloud data generated in the point cloud generation process P14 up to now in the main photography, and acquires the progress rate of the photography in the main photography based on the photographed range and the target photography range acquired in the target photography range acquisition process P41. The progress rate is the ratio of the photographed range to the target photography range.

[0107] In the recommended route generation process P43, the processor 16 generates a recommended route for the actual shooting, i.e., a route that is recommended for the user to take during the actual shooting, based on the target shooting range acquired in the target shooting range acquisition process P41. Specifically, a route that covers the entire target shooting range and has the shortest possible travel distance is generated as the recommended route. In addition, in the recommended route generation process P43, a recommended route is generated in response to a user's instruction so as to avoid areas that require attention when shooting, i.e., areas with strong reflections, areas with insufficient features, and areas with sudden movements.

[0108] In the progress status image generation process P44, the processor 16 generates an image that visualizes the progress status based on the progress rate acquired in the progress rate acquisition process P42. Specifically, an indicator image 621 (see FIG. 25, etc.) is generated. The indicator image 621 is, for example, a progress bar, and is drawn so that the portion indicating the photographed range changes as the actual photographing progresses.

[0109] In the map image generation process P45, the processor 16 generates an overhead point cloud image that becomes the map image 32 representing the measurement target location on the confirmation map 31 based on the point cloud data generated in the point cloud generation process P14. Specifically, after shooting in the preliminary shooting mode, an overhead point cloud image 35 by preliminary shooting, i.e., an overhead point cloud image is generated based on the point cloud data generated in the point cloud generation process P14 in the preliminary shooting (see FIG. 16, etc.). During shooting in the main shooting mode, an overhead point cloud image 36 by main shooting, i.e., an overhead point cloud image is generated based on the point cloud data generated in the point cloud generation process P14 from the start of the main shooting to the present (see FIG. 24, etc.).

[0110] In the route image generation process P46, the processor 16 generates a route image 51 representing the recommended route based on the recommended route generated in the recommended route generation process P43 (see Figs. 22, 30, etc.). The route image 51 visualizes the progress of the actual photography. Specifically, as the actual photography progresses, the photographed sections of the route image 51 are erased. Also, the photographed sections of the route image 51 may not be erased and may be drawn in a different drawing mode (e.g., color) from the unphotographed sections. Note that, although an example is shown in which the start position mark (S) is displayed on the confirmation map 31 even after photography has started, for example, after photography has started, the confirmation map 31 (see Fig. 31) from which the start position mark (S) has been erased may be displayed.

[0111] In the confirmation map generation process P47, the processor 16 generates a confirmation map 31 for the user to confirm matters to be aware of during actual photography, based on the map image 32 acquired in the map image generation process P45, the route image 51 acquired in the route image generation process P46, and the position information of the points to be careful about during photography detected in the point to be careful about photography detection process P3. The confirmation map 31 is obtained by superimposing an image that visualizes various support information on the map image 32 that shows the measurement target location. The confirmation map 31 is displayed after photography in the preliminary photography mode and during photography in the main photography mode.

[0112] In the confirmation map 31 displayed after shooting in the preliminary shooting mode, an image visualizing points to be careful of when shooting, that is, points with insufficient point clouds, points with strong reflections, points with insufficient features, and points with sudden movements, is superimposed on a map image 32. The map image 32 is an image visualizing the results of the preliminary shooting, specifically, an overhead point cloud image 35 generated from point cloud data acquired in the preliminary shooting. The image visualizing points to be careful of when shooting, specifically, are mark images 41, 42, and 43 representing points with strong reflections, points with insufficient features, and points with sudden movements, and an area image 44 representing points with insufficient point clouds.

[0113] In addition, in the confirmation map 31 displayed during shooting in the main shooting mode, images indicating points to be careful of when shooting (mark images 41, 42, 43, area image 44) and a route image 51 indicating a recommended route are superimposed on the map image 32.

[0114] At this time, the progress is visualized on the map image 32. Specifically, the map image 32 (see FIG. 24, etc.) includes an overhead point cloud image 35 from preliminary shooting and an overhead point cloud image 36 from main shooting, and is drawn so that as the main shooting progresses, the already-captured range in the overhead point cloud image 35 from preliminary shooting is gradually erased and replaced with the overhead point cloud image 36 from main shooting.

[0115] The progress is also visualized in images showing areas requiring attention when photographing (mark images 41, 42, 43, area image 44). Specifically, the mark images 41, 42, 43 showing areas with strong reflection, areas lacking features, and areas with sudden movements, and the area image 44 showing areas lacking point clouds are replaced with those generated during the actual photographing from those generated during the preliminary photographing as the actual photographing progresses. The progress is also visualized in the route image 51. Specifically, the photographed sections of the route image 51 are erased as the actual photographing progresses.

[0116] In the display information generation process P5, the processor 16 generates display information for a screen (see FIGS. 12 to 31) to be displayed on the display 13 based on the image captured by the visible camera 21, the confirmation map 31 generated in the confirmation map generation process P47, and the like.

[0117] In the display process P6, the processor 16 displays a screen (see FIGS. 12 to 31) on the display 13 based on the display information generated in the display information generation process P5.

[0118] Next, a description will be given of the procedure of the processing performed by the processor 16 during shooting (during preliminary shooting and main shooting). Fig. 8 is a flow chart showing the procedure of the processing performed by the processor 16 during shooting.

[0119] First, the processor 16 acquires an image captured by the visible camera 21, distance information from the depth camera 22, and detection data from the IMU 23 from the sensor unit 12 (ST101).

[0120] Next, the processor 16 extracts feature points from the image captured by the visible camera 21 (feature extraction process P11) (ST102).

[0121] Next, the processor 16 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition regarding the position and orientation of the image capturing device 1, and updates the position and orientation trajectory data based on the amount of transition (tracking process P12) (ST103).

[0122] Next, the processor 16 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23 (position and orientation correction process P13) (ST104).

[0123] Next, the processor 16 generates point cloud data based on the distance information of the depth camera 22 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13 (point cloud generation process P14) (ST105).

[0124] Next, the processor 16 determines whether or not preliminary shooting is in progress (ST106). If preliminary shooting is in progress (Yes in ST106), the processor 16 then performs processing during preliminary shooting (ST107). On the other hand, if preliminary shooting is not in progress (No in ST106), the processor 16 then determines whether or not main shooting is in progress (ST108). If main shooting is in progress (Yes in ST108), the processor 16 then performs processing during main shooting (ST109).

[0125] Next, the processor 16 generates display information regarding the screen being photographed, including the image photographed by the visible camera 21 (display information generation process P5) (ST110). At this time, if preliminary photographing is in progress, display information regarding the screen being photographed, including the overhead point cloud image 35 generated in the preliminary photographing, is generated in addition to the image photographed by the visible camera 21. Also, if main photographing is in progress, display information regarding the screen being photographed, including the confirmation map 31, is generated in addition to the image photographed by the visible camera 21. The confirmation map 31 includes a map image 32 consisting of the overhead point cloud image 35, mark images 41, 42, 43 and an area image 44 indicating points to be careful of when photographing, and a route image 51 indicating a recommended route (see FIG. 2 and FIG. 3).

[0126] Next, the processor 16 displays a screen on the display 13 based on the generated display information (display process P6) (ST111).

[0127] Next, a procedure for the process during preliminary photographing in ST107 shown in Fig. 8 will be described. Fig. 9 is a flow diagram showing the procedure for the process during preliminary photographing.

[0128] First, the processor 16 obtains an evaluation value (reflection evaluation value) that indicates the intensity of reflected light that appears in a captured image (reflection evaluation process P21) (ST201).

[0129] Next, processor 16 counts the feature points within the target range based on the extraction result of feature extraction process P11, and obtains an evaluation value (the number of feature points) representing the extraction status of the feature points (feature point evaluation process P22) (ST202).

[0130] Next, the processor 16 acquires an evaluation value (sudden movement evaluation value) that indicates the abruptness of the user's movement of the photographing device 1 (sudden movement evaluation process P23) (ST203). At this time, the sudden movement evaluation value is calculated based on the detection data of the IMU 23. In addition, the sudden movement evaluation value is calculated based on the amount of transition regarding the position and orientation of the photographing device 1 acquired in the tracking process P12.

[0131] Next, the processor 16 counts the point clouds within the target range based on the point cloud data generated in the point cloud generation process P14, and obtains an evaluation value (point cloud density evaluation value) representing the point cloud generation status (point cloud density evaluation process P24) (ST204).

[0132] Next, processor 16 detects areas to be careful of when photographing (areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements) based on the evaluation values ​​obtained in each of the reflection evaluation process P21 (ST201), the feature point evaluation process P22 (ST202), the sudden movement evaluation process P23 (ST203), and the point cloud density evaluation process P24 (ST204) (areas to be careful of when photographing) (detection process P3) (ST205).

[0133] At this time, the reflection evaluation value is compared with a predetermined threshold to detect strong reflection points (strong reflection point detection process P31). The number of feature points is also compared with a predetermined threshold to detect feature-lacking points (feature-lacking point detection process P32). The sudden movement evaluation value is also compared with a predetermined threshold to detect sudden movement points (sudden movement point detection process P33). The point cloud density evaluation value is also compared with a predetermined threshold to detect point cloud-lacking points (point cloud-lacking point detection process P34). The thresholds used in each detection process may be preset or may be appropriately specified by the user.

[0134] Next, the processor 16 judges whether or not a region to be careful of when photographing (a region lacking a point cloud, a region with strong reflection, a region lacking a feature, and a region with sudden movement) has been detected (ST206). If a region to be careful of when photographing has been detected (Yes in ST206), the processor 16 stores the position of the detected region to be careful of when photographing and the content of a warning to the user in the memory 15 (ST207).

[0135] Next, the processor 16 accumulates the point cloud data generated in the point cloud generation process P14 (ST105 in FIG. 8) in the memory 15 (ST208). Next, the processor 16 generates display information for a screen including a highlight or a text display indicating that preliminary imaging is in progress (display information generation process P5) (ST209).

[0136] Next, a description will be given of the procedure of the process during actual shooting in ST109 shown in Fig. 8. Fig. 10 is a flow diagram showing the procedure of the process during actual shooting.

[0137] First, the processor 16 compares the point cloud data generated this time in the point cloud generation process P14 (ST105 in FIG. 8) with the point cloud data at the time of preliminary shooting stored in the memory 15, and determines whether or not the range of the point cloud data generated this time overlaps with the range of the point cloud data at the time of preliminary shooting (ST301).

[0138] Here, if the range of the current point cloud data overlaps with the range of the point cloud data at the time of preliminary shooting (Yes in ST302), then processor 16 updates map image 32 consisting of overhead point cloud images 35, 36 (see FIG. 24, etc.) for the range where the current point cloud data and the point cloud data at the time of preliminary shooting overlap (map image generation process P45) (ST303). At this time, map image 32 is drawn so that overhead point cloud image 35 from preliminary shooting is deleted and replaced with overhead point cloud image 36 from main shooting. Note that, even for point cloud data at the time of preliminary shooting that has some error with respect to the point cloud data at the time of main shooting, it may be determined that both point cloud data overlap, and the overhead point cloud image 35 from preliminary shooting may be deleted.

[0139] Next, the processor 16 generates a confirmation map 31 including an updated or unupdated map image 32 (confirmation map generation process P47) (ST304). At this time, the confirmation map 31 is generated by superimposing the mark images 41, 42, 43 and area image 44 representing points to be careful of when taking photographs, and the route image 51 representing a recommended route on the map image 32 in accordance with the display setting contents based on the user's designation.

[0140] Next, a description will be given of the procedure of the process carried out when the preliminary photographing is completed in the processor 16. Fig. 11 is a flow chart showing the procedure of the process carried out when the preliminary photographing is completed.

[0141] First, the processor 16 detects points where the point cloud is lacking based on the point cloud data acquired by the preliminary photographing, i.e., the point cloud data generated by the point cloud generation process P14 during the preliminary photographing and stored in the memory 15 (point cloud lacking area detection process P34) (ST401).

[0142] Next, the processor 16 generates a recommended route based on the target shooting range set from the point cloud data acquired by the preliminary shooting and the position and contents of the detected shooting caution points (recommended route generation process P43) (ST402). Specifically, a route that covers the target shooting range and has the shortest possible travel distance is generated as the recommended route. At this time, the recommended route is generated according to setting information related to the generation of the recommended route. For example, when the user is instructed to avoid the shooting caution points, a recommended route is generated to avoid the shooting caution points.

[0143] Next, the processor 16 generates a confirmation map 31 including an overhead point cloud image 35 (map image 32) generated from the point cloud data acquired in the preliminary photographing, mark images 41, 42, 43 and an area image 44 indicating areas to be careful of when photographing (areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements), and a route image 51 indicating a recommended route (confirmation map generation process P47) (ST403).

[0144] Next, the processor 16 generates display information for a screen (confirmation screen) including the confirmation map 31 as a screen for the user to confirm the preliminary shooting result (display information generation process P5) (ST404). Next, the processor 16 displays the screen on the display 13 based on the generated display information (display process P6) (ST405).

[0145] Next, processor 16 determines whether or not a user operation (e.g., tap operation, click operation) has been performed to select areas to be careful about when photographing (areas lacking point clouds, areas with strong reflections, areas lacking features, and areas with sudden movements) (ST406). If a user operation to select areas to be careful about when photographing has not been performed (No in ST406), processor 16 then determines whether or not a user operation to close the screen for checking the preliminary photographing results has been performed (ST407). If a user operation to close the screen for checking the preliminary photographing results has not been performed (No in ST407), the process returns to ST406.

[0146] On the other hand, when a user operation to select a point to be careful about photographing is performed (Yes in ST406), display information for a screen (confirmation screen) including a message that provides guidance (such as a suggestion of an improvement measure) according to the type of the selected point to be careful about photographing is generated (display information generation process P5) (ST408). Next, returning to ST404, processor 16 displays a screen including a message on display 13 based on the generated display information.

[0147] Next, a screen 101 displayed on the display 13 in the preliminary capture mode will be described. Fig. 12 is an explanatory diagram showing the screen 101 displayed before shooting in the preliminary capture mode. Figs. 13 and 14 are explanatory diagrams showing a screen 201 displayed during shooting in the preliminary capture mode. Fig. 15 is an explanatory diagram showing a screen 301 displayed immediately after shooting in the preliminary capture mode.

[0148] 12 to 15, a screen 101 in the preliminary capture mode is provided with a main window 102, a sub-window 103, and an operation window 104. The sub-window 103 is provided so as to be overlapped with a part of the main window 102.

[0149] The main window 102 and the sub window 103 each display a captured image 111 and an overhead point cloud image 35. The captured image 111 is a current captured image, that is, an image captured in real time and output from the visible camera 21. The overhead point cloud image 35 is an image (rendering) of each point of the point cloud data as seen from a viewpoint in the sky.

[0150] In addition, on screen 101 before shooting shown in Fig. 12, point cloud data of the measurement target location has not been generated, so overhead point cloud image 35 is not displayed. Also, on screen 101 during shooting shown in Fig. 13 and Fig. 14, point cloud data of the measurement target location has been partially generated, so overhead point cloud image 35 is partially displayed. Also, on screen 101 immediately after shooting shown in Fig. 15, point cloud data of the measurement target location has been generated overall, so overhead point cloud image 35 is displayed overall.

[0151] The operation window 104 is provided with a "start preliminary shooting" button 121 and an "end preliminary shooting" button 122. When the user operates the "start preliminary shooting" button 121, preliminary shooting is started, and images captured by the visible camera 21 are stored in the memory 15. When the user operates the "end preliminary shooting" button 122, preliminary shooting is ended. Note that in a configuration including the depth camera 22 and the IMU 23, distance information from the depth camera 22 and detection data from the IMU 23 are stored in the memory 15 together with images captured by the visible camera 21.

[0152] The operation window 104 is also provided with a "screen switching" button 123 and a check box 124 related to display of the sub-window 103. When the user operates the "screen switching" button 123, a state in which the captured image 111 is enlarged and displayed in the main window 102 and the overhead point cloud image 35 is reduced and displayed in the sub-window 103 (the state shown in Figs. 13 and 14) is switched to a state in which the overhead point cloud image 35 is enlarged and displayed in the main window 102 and the captured image 111 is reduced and displayed in the sub-window 103. When the user operates the check box 124, the display state and non-display state of the sub-window 103 can be switched.

[0153] In addition, the operation window 104 is provided with a tab 131 of "Main Shooting", a tab 132 of "Preliminary Shooting", and a tab 133 of "Change Settings" as operation sections for selecting a mode. The screen 101 shown in Figs. 12 to 19 is for the preliminary shooting mode (first shooting mode), and is displayed when the user operates the tab 132 of "Preliminary Shooting". When the user operates the tab 133 of "Change Settings", the mode is switched to the change settings mode, and a transition is made to screens 301, 401, and 501 (see Figs. 20 to 23). When the user operates the tab 131 of "Main Shooting", the mode is switched to the main shooting mode (second shooting mode), and a transition is made to screen 601 (see Figs. 24 to 31).

[0154] In this embodiment, the modes are switched by operating the tabs 131, 132, and 133, but the modes may be switched by other types of operation units such as buttons.

[0155] When shooting is started in the preliminary shooting mode, the display mode of the screen 101 changes to inform the user that preliminary shooting is in progress. In the example shown in FIG. 13, the screen 101 is highlighted. Specifically, frame images 141 and 142 are displayed in the main window 102 and the operation window 104. The frame images 141 and 142 are drawn in a predetermined color (e.g., red). In addition, the background of the operation window 104 is displayed in a predetermined color (e.g., green). On the other hand, in the example shown in FIG. 14, characters 151 indicating that preliminary shooting is in progress are displayed at the top of the main window 102. Note that the screen 101 specific to each mode may be highlighted by switching between the tabs 131, 132, and 133.

[0156] During preliminary shooting, message display windows containing various messages are displayed on the screen as necessary. Messages may be of relatively high importance (urgency), such as instructions or notifications that require immediate action, or of relatively low importance, such as reference information or content requiring attention. Message display windows containing messages of relatively high importance are displayed in the center of the main window 102, while message display windows containing messages of relatively low importance are displayed at the bottom of the main window 102. Message display windows disappear after a predetermined display time has elapsed or when the need for displaying them has disappeared.

[0157] For example, when preliminary photographing starts, a confirmation message such as "Preliminary photographing will start. Please take photographs of the entire 3D point cloud conversion range" is displayed at the bottom of the main window 102. Also, in the example shown in Fig. 15, when preliminary photographing ends, a confirmation message is displayed in the message display window 161 in the center of the main window 102. For example, a message such as "Processing. Please wait a moment" is displayed.

[0158] Next, a screen 201 that is displayed on the display 13 when checking after shooting in the preliminary shooting mode will be described. Figures 16, 17, 18, and 19 are explanatory diagrams showing the screen 201 that is displayed when checking after shooting in the preliminary shooting mode.

[0159] 16, 17, and 18 are examples of a screen 201 (confirmation screen) that is displayed after shooting in the preliminary shooting mode. In the screen 201, a confirmation map 31 is displayed in the main window 102 so that the user can check points to be careful about before the actual shooting.

[0160] 16 shows an example in which strong reflection areas and point cloud insufficient areas are detected. In this case, a mark image 41 representing the strong reflection area is superimposed on the position where the strong reflection area is detected on the map image 32 (bird's-eye view point cloud image 35) in the confirmation map 31. In addition, an area image 44 representing the point cloud insufficient area is superimposed on the range where the point cloud insufficient area is detected on the map image 32.

[0161] When the user selects a strong reflection point on the confirmation map 31, specifically, when the user performs a predetermined operation (tap operation, click operation) on the mark image 41 representing the strong reflection point, a message is displayed in the message display window 211 to prompt the user to take measures to improve the shooting environment related to the strong reflection point. In this example, the selected strong reflection point is a case where the reflected light is assumed to be caused by indoor lighting, and a message is displayed to prompt the user to improve the shooting environment related to lighting, for example, a message saying "If the reflection is strong, please reduce the lighting at the shooting point." In addition, when the reflected light is assumed to be caused by external light entering through a window, a message is displayed to prompt the user to improve the shooting environment related to external light, for example, a message saying "If the curtains are open and external light is entering, please close the curtains." Note that, when both the indoor lighting and the external light entering through the window may be the cause, a message may be displayed to prompt the user to improve both, for example, a message saying "Please reduce the lighting at the shooting point and close the curtains."

[0162] In addition, when the user selects a point cloud deficiency area on the confirmation map 31, specifically, performs a specified operation (tap operation, click operation) on the area image 44 representing the point cloud deficiency area, a message may be displayed in the message display window 211 urging the user to improve the shooting environment or shooting method related to the point cloud deficiency area.

[0163] 17 shows an example in which a feature-lacking portion is detected. In this case, in the confirmation map 31, a mark image 42 representing the feature-lacking portion is superimposed at the position where the feature-lacking portion is detected on the map image 32 (bird's-eye view point cloud image 35).

[0164] When the user performs an operation to select a feature-lacking portion on the confirmation map 31, specifically, a predetermined operation (tap operation, click operation) on the mark image 42 representing the feature-lacking portion, a message is displayed in the message display window 211 to prompt the user to take measures to improve the shooting environment or shooting method related to the feature-lacking portion. In this example, the selected feature-lacking portion is caused by the shooting environment of the measurement target location, and a message is displayed to prompt the user to improve the shooting environment, for example, a message saying "Please place something with texture." In addition, when the selected feature-lacking portion is caused by the user's shooting method, a message is displayed to prompt the user to improve the shooting method, for example, a message saying "Please set the angle of view to include a distinctive pattern or object."

[0165] 18 shows an example of a case where a sudden movement part is detected. In this case, in the confirmation map 31, a mark image 43 representing the sudden movement part is superimposed on the position where the sudden movement part is detected on the map image 32 (overhead point cloud image 35).

[0166] When the user performs an operation to select a sudden movement location on the confirmation map 31, specifically, a predetermined operation (tap operation, click operation) on the mark image 43 representing the sudden movement location, a message urging the user to take measures to improve the shooting method related to the sudden movement is displayed in the message display window 211. Specifically, a message urging the user not to make a sudden movement (a sudden turn), for example, a message saying "When shooting, move slowly to avoid making a sudden turn" is displayed.

[0167] Fig. 19 is another example of the screen 201 displayed after shooting in the preliminary shooting mode. In the examples shown in Figs. 16, 17, and 18, a message is displayed in the message display window 211 to prompt the user to improve the shooting environment or shooting method by operating the mark images 41, 42, and 43 representing the points to be careful about when shooting. On the other hand, in this example, the contents of the points to be careful about when shooting are displayed in a list in the legend display section 221. This allows the user to understand the contents of the mark images 41, 42, and 43 and area image 44 representing the points to be careful about when shooting, which are drawn on the confirmation map 31.

[0168] In this embodiment, points to be careful about when photographing are detected based on the results of preliminary photographing, and the confirmation map 31 on which the mark images 41, 42, 43 and the area image 44 representing the points to be careful when photographing are displayed on the screen after the preliminary photographing, and by selecting a point to be careful when photographing on the confirmation map 31, a message providing guidance (such as a suggestion of an improvement measure) regarding the selected point to be careful when photographing is displayed. This allows the user to understand the location and content of the points to be careful when photographing, and improve the photographing environment and photographing method to suppress the occurrence of events that cause a decrease in point cloud accuracy, thereby enabling the user to obtain highly accurate point cloud data with only one actual photographing.

[0169] In this embodiment, by selecting a point to be photographed on the confirmation map 31 on the screen 201 after preliminary photographing, a message providing guidance regarding the selected point to be photographed is displayed, but a message providing guidance regarding the detected point to be photographed may also be displayed when a point to be photographed is detected during preliminary photographing.

[0170] 16 to 19, in the confirmation map 31, mark images 41, 42, and 43 representing areas to be careful of when photographing (high reflection areas, areas lacking features, and areas of sudden movements) are superimposed on the map image 32 (bird's-eye view point cloud image 35), but if multiple areas to be careful of when photographing are detected at a high density, an area image drawn to include the multiple areas to be careful of when photographing may be superimposed. Also, if the mark image representing the areas to be careful of when photographing is, for example, circular, multiple mark images can be superimposed so as to overlap each other, so that they can be recognized as one area.

[0171] Next, the screens 301, 401, 501 (setting screens) in the setting change mode related to the preliminary shooting results displayed on the display 13 will be described. Fig. 20 is an explanatory diagram showing the screen 301 related to the progress status setting. Fig. 21 is an explanatory diagram 401 showing the screen related to the shooting caution point display setting. Figs. 22 and 23 are explanatory diagrams showing the screen 501 related to the recommended route setting.

[0172] As shown in Figures 20 to 23, the setting change mode screens 301, 401, and 501 are provided with a "Progress setting" tab 311, a "Caution for Shooting Areas Display Setting" tab 312, and a "Recommended Route Setting" tab 313 as operation sections for switching settings.

[0173] When the user operates the "progress setting" tab 311, a screen 301 (see FIG. 20) relating to the progress setting is displayed. When the user operates the "noticeable points for photographing display setting" tab 312, a screen 401 (see FIG. 21) relating to the noticeable points for photographing display setting is displayed. When the user operates the "recommended route setting" tab 313, a screen 501 (see FIG. 22 and FIG. 23) relating to the recommended route setting is displayed.

[0174] In this embodiment, the settings are switched by operating the tabs 311, 312, and 313, but the settings may be switched by another type of operation unit such as a button.

[0175] 20 is displayed when the user operates the "Progress Status Setting" tab 311. In this screen 301, the user can specify whether or not to display the progress status. In addition, in this screen 301, the user can specify a target shooting range that will be the reference when acquiring the progress status.

[0176] A check box 321 for displaying the progress status is provided on the screen 301. When the user operates the check box 321, the user can switch the setting as to whether or not to display the progress status on the screen 601 (see FIG. 25, etc.) during actual shooting. Specifically, the user can switch between displaying and not displaying an indicator image 621 indicating the progress rate on the screen 601 during actual shooting.

[0177] Furthermore, the screen 301 is provided with a "Start range input" button 323 and an "Execute range setting" button 324 for setting a target shooting range designated by the user.

[0178] When the user operates the "Start range input" button 323, the mode transitions to range input mode. In the range input mode, the user can input the target shooting range on the overhead point cloud image 35 displayed on the main window 102. Specifically, for example, the user can input the target shooting range by performing an operation (for example, a tap operation or a click operation) to sequentially specify the vertices of a polygon representing the target shooting range. At this time, an area image 61 representing the target shooting range is displayed on the main window 102 together with the overhead point cloud image 35. This allows the user to specify a preliminary shooting range, that is, a range different from the target shooting range set based on the point cloud data generated in the preliminary shooting, as a new target shooting range.

[0179] When the user operates the "Execute range setting" button 324, the range input mode is cancelled and the target shooting range input by the user is registered.

[0180] 21 is displayed when the user operates the "Caution Points for Shooting" tab 312. In this screen 401, the user can specify whether or not to display each type of caution points for shooting on the confirmation map 31, and can check the confirmation map 31 generated based on the specification.

[0181] The screen 401 is provided with check boxes 411 to 414 for displaying areas requiring attention when photographing, that is, areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements.

[0182] The user can switch between displaying or not displaying a mark image 41 representing a strong reflection portion on the confirmation map 31 by operating a check box 411 related to the display of a strong reflection portion. The user can switch between displaying or not displaying a mark image 42 representing a feature-lacking portion on the confirmation map 31 by operating a check box 412 related to the display of a feature-lacking portion. The user can switch between displaying or not displaying a mark image 43 representing a sudden movement portion on the confirmation map 31 by operating a check box 413 related to the display of a sudden movement portion. The user can switch between displaying or not displaying an area image 44 representing a point cloud-lacking portion on the confirmation map 31 by operating a check box 414 related to the display of a point cloud-lacking portion.

[0183] Moreover, the screen 401 is provided with a "execute display setting" button 415. When the user operates the "execute display setting" button 415, display conditions for the points to be careful of when taking pictures are set according to the input contents of the check boxes 411 to 414. At this time, in the main window 102, a confirmation map 31 is generated in which the mark images 41, 42, 43 and the area image 44 are superimposed on the map image 32 (bird's-eye view point cloud image 35) according to the settings, and the confirmation map 31 is displayed in the main window 102. Note that, as an initial setting state, all the check boxes 411 to 414 may be checked, and the user may uncheck them at will. Also, the display settings may be saved in association with the shooting locations for each user, and the saved display settings may be reused.

[0184] 22 is displayed when the user operates the "recommended route setting" tab 313. In this screen 501, the user can specify conditions for generating a recommended route and check the recommended route generated based on those conditions.

[0185] The screen 501 is provided with check boxes 511 to 513 for avoiding locations that require attention when photographing when generating a recommended route, that is, locations with strong reflections, locations with insufficient features, and locations with sudden movements.

[0186] The user can switch whether or not to avoid strong reflection points when generating a recommended route by operating a check box 511 related to avoiding strong reflection points. The user can switch whether or not to avoid feature-lacking points when generating a recommended route by operating a check box 512 related to avoiding feature-lacking points. The user can switch whether or not to avoid feature-lacking points when generating a recommended route by operating a check box 513 related to avoiding sudden movement points.

[0187] Furthermore, the screen 501 is provided with a “generate recommended route” button 514. When the user operates the “generate recommended route” button 514, a recommended route is generated according to the input contents of the check boxes 511 to 513. Furthermore, a route image 51 representing the recommended route is generated, and a confirmation map 31 is generated in which the route image 51 is superimposed on the map image 32 (bird's-eye view point cloud image 35), and the confirmation map 31 is displayed in the main window 102.

[0188] In the example shown in FIG. 22, the strong reflection parts, the feature-lacking parts, and the sudden movement parts are set not to be avoided by the operation input of the check boxes 511 to 513. In this case, a route that covers the entire target shooting range and has the shortest possible moving distance is generated as the recommended route, regardless of the strong reflection parts, the feature-lacking parts, and the sudden movement parts. On the other hand, in the example shown in FIG. 23, the strong reflection parts are set to be avoided. In this case, the recommended route is generated so as to avoid the strong reflection parts. Note that the setting for generating the recommended route may be saved in association with the shooting location for each user, and the saved setting for generating the recommended route may be reused.

[0189] Next, a description will be given of a screen 601 in the actual shooting mode displayed on the display 13. Figures 24, 25, 26, 27, 28, 29, 30, and 31 are explanatory diagrams showing the screen 601 displayed on the display 13 in the actual shooting mode.

[0190] As shown in FIG. 24, a screen 601 in the main shooting mode is provided with a main window 102, a sub-window 103, and an operation window 104, similar to the screen 101 in the preliminary shooting mode (see FIG. 12, etc.).

[0191] A confirmation map 31 is displayed in the sub-window 103 to allow the user to confirm points that need attention when taking an actual photograph.

[0192] The operation window 104 is provided with a "Start Main Shooting" button 611 and an "End Main Shooting" button 612. When the user operates the "Start Main Shooting" button 611, the main shooting is started, and the image captured by the visible camera 21, the distance information by the depth camera 22, and the detection data by the IMU 23 are stored in the memory 15. When the user operates the "End Main Shooting" button 612, the main shooting is ended.

[0193] Immediately after the screen 601 of the actual shooting mode is displayed or immediately after the user operates the "Start actual shooting" button 611, a confirmation message such as "Start actual shooting. Please shoot from the starting point" may be displayed. Also, actual shooting may start from the point where preliminary shooting ended, or the starting point of actual shooting may be suggested based on the preliminary shooting result. Also, in order to obtain the progress during actual shooting, alignment between the preliminary shooting result (point cloud) and the actual shooting result (point cloud) is required. To facilitate this alignment, during preliminary shooting, shooting may be ended with the photographing device 1 placed on a desk or the like, and the photographing device 1 may not be moved until actual shooting is started.

[0194] In addition, the operation window 104 is provided with a "screen switch" button 123 and a check box 124 for displaying the sub-window 103, similar to the screen 101 in the preliminary capture mode (see FIG. 12, etc.).

[0195] The operation window 104 also has a check box 615 for displaying the progress status, a check box 616 for displaying the preliminary shooting results, i.e., the point cloud acquired in the preliminary shooting, a check box 617 for displaying areas to be careful about when shooting, and a check box 618 for displaying a recommended route.

[0196] The user can specify whether or not to display the progress status (enable or disable) by operating a check box 615 related to the display of the progress status. The user can also specify whether or not to display the preliminary shooting results by operating a check box 616 related to the display of the preliminary shooting results (point cloud). The user can also specify whether or not to display the points to be careful about when shooting by operating a check box 617 related to the display of the points to be careful about when shooting. The user can also specify whether or not to display the recommended route by operating a check box 618 related to the display of the recommended route.

[0197] Here, the areas to be careful about when photographing include areas with strong reflection, areas with insufficient features, areas with sudden movements, and areas with insufficient point clouds, and the presence or absence of display for each type of area to be careful about when photographing can be specified collectively using the checkboxes 617 related to display of areas to be careful about when photographing. Note that the presence or absence of display may be specified individually for each type of area to be careful about when photographing (areas with strong reflection, areas with insufficient features, areas with sudden movements, and areas with insufficient point clouds).

[0198] Note that the check boxes 615-618 on the screen in the actual shooting mode may be operated before shooting starts, but may also be operated after shooting starts. For example, if the user feels bothered by the display of the progress status, preliminary shooting results, points to be careful about in shooting, and recommended route, the user may operate the check boxes 615-618 during actual shooting to change whether or not to display them.

[0199] The example shown in Figure 24 is a case where, by operating check boxes 615 to 618, the display of the progress status is disabled, the display of the preliminary shooting results is enabled, the display of areas to be careful about when shooting is disabled, and the display of the recommended route is disabled.

[0200] In this case, a map image 32 showing the measurement target location is displayed on the confirmation map 31 displayed in the sub window 103. The map image 32 includes an overhead point cloud image 35 from preliminary shooting and an overhead point cloud image 36 from main shooting. The overhead point cloud image 35 from preliminary shooting is an overhead point cloud image in which the point cloud data acquired in the preliminary shooting is visualized, and the overhead point cloud image 36 from main shooting is an overhead point cloud image in which the point cloud data acquired up to the present in main shooting is visualized.

[0201] The overhead point cloud image 35 from the preliminary shooting and the overhead point cloud image 36 from the main shooting are drawn in different drawing manners (e.g., colors). Specifically, for example, the overhead point cloud image 35 from the preliminary shooting is drawn in purple, and the overhead point cloud image 36 from the main shooting is drawn in black. Note that, if the captured color is reflected in the acquired point cloud, for example, the overhead point cloud image 35 from the preliminary shooting may be drawn in purple, and the overhead point cloud image 36 from the main shooting may be drawn in the acquired color. Also, each predetermined color may be blended (mixed) into the color of each acquired point cloud.

[0202] The map image 32 visualizes the progress of the actual photography. Specifically, as the actual photography progresses, the already-photographed range in the overhead point cloud image 35 from the preliminary photography is gradually erased and replaced with the overhead point cloud image 36 from the actual photography. Here, the overhead point cloud image 35 from the preliminary photography that remains without being erased represents the range that has not been photographed, and the overhead point cloud image 36 from the actual photography represents the range that has been photographed. Also, as the actual photography progresses, the range of the overhead point cloud image 35 from the preliminary photography gradually becomes smaller, and the range of the overhead point cloud image 36 from the actual photography gradually becomes larger. This allows the user to grasp the progress of the actual photography.

[0203] In addition, in the confirmation map 31, the areas that have been captured in the overhead point cloud image 35 from the preliminary shooting may not be erased, but may be drawn in a manner (e.g., a color) different from the areas that have not been captured in the overhead point cloud image 35 from the preliminary shooting or the overhead point cloud image 36 from the main shooting.

[0204] The example shown in Fig. 25 is a case where the display of the progress status is enabled, the display of the preliminary shooting results is enabled, the display of points to be careful about in shooting is disabled, and the display of the recommended route is disabled by operating the check boxes 615 to 618. In this example, the settings other than the progress status are the same as the example shown in Fig. 24.

[0205] In this case, an indicator image 621 that visualizes the progress of the actual shooting is displayed in the main window 102. The confirmation map 31 displayed in the sub-window 103 is similar to the example shown in FIG.

[0206] The indicator image 621 is a so-called progress bar, and is drawn so that the portion indicating the photographed range changes as the actual photographing progresses. That is, as the actual photographing progresses, the photographed portion is drawn gradually larger and the unphotographed portion is drawn gradually smaller. In addition, in the indicator image 621, the photographed portion and the unphotographed portion are drawn in different modes (e.g., colors or patterns). For example, the photographed portion is drawn in green, and the unphotographed portion is drawn in white.

[0207] The example shown in Fig. 26 is a case where the user has specified a target shooting range. The display settings for the progress status, preliminary shooting results, points to be careful about when shooting, and recommended route by operating check boxes 615 to 618 are the same as those in the example shown in Fig. 25.

[0208] In this case, an area image 61 representing the target shooting range designated by the user is drawn on the confirmation map 31 displayed on the sub window 103. The area image 61 is drawn in a predetermined color (e.g., yellow), for example, and is superimposed in a semi-transparent state on the overhead point cloud image 35 obtained by preliminary shooting, which represents the initial target shooting range. Note that the area image 61 only needs to be able to show the range, and for example, only a line showing the range may be drawn.

[0209] The example shown in Fig. 27 is a case where the display of the progress status is enabled, the display of the preliminary shooting results is enabled, the display of the points to be careful about when shooting is enabled, and the display of the recommended route is disabled by operating the check boxes 615 to 618. In this example, the settings other than the points to be careful about when shooting are the same as the example shown in Fig. 25. This example is also a case where a point cloud insufficient area is detected.

[0210] In this case, in the confirmation map 31 displayed in the subwindow 103, an area image 44 representing the point cloud insufficient portion is superimposed on the overhead point cloud images 35, 36 constituting the map image 32. The area image 44 representing the point cloud insufficient portion is drawn in a predetermined color (for example, red).

[0211] The example shown in Fig. 28 is a case where a strong reflection part is detected together with a point cloud shortage part. The display settings related to the progress status, preliminary shooting results, points to be careful about when shooting, and recommended route by operating the check boxes 615 to 618 are the same as the example shown in Fig. 27.

[0212] In this case, in the confirmation map 31 displayed in the sub-window 103, a mark image 41 indicating a strong reflection area and an area image 44 indicating an area where the point cloud is lacking are superimposed on the overhead point cloud images 35, 36 that constitute the map image 32.

[0213] Here, when the user performs an operation to select a strong reflection point on the confirmation map 31, specifically, a predetermined operation (tap operation, click operation) on the mark image 41 representing the strong reflection point, a message prompting the user to take measures to improve the shooting environment related to the strong reflection point is displayed in the message display window 631. In this example, the selected strong reflection point is a case where the reflected light is assumed to be caused by indoor lighting, and a message prompting the user to improve the shooting environment related to lighting, for example, a message saying "If the reflection is strong, please reduce the lighting at the shooting point", is displayed. In addition, when the reflected light is assumed to be caused by external light entering through a window, a message prompting the user to improve the shooting environment related to external light, for example, a message saying "If the curtains are open and external light is entering, please close the curtains", is displayed. Note that, when both the indoor lighting and the external light entering through the window are likely to be the cause, a message prompting the user to improve both, for example, a message saying "Please reduce the lighting at the shooting point and close the curtains", may be displayed.

[0214] The example shown in Fig. 29 is a case where a point cloud lacking part and a feature lacking part are detected. The display settings for the progress status, preliminary shooting results, points to be careful about in shooting, and recommended route by operating the check boxes 615 to 618 are the same as the examples shown in Figs. 27 and 28.

[0215] In this case, in the confirmation map 31 displayed in the sub-window 103, a mark image 42 indicating an area lacking features and an area image 44 indicating an area lacking point cloud are superimposed on the overhead point cloud images 35, 36 that constitute the map image 32.

[0216] Here, when the user performs an operation to select a feature-lacking portion on the confirmation map 31, specifically, a predetermined operation (tap operation, click operation) on the mark image 42 representing the feature-lacking portion, a message is displayed in the message display window 631 to prompt the user to take measures to improve the shooting environment or shooting method related to the feature-lacking portion. In this example, the selected feature-lacking portion is caused by the user's shooting method, and a message is displayed to prompt the user to improve the shooting method, for example, a message saying "When shooting a wall or floor, please set the angle of view to include other features." In addition, when the selected feature-lacking portion is caused by the shooting environment of the measurement target location, a message is displayed to prompt the user to improve the shooting environment, for example, a message saying "Please place something with texture."

[0217] The examples shown in Figs. 30 and 31 are when the display of all of the progress status, preliminary photography results, points to be careful about in photography, and recommended route is enabled by operating the check boxes 615 to 618. In this example, the settings other than the progress status are the same as the example shown in Fig. 24. This example is also a case where a strong reflection area, an area lacking features, an area of ​​sudden movement, and an area lacking point clouds have been detected. Fig. 30 shows the time immediately after the start of actual photography, and Fig. 31 shows the time when actual photography has progressed to a certain extent.

[0218] In this case, in the confirmation map 31 displayed in the subwindow 103, a route image 51 indicating the recommended route is superimposed on the overhead point cloud images 35, 36 constituting the map image 32. In addition, a mark image 41 indicating a strong reflection portion, a mark image 42 indicating a feature-lacking portion, a mark image 43 indicating a sudden movement portion, and an area image 44 indicating a point cloud-lacking portion are superimposed on the overhead point cloud images 35, 36 constituting the map image 32.

[0219] 30 and 31, a navigation icon 641 is superimposed on the captured image 111 displayed in the main window 102. The navigation icon 641 is an image representing the user's route, and is created based on the recommended route and the current position.

[0220] Here, if the shooting environment and shooting method are appropriately improved in the main shooting compared to the preliminary shooting, the mark images 41, 42, 43 and the area image 44 change as the main shooting progresses. Specifically, the area image 44 representing the point cloud insufficient area shrinks as the main shooting progresses. This allows the user to understand that the point cloud insufficient area has decreased. Also, in the already shot section, the mark images 41, 42, 43 representing the strong reflection area, the feature insufficient area, and the sudden movement area are erased. This allows the user to confirm that no event that reduces the accuracy of the point cloud data has occurred in the strong reflection area, the feature insufficient area, and the sudden movement area.

[0221] In addition, the route image 51 showing the recommended route changes as the actual photography progresses. Specifically, the photographed sections of the route image 51 are erased. This allows the user to grasp the progress of the actual photography. The photographed sections in the route image 51 may be drawn in a different drawing style from the unphotographed sections without being erased. For example, the photographed sections may be drawn in a lighter color than the unphotographed sections.

[0222] In addition, when the photographing device 1 is configured as a tablet terminal (see FIG. 1), it is difficult to operate the screen during photographing. Therefore, operations on the screen 601 in the actual photographing mode, specifically, operations on the check boxes 615 to 618 and operations on the points to be noted for photographing on the confirmation map 31, may be performed before photographing starts, and functions may be restricted so that screen operations cannot be performed once photographing starts. On the other hand, when the photographing device 1 is configured as a wearable device such as a head-mounted device equipped with a camera and a display, screen operations during photographing are easy, so that it may be possible to operate the screen in the actual photographing mode.

[0223] However, if the user is unfamiliar with the operation, it is expected that the user will start actual shooting without taking preliminary shooting. Therefore, when the user operates the tab 131 for the actual shooting mode before shooting in the preliminary shooting mode, a confirmation message regarding preliminary shooting, that is, a message for confirming whether or not preliminary shooting is possible and prompting the user to perform preliminary shooting, may be displayed. For example, a message may be displayed stating, "If you want to perform actual shooting without taking preliminary shooting, continue as is. If you want to perform preliminary shooting, select the preliminary shooting tab."

[0224] Also, there are cases where preliminary photography has already been performed at the same measurement target location. In this case, if the photography data and point cloud data (preliminary photography results) acquired in the previous preliminary photography can be used, the current preliminary photography can be omitted. Therefore, a screen operation for saving and reading the preliminary photography results may be made possible. In this way, if the preliminary photography results have been saved, the current preliminary photography can be omitted by reading out the preliminary photography results. In this way, for example, if the main photography fails after the preliminary photography has been performed, the main photography can be redone without performing the preliminary photography again.

[0225] In addition, if the user starts the actual shooting without taking a preliminary shot, the target shooting range cannot be set based on the preliminary shooting result, and therefore the progress status during the actual shooting cannot be presented to the user. Therefore, when the user tries to start the actual shooting without taking a preliminary shot, a confirmation message regarding the progress status may be displayed. For example, a message saying "Progress status is not displayed" may be displayed.

[0226] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0227] The photographing device, photographing work support method, and photographing work support program of the present invention have the effect of enabling a user to check in advance the locations where various phenomena that cause a decrease in the accuracy of point cloud data will occur when performing actual photographing, thereby enabling highly accurate measurement results to be obtained with a single actual photographing.They are useful as a photographing device that photographs a location to be measured, and a photographing work support method and photographing work support program that support the photographing work of a user using the photographing device, in order to perform 3D measurement processing that generates 3D spatial information of the location to be measured based on the photographing data of the location to be measured. [Explanation of symbols]

[0228] 1: Imaging device 11: Device body 12: Sensor section 13: Display (display unit) 14: Input devices 15: Memory 16: Processor 21: Visible camera 22: Depth camera 23: IMU 31: Confirmation map 32: Map image 35: Preliminary overhead point cloud image (first overhead point cloud image) 36: Overhead point cloud image taken during actual shooting (second overhead point cloud image) 41: Mark image showing strong reflection area 42: Mark image showing feature deficiency 43: Mark image showing sudden movement 44: Area image showing point cloud shortage areas 51: Route image showing the recommended route 61: Area image showing the target shooting range 101: Pre-shooting mode screen 102: Main window 103: Sub window 104: Operation window 111: Photographed image 131: "Main Shooting" tab 132: "Preliminary Shooting" tab 133: "Change Settings" tab 161: Message display window 201: Pre-shooting mode screen 211: Message display window 301, 401, 501: Setting change mode screen 311: "Progress Settings" tab 312: "Cautions for Shooting" tab 313: "Recommended Route Settings" tab 601: Main shooting mode screen 621: Indicator image 631: Message display window 641: Navigation icon

Claims

1. An imaging device that images a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on imaging data of the measurement target location, A device body held by a user; An imaging unit provided in the device body for imaging a measurement target location; a display unit that displays work support information related to an image capturing operation in which a user moves around a measurement target location and causes the image capturing unit to capture the measurement target location; A processor that controls the image capturing unit and the display unit, The processor, Based on the shooting data acquired in the preliminary shooting, areas that require attention in the actual shooting are detected, A confirmation map is generated on which the points to be photographed are drawn on a map image representing the measurement target location; A photographing device that displays a screen including the confirmation map on the display unit.

2. The processor, generating point cloud data as a preliminary photographing result based on the photographing data acquired in the preliminary photographing; A bird's-eye view point cloud image is generated based on the point cloud data.

2. The photographing device according to claim 1, wherein the confirmation map is generated by superimposing an image representing the points to be careful of photographing on the map image including the overhead point cloud image.

3. The processor, 2. The photographing apparatus according to claim 1, wherein the photographing mode is switched between a first photographing mode in which preliminary photographing is performed and a second photographing mode in which main photographing is performed in response to a user's operation to select a photographing mode.

4. The processor, The photographing device according to claim 1, characterized in that, during actual photographing, a screen including a real-time photographed image output from the photographing unit and the confirmation map on which the areas to be careful of when photographing are drawn is displayed on the display unit.

5. The processor, 2. The photographing apparatus according to claim 1, wherein the display unit displays a screen including the confirmation map on which the points to be careful of during photographing are depicted before the main photographing operation is performed after the preliminary photographing operation.

6. The processor, Using a simplified process compared to the actual shooting, simple point cloud data was generated as a preliminary shooting result.

2. The photographing apparatus according to claim 1, wherein the areas to be noted for photographing are detected based on the simplified point cloud data.

7. The processor, 2. The imaging device according to claim 1, wherein the confirmation map is generated in which at least one of strong reflection areas, areas lacking features, areas of sudden movement, and areas lacking point clouds is depicted as the areas to be careful of when imaging.

8. A photographing operation support method for performing a process by a processor to support a photographing operation of a user who holds a photographing device and moves within a measurement target location to photograph the measurement target location with the photographing device, in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on photographing data of the measurement target location, comprising: Based on the shooting data acquired in the preliminary shooting, areas that require attention in the actual shooting are detected, A confirmation map is generated on which the points to be photographed are drawn on a map image representing the measurement target location; A photographing operation supporting method, comprising: displaying a screen including the confirmation map on a display unit.

9. A photography operation support program that causes a processor to execute a process for supporting a photography operation of a user who holds a photography device and moves within a measurement target location to photograph the measurement target location with the photography device, in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on photography data of the measurement target location, the process comprising: Based on the shooting data acquired in the preliminary shooting, areas that require attention in the actual shooting are detected, A confirmation map is generated on which the points to be photographed are drawn on a map image representing the measurement target location; A photographing operation support program, comprising: a display unit that displays a screen including the confirmation map.

Images