Imaging device, imaging operation support method, and imaging operation support program

Abstract

To enable a user to proceed with imaging operation while grasping a progress situation of the imaging.SOLUTION: In an imaging device for imaging a measurement target location in order to perform 3D measurement processing for generating 3D spatial information of the measurement target location on the basis of imaging data of the measurement target location, work support information related to imaging work for imaging a measurement target location where a user performs imaging while moving is displayed on a display. In particular, a target imaging range for the actual imaging is set on the basis of imaging data acquired in preliminary imaging, an imaging-completed range is acquired on the basis of imaging data obtained by the actual imaging, information on a progress situation of the actual imaging is acquired on the basis of the imaging-completed range and the target imaging range, and a screen including the information on the progress situation of the actual imaging is displayed on the display.SELECTED DRAWING: Figure 31

Description

[Technical Field]

[0001] The present invention relates to an imaging device that captures images of a measurement target location in order to perform 3D measurement processing that generates 3D spatial information of the measurement target location based on imaging data of the measurement target location, as well as to a method and program for supporting a user's imaging work using the imaging device. [Background technology]

[0002] There is a known 3D measurement technology that generates 3D spatial information (map data) about a measurement target location based on photographic data of the measurement target location. In recent years, the SLAM (Simultaneous Localization and Mapping) method has been attracting attention for this type of 3D measurement. The SLAM method generates 3D spatial information and the location information of a mobile object as a measurement result based on photographic data acquired by carrying a photographic device on the mobile object and photographing the measurement target location. In particular, 3D measurement can be easily performed by a user acquiring photographic data of the measurement target location by using a handheld photographic device to photograph the measurement target location while moving around the measurement target location.

[0003] On the other hand, when a user photographs a measurement target location with a handheld camera while walking around the measurement target location, it is desirable to provide work support that encourages the user to perform appropriate photographing work so that highly accurate and stable 3D measurements can be performed.From this perspective, a technology has been known in the past that displays multiple marks, such as arrows, that indicate the direction and speed of movement of the photographing device (sensor) held by the user, to guide the user on how to proceed with the photographing work (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6489566 Summary of the Invention [Problem to be solved by the invention]

[0005] In photography work in which a user holds a photography device and photographs a measurement target location while moving around the measurement target location, it can be difficult to proceed with the photography work efficiently depending on the user's level of proficiency, the characteristics of the measurement target location, etc. In other words, various events that can reduce the accuracy of point cloud data can occur during photography, and it is necessary to proceed with the photography work while avoiding the occurrence of such events.

[0006] During such photography work, it is desirable for the user to be able to grasp the progress of photography in order to carry out the work appropriately. For example, if the user proceeds with the photography work too carefully and the photography work takes longer, the processing load increases, and measurement errors accumulate (cumulate), reducing the accuracy of the point cloud data. For this reason, a technology is desired that allows the user to proceed with the photography work while grasping the progress of photography. However, conventional technology does not take such a demand into consideration at all.

[0007] Therefore, a main object of the present invention is to provide a photography apparatus, a photography operation support method, and a photography operation support program that allow a user to proceed with photography operations while understanding the progress of photography. [Means for solving the problem]

[0008] The photographing device of the present invention is a photographing device that photographs a location to be measured in order to perform 3D measurement processing to generate 3D spatial information of the location to be measured based on photographing data of the location to be measured, and is equipped with a device main body held by a user, a photographing unit provided in the device main body that photographs the location to be measured, a display unit that displays work support information related to the photographing work in which the user causes the photographing unit to photograph the location to be measured while moving around the location to be measured, and a processor that controls the photographing unit and the display unit, and is configured so that the processor sets a target photographing range for the actual photographing based on the photographing data acquired in preliminary photographing, acquires the photographed range based on the photographing data acquired in actual photographing, acquires information regarding the progress of the actual photographing based on the photographed range and the target photographing range, and displays a screen including information regarding the progress of the actual photographing on the display unit.

[0009] In addition, the photography support method of the present invention is a photography support method in which a processor performs processing to support the photography work of a user who holds a photography device and moves around the measurement target location while having the photography device photograph the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photography data of the measurement target location.The method is configured to set a target photography range for the actual photography based on the photography data acquired in preliminary photography, obtain a photographed range based on the photography data acquired in main photography, obtain information regarding the progress of the main photography based on the photographed range and the target photography range, and display a screen including information regarding the progress of the main photography on a display unit.

[0010] In addition, the photography work support program of the present invention is a photography work support program that causes a processor to execute a process to support the photography work of a user who holds a photography device and moves around the measurement target location while having the photography device photograph the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photography data of the measurement target location.The photography work support program is configured to set a target photography range for the actual photography based on the photography data acquired in preliminary photography, obtain a photographed range based on the photography data acquired in actual photography, obtain information regarding the progress of the actual photography based on the photographed range and the target photography range, and display a screen including information regarding the progress of the actual photography on a display unit. [Effects of the Invention]

[0011] According to the present invention, information relating to the progress of the actual photography is presented to the user, so that the user can proceed with the photography work while understanding the progress of the actual photography. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram showing a situation of a photographing operation using the photographing device according to the present embodiment; [Figure 2] An explanatory diagram showing an example of a confirmation map that includes points to be careful about when taking photos. [Figure 3] An explanatory diagram showing an example of a confirmation map including a recommended route. [Figure 4] An explanatory diagram showing the photographing situation of a strongly reflective area by a photographing device. [Figure 5] A block diagram showing the schematic configuration of an imaging device. [Figure 6] Block diagram showing the overall processing performed by the processor [Figure 7] FIG. 10 is an explanatory diagram showing an overlapping region used in the point cloud density evaluation process and the feature point evaluation process. [Figure 8] Flow diagram showing the processing steps performed by the processor during shooting [Figure 9] 1 is a flowchart showing the processing steps performed by the processor during preliminary photography. [Figure 10]Flow diagram showing the processing steps performed by the processor during actual imaging [Figure 11] A flowchart showing the procedure of processing performed when the processor finishes preliminary photography. [Figure 12] FIG. 10 is an explanatory diagram showing a screen displayed before shooting in the preliminary shooting mode; [Figure 13] FIG. 10 is an explanatory diagram showing a screen displayed during shooting in the preliminary shooting mode. [Figure 14] FIG. 10 is an explanatory diagram showing a screen displayed during shooting in the preliminary shooting mode. [Figure 15] FIG. 10 is an explanatory diagram showing a screen displayed immediately after shooting in the preliminary shooting mode. [Figure 16] FIG. 10 is an explanatory diagram showing a screen displayed when confirming a photograph taken in the preliminary photographing mode; [Figure 17] FIG. 10 is an explanatory diagram showing a screen displayed when confirming a photograph taken in the preliminary photographing mode; [Figure 18] FIG. 10 is an explanatory diagram showing a screen displayed when confirming a photograph taken in the preliminary photographing mode; [Figure 19] FIG. 10 is an explanatory diagram showing a screen displayed when confirming a photograph taken in the preliminary photographing mode; [Figure 20] FIG. 10 is an explanatory diagram showing a screen related to progress setting displayed in the setting change mode; [Figure 21] FIG. 10 is an explanatory diagram showing a screen for setting the display of points to be careful about when taking photographs, which is displayed in the setting change mode; [Figure 22] FIG. 10 is an explanatory diagram showing a screen related to recommended route settings displayed in the setting change mode; [Figure 23] FIG. 10 is an explanatory diagram showing a screen related to recommended route settings displayed in the setting change mode; [Figure 24] FIG. 10 is an explanatory diagram showing a screen displayed in the actual shooting mode; [Figure 25] FIG. 10 is an explanatory diagram showing a screen displayed in the actual shooting mode; [Figure 26] FIG. 10 is an explanatory diagram showing a screen displayed in the actual shooting mode; [Figure 27] FIG. 10 is an explanatory diagram showing a screen displayed in the actual shooting mode; [Figure 28]FIG. 10 is an explanatory diagram showing a screen displayed in the actual shooting mode; [Figure 29] FIG. 10 is an explanatory diagram showing a screen displayed in the actual shooting mode; [Figure 30] FIG. 10 is an explanatory diagram showing a screen displayed in the actual shooting mode; [Figure 31] FIG. 10 is an explanatory diagram showing a screen displayed in the actual shooting mode; DETAILED DESCRIPTION OF THE INVENTION

[0013] The first invention made to solve the above problem is a photography device that photographs a measurement target location in order to perform 3D measurement processing that generates 3D spatial information of the measurement target location based on photography data of the measurement target location, and includes a device main body held by a user, a photography unit provided in the device main body that photographs the measurement target location, a display unit that displays work support information related to the photography work in which the user moves around the measurement target location and has the photography unit photograph the measurement target location, and a processor that controls the photography unit and the display unit, wherein the processor sets a target photography range for the actual photography based on the photography data acquired in preliminary photography, acquires the photographed range based on the photography data acquired in actual photography, acquires information about the progress of the actual photography based on the photographed range and the target photography range, and displays a screen including information about the progress of the actual photography on the display unit.

[0014] This allows the user to be presented with information about the progress of the actual photography, allowing the user to proceed with the photography work while understanding the progress of the actual photography.

[0015] In addition, in a second invention, the processor is configured to switch between a first shooting mode in which preliminary shooting is performed and a second shooting mode in which main shooting is performed in response to a user operation to select a shooting mode.

[0016] This allows the user to select and execute the preliminary photography and the main photography as appropriate. For example, the user can skip the preliminary photography and execute the main photography as appropriate.

[0017] In addition, a third invention is configured such that, during the actual shooting, the processor causes the display unit to display a screen including a real-time captured image output from the shooting unit and an image visualizing the progress of the actual shooting.

[0018] This allows the user to check the shooting status based on the captured image during the actual shooting, and to proceed with the shooting work while checking the progress of the actual shooting based on an image that visualizes the progress.

[0019] In addition, in a fourth invention, the processor generates a map image representing the location to be measured as an image visualizing the progress of the actual photography, and the map image includes a first overhead point cloud image representing the target photography range generated by preliminary photography and a second overhead point cloud image representing the photographed range generated by the actual photography, and the first overhead point cloud image and the second overhead point cloud image are drawn in different ways.

[0020] This allows the progress to be expressed in terms of the ratio of the first and second overhead point cloud images on the map image showing the location to be measured, so that the user can distinguish between the areas that have been photographed and the areas that have not been photographed in the actual photography, and can also grasp the progress of the actual photography.

[0021] In addition, a fifth invention is configured such that the processor generates an indicator image as an image visualizing the progress of the main photography, in which the part indicating the photographed range changes as the main photography progresses.

[0022] This allows the user to easily grasp the progress of the actual photography.

[0023] In addition, in a sixth aspect of the present invention, the processor sets an initial target shooting range based on point cloud data generated by preliminary shooting, sets a user-specified target shooting range in response to a user operation that changes the range from the initial target shooting range, and acquires information regarding the progress of the main shooting based on the user-specified target shooting range.

[0024] This allows the user to specify a range different from the initial target shooting range as a new target shooting range.

[0025] Furthermore, a seventh invention is a method for supporting photography operations in which a processor performs processing to support a user's photography operation by having the user hold a photography device and move around the measurement target location while having the photography device photograph the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photography data of the measurement target location. The method is configured to set a target photography range for the actual photography based on the photography data acquired in preliminary photography, obtain a photographed range based on the photography data acquired in the actual photography, obtain information regarding the progress of the actual photography based on the photographed range and the target photography range, and display a screen including information regarding the progress of the actual photography on a display unit.

[0026] According to this, as in the first aspect of the invention, information relating to the progress of the actual photography is presented to the user, so that the user can proceed with the photography work while understanding the progress of the actual photography.

[0027] Furthermore, an eighth aspect of the present invention is a photography support program that causes a processor to execute a process to support a user's photography work by having the photography device photograph the measurement target location while holding the photography device and moving within the measurement target location, in order to perform 3D measurement processing that generates 3D spatial information of the measurement target location based on photography data of the measurement target location. The program is configured to set a target photography range for the actual photography based on the photography data acquired in preliminary photography, obtain a photographed range based on the photography data acquired in actual photography, obtain information regarding the progress of the actual photography based on the photographed range and the target photography range, and display a screen including information regarding the progress of the actual photography on a display unit.

[0028] According to this, as in the first aspect of the invention, information relating to the progress of the actual photography is presented to the user, so that the user can proceed with the photography work while understanding the progress of the actual photography.

[0029] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0030] FIG. 1 is an explanatory diagram showing a situation in which a user performs a photographing operation using a photographing device 1 according to this embodiment.

[0031] The photographing device 1 includes a device main body 11 and a sensor unit 12. The sensor unit 12 includes a visible light camera 21 (photographing unit). The visible light camera 21 is a monocular camera that detects visible light to photograph a subject, and outputs a photographed image, for example, an RGB color image. The photographing device 1 can be configured as a tablet terminal or a notebook PC.

[0032] A user (worker) walks through a location to be measured while holding the device body 11 of the image capture device 1. At this time, the location to be measured is captured by the visible camera 21 of the image capture device 1, and image capture data is sequentially acquired for each point in the location to be measured. The image capture data includes data such as the captured image, the sensor detection results, and the time of capture.

[0033] In this embodiment, the photographing device 1 is a three-dimensional measuring device. That is, the photographing device 1 performs three-dimensional measurement processing based on photographed images of each point sequentially acquired by the visible camera 21, and generates three-dimensional spatial information about the measurement target location. In the three-dimensional measurement processing, point cloud data (environmental map) is generated as three-dimensional spatial information about the measurement target location using the SLAM method.

[0034] 1 shows an example in which a user holding the device main body 11 walks through the measurement target location to photograph the measurement target location, but the measurement target location may also be photographed using a self-propelled robot or the like equipped with the photographing device 1. In this embodiment, work assistance is provided to the user, and when the user himself holds the device main body 11 and performs the photographing work, work assistance is provided to the user. On the other hand, when a self-propelled robot photographs, work assistance is provided to the user operating the robot.

[0035] In addition, in this embodiment, the photographing device 1 equipped with a sensor unit 12 such as a visible camera 21 performs the 3D measurement processing, but the sensor unit 12 may be configured as a photographing device 1 independent of a PC (information processing device) that performs the 3D measurement processing.

[0036] The user also photographs the measurement target location while walking through the measurement target location with the handheld camera 1. Therefore, the photographing operation includes actions such as walking (moving) while holding the camera 1 and pointing the camera 1 in an appropriate direction.

[0037] Next, we will explain the points to be careful about when photographing and the confirmation map 31. Fig. 2 is an explanatory diagram showing an example of the confirmation map 31 including the points to be careful about when photographing. Fig. 3 is an explanatory diagram showing an example of the confirmation map 31 including the recommended route. Fig. 4 is an explanatory diagram showing the photographing situation by the photographing device 1 in a strongly reflective point.

[0038] In photography work in which a user holds the photography device 1 and photographs a measurement target location while moving around the measurement target location, it may be difficult to obtain highly accurate measurement results with just one photograph, depending on the user's level of proficiency and the characteristics of the measurement target location. In other words, various events that can reduce the accuracy of the point cloud data may occur during photography, and a high level of skill is required to proceed with the photography work while avoiding such events. For this reason, inexperienced users will have to repeat the photography work many times through trial and error.

[0039] Therefore, in this embodiment, preliminary photography is performed in addition to the main photography (regular photography) for acquiring three-dimensional spatial information (point cloud data) as a measurement result, and based on the results of the preliminary photography, areas to be careful about in the main photography, i.e., areas that require the user's attention in the main photography, are detected and presented to the user. This makes it possible to improve the accuracy of the main photography results, i.e., the point cloud data acquired in the main photography.

[0040] Areas requiring caution when photographing include areas with strong reflections, areas lacking features, areas with sudden movements, and areas with insufficient point clouds. Areas requiring caution when photographing also include areas where an event that could cause a decrease in the accuracy of the point cloud data actually occurred during preliminary photography, and areas where an event that could cause a decrease in the accuracy of the point cloud data is likely to occur.

[0041] Here, for example, assume that lighting is installed on the ceiling and the floor is made of a reflective material (see FIG. 4). In this case, the lighting light is reflected by the floor, and this reflected light enters the image capture device 1, causing the reflected light from the floor to appear in the captured image. At this time, the reflected light from the floor is extracted as a feature point, and position estimation is performed based on this feature point. However, as the image capture device 1 moves, the position of the reflected light from the floor also moves. As a result, position estimation based on the feature points cannot be performed appropriately, and the accuracy of the point cloud data as a measurement result is significantly reduced.

[0042] Therefore, in this embodiment, a location where strong reflected light is estimated to have appeared in an image captured by the visible light camera 21 or a location where strong reflected light actually appeared in an image captured in preliminary shooting is detected as a strong reflection location and presented to the user. Furthermore, guidance (such as suggestions for improvement) regarding the strong reflection location is provided to the user. Specifically, if the reflected light is estimated to be caused by indoor lighting, the user is prompted to improve the shooting environment regarding lighting. For example, the user is prompted to dim the lighting. Furthermore, if the reflected light is estimated to be caused by external light entering through a window, the user is prompted to improve the shooting environment regarding external light. For example, the user is prompted to close the curtains. Note that if the reflected light is likely to be caused by both indoor lighting and external light entering through a window, the user may also be prompted to dim the lighting or close the curtains. This allows the user to appropriately improve the shooting environment regarding the strong reflection location.

[0043] Furthermore, if an area containing many areas without distinctive patterns or objects, such as walls, floors, or ceilings, is photographed, feature point extraction will fail and fewer feature points will be extracted, preventing proper self-position estimation and reducing the accuracy of the point cloud data.

[0044] Therefore, in this embodiment, areas where feature points could not be sufficiently extracted from the image captured by the visible camera 21 are detected as feature-deficient areas and presented to the user. Furthermore, guidance regarding feature-deficient areas (such as suggestions for improvement) is provided to the user. For example, if the feature-deficient areas are caused by the shooting environment of the measurement target location, the user is prompted to improve the shooting environment. For example, the user is prompted to place something with texture (e.g., distinctive wallpaper). Furthermore, if the feature-deficient areas are caused by the user's shooting method, the user is prompted to improve the shooting method. For example, the user is prompted to shoot with an angle of view that includes distinctive patterns or objects.

[0045] Furthermore, if the user moves the image capturing device 1 too quickly while capturing an image, errors will occur in the position estimation and the point cloud will not be generated appropriately, resulting in a decrease in the accuracy of the point cloud data.

[0046] Therefore, in this embodiment, a location where the user actually made a sudden movement during preliminary shooting or a location where it is estimated that the user is likely to make a sudden movement is detected as a sudden movement location and presented to the user. For example, a location where a sudden acceleration / deceleration state in which the acceleration generated in the image capture device 1 falls outside a predetermined range or a sudden rotation state in which the angular velocity generated in the image capture device 1 falls outside a predetermined range is detected as a sudden movement location and presented to the user. Furthermore, a location where the space for the user to pass through is narrow or where the route the user takes makes a sudden turn is detected as a sudden movement location and presented to the user because it is likely that the user will make a sudden movement. Furthermore, guidance (such as suggestions for improvement) regarding the sudden movement location is presented to the user. For example, the user is urged not to make a sudden movement.

[0047] Furthermore, areas hidden in the shadow of objects, such as the back of a desk monitor, will not be photographed, and areas with insufficient photography will not generate a point cloud with sufficient density, resulting in a decrease in the accuracy of the point cloud data.Furthermore, if the user does not move the photographing device 1 appropriately while photographing, the point cloud will not be generated with sufficient density, resulting in a decrease in the accuracy of the point cloud data.

[0048] Therefore, in this embodiment, areas where a point cloud was not generated because they were not photographed in the preliminary photography (unphotographed areas), or areas where a point cloud was not generated with sufficient density despite being photographed, are detected as areas with insufficient point clouds and presented to the user.

[0049] 2 and 3, in this embodiment, a confirmation map 31 is generated and presented to the user to allow the user to check areas requiring careful shooting before or during actual shooting. In confirmation map 31, the results of preliminary shooting, i.e., an overhead point cloud image 35 generated from point cloud data obtained by preliminary shooting, is used as map image 32 representing the location to be measured, and an image representing areas requiring careful shooting is superimposed on map image 32. Note that overhead point cloud image 35 is an image (rendering) of each point of the point cloud data as viewed from an aerial viewpoint.

[0050] In the example shown in FIG. 2(A), strong reflection areas, feature-lacking areas, sudden movement areas, and point cloud-lacking areas are detected as areas requiring caution when photographing, and mark images 41, 42, and 43 representing the strong reflection areas, feature-lacking areas, and sudden movement areas, and an area image 44 representing an area requiring caution when photographing, are drawn on the confirmation map 31. The mark images 41, 42, and 43 are drawn in different shapes depending on their type (strong reflection areas, feature-lacking areas, and sudden movement areas), but they may also be drawn in the same shape with a different drawing aspect (e.g., color) other than the shape. The area image 44 is drawn in a predetermined color (e.g., red). The area image 44 may also be drawn in a pattern (design) or a semi-transparent color. Note that the mark images 41, 42, and 43 and the area image 44 are not limited to mark images and area images, respectively. For example, the mark images 41, 42, and 43 may be drawn in a predetermined color different from the area image 44 as area images indicating areas requiring caution when photographing, or may be drawn in a pattern (design) or a semi-transparent color.

[0051] In this example, areas that were not captured because they were hidden by the shadow of an object, such as the back of a monitor on a desk, are detected as areas that lack a point cloud. Additionally, the area on the periphery of the measurement target location is detected as an area that lacks a point cloud. This area appears in the captured image, but because the point cloud density is low, it is detected as an area that lacks a point cloud.

[0052] In the example shown in Fig. 2(B), a point cloud insufficient area is detected as an area requiring careful photography, and an area image 44 representing the point cloud insufficient area is drawn on the confirmation map 31. In this example, similar to the example shown in Fig. 2(A), an area on the periphery of the measurement target location is detected as a point cloud insufficient area.

[0053] Here, as in the example shown in Figure 2(A), in many cases the user does not need to pay particular attention to the point cloud insufficient areas on the periphery of the measurement target location when taking the actual photograph. On the other hand, as in the example shown in Figure 2(B), when the point cloud insufficient areas extend from the periphery of the measurement target location toward the inside, the user needs to pay particular attention when taking the actual photograph.

[0054] In this embodiment, strong reflection areas, areas lacking features, areas with sudden movements, and areas lacking point clouds are detected and presented to the user as areas requiring attention in photography, but in addition, blurring areas, i.e., areas estimated to be highly likely to experience blurring (out-of-focus) in the image captured by the visible camera 21, or areas where blurring actually occurred in the captured image during preliminary photography, may also be detected and presented to the user. If blurring occurs in the captured image, point cloud generation will not be performed appropriately, but by presenting the areas where blurring occurs to the user before actual photography, it is possible to prevent blurring from occurring in the captured image.

[0055] In this case, an evaluation value (blur evaluation value) that indicates the occurrence of blur in the captured image is calculated, and the evaluation value is compared with a predetermined threshold to determine the location where blur has occurred. The blur evaluation value is calculated by image analysis of the captured image. The blur evaluation value is also calculated based on the exposure time of the visible light camera 21 and the detection data (acceleration, angular velocity) of the IMU 23. The blur evaluation value is also calculated based on the shooting environment, such as brightness.

[0056] In this embodiment, a target imaging range to be the target of actual imaging is set as shown in Fig. 3. The target imaging range is set based on the preliminary imaging results, i.e., the point cloud data acquired in the preliminary imaging. For example, the range in which a point cloud is generated in the preliminary imaging is set as the target imaging range. At this time, if there is an unphotographed area (area in which a point cloud is not generated) surrounded by an imaged area (area in which a point cloud is generated), the unphotographed area may be included in the target imaging range as an unphotographed area.

[0057] In this embodiment, a recommended route is presented to the user to encourage them to take during actual photography. The recommended route is generated based on the target photography range. Specifically, a route that covers the entire target photography range and has as short a travel distance as possible is generated as the recommended route.

[0058] In the confirmation map 31, a route image 51 representing the recommended route is superimposed on a map image 32 consisting of an overhead point cloud image 35. A line representing the recommended route, an arrow representing the direction of travel, a mark representing the start position, a mark representing the current position, and a mark representing the end position are drawn on the route image 51 representing the recommended route.

[0059] Here, when a point requiring attention for photography is presented to the user, the user can be prompted to improve the photography environment or photography method for the point requiring attention, allowing the user to take appropriate measures to improve the photography environment or photography method, thereby improving the accuracy of the measurement results (point cloud). However, there are cases where it is difficult to improve the photography environment or photography method for the point requiring attention for photography.

[0060] In this case, in this embodiment, a recommended route is generated in accordance with a user's instruction to avoid the location where shooting is required. This makes it impossible to acquire a point cloud generated by shooting at the location where shooting is required, but it is possible to avoid a decrease in the accuracy of the measurement results (point cloud) caused by passing through the location where shooting is required without improving the shooting environment or shooting method.

[0061] The example shown in Figure 3(A) is a case where areas requiring careful shooting are not avoided, and a route that covers the entire target shooting range and has the shortest possible travel distance is generated as the recommended route, regardless of areas that lack features. On the other hand, the example shown in Figure 3(B) is a case where areas with strong reflections are avoided, and a recommended route that avoids areas with strong reflections is generated.

[0062] Next, we will explain the general configuration of the photographing device 1. Fig. 5 is a block diagram showing the general configuration of the photographing device 1. Fig. 6 is a block diagram showing an overview of the processing performed by the processor 16. Fig. 7 is an explanatory diagram showing the overlapping region used in the feature point evaluation process P22 and the point cloud density evaluation process P24.

[0063] As shown in FIG. 5, the photographing device 1 includes, in addition to the sensor unit 12, a display 13 (display unit), an input device 14, a memory 15 (storage unit), and a processor 16 (CPU).

[0064] In addition to the visible camera 21, the sensor unit 12 also includes a depth camera 22 and an IMU 23 (Inertial Measurement Unit).

[0065] The depth camera 22 is a stereo camera that detects infrared light to capture an image of a subject, and outputs depth information (distance image) as the detection result. The distance to the subject can be measured based on the detection result of the depth camera 22. Note that the depth camera 22 may be replaced by a sensor capable of acquiring depth information using other methods, such as LiDAR, instead of a stereo camera.

[0066] The IMU 23 (detector) detects the motion state of the image capturing device 1, specifically, three-dimensional angular velocity and acceleration. Based on the detection results of the IMU 23, the movement amount and rotation amount of the image capturing device 1 are measured.

[0067] The visible camera 21, the depth camera 22, and the IMU 23 may not be integrated into a sensor unit. Alternatively, the depth camera 22 and the IMU 23 may be omitted, and only the visible camera 21 may be provided. Alternatively, the visible camera 21 may be provided in addition to either the depth camera 22 or the IMU 23.

[0068] The display 13 presents various information related to the shooting operation to the user, and displays screens 101, 201 for the preliminary shooting mode (see Figures 12, 16, etc.), screens 301, 401, 501 for the setting change mode (see Figures 20, 21, 22, etc.), and screen 601 for the main shooting mode (see Figure 24, etc.).

[0069] The input device 14 is used by the user to perform input operations. The input device 14 may be a keyboard, a mouse, a touchpad, a touch panel, etc. If the photographing device 1 is configured as a tablet terminal, a touch panel display is provided in which the touch panel as the input device 14 and the display panel as the display 13 are integrated.

[0070] The memory 15 stores programs and the like to be executed by the processor 16. The memory 15 also stores images captured by the visible camera 21, distance information from the depth camera 22, and detection data from the IMU 23. The memory 15 also stores measurement results (point cloud data) generated by the processor 16.

[0071] The processor 16 performs various processes by executing programs stored in the memory 15. In this embodiment, the processor 16 performs a three-dimensional measurement process P1, a preliminary photography evaluation process P2, a process P3 for detecting areas to be careful about when photographing, a support information generation process P4, a display information generation process P5, and a display process P6.

[0072] In this embodiment, the photographing device 1 performs the 3D measurement process P1, the preliminary photographing evaluation process P2, the photographing point detection process P3, the support information generation process P4, and the display information generation process P5, but all or part of these processes may be performed by a server device (not shown) that can communicate with the photographing device 1.

[0073] As shown in FIG. 6, the three-dimensional measurement process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0074] In the feature extraction process P11, the processor 16 extracts feature information (feature points, etc.) from the image (frame) captured by the visible camera 21.

[0075] In tracking process P12, processor 16 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition related to the position and orientation of the image capture device 1, and updates the position and orientation trajectory data based on the amount of transition. Note that the position and orientation trajectory data is obtained by tracking the position and orientation of the image capture device 1, and includes the tracking results of the position and orientation at the time of capturing each captured image (frame), i.e., information regarding the position and orientation of the image capture device 1 at each time of capturing.

[0076] In the position and orientation correction process P13, the processor 16 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, the position and orientation trajectory data is corrected so as to compensate for the measurement results of locations with few features, such as walls and ceilings.

[0077] In the point cloud generation process P14, the processor 16 generates point cloud data based on the distance information of the depth camera 22 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13.

[0078] Here, during preliminary photography, processing is simplified compared to main photography, and simple point cloud data is generated as the preliminary photography result. This reduces the user's waiting time when the point cloud data as the preliminary photography result is presented to the user after the preliminary photography is completed. In addition, since the processing load on the processor 16 is reduced, battery consumption is reduced and photography can be continued for a long time. Note that in a configuration in which the point cloud generation process P14 and the like are performed on a server device separate from the photography device 1, the amount of data communicated is reduced, thereby reducing the communication load.

[0079] Furthermore, the point cloud generation process P14 performed during preliminary photography may be even simpler depending on the purpose of the preliminary photography. For example, if the point cloud data generated during preliminary photography is used only to acquire the target photography range and not to detect areas of interest for photography or to generate a recommended route based on the point cloud data, a point cloud sufficient to grasp the general positional relationships of objects is sufficient, and therefore even simpler processing may be used. In this case, for example, the photography data may be thinned out, i.e., the frame rate may be artificially lowered, thereby further reducing the load on the processor 16.

[0080] Furthermore, the point cloud generation process P14 performed during preliminary photography reduces the load on the processor 16 if only a small number of point clouds need to be generated, and therefore the working time for preliminary photography can be shortened by making the movement speed of the photography device 1 faster than that during main photography.

[0081] The preliminary photography evaluation process P2 includes a reflection evaluation process P21, a feature point evaluation process P22, a sudden movement evaluation process P23, and a point cloud density evaluation process P24.

[0082] In the reflection evaluation process P21, the processor 16 acquires an evaluation value (reflection evaluation value) representing the intensity of the reflected light appearing in the captured image based on the detection data of the IMU 23 and the transition amount regarding the position and posture of the image capturing device 1 acquired in the tracking process P12.

[0083] Specifically, the processor 16 compares the amount of movement of the image capture device 1 based on the detection data of the IMU 23 with the amount of movement (transition amount) of the image capture device 1 acquired in the tracking process P12, and acquires a reflection evaluation value that indicates the degree of inconsistency between the two. Here, the amount of movement of the image capture device 1 based on the detection data of the IMU 23 is determined in the position and orientation correction process P13. Furthermore, in the tracking process P12, the amount of movement (transition amount) of the image capture device 1 is determined by comparing feature points extracted from the current captured image with feature points extracted from a previously captured image. In this case, the reflection evaluation value indicates the degree of abnormality in the position estimation based on the captured image, i.e., the magnitude of the error occurring in the position estimation based on the feature points extracted from the captured image, and indirectly indicates the appearance of strong reflected light in the captured image.

[0084] In another embodiment, the processor 16 acquires a reflectance evaluation value that represents the appearance of reflected light in the captured image. That is, the size of the area where the brightness value is locally high in the captured image, specifically, the number of pixels whose brightness value deviation compared to the surrounding pixels is equal to or exceeds a predetermined threshold, is acquired as the reflectance evaluation value. In this case, the reflectance evaluation value directly represents the appearance of strong reflected light in the captured image. Note that the reflectance evaluation value in this case can also be used as a criterion for detecting shine or blown-out highlights in the captured image.

[0085] In the feature point evaluation process P22, the processor 16 counts the feature points within the target range based on the extraction result of the feature extraction process P11, and obtains an evaluation value (number of feature points) that indicates the extraction status of the feature points.

[0086] The number of feature points may be the number of feature points extracted in one captured image (one frame), or may be the number of feature points extracted in the overlapping area (see FIG. 7) between the capture range at the current capture time t and the capture range at the previous capture time t-1.

[0087] In the sudden movement evaluation process P23, the processor 16 acquires an evaluation value (sudden movement evaluation value) that indicates the degree of suddenness of the movement of the user moving the image capturing device 1, based on the detection data (acceleration, angular velocity) of the IMU 23. The degree of suddenness of the movement indicates the degree of suddenness of the change in the position and orientation of the image capturing device 1, and specifically indicates the degree of acceleration (including deceleration) that has occurred in the image capturing device 1, or the degree of angular velocity that has occurred in the image capturing device 1.

[0088] In another embodiment, in the sudden movement evaluation process P23, the processor 16 acquires a sudden movement evaluation value based on the amount of transition related to the position and orientation of the image capturing device 1 acquired in the tracking process P12. Here, the amount of transition represents the change in the position and orientation of the image capturing device 1, and the suddenness of the movement can be evaluated based on the magnitude of the amount of transition.

[0089] In another aspect, in the sudden movement evaluation process P23, the processor 16 acquires the movement route (traffic line) of the user (photography device 1) during preliminary photography, and acquires a sudden movement evaluation value as an evaluation value representing the likelihood that the user will make a sudden movement based on the movement route and the positioning status of surrounding objects estimated from the generated point cloud.

[0090] In the point cloud density evaluation process P24, the processor 16 counts the number of point clouds within the target area based on the point cloud data generated in the point cloud generation process P14, and obtains an evaluation value (point cloud density evaluation value) that indicates the state of point cloud generation. The point cloud density evaluation value indicates whether real-time point cloud generation is being performed appropriately. The smaller the point cloud density evaluation value, the lower the point cloud density.

[0091] The point cloud density evaluation value may be calculated from the number of point clouds generated in a predetermined unit time (for example, one second) (total number of points generated in the unit time) and the size of the shooting range of the visible light camera 21, as shown in the following formula. In this case, if the number of point clouds generated in the entire shooting range of the visible light camera 21 is small, the point cloud density evaluation value will be small. Point cloud density evaluation value = (number of point clouds generated per unit time) / (shooting range)

[0092] The point cloud density evaluation value may also be the number of generated points (the total number of points included in the overlapping area) in the overlapping area (see FIG. 7) between the shooting range at the current shooting time t and the shooting range at the previous shooting time t-1. In this case, too, the smaller the point cloud density evaluation value, the lower the point cloud density. In addition to the fact that a small number of point clouds are generated in the shooting range of the visible camera 21, the point cloud density evaluation value also decreases when the overlapping area between the shooting range at the current shooting time t and the shooting range at the previous shooting time t-1 is narrow. Note that the point cloud density may increase as the same location is repeatedly photographed. For this reason, the point cloud density evaluation value may also be calculated taking into account the existing point cloud generation status at that location. In this case, the number of generated points is not only the point cloud generated at the current shooting time t, but also the total number of all point clouds included in the overlapping area.

[0093] In another aspect, in the point cloud density evaluation process P24, the processor 16 calculates normal vectors perpendicular to the surface of the object by surface approximation or meshing of the point cloud data, and detects areas where the normal vectors are not continuous as areas where the point cloud is lacking.

[0094] In another aspect, in the point cloud density evaluation process P24, the processor 16 calculates the density for each position of the point cloud data, and detects areas where a point cloud exists but the density of the point cloud is extremely low (for example, when the density is within an interval between two predetermined thresholds) as areas where the point cloud is lacking.

[0095] The processing P3 for detecting areas requiring careful photography includes processing P31 for detecting strong reflection areas, processing P32 for detecting areas lacking features, processing P33 for detecting areas lacking sudden movements, and processing P34 for detecting areas lacking points.

[0096] In the strong reflection area detection process P31, the processor 16 detects strong reflection areas, i.e., areas where significant reflected light appears in the image captured by the visible light camera 21, based on the reflection evaluation value acquired in the reflection evaluation process P21. Specifically, the reflection evaluation value is defined so that the larger the value, the stronger the reflection, and an area is determined to be a strong reflection area when the reflection evaluation value is greater than a predetermined threshold. When a strong reflection area is detected, the position of the image capture device 1 at that time is stored in the memory 15 as the position of the strong reflection area. The strong reflection area detection process P31 is performed during preliminary image capture.

[0097] In the feature-lacking portion detection process P32, the processor 16 detects feature-lacking portions, i.e., portions where the feature points necessary to generate point cloud data with sufficient accuracy could not be extracted sufficiently, based on the number of feature points acquired in the feature point evaluation process P22. Specifically, a portion is determined to be a feature-lacking portion when the number of feature points is less than a predetermined threshold. When a feature-lacking portion is detected, the position of the image capture device 1 at that time is stored in the memory 15 as the position of the feature-lacking portion. At this time, because the reliability of the position estimation result based on the feature points is low, the position estimation result based on the detection data of the IMU 23 is adopted. Note that the feature-lacking portion detection process P32 is performed during preliminary image capture.

[0098] In the sudden movement location detection process P33, the processor 16 detects a sudden movement location, i.e., a location where the user actually made a sudden movement during preliminary shooting or a location where it is estimated that the user is likely to make a sudden movement, based on the sudden movement evaluation value acquired in the sudden movement evaluation process P23. Specifically, the sudden movement evaluation value is defined so that the larger the value, the suddener the movement, and a sudden movement location is determined to be a sudden movement location when the sudden movement evaluation value is greater than a predetermined threshold. When a sudden movement location is detected using a method in which the sudden movement evaluation value is acquired based on detection data from the IMU 23 or the amount of transition acquired in the tracking process P12, the position of the image capture device 1 at that time is stored in the memory 15 as the location of the sudden movement location. On the other hand, in a method in which the sudden movement evaluation value is acquired based on the movement route (traffic line) of the image capture device 1 during preliminary shooting, a location determined to be a sudden movement location is stored in the memory 15 as the location of the sudden movement location based on the movement route and the location of surrounding objects estimated from the generated point cloud.

[0099] In a method in which the sudden movement evaluation value is acquired based on the detection data of the IMU 23 or the transition amount acquired in the tracking process P12, the sudden movement location detection process P33 is performed during the preliminary shooting. On the other hand, in a method in which the sudden movement evaluation value is acquired based on the movement route (traffic line) of the image capturing device 1 during the preliminary shooting, the sudden movement location detection process P33 is performed when the preliminary shooting is completed.

[0100] In the point cloud insufficient portion detection process P34, the processor 16 detects a point cloud insufficient portion, i.e., a portion where a sufficient number of point clouds were not generated in the preliminary imaging, based on the point cloud density evaluation value acquired in the point cloud density evaluation process P24. Specifically, a portion is determined to be a point cloud insufficient portion when the point cloud density evaluation value is less than a predetermined threshold. When a point cloud insufficient portion is detected using a method in which the point cloud density evaluation value is calculated by counting the number of point clouds generated in real time (which may also take into account the existing point cloud generation status at that portion), the position of the image capture device 1 at that time is stored in memory 15 as the position of the point cloud insufficient portion. On the other hand, in a method in which a portion where normal vectors are discontinuous is detected or a portion where a point cloud exists but the point cloud density is extremely low (for example, when the density is within a range between two predetermined thresholds), the position determined to be a portion where the normal vectors are discontinuous or where the density is extremely low is stored in memory 15 as the position of the point cloud insufficient portion.

[0101] In a method in which the point cloud density evaluation value is calculated by counting the number of generated point clouds in real time (which may take into account the existing point cloud generation status at that location), the point cloud insufficiency location detection process P34 is performed during the preliminary imaging. On the other hand, in a method in which a location where normal vectors are not continuous is detected or a location where a point cloud exists but the point cloud density is extremely low (for example, when the density is within an interval between two predetermined thresholds), the point cloud insufficiency location detection process P34 is performed when the preliminary imaging is completed.

[0102] The support information generation process P4 includes a target shooting range acquisition process P41, a progress rate acquisition process P42, a recommended route generation process P43, a progress status image generation process P44, a map image generation process P45, a route image generation process P46, and a confirmation map generation process P47.

[0103] In target shooting range acquisition process P41, processor 16 sets a target shooting range to be the target of main shooting based on the point cloud data generated in point cloud generation process P14 during preliminary shooting. Processor 16 also sets a user-specified target shooting range in response to a user operation that changes the range from the initial target shooting range set based on the point cloud data.

[0104] In the progress rate acquisition process P42, the processor 16 acquires the photographed range for the actual photography based on the point cloud data generated in the point cloud generation process P14 up to now in the actual photography, and acquires the progress rate of the photography for the actual photography based on this photographed range and the target photography range acquired in the target photography range acquisition process P41. The progress rate is the ratio of the photographed range to the target photography range.

[0105] In the recommended route generation process P43, the processor 16 generates a recommended route for the actual shooting, i.e., a route that is recommended for the user to take during the actual shooting, based on the target shooting range acquired in the target shooting range acquisition process P41. Specifically, a route that covers the entire target shooting range and has as short a travel distance as possible is generated as the recommended route. Furthermore, in the recommended route generation process P43, a recommended route is generated in accordance with the user's instructions so as to avoid areas that require caution when shooting, i.e., areas with strong reflections, areas with insufficient features, and areas with sudden movements.

[0106] In the progress status image generation process P44, the processor 16 generates an image that visualizes the progress status based on the progress rate acquired in the progress rate acquisition process P42. Specifically, an indicator image 621 (see FIG. 25, etc.) is generated. The indicator image 621 is, for example, a progress bar, and is drawn so that the portion indicating the photographed range changes as the actual photographing progresses.

[0107] In map image generation processing P45, processor 16 generates an overhead point cloud image that becomes map image 32 representing the measurement target location on confirmation map 31, based on the point cloud data generated in point cloud generation processing P14. Specifically, after shooting in preliminary shooting mode, an overhead point cloud image 35 by preliminary shooting, i.e., an overhead point cloud image is generated based on the point cloud data generated in point cloud generation processing P14 during preliminary shooting (see FIG. 16, etc.). Also, during shooting in main shooting mode, an overhead point cloud image 36 by main shooting, i.e., an overhead point cloud image is generated based on the point cloud data generated in point cloud generation processing P14 from the start of main shooting to the present (see FIG. 24, etc.).

[0108] In the route image generation process P46, the processor 16 generates a route image 51 representing the recommended route based on the recommended route generated in the recommended route generation process P43 (see, for example, FIGS. 22 and 30). The route image 51 visualizes the progress of the actual photography. Specifically, as the actual photography progresses, photographed sections of the route image 51 are erased. Alternatively, photographed sections of the route image 51 may be drawn in a different manner (e.g., color) from unphotographed sections without being erased. Note that, although an example is shown in which the start position mark (S) is displayed on the confirmation map 31 even after photography has started, for example, a confirmation map 31 (see FIG. 31) from which the start position mark (S) has been erased may be displayed after photography has started.

[0109] In the confirmation map generation process P47, the processor 16 generates a confirmation map 31 for the user to confirm matters to be aware of during actual photography, based on the map image 32 acquired in the map image generation process P45, the route image 51 acquired in the route image generation process P46, and the location information of the points to be careful about when photographing detected in the point to be careful about when photographing process P3. The confirmation map 31 is obtained by superimposing an image that visualizes various types of support information on the map image 32 that represents the location to be measured. The confirmation map 31 is displayed after photography in the preliminary photography mode and during photography in the main photography mode.

[0110] In confirmation map 31 displayed after shooting in preliminary shooting mode, an image visualizing areas requiring attention for shooting, i.e., areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements, is superimposed on map image 32. Map image 32 is an image visualizing the results of preliminary shooting, specifically, an overhead point cloud image 35 generated from the point cloud data acquired in the preliminary shooting. The image visualizing areas requiring attention for shooting specifically consists of mark images 41, 42, and 43 representing areas with strong reflections, areas with insufficient features, and areas with sudden movements, and an area image 44 representing areas with insufficient point clouds.

[0111] In addition, in the confirmation map 31 displayed during shooting in the main shooting mode, images indicating points to be careful about when shooting (mark images 41, 42, 43, area image 44) and a route image 51 indicating a recommended route are superimposed on the map image 32.

[0112] At this time, the progress is visualized on the map image 32. Specifically, the map image 32 (see FIG. 24, etc.) includes an overhead point cloud image 35 from the preliminary shooting and an overhead point cloud image 36 from the main shooting, and is drawn so that as the main shooting progresses, the already-captured range in the overhead point cloud image 35 from the preliminary shooting is gradually erased and replaced with the overhead point cloud image 36 from the main shooting.

[0113] The progress is also visualized in images showing areas requiring careful shooting (mark images 41, 42, 43, area image 44). Specifically, the mark images 41, 42, 43 showing areas with strong reflections, areas lacking features, and areas with sudden movements, and the area image 44 showing areas lacking point clouds, are replaced with images generated during the actual shooting from those generated during the preliminary shooting as the actual shooting progresses. The progress is also visualized in the route image 51. Specifically, as the actual shooting progresses, the already-photographed sections of the route image 51 are erased.

[0114] In the display information generation process P5, the processor 16 generates display information for the screen (see Figures 12 to 31) to be displayed on the display 13 based on the image captured by the visible camera 21 and the confirmation map 31 generated in the confirmation map generation process P47.

[0115] In the display process P6, the processor 16 displays a screen (see FIGS. 12 to 31) on the display 13 based on the display information generated in the display information generation process P5.

[0116] Next, a description will be given of the processing procedure performed during shooting (during preliminary shooting and main shooting) in processor 16. Fig. 8 is a flowchart showing the processing procedure performed by processor 16 during shooting.

[0117] First, the processor 16 acquires the image captured by the visible camera 21, the distance information from the depth camera 22, and the detection data from the IMU 23 from the sensor unit 12 (ST101).

[0118] Next, the processor 16 extracts feature points from the image captured by the visible camera 21 (feature extraction process P11) (ST102).

[0119] Next, the processor 16 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition regarding the position and orientation of the image capturing device 1, and updates the position and orientation trajectory data based on the amount of transition (tracking process P12) (ST103).

[0120] Next, the processor 16 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23 (position and orientation correction process P13) (ST104).

[0121] Next, the processor 16 generates point cloud data based on the distance information of the depth camera 22 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13 (point cloud generation process P14) (ST105).

[0122] Next, processor 16 determines whether preliminary photography is in progress (ST106). If preliminary photography is in progress (Yes in ST106), processor 16 then performs preliminary photography processing (ST107). On the other hand, if preliminary photography is not in progress (No in ST106), processor 16 then determines whether main photography is in progress (ST108). If main photography is in progress (Yes in ST108), processor 16 then performs main photography processing (ST109).

[0123] Next, the processor 16 generates display information for the screen being photographed, including the image captured by the visible camera 21 (display information generation process P5) (ST110). At this time, if preliminary photographing is in progress, display information for the screen being photographed, including the overhead point cloud image 35 generated in the preliminary photographing, is generated in addition to the image captured by the visible camera 21. Also, if main photographing is in progress, display information for the screen being photographed, including the confirmation map 31, is generated in addition to the image captured by the visible camera 21. The confirmation map 31 includes a map image 32 made up of the overhead point cloud image 35, mark images 41, 42, 43 and an area image 44 indicating points to be careful of when photographing, and a route image 51 indicating a recommended route (see FIGS. 2 and 3).

[0124] Next, the processor 16 displays a screen on the display 13 based on the generated display information (display process P6) (ST111).

[0125] Next, a description will be given of the procedure of the process during preliminary imaging in ST107 shown in Fig. 8. Fig. 9 is a flowchart showing the procedure of the process during preliminary imaging.

[0126] First, the processor 16 acquires an evaluation value (reflection evaluation value) that indicates the intensity of reflected light that appears in a captured image (reflection evaluation process P21) (ST201).

[0127] Next, processor 16 counts the feature points within the target range based on the extraction results of feature extraction process P11, and obtains an evaluation value (number of feature points) representing the extraction status of the feature points (feature point evaluation process P22) (ST202).

[0128] Next, the processor 16 acquires an evaluation value (sudden movement evaluation value) that indicates the abruptness of the user's movement of the image capturing device 1 (sudden movement evaluation process P23) (ST203). At this time, the sudden movement evaluation value is calculated based on the detection data of the IMU 23. The sudden movement evaluation value is also calculated based on the amount of transition related to the position and orientation of the image capturing device 1 acquired in the tracking process P12.

[0129] Next, processor 16 counts the point clouds within the target range based on the point cloud data generated in point cloud generation process P14, and obtains an evaluation value (point cloud density evaluation value) representing the point cloud generation status (point cloud density evaluation process P24) (ST204).

[0130] Next, processor 16 detects areas to be careful about when photographing (areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements) based on the evaluation values ​​obtained in each of the reflection evaluation process P21 (ST201), the feature point evaluation process P22 (ST202), the sudden movement evaluation process P23 (ST203), and the point cloud density evaluation process P24 (ST204) (area detection process P3 for areas to be careful about when photographing) (ST205).

[0131] At this time, the reflection evaluation value is compared with a predetermined threshold to detect strong reflection areas (strong reflection area detection process P31). Also, the number of feature points is compared with a predetermined threshold to detect feature-lacking areas (feature-lacking area detection process P32). Also, the sudden movement evaluation value is compared with a predetermined threshold to detect sudden movement areas (sudden movement area detection process P33). Also, the point cloud density evaluation value is compared with a predetermined threshold to detect point cloud-lacking areas (point cloud-lacking area detection process P34). The thresholds used in each detection process may be set in advance or may be specified by the user as appropriate.

[0132] Next, processor 16 determines whether or not a region requiring caution in photography (a region lacking a point cloud, a region with strong reflection, a region lacking a feature, and a region with sudden movement) has been detected (ST206). If a region requiring caution in photography has been detected (Yes in ST206), processor 16 stores the position of the detected region requiring caution in photography and the content of a warning to the user in memory 15 (ST207).

[0133] Next, the processor 16 stores the point cloud data generated in the point cloud generation process P14 (ST105 in FIG. 8) in the memory 15 (ST208). Next, the processor 16 generates display information for the screen including a highlight or a text display indicating that preliminary imaging is in progress (display information generation process P5) (ST209).

[0134] Next, a description will be given of the procedure of the process during actual imaging in ST109 shown in Fig. 8. Fig. 10 is a flow chart showing the procedure of the process during actual imaging.

[0135] First, the processor 16 compares the point cloud data generated this time in the point cloud generation process P14 (ST105 in Figure 8) with the point cloud data at the time of preliminary photography stored in the memory 15, and determines whether the range of the point cloud data generated this time overlaps with the range of the point cloud data at the time of preliminary photography (ST301).

[0136] If the range of the current point cloud data overlaps with the range of the point cloud data from the preliminary photographing (Yes in ST302), processor 16 then updates map image 32 consisting of overhead point cloud images 35, 36 (see FIG. 24, etc.) for the range where the current point cloud data and the point cloud data from the preliminary photographing overlap (map image generation process P45) (ST303). At this time, map image 32 is drawn so that overhead point cloud image 35 from the preliminary photographing is deleted and replaced with overhead point cloud image 36 from the main photographing. Note that even for point cloud data from the preliminary photographing that has some error with respect to the point cloud data from the main photographing, it may be determined that both point cloud data overlap, and the overhead point cloud image 35 from the preliminary photographing may be deleted.

[0137] Next, the processor 16 generates a confirmation map 31 including an updated or unupdated map image 32 (confirmation map generation process P47) (ST304). At this time, mark images 41, 42, 43 indicating points to be careful of when taking photographs, an area image 44, and a route image 51 indicating a recommended route are superimposed on the map image 32 in accordance with the display setting contents specified by the user, to generate the confirmation map 31.

[0138] Next, a description will be given of the procedure of the process carried out when the preliminary photographing is completed in the processor 16. Fig. 11 is a flowchart showing the procedure of the process carried out when the preliminary photographing is completed.

[0139] First, the processor 16 detects areas where the point cloud is lacking based on the point cloud data acquired by the preliminary photographing, i.e., the point cloud data generated by the point cloud generation process P14 during the preliminary photographing and stored in the memory 15 (point cloud lacking area detection process P34) (ST401).

[0140] Next, processor 16 generates a recommended route based on the target shooting range set from the point cloud data acquired by the preliminary shooting and the positions and details of the detected places to be careful about when shooting (recommended route generation process P43) (ST402). Specifically, a route that covers the target shooting range and has the shortest possible travel distance is generated as the recommended route. At this time, the recommended route is generated according to setting information related to recommended route generation. For example, if the user is instructed to avoid places to be careful about when shooting, a recommended route is generated that avoids the places to be careful about when shooting.

[0141] Next, the processor 16 generates a confirmation map 31 including an overhead point cloud image 35 (map image 32) generated from the point cloud data acquired in the preliminary photographing, mark images 41, 42, 43 and an area image 44 indicating areas requiring caution in photographing (areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements), and a route image 51 indicating the recommended route (confirmation map generation process P47) (ST403).

[0142] Next, processor 16 generates display information for a screen (confirmation screen) including confirmation map 31 as a screen for the user to confirm the preliminary shooting results (display information generation process P5) (ST404). Next, processor 16 displays a screen on display 13 based on the generated display information (display process P6) (ST405).

[0143] Next, processor 16 determines whether or not a user operation (e.g., tap operation, click operation) has been performed to select areas to be careful about when photographing (areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements) (ST406). If a user operation to select areas to be careful about when photographing has not been performed (No in ST406), processor 16 then determines whether or not a user operation to close the screen for checking the preliminary photographing results has been performed (ST407). If a user operation to close the screen for checking the preliminary photographing results has not been performed (No in ST407), the process returns to ST406.

[0144] On the other hand, if the user has performed an operation to select a point to be careful about when photographing (Yes in ST406), display information for a screen (confirmation screen) including a message that provides guidance (such as a suggestion of an improvement measure) according to the type of the selected point to be careful about when photographing is generated (display information generation process P5) (ST408). Next, returning to ST404, processor 16 displays a screen including the message on display 13 based on the generated display information.

[0145] Next, a screen 101 displayed on the display 13 in the preliminary capture mode will be described. Fig. 12 is an explanatory diagram showing the screen 101 displayed before capture in the preliminary capture mode. Figs. 13 and 14 are explanatory diagrams showing a screen 201 displayed during capture in the preliminary capture mode. Fig. 15 is an explanatory diagram showing a screen 301 displayed immediately after capture in the preliminary capture mode.

[0146] 12 to 15, a screen 101 in the preliminary capture mode is provided with a main window 102, a sub-window 103, and an operation window 104. The sub-window 103 is provided so as to be superimposed on a part of the main window 102.

[0147] The main window 102 and the sub-window 103 respectively display a captured image 111 and an overhead point cloud image 35. The captured image 111 is a current captured image, that is, an image captured in real time and output from the visible camera 21. The overhead point cloud image 35 is an image (rendering) of each point of the point cloud data as seen from a viewpoint in the sky.

[0148] Note that on the screen 101 before shooting shown in Fig. 12, point cloud data for the measurement target location has not been generated, so the overhead point cloud image 35 is not displayed. On the screen 101 during shooting shown in Figs. 13 and 14, point cloud data for the measurement target location has been partially generated, so the overhead point cloud image 35 is partially displayed. On the screen 101 immediately after shooting shown in Fig. 15, point cloud data for the measurement target location has been generated overall, so the overhead point cloud image 35 is displayed overall.

[0149] The operation window 104 is provided with a "start preliminary shooting" button 121 and an "end preliminary shooting" button 122. When the user operates the "start preliminary shooting" button 121, preliminary shooting begins, and images captured by the visible camera 21 are stored in the memory 15. When the user operates the "end preliminary shooting" button 122, preliminary shooting ends. In a configuration including a depth camera 22 and an IMU 23, distance information from the depth camera 22 and detection data from the IMU 23 are stored in the memory 15 along with images captured by the visible camera 21.

[0150] The operation window 104 also has a "screen switch" button 123 and a check box 124 related to the display of the sub-window 103. When the user operates the "screen switch" button 123, a state is switched between one in which the captured image 111 is enlarged and displayed in the main window 102 and the overhead point cloud image 35 is reduced and displayed in the sub-window 103 (the state shown in FIGS. 13 and 14), and another state in which the overhead point cloud image 35 is enlarged and displayed in the main window 102 and the captured image 111 is reduced and displayed in the sub-window 103. When the user operates the check box 124, the sub-window 103 can be switched between a display state and a non-display state.

[0151] Furthermore, the operation window 104 is provided with a "Main Shooting" tab 131, a "Preliminary Shooting" tab 132, and a "Change Settings" tab 133 as operation sections for selecting a mode. The screen 101 shown in FIGS. 12 to 19 is for the preliminary shooting mode (first shooting mode), and is displayed when the user operates the "Preliminary Shooting" tab 132. When the user operates the "Change Settings" tab 133, the mode switches to the setting change mode, and the screen transitions to screens 301, 401, and 501 (see FIGS. 20 to 23). When the user operates the "Main Shooting" tab 131, the mode switches to the main shooting mode (second shooting mode), and the screen transitions to screen 601 (see FIGS. 24 to 31).

[0152] In this embodiment, the modes are switched by operating the tabs 131, 132, and 133, but the modes may be switched by operating other types of operation units such as buttons.

[0153] When shooting starts in the preliminary shooting mode, the display mode of screen 101 changes to notify the user that preliminary shooting is in progress. In the example shown in FIG. 13, screen 101 is highlighted. Specifically, frame images 141 and 142 are displayed in main window 102 and operation window 104. Frame images 141 and 142 are drawn in a predetermined color (e.g., red). The background of operation window 104 is displayed in a predetermined color (e.g., green). On the other hand, in the example shown in FIG. 14, text 151 indicating that preliminary shooting is in progress is displayed at the top of main window 102. Note that by switching between tabs 131, 132, and 133, a specific screen 101 for each mode may be highlighted.

[0154] During preliminary shooting, message display windows containing various messages are displayed on the screen as needed. Some messages are relatively important (urgent), such as instructions or notifications that require immediate attention, while others are relatively less important, such as reference information or content requiring attention. Message display windows containing messages of relatively high importance are displayed in the center of the main window 102, while message display windows containing messages of relatively low importance are displayed at the bottom of the main window 102. Message display windows disappear after a predetermined display time has elapsed or when there is no longer any need for their display.

[0155] For example, when the preliminary photographing starts, a confirmation message such as "Preliminary photographing will begin. Please take photographs evenly throughout the 3D point cloud generation range" is displayed at the bottom of the main window 102. In the example shown in Fig. 15, when the preliminary photographing ends, a confirmation message is displayed in the message display window 161 in the center of the main window 102. For example, a message such as "Processing. Please wait a moment" is displayed.

[0156] Next, a screen 201 that is displayed on the display 13 when checking an image after shooting in the preliminary shooting mode will be described. Figures 16, 17, 18, and 19 are explanatory diagrams showing the screen 201 that is displayed when checking an image after shooting in the preliminary shooting mode.

[0157] 16, 17, and 18 are examples of a screen 201 (confirmation screen) that is displayed after shooting in the preliminary shooting mode. In the screen 201, a confirmation map 31 is displayed in the main window 102 so that the user can check points to be careful about before shooting.

[0158] 16 shows an example in which a strong reflection area and a point cloud insufficient area are detected. In this case, a mark image 41 representing the strong reflection area is superimposed on the map image 32 (bird's-eye view point cloud image 35) at the position where the strong reflection area is detected on the confirmation map 31. In addition, an area image 44 representing the point cloud insufficient area is superimposed on the range on the map image 32 where the point cloud insufficient area is detected.

[0159] When the user selects a strong reflection location on the confirmation map 31, specifically, when the user performs a predetermined operation (tap operation, click operation) on the mark image 41 representing the strong reflection location, a message prompting the user to take measures to improve the shooting environment related to the strong reflection location is displayed in the message display window 211. In this example, the selected strong reflection location is a location where reflected light is assumed to be caused by indoor lighting, and a message prompting the user to improve the shooting environment regarding lighting, such as a message saying, "If the reflection is strong, please reduce the lighting at the shooting location," is displayed. Also, if the reflected light is assumed to be caused by external light entering through a window, a message prompting the user to improve the shooting environment regarding external light, such as a message saying, "If the curtains are open and external light is entering, please close the curtains," is displayed. Note that if both indoor lighting and external light entering through a window are likely to be the cause, a message prompting the user to improve both may be displayed, such as a message saying, "Please reduce the lighting at the shooting location and close the curtains."

[0160] In addition, when the user selects a point cloud deficiency area on the confirmation map 31, specifically, performs a predetermined operation (tap operation, click operation) on the area image 44 representing the point cloud deficiency area, a message may be displayed in the message display window 211 urging the user to improve the shooting environment or shooting method related to the point cloud deficiency area.

[0161] 17 shows an example in which a feature-lacking portion is detected. In this case, in the confirmation map 31, a mark image 42 representing the feature-lacking portion is superimposed at the position where the feature-lacking portion is detected on the map image 32 (bird's-eye view point cloud image 35).

[0162] When the user selects a feature-deficient portion on the confirmation map 31, specifically, when the user performs a predetermined operation (tap operation, click operation) on the mark image 42 representing the feature-deficient portion, a message prompting the user to take measures to improve the shooting environment or shooting method related to the feature-deficient portion is displayed in the message display window 211. In this example, the selected feature-deficient portion is caused by the shooting environment of the measurement target location, and a message prompting the user to improve the shooting environment, such as a message saying "Please place something with texture," is displayed. Furthermore, when the selected feature-deficient portion is caused by the user's shooting method, a message prompting the user to improve the shooting method, such as a message saying "Please set the angle of view to include a distinctive pattern or object," is displayed.

[0163] 18 shows an example in which a sudden movement location is detected. In this case, in the confirmation map 31, a mark image 43 representing the sudden movement location is superimposed at the position where the sudden movement location is detected on the map image 32 (bird's-eye view point cloud image 35).

[0164] When the user performs an operation to select a sudden movement location on the confirmation map 31, specifically, a predetermined operation (tap operation, click operation) on the mark image 43 representing the sudden movement location, a message urging the user to take measures to improve the shooting method related to the sudden movement is displayed in the message display window 211. Specifically, a message urging the user not to make sudden movements (sharp turns), for example, a message saying "When shooting, move slowly to avoid making sharp turns," is displayed.

[0165] 19 is another example of the screen 201 displayed after shooting in the preliminary shooting mode. In the examples shown in FIGS. 16, 17, and 18, by operating the mark images 41, 42, and 43 indicating areas requiring caution when shooting, a message is displayed in the message display window 211 urging the user to improve the shooting environment or shooting method. On the other hand, in this example, the details of the areas requiring caution when shooting are displayed in a list in the legend display section 221. This allows the user to understand the details of the mark images 41, 42, and 43 indicating areas requiring caution when shooting and the area image 44 drawn on the confirmation map 31.

[0166] As described above, in this embodiment, points requiring attention for photography are detected based on the results of the preliminary photography, and on the screen after the preliminary photography, a confirmation map 31 is displayed on which mark images 41, 42, 43 indicating the points requiring attention for photography and an area image 44 are drawn, and by selecting a point requiring attention for photography on the confirmation map 31, a message providing guidance (such as a suggestion of improvement measures) regarding the selected point requiring attention for photography is displayed. This allows the user to understand the location and details of the points requiring attention for photography, and improve the photography environment and photography method to prevent the occurrence of events that could reduce point cloud accuracy, thereby enabling highly accurate point cloud data to be obtained with just one actual photography session.

[0167] In this embodiment, by selecting a point of caution for photography on the confirmation map 31 on the screen 201 after preliminary photography, a message providing guidance regarding the selected point of caution for photography is displayed, but a message providing guidance regarding the detected point of caution for photography may also be displayed at the time when a point of caution for photography is detected during preliminary photography.

[0168] 16 to 19, in the confirmation map 31, mark images 41, 42, and 43 representing areas to be careful of when taking photographs (areas with strong reflections, areas lacking features, and areas of sudden movements) are superimposed on the map image 32 (bird's-eye view point cloud image 35), but if multiple areas to be careful of when taking photographs are detected at a high density, an area image drawn to include the multiple areas to be careful of when taking photographs may be superimposed. Also, if the mark images representing areas to be careful of when taking photographs are circular, for example, multiple mark images may be superimposed so that they overlap each other, allowing them to be recognized as a single area.

[0169] Next, we will explain the screens 301, 401, 501 (setting screens) in the setting change mode related to the preliminary shooting results displayed on the display 13. Fig. 20 is an explanatory diagram showing the screen 301 related to progress status setting. Fig. 21 is an explanatory diagram 401 showing the screen related to setting the display of points to be careful about when shooting. Figs. 22 and 23 are explanatory diagrams showing the screen 501 related to recommended route setting.

[0170] As shown in Figures 20 to 23, the setting change mode screens 301, 401, and 501 are provided with a "Progress Status Settings" tab 311, a "Caution Points for Shooting Display Settings" tab 312, and a "Recommended Route Settings" tab 313 as operation sections for switching settings.

[0171] When the user operates the "Progress Status Settings" tab 311, a screen 301 (see FIG. 20) relating to progress status settings is displayed. When the user operates the "Caution Points for Photography Display Settings" tab 312, a screen 401 (see FIG. 21) relating to caution points for photography display settings is displayed. When the user operates the "Recommended Route Settings" tab 313, a screen 501 (see FIGS. 22 and 23) relating to recommended route settings is displayed.

[0172] In this embodiment, the setting items are switched by operating the tabs 311, 312, and 313, but the setting items may be switched by operating other types of operation units such as buttons.

[0173] 20 is displayed when the user operates the "Progress Status Settings" tab 311. On this screen 301, the user can specify whether or not to display the progress status. Also, on this screen 301, the user can specify the target shooting range that will be used as a reference when obtaining the progress status.

[0174] A check box 321 for displaying the progress status is provided on the screen 301. When the user operates the check box 321, the user can switch the setting as to whether or not to display the progress status on the screen 601 (see FIG. 25, etc.) during actual shooting. Specifically, the user can switch whether or not to display an indicator image 621 indicating the progress rate on the screen 601 during actual shooting.

[0175] The screen 301 also has a "Start range input" button 323 and a "Execute range setting" button 324 for setting a target shooting range designated by the user.

[0176] When the user operates the "Start range input" button 323, the system transitions to range input mode. In range input mode, the user can input the target shooting range on the overhead point cloud image 35 displayed in the main window 102. Specifically, for example, the user can input the target shooting range by sequentially specifying the vertices of a polygon representing the target shooting range (for example, by tapping or clicking). At this time, an area image 61 representing the target shooting range is displayed in the main window 102 together with the overhead point cloud image 35. This allows the user to specify a preliminary shooting range, i.e., a range different from the target shooting range set based on the point cloud data generated in the preliminary shooting, as a new target shooting range.

[0177] When the user operates the "execute range setting" button 324, the range input mode is cancelled and the target shooting range input by the user is registered.

[0178] 21 is displayed when the user operates the "Caution Points for Photography Display Settings" tab 312. On this screen 401, the user can specify whether or not to display each type of caution points for photography on the confirmation map 31, and can check the confirmation map 31 generated based on the specification.

[0179] The screen 401 is provided with check boxes 411 to 414 for displaying areas requiring attention when photographing, that is, areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements.

[0180] By operating a check box 411 related to the display of strong reflection areas, the user can switch between displaying and not displaying a mark image 41 representing a strong reflection area on the confirmation map 31. Furthermore, by operating a check box 412 related to the display of feature-lacking areas, the user can switch between displaying and not displaying a mark image 42 representing a feature-lacking area on the confirmation map 31. Furthermore, by operating a check box 413 related to the display of sudden action areas, the user can switch between displaying and not displaying a mark image 43 representing a sudden action area on the confirmation map 31. Furthermore, by operating a check box 414 related to the display of point cloud-lacking areas, the user can switch between displaying and not displaying an area image 44 representing a point cloud-lacking area on the confirmation map 31.

[0181] Furthermore, the screen 401 is provided with a "execute display settings" button 415. When the user operates the "execute display settings" button 415, display conditions for points of caution for photography are set according to the input contents of the check boxes 411 to 414. At this time, in the main window 102, a confirmation map 31 is generated in which mark images 41, 42, 43 and an area image 44 are superimposed on the map image 32 (bird's-eye view point cloud image 35) according to the settings, and the confirmation map 31 is displayed in the main window 102. Note that it is preferable to check all the check boxes 411 to 414 as an initial setting, and allow the user to uncheck them at will. Furthermore, display settings may be saved for each user in association with a photography location, and the saved display settings may be reused.

[0182] 22 is displayed when the user operates the "Recommended Route Settings" tab 313. On this screen 501, the user can specify conditions for generating a recommended route and check the recommended route generated based on those conditions.

[0183] The screen 501 is provided with check boxes 511 to 513 for avoiding areas to be careful of when photographing when generating a recommended route, that is, areas with strong reflections, areas with insufficient features, and areas with sudden movements.

[0184] The user can switch whether or not to avoid strong reflection points when generating a recommended route by operating a check box 511 related to avoiding strong reflection points. Also, the user can switch whether or not to avoid feature-insufficient points when generating a recommended route by operating a check box 512 related to avoiding feature-insufficient points. Also, the user can switch whether or not to avoid feature-insufficient points when generating a recommended route by operating a check box 513 related to avoiding sudden movement points.

[0185] Screen 501 also has a "Generate recommended route" button 514. When the user operates "Generate recommended route" button 514, a recommended route is generated in accordance with the input contents of check boxes 511 to 513. Furthermore, a route image 51 representing the recommended route is generated, and a confirmation map 31 is generated in which route image 51 is superimposed on map image 32 (bird's-eye view point cloud image 35), and confirmation map 31 is displayed in main window 102.

[0186] In the example shown in FIG. 22, check boxes 511 to 513 are set by operation input so as not to avoid strong reflection areas, feature-insufficient areas, and sudden movement areas. In this case, a route that covers the entire target shooting range and has as short a travel distance as possible is generated as the recommended route, regardless of the strong reflection areas, feature-insufficient areas, and sudden movement areas. On the other hand, in the example shown in FIG. 23, it is set so as to avoid strong reflection areas. In this case, a recommended route is generated so as to avoid strong reflection areas. Note that the settings for generating the recommended route may be saved in association with the shooting location for each user, and the saved settings for generating the recommended route may be reused.

[0187] Next, we will explain the screen 601 in the actual shooting mode that is displayed on the display 13. Figures 24, 25, 26, 27, 28, 29, 30, and 31 are explanatory diagrams showing the screen 601 that is displayed on the display 13 in the actual shooting mode.

[0188] As shown in FIG. 24, the main shooting mode screen 601 is provided with a main window 102, a sub-window 103, and an operation window 104, similar to the preliminary shooting mode screen 101 (see FIG. 12, etc.).

[0189] A confirmation map 31 is displayed in the sub-window 103 to allow the user to check points that require attention during actual photography.

[0190] The operation window 104 is provided with a "Start Main Shooting" button 611 and an "End Main Shooting" button 612. When the user operates the "Start Main Shooting" button 611, the main shooting starts, and the image captured by the visible camera 21, the distance information by the depth camera 22, and the detection data by the IMU 23 are stored in the memory 15. When the user operates the "End Main Shooting" button 612, the main shooting ends.

[0191] Immediately after transitioning to the actual shooting mode screen 601 or immediately after the user operates the "Start actual shooting" button 611, a confirmation message, for example, a message stating "Actual shooting will begin. Please shoot from the starting point," may be displayed. Also, actual shooting may start from the point where preliminary shooting ended, or a starting point for actual shooting may be suggested based on the preliminary shooting results. Also, in order to obtain the progress during actual shooting, it is necessary to align the preliminary shooting results (point cloud) with the actual shooting results (point cloud). To facilitate this alignment, during preliminary shooting, shooting may be completed with the photographing device 1 placed on a desk or the like, and the photographing device 1 may not be moved until actual shooting begins.

[0192] Also, the operation window 104 is provided with a "screen switch" button 123 and a check box 124 for displaying the sub-window 103, similar to the screen 101 of the preliminary photographing mode (see FIG. 12, etc.).

[0193] The operation window 104 also has a check box 615 for displaying the progress status, a check box 616 for displaying the preliminary shooting results, i.e., the point cloud acquired in the preliminary shooting, a check box 617 for displaying points to be careful about when shooting, and a check box 618 for displaying the recommended route.

[0194] The user can specify whether or not to display the progress status (enable or disable) by operating a check box 615 related to the display of the progress status. The user can also specify whether or not to display the preliminary shooting results by operating a check box 616 related to the display of the preliminary shooting results (point cloud). The user can also specify whether or not to display the points to be careful about when shooting by operating a check box 617 related to the display of the points to be careful about when shooting. The user can also specify whether or not to display the recommended route by operating a check box 618 related to the display of the recommended route.

[0195] Here, the areas requiring caution in photography include areas requiring strong reflection, areas lacking features, areas requiring sudden movement, and areas requiring insufficient point clouds, and whether or not to display each type of area requiring caution in photography can be specified collectively using checkboxes 617 related to the display of areas requiring caution in photography. Note that whether or not to display each type of area requiring caution in photography (areas requiring strong reflection, areas lacking features, areas requiring sudden movement, and areas requiring insufficient point clouds) may also be specified individually.

[0196] Note that the check boxes 615 to 618 on the screen for actual photography mode may be operated before photography starts, but may also be operated after photography starts. For example, if the user finds the displays of the progress status, preliminary photography results, points to be careful about when photography is started, the user may operate the check boxes 615 to 618 during actual photography to change whether or not to display them.

[0197] The example shown in Figure 24 is a case where, by operating check boxes 615 to 618, the display of progress status is disabled, the display of preliminary shooting results is enabled, the display of areas to be careful about when shooting is disabled, and the display of the recommended route is disabled.

[0198] In this case, a map image 32 showing the measurement target location is displayed on the confirmation map 31 displayed in the subwindow 103. The map image 32 includes an overhead point cloud image 35 obtained by preliminary photography and an overhead point cloud image 36 obtained by main photography. The overhead point cloud image 35 obtained by preliminary photography is an overhead point cloud image that visualizes the point cloud data acquired in the preliminary photography, and the overhead point cloud image 36 obtained by main photography is an overhead point cloud image that visualizes the point cloud data acquired up to the present time in the main photography.

[0199] The overhead point cloud image 35 obtained by preliminary shooting and the overhead point cloud image 36 obtained by main shooting are drawn in different drawing modes (e.g., colors). Specifically, for example, the overhead point cloud image 35 obtained by preliminary shooting is drawn in purple, and the overhead point cloud image 36 obtained by main shooting is drawn in black. Note that if the captured color is reflected in the acquired point clouds, for example, the overhead point cloud image 35 obtained by preliminary shooting may be drawn in purple, and the overhead point cloud image 36 obtained by main shooting may be drawn in the acquired color. Furthermore, each predetermined color may be blended (mixed) into the color of each acquired point cloud.

[0200] The map image 32 visualizes the progress of the actual photography. Specifically, as the actual photography progresses, the already-captured range in the overhead point cloud image 35 from the preliminary photography is gradually erased and replaced with the overhead point cloud image 36 from the actual photography. Here, the remaining overhead point cloud image 35 from the preliminary photography represents the range that has not yet been captured, and the overhead point cloud image 36 from the actual photography represents the range that has already been captured. Furthermore, as the actual photography progresses, the range of the overhead point cloud image 35 from the preliminary photography gradually becomes smaller, and the range of the overhead point cloud image 36 from the actual photography gradually becomes larger. This allows the user to grasp the progress of the actual photography.

[0201] In addition, in the confirmation map 31, the areas that have been photographed in the overhead point cloud image 35 obtained by preliminary photography may not be erased, but may be drawn in a manner (e.g., color) different from the areas that have not been photographed in the overhead point cloud image 35 obtained by preliminary photography or the overhead point cloud image 36 obtained by main photography.

[0202] The example shown in Fig. 25 is a case where the display of the progress status is enabled, the display of the preliminary photography results is enabled, the display of points to be careful about when taking photos is disabled, and the display of the recommended route is disabled by operating check boxes 615 to 618. In this example, the settings other than the progress status are the same as those in the example shown in Fig. 24.

[0203] In this case, an indicator image 621 that visualizes the progress of the actual shooting is displayed in the main window 102. The confirmation map 31 displayed in the sub-window 103 is the same as the example shown in FIG.

[0204] The indicator image 621 is a so-called progress bar, and is drawn so that the portion indicating the photographed range changes as the actual photographing progresses. That is, as the actual photographing progresses, the photographed portion is drawn gradually larger and the unphotographed portion is drawn gradually smaller. Furthermore, in the indicator image 621, the photographed portion and the unphotographed portion are drawn in different modes (for example, colors or patterns). For example, the photographed portion is drawn in green, and the unphotographed portion is drawn in white.

[0205] The example shown in Fig. 26 is a case where the user has specified a target shooting range. The display settings for the progress status, preliminary shooting results, points to note when shooting, and recommended route by operating check boxes 615 to 618 are the same as those in the example shown in Fig. 25.

[0206] In this case, an area image 61 representing the target shooting range designated by the user is drawn on the confirmation map 31 displayed in the subwindow 103. The area image 61 is drawn in a predetermined color (e.g., yellow), for example, and is superimposed in a semi-transparent state on the overhead point cloud image 35 obtained by preliminary shooting, which represents the initial target shooting range. Note that the area image 61 only needs to be able to show the range, and for example, only a line indicating the range may be drawn.

[0207] The example shown in Fig. 27 is a case where the display of progress status is enabled, the display of preliminary photography results is enabled, the display of points to be careful about when photographing is enabled, and the display of recommended route is disabled by operating check boxes 615 to 618. In this example, the settings other than the points to be careful about when photographing are the same as those in the example shown in Fig. 25. In addition, this example is a case where a point cloud insufficient area is detected.

[0208] In this case, in the confirmation map 31 displayed in the subwindow 103, an area image 44 representing the point cloud insufficient portion is superimposed on the overhead point cloud images 35 and 36 constituting the map image 32. The area image 44 representing the point cloud insufficient portion is drawn in a predetermined color (for example, red).

[0209] The example shown in Fig. 28 is a case where a point cloud insufficient area and a strongly reflective area are detected. The display settings for the progress status, preliminary shooting results, areas to be careful about when shooting, and recommended route by operating check boxes 615 to 618 are the same as the example shown in Fig. 27.

[0210] In this case, in the confirmation map 31 displayed in the sub-window 103, a mark image 41 representing a strongly reflective area and an area image 44 representing an area lacking a point cloud are superimposed on the overhead point cloud images 35, 36 that constitute the map image 32.

[0211] Here, when the user selects a strong reflection location on the confirmation map 31, specifically, when the user performs a predetermined operation (tap operation, click operation) on the mark image 41 representing the strong reflection location, a message prompting the user to take measures to improve the shooting environment related to the strong reflection location is displayed in the message display window 631. In this example, the selected strong reflection location is a location where reflected light is assumed to be caused by indoor lighting, and a message prompting the user to improve the shooting environment regarding lighting, such as a message saying, "If the reflection is strong, please reduce the lighting at the shooting location," is displayed. Also, if the reflected light is assumed to be caused by external light entering through a window, a message prompting the user to improve the shooting environment regarding external light, such as a message saying, "If the curtains are open and external light is entering, please close the curtains," is displayed. Note that if both indoor lighting and external light entering through a window are likely to be the cause, a message prompting the user to improve both may be displayed, such as a message saying, "Please reduce the lighting at the shooting location and close the curtains."

[0212] The example shown in Fig. 29 is a case where a point cloud-deficient area and a feature-deficient area are detected. The display settings for the progress status, preliminary photography results, areas to be careful about when photographing, and recommended route by operating check boxes 615 to 618 are the same as the examples shown in Figs. 27 and 28.

[0213] In this case, in the confirmation map 31 displayed in the sub-window 103, a mark image 42 indicating an area lacking features and an area image 44 indicating an area lacking point cloud are superimposed on the overhead point cloud images 35, 36 that constitute the map image 32.

[0214] Here, when the user selects a feature-deficient portion on the confirmation map 31, specifically, when the user performs a predetermined operation (tap operation, click operation) on the mark image 42 representing the feature-deficient portion, a message urging the user to take measures to improve the shooting environment or shooting method related to the feature-deficient portion is displayed in the message display window 631. In this example, the selected feature-deficient portion is caused by the user's shooting method, and a message urging the user to improve the shooting method is displayed, for example, a message saying, "When shooting walls or floors, please use an angle of view that includes other features." Furthermore, when the selected feature-deficient portion is caused by the shooting environment of the measurement target location, a message urging the user to improve the shooting environment is displayed, for example, a message saying, "Please place something with texture."

[0215] The examples shown in Figures 30 and 31 are when the check boxes 615 to 618 are operated to enable display of all of the progress status, preliminary photography results, areas to be careful about when photographing, and recommended route. In this example, the settings other than the progress status are the same as those in the example shown in Figure 24. This example also shows a case where a strong reflection area, an area lacking features, an area with sudden movement, and an area lacking point clouds have been detected. Figure 30 shows the time immediately after the actual photography has started, and Figure 31 shows the time when the actual photography has progressed to a certain extent.

[0216] In this case, in the confirmation map 31 displayed in the subwindow 103, a route image 51 representing the recommended route is superimposed on the overhead point cloud images 35, 36 constituting the map image 32. In addition, a mark image 41 representing a strong reflection area, a mark image 42 representing a feature-lacking area, a mark image 43 representing a sudden movement area, and an area image 44 representing a point cloud-lacking area are superimposed on the overhead point cloud images 35, 36 constituting the map image 32.

[0217] 30 and 31, a navigation icon 641 is superimposed on the captured image 111 displayed in the main window 102. The navigation icon 641 is an image that represents the user's route, and is created based on the recommended route and the current location.

[0218] If the shooting environment and shooting method are appropriately improved in the actual shooting compared to the preliminary shooting, the mark images 41, 42, 43 and the area image 44 change as the actual shooting progresses. Specifically, the area image 44 representing the point cloud insufficient areas shrinks as the actual shooting progresses. This allows the user to understand that the number of point cloud insufficient areas has decreased. Furthermore, in the sections that have already been photographed, the mark images 41, 42, 43 representing strong reflection areas, feature insufficient areas, and sudden movement areas are erased. This allows the user to confirm that no events that reduce the accuracy of the point cloud data have occurred in the strong reflection areas, feature insufficient areas, and sudden movement areas.

[0219] Furthermore, the route image 51 representing the recommended route changes as the actual photography progresses. Specifically, photographed sections of the route image 51 are erased. This allows the user to understand the progress of the actual photography. It should be noted that photographed sections of the route image 51 may not be erased, but may be drawn in a different style from unphotographed sections. For example, photographed sections may be drawn in a lighter color than unphotographed sections.

[0220] Note that when the photographing device 1 is configured as a tablet terminal (see FIG. 1), it is difficult to operate the screen while photographing. Therefore, operations on the screen 601 in the actual photographing mode, specifically, operations on the check boxes 615 to 618 and operations on the points to be noted when photographing on the confirmation map 31, may be performed before photographing starts, and functions may be restricted so that screen operations cannot be performed once photographing starts. On the other hand, when the photographing device 1 is configured as a wearable device such as a head-mounted device equipped with a camera and a display, screen operations during photographing are easier, so it may be possible to enable screen operations in the actual photographing mode.

[0221] Incidentally, if the user is unfamiliar with the system, it is expected that the user will start actual shooting without taking preliminary photographs. Therefore, if the user operates the main shooting mode tab 131 before shooting in the preliminary shooting mode, a confirmation message regarding preliminary shooting, i.e., a message for confirming whether or not preliminary shooting is to be performed and prompting the user to perform preliminary shooting, may be displayed. For example, a message such as "If you want to perform actual shooting without taking preliminary photographs, continue as is. If you want to perform preliminary photographs, select the preliminary shooting tab." may be displayed.

[0222] In addition, there may be cases where preliminary photography has already been performed at the same measurement location. In this case, if the photography data and point cloud data (preliminary photography results) acquired in the previous preliminary photography can be used, the current preliminary photography can be omitted. Therefore, a screen operation for saving and loading the preliminary photography results may be provided. In this way, if the preliminary photography results have been saved, the current preliminary photography can be omitted by loading the preliminary photography results. This allows, for example, if the main photography fails after the preliminary photography has been performed, the main photography can be redone without performing another preliminary photography.

[0223] Furthermore, if a user starts actual shooting without taking preliminary photographs, the target shooting range cannot be set based on the results of the preliminary photographs, and therefore the progress status of the actual photographs cannot be presented to the user. Therefore, when an attempt is made to start actual photographs without taking preliminary photographs, a confirmation message regarding the progress status may be displayed. For example, a message stating "Progress status will not be displayed" may be displayed.

[0224] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0225] The photographing device, photographing work support method, and photographing work support program according to the present invention have the effect of allowing a user to proceed with photographing work while grasping the progress of the photographing, and are useful as a photographing device that photographs a measurement target location in order to perform 3D measurement processing that generates 3D spatial information of the measurement target location based on photographing data of the measurement target location, as well as a photographing work support method and photographing work support program that support a user's photographing work using the photographing device. [Explanation of symbols]

[0226] 1: Imaging device 11: Device body 12: Sensor unit 13: Display (display unit) 14: Input device 15: Memory 16: Processor 21: Visible camera 22: Depth camera 23:IMU 31: Confirmation map 32: Map image 35: Preliminary overhead point cloud image (first overhead point cloud image) 36: Overhead point cloud image taken during actual shooting (second overhead point cloud image) 41: Mark image showing strong reflection areas 42: Mark image showing feature deficiency 43: Mark image showing sudden movement area 44: Area image showing missing points 51: Route image showing the recommended route 61: Area image showing the target shooting range 101: Pre-shooting mode screen 102: Main window 103: Subwindow 104: Operation window 111: Photographed image 131: "Main Shooting" tab 132: "Preliminary Shooting" tab 133: "Change Settings" tab 161: Message display window 201: Pre-shooting mode screen 211: Message display window 301, 401, 501: Setting change mode screen 311: "Progress Settings" tab 312: "Caution Area Display Settings" tab 313: "Recommended Route Settings" tab 601: Main shooting mode screen 621: Indicator image 631: Message display window 641: Navigation icon

Claims

1. An imaging device that images a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on imaging data of the measurement target location, A device body held by a user; An imaging unit provided in the device body for imaging a measurement target location; a display unit that displays work support information related to an image capturing operation in which a user moves around a measurement target location and causes the image capturing unit to capture the measurement target location; A processor that controls the image capturing unit and the display unit, The processor, Based on the shooting data obtained in the preliminary shooting, the target shooting range for the actual shooting is set, A photographed range is acquired based on the photographing data acquired in the main photographing, and information regarding the progress of the main photographing is acquired based on the photographed range and the target photographing range; The photographing apparatus is characterized in that a screen including information regarding the progress of the actual photographing is displayed on the display unit.

2. The processor, 2. The photographing apparatus according to claim 1, wherein the photographing mode is switched between a first photographing mode in which preliminary photographing is performed and a second photographing mode in which main photographing is performed in response to a user's operation to select a photographing mode.

3. The processor, The photographing device according to claim 1, characterized in that, during the actual photographing, a screen including a real-time photographed image output from the photographing unit and an image visualizing the progress of the actual photographing is displayed on the display unit.

4. The processor, A map image showing the measurement target locations is generated as an image visualizing the progress of the actual photography. The photographing device according to claim 3, characterized in that the map image includes a first overhead point cloud image representing the target photographing range generated by preliminary photographing, and a second overhead point cloud image representing the photographed range generated by main photographing, and the first overhead point cloud image and the second overhead point cloud image are drawn in different manners.

5. The processor, 4. The photographing device according to claim 3, further comprising an indicator image that visualizes the progress of the actual photographing, the indicator image having a portion indicating the photographed range changing as the actual photographing progresses.

6. The processor, Set the initial target shooting range based on the point cloud data generated by the preliminary shooting, A user-specified target shooting range is set in response to a user operation that changes the range from the initial target shooting range; 2. The photographing apparatus according to claim 1, further comprising: a step of acquiring information regarding a progress status of the actual photographing based on the target photographing range designated by the user.

7. A photographing operation support method for performing a process by a processor to support a photographing operation of a user who holds a photographing device and moves within a measurement target location while photographing the measurement target location with the photographing device, in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on photographing data of the measurement target location, comprising: Based on the shooting data obtained in the preliminary shooting, the target shooting range for the actual shooting is set, A photographed range is acquired based on the photographing data acquired in the main photographing, and information regarding the progress of the main photographing is acquired based on the photographed range and the target photographing range; The method for supporting radiography work comprises displaying on a display unit a screen including information regarding the progress of the actual radiography.

8. A photography operation support program that causes a processor to execute a process for supporting a photography operation of a user who holds a photography device and moves within a measurement target location to photograph the measurement target location with the photography device, in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on photography data of the measurement target location, the process comprising: Based on the shooting data obtained in the preliminary shooting, the target shooting range for the actual shooting is set, A photographed range is acquired based on the photographing data acquired in the main photographing, and information regarding the progress of the main photographing is acquired based on the photographed range and the target photographing range; A photography operation support program that displays a screen including information regarding the progress of the actual photography on a display unit.

Images