Imaging device, imaging work assistance method, and imaging work assistance program

Abstract

[Problem] To enable a user to preliminarily confirm a site where various events causing deterioration in the accuracy of point group data occur when performing main imaging, and to obtain a highly accurate measurement result in one main imaging. [Solution] In order to perform three-dimensional measurement processing for generating three-dimensional spatial information of a measurement target location on the basis of imaging data of the measurement target location, this imaging device, which captures an image of the measurement target location: displays, on a display, work assistance information relating to imaging work in which a user captures an image of the measurement target location while moving therein; detects, on the basis of imaging data acquired by preliminary imaging in particular, a site requiring care during imaging when main imaging is performed; generates a confirmation map 31 in which the site requiring care during imaging is drawn over a map image 32 representing the measurement target location; and displays a screen including the confirmation map on the display.

Description

Photographing device, photographing work support method, and photographing work support program

[0001] The present disclosure relates to an imaging device that captures images of a measurement target location in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on imaging data of the measurement target location, as well as a method and program for supporting a user's imaging work using the imaging device.

[0002] Three-dimensional measurement techniques are known that generate three-dimensional spatial information (map data) about a measurement target location based on photographic data of the measurement target location. In recent years, the SLAM (Simultaneous Localization and Mapping) method has been attracting attention for this type of three-dimensional measurement. The SLAM method generates three-dimensional spatial information and the location information of a mobile object as a measurement result based on photographic data acquired by carrying a photographing device on the mobile object and photographing the measurement target location. In particular, three-dimensional measurement can be easily performed by a user acquiring photographic data of the measurement target location by using a handheld photographing device to photograph the measurement target location while moving around the measurement target location.

[0003] On the other hand, when a user photographs a measurement target location with a handheld imaging device while walking through the measurement target location, it is desirable to provide work assistance to the user that encourages the user to perform the imaging work appropriately so that highly accurate and stable three-dimensional measurements can be performed.From this perspective, a technology has been known in the past that displays multiple marks, such as arrows, that indicate the direction and speed of movement of the imaging device (sensor) held by the user to guide the user on how to proceed with the imaging work (see Patent Document 1).

[0004] Patent No. 6489566

[0005] In photography work in which a user holds a photography device and photographs a measurement target location while moving around the measurement target location, it may be difficult to obtain highly accurate measurement results with just one photograph, depending on the user's level of proficiency, the characteristics of the measurement target location, etc. In other words, various events that can reduce the accuracy of the point cloud data can occur during photography, and a high level of skill is required to proceed with the photography work while avoiding the occurrence of such events.

[0006] Therefore, there is a need for a technology that can notify users in advance of locations where various events that may cause a decrease in the accuracy of point cloud data occur as locations where they should be careful when photographing. However, conventional technologies have not taken such a need into consideration.

[0007] Therefore, the main purpose of the present disclosure is to provide an imaging device, a method for supporting imaging work, and a program for supporting imaging work that enable a user to check in advance the locations where various events that may cause a decrease in the accuracy of point cloud data will occur when performing actual imaging, thereby enabling highly accurate measurement results to be obtained with a single actual imaging session.

[0008] The photography device disclosed herein is a photography device that photographs a location to be measured in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the location to be measured based on photography data of the location to be measured, and is equipped with a device main body held by a user, a photography unit that is provided in the device main body and photographs the location to be measured, a display unit that displays work support information related to the photography work in which the user causes the photography unit to photograph the location to be measured while moving around the location to be measured, and a processor that controls the photography unit and the display unit, and is configured such that the processor detects areas that require attention for photography in the actual photography based on the photography data obtained in preliminary photography, generates a confirmation map in which the areas that require attention for photography are drawn on a map image representing the location to be measured, and displays a screen including the confirmation map on the display unit.

[0009] In addition, the photography support method disclosed herein is a photography support method in which a processor performs processing to support the photography work of a user who holds a photography device and moves around the measurement target location while having the photography device photograph the measurement target location, in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on photography data of the measurement target location.The method is configured to detect areas that require careful photography in the actual photography based on the photography data obtained in preliminary photography, generate a confirmation map in which the areas that require careful photography are drawn on a map image representing the measurement target location, and display a screen including the confirmation map on a display unit.

[0010] In addition, the photography support program disclosed herein is a photography support program that causes a processor to execute a process to support the photography work of a user who holds a photography device and moves around the measurement target location while having the photography device photograph the measurement target location, in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on photography data of the measurement target location.The photography support program is configured to detect areas that require careful photography in the actual photography based on the photography data acquired in preliminary photography, generate a confirmation map in which the areas require careful photography are drawn on a map image representing the measurement target location, and display a screen including the confirmation map on a display unit.

[0011] According to the present disclosure, when taking actual photographs, the user can check in advance the areas where caution is required for photography, i.e., the areas where various events that may cause a decrease in the accuracy of the point cloud data may occur, thereby enabling highly accurate measurement results to be obtained with a single actual photograph.

[0012] Explanatory diagram showing the situation of shooting work using the imaging device according to this embodiment, Explanatory diagram showing an example of a confirmation map including points to note during shooting, Explanatory diagram showing an example of a confirmation map including a recommended route, Explanatory diagram showing the shooting situation by the imaging device at a strongly reflecting location, Block diagram showing the schematic configuration of the imaging device, Block diagram showing the outline of the processing performed by the processor, Explanatory diagram showing the overlapping area used in the point group density evaluation process and the feature point evaluation process, Flow diagram showing the procedure of the processing performed during shooting by the processor, Flow diagram showing the procedure of the processing performed during preliminary shooting by the processor, Flow diagram showing the procedure of the processing performed during main shooting by the processor, Flow diagram showing the procedure of the processing performed when the preliminary shooting is completed by the processor, Explanatory diagram showing the screen displayed before shooting in the preliminary shooting mode, Explanatory diagram showing the screen displayed during shooting in the preliminary shooting mode, Explanatory diagram showing the screen displayed during shooting in the preliminary shooting mode, Explanatory diagram showing the screen displayed immediately after shooting in the preliminary shooting mode, Explanatory diagram showing the screen displayed at the time of confirmation after shooting in the preliminary shooting mode, Explanatory diagram showing the screen displayed at the time of confirmation after shooting in the preliminary shooting mode, Explanatory diagram showing the screen displayed at the time of confirmation after shooting in the preliminary shooting mode, Explanatory diagram showing the screen displayed at the time of confirmation after shooting in the preliminary shooting mode, Explanatory diagram showing the screen related to the progress setting displayed in the setting change mode, Explanatory diagram showing the screen related to the display setting of points to note during shooting displayed in the setting change mode, Explanatory diagram showing the screen related to the recommended route setting displayed in the setting change mode, Explanatory diagram showing the screen related to the recommended route setting displayed in the setting change mode, Explanatory diagram showing the screen displayed in the main shooting mode, Explanatory diagram showing the screen displayed in the main shooting mode, Explanatory diagram showing the screen displayed in the main shooting mode, Explanatory diagram showing the screen displayed in the main shooting mode, Explanatory diagram showing the screen displayed in the main shooting mode, Explanatory diagram showing the screen displayed in the main shooting mode, Explanatory diagram showing the screen displayed in the main shooting mode, Explanatory diagram showing the screen displayed in the main shooting mode, Explanatory diagram showing the screen displayed in the main shooting mode, Explanatory diagram showing the screen displayed in the main shooting mode

[0013] The first invention made to solve the above problem is a photography device that photographs a location to be measured in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the location to be measured based on photography data of the location to be measured, and is equipped with a device main body held by a user, a photography unit provided in the device main body that photographs the location to be measured, a display unit that displays work support information related to the photography work in which the user moves around the location to be measured and has the photography unit photograph the location to be measured, and a processor that controls the photography unit and the display unit, and the processor detects areas that require attention for photography in the actual photography based on the photography data obtained in preliminary photography, generates a confirmation map in which the areas that require attention for photography are drawn on a map image representing the location to be measured, and displays a screen including the confirmation map on the display unit.

[0014] This allows the user to check in advance the areas that require special attention when taking the actual photograph, i.e., the areas where various phenomena that may reduce the accuracy of the point cloud data may occur, thereby enabling highly accurate measurement results to be obtained with a single actual photograph.

[0015] In addition, in the second invention, the processor generates point cloud data as a preliminary shooting result based on the shooting data acquired in the preliminary shooting, generates an overhead point cloud image based on the point cloud data, and generates the confirmation map by superimposing an image representing the areas to be careful about when shooting on the map image including the overhead point cloud image.

[0016] This makes it possible to easily generate an appropriate map image showing the measurement target location and present a confirmation map that is easy for the user to understand.

[0017] In addition, in a third invention, the processor is configured to switch between a first shooting mode in which preliminary shooting is performed and a second shooting mode in which main shooting is performed in response to a user's operation to select a shooting mode.

[0018] This allows the user to select and execute the preliminary photography and the main photography as appropriate. For example, the user can skip the preliminary photography and execute the main photography as appropriate.

[0019] In addition, a fourth invention is configured such that, during actual photography, the processor causes the display unit to display a screen including a real-time photographed image output from the photography unit and the confirmation map on which the areas to be careful about when photographing are drawn.

[0020] This allows the user to proceed with the photographing work while checking the photographing status of the photographing unit as well as points along the route where caution is required for photographing.

[0021] In a fifth aspect of the present invention, the processor is configured to cause the display unit to display a screen including the confirmation map on which the points to be careful about in photography are depicted before the main photography after the preliminary photography.

[0022] This allows the user to check points that need to be taken into consideration when photographing, which are present in the measurement location, before starting actual photographing after the preliminary photographing is completed.

[0023] In addition, in a sixth aspect of the present invention, the processor generates simple point cloud data as a preliminary shooting result using processing that is simpler than that used in actual shooting, and detects the areas that require attention when shooting based on the simple point cloud data.

[0024] This makes it possible to reduce the waiting time of the user when point cloud data as a result of the preliminary photography is presented to the user after the preliminary photography is completed.

[0025] In addition, in a seventh invention, the processor is configured to generate the confirmation map in which at least one of strong reflection areas, areas lacking features, areas of sudden movement, and areas lacking point clouds is depicted as the areas to be careful of when photographing.

[0026] This allows the user to be presented with appropriate points to be careful about when taking the actual photograph.

[0027] Furthermore, the eighth invention is a photography support method in which a processor performs processing to support a user's photography work by having the user hold a photography device and move around the measurement target location while having the photography device photograph the measurement target location, in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the measurement target location based on photography data of the measurement target location.The method is configured to detect areas that require careful photography in the actual photography based on the photography data obtained in preliminary photography, generate a confirmation map in which the areas that require careful photography are drawn on a map image representing the measurement target location, and display a screen including the confirmation map on a display unit.

[0028] As with the first invention, this allows the user to check in advance the areas that require careful shooting when taking the actual photograph, i.e., the areas where various events that may cause a decrease in the accuracy of the point cloud data may occur, thereby enabling highly accurate measurement results to be obtained with a single actual photograph.

[0029] In addition, a ninth invention is a photography operation support program that causes a processor to execute a process to support the photography operation of a user who holds a photography device and moves around the measurement target location while having the photography device photograph the measurement target location, in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on photography data of the measurement target location.The program is configured to detect areas that require careful photography in the actual photography based on the photography data obtained in preliminary photography, generate a confirmation map in which the areas that require careful photography are drawn on a map image representing the measurement target location, and display a screen including the confirmation map on a display unit.

[0030] As with the first invention, this allows the user to check in advance the areas that require careful shooting when taking the actual photograph, i.e., the areas where various events that may cause a decrease in the accuracy of the point cloud data may occur, thereby enabling highly accurate measurement results to be obtained with a single actual photograph.

[0031] The tenth invention made to solve the above problem is an imaging device that photographs a measurement target location in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on imaging data of the measurement target location, the imaging device comprising: a device main body held by a user; an imaging unit provided in the device main body that photographs the measurement target location; a display unit that displays work support information related to the imaging work in which the user causes the imaging unit to photograph the measurement target location while moving around the measurement target location; and a processor that controls the imaging unit and the display unit, wherein the processor sets a target imaging range for the actual imaging based on the imaging data acquired in preliminary imaging, acquires the photographed range based on the imaging data acquired in actual imaging, acquires information regarding the progress of the actual imaging based on the photographed range and the target imaging range, and causes the display unit to display a screen including information regarding the progress of the actual imaging.

[0032] This allows the user to be presented with information about the progress of the actual photography, allowing the user to proceed with the photography work while understanding the progress of the actual photography.

[0033] In addition, an eleventh invention is configured such that the processor switches between a first shooting mode in which preliminary shooting is performed and a second shooting mode in which main shooting is performed in response to a user operation to select a shooting mode.

[0034] This allows the user to select and execute the preliminary photography and the main photography as appropriate. For example, the user can skip the preliminary photography and execute the main photography as appropriate.

[0035] In addition, a twelfth invention is configured such that, during the actual shooting, the processor causes the display unit to display a screen including a real-time captured image output from the shooting unit and an image visualizing the progress of the actual shooting.

[0036] This allows the user to check the shooting status based on the captured image during the actual shooting, and to proceed with the shooting work while checking the progress of the actual shooting based on an image that visualizes the progress.

[0037] In addition, in a thirteenth invention, the processor generates a map image representing the location to be measured as an image visualizing the progress of the actual photography, and the map image includes a first overhead point cloud image representing the target photography range generated by preliminary photography and a second overhead point cloud image representing the photographed range generated by the actual photography, and the first overhead point cloud image and the second overhead point cloud image are drawn in different ways.

[0038] This allows the progress to be expressed in terms of the ratio between the first and second aerial point cloud images on the map image showing the location to be measured, so that the user can distinguish between the areas that have been photographed and the areas that have not been photographed in the actual photography, and can also grasp the progress of the actual photography.

[0039] In addition, in a fourteenth aspect of the present invention, the processor is configured to generate an indicator image as an image visualizing the progress of the actual photography, in which the part indicating the photographed range changes as the actual photography progresses.

[0040] This allows the user to easily grasp the progress of the actual photography.

[0041] In addition, in a fifteenth aspect of the present invention, the processor sets an initial target shooting range based on point cloud data generated by preliminary shooting, sets a user-specified target shooting range in response to a user operation that changes the range from the initial target shooting range, and acquires information regarding the progress of the main shooting based on the user-specified target shooting range.

[0042] This allows the user to specify a range different from the initial target shooting range as a new target shooting range.

[0043] In addition, a sixteenth aspect of the present invention is a method for supporting photography operations in which a processor performs processing to support a user's photography operation by having the user hold a photography device and move around the measurement target location while having the photography device photograph the measurement target location, in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the measurement target location based on photography data of the measurement target location, the method is configured to set a target photography range for the actual photography based on the photography data acquired in preliminary photography, obtain a photographed range based on the photography data acquired in the actual photography, obtain information regarding the progress of the actual photography based on the photographed range and the target photography range, and display a screen including information regarding the progress of the actual photography on a display unit.

[0044] According to this, as in the tenth aspect of the invention, information regarding the progress of the actual photography is presented to the user, so that the user can proceed with the photography work while understanding the progress of the actual photography.

[0045] In addition, a seventeenth aspect of the present invention is a photography operation support program that causes a processor to execute a process to support a user's photography operation by having the photography device photograph the measurement target location while holding the photography device and moving within the measurement target location, in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on photography data of the measurement target location, and is configured to set a target photography range for the actual photography based on the photography data acquired in preliminary photography, obtain a photographed range based on the photography data acquired in the actual photography, obtain information regarding the progress of the actual photography based on the photographed range and the target photography range, and display a screen including information regarding the progress of the actual photography on a display unit.

[0046] According to this, as in the tenth aspect of the invention, information regarding the progress of the actual photography is presented to the user, so that the user can proceed with the photography work while understanding the progress of the actual photography.

[0047] The eighteenth invention made to solve the above problem is an imaging device that photographs a measurement target location in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on imaging data of the measurement target location, the imaging device comprising: a device main body held by a user; an imaging unit provided in the device main body that photographs the measurement target location; a display unit that displays work support information related to the imaging work in which the user moves around the measurement target location and has the imaging unit photograph the measurement target location; and a processor that controls the imaging unit and the display unit, wherein the processor detects areas that need to be photographed in the actual imaging based on the imaging data obtained in preliminary imaging, displays a screen on the display unit that includes an operation unit where the user operates to specify display conditions for the areas that need to be photographed, generates a confirmation map in which the areas that need to be photographed are drawn on a map image representing the measurement target location according to the specified display conditions, and displays a screen including the confirmation map on the display unit.

[0048] This allows the user to appropriately change the display conditions for the points of caution for photography as needed on the confirmation map used by the user to check the points of caution for photography in the actual photography. Note that the operation unit may be a unit that switches whether or not the points of caution for photography are displayed on the confirmation map, for example, a check box.

[0049] In addition, a 19th invention is configured such that the processor displays on the display unit a screen including a real-time captured image output from the photographing unit, the confirmation map, and the operation unit during actual photographing.

[0050] This allows the user to view the confirmation map by changing the display conditions of the areas to be careful about when taking a photograph, which are drawn on the confirmation map during actual photography. In this case, the display conditions for each type of areas to be careful about when taking a photograph (for example, areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements) may be specified all at once.

[0051] In a twentieth aspect of the present invention, the processor is configured to cause the display unit to display a screen including the confirmation map and the operation unit at the time of confirmation before the main photographing after the preliminary photographing.

[0052] This allows the user to view the confirmation map by changing the display conditions of the areas to be careful about when shooting, which are drawn on the confirmation map after the preliminary shooting and before the actual shooting. In this case, the display conditions may be set individually for each type of areas to be careful about when shooting (for example, areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements).

[0053] In addition, a 21st invention is configured such that the processor generates a recommended route for the actual shooting based on a target shooting range based on point cloud data generated in the preliminary shooting, and displays a screen including the recommended route on the display unit.

[0054] This allows the actual photography to be carried out efficiently and without omission. In this case, for example, a route that covers the entire target photography range and has as short a travel distance as possible may be generated as the recommended route.

[0055] In a twenty-second aspect of the present invention, the processor generates the recommended route in accordance with an instruction from a user so as to avoid the important photography location.

[0056] According to this, when it is difficult to improve the shooting environment or shooting method for a point requiring careful shooting, a recommended route is generated that avoids the point requiring careful shooting, thereby preventing the accuracy of the measurement results (point cloud) from decreasing due to passing through the point requiring careful shooting during actual shooting without improving the shooting environment or shooting method.

[0057] In addition, in a 23rd invention, the processor is configured to cause the display unit to display, at the time of actual photographing, a screen including the confirmation map on which the recommended route is drawn and the photographed image on which an image showing a course based on the recommended route is superimposed.

[0058] This allows the user to check their route from a bird's-eye view by viewing the recommended route drawn on the confirmation map, and at the same time, they can directly check their route by viewing an image showing their route superimposed on the captured image.

[0059] In addition, in a 24th invention, the processor is configured to generate the confirmation map in which at least one of strong reflection areas, areas with insufficient features, areas with sudden movements, and areas with insufficient point clouds is depicted as the areas to be careful of when photographing.

[0060] This allows the user to be presented with appropriate points to be careful about when taking the actual photograph.

[0061] In addition, a 25th invention is a photography support method in which a processor performs processing to support a user's photography work by having the photography device photograph the measurement target location while holding the photography device and moving within the measurement target location, in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the measurement target location based on photography data of the measurement target location.The method is configured to detect areas that need to be photographed in the actual photography based on the photography data obtained in preliminary photography, display on a display unit a screen including an operation unit where the user operates to specify display conditions for the areas that need to be photographed, generate a confirmation map in which the areas that need to be photographed are drawn on a map image representing the measurement target location according to the specified display conditions, and display a screen including the confirmation map on the display unit.

[0062] According to this, as in the eighteenth invention, the user can appropriately change the display conditions of points to be careful about when photographing, as needed, on the confirmation map for the user to check points to be careful about when photographing.

[0063] In addition, a 26th invention is a photography operation support program that causes a processor to execute a process to support the photography work of a user who holds a photography device and moves around the measurement target location while having the photography device photograph the measurement target location, in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on photography data of the measurement target location.The program is configured to detect areas that need to be photographed in the actual photography based on the photography data obtained in preliminary photography, display on a display unit a screen including an operation unit where the user operates to specify display conditions for the areas that need to be photographed, generate a confirmation map in which the areas that need to be photographed are drawn on a map image representing the measurement target location according to the specified display conditions, and display a screen including the confirmation map on the display unit.

[0064] According to this, as in the eighteenth invention, the user can appropriately change the display conditions of points to be careful about when photographing, as needed, on the confirmation map for the user to check points to be careful about when photographing.

[0065] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0066] FIG. 1 is an explanatory diagram showing a situation in which a user performs a photographing operation using a photographing device 1 according to this embodiment.

[0067] The photographing device 1 includes a device main body 11 and a sensor unit 12. The sensor unit 12 includes a visible light camera 21 (photographing unit). The visible light camera 21 is a monocular camera that detects visible light to photograph a subject and outputs a photographed image, for example, an RGB color image. The photographing device 1 can be configured as a tablet terminal or a notebook PC.

[0068] A user (worker) walks through a location to be measured while holding the device body 11 of the image capture device 1. At this time, the location to be measured is captured by the visible camera 21 of the image capture device 1, and image capture data is sequentially acquired for each point in the location to be measured. The image capture data includes data such as the captured image, the sensor detection results, and the time of capture.

[0069] In this embodiment, the image capture device 1 serves as a three-dimensional measurement device. That is, the image capture device 1 performs three-dimensional measurement processing based on images of each point sequentially acquired by the visible camera 21, and generates three-dimensional spatial information about the measurement target location. In the three-dimensional measurement processing, point cloud data (environmental map) is generated as three-dimensional spatial information about the measurement target location using the SLAM method.

[0070] 1 shows an example in which a user holding the device main body 11 walks through the measurement target location to photograph the measurement target location, but the measurement target location may also be photographed using a self-propelled robot or the like equipped with the photographing device 1. In this embodiment, work assistance is provided to the user, and when the user himself / herself holds the device main body 11 and performs the photographing work, work assistance is provided to the user. On the other hand, when a self-propelled robot photographs, work assistance is provided to the user operating the robot.

[0071] In addition, in this embodiment, the photographing device 1 equipped with a sensor unit 12 such as a visible camera 21 performs three-dimensional measurement processing, but the sensor unit 12 may be configured as a photographing device 1 independent of a PC (information processing device) that performs three-dimensional measurement processing.

[0072] The user also takes a photograph of the measurement target location while walking through the measurement target location using the handheld camera 1. Therefore, the photographing operation includes actions such as walking (moving) while holding the camera 1 and pointing the camera 1 in an appropriate direction.

[0073] Next, we will explain the points to be careful about when taking photos and the confirmation map 31. Fig. 2 is an explanatory diagram showing an example of the confirmation map 31 including the points to be careful about when taking photos. Fig. 3 is an explanatory diagram showing an example of the confirmation map 31 including the recommended route. Fig. 4 is an explanatory diagram showing the situation when taking photos by the photographing device 1 in a strongly reflective area.

[0074] In a photographing operation in which a user photographs a measurement target location while holding the photographing device 1 and moving around the measurement target location, it may be difficult to obtain highly accurate measurement results with just one photograph, depending on the user's level of proficiency and the characteristics of the measurement target location. In other words, various events that can reduce the accuracy of the point cloud data may occur during photographing, and a high level of skill is required to proceed with the photographing operation while avoiding such events. Therefore, an inexperienced user will have to repeat the photographing operation many times through trial and error.

[0075] Therefore, in this embodiment, preliminary photography is performed in addition to the main photography (regular photography) for obtaining three-dimensional spatial information (point cloud data) as a measurement result, and based on the results of the preliminary photography, areas to be careful about in the main photography, i.e., areas that require the user's attention in the main photography, are detected and presented to the user. This makes it possible to improve the accuracy of the main photography results, i.e., the point cloud data obtained in the main photography.

[0076] Areas requiring caution when photographing include areas with strong reflections, areas lacking features, areas with sudden movements, and areas with insufficient point clouds. Areas requiring caution when photographing also include areas where an event that could cause a decrease in the accuracy of the point cloud data actually occurred during preliminary photography, and areas where an event that could cause a decrease in the accuracy of the point cloud data is likely to occur.

[0077] For example, consider a case where lighting is installed on the ceiling and the floor is made of a reflective material (see FIG. 4). In this case, the lighting light is reflected by the floor, and this reflected light enters the image capture device 1, causing the reflected light from the floor to appear in the captured image. At this time, the reflected light from the floor is extracted as a feature point, and position estimation is performed based on this feature point. However, as the image capture device 1 moves, the position of the reflected light from the floor also moves. As a result, position estimation based on the feature points is not performed appropriately, and the accuracy of the point cloud data as a measurement result is significantly reduced.

[0078] Therefore, in this embodiment, locations where strong reflected light is estimated to appear in the image captured by the visible light camera 21 or locations where strong reflected light actually appeared in the captured image in preliminary shooting are detected as strong reflection locations and presented to the user. Furthermore, guidance (such as suggestions for improvement) regarding the strong reflection locations is provided to the user. Specifically, if the reflected light is estimated to be caused by indoor lighting, the user is prompted to improve the shooting environment regarding lighting. For example, the user is prompted to dim the lighting. Furthermore, if the reflected light is estimated to be caused by external light entering through a window, the user is prompted to improve the shooting environment regarding external light. For example, the user is prompted to close the curtains. Note that if the reflected light is likely to be caused by both indoor lighting and external light entering through a window, the user may also be prompted to dim the lighting or close the curtains. This allows the user to appropriately improve the shooting environment regarding the strong reflection locations.

[0079] Furthermore, if an area containing many areas without distinctive patterns or objects, such as walls, floors, or ceilings, is photographed, feature point extraction will fail and fewer feature points will be extracted, preventing proper self-position estimation and reducing the accuracy of the point cloud data.

[0080] Therefore, in this embodiment, areas where feature points could not be sufficiently extracted from the image captured by the visible camera 21 are detected as feature-deficient areas and presented to the user. Furthermore, guidance regarding the feature-deficient areas (such as suggestions for improvement) is provided to the user. For example, if the feature-deficient areas are caused by the shooting environment of the measurement target location, the user is prompted to improve the shooting environment. For example, the user is prompted to place something with texture (e.g., distinctive wallpaper). Furthermore, if the feature-deficient areas are caused by the user's shooting method, the user is prompted to improve the shooting method. For example, the user is prompted to shoot with an angle of view that includes distinctive patterns or objects.

[0081] Furthermore, if the user moves the image capturing device 1 too quickly while capturing an image, errors will occur in the position estimation and the point cloud will not be generated appropriately, resulting in a decrease in the accuracy of the point cloud data.

[0082] Therefore, in this embodiment, a location where the user actually made a sudden movement during preliminary shooting or a location where it is estimated that the user is likely to make a sudden movement is detected as a sudden movement location and presented to the user. For example, a location where a sudden acceleration / deceleration state in which the acceleration generated in the image capture device 1 falls outside a predetermined range or a sudden rotation state in which the angular velocity generated in the image capture device 1 falls outside a predetermined range is detected as a sudden movement location and presented to the user. Furthermore, a location where the space for the user to pass through is narrow or where the route the user takes makes a sudden turn is detected as a sudden movement location and presented to the user because it is likely that the user will make a sudden movement. Furthermore, guidance regarding the sudden movement location (such as suggestions for improvement) is presented to the user. For example, the user is urged not to make a sudden movement.

[0083] Furthermore, areas hidden by the shadow of an object, such as the back of a desk monitor, will be missed from being photographed, and areas with insufficient photography will not generate a point cloud with sufficient density, resulting in a decrease in the accuracy of the point cloud data.Furthermore, if the user does not move the photographing device 1 appropriately while photographing, the point cloud will not be generated with sufficient density, resulting in a decrease in the accuracy of the point cloud data.

[0084] Therefore, in this embodiment, areas where a point cloud was not generated because they were not photographed in the preliminary photography (unphotographed areas), or areas where a point cloud was not generated with sufficient density despite being photographed, are detected as areas with insufficient point clouds and presented to the user.

[0085] 2 and 3, in this embodiment, a confirmation map 31 is generated and presented to the user to allow the user to check areas requiring attention before or during actual photography. In the confirmation map 31, the results of the preliminary photography, i.e., an overhead point cloud image 35 generated from the point cloud data obtained by the preliminary photography, is used as a map image 32 representing the measurement target location, and an image representing the areas requiring attention for photography is superimposed on the map image 32. The overhead point cloud image 35 is an image (rendering) of each point of the point cloud data as viewed from an aerial viewpoint.

[0086] In the example shown in FIG. 2A , strong reflection areas, feature-deficient areas, sudden movement areas, and point cloud-deficient areas are detected as areas requiring caution when photographing, and mark images 41, 42, and 43 representing the strong reflection areas, feature-deficient areas, and sudden movement areas, and an area image 44 representing the point cloud-deficient areas are drawn on the confirmation map 31. The mark images 41, 42, and 43 are drawn in different shapes depending on their type (strong reflection areas, feature-deficient areas, and sudden movement areas), but they may also be drawn in the same shape with a different drawing aspect (e.g., color) other than the shape. The area image 44 is drawn in a predetermined color (e.g., red). The area image 44 may also be drawn in a pattern (design) or a semi-transparent color. Note that the mark images 41, 42, and 43 and the area image 44 are not limited to mark images and area images, respectively. For example, the mark images 41, 42, and 43 may be drawn in a predetermined color different from the area image 44 as area images indicating areas requiring caution when photographing, or may be drawn in a pattern (design) or a semi-transparent color.

[0087] In this example, areas that were not captured because they were hidden by the shadow of an object, such as the back of a monitor on a desk, are detected as areas that lack a point cloud. Additionally, the area on the periphery of the measurement target location is detected as an area that lacks a point cloud. This area appears in the captured image, but because the point cloud density is low, it is detected as an area that lacks a point cloud.

[0088] In the example shown in Fig. 2(B), a point cloud insufficient area is detected as an area requiring careful photography, and an area image 44 representing the point cloud insufficient area is drawn on the confirmation map 31. In this example, similar to the example shown in Fig. 2(A), an area on the periphery of the measurement target location is detected as a point cloud insufficient area.

[0089] Here, as in the example shown in Fig. 2(A), in many cases the user does not need to pay particular attention to the point cloud insufficient areas on the periphery of the measurement target location when taking the actual photograph. On the other hand, as in the example shown in Fig. 2(B), when the point cloud insufficient areas extend from the periphery of the measurement target location toward the inside, the user needs to pay particular attention when taking the actual photograph.

[0090] In this embodiment, strong reflection areas, areas lacking features, areas with sudden movements, and areas lacking point clouds are detected and presented to the user as areas requiring attention during photography, but in addition, blurring areas, i.e., areas estimated to be highly likely to experience blurring (out-of-focus) in the image captured by the visible camera 21, or areas where blurring actually occurred in the captured image during preliminary photography, may also be detected and presented to the user. If blurring occurs in the captured image, point cloud generation will not be performed appropriately, but by presenting the areas where blurring occurs to the user before actual photography, it is possible to prevent blurring from occurring in the captured image.

[0091] In this case, an evaluation value (blur evaluation value) that indicates the occurrence of blur in the captured image is calculated, and the evaluation value is compared with a predetermined threshold to determine the location where blur has occurred. The blur evaluation value is calculated by image analysis of the captured image. The blur evaluation value is also calculated based on the exposure time of the visible light camera 21 and the detection data (acceleration, angular velocity) of the IMU 23. The blur evaluation value is also calculated based on the shooting environment, such as brightness.

[0092] In this embodiment, a target imaging range to be the target of actual imaging is set as shown in Fig. 3. The target imaging range is set based on the results of preliminary imaging, i.e., the point cloud data acquired in the preliminary imaging. For example, the range in which a point cloud is generated in the preliminary imaging is set as the target imaging range. In this case, if there is an unphotographed area (an area in which a point cloud is not generated) surrounded by an imaged area (an area in which a point cloud is generated), the unphotographed area may be included in the target imaging range as an unphotographed area.

[0093] In this embodiment, a recommended route is presented to the user to encourage them to take during actual photography. The recommended route is generated based on the target photography range. Specifically, a route that covers the entire target photography range and has as short a travel distance as possible is generated as the recommended route.

[0094] In the confirmation map 31, a route image 51 representing the recommended route is superimposed on a map image 32 consisting of an overhead point cloud image 35. A line representing the recommended route, an arrow representing the direction of travel, a mark representing the start position, a mark representing the current position, and a mark representing the end position are drawn on the route image 51 representing the recommended route.

[0095] Here, when a point of caution for photography is presented to the user, the user can be prompted to improve the photography environment or photography method for the point of caution, allowing the user to take appropriate measures to improve the photography environment or photography method, thereby improving the accuracy of the measurement results (point cloud). However, there are cases where it is difficult to improve the photography environment or photography method for the point of caution for photography.

[0096] In this case, in this embodiment, a recommended route is generated in accordance with a user's instruction to avoid the location where shooting is required. This makes it impossible to obtain a point cloud generated by shooting at the location where shooting is required, but it is possible to avoid a decrease in the accuracy of the measurement results (point cloud) caused by passing through the location where shooting is required without improving the shooting environment or shooting method.

[0097] The example shown in Fig. 3(A) is a case where cautionary shooting locations are not avoided, and a route that covers the entire target shooting range and has the shortest possible travel distance is generated as the recommended route, regardless of locations that lack features. On the other hand, the example shown in Fig. 3(B) is a case where strong reflection locations are avoided, and a recommended route that avoids strong reflection locations is generated.

[0098] Next, we will explain the general configuration of the photographing device 1. Fig. 5 is a block diagram showing the general configuration of the photographing device 1. Fig. 6 is a block diagram showing an overview of the processing performed by the processor 16. Fig. 7 is an explanatory diagram showing overlapping regions used in the feature point evaluation process P22 and the point cloud density evaluation process P24.

[0099] As shown in FIG. 5, the photographing device 1 includes a sensor unit 12, a display 13 (display unit), an input device 14, a memory 15 (storage unit), and a processor 16 (CPU).

[0100] In addition to the visible camera 21, the sensor unit 12 also includes a depth camera 22 and an IMU (Inertial Measurement Unit) 23.

[0101] The depth camera 22 is a stereo camera that detects infrared light to capture an image of a subject and outputs depth information (distance image) as the detection result. The distance to the subject can be measured based on the detection result of the depth camera 22. Note that instead of a stereo camera, a sensor capable of acquiring depth information using another method, such as LiDAR, may be used as the depth camera 22.

[0102] The IMU 23 (detector) detects the motion state of the image capturing device 1 itself, specifically, three-dimensional angular velocity and acceleration. Based on the detection results of the IMU 23, the movement amount and rotation amount of the image capturing device 1 are measured.

[0103] The visible camera 21, the depth camera 22, and the IMU 23 may not be integrated into a sensor unit. Alternatively, the depth camera 22 and the IMU 23 may be omitted, and only the visible camera 21 may be provided. Alternatively, the visible camera 21 may be provided in addition to either the depth camera 22 or the IMU 23.

[0104] The display 13 presents various information related to the photographing operation to the user, and displays screens 101 and 201 for the preliminary photographing mode (see Figures 12 and 16, etc.), screens 301, 401 and 501 for the setting change mode (see Figures 20, 21 and 22, etc.), and screen 601 for the main photographing mode (see Figure 24, etc.).

[0105] The input device 14 is used by the user to perform input operations. The input device 14 may be a keyboard, a mouse, a touchpad, a touch panel, etc. If the imaging device 1 is configured as a tablet terminal, a touch panel display is provided in which the touch panel as the input device 14 and the display panel as the display 13 are integrated.

[0106] The memory 15 stores programs and the like to be executed by the processor 16. The memory 15 also stores images captured by the visible camera 21, distance information from the depth camera 22, and detection data from the IMU 23. The memory 15 also stores measurement results (point cloud data) generated by the processor 16.

[0107] The processor 16 performs various processes by executing programs stored in the memory 15. In this embodiment, the processor 16 performs a three-dimensional measurement process P1, a preliminary photography evaluation process P2, a process P3 for detecting areas to be careful about when photographing, a support information generation process P4, a display information generation process P5, and a display process P6.

[0108] In this embodiment, the photographing device 1 performs three-dimensional measurement processing P1, preliminary photographing evaluation processing P2, photographing point detection processing P3, support information generation processing P4, and display information generation processing P5, but all or part of these processes may be performed by a server device (not shown) that can communicate with the photographing device 1.

[0109] As shown in FIG. 6, the three-dimensional measurement process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0110] In the feature extraction process P11, the processor 16 extracts feature information (feature points, etc.) from the image (frame) captured by the visible camera 21.

[0111] In tracking process P12, processor 16 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition related to the position and orientation of the image capture device 1, and updates the position and orientation trajectory data based on the amount of transition. Note that the position and orientation trajectory data is obtained by tracking the position and orientation of the image capture device 1, and includes the tracking results of the position and orientation at the time of capturing each captured image (frame), i.e., information regarding the position and orientation of the image capture device 1 at each time of capturing.

[0112] In the position and orientation correction process P13, the processor 16 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, the position and orientation trajectory data is corrected so as to compensate for the measurement results of locations with few features, such as walls and ceilings.

[0113] In the point cloud generation process P14, the processor 16 generates point cloud data based on the distance information of the depth camera 22 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13.

[0114] Here, during preliminary photography, simplified processing is performed compared to main photography, and simple point cloud data is generated as the preliminary photography result. This reduces the user's waiting time when the point cloud data as the preliminary photography result is presented to the user after the preliminary photography is completed. Furthermore, since the processing load on the processor 16 is reduced, battery consumption is reduced and photography can be performed for a long time. Note that in a configuration in which the point cloud generation process P14 and the like are performed on a server device separate from the photography device 1, the amount of data communicated is reduced, thereby reducing the communication load.

[0115] Furthermore, the point cloud generation process P14 performed during preliminary photography may be even simpler depending on the purpose of the preliminary photography. For example, if the point cloud data generated during preliminary photography is used only to acquire the target photography range and not to detect areas of interest for photography or generate a recommended route based on the point cloud data, a point cloud sufficient to grasp the general positional relationships of objects may be sufficient, and therefore even simpler processing may be used. In this case, for example, the load on the processor 16 can be further reduced by thinning out the photography data, i.e., by performing processing at a pseudo-lower frame rate.

[0116] Furthermore, the point cloud generation process P14 performed during preliminary photography reduces the load on the processor 16 if only a small number of point clouds need to be generated, and therefore the working time for preliminary photography can be shortened by making the movement speed of the photography device 1 faster than that for the main photography.

[0117] The preliminary photography evaluation process P2 includes a reflection evaluation process P21, a feature point evaluation process P22, a sudden movement evaluation process P23, and a point cloud density evaluation process P24.

[0118] In the reflection evaluation process P21, the processor 16 obtains an evaluation value (reflection evaluation value) representing the intensity of the reflected light appearing in the captured image based on the detection data of the IMU 23 and the transition amount regarding the position and posture of the imaging device 1 obtained in the tracking process P12.

[0119] Specifically, the processor 16 compares the amount of movement of the image capture device 1 based on the detection data of the IMU 23 with the amount of movement (transition amount) of the image capture device 1 acquired in the tracking process P12 to obtain a reflection evaluation value that indicates the degree of inconsistency between the two. Here, the amount of movement of the image capture device 1 based on the detection data of the IMU 23 is determined in the position and orientation correction process P13. Furthermore, in the tracking process P12, the amount of movement (transition amount) of the image capture device 1 is determined by comparing feature points extracted from the current captured image with feature points extracted from a previously captured image. In this case, the reflection evaluation value indicates the degree of abnormality in the position estimation based on the captured image, i.e., the magnitude of the error occurring in the position estimation based on the feature points extracted from the captured image, and indirectly indicates the appearance of strong reflected light in the captured image.

[0120] In another embodiment, the processor 16 acquires a reflection evaluation value that represents the appearance of reflected light in the captured image. That is, the size of the area in the captured image where the brightness value is locally high, specifically, the number of pixels whose brightness value deviates from that of surrounding pixels by a predetermined threshold or more, is acquired as the reflection evaluation value. In this case, the reflection evaluation value directly represents the appearance of strong reflected light in the captured image. Note that the reflection evaluation value in this case can also be used as a criterion for detecting shine or blown-out highlights in the captured image.

[0121] In the feature point evaluation process P22, the processor 16 counts the feature points within the target range based on the extraction results of the feature extraction process P11, and obtains an evaluation value (number of feature points) that indicates the extraction status of the feature points.

[0122] The number of feature points may be the number of feature points extracted in one captured image (one frame), or may be the number of feature points extracted in the overlapping area between the capture range at the current capture time t and the capture range at the previous capture time t-1 (see FIG. 7).

[0123] In the sudden movement evaluation process P23, the processor 16 acquires an evaluation value (sudden movement evaluation value) that indicates the degree of suddenness of the movement of the user moving the image capture device 1, based on the detection data (acceleration, angular velocity) of the IMU 23. The degree of suddenness of the movement indicates the degree of suddenness of the change in the position and orientation of the image capture device 1, and specifically indicates the degree of acceleration (including deceleration) that has occurred in the image capture device 1, or the degree of angular velocity that has occurred in the image capture device 1.

[0124] In another aspect, in the sudden movement evaluation process P23, the processor 16 acquires a sudden movement evaluation value based on the amount of transition regarding the position and orientation of the image capture device 1 acquired in the tracking process P12. Here, the amount of transition represents the change in the position and orientation of the image capture device 1, and the suddenness of the movement can be evaluated from the magnitude of the amount of transition.

[0125] In another aspect, in the sudden movement evaluation process P23, the processor 16 acquires the movement route (traffic line) of the user (photography device 1) during preliminary photography, and acquires a sudden movement evaluation value as an evaluation value representing the likelihood that the user will make a sudden movement based on the movement route and the positioning status of surrounding objects estimated from the generated point cloud.

[0126] In the point cloud density evaluation process P24, the processor 16 counts the number of point clouds within the target area based on the point cloud data generated in the point cloud generation process P14, and obtains an evaluation value (point cloud density evaluation value) that indicates the state of point cloud generation. The point cloud density evaluation value indicates whether real-time point cloud generation is being performed appropriately. The smaller the point cloud density evaluation value, the lower the point cloud density.

[0127] The point cloud density evaluation value may be calculated from the number of point clouds generated in a predetermined unit time (for example, one second) (total number of points generated in that unit time) and the size of the imaging range of the visible camera 21, as in the following formula. In this case, if the number of point clouds generated in the entire imaging range of the visible camera 21 is small, the point cloud density evaluation value will be small. Point cloud density evaluation value = (number of point clouds generated in a predetermined unit time) / (imaging range)

[0128] The point cloud density evaluation value may also be the number of generated points (the total number of points included in the overlapping area) in the overlapping area (see FIG. 7 ) between the shooting range at the current shooting time t and the shooting range at the previous shooting time t-1. In this case, too, the smaller the point cloud density evaluation value, the lower the point cloud density. Furthermore, in addition to the small number of point clouds generated in the shooting range of the visible camera 21, the smaller the overlapping area between the shooting range at the current shooting time t and the shooting range at the previous shooting time t-1, the smaller the point cloud density evaluation value. It is also possible that the point cloud density may increase as the same location is repeatedly photographed. Therefore, the point cloud density evaluation value may be calculated taking into account the existing point cloud generation status at that location. In this case, the number of generated points is not just the point cloud generated at the current shooting time t, but the total number of all point clouds included in the overlapping area.

[0129] In another aspect, in the point cloud density evaluation process P24, the processor 16 calculates normal vectors perpendicular to the surface of the object by surface approximation or meshing of the point cloud data, and detects areas where the normal vectors are not continuous as areas where the point cloud is lacking.

[0130] In another aspect, in the point cloud density evaluation process P24, the processor 16 calculates the density for each position of the point cloud data, and detects areas where a point cloud exists but the density of the point cloud is extremely low (for example, when the density is within an interval between two predetermined thresholds) as areas where the point cloud is lacking.

[0131] The processing P3 for detecting areas requiring attention in photography includes processing P31 for detecting strong reflection areas, processing P32 for detecting areas lacking features, processing P33 for detecting sudden movement areas, and processing P34 for detecting areas lacking point clouds.

[0132] In the strong reflection area detection process P31, the processor 16 detects strong reflection areas, i.e., areas where significant reflected light appears in the image captured by the visible light camera 21, based on the reflection evaluation value acquired in the reflection evaluation process P21. Specifically, the reflection evaluation value is defined so that the larger the value, the stronger the reflection, and an area is determined to be a strong reflection area when the reflection evaluation value is greater than a predetermined threshold. When a strong reflection area is detected, the position of the image capture device 1 at that time is stored in the memory 15 as the position of the strong reflection area. The strong reflection area detection process P31 is performed during preliminary image capture.

[0133] In the feature-lacking portion detection process P32, the processor 16 detects feature-lacking portions, i.e., portions where the feature points necessary to generate point cloud data with sufficient accuracy could not be extracted sufficiently, based on the number of feature points acquired in the feature point evaluation process P22. Specifically, a portion is determined to be feature-lacking when the number of feature points is less than a predetermined threshold. When a feature-lacking portion is detected, the position of the image capture device 1 at that time is stored in the memory 15 as the position of the feature-lacking portion. At this time, because the reliability of the position estimation result based on the feature points is low, the position estimation result based on the detection data of the IMU 23 is adopted. Note that the feature-lacking portion detection process P32 is performed during preliminary image capture.

[0134] In the sudden movement location detection process P33, the processor 16 detects a sudden movement location, i.e., a location where the user actually made a sudden movement during preliminary image capture, or a location where it is estimated that the user is likely to make a sudden movement, based on the sudden movement evaluation value acquired in the sudden movement evaluation process P23. Specifically, the sudden movement evaluation value is defined so that the larger the value, the more sudden the movement, and a sudden movement location is determined to be a sudden movement location when the sudden movement evaluation value is greater than a predetermined threshold. When a sudden movement location is detected using a method in which the sudden movement evaluation value is acquired based on the detection data of the IMU 23 or the transition amount acquired in the tracking process P12, the position of the image capture device 1 at that time is stored in the memory 15 as the location of the sudden movement location. On the other hand, in a method in which the sudden movement evaluation value is acquired based on the movement route (traffic line) of the image capture device 1 during preliminary image capture, a location determined to be a sudden movement location based on the movement route and the location of surrounding objects estimated from the generated point cloud is stored in the memory 15 as the location of the sudden movement location.

[0135] In a method in which the sudden movement evaluation value is acquired based on the detection data of the IMU 23 or the transition amount acquired in the tracking process P12, the sudden movement location detection process P33 is performed during the preliminary shooting. On the other hand, in a method in which the sudden movement evaluation value is acquired based on the movement route (traffic line) of the image capturing device 1 during the preliminary shooting, the sudden movement location detection process P33 is performed when the preliminary shooting is completed.

[0136] In the point cloud insufficient portion detection process P34, the processor 16 detects a point cloud insufficient portion, i.e., a portion where a sufficient number of point clouds were not generated in the preliminary image capture, based on the point cloud density evaluation value acquired in the point cloud density evaluation process P24. Specifically, a point cloud insufficient portion is determined when the point cloud density evaluation value is less than a predetermined threshold. When a point cloud insufficient portion is detected using a method in which the point cloud density evaluation value is calculated by counting the number of point clouds generated in real time (which may also take into account the existing point cloud generation status at that portion), the position of the image capture device 1 at that time is stored in memory 15 as the position of the point cloud insufficient portion. On the other hand, in a method in which a portion where normal vectors are discontinuous is detected or a portion where a point cloud exists but the point cloud density is extremely low (e.g., when the density is within a range between two predetermined thresholds), the position determined to be a discontinuous portion or a portion where the density is extremely low is stored in memory 15 as the position of the point cloud insufficient portion.

[0137] In a method in which the point cloud density evaluation value is calculated by counting the number of generated point clouds in real time (which may take into account the existing point cloud generation status at that location), the point cloud insufficiency location detection process P34 is performed during the preliminary imaging. On the other hand, in a method in which a location where normal vectors are not continuous is detected or a location where a point cloud exists but the point cloud density is extremely low (for example, when the density is within an interval between two predetermined thresholds), the point cloud insufficiency location detection process P34 is performed when the preliminary imaging is completed.

[0138] The support information generation process P4 includes a target shooting range acquisition process P41, a progress rate acquisition process P42, a recommended route generation process P43, a progress status image generation process P44, a map image generation process P45, a route image generation process P46, and a confirmation map generation process P47.

[0139] In the target shooting range acquisition process P41, the processor 16 sets a target shooting range to be the target of main shooting based on the point cloud data generated in the point cloud generation process P14 during preliminary shooting. The processor 16 also sets a user-specified target shooting range in response to a user operation that changes the range from the initial target shooting range set based on the point cloud data.

[0140] In the progress rate acquisition process P42, the processor 16 acquires the photographed range for the actual photography based on the point cloud data generated in the point cloud generation process P14 up to now in the actual photography, and acquires the progress rate of the photography for the actual photography based on this photographed range and the target photography range acquired in the target photography range acquisition process P41. The progress rate is the ratio of the photographed range to the target photography range.

[0141] In the recommended route generation process P43, the processor 16 generates a recommended route for the actual shooting, i.e., a route that is recommended for the user to take during the actual shooting, based on the target shooting range acquired in the target shooting range acquisition process P41. Specifically, a route that covers the entire target shooting range and has as short a travel distance as possible is generated as the recommended route. Furthermore, in the recommended route generation process P43, a recommended route is generated in accordance with the user's instructions so as to avoid areas that require caution when shooting, i.e., areas with strong reflections, areas with insufficient features, and areas with sudden movements.

[0142] In the progress status image generation process P44, the processor 16 generates an image that visualizes the progress status based on the progress rate acquired in the progress rate acquisition process P42. Specifically, an indicator image 621 (see FIG. 25, etc.) is generated. The indicator image 621 is, for example, a progress bar, and is drawn so that the portion indicating the captured range changes as the actual shooting progresses.

[0143] In the map image generation process P45, the processor 16 generates an overhead point cloud image that serves as the map image 32 representing the measurement target location on the confirmation map 31, based on the point cloud data generated in the point cloud generation process P14. Specifically, after photographing in the preliminary photographing mode, an overhead point cloud image 35 from the preliminary photographing, i.e., an overhead point cloud image, is generated based on the point cloud data generated in the point cloud generation process P14 during the preliminary photographing (see FIG. 16, etc.). During photographing in the main photographing mode, an overhead point cloud image 36 from the main photographing, i.e., an overhead point cloud image, is generated based on the point cloud data generated in the point cloud generation process P14 from the start of the main photographing to the present (see FIG. 24, etc.).

[0144] In the route image generation process P46, the processor 16 generates a route image 51 representing the recommended route based on the recommended route generated in the recommended route generation process P43 (see, for example, Figures 22 and 30). The route image 51 visualizes the progress of the actual photography. Specifically, as the actual photography progresses, photographed sections of the route image 51 are erased. Alternatively, photographed sections of the route image 51 may be depicted in a different manner (e.g., color) from unphotographed sections without being erased. Note that, although an example is shown in which the start position mark (S) is displayed on the confirmation map 31 even after photography has begun, for example, a confirmation map 31 (see Figure 31) from which the start position mark (S) has been erased may be displayed after photography has begun.

[0145] In the confirmation map generation process P47, the processor 16 generates a confirmation map 31 for the user to confirm matters to be aware of during actual photography, based on the map image 32 acquired in the map image generation process P45, the route image 51 acquired in the route image generation process P46, and the location information of the points to be careful about when photographing detected in the point to be careful about when photographing process P3. The confirmation map 31 is obtained by superimposing an image that visualizes various types of support information on the map image 32 that represents the location to be measured. The confirmation map 31 is displayed after photography in the preliminary photography mode and during photography in the main photography mode.

[0146] In the confirmation map 31 displayed after shooting in the preliminary shooting mode, an image visualizing areas requiring attention for shooting, i.e., areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements, is superimposed on a map image 32. The map image 32 is an image visualizing the results of the preliminary shooting, specifically, an overhead point cloud image 35 generated from the point cloud data acquired in the preliminary shooting. The image visualizing areas requiring attention for shooting specifically includes mark images 41, 42, and 43 representing areas with strong reflections, areas with insufficient features, and areas with sudden movements, as well as an area image 44 representing areas with insufficient point clouds.

[0147] In addition, in the confirmation map 31 displayed during shooting in this shooting mode, images indicating points to be careful about when shooting (mark images 41, 42, 43, area image 44) and a route image 51 indicating the recommended route are superimposed on the map image 32.

[0148] At this time, the progress is visualized on the map image 32. Specifically, the map image 32 (see FIG. 24 and the like) includes an overhead point cloud image 35 obtained by the preliminary shooting and an overhead point cloud image 36 obtained by the main shooting, and is drawn so that as the main shooting progresses, the already-captured range in the overhead point cloud image 35 obtained by the preliminary shooting is gradually erased and replaced with the overhead point cloud image 36 obtained by the main shooting.

[0149] The progress is also visualized in images representing areas requiring careful shooting (mark images 41, 42, 43, area image 44). Specifically, the mark images 41, 42, 43 representing areas with strong reflections, areas lacking features, and areas with sudden movements, and the area image 44 representing areas lacking point clouds, are replaced with images generated during the actual shooting from those generated during the preliminary shooting as the actual shooting progresses. The progress is also visualized in the route image 51. Specifically, as the actual shooting progresses, sections of the route image 51 that have already been photographed are erased.

[0150] In the display information generation process P5, the processor 16 generates display information for the screen (see Figures 12 to 31) to be displayed on the display 13 based on the image captured by the visible camera 21 and the confirmation map 31 generated in the confirmation map generation process P47.

[0151] In the display process P6, the processor 16 displays a screen (see FIGS. 12 to 31) on the display 13 based on the display information generated in the display information generation process P5.

[0152] Next, a description will be given of the processing procedure performed by the processor 16 during shooting (during preliminary shooting and main shooting). Fig. 8 is a flowchart showing the processing procedure performed by the processor 16 during shooting.

[0153] First, the processor 16 acquires the image captured by the visible camera 21, the distance information from the depth camera 22, and the detection data from the IMU 23 from the sensor unit 12 (ST101).

[0154] Next, the processor 16 extracts feature points from the image captured by the visible camera 21 (feature extraction process P11) (ST102).

[0155] Next, the processor 16 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition regarding the position and orientation of the image capturing device 1, and updates the position and orientation trajectory data based on the amount of transition (tracking process P12) (ST103).

[0156] Next, the processor 16 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23 (position and orientation correction process P13) (ST104).

[0157] Next, the processor 16 generates point cloud data based on the distance information of the depth camera 22 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13 (point cloud generation process P14) (ST105).

[0158] Next, the processor 16 determines whether or not preliminary photography is in progress (ST106). If preliminary photography is in progress (Yes in ST106), the processor 16 then performs preliminary photography processing (ST107). On the other hand, if preliminary photography is not in progress (No in ST106), the processor 16 then determines whether or not actual photography is in progress (ST108). If actual photography is in progress (Yes in ST108), the processor 16 then performs actual photography processing (ST109).

[0159] Next, the processor 16 generates display information related to the screen being photographed, including the image captured by the visible camera 21 (display information generation process P5) (ST110). At this time, if preliminary photographing is in progress, display information related to the screen being photographed, including the overhead point cloud image 35 generated in the preliminary photographing, in addition to the image captured by the visible camera 21, is generated. If main photographing is in progress, display information related to the screen being photographed, including the image captured by the visible camera 21, in addition to the image captured by the visible camera 21, is generated. The confirmation map 31 includes a map image 32 made up of the overhead point cloud image 35, mark images 41, 42, 43 and an area image 44 indicating points to be careful of when photographing, and a route image 51 indicating a recommended route (see FIGS. 2 and 3).

[0160] Next, the processor 16 displays a screen on the display 13 based on the generated display information (display process P6) (ST111).

[0161] Next, the procedure of the process during preliminary imaging in ST107 shown in Fig. 8 will be described. Fig. 9 is a flow chart showing the procedure of the process during preliminary imaging.

[0162] First, the processor 16 acquires an evaluation value (reflection evaluation value) that indicates the intensity of reflected light that appears in the captured image (reflection evaluation process P21) (ST201).

[0163] Next, the processor 16 counts the feature points within the target range based on the extraction results of the feature extraction process P11, and obtains an evaluation value (number of feature points) representing the extraction status of the feature points (feature point evaluation process P22) (ST202).

[0164] Next, the processor 16 acquires an evaluation value (sudden movement evaluation value) that indicates the abruptness of the user's movement of the image capture device 1 (sudden movement evaluation process P23) (ST203). At this time, the sudden movement evaluation value is calculated based on the detection data of the IMU 23. The sudden movement evaluation value is also calculated based on the amount of transition related to the position and orientation of the image capture device 1 acquired in the tracking process P12.

[0165] Next, the processor 16 counts the point clouds within the target range based on the point cloud data generated in the point cloud generation process P14, and obtains an evaluation value (point cloud density evaluation value) representing the point cloud generation status (point cloud density evaluation process P24) (ST204).

[0166] Next, the processor 16 detects areas to be careful about when photographing (areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements) based on the evaluation values ​​obtained in each of the reflection evaluation process P21 (ST201), the feature point evaluation process P22 (ST202), the sudden movement evaluation process P23 (ST203), and the point cloud density evaluation process P24 (ST204) (detection process P3 of areas to be careful about when photographing) (ST205).

[0167] At this time, the reflection evaluation value is compared with a predetermined threshold to detect strong reflection areas (strong reflection area detection process P31). The number of feature points is compared with a predetermined threshold to detect feature-lacking areas (feature-lacking area detection process P32). The sudden movement evaluation value is compared with a predetermined threshold to detect sudden movement areas (sudden movement area detection process P33). The point cloud density evaluation value is compared with a predetermined threshold to detect point cloud-lacking areas (point cloud-lacking area detection process P34). The thresholds used in each detection process may be preset or may be specified by the user as appropriate.

[0168] Next, the processor 16 determines whether or not any areas requiring caution for photography (areas lacking point clouds, areas with strong reflections, areas lacking features, and areas requiring sudden movements) have been detected (ST206). If any areas requiring caution for photography have been detected (Yes in ST206), the processor 16 stores the positions of the detected areas requiring caution for photography and the content of a warning to the user in the memory 15 (ST207).

[0169] Next, the processor 16 stores the point cloud data generated in the point cloud generation process P14 (ST105 in FIG. 8) in the memory 15 (ST208). Next, the processor 16 generates display information for the screen including a highlight or text display indicating that preliminary photography is in progress (display information generation process P5) (ST209).

[0170] Next, a description will be given of the procedure of the process during actual photography in ST109 shown in Fig. 8. Fig. 10 is a flow chart showing the procedure of the process during actual photography.

[0171] First, the processor 16 compares the point cloud data generated this time in the point cloud generation process P14 (ST105 in FIG. 8) with the point cloud data at the time of preliminary photography stored in the memory 15, and determines whether the range of the point cloud data generated this time overlaps with the range of the point cloud data at the time of preliminary photography (ST301).

[0172] If the range of the current point cloud data overlaps with the range of the point cloud data from the preliminary photographing (Yes in ST302), the processor 16 then updates the map image 32 consisting of the overhead point cloud images 35, 36 (see FIG. 24, etc.) for the overlapping range of the current point cloud data and the point cloud data from the preliminary photographing (map image generation process P45) (ST303). At this time, the map image 32 is drawn so that the overhead point cloud image 35 from the preliminary photographing is deleted and replaced with the overhead point cloud image 36 from the main photographing. Note that even if the point cloud data from the preliminary photographing has some error compared to the point cloud data from the main photographing, it may be determined that the two point cloud data overlap, and the overhead point cloud image 35 from the preliminary photographing may be deleted.

[0173] Next, the processor 16 generates a confirmation map 31 including an updated or unupdated map image 32 (confirmation map generation process P47) (ST304). At this time, mark images 41, 42, and 43 indicating points to be careful of when taking photographs, an area image 44, and a route image 51 indicating a recommended route are superimposed on the map image 32 in accordance with the display settings specified by the user, thereby generating the confirmation map 31.

[0174] Next, a description will be given of the procedure of the process carried out when the preliminary photographing is completed in the processor 16. Fig. 11 is a flowchart showing the procedure of the process carried out when the preliminary photographing is completed.

[0175] First, the processor 16 detects areas where the point cloud is lacking based on the point cloud data acquired by the preliminary photographing, i.e., the point cloud data generated by the point cloud generation process P14 during the preliminary photographing and stored in the memory 15 (point cloud lacking area detection process P34) (ST401).

[0176] Next, the processor 16 generates a recommended route based on the target shooting range set from the point cloud data acquired by the preliminary shooting and the location and details of the detected points of caution for shooting (recommended route generation process P43) (ST402). Specifically, a route that covers the target shooting range and has the shortest possible travel distance is generated as the recommended route. At this time, the recommended route is generated according to setting information related to recommended route generation. For example, if the user is instructed to avoid points of caution for shooting, a recommended route is generated that avoids the points of caution for shooting.

[0177] Next, the processor 16 generates a confirmation map 31 including an overhead point cloud image 35 (map image 32) generated from the point cloud data acquired in the preliminary photographing, mark images 41, 42, 43 and an area image 44 indicating areas requiring caution in photographing (areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements), and a route image 51 indicating the recommended route (confirmation map generation process P47) (ST403).

[0178] Next, processor 16 generates display information for a screen (confirmation screen) including confirmation map 31 as a screen for the user to confirm the preliminary shooting results (display information generation process P5) (ST404). Next, processor 16 displays a screen on display 13 based on the generated display information (display process P6) (ST405).

[0179] Next, processor 16 determines whether or not a user operation (e.g., tap operation, click operation) has been performed to select areas requiring attention for photography (areas lacking point clouds, areas with strong reflections, areas lacking features, and areas with sudden movements) (ST406). If a user operation to select areas requiring attention for photography has not been performed (No in ST406), processor 16 then determines whether or not a user operation to close the screen for checking the preliminary photography results has been performed (ST407). If a user operation to close the screen for checking the preliminary photography results has not been performed (No in ST407), the process returns to ST406.

[0180] On the other hand, if the user has performed an operation to select a location to be careful about when photographing (Yes in ST406), display information for a screen (confirmation screen) including a message providing guidance (such as a suggestion of an improvement measure) according to the type of the selected location to be careful about when photographing is generated (display information generation process P5) (ST408). Next, returning to ST404, processor 16 displays a screen including the message on display 13 based on the generated display information.

[0181] Next, a screen 101 displayed on the display 13 in the preliminary capture mode will be described. Fig. 12 is an explanatory diagram showing the screen 101 displayed before capture in the preliminary capture mode. Figs. 13 and 14 are explanatory diagrams showing a screen 201 displayed during capture in the preliminary capture mode. Fig. 15 is an explanatory diagram showing a screen 301 displayed immediately after capture in the preliminary capture mode.

[0182] 12 to 15, a screen 101 in the preliminary shooting mode is provided with a main window 102, a sub-window 103, and an operation window 104. The sub-window 103 is provided so as to be superimposed on a part of the main window 102.

[0183] The main window 102 and the sub-window 103 respectively display a captured image 111 and an overhead point cloud image 35. The captured image 111 is a current captured image, i.e., an image captured in real time and output from the visible camera 21. The overhead point cloud image 35 is an image (rendering) of each point of the point cloud data as seen from a viewpoint in the sky.

[0184] Note that on the screen 101 before shooting shown in Fig. 12, point cloud data for the measurement target location has not been generated, so the overhead point cloud image 35 is not displayed. Also, on the screen 101 during shooting shown in Figs. 13 and 14, point cloud data for the measurement target location has been partially generated, so the overhead point cloud image 35 is partially displayed. Also, on the screen 101 immediately after shooting shown in Fig. 15, point cloud data for the measurement target location has been entirely generated, so the overhead point cloud image 35 is entirely displayed.

[0185] The operation window 104 is provided with a "Start preliminary image capture" button 121 and an "End preliminary image capture" button 122. When the user operates the "Start preliminary image capture" button 121, preliminary image capture is started, and images captured by the visible camera 21 are stored in the memory 15. When the user operates the "End preliminary image capture" button 122, preliminary image capture is ended. Note that in a configuration including a depth camera 22 and an IMU 23, distance information from the depth camera 22 and detection data from the IMU 23 are stored in the memory 15 along with images captured by the visible camera 21.

[0186] The operation window 104 also has a "screen switch" button 123 and a check box 124 related to the display of the sub-window 103. When the user operates the "screen switch" button 123, the display switches between a state in which the captured image 111 is enlarged and displayed in the main window 102 and the overhead point cloud image 35 is reduced and displayed in the sub-window 103 (the state shown in FIGS. 13 and 14), and a state in which the overhead point cloud image 35 is enlarged and displayed in the main window 102 and the captured image 111 is reduced and displayed in the sub-window 103. When the user operates the check box 124, the display state and non-display state of the sub-window 103 can be switched.

[0187] The operation window 104 also has an "actual shooting" tab 131, a "preliminary shooting" tab 132, and a "settings change" tab 133 as operation sections for selecting a mode. The screen 101 shown in FIGS. 12 to 19 is for the preliminary shooting mode (first shooting mode), and is displayed when the user operates the "preliminary shooting" tab 132. When the user operates the "settings change" tab 133, the screen switches to the setting change mode, and transitions to screens 301, 401, and 501 (see FIGS. 20 to 23). When the user operates the "actual shooting" tab 131, the screen switches to the actual shooting mode (second shooting mode), and transitions to screen 601 (see FIGS. 24 to 31).

[0188] In this embodiment, the modes are switched by operating the tabs 131, 132, and 133, but the modes may also be switched by operating other types of operation units such as buttons.

[0189] When shooting begins in the preliminary shooting mode, the display mode of the screen 101 changes to notify the user that preliminary shooting is in progress. In the example shown in FIG. 13 , the screen 101 is highlighted. Specifically, frame images 141 and 142 are displayed in the main window 102 and the operation window 104. The frame images 141 and 142 are drawn in a predetermined color (e.g., red). The background of the operation window 104 is displayed in a predetermined color (e.g., green). On the other hand, in the example shown in FIG. 14 , text 151 indicating that preliminary shooting is in progress is displayed at the top of the main window 102. Note that the screen 101 specific to each mode may be highlighted by switching between tabs 131, 132, and 133.

[0190] During preliminary shooting, message display windows containing various messages are displayed on the screen as needed. Some messages are relatively important (urgent), such as instructions or notifications that require immediate attention, while others are relatively less important, such as reference information or important content. Message display windows containing messages of relatively high importance are displayed in the center of the main window 102, while message display windows containing messages of relatively low importance are displayed at the bottom of the main window 102. Message display windows disappear after a predetermined display time has elapsed or when the need for their display has passed.

[0191] For example, when the preliminary image capture is started, a confirmation message such as "Preliminary image capture is starting. Please capture images evenly throughout the 3D point cloud creation range" is displayed at the bottom of the main window 102. In the example shown in Fig. 15, when the preliminary image capture is completed, a confirmation message is displayed in the message display window 161 in the center of the main window 102. For example, a message such as "Processing. Please wait a moment" is displayed.

[0192] Next, a screen 201 that is displayed on the display 13 when checking an image after shooting in the preliminary shooting mode will be described. Figures 16, 17, 18, and 19 are explanatory diagrams showing the screen 201 that is displayed when checking an image after shooting in the preliminary shooting mode.

[0193] 16, 17, and 18 show examples of a screen 201 (confirmation screen) that is displayed after shooting in the preliminary shooting mode. In the screen 201, a confirmation map 31 is displayed in the main window 102, allowing the user to check points that require attention when shooting before actual shooting.

[0194] 16 shows an example in which strong reflection areas and point cloud insufficient areas are detected. In this case, in the confirmation map 31, a mark image 41 representing the strong reflection area is superimposed on the position where the strong reflection area is detected on the map image 32 (bird's-eye view point cloud image 35). In addition, an area image 44 representing the point cloud insufficient area is superimposed on the range where the point cloud insufficient area is detected on the map image 32.

[0195] When a user selects a strong reflection location on the confirmation map 31, specifically, when a predetermined operation (tap operation, click operation) is performed on the mark image 41 representing the strong reflection location, a message prompting the user to take measures to improve the shooting environment related to the strong reflection location is displayed in the message display window 211. In this example, the selected strong reflection location is a location where reflected light is assumed to be caused by indoor lighting, and a message prompting the user to improve the shooting environment regarding lighting, such as a message saying, "If the reflection is strong, please reduce the lighting at the shooting location," is displayed. Furthermore, when the reflected light is assumed to be caused by external light entering through a window, a message prompting the user to improve the shooting environment regarding external light, such as a message saying, "If the curtains are open and external light is entering, please close the curtains," is displayed. Note that when both indoor lighting and external light entering through a window are likely to be the cause, a message prompting the user to improve both, such as a message saying, "Please reduce the lighting at the shooting location and close the curtains," may be displayed.

[0196] In addition, when the user selects an area where the point cloud is lacking on the confirmation map 31, specifically, performs a predetermined operation (tap operation, click operation) on the area image 44 representing the area where the point cloud is lacking, a message may be displayed in the message display window 211 urging the user to improve the shooting environment or shooting method related to the area where the point cloud is lacking.

[0197] 17 shows an example in which a feature-insufficient portion is detected. In this case, in the confirmation map 31, a mark image 42 representing the feature-insufficient portion is superimposed at the position on the map image 32 (bird's-eye view point cloud image 35) where the feature-insufficient portion is detected.

[0198] When the user selects a feature-deficient portion on the confirmation map 31, specifically, when the user performs a predetermined operation (tap operation, click operation) on the mark image 42 representing the feature-deficient portion, a message urging the user to take measures to improve the shooting environment or shooting method related to the feature-deficient portion is displayed in the message display window 211. In this example, the selected feature-deficient portion is caused by the shooting environment of the measurement target location, and a message urging the user to improve the shooting environment, such as a message saying "Please place something with texture," is displayed. Furthermore, when the selected feature-deficient portion is caused by the user's shooting method, a message urging the user to improve the shooting method, such as a message saying "Please adjust the angle of view to include a distinctive pattern or object," is displayed.

[0199] 18 shows an example in which a sudden movement location is detected. In this case, in the confirmation map 31, a mark image 43 representing the sudden movement location is superimposed at the position on the map image 32 (bird's-eye view point cloud image 35) where the sudden movement location is detected.

[0200] When the user performs an operation to select a sudden movement location on the confirmation map 31, specifically, a predetermined operation (tap operation, click operation) on the mark image 43 representing the sudden movement location, a message urging the user to take measures to improve the shooting method related to the sudden movement is displayed in the message display window 211. Specifically, a message urging the user not to make sudden movements (sharp turns), for example, a message saying "When shooting, please move slowly to avoid making sharp turns," is displayed.

[0201] 19 is another example of a screen 201 displayed after shooting in the preliminary shooting mode. In the examples shown in FIGS. 16, 17, and 18, a message urging the user to improve the shooting environment or shooting method is displayed in the message display window 211 by operating the mark images 41, 42, and 43 indicating areas requiring caution when shooting. In this example, on the other hand, the details of the areas requiring caution when shooting are displayed in a list in the legend display section 221. This allows the user to understand the details of the mark images 41, 42, and 43 and area image 44 indicating areas requiring caution when shooting that are drawn on the confirmation map 31.

[0202] As described above, in this embodiment, points requiring attention for photography are detected based on the results of the preliminary photography, and on the screen after the preliminary photography, a confirmation map 31 is displayed on which mark images 41, 42, and 43 indicating the points requiring attention for photography and an area image 44 are drawn, and by selecting a point requiring attention for photography on the confirmation map 31, a message providing guidance (such as a suggestion of an improvement measure) regarding the selected point requiring attention for photography is displayed. This allows the user to understand the location and details of the points requiring attention for photography and improve the photography environment and photography method to prevent the occurrence of events that could reduce point cloud accuracy, thereby enabling highly accurate point cloud data to be obtained with just one actual photography session.

[0203] In this embodiment, by selecting a point of caution for photography on the confirmation map 31 on the screen 201 after preliminary photography, a message providing guidance regarding the selected point of caution for photography is displayed, but a message providing guidance regarding the detected point of caution for photography may also be displayed at the time when a point of caution for photography is detected during preliminary photography.

[0204] 16 to 19, in the confirmation map 31, mark images 41, 42, and 43 representing areas to be careful of when taking photographs (areas with strong reflections, areas lacking features, and areas with sudden movements) are superimposed on the map image 32 (bird's-eye view point cloud image 35), but if multiple areas to be careful of when taking photographs are detected at a high density, an area image drawn to include the multiple areas to be careful of when taking photographs may be superimposed. Also, if the mark images representing the areas to be careful of when taking photographs are, for example, circular, multiple mark images may be superimposed so that they overlap each other, allowing them to be recognized as a single area.

[0205] Next, we will explain the screens 301, 401, and 501 (setting screens) in the setting change mode for the preliminary photography results displayed on the display 13. Fig. 20 is an explanatory diagram showing the screen 301 for progress status setting. Fig. 21 is an explanatory diagram showing the screen 401 for setting the display of points to be careful about when taking photos. Figs. 22 and 23 are explanatory diagrams showing the screen 501 for setting the recommended route.

[0206] As shown in Figures 20 to 23, the setting change mode screens 301, 401, and 501 are provided with a "Progress Status Settings" tab 311, a "Caution Points for Shooting Display Settings" tab 312, and a "Recommended Route Settings" tab 313 as operation sections for switching settings.

[0207] When the user operates the "progress setting" tab 311, a screen 301 (see FIG. 20) for setting the progress status is displayed. When the user operates the "noticeable area display setting" tab 312, a screen 401 (see FIG. 21) for setting the noticeable area display is displayed. When the user operates the "recommended route setting" tab 313, a screen 501 (see FIGS. 22 and 23) for setting the recommended route is displayed.

[0208] In this embodiment, the setting items are switched by operating the tabs 311, 312, and 313, but the setting items may be switched by operating other types of operation units such as buttons.

[0209] 20 is displayed when the user operates the "Progress Status Settings" tab 311. On this screen 301, the user can specify whether or not to display the progress status. Also, on this screen 301, the user can specify a target shooting range that will be used as a reference when acquiring the progress status.

[0210] A check box 321 related to the display of the progress status is provided on the screen 301. When the user operates the check box 321, the user can switch the setting as to whether or not to display the progress status on the screen 601 (see FIG. 25 etc.) during actual shooting. Specifically, the user can switch whether or not to display an indicator image 621 indicating the progress rate on the screen 601 during actual shooting.

[0211] The screen 301 also has a "Start range input" button 323 and a "Execute range setting" button 324 for setting a target shooting range designated by the user.

[0212] When the user operates the "Start Range Input" button 323, the system transitions to range input mode. In range input mode, the user can input a target capture range on the overhead point cloud image 35 displayed in the main window 102. Specifically, for example, the user can input the target capture range by sequentially specifying the vertices of a polygon representing the target capture range (e.g., by tapping or clicking). At this time, an area image 61 representing the target capture range is displayed in the main window 102 together with the overhead point cloud image 35. This allows the user to specify a preliminary capture range, i.e., a range different from the target capture range set based on the point cloud data generated in the preliminary capture, as a new target capture range.

[0213] When the user operates the "execute range setting" button 324, the range input mode is cancelled and the target shooting range input by the user is registered.

[0214] 21 is displayed when the user operates the "Caution Points for Photography Display Settings" tab 312. On this screen 401, the user can specify whether or not to display each type of caution points for photography on the confirmation map 31, and can check the confirmation map 31 generated based on the specification.

[0215] The screen 401 is provided with check boxes 411 to 414 for displaying areas requiring attention when photographing, that is, areas with insufficient point clouds, areas with strong reflections, areas with insufficient features, and areas with sudden movements.

[0216] The user can switch between displaying and deactivating mark images 41 representing strong reflection areas on the confirmation map 31 by operating a check box 411 related to the display of strong reflection areas. The user can switch between displaying and deactivating mark images 42 representing feature-lacking areas on the confirmation map 31 by operating a check box 412 related to the display of feature-lacking areas. The user can switch between displaying and deactivating mark images 43 representing sudden movement areas on the confirmation map 31 by operating a check box 413 related to the display of sudden movement areas. The user can switch between displaying and deactivating area images 44 representing point cloud-lacking areas on the confirmation map 31 by operating a check box 414 related to the display of point cloud-lacking areas.

[0217] Screen 401 also has a "Execute Display Settings" button 415. When the user operates the "Execute Display Settings" button 415, display conditions for points of caution for photography are set in accordance with the input contents of check boxes 411 to 414. At this time, in main window 102, a confirmation map 31 is generated in which mark images 41, 42, 43 and area image 44 are superimposed on map image 32 (bird's-eye view point cloud image 35) in accordance with the settings, and confirmation map 31 is displayed in main window 102. Note that it is preferable to check all check boxes 411 to 414 as the initial setting, and allow the user to uncheck them at will. Furthermore, display settings may be saved for each user in association with a photography location, and the saved display settings may be reused.

[0218] 22 is displayed when the user operates the "recommended route setting" tab 313. On this screen 501, the user can specify conditions for generating a recommended route and check the recommended route generated based on those conditions.

[0219] The screen 501 is provided with check boxes 511 to 513 for avoiding areas to be careful of when photographing when generating a recommended route, that is, areas with strong reflections, areas with insufficient features, and areas with sudden movements.

[0220] The user can switch whether or not to avoid strong reflection points when generating a recommended route by operating a check box 511 related to avoiding strong reflection points. The user can also switch whether or not to avoid feature-insufficient points when generating a recommended route by operating a check box 512 related to avoiding feature-insufficient points. The user can also switch whether or not to avoid feature-insufficient points when generating a recommended route by operating a check box 513 related to avoiding sudden movement points.

[0221] Screen 501 also has a "Generate recommended route" button 514. When the user operates "Generate recommended route" button 514, a recommended route is generated in accordance with the input contents of check boxes 511 to 513. Furthermore, a route image 51 representing the recommended route is generated, and a confirmation map 31 is generated in which route image 51 is superimposed on map image 32 (bird's-eye view point cloud image 35), and confirmation map 31 is displayed in main window 102.

[0222] In the example shown in Fig. 22, the check boxes 511 to 513 are set to not avoid strong reflection areas, feature-insufficient areas, and sudden movement areas. In this case, a route that covers the entire target shooting range and has the shortest possible travel distance is generated as the recommended route, regardless of the strong reflection areas, feature-insufficient areas, and sudden movement areas. On the other hand, in the example shown in Fig. 23, the setting is made to avoid strong reflection areas. In this case, the recommended route is generated so as to avoid strong reflection areas. Note that the settings for generating the recommended route may be saved for each user in association with the shooting location, and the saved settings for generating the recommended route may be reused.

[0223] Next, we will explain the screen 601 in the actual photography mode that is displayed on the display 13. Figures 24, 25, 26, 27, 28, 29, 30, and 31 are explanatory diagrams showing the screen 601 that is displayed on the display 13 in the actual photography mode.

[0224] As shown in FIG. 24, the main shooting mode screen 601 is provided with a main window 102, a sub-window 103, and an operation window 104, similar to the preliminary shooting mode screen 101 (see FIG. 12, etc.).

[0225] A confirmation map 31 is displayed in the sub-window 103 to allow the user to check points that require attention during actual photography.

[0226] The operation window 104 is provided with a "Start Main Shooting" button 611 and an "End Main Shooting" button 612. When the user operates the "Start Main Shooting" button 611, the main shooting starts, and the image captured by the visible camera 21, the distance information by the depth camera 22, and the detection data by the IMU 23 are stored in the memory 15. When the user operates the "End Main Shooting" button 612, the main shooting ends.

[0227] Immediately after the transition to the actual photography mode screen 601 or immediately after the user operates the "Start actual photography" button 611, a confirmation message, for example, a message stating "Actual photography will begin. Please start photography from the starting point," may be displayed. The actual photography may start from the point where the preliminary photography ended, or a starting point for the actual photography may be suggested based on the preliminary photography results. To obtain the progress during the actual photography, it is necessary to align the preliminary photography results (point clouds) with the actual photography results (point clouds). To facilitate this alignment, the photographing may be completed with the photographing device 1 placed on a desk or the like during the preliminary photography, and the photographing device 1 may not be moved until the actual photography starts.

[0228] The operation window 104 also has a "screen change" button 123 and a check box 124 for displaying the sub-window 103, similar to the screen 101 of the preliminary shooting mode (see FIG. 12, etc.).

[0229] The operation window 104 also has a check box 615 for displaying the progress status, a check box 616 for displaying the preliminary shooting results, i.e., the point cloud acquired in the preliminary shooting, a check box 617 for displaying points to be careful about when shooting, and a check box 618 for displaying the recommended route.

[0230] The user can specify whether or not to display the progress status (enable or disable) by operating a check box 615 related to the display of the progress status. The user can also specify whether or not to display the preliminary imaging results by operating a check box 616 related to the display of the preliminary imaging results (point cloud). The user can also specify whether or not to display the points of caution for imaging by operating a check box 617 related to the display of the points of caution for imaging. The user can also specify whether or not to display the recommended route by operating a check box 618 related to the display of the recommended route.

[0231] Here, the areas requiring caution for photography include areas requiring strong reflection, areas lacking features, areas requiring sudden movement, and areas requiring insufficient point clouds, and whether or not to display each type of area requiring caution for photography can be specified collectively using the check boxes 617 related to the display of areas requiring caution for photography. Note that whether or not to display each type of area requiring caution for photography (areas requiring strong reflection, areas lacking features, areas requiring sudden movement, and areas requiring insufficient point clouds) may also be specified individually.

[0232] Note that the check boxes 615 to 618 on the main photography mode screen may be operated before photography starts, but may also be operated after photography starts. For example, if the user finds the displays of the progress status, preliminary photography results, points to be careful about when photographing, and recommended routes bothersome, the user may operate the check boxes 615 to 618 during main photography to change whether or not to display them.

[0233] The example shown in Figure 24 is a case where, by operating check boxes 615 to 618, the display of progress status is disabled, the display of preliminary shooting results is enabled, the display of areas to be careful about when shooting is disabled, and the display of the recommended route is disabled.

[0234] In this case, a map image 32 showing the measurement target location is displayed on the confirmation map 31 displayed in the subwindow 103. The map image 32 includes an overhead point cloud image 35 obtained by preliminary photography and an overhead point cloud image 36 obtained by main photography. The overhead point cloud image 35 obtained by preliminary photography is an overhead point cloud image that visualizes the point cloud data acquired in the preliminary photography, and the overhead point cloud image 36 obtained by main photography is an overhead point cloud image that visualizes the point cloud data acquired up to the present time in the main photography.

[0235] The overhead point cloud image 35 obtained by preliminary photography and the overhead point cloud image 36 obtained by main photography are drawn in different drawing modes (e.g., colors). Specifically, for example, the overhead point cloud image 35 obtained by preliminary photography is drawn in purple, and the overhead point cloud image 36 obtained by main photography is drawn in black. Note that if the captured color is reflected in the acquired point clouds, for example, the overhead point cloud image 35 obtained by preliminary photography may be drawn in purple, and the overhead point cloud image 36 obtained by main photography may be drawn in the acquired color. Furthermore, each predetermined color may be blended (mixed) into the color of each acquired point cloud.

[0236] The map image 32 visualizes the progress of the actual photography. Specifically, as the actual photography progresses, the captured range in the overhead point cloud image 35 from the preliminary photography is gradually erased and replaced with the overhead point cloud image 36 from the actual photography. Here, the remaining overhead point cloud image 35 from the preliminary photography represents the range that has not yet been captured, and the overhead point cloud image 36 from the actual photography represents the range that has already been captured. Furthermore, as the actual photography progresses, the range of the overhead point cloud image 35 from the preliminary photography gradually becomes smaller, and the range of the overhead point cloud image 36 from the actual photography gradually becomes larger. This allows the user to grasp the progress of the actual photography.

[0237] In addition, in the confirmation map 31, the areas that have been photographed in the overhead point cloud image 35 obtained by preliminary photography may not be erased, but may be drawn in a manner (e.g., color) different from the areas that have not been photographed in the overhead point cloud image 35 obtained by preliminary photography or the overhead point cloud image 36 obtained by main photography.

[0238] 25 shows an example in which the progress status display is enabled, the preliminary photography result display is enabled, the points to be careful about when taking photos are disabled, and the recommended route display is disabled by operating the check boxes 615 to 618. In this example, the settings other than the progress status are the same as those in the example shown in FIG.

[0239] In this case, an indicator image 621 visualizing the progress of the actual shooting is displayed in the main window 102. The confirmation map 31 displayed in the sub-window 103 is the same as the example shown in FIG.

[0240] The indicator image 621 is a so-called progress bar, and is drawn so that the portion indicating the captured range changes as the actual shooting progresses. That is, as the actual shooting progresses, the captured portion is drawn gradually larger and the uncaptured portion is drawn gradually smaller. In addition, in the indicator image 621, the captured portion and the uncaptured portion are drawn in different modes (e.g., colors or patterns). For example, the captured portion is drawn in green, and the uncaptured portion is drawn in white.

[0241] The example shown in Fig. 26 is a case where the user has specified a target shooting range. The display settings for the progress status, preliminary shooting results, points to note when shooting, and recommended route, which are set by operating check boxes 615 to 618, are the same as those in the example shown in Fig. 25.

[0242] In this case, an area image 61 representing the target shooting range specified by the user is drawn on the confirmation map 31 displayed in the subwindow 103. The area image 61 is drawn in a predetermined color (e.g., yellow) and is semi-transparently superimposed on the overhead point cloud image 35 obtained by preliminary shooting, which represents the initial target shooting range. Note that the area image 61 only needs to be able to show the range, and for example, only a line indicating the range may be drawn.

[0243] The example shown in Fig. 27 is a case where the display of progress status is enabled, the display of preliminary photography results is enabled, the display of points to be careful about when photographing is enabled, and the display of recommended routes is disabled, by operating check boxes 615 to 618. In this example, the settings other than the points to be careful about when photographing are the same as those in the example shown in Fig. 25. This example also shows a case where a point cloud shortage area has been detected.

[0244] In this case, in the confirmation map 31 displayed in the sub-window 103, an area image 44 representing the point cloud insufficiency area is superimposed on the overhead point cloud images 35 and 36 constituting the map image 32. The area image 44 representing the point cloud insufficiency area is drawn in a predetermined color (e.g., red).

[0245] The example shown in Fig. 28 is a case where a point cloud shortage area and a strong reflection area are detected. The display settings for the progress status, preliminary shooting results, areas to be careful about when shooting, and recommended route by operating check boxes 615 to 618 are the same as the example shown in Fig. 27.

[0246] In this case, in the confirmation map 31 displayed in the sub-window 103, a mark image 41 representing a strongly reflective area and an area image 44 representing an area lacking a point cloud are superimposed on the overhead point cloud images 35 and 36 that constitute the map image 32.

[0247] Here, when the user selects a strong reflection location on the confirmation map 31, specifically, when the user performs a predetermined operation (tap operation, click operation) on the mark image 41 representing the strong reflection location, a message prompting the user to take measures to improve the shooting environment related to the strong reflection location is displayed in the message display window 631. In this example, the selected strong reflection location is a location where reflected light is assumed to be caused by indoor lighting, and a message prompting the user to improve the shooting environment regarding lighting, such as a message saying, "If the reflection is strong, please reduce the lighting at the shooting location," is displayed. Furthermore, if the reflected light is assumed to be caused by external light entering through a window, a message prompting the user to improve the shooting environment regarding external light, such as a message saying, "If the curtains are open and external light is entering, please close the curtains," is displayed. Note that if both indoor lighting and external light entering through a window are likely to be the cause, a message prompting the user to improve both, such as a message saying, "Please reduce the lighting at the shooting location and close the curtains," may be displayed.

[0248] The example shown in Fig. 29 is a case where a point cloud-deficient area and a feature-deficient area are detected. The display settings for the progress status, preliminary photography results, areas to be careful about when photographing, and recommended route by operating check boxes 615 to 618 are the same as those in the examples shown in Figs. 27 and 28.

[0249] In this case, in the confirmation map 31 displayed in the sub-window 103, a mark image 42 representing an area lacking features and an area image 44 representing an area lacking point cloud are superimposed on the overhead point cloud images 35, 36 that constitute the map image 32.

[0250] Here, when the user selects a feature-deficient portion on the confirmation map 31, specifically, when the user performs a predetermined operation (tap operation, click operation) on the mark image 42 representing the feature-deficient portion, a message urging the user to take measures to improve the shooting environment or shooting method related to the feature-deficient portion is displayed in the message display window 631. In this example, the selected feature-deficient portion is caused by the user's shooting method, and a message urging the user to improve the shooting method is displayed, such as a message saying, "When shooting walls or floors, please use an angle of view that includes other features." Furthermore, when the selected feature-deficient portion is caused by the shooting environment of the measurement target location, a message urging the user to improve the shooting environment is displayed, such as a message saying, "Please place something with texture."

[0251] The examples shown in Figures 30 and 31 are when the check boxes 615 to 618 are operated to enable display of all of the progress status, preliminary photography results, areas to be careful about when photographing, and recommended route. In this example, the settings other than the progress status are the same as those in the example shown in Figure 24. This example also shows a case where a strong reflection area, an area lacking features, an area with sudden movement, and an area lacking point clouds have been detected. Figure 30 shows the time immediately after the actual photography has started, and Figure 31 shows the time when the actual photography has progressed to a certain extent.

[0252] In this case, in the confirmation map 31 displayed in the sub-window 103, a route image 51 representing the recommended route is superimposed on the overhead point cloud images 35, 36 constituting the map image 32. In addition, a mark image 41 representing a strong reflection point, a mark image 42 representing a feature-lacking point, a mark image 43 representing a sudden movement point, and an area image 44 representing a point cloud-lacking point are superimposed on the overhead point cloud images 35, 36 constituting the map image 32.

[0253] 30 and 31, a navigation icon 641 is superimposed on the captured image 111 displayed in the main window 102. The navigation icon 641 is an image representing the user's route, and is created based on the recommended route and the current location.

[0254] Here, if the shooting environment and shooting method are appropriately improved in the actual shooting compared to the preliminary shooting, the mark images 41, 42, 43 and the area image 44 change as the actual shooting progresses. Specifically, the area image 44 representing the point cloud insufficiency area shrinks as the actual shooting progresses. This allows the user to understand that the point cloud insufficiency area has decreased. Furthermore, in the already-photographed sections, the mark images 41, 42, 43 representing the strong reflection area, the feature insufficiency area, and the sudden movement area are erased. This allows the user to confirm that no events that reduce the accuracy of the point cloud data have occurred in the strong reflection area, the feature insufficiency area, and the sudden movement area.

[0255] Furthermore, the route image 51 representing the recommended route changes as the actual photography progresses. Specifically, photographed sections of the route image 51 are erased. This allows the user to understand the progress of the actual photography. Photographed sections of the route image 51 may not be erased, but may be drawn in a different style from unphotographed sections. For example, photographed sections may be drawn in a lighter color than unphotographed sections.

[0256] Note that when the image capture device 1 is configured as a tablet terminal (see FIG. 1 ), it is difficult to operate the screen while capturing images. Therefore, operations on the screen 601 in the actual capture mode, specifically, operations on the check boxes 615 to 618 and operations on the points to be noted for capture on the confirmation map 31, may be performed before the start of capture, and functions may be restricted so that screen operations are not possible once capture has begun. On the other hand, when the image capture device 1 is configured as a wearable device such as a head-mounted device equipped with a camera and a display, screen operations during capture are easier, so it may be possible to enable screen operations in the actual capture mode.

[0257] Incidentally, if a user is unfamiliar with the system, it is expected that the user will start actual shooting without taking preliminary photographs. Therefore, if the user operates the main shooting mode tab 131 before shooting in the preliminary shooting mode, a confirmation message regarding preliminary shooting, i.e., a message that asks whether or not preliminary shooting is to be performed and prompts the user to perform preliminary shooting, may be displayed. For example, a message such as "If you want to perform actual shooting without taking preliminary photographs, continue as is. If you want to perform preliminary photographs, select the preliminary shooting tab." may be displayed.

[0258] In addition, there may be cases where preliminary photography has already been performed at the same measurement location. In this case, if the photography data and point cloud data (preliminary photography results) acquired in the previous preliminary photography can be used, the current preliminary photography can be omitted. Therefore, a screen operation for saving and loading the preliminary photography results may be provided. In this way, if the preliminary photography results have been saved, the current preliminary photography can be omitted by loading the preliminary photography results. This allows, for example, if the main photography fails after the preliminary photography has been performed, the main photography can be redone without performing another preliminary photography.

[0259] Furthermore, if a user starts actual shooting without taking preliminary photographs, the target shooting range cannot be set based on the results of the preliminary photographs, and therefore the progress status of the actual photographs cannot be presented to the user. Therefore, when an attempt is made to start actual photographs without taking preliminary photographs, a confirmation message regarding the progress status may be displayed. For example, a message stating "Progress status will not be displayed" may be displayed.

[0260] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments.

[0261] The photographing device, photographing work support method, and photographing work support program disclosed herein have the effect of enabling a user to check in advance the locations where various phenomena that can cause a decrease in the accuracy of point cloud data will occur when performing actual photographing, thereby enabling highly accurate measurement results to be obtained with a single actual photographing session.They are useful as a photographing device that photographs a location to be measured, and as a photographing work support method and photographing work support program that support the photographing work of a user using the photographing device, in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the location to be measured based on the photographing data of the location to be measured.

[0262] 1: Imaging device 11: Device main body 12: Sensor unit 13: Display (display unit) 14: Input device 15: Memory 16: Processor 21: Visible camera 22: Depth camera 23: IMU 31: Confirmation map 32: Map image 35: Aerial view point cloud image from preliminary imaging (first aerial view point cloud image) 36: Aerial view point cloud image from actual imaging (second aerial view point cloud image) 41: Mark image showing strong reflection area 42: Mark image showing area with insufficient features 43: Mark image showing area with sudden movement 44: Area image showing area with insufficient point cloud 51: Route image showing recommended route 61: Area image showing target imaging range 101: Screen of preliminary imaging mode 102: Main window 103: Sub-window 104: Operation window 111: Photographed image 131: "Main imaging" tab 132: "Preliminary imaging" tab 133: "Change settings" tab 161: Message display window 201: Screen in preliminary shooting mode 211: Message display window 301, 401, 501: Screen in setting change mode 311: "Progress setting" tab 312: "Display setting of points to be careful about when shooting" tab 313: "Recommended route setting" tab 601: Screen in actual shooting mode 621: Indicator image 631: Message display window 641: Navigation icon

Claims

1. A photographing device that photographs a location to be measured in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the location to be measured based on photographing data of the location to be measured, comprising: a device main body held by a user; a photographing unit provided in the device main body that photographs the location to be measured; a display unit that displays work support information related to the photographing work in which the user moves around the location to be measured and has the photographing unit photograph the location to be measured; and a processor that controls the photographing unit and the display unit, wherein the processor detects areas that need to be photographed in the actual photographing based on the photographing data obtained in preliminary photographing, generates a confirmation map in which the areas that need to be photographed are drawn on a map image representing the location to be measured, and displays a screen including the confirmation map on the display unit.

2. The photographing device described in claim 1, characterized in that the processor generates point cloud data as a preliminary photographing result based on the photographing data acquired in the preliminary photographing, generates an overhead point cloud image based on the point cloud data, and generates the confirmation map by superimposing an image representing the areas to be careful of in photographing on the map image including the overhead point cloud image.

3. The photographing device according to claim 1, characterized in that the processor switches between a first photographing mode in which preliminary photographing is performed and a second photographing mode in which actual photographing is performed in response to a user's operation to select a photographing mode.

4. The photographing device described in claim 1, characterized in that the processor causes the display unit to display a screen including a real-time photographed image output from the photographing unit during actual photographing and the confirmation map on which the areas to be careful of when photographing are depicted.

5. The photographing device according to claim 1, characterized in that the processor causes the display unit to display a screen including the confirmation map on which the areas to be careful of during photographing are depicted before the actual photographing process after the preliminary photographing process.

6. The photographing device according to claim 1, characterized in that the processor generates simplified point cloud data as a preliminary photographing result by processing that is simplified compared to that used in actual photographing, and detects the areas to be photographed based on the simplified point cloud data.

7. The photographing device according to claim 1, characterized in that the processor generates the confirmation map in which at least one of strong reflection areas, areas lacking features, areas of sudden movement, and areas lacking point clouds is depicted as the areas to be careful of when photographing.

8. A method for supporting photography operations in which a processor performs processing to support a user's photography operation by having the user hold a photography device and move around the location to be measured while having the photography device photograph the location to be measured, in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured based on photography data of the location to be measured, the method comprising the steps of: detecting areas that need to be photographed in the actual photography based on photography data acquired in preliminary photography; generating a confirmation map in which the areas that need to be photographed are drawn on a map image representing the location to be measured; and displaying a screen including the confirmation map on a display unit.

9. A photography support program that causes a processor to execute a process to support a user's photography work by having the user hold a photography device and move around the measurement target location while having the photography device photograph the measurement target location in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on photography data of the measurement target location, the photography support program detecting areas that need to be photographed in the actual photography based on photography data acquired in preliminary photography, generating a confirmation map in which the areas that need to be photographed are drawn on a map image representing the measurement target location, and displaying a screen including the confirmation map on a display unit.

10. A photographing device that photographs a location to be measured in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured based on photographing data of the location to be measured, comprising: a device main body held by a user; a photographing unit provided in the device main body for photographing the location to be measured; a display unit that displays work support information related to the photographing work in which the user moves around the location to be measured and has the photographing unit photograph the location to be measured; and a processor that controls the photographing unit and the display unit, wherein the processor sets a target photographing range for the actual photographing based on the photographing data acquired in preliminary photographing, acquires a photographed range based on the photographing data acquired in actual photographing, acquires information regarding the progress of the actual photographing based on the photographed range and the target photographing range, and causes a screen including information regarding the progress of the actual photographing to be displayed on the display unit.

11. The photographing device according to claim 10, characterized in that the processor switches between a first photographing mode in which preliminary photographing is performed and a second photographing mode in which actual photographing is performed in response to a user's operation to select a photographing mode.

12. The photographing device according to claim 10, characterized in that the processor causes the display unit to display a screen including a real-time photographed image output from the photographing unit during the actual photographing and an image visualizing the progress of the actual photographing.

13. The photographing device described in claim 12, characterized in that the processor generates a map image representing the measurement target location as an image visualizing the progress of the actual photographing, the map image including a first overhead point cloud image representing the target photographing range generated by preliminary photographing and a second overhead point cloud image representing the photographed range generated by actual photographing, and the first overhead point cloud image and the second overhead point cloud image are rendered in different manners.

14. The photographing device according to claim 12, characterized in that the processor generates an indicator image, as an image visualizing the progress of the actual photographing, in which the part indicating the photographed range changes as the actual photographing progresses.

15. The photographing device according to claim 10, characterized in that the processor sets an initial target shooting range based on point cloud data generated by preliminary photographing, sets a user-specified target shooting range in response to a user's operation to change the range from the initial target shooting range, and obtains information regarding the progress of the actual photographing based on the user-specified target shooting range.

16. A method for supporting photography by a processor that supports a user in photographing a location to be measured by having the photography device photograph the location to be measured while moving within the location to be measured, in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured based on photography data of the location to be measured, the method comprising the steps of: setting a target photography range for the actual photography based on photography data acquired in preliminary photography; acquiring a photographed range based on photography data acquired in the actual photography; acquiring information regarding the progress of the actual photography based on the photographed range and the target photography range; and displaying a screen including information regarding the progress of the actual photography on a display unit.

17. A photography support program that causes a processor to execute a process to support a user's photography work by having the user hold a photography device and move around the measurement target location while having the photography device photograph the measurement target location in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the measurement target location based on photography data of the measurement target location, the photography support program setting a target photography range for actual photography based on photography data acquired in preliminary photography, acquiring a photographed range based on photography data acquired in actual photography, acquiring information regarding the progress of the actual photography based on the photographed range and the target photography range, and displaying a screen including information regarding the progress of the actual photography on a display unit.

18. A photographing device that photographs a location to be measured in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured based on photographing data of the location to be measured, comprising: a device main body held by a user; a photographing unit provided in the device main body for photographing the location to be measured; a display unit that displays work support information related to the photographing work in which the user moves around the location to be measured and has the photographing unit photograph the location to be measured; and a processor that controls the photographing unit and the display unit, wherein the processor detects areas that should be photographed in actual photographing based on photographing data obtained in preliminary photographing, displays on the display unit a screen including an operation unit through which a user operates to specify display conditions for the areas that should be photographed, generates a confirmation map in which the areas that should be photographed are drawn on a map image representing the location to be measured in accordance with the specified display conditions, and displays a screen including the confirmation map on the display unit.

19. The photographing device according to claim 18, characterized in that the processor causes the display unit to display a screen including a real-time photographed image output from the photographing unit, the confirmation map, and the operation unit during actual photographing.

20. The photographing device according to claim 18, characterized in that the processor causes the display unit to display a screen including the confirmation map and the operation unit when checking after preliminary photographing and before actual photographing.

21. The photographing device according to claim 18, characterized in that the processor generates a recommended route for the actual photographing based on a target photographing range based on point cloud data generated in preliminary photographing, and displays a screen including the recommended route on the display unit.

22. The photographing device according to claim 21, characterized in that the processor generates the recommended route in accordance with a user's instruction so as to avoid the locations that require attention for photographing.

23. The photographing device according to claim 21, characterized in that the processor, during actual photographing, causes the display unit to display a screen including the confirmation map on which the recommended route is drawn and a photographed image on which an image showing a course based on the recommended route is superimposed.

24. The photographing device according to claim 18, characterized in that the processor generates the confirmation map in which at least one of strong reflection areas, areas lacking features, areas of sudden movement, and areas lacking point clouds is depicted as the areas to be careful about when photographing.

25. A method for supporting photography operations, in which a processor performs processing to support a user's photography operation by having the photography device photograph the location to be measured while moving within the location to be measured, in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured based on photography data of the location to be measured, the method comprising the steps of: detecting areas that need to be photographed in actual photography based on photography data acquired in preliminary photography; displaying on a display unit a screen including an operation unit through which a user operates to specify display conditions for the areas that need to be photographed; generating a confirmation map in which the areas that need to be photographed are drawn on a map image representing the location to be measured according to the specified display conditions; and displaying on the display unit a screen including the confirmation map.

26. A photography support program that causes a processor to execute a process to support a user's photography work by having the photography device photograph the location to be measured while moving within the location to be measured, in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured based on photography data of the location to be measured, the photography support program comprising: detecting areas that need to be photographed in actual photography based on photography data acquired in preliminary photography; displaying on a display unit a screen including an operation unit through which a user operates to specify display conditions for the areas that need to be photographed; generating a confirmation map in which the areas that need to be photographed are drawn on a map image representing the location to be measured according to the specified display conditions; and displaying a screen including the confirmation map on the display unit.

Images