Photographing device, photographing operation support method, and photographing operation support program

The photographing device and method offer customizable work support to enhance three-dimensional measurement accuracy by adapting to user proficiency, providing targeted guidance for improved photographing techniques.

JP2024159139AActive Publication Date: 2024-11-08PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2023074934
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-28
Publication Date
2024-11-08
Estimated Expiration
2043-04-28

AI Technical Summary

Technical Problem

Conventional three-dimensional measurement techniques lack intuitive guidance for users, leading to inconsistent and potentially inappropriate photographing methods due to varying user proficiency levels, necessitating personalized work support.

Method used

A photographing device and method that provides customizable work support through a processor-controlled display unit, allowing users to select and adjust settings based on their proficiency level, displaying appropriate guidance and messages to improve photographing quality.

Benefits of technology

Enables tailored work support for users, ensuring accurate and efficient three-dimensional spatial information generation by addressing user-specific needs and improving photographing techniques.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159139000001_ABST
    Figure 2024159139000001_ABST
Patent Text Reader

Abstract

To provide a photographing device, a photographing operation support method, and a photographing operation support program that can set a condition for executing operation support for a user performing photographing operations individually for every user, and thereby can perform appropriate operation support according to the skill level of the user, or the like.SOLUTION: In order to perform three-dimensional measurement processing of generating three-dimensional space information of a measurement target place on the basis of a photographed image of the measurement target place, a photographing device that photographs the measurement target place, displays, on a display, operation support information related to photographing operations performed by a user causing a photographing unit to photograph the measurement target place while moving in the measurement target place, especially, displays, on the display, a customized screen 201 including check boxes 232, 242, 252 for selecting items to be executed from a plurality of items, determines the necessity and details of the operation support for the user in relation to the items selected on the customized screen, and when the operation support is required, executes operation support processing of displaying the operation support information on the display.SELECTED DRAWING: Figure 15
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an imaging device that captures images of each point of a measurement target location in order to perform 3D measurement processing that generates 3D spatial information of the measurement target location based on images captured at each point of the measurement target location, and to a photography work support method and photography work support program that support a user's photography work using the imaging device. [Background technology]

[0002] There is known a 3D measurement technique that generates 3D spatial information (map data) about a measurement target location based on images of the measurement target location. In recent years, the SLAM (Simultaneous Localization And Mapping) method has been attracting attention for this 3D measurement. In the SLAM method, a photographing device is held by a moving object, and based on images of each point in the measurement target location, 3D spatial information and the position information of the object can be generated as a measurement result. In particular, a user (worker) can easily perform 3D measurement by performing a photographing task of photographing each point in the measurement target location while moving within the measurement target location using a handheld photographing device to obtain photographed images of each point in the measurement target location.

[0003] On the other hand, when a user photographs a measurement target location with a photographing device held in his / her hand while walking through the measurement target location, it is desirable to provide work support that prompts the user to perform appropriate photographing work during the photographing work so that highly accurate and stable 3D measurement can be performed.From this perspective, a technology has been known in the past that displays multiple marks such as arrows that indicate the moving direction and moving speed of the photographing device (sensor) held by the user to guide the user on how to proceed with the photographing work (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6489566 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in conventional technology, since a user cannot intuitively grasp how to take a photograph, the specific procedure for taking a photograph is determined according to the photographer's subjective judgment based on past experience. For this reason, depending on the user's level of proficiency in the photographing work for 3D measurement, the photographing work may be performed in an inappropriate manner. Therefore, it is desirable to provide work support so that the user can take a photographing work in an appropriate manner, regardless of the user's level of proficiency. In particular, since the conditions and timing in which work support is required differ depending on the user, it is desirable to be able to set conditions for work support individually for each user.

[0006] Therefore, the main object of the present invention is to provide a photography device, a photography work support method, and a photography work support program that enable the conditions for providing work support to a user during photography work to be set individually for each user, thereby providing appropriate work support according to the user's level of proficiency, etc. [Means for solving the problem]

[0007] The photographing device of the present invention is an imaging device that photographs a location to be measured in order to perform 3D measurement processing to generate 3D spatial information of the location to be measured based on a photographed image of the location to be measured, and is equipped with a device main body held by a user, a photographing unit provided in the device main body for photographing the location to be measured, a display unit that displays work support information related to a photographing task in which the user moves around the location to be measured and has the photographing unit photograph the location to be measured, and a processor that controls the photographing unit and the display unit, and the processor is configured to display a customization screen on the display unit including an operation input unit for selecting an item to be performed from a plurality of items, determine whether or not work support is required for the user and the content of the item selected on the customization screen, and if work support is required, execute a work support process that displays the work support information on the display unit.

[0008] In addition, the photography support method of the present invention is a photography support method in which a processor performs processing to support a user's photography work by having the photography device photograph the location to be measured while holding the photography device and moving around the location to be measured, in order to perform 3D measurement processing to generate 3D spatial information of the location to be measured based on a photographed image of the location to be measured, and is configured to display a customization screen on a display unit including an operation input unit for selecting an item to be performed from a plurality of items, determine whether or not work support is required for the user and the content of the support, and if work support is required, execute work support processing to display work support information on the display unit.

[0009] In addition, the photography work support program of the present invention is a photography work support program that causes a processor to execute a process to support the photography work of a user who holds a photography device and moves around the measurement target location while having the photography device photograph the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on a photographed image of the measurement target location, and is configured to display a customization screen on a display unit including an operation input unit for selecting an item to be performed from a plurality of items, determine whether or not and what content of work support is required for the user with respect to the item selected on the customization screen, and execute a work support process that displays work support information on the display unit if work support is required. Effect of the Invention

[0010] According to the present invention, a user can select an item for which work assistance is to be provided according to the user's own level of proficiency, etc., so that appropriate work assistance can be provided according to the user's level of proficiency, etc. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram showing a situation of a photographing operation using the photographing device according to the present embodiment; [Diagram 2] FIG. 13 is an explanatory diagram showing a support condition table used in processing by a processor; [Diagram 3] FIG. 1 is a block diagram showing a schematic configuration of an imaging device. [Figure 4] Block diagram showing an overview of the processing performed by the processor [Diagram 5] FIG. 1 is an explanatory diagram showing an overlapping region used in the point cloud density evaluation process and the feature point evaluation process. [Figure 6] A flow chart showing the steps of processing performed by a processor. [Figure 7] FIG. 1 is a flow diagram showing the procedure of blur-related support condition determination processing performed by a processor. [Figure 8] FIG. 11 is a flow diagram showing the procedure of a support condition determination process for point cloud density performed by a processor. [Figure 9]FIG. 11 is a flowchart showing the procedure of a process for determining an assistance condition regarding a feature point, which is performed by a processor. [Figure 10] FIG. 11 is a flow chart showing the procedure of a process for determining a support condition regarding a sudden action, which is executed by the processor. [Figure 11] FIG. 13 is a flowchart showing the procedure of a process for determining an assistance condition regarding elapsed time, which is performed by a processor. [Figure 12] A diagram showing the shooting mode screen displayed on the display. [Figure 13] FIG. 11 is an explanatory diagram showing a display state of a warning message on a screen in a shooting mode; [Figure 14] FIG. 11 is an explanatory diagram showing how a notification message is displayed on the screen in shooting mode. [Figure 15] An explanatory diagram showing the setting mode screen displayed on the display [Figure 16] FIG. 11 is an explanatory diagram showing an operation for changing the priority of each item on the setting mode screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] A first invention made to solve the above problem is an imaging device that photographs a location to be measured in order to perform 3D measurement processing to generate 3D spatial information of the location to be measured based on a photographed image of the location to be measured, the imaging device comprising: a device main body held by a user; an imaging unit provided in the device main body for photographing the location to be measured; a display unit that displays work support information related to a photography task in which the user moves around the location to be measured and has the imaging unit photograph the location to be measured; and a processor that controls the imaging unit and the display unit, the processor being configured to display on the display unit a customization screen including an operation input unit for selecting an item to be performed from a plurality of items, determine whether or not work support is required for the user and the content of the item selected on the customization screen, and if work support is required, execute a work support process that displays the work support information on the display unit.

[0013] This allows the user to select the item for which work assistance will be provided according to the user's own level of proficiency, etc., so that appropriate work assistance can be provided according to the user's level of proficiency, etc.

[0014] In addition, in a second invention, the processor is configured to display the customization screen including an operation input unit for individually specifying whether or not to enable processing for each of a plurality of items regarding the work support information that prompts a user to take action to improve the state of the photographing work.

[0015] According to this, the user can individually specify whether or not to enable processing for each item of work support information (e.g., warning messages) that prompts the user to take an action to improve the state of the user's photographing work. In this case, the operation input unit may be, for example, a check box provided on the screen.

[0016] In addition, in a third invention, the processor is configured to display the customization screen including an operation input unit for individually specifying whether or not to enable processing for each of a plurality of items regarding the work support information that prompts a user to take an action to improve the state of the photographing work.

[0017] According to this, the user can individually specify whether or not to enable processing for each item of work support information (e.g., a notification message) that prompts the user to take an action to improve the state of the user's photographing work. In this case, the operation input unit may be, for example, a check box.

[0018] In addition, a fourth invention is configured such that the processor displays the customization screen including an operation input unit that instructs saving of setting information obtained by a user's operation input on the customization screen and loading of the saved setting information.

[0019] This allows the user to instruct the saving and loading of setting information, thereby reducing the labor required for setting up when starting a photographing operation.

[0020] In a fifth aspect of the present invention, the processor displays the customization screen including an operation input unit for inputting a judgment condition for judging whether or not a task support is required for a user.

[0021] According to this, the user can specify the judgment conditions for judging the necessity of the work support. In this case, the operation input unit for the judgment conditions may be, for example, an input unit for numbers representing a threshold value as a judgment criterion for judging the necessity of the work support. Furthermore, the operation input unit for the judgment conditions may be one for adjusting the level of the judgment criterion.

[0022] In addition, in a sixth invention, the processor is configured to display on the display unit the customization screen, which includes an operation input unit for specifying a priority for implementing work support for each of a plurality of items based on a determination result of the need for work support for each of a plurality of items.

[0023] This allows the user to specify the priority of each item. In this case, the priority may be changed by, for example, changing the order of the display parts of each item by drag and drop.

[0024] Further, a seventh invention is a photography support method in which a processor performs processing to support a user's photography work by having the photography device photograph the location to be measured while moving within the location to be measured while holding an photography device, in order to perform 3D measurement processing for generating 3D spatial information of the location to be measured based on photographed images of the location to be measured, the method being configured to display a customization screen on a display unit including an operation input unit for selecting an item to be performed from a plurality of items, determine whether or not work support is required for the user and the content of the support, and if work support is required, execute work support processing to display work support information on the display unit.

[0025] According to this, as in the first invention, the user can select the items for which work assistance will be provided based on his / her own level of proficiency, etc., so that appropriate work assistance can be provided based on the user's level of proficiency, etc.

[0026] Further, an eighth invention is a photography work support program that causes a processor to execute a process to support a user's photography work by having the photography device photograph the location to be measured while moving within the location to be measured while holding an photography device, in order to perform 3D measurement processing for generating 3D spatial information of the location to be measured based on photographed images of the location to be measured, and is configured to display a customization screen on a display unit including an operation input unit for selecting an item to be performed from a plurality of items, determine whether or not work support is required for the user and the content of the work support for the item selected on the customization screen, and execute a work support process that displays work support information on the display unit if work support is required.

[0027] According to this, as in the first invention, the user can select the items for which work assistance will be provided based on his / her own level of proficiency, etc., so that appropriate work assistance can be provided based on the user's level of proficiency, etc.

[0028] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0029] FIG. 1 is an explanatory diagram showing the situation of a photographing operation performed by a user using a photographing device 1 according to this embodiment.

[0030] The photographing device 1 includes a device body 11 and a sensor unit 12. The sensor unit 12 includes a visible camera 21 (photographing unit). The visible camera 21 is a monocular camera that detects visible light to photograph a subject, and outputs a photographed image, for example, a color image in an RGB format. The photographing device 1 can be configured as a tablet terminal or a notebook PC.

[0031] A user (worker) walks through a location to be measured while holding the device body 11 of the photographing device 1. At this time, the photographing device 1 photographs the location to be measured with the visible camera 21, and photographed images of each point in the location to be measured are sequentially acquired.

[0032] In this embodiment, the photographing device 1 is a three-dimensional measuring device. That is, the photographing device 1 performs three-dimensional measurement processing based on photographed images of each point sequentially acquired by the visible camera 21, and generates three-dimensional spatial information on the measurement target location. In the three-dimensional measurement processing, point cloud data (environmental map) is generated as three-dimensional spatial information on the measurement target location using the SLAM method.

[0033] 1 shows an example in which a user holding the device main body 11 walks through the measurement target location to photograph the measurement target location, but the measurement target location may also be photographed using a self-propelled robot equipped with the photographing device 1. In this embodiment, the user is assisted in the work, and when the user himself holds the device main body 11 and performs the photographing work, the work assistance is provided to the user. On the other hand, when a self-propelled robot photographs, the work assistance is provided to the user operating the robot.

[0034] In this embodiment, the photographing device 1 equipped with a sensor unit 12 such as a visible camera 21 performs 3D measurement processing, but the sensor unit 12 may be configured as a photographing device 1 independent of a PC (information processing device) that performs the 3D measurement processing.

[0035] In addition, the user photographs the measurement target location with the imaging device 1 held in the hand while walking through the measurement target location. Therefore, the imaging work includes the action of walking (moving) while holding the imaging device 1, the action of pointing the imaging device 1 in an appropriate direction, etc.

[0036] Next, there will be explained an outline of the processing performed by the processor 16. FIG.

[0037] In this embodiment, messages (work support information) are displayed on the screen of the photographing device 1 as work support. The messages include warning messages, attention messages, and notification messages. Warning messages encourage the user to take action to improve the state of photographing work that needs to be improved immediately. Attention messages encourage the user to pay attention to a specific matter. Notification messages encourage the user to take action to improve the state of photographing work that does not need to be improved immediately but has room for improvement, for example, action to make the photographing work more efficient.

[0038] In this embodiment, the support condition table is referenced, and the necessity and content of work support for each item is determined based on the evaluation value for each item (blur, point cloud density, feature points, sudden movements, elapsed time, etc.) (support condition determination process). In the support condition table, threshold values ​​are registered as determination conditions for each item, and the evaluation value of each item is compared with the threshold value for each item to determine the necessity and content of work support for each item.

[0039] As shown in FIG. 2(A), the assistance condition table sets, for each warning message item (blur, point cloud density, feature points, sudden movements, etc.), the criteria for determining whether or not work assistance is required through a warning message, and the content of the message to be displayed on the screen.

[0040] Here, if the user moves the image capturing device 1 too fast during shooting, blurring occurs in the image captured by the visible camera 21, and the accuracy of the point cloud data decreases. Therefore, in this embodiment, an evaluation value (blur evaluation value) that indicates the occurrence of blur in the captured image is calculated, and a state in which the user is moving the image capturing device 1 too fast is detected based on the blur evaluation value. In this case, a warning message is displayed on the screen as work support, urging the user to slow down the speed at which the image capturing device 1 is moved.

[0041] Specifically, the blur evaluation value is defined so that the smaller the value, the stronger the blur. A threshold value B1 is set for the blur evaluation value, and if the blur evaluation value is less than the threshold value B1 (blur evaluation value<threshold value B1), it is determined that the user is moving the image capture device 1 too fast. In this case, a warning message urging the user to slow down the speed at which the image capture device 1 is moved, for example, a warning message saying "Blurring is occurring (please move slowly)", is displayed on the screen.

[0042] Furthermore, if the user moves the image capture device 1 too fast during image capture, the point cloud will not be generated with sufficient density, resulting in a decrease in the accuracy of the point cloud data. Therefore, in this embodiment, an evaluation value (point cloud density evaluation value) that indicates the point cloud generation status is calculated, and a state in which the user is moving the image capture device 1 too fast is detected based on the point cloud density evaluation value. In this case, a warning message is displayed on the screen to prompt the user to slow down the speed at which the image capture device 1 is moved, as work support.

[0043] Specifically, the point cloud density evaluation value is defined such that the smaller the value, the lower the point cloud density. In addition, a threshold D1 is set for the point cloud density evaluation value, and when the point cloud density evaluation value is less than the threshold D1 (point cloud density evaluation value<threshold D1), it is determined that the user moves the image capture device 1 too fast. In this case, a warning message such as "Point cloud density decrease (please move slowly)" is displayed on the screen as a warning message to encourage the user to slow down the speed at which the image capture device 1 is moved. It is also possible that the point cloud density can be improved by repeatedly photographing the same location. For this reason, the point cloud density evaluation value may be calculated only from a newly generated point cloud, or may be calculated including not only a newly generated point cloud but also point clouds generated in the past.

[0044] Furthermore, when an area including many areas without many distinctive patterns or objects, such as walls, floors, and ceilings, is photographed, the extraction of feature points fails and the number of extracted feature points decreases, resulting in a decrease in the accuracy of the point cloud data. Therefore, in this embodiment, an evaluation value (the number of feature points) that indicates the extraction status of feature points is calculated, and a state in which the user is pointing the image capture device 1 in an inappropriate direction is detected based on the number of feature points. In this case, a warning message is displayed on the screen as work support, urging the user to point the image capture device 1 in an appropriate direction.

[0045] Specifically, a threshold Q1 is set for the number of feature points, and when the number of feature points is less than the threshold Q1 (number of feature points<threshold Q1), it is determined that the direction in which the user points the image capturing device 1 is not appropriate. In this case, a warning message such as "fewer feature points (please set an angle of view that includes distinctive patterns or objects)" is displayed on the screen as a warning message to encourage the user to point the image capturing device 1 in an appropriate direction. In addition, a warning message such as "fewer feature points (please set an angle of view that includes things other than walls and floors)" may also be displayed on the screen.

[0046] Furthermore, within the range in which the IMU 23 (see FIG. 3) operates normally, position and orientation tracking is less likely to fail due to correction of position and orientation information based on detection data from the IMU 23. Therefore, even when an area including many locations with few features such as walls, floors, and ceilings is captured, position and orientation tracking may be able to continue with little error. On the other hand, if the user moves the image capture device 1 too fast during capture, causing the image capture device 1 to move outside the range in which the IMU 23 operates normally, position and orientation tracking is more likely to fail.

[0047] Therefore, when the number of feature points is less than the threshold Q1, it may be determined that the situation is such that tracking of the position and orientation is likely to fail if the user moves the image capture device 1 too fast, and a warning message urging the user to slow down the speed at which the image capture device 1 is moved, for example, a warning message stating "fewer feature points (please move slowly)", may be displayed on the screen. This allows the user to slow down the speed at which the image capture device 1 is moved, and the image capture device 1 is moved within a range in which the IMU 23 operates normally, thereby stabilizing point cloud generation.

[0048] In addition, if the user moves the camera device 1 too quickly while taking a picture, the accuracy of the point cloud data will decrease. However, in addition to blur, point cloud density, and feature points as described above, it is possible to detect whether the user is moving the camera device 1 too quickly based on the acceleration and angular velocity generated in the camera device 1 itself as the user moves the camera device 1.

[0049] Therefore, in this embodiment, a sudden acceleration / deceleration state in which the acceleration generated in the photographing device 1 falls outside a specified range, or a sudden rotation state in which the angular velocity generated in the photographing device 1 falls outside a specified range is detected as a sudden movement by the user, and when the user makes a sudden movement, a warning message is displayed on the screen as work assistance, urging the user to slow down the speed at which the photographing device 1 is moved.

[0050] Specifically, the sudden movement evaluation value (an index based on data detected by the IMU 23) is defined so that the larger the value, the suddener the movement. A threshold value S is set for the sudden movement evaluation value, and when the sudden movement evaluation value is greater than the threshold value S (sudden movement evaluation value>threshold value S), it is determined that the user is moving the image capture device 1 too fast. In this case, a warning message urging the user to slow down the speed at which the image capture device 1 is moved is displayed on the screen, for example, a warning message stating "Sudden acceleration / deceleration / sudden rotation has occurred (please move at a slower speed)."

[0051] In addition, as shown in FIG. 2(B), the assistance condition table sets, for each item (such as elapsed time) of a warning message, a judgment condition for determining whether or not work assistance is required through the warning message, and the content of the message to be displayed on the screen.

[0052] When the shooting time becomes long, the processing load increases, and measurement errors accumulate (pile up), decreasing the accuracy of the point cloud data. Therefore, in this embodiment, an upper limit time is set for shooting, and when the elapsed time from the start of shooting approaches the upper limit time, a warning message is displayed on the screen to alert the user as work support.

[0053] Specifically, a threshold value T is set for the elapsed time, and when the elapsed time is greater than the threshold value T (elapsed time>threshold value T), a warning message such as "The upper limit of shooting time is approaching" is displayed on the screen. Note that the threshold value T may be calculated by multiplying the upper limit time by a predetermined ratio (e.g., 90%).

[0054] Furthermore, as shown in FIG. 2(C), the assistance condition table sets, for each item (such as blur, point cloud density, feature points, etc.) of notification messages, the judgment conditions for determining whether or not work assistance is required through a notification message, and the contents of the message to be displayed on the screen.

[0055] If the user moves the photographing device 1 too slowly during photographing, the efficiency of the photographing work decreases. Therefore, in this embodiment, a state in which the user moves the photographing device 1 too slowly or a state in which the moving speed does not need to be slowed down are detected based on the blur evaluation value, the point cloud density evaluation value, and the number of feature points. If the user moves the photographing device 1 too slowly or if the moving speed does not need to be slowed down, a notification message is displayed on the screen as work assistance, urging the user to move the photographing device 1 faster.

[0056] Specifically, a threshold value B2 is set for the blur evaluation value, and if the blur evaluation value is greater than the threshold value B2 (blur evaluation value>threshold value B2), it is determined that the user moves the photographing device 1 too slowly. Also, a threshold value D2 is set for the point cloud density evaluation value, and if the point cloud density evaluation value is greater than the threshold value D2 (point cloud density evaluation value>threshold value D2), it is determined that the user moves the photographing device 1 too slowly. Also, a threshold value Q2 is set for the number of feature points, and if the number of feature points is greater than the threshold value Q2 (number of feature points>threshold value Q2), it is determined that the user does not need to slow down the speed at which the photographing device 1 is moved. When it is determined that the user moves the photographing device 1 too slowly or that the moving speed does not need to be slowed down, a notification message urging the user to move the photographing device 1 faster, for example, a message saying "Good shooting (movement can be sped up)" is displayed on the screen.

[0057] In addition, in this embodiment, the assistance condition table is referenced, and based on the priority (order of priority) of each item, an item for which assistance is to be provided is selected from among the items determined to require assistance, and the content of the work assistance to be provided, i.e., a message to be displayed as an image (warning message, attention message, notification message), is determined (assistance content determination process).

[0058] As shown in Fig. 2 (A), (B), and (C), the support condition table sets priorities for displaying messages. When it is determined that work support is required for multiple items, work support is provided only for items with high priority. At this time, only one item with the highest priority may be selected, or a predetermined number of items may be selected starting from the highest priority. Priorities are set based on the level of urgency, etc.

[0059] In addition, when a priority is not set and it is determined that work support is required for multiple items, work support (display of a message) may be performed for all of the items. In addition, when it is determined that work support is required for multiple items, work support is performed for all of the items, but messages may be displayed on the screen in order from top to bottom based on the priority.

[0060] In addition, in the assistance condition table, a display position of the message on the screen is set for each of the warning message, attention message, and notification message.

[0061] Warning messages are displayed in the center of the screen (see FIG. 13). In this example, warning messages for the items blur, point cloud density, feature points, and sudden movements are displayed in the center of the screen. Meanwhile, notification messages and attention messages are displayed at the bottom of the screen (see FIG. 14). In this example, notification messages for blur, point cloud density, and feature points, and attention messages for elapsed time are displayed at the bottom of the screen.

[0062] The thresholds and priorities registered in the assistance condition table may be fixed values, but in this embodiment, the user can specify them on the setting mode screen (see FIG. 15).

[0063] Next, a description will be given of a schematic configuration of the photographing device 1. Fig. 3 is a block diagram showing a schematic configuration of the photographing device 1. Fig. 4 is a block diagram showing an overview of the processing performed by the processor 16. Fig. 5 is an explanatory diagram showing an overlapping region used in the point cloud density evaluation processing and the feature point evaluation processing.

[0064] As shown in FIG. 3, in addition to the sensor unit 12, the photographing device 1 includes a display 13 (display unit), an input device 14, a memory 15 (storage unit), and a processor 16 (CPU).

[0065] In addition to the visible camera 21, the sensor unit 12 includes a depth camera 22 and an IMU 23 (Inertial Measurement Unit).

[0066] The depth camera 22 is a stereo camera that detects infrared light to capture an image of a subject, and outputs depth information (distance image) as a detection result. The distance to the subject can be measured based on the detection result of the depth camera 22. Note that the depth camera 22 may be a sensor capable of acquiring depth information by other methods, such as LiDAR, in addition to a stereo camera.

[0067] The IMU 23 (detection unit) detects the motion state of the device itself, specifically, three-dimensional angular velocity and acceleration. Based on the detection result of the IMU 23, the amount of movement and rotation of the image capturing device 1 can be measured.

[0068] The visible camera 21, the depth camera 22, and the IMU 23 may not be integrated into a sensor unit. Alternatively, the depth camera 22 and the IMU 23 may be omitted, and only the visible camera 21 may be provided. Alternatively, the visible camera 21 may be provided together with either the depth camera 22 or the IMU 23.

[0069] The display 13 presents various information related to the photographing operation to the user, and displays a photographing mode screen 101 (see FIG. 12) and a setting mode screen 201 (see FIG. 15), etc.

[0070] The input device 14 is used by the user to perform input operations. The input device 14 may be a keyboard, a mouse, a touch pad, a touch panel, etc. When the photographing device 1 is configured as a tablet terminal, a touch panel display in which the touch panel as the input device 14 and the display panel as the display 13 are integrated is provided.

[0071] The memory 15 stores programs and the like executed by the processor 16. The memory 15 also stores images captured by the visible camera 21, distance information from the depth camera 22, and detection data from the IMU 23. The memory 15 also stores measurement results (point cloud data) generated by the processor 16.

[0072] The processor 16 performs various processes by executing the programs stored in the memory 15. In this embodiment, the processor 16 performs a three-dimensional measurement process P1, a task support process P2, a display information generation process P3, a display process P4, and the like.

[0073] The three-dimensional measurement process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0074] The work support process P2 includes a blur evaluation process P21, a point cloud density evaluation process P22, a feature point evaluation process P23, a sudden movement evaluation process P24, a time passage evaluation process P25, an assistance condition determination process P26, and an assistance content determination process P27.

[0075] In this embodiment, the photographing device 1 performs the 3D measurement process P1, the work support process P2, and the display information generation process P3, but all or part of these processes may be performed by a server device (not shown) capable of communicating with the photographing device 1.

[0076] As shown in FIG. 4, in a feature extraction process P11, the processor 16 extracts feature information (feature points, etc.) from an image (frame) captured by the visible camera 21.

[0077] In tracking process P12, processor 16 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition related to the position and orientation of the image capture device 1, and updates the position and orientation trajectory data based on the amount of transition. Note that the position and orientation trajectory data is obtained by tracking the position and orientation of the image capture device 1, and includes the tracking results of the position and orientation at the time of capturing each captured image (frame), i.e., information on the position and orientation of the image capture device 1 at each time of capturing.

[0078] In the position and orientation correction process P13, the processor 16 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, the position and orientation trajectory data is corrected so as to compensate for the measurement results of places with few features, such as walls and ceilings.

[0079] In the point cloud generation process P14, the processor 16 generates point cloud data based on the distance information of the depth camera 22 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13.

[0080] In the blur evaluation process P21, the processor 16 obtains an evaluation value (blur evaluation value) that represents the occurrence of blur in the captured image by image analysis of the captured image. The blur evaluation value evaluates the occurrence of blur in the captured image, and the smaller the value, the stronger the blur. Note that the image analysis performed in the blur evaluation process P21 can employ various known techniques, and for example, evaluates the occurrence of blur based on edges included in the captured image.

[0081] In another pattern of the blur evaluation process P21, the processor 16 acquires an evaluation value (blur evaluation value) representing the occurrence of blur in the captured image based on the exposure time of the visible camera 21 and the detection data (acceleration, angular velocity) of the IMU 23. Here, since it is highly likely that blur is occurring strongly when the exposure time is long, and it is also highly likely that blur is occurring strongly when the acceleration and angular velocity occurring in the image capture device 1 are large, it is possible to estimate the occurrence of blur based on the exposure time of the visible camera 21 and the detection data (acceleration, angular velocity) of the IMU 23.

[0082] The blur evaluation process P21 may be performed based on only one of the exposure time of the visible camera 21 and the detection data of the IMU 23. The exposure time of the visible camera 21 itself has little to do with the user's photographing work, but the effect of the motion state of the photographing device 1 on the occurrence of blur varies depending on the length of the exposure time. For example, if the exposure time is long, significant blur occurs if the acceleration and angular velocity of the photographing device 1 are not sufficiently small, and conversely, if the exposure time is short, significant blur does not occur even if the acceleration and angular velocity of the photographing device 1 are somewhat large.

[0083] In the point cloud density evaluation process P22, the processor 16 counts the number of point clouds in the target range based on the point cloud data generated in the point cloud generation process P14, and obtains an evaluation value (point cloud density evaluation value) that indicates the point cloud generation status. The point cloud density evaluation value indicates whether the real-time point cloud generation is being performed appropriately. The smaller the point cloud density evaluation value, the lower the point cloud density.

[0084] The point cloud density evaluation value may be calculated from the number of point clouds generated in a predetermined unit time (for example, one second) (the total number of points generated in the unit time) and the size of the shooting range of the visible camera 21, as shown in the following formula. In this case, if the number of point clouds generated in the entire shooting range of the visible camera 21 is small, the point cloud density evaluation value will be small. Point cloud density evaluation value = (number of point clouds generated per unit time) / (shooting range)

[0085] The point cloud density evaluation value may be the number of generated points (total number of points included in the overlapping area) in the overlapping area (see FIG. 5) between the shooting range at the current shooting time t and the shooting range at the previous shooting time t-1. In this case, the smaller the numerical value of the point cloud density evaluation value, the lower the point cloud density. In addition to the fact that the number of generated points in the shooting range of the visible camera 21 is small, the point cloud density evaluation value becomes small when the overlapping area between the shooting range at the current shooting time t and the shooting range at the previous shooting time t-1 is narrow. It is also considered that the density of the point cloud is improved by repeatedly shooting the same place. For this reason, the point cloud density evaluation value may be obtained only from the newly generated point cloud, or may be obtained by including not only the newly generated point cloud but also the point cloud generated in the past. In this case, the number of generated points is the total number of all point clouds included in the overlapping area, not only the point cloud generated at the current shooting time t.

[0086] In the feature point evaluation process P23, the processor 16 counts the feature points within the target range based on the extraction result of the feature extraction process P11, and obtains an evaluation value (the number of feature points) that indicates the extraction status of the feature points.

[0087] The number of feature points may be the number of feature points extracted in one captured image (one frame), or the number of feature points extracted in an overlapping area (see FIG. 5) between the capture range at the current capture time t and the capture range at the previous capture time t-1.

[0088] In the sudden movement evaluation process P24, the processor 16 acquires an evaluation value (sudden movement evaluation value) representing the degree of suddenness of the movement of the user moving the photographing device 1 based on the detection data (acceleration, angular velocity) of the IMU 23. Note that the degree of suddenness of the movement represents the degree of suddenness of the change in the position and orientation of the own device, and specifically represents the degree of acceleration (including deceleration) occurring in the photographing device 1, or the degree of angular velocity occurring in the photographing device 1.

[0089] In another pattern of the sudden movement evaluation process P24, the processor 16 acquires a sudden movement evaluation value based on the amount of transition regarding the position and orientation of the image capturing device 1 acquired in the tracking process P12. Here, the amount of transition represents a change in the position and orientation of the image capturing device 1, and the suddenness of the movement can be evaluated based on the magnitude of the amount of transition.

[0090] In the time lapse evaluation process P25, the processor 16 measures the time elapsed from the start of shooting and obtains an evaluation value (elapsed time) that indicates the lapse of shooting time. Specifically, when shooting starts, a timer starts counting, and the time measured by the timer at the time of measurement is obtained as the elapsed time.

[0091] In the assistance condition determination process P26, the processor 16 refers to the assistance condition table (see FIG. 2) and determines the necessity and content of work assistance for each item based on the evaluation value for each item (blur, point cloud density, feature points, sudden movement, and elapsed time). In the assistance condition table, a threshold value is registered as a determination condition for each item, and the evaluation value of each item is compared with the threshold value for each item.

[0092] In the support content determination process P27, processor 16 refers to the support condition table (see FIG. 2) and selects an item for which work support is to be performed from among the items determined to require work support based on the priority of each item, and determines the content of the work support, i.e., a message to be displayed as an image (warning message, attention message, notification message).

[0093] In the display information generation process P3, the processor 16 generates display information for a screen to be displayed on the display 13 based on the image captured by the visible camera 21, the point cloud data generated in the point cloud generation process P14, and the elapsed time acquired in the time passage evaluation process P25. Also, in the display information generation process P3, when work support is necessary, the processor 16 generates display information for a screen including the content of work support (warning message, attention message, notification message) determined in the support content determination process P27.

[0094] In the display process P4, the processor 16 displays a screen (see FIGS. 12 to 15) on the display 13 based on the display information generated in the display information generation process P3.

[0095] In this embodiment, blur evaluation processing P21, point cloud density evaluation processing P22, feature point evaluation processing P23, sudden movement evaluation processing P24, and time passage evaluation processing P25 are performed as processes for obtaining evaluation values ​​for determining whether or not work assistance is required, but only some of these processes may be performed.

[0096] Next, a description will be given of the procedure of the process performed by the processor 16. FIG.

[0097] First, the processor 16 acquires an image captured by the visible camera 21, distance information from the depth camera 22, and detection data from the IMU 23 from the sensor unit 12 (ST101).

[0098] Next, the processor 16 extracts feature points from the image captured by the visible camera 21 (feature extraction process) (ST102).

[0099] Next, the processor 16 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition regarding the position and orientation of the image capturing device 1, and updates the position and orientation trajectory data based on the amount of transition (tracking process) (ST103).

[0100] Next, the processor 16 corrects the position and orientation trajectory data acquired in the tracking process based on the detection data of the IMU 23 (position and orientation correction process) (ST104).

[0101] Next, the processor 16 generates point cloud data based on the distance information of the depth camera 22 and the position and orientation trajectory data acquired in the tracking process and the position and orientation correction process (point cloud generation process) (ST105).

[0102] Next, the processor 16 acquires an evaluation value (blur evaluation value) that indicates the occurrence state of blur in the captured image (blur evaluation process) (ST106). At this time, the blur evaluation value is obtained by image analysis of the captured image. In addition, the blur evaluation value is obtained based on the exposure time of the visible camera 21 and the detection data of the IMU 23.

[0103] Next, processor 16 counts the point clouds within the target range based on the point cloud data generated in the point cloud generation process, and obtains an evaluation value (point cloud density evaluation value) representing the point cloud generation status (point cloud density evaluation process) (ST107).

[0104] Next, processor 16 counts the feature points within the target range based on the extraction result of the feature extraction process, and obtains an evaluation value (the number of feature points) that represents the extraction status of the feature points (feature point evaluation process) (ST108).

[0105] Next, the processor 16 acquires an evaluation value (sudden movement evaluation value) that indicates the abruptness of the user's movement of the photographing device 1 (sudden movement evaluation process) (ST109). At this time, the sudden movement evaluation value is calculated based on the detection data of the IMU 23. In addition, the sudden movement evaluation value is calculated based on the amount of transition related to the position and orientation of the photographing device 1 acquired in the tracking process.

[0106] Next, the processor 16 acquires an evaluation value (elapsed time) that indicates the lapse of the shooting time (time lapse evaluation process) (ST110). At this time, the timer is used to measure the elapsed time from the start of shooting.

[0107] The processing in ST106 to ST110 is not limited to the order shown in the figure.

[0108] Next, the processor 16 refers to the support condition table (see FIG. 2) and determines the necessity and content of work support for each item based on the evaluation value for each item (blur, point cloud density, feature points, sudden movement, and elapsed time) (support condition determination process) (ST111). Note that the support condition determination process for each item is shown in FIG. 7 to FIG. 11.

[0109] Next, the processor 16 determines whether or not it has been determined that work assistance is required for one or more items (ST112).

[0110] Here, if it is determined that work support is required for one or more items (Yes in ST112), processor 16 then refers to the support condition table (see FIG. 2) and selects an item for which work support is to be performed from among the items determined to require work support based on the priority of each item, and determines the content of the work support, i.e., a message to be displayed as an image (warning message, attention message, notification message) (support content determination process) (ST113).

[0111] Here, when it is determined that work support is required for multiple items, work support (display of a message) may be provided for all of the items. Also, from among the multiple items determined to require work support, only one item with the highest priority may be selected and work support may be provided. Also, from among the multiple items determined to require work support, a predetermined number of items may be selected in descending order of priority and work support may be provided.

[0112] Next, the processor 16 generates display information for the screen based on the determined work support content (display information generation process), and displays the screen on the display 13 based on the display information (display process) (ST114).

[0113] Next, there will be described the assistance condition determination process for blurring, which is performed by the processor 16. Fig. 7 is a flow chart showing the procedure of the assistance condition determination process for blurring.

[0114] Processor 16 refers to the support condition table (see FIG. 2) to determine whether or not work support regarding blur is necessary and the content of the support. In the support condition table, threshold values ​​B1 and B2 that are the determination conditions for a warning message and a notice message regarding blur are registered.

[0115] Specifically, first, processor 16 determines whether or not the blur evaluation value is less than threshold value B1 (ST201). If the blur evaluation value is less than threshold value B1 (Yes in ST201), processor 16 determines that a work support regarding blur is required and the content of the support is a warning message (ST202).

[0116] On the other hand, if the blur evaluation value is equal to or greater than threshold B1 (No in ST201), processor 16 then determines whether or not the blur evaluation value is greater than threshold B2 (ST203). If the blur evaluation value is greater than threshold B2 (Yes in ST203), processor 16 determines that blur-related task assistance is required and that the content is a notification message (ST204).

[0117] On the other hand, if the blur evaluation value is equal to or less than the threshold value B2 (No in ST203), the processor 16 determines that task support regarding blur is not required (ST205).

[0118] Next, a description will be given of the support condition determination process related to the point cloud density, which is performed by the processor 16. Fig. 8 is a flow diagram showing the procedure of the support condition determination process related to the point cloud density.

[0119] The processor 16 refers to the support condition table (see FIG. 2) to determine the necessity and content of work support related to the point cloud density. The support condition table has registered therein a threshold value D1 and a threshold value D2 that are the determination conditions for a warning message and a notice message related to the point cloud density, respectively.

[0120] Specifically, first, the processor 16 determines whether the point cloud density evaluation value is less than the threshold value D1 (ST301). If the point cloud density evaluation value is less than the threshold value D1 (Yes in ST301), the processor 16 determines that a task support related to the point cloud density is required and the content of the support is a warning message (ST302).

[0121] On the other hand, if the point cloud density evaluation value is equal to or greater than the threshold D1 (No in ST301), then processor 16 determines whether or not the point cloud density evaluation value is greater than threshold D2 (ST303). If the point cloud density evaluation value is greater than threshold D2 (Yes in ST303), processor 16 determines that task support related to point cloud density is required and that the content is a notification message (ST304).

[0122] On the other hand, if the point cloud density evaluation value is equal to or less than the threshold value D2 (No in ST303), the processor 16 determines that task support related to the point cloud density is not required (ST305).

[0123] Next, there will be described the assistance condition determination process for feature points, which is performed by the processor 16. Fig. 9 is a flowchart showing the procedure of the assistance condition determination process for feature points.

[0124] The processor 16 refers to the support condition table (see FIG. 2) to determine the necessity and content of work support related to the feature points. In the support condition table, threshold values ​​Q1 and Q2 that are the determination conditions for each of the warning message and the notification message related to the feature points are registered.

[0125] Specifically, first, processor 16 determines whether the number of feature points is less than threshold value Q1 (ST401). If the number of feature points is less than threshold value Q1 (Yes in ST401), processor 16 determines that task support related to feature points is required and that the content of the support is a warning message (ST402).

[0126] On the other hand, if the number of feature points is equal to or greater than threshold value Q1 (No in ST401), processor 16 then determines whether the number of feature points is greater than threshold value Q2 (ST403). If the number of feature points is greater than threshold value Q2 (Yes in ST403), processor 16 determines that task support related to feature points is required and that the content is a notification message (ST404).

[0127] On the other hand, if the number of feature points is equal to or less than the threshold value Q2 (No in ST403), the processor 16 determines that task support related to feature points is not required (ST405).

[0128] Next, there will be described the support condition determination process for a sudden movement, which is performed by the processor 16. Fig. 10 is a flow chart showing the procedure of the support condition determination process for a sudden movement.

[0129] The processor 16 refers to the support condition table (see FIG. 2) to determine the necessity and content of work support regarding sudden movements. The support condition table stores a threshold value S that is a determination condition for a warning message regarding a sudden movement.

[0130] Specifically, first, processor 16 determines whether or not the sudden movement evaluation value is greater than threshold value S (ST501). If the sudden movement evaluation value is greater than threshold value S (Yes in ST501), processor 16 determines that task support regarding the sudden movement is required and that the content of the support is a warning message (ST502).

[0131] On the other hand, if the sudden movement evaluation value is equal to or less than the threshold value S (No in ST501), the processor 16 determines that task support related to the sudden movement is not required (ST503).

[0132] In this embodiment, if the user moves the photographing device 1 too quickly while taking a picture, a warning message is displayed as work assistance to urge the user to slow down the speed at which the photographing device 1 is moved. However, if the user's movements are too slow based on the sudden movement evaluation value, a notification message may be displayed as work assistance to improve work efficiency to urge the user to move the photographing device 1 more quickly.

[0133] In this case, threshold values ​​S1 and S2 are set as judgment conditions for a warning message and a notification message, respectively, and if the sudden movement evaluation value is greater than threshold value S1, the content of the work support is judged to be a warning message, and if the sudden movement evaluation value is less than threshold value S2, the content of the work support is judged to be a notification message.

[0134] Next, a description will be given of assistance condition determination processing related to elapsed time, which is performed by the processor 16. Fig. 11 is a flow diagram showing the procedure of assistance condition determination processing related to elapsed time.

[0135] The processor 16 refers to the support condition table (see FIG. 2) to determine the necessity and content of work support related to the elapsed time. The support condition table has registered therein a threshold value T that is a determination condition for a warning message related to the elapsed time.

[0136] Specifically, first, processor 16 determines whether the elapsed time is greater than threshold value T (ST601). If the elapsed time is greater than threshold value T (Yes in ST601), processor 16 determines that task support related to the elapsed time is necessary and that the content of the support is a warning message (ST602).

[0137] On the other hand, if the elapsed time is equal to or less than the threshold value T (No in ST601), the processor 16 determines that task support related to the elapsed time is not required (ST603).

[0138] Next, a description will be given of the screen 101 in the shooting mode displayed on the display 13. Fig. 12 is an explanatory diagram showing the screen 101 in the shooting mode.

[0139] A shooting mode screen 101 is provided with a main window 102 and a sub-window 103. The display magnifications of the main window 102 and the sub-window 103 are different, so that the image is displayed enlarged in the main window 102 and reduced in the sub-window 103.

[0140] Furthermore, a captured image 121 and a map image 122 are displayed in the main window 102 and the sub window 103. The captured image 121 is a current captured image, that is, an image captured in real time and output from the visible camera 21. The map image 122 is an overhead point cloud image, that is, an image (rendering) of each point of the point cloud data as viewed from a viewpoint in the sky.

[0141] Furthermore, the shooting mode screen 101 is provided with a "start shooting" button 105 and an "end shooting" button 106. When the user operates the "start shooting" button 105, shooting by the visible camera 21 starts, and the captured image is stored in the memory 15. When the user operates the "end shooting" button 106, shooting by the visible camera 21 ends. Note that, if a depth camera 22 and an IMU 23 are present, distance information by the depth camera 22 and detection data by the IMU 23 are acquired in the same manner as shooting by the visible camera 21.

[0142] Furthermore, the image capture mode screen 101 is provided with a "screen switch" button 107 and a check box 108 related to display of the sub-window 103. When the user operates the "screen switch" button 107, a state is switched between a state in which the captured image 121 is displayed enlarged in the main window 102 and a map image 122 is displayed reduced in the sub-window 103 (the state shown in FIG. 12) and a state in which the map image 122 is displayed enlarged in the main window 102 and the captured image 121 is displayed reduced in the sub-window 103. When the user operates the check box 108, the sub-window 103 can be switched between a display state and a non-display state.

[0143] Furthermore, the shooting mode screen 101 is provided with a time elapsed status display section 111 and a check box 112 for displaying the time elapsed status display section 111. The time elapsed status display section 111 displays the elapsed time from the start of shooting. When the user operates the check box 112, the time elapsed status display section 111 can be switched between a displayed state and a non-displayed state.

[0144] In this example, the time elapsed from the start of shooting is displayed on the time elapsed status display unit 111, but instead of or together with the elapsed time, the remaining time until the upper limit time may be displayed. Also, the ratio of the elapsed time and the remaining time to the upper limit time may be displayed as the time elapsed status. In this case, an image showing the time elapsed status, for example, a level indicator whose display changes with the passage of time, may be displayed.

[0145] Furthermore, the shooting mode screen 101 is provided with a "Shooting" tab 131 and a "Settings" tab 132. The user can switch between the shooting mode and the setting mode by operating the tabs 131, 132. The shooting mode screen 101 shown in FIG. 12 is when the user operates the "Shooting" tab 131. When the user operates the "Settings" tab 132, the screen transitions to a setting mode screen 201 (see FIG. 15).

[0146] In this embodiment, the shooting mode and the setting mode are switched by operating the tabs 131 and 132, but the shooting mode and the setting mode may be switched by another type of operation input unit such as a button.

[0147] Next, the display status of the warning message, the attention message, and the notification message will be described. Fig. 13 is an explanatory diagram showing the display status of the warning message on the screen 101 in the shooting mode. Fig. 14 is an explanatory diagram showing the display status of the notification message on the screen 101 in the shooting mode.

[0148] As shown in Fig. 13, when it is determined that work assistance is necessary and the content of the work assistance is a warning message, a first message display window 141 including the warning message is superimposed on the center of the shooting mode screen 101. The example shown in Fig. 13 is a case where the content of the work assistance is a warning message regarding the blur item, and a warning message saying "Blurring occurring (please move slowly)" is displayed.

[0149] When the content of the work support is a warning message related to the point cloud density item, the first message display window 141 containing the warning message "Point cloud density decreasing (please move slowly)" is displayed in the center of the screen. When the content of the work support is a warning message related to the feature point item, the first message display window 141 containing the warning message "Feature points decreasing (please set the angle of view to include distinctive patterns or objects)" is displayed in the center of the screen. When the content of the work support is a warning message related to the sudden movement item, the first message display window 141 containing the warning message "Sudden movement occurred (please move slowly)" is displayed in the center of the screen.

[0150] The first message display window 141 continues to be displayed unless the user's photographing work status improves, and disappears when the user's photographing work status improves.

[0151] As shown in Fig. 14, when it is determined that work support is necessary and the content of the work support is a notification message, a second message display window 142 including the notification message is superimposed on the bottom of the shooting mode screen 101. The example shown in Fig. 14 is a case where the content of the work support is a notification message related to any one of blur, point cloud density, feature points, and sudden movements, and a notification message saying "Good shooting (movement speed can be increased)" is displayed.

[0152] Furthermore, when the content of the work support is a warning message regarding the elapsed time item, a second message display window 142 including a warning message saying "The upper limit of shooting time is approaching" is displayed at the bottom of the screen.

[0153] The second message display window 142 continues to be displayed if the user's photographing work status does not improve, and disappears when the user's photographing work status improves. Note that the second message display window 142 may display information that is helpful for the user's photographing work or information that calls attention to the user's work, in addition to a notice message as a work support.

[0154] In this manner, in this embodiment, warning messages for each of the items of blur, point cloud density, feature points, and sudden movements are displayed in the center of the screen 101 (see FIG. 13). This allows the user in the middle of shooting work to be informed in an easy-to-understand manner of the points that need to be improved immediately in the user's shooting work. In particular, the warning message provides textual information on the points that need to be improved for each item along with the state and cause, so the user can immediately recognize the points that need to be improved in the shooting work.

[0155] Meanwhile, notification messages regarding each of the items of blur, point cloud density, and feature points are displayed at the bottom of the screen 101 (see FIG. 14). This allows the user in the middle of shooting work to be guided in an easy-to-understand manner on the desired shooting method when the user's shooting work is good but there is room for improvement. In particular, the notification message, like the warning message, guides the points to be improved with text along with the state and cause, so that the user can immediately recognize the points to be improved in the shooting work.

[0156] Also, a message to alert the user regarding the item of elapsed time is displayed at the bottom of the screen 101 (see FIG. 14). This makes it possible to alert the user during shooting work to the elapsed time, which is an item that the user should be careful of when proceeding with shooting work.

[0157] In this embodiment, a warning message, a caution message, or a notification message is displayed on the screen 101 as the work support, but the work support is not limited to the message displayed on the screen 101. For example, an image (icon, mark, change in color scheme, animation, etc.) showing the same work support content as the warning message, caution message, or notification message may be displayed on the screen 101. Furthermore, the same work support content as the warning message, caution message, or notification message may be expressed by a change in the lighting color or blinking of a lamp provided in the photographing device 1. Furthermore, the same work support content as the warning message, caution message, or notification message may be expressed by a sound or voice message output from a speaker provided in the photographing device 1.

[0158] Next, a description will be given of the setting mode screen 201 (customization screen) displayed on the display 13. Fig. 15 is an explanatory diagram showing the setting mode screen 201.

[0159] The setting mode screen 201 displays a first setting item input section 211 for a warning message, a second setting item input section 212 for a reminder message, and a third setting item input section 213 for a notice message.

[0160] The first setting item input section 211 is provided with a check box 231 related to a warning message. The user can switch between enabling and disabling the processing of all items related to the warning message collectively by operating the check box 231. The first setting item input section 211 is also provided with a check box 232 for each item (blur, point cloud density, feature points, sudden movement). The user can switch between enabling and disabling the processing of each item individually by operating the check box 232. The first setting item input section 211 is also provided with an input field 233 for a threshold for each item. The user can input a number representing the threshold for each item into the input field 233.

[0161] The second setting item input section 212 is provided with a check box 241 related to the attention message. The user can switch between enabling and disabling the processing of all items related to the attention message collectively by operating the check box 241. The second setting item input section 212 is also provided with a check box 242 for each item (elapsed time). The user can switch between enabling and disabling the processing of each item individually by operating the check box 242. The second setting item input section 212 is also provided with an input field 243 for a threshold for each item. The user can input a number representing the threshold for each item into the input field 243. Note that, although only one item (elapsed time) can be set for the attention message in the example shown in FIG. 15, multiple items may be set.

[0162] The third setting item input section 213 is provided with a check box 251 related to the notification message. The user can switch between enabling and disabling the processing of all items related to the notification message collectively by operating the check box 251. The third setting item input section 213 is also provided with a check box 252 for each item (blur, point cloud density, feature point). The user can switch between enabling and disabling the processing of each item individually by operating the check box 252. The third setting item input section 213 is also provided with an input field 253 for a threshold value for each item. The user can input a number representing the threshold value for each item into the input field 253.

[0163] In this example, numbers representing threshold values ​​are input into input fields 233, 243, and 253 as operation input units for judgment conditions, but the user may adjust the level of the judgment criteria using an operation input unit such as a slide bar or button.

[0164] Furthermore, the setting mode screen 201 is provided with a "Save Settings" button 261 and a "Load Settings" button 262. When the user operates the "Save Settings" button 261, the setting information is saved with the current input contents. When the user operates the "Load Settings" button 262, the previous setting information is loaded and displayed in the setting item input areas 211, 212, and 213.

[0165] In this manner, in this embodiment, the user can specify the judgment condition (threshold) when judging the necessity of work support for each item. Therefore, the user can set an appropriate threshold according to the characteristics of the measurement target location. For example, if the measurement target location is a large space and there are not many subjects within a distance where a point cloud can be generated, it is difficult to generate a point cloud, and the point cloud density evaluation value may be equal to or less than the threshold, causing frequent warning messages. In this case, if the user finds the warning messages annoying, the user can avoid frequent warning messages by resetting the threshold to a smaller value.

[0166] Moreover, whether the process for each item is enabled or disabled and the judgment conditions (thresholds) for each item differ depending on the measurement target location. Therefore, setting information may be registered in advance for each type of measurement target location, and setting information suitable for the measurement target location may be loaded when starting the image capture work. This can reduce the labor required for setting work when starting the image capture work. Furthermore, by saving setting information customized by the user in association with information on the measurement target location at the end of image capture, the next time the same user captures the measurement target location, the setting information can be loaded and image capture can be started promptly.

[0167] Next, a description will be given of an operation for changing the priority of each item on the setting mode screen 201. Fig. 16 is an explanatory diagram showing the status of an operation for changing the priority of each item.

[0168] In the first setting item input section 211, a plurality of items (blur, point cloud density, feature points, and sudden movements) are displayed in order of priority from the top. In the third setting item input section 213, a plurality of items (blur, point cloud density, and feature points) are displayed in order of priority from the top. In the example shown in FIG. 15, only one item (elapsed time) is displayed in the second setting item input section 212, but a plurality of items may be displayed, in which case the plurality of items are displayed in order of priority from the top.

[0169] In the setting item input sections 211, 212, and 213 (operation input section), the user can change the priority of each item by performing a predetermined operation. For example, when the user performs a drag-and-drop operation on the display portion of each item in the setting item input sections 211, 212, and 213 to rearrange the order of the items, the priority is changed to the rearranged order.

[0170] Note that priority-based processing is performed when multiple items are enabled by operating check boxes 231, 241, and 251, and the content of the work support, i.e., warning messages, caution messages, and notification messages to be displayed as images, is determined for each of the multiple enabled items based on the priority.

[0171] Furthermore, the setting item input sections 211, 212, and 213 may be provided with an input field for inputting the priority, allowing the user to input the priority numerically.

[0172] Incidentally, it is difficult to imagine a warning message and a notice message being displayed at the same time, but it is possible that a warning message and a warning message will be displayed at the same time, and it is also possible that a notice message and a warning message will be displayed at the same time. In this case, if the display positions of the messages displayed at the same time are different (for example, the center and the bottom), they may be displayed at the same time, but if the display positions are the same (for example, both at the bottom), it is better to give priority to one of the messages or to display each message alternately.

[0173] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0174] The photographing device, photographing work support method, and photographing work support program of the present invention have the effect of enabling the conditions for providing work support to a user during photographing work to be set individually for each user, thereby enabling appropriate work support to be provided according to the user's level of proficiency, etc., and are useful as a photographing device that photographs each point of a measurement target location in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location, and as a photographing work support method and photographing work support program that support the photographing work of a user using the photographing device. [Explanation of symbols]

[0175] 1: Imaging device 11: Device body 12: Sensor section 13: Display 14: Input devices 15: Memory 16: Processor 21: Visible camera 22: Depth camera 23: IMU 101: Shooting mode screen 111: Time progress display section 121: Photographed image 122: Map image 141: First message display window 142: Second message display window 201: Setting mode screen 211: First setting item input section 212: Second setting item input section 213: Third setting item input section P1: 3D measurement processing P2: Work support processing P3: Display information generation process P4: Display processing P11: Feature extraction processing P12: Tracking process P13: Position and orientation correction processing P14: Point cloud generation processing P21: Blur evaluation processing P22: Point cloud density evaluation processing P23: Feature point evaluation processing P24: Sudden movement evaluation processing P25: Time-lapse evaluation process P26: Support condition determination process P27: Support content decision process

Claims

1. An imaging device that captures an image of a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on a captured image of the measurement target location, A device body held by a user; An imaging unit provided in the device body for imaging a measurement target location; a display unit that displays work support information related to an image capturing operation in which a user moves around a measurement target location and causes the image capturing unit to capture the measurement target location; A processor that controls the image capturing unit and the display unit, The processor, displaying a customization screen on the display unit, the customization screen including an operation input unit for selecting an item to be implemented from a plurality of items; The photographing device is characterized in that it determines whether or not work assistance is required for the user and the content of the assistance for the user regarding the item selected on the customization screen, and if work assistance is required, executes a work assistance process to display the work assistance information on the display unit.

2. The processor, The photographing device according to claim 1, characterized in that the customization screen is displayed including an operation input unit for individually specifying whether or not to enable processing for each of a plurality of items regarding the work support information that prompts a user to take action to improve the state of the photographing work.

3. The processor, The photographing device according to claim 1, characterized in that the customization screen is displayed including an operation input unit for individually specifying whether or not to enable processing for each of a plurality of items regarding the work support information that prompts a user to take action to improve the state of the photographing work.

4. The processor, The photographing device according to claim 1, characterized in that the customization screen is displayed including an operation input unit that instructs the saving of setting information acquired by a user's operation input on the customization screen and the reading of the saved setting information.

5. The processor, 2. The photographing apparatus according to claim 1, wherein the customization screen includes an operation input section for inputting a judgment condition for judging whether or not a user needs work assistance.

6. The processor, The photographing device according to claim 1, characterized in that the customization screen is displayed on the display unit, the customization screen including an operation input unit for specifying a priority for each item when implementing work assistance based on a judgment result obtained by judging whether or not work assistance is necessary for each of a plurality of items.

7. A photographing operation support method for performing a process by a processor to support a photographing operation of a user who holds a photographing device and moves within a measurement target location while photographing the measurement target location with the photographing device, in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on a photographed image of the measurement target location, the process comprising: displaying on the display unit a customization screen including an operation input unit for selecting an item to be implemented from among a plurality of items; A photographing work support method, characterized in that it determines whether or not work support is required for a user and the content of the support for that user with respect to an item selected on the customization screen, and if work support is required, executes work support processing to display work support information on the display unit.

8. A photography operation support program that causes a processor to execute a process for supporting a photography operation of a user who holds a photography device and moves within a measurement target location to photograph the measurement target location with the photography device, in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on a photographed image of the measurement target location, the process comprising: displaying on the display unit a customization screen including an operation input unit for selecting an item to be implemented from among a plurality of items; A photography work support program characterized by executing a work support process that determines whether or not work support is required for a user and the content of the work support for the user with respect to an item selected on the customization screen, and displays work support information on the display unit if work support is required.

Citation Information

Patent Citations

  • Information presenting device and information presenting method

    JP2003281297A

  • Photographing support device, photographing support system, photographing support method, and photographing support program

    JP2016051904A

  • Driving support apparatus and driving support system

    JP2018160136A

  • Surveying instrument with support function

    JP2019203712A

  • Three-dimensional measurement device and its measurement support processing method

    JP6489566B2