Photographing device, photographing work support method, and photographing work support program

Abstract

To provide a photographing device, a photographing operation support method, and a photographing operation support program that guide a point to be immediately improved in relation to a user's photographing operations in an easy-to-understand manner to the user performing the photographing operations, and thereby allow the user to execute the photographing operations according to an appropriate method regardless of the skill level of the user, or the like.SOLUTION: In order to perform three-dimensional measurement processing of generating three-dimensional space information of a measurement target place on the basis of a photographed image of the measurement target place, a photographing device that photographs the measurement target place, displays, on a display, operation support information related to photographing operations performed by a user photographing the measurement target place while moving in the measurement target place, especially, determines the necessity of operation support for improving the state of the user's photographing operations on the basis of the state of the user's photographing operations, specifically, the state of occurrence of a rapid movement that is a state where the user rapidly moves the device, and the time elapsed from the start of photographing, and when the operation support is required, displays a warning message or an attention attracting message for encouraging the user to take an action to improve the state of the user's photographing operations as the operation support information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an imaging device that captures images of each point in a measurement location in order to perform 3D measurement processing that generates 3D spatial information of the measurement location based on the captured images of each point in the measurement location, as well as a imaging work support method and imaging work support program that support a user's imaging work using the imaging device. [Background technology]

[0002] There is a known 3D measurement technology that generates 3D spatial information (map data) about a measurement target location based on images of the measurement target location. In recent years, the SLAM (Simultaneous Localization and Mapping) method has been attracting attention for this type of 3D measurement. The SLAM method can generate 3D spatial information and the location information of a mobile object as a measurement result based on images of each point in the measurement target location captured by a camera device attached to the mobile object. In particular, 3D measurement can be easily performed by a user (worker) using a handheld camera device to take photos while moving around the measurement target location, thereby acquiring images of each point in the measurement target location.

[0003] On the other hand, when a user photographs a measurement target location with a handheld camera while walking around the measurement target location, it is desirable to provide work support that encourages the user to perform appropriate photographing work so that highly accurate and stable 3D measurements can be performed.From this perspective, a technology has been known in the past that displays multiple marks, such as arrows, that indicate the direction and speed of movement of the photographing device (sensor) held by the user, to guide the user on how to proceed with the photographing work (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6489566 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, users are unable to intuitively grasp how to perform photography, and so the specific procedure for performing photography is determined subjectively by the photographer based on past experience. This poses a problem in that if a user is not sufficiently skilled in photography for 3D measurement, the photography procedure may be performed in an inappropriate manner. In particular, if the user's working conditions are such that they reduce the accuracy of 3D measurement, it is desirable to provide clear guidance on areas in the user's photography procedure that need to be improved immediately.

[0006] The photographing device of the present invention is a photographing device that photographs a measurement target location in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photographed images of the measurement target location, and includes: a device main body held by a user; a photographing unit provided in the device main body that photographs the measurement target location; and a display unit that displays work support information related to a photographing operation in which the user causes the photographing unit to photograph the measurement target location while moving around the measurement target location. an acquisition unit that acquires an evaluation value representing the state of an action of a user moving the device; and a control unit that controls the image capture unit, the display unit, and the acquisition unit. a processor, the processor comprising: Based on the evaluation value acquired by the acquisition unit, The system is configured to determine whether work assistance is necessary to improve the state of the photography work, and if work assistance is necessary, to display on the display unit as work assistance information information encouraging the user to take action to improve the photography work. [Means for solving the problem]

[0007] Furthermore, a photography operation support method of the present invention is a photography operation support method in which a processor performs a process to support a photography operation of a user who holds a photography device and moves within the measurement target location while causing the photography device to photograph the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on a photographed image of the measurement target location, and An evaluation value representing the state of the user's operation of moving the photographing device is obtained, and based on the evaluation value,The system determines whether work assistance is necessary to improve the state of the photography work, and if work assistance is necessary, the system displays work assistance information on the display unit, encouraging the user to take action to improve the photography work.

[0008] Furthermore, a photography operation support program of the present invention is a photography operation support program that causes a processor to execute a process for supporting a photography operation of a user who holds a photography device and moves within the measurement target location while causing the photography device to photograph the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on a photographed image of the measurement target location, and An evaluation value representing the state of the user's operation of moving the photographing device is obtained, and based on the evaluation value, The system determines whether work assistance is necessary to improve the state of the photography work, and if work assistance is necessary, the system displays work assistance information on the display unit, encouraging the user to take action to improve the photography work.

[0009] In addition, the photography work support program of the present invention is a photography work support program that causes a processor to execute a process to support the photography work of a user who holds a photography device and moves around the measurement target location while having the photography device photograph the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photographed images of the measurement target location.The photography work support program is configured to determine whether or not work support is needed to improve the status of the user's photography work based on the status of the user's photography work, and if work support is needed, to display information on the display unit as work support information to encourage the user to take action to improve the photography work. [Effects of the Invention]

[0010] According to the present invention, a user who is taking photographs can be given easy-to-understand guidance on points that need to be improved immediately regarding the user's photographing work, so that the user can carry out the photographing work in an appropriate manner regardless of the user's level of proficiency, etc. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram showing a situation of a photographing operation using the photographing device according to the present embodiment; [Figure 2] FIG. 10 is an explanatory diagram showing a support condition table used in the processing of the processor; [Figure 3] A block diagram showing the schematic configuration of an imaging device. [Figure 4] Block diagram showing the overall processing performed by the processor [Figure 5] FIG. 10 is an explanatory diagram showing an overlapping region used in the point cloud density evaluation process and the feature point evaluation process. [Figure 6] A flowchart showing the processing steps performed by the processor. [Figure 7] FIG. 1 is a flowchart showing the procedure for blur support condition determination processing performed by a processor. [Figure 8] FIG. 1 is a flowchart showing the procedure of a support condition determination process related to point cloud density performed by a processor. [Figure 9] FIG. 1 is a flowchart showing the procedure of a process for determining support conditions for feature points, which is performed by a processor. [Figure 10] FIG. 1 is a flowchart showing the procedure of a process for determining a support condition related to a sudden movement, which is performed by a processor. [Figure 11] FIG. 1 is a flowchart showing the procedure of a process for determining an assistance condition regarding elapsed time, which is performed by a processor. [Figure 12] An explanatory diagram showing the shooting mode screen displayed on the display [Figure 13] FIG. 10 is an explanatory diagram showing a display state of a warning message on a screen in shooting mode. [Figure 14] An explanatory diagram showing the display status of a notification message on the shooting mode screen. [Figure 15] An explanatory diagram showing the setting mode screen displayed on the display [Figure 16] FIG. 10 is an explanatory diagram showing an operation for changing the priority of each item on the setting mode screen. DETAILED DESCRIPTION OF THE INVENTION

[0012] The first invention made to solve the above problems is a photographing device for photographing a measurement target location in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photographed images of the measurement target location, the photographing device comprising: a device main body held by a user; a photographing unit provided in the device main body for photographing the measurement target location; and a display unit for displaying work support information related to a photographing operation in which the user causes the photographing unit to photograph the measurement target location while moving around the measurement target location. an acquisition unit that acquires an evaluation value representing the state of an action of a user moving the device; and a control unit that controls the image capture unit, the display unit, and the acquisition unit. a processor, the processor comprising: Based on the evaluation value acquired by the acquisition unit, The system is configured to determine whether work assistance is necessary to improve the state of the photography work, and if work assistance is necessary, to display on the display unit as work assistance information information encouraging the user to take action to improve the photography work.

[0015] Also, Second The invention further comprises: An inertial measurement unit that detects the acceleration and angular velocity of the device itself wherein the processor: The evaluation value is obtained based on the detection result of the inertial measurement unit. The composition is as follows.

[0016] According to this, Based on the evaluation values ​​of the acceleration (including deceleration) and angular velocity generated in the device itself, It is possible to appropriately determine whether or not work support is required for sudden movements. Cut.

[0017] Also, Third The invention is characterized in that the processor measures the position and orientation of the own device acquired in the 3D measurement process. Obtain the evaluation value from The composition is as follows.

[0018] According to this, the position and posture of the device, which are affected by the motion state of the device, Based on the evaluation value, It is possible to appropriately determine whether or not work assistance is required for sudden movements.

[0019] Also, Fourth The invention is In order to perform three-dimensional measurement processing to generate three-dimensional spatial information of a measurement target location based on a photographed image of the measurement target location, an imaging device for photographing the measurement target location includes: a device main body held by a user; an imaging unit provided in the device main body for photographing the measurement target location; a display unit for displaying work support information relating to a photographing operation in which the user causes the imaging unit to photograph the measurement target location while moving around the measurement target location; and a processor for controlling the imaging unit and the display unit, wherein the processor measures the elapsed time from the start of photographing, and determines whether work support is required to improve the status of the photographing operation based on an evaluation value that indicates the elapsed status of the photographing time, and when work support is required, displays information on the display unit that prompts the user to take action to improve the photographing operation as the work support information. The composition is as follows.

[0020] According to this, For users who are taking photos, By providing support for the elapsed time, it is possible to avoid the situation where the imaging time becomes too long, which increases the processing load, and the measurement errors accumulate, which reduces the accuracy of the 3D measurement. This allows the user to perform appropriate photographing work regardless of the user's level of proficiency. In this case, for example, work assistance may be provided when the shooting time approaches the upper limit time, specifically, when the shooting time exceeds a threshold calculated by multiplying the upper limit time by a predetermined ratio.

[0021] Also, Fifth The invention further comprises the processor: Information urging the user to take the action, such as information that the maximum shooting time is approaching is displayed on the display unit.

[0022] According to this, the user The remaining time until the upper limit, the percentage of the elapsed time and the remaining time in relation to the upper limit, etc. You can keep track of the progress of the shooting time. can.

[0023] Also, Sixth The invention further comprises the processor: To determine whether work support is required for each of multiple items, the evaluation value for each of the multiple items is acquired, and a table in which the evaluation conditions for the evaluation values ​​for each of the multiple items are registered is referenced. The composition is as follows.

[0024] In addition, a seventh aspect of the present invention is configured such that the processor refers to a table in which determination conditions for each of a plurality of items are registered, and determines whether or not work support is required for each of the plurality of items.

[0025] Also, Seventh The invention is a photography operation support method in which a processor performs a process to support a user's photography operation by having the user hold a photography device and photograph the measurement target location while moving within the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photographed images of the measurement target location, and the method comprises the steps of: An evaluation value representing the state of the user's operation of moving the photographing device is obtained, and based on the evaluation value, The system determines whether work assistance is necessary to improve the state of the photography work, and if work assistance is necessary, the system displays work assistance information on the display unit, encouraging the user to take action to improve the photography work.

[0026] Furthermore, the eighth invention is a photography support method in which a processor performs processing to support a user's photography work by having the user hold a photography device and photograph the measurement target location while moving within the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photographed images of the measurement target location, and the method is configured to determine whether or not work support is needed to improve the status of the user's photography work based on the status of the user's photography work, and if work support is needed, to display information on a display unit as work support information encouraging the user to take action to improve the photography work.

[0027] Also, 8th The invention provides a photography operation support program that causes a processor to execute a process for supporting a user's photography operation in which the user holds a photography device and photographs the measurement target location while moving within the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photographed images of the measurement target location, the program comprising: An evaluation value representing the state of the user's operation of moving the photographing device is obtained, and based on the evaluation value, The system determines whether work assistance is necessary to improve the state of the photography work, and if work assistance is necessary, the system displays work assistance information on the display unit, encouraging the user to take action to improve the photography work.

[0028] Furthermore, a ninth aspect of the present invention is a photography work support program that causes a processor to execute a process to support a user's photography work by having the photography device photograph the measurement target location while holding the photography device and moving within the measurement target location, in order to perform 3D measurement processing to generate 3D spatial information of the measurement target location based on photographed images of the measurement target location.The program is configured to determine whether or not work support is needed to improve the status of the user's photography work based on the status of the user's photography work, and if work support is needed, to display information on a display unit as work support information encouraging the user to take action to improve the photography work.

[0029] As with the first invention, this allows a user who is currently taking photographs to be clearly guided as to areas in their photography work that need to be improved immediately, allowing the user to carry out photography work in an appropriate manner regardless of their level of proficiency.

[0030] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0031] FIG. 1 is an explanatory diagram showing a situation in which a user performs a photographing operation using a photographing device 1 according to this embodiment.

[0032] The photographing device 1 includes a device main body 11 and a sensor unit 12. The sensor unit 12 includes a visible light camera 21 (photographing unit). The visible light camera 21 is a monocular camera that detects visible light to photograph a subject, and outputs a photographed image, for example, an RGB color image. The photographing device 1 can be configured as a tablet terminal or a notebook PC.

[0033] A user (worker) walks through a location to be measured while holding the device body 11 of the photographing device 1. At this time, the location to be measured is photographed by the visible camera 21 of the photographing device 1, and photographed images of each point in the location to be measured are sequentially acquired.

[0034] In this embodiment, the photographing device 1 is a three-dimensional measuring device. That is, the photographing device 1 performs three-dimensional measurement processing based on photographed images of each point sequentially acquired by the visible camera 21, and generates three-dimensional spatial information about the measurement target location. In the three-dimensional measurement processing, point cloud data (environmental map) is generated as three-dimensional spatial information about the measurement target location using the SLAM method.

[0035] 1 shows an example in which a user holding the device main body 11 walks through the measurement target location to photograph the measurement target location, but the measurement target location may also be photographed using a self-propelled robot or the like equipped with the photographing device 1. In this embodiment, work assistance is provided to the user, and when the user himself holds the device main body 11 and performs the photographing work, work assistance is provided to the user. On the other hand, when a self-propelled robot photographs, work assistance is provided to the user operating the robot.

[0036] In this embodiment, the photographing device 1 equipped with a sensor unit 12 such as a visible camera 21 performs the 3D measurement processing, but the sensor unit 12 may be configured as a photographing device 1 independent of a PC (information processing device) that performs the 3D measurement processing.

[0037] The user also photographs the measurement target location while walking through the measurement target location with the handheld camera 1. Therefore, the photographing operation includes actions such as walking (moving) while holding the camera 1 and pointing the camera 1 in an appropriate direction.

[0038] Next, an outline of the processing performed by the processor 16 will be described. FIG.

[0039] In this embodiment, messages (work support information) are displayed on the screen of the photographing device 1 as work support. The messages include warning messages, attention-calling messages, and notification messages. Warning messages encourage the user to take action to improve the state of photographing work that needs to be improved immediately. Attention-calling messages encourage the user to pay attention to a specific matter. Notification messages encourage the user to take action to improve the state of photographing work that does not need to be improved immediately but has room for improvement, for example, action to make the photographing work more efficient.

[0040] Furthermore, in this embodiment, the support condition table is referenced, and the necessity and content of work support for each item is determined based on the evaluation value for each item (blur, point cloud density, feature points, sudden movements, elapsed time, etc.) (support condition determination process). Thresholds are registered as determination conditions for each item in the support condition table, and the evaluation value of each item is compared with the threshold value for each item to determine the necessity and content of work support for each item.

[0041] As shown in Figure 2(A), the assistance condition table sets, for each item (blur, point cloud density, feature points, sudden movements, etc.) of warning messages, the judgment conditions for determining whether or not work assistance is required through a warning message, and the content of the message to be displayed on the screen.

[0042] Here, if the user moves the image capturing device 1 too fast while capturing an image, blurring occurs in the image captured by the visible camera 21, reducing the accuracy of the point cloud data. Therefore, in this embodiment, an evaluation value (blur evaluation value) that indicates the occurrence of blur in the captured image is calculated, and a state in which the user is moving the image capturing device 1 too fast is detected based on the blur evaluation value. In this case, a warning message urging the user to slow down the speed at which they move the image capturing device 1 is displayed on the screen as work assistance.

[0043] Specifically, the blur evaluation value is defined so that the smaller the value, the stronger the blur. Furthermore, a threshold value B1 is set for the blur evaluation value, and if the blur evaluation value is less than the threshold value B1 (blur evaluation value<threshold value B1), it is determined that the user is moving the image capturing device 1 too fast. In this case, a warning message urging the user to slow down the speed at which they move the image capturing device 1, such as a warning message saying "Blur occurring (please move slowly)," is displayed on the screen.

[0044] Furthermore, if the user moves the image capture device 1 too fast while capturing an image, the point cloud will not be generated with sufficient density, resulting in a decrease in the accuracy of the point cloud data. Therefore, in this embodiment, an evaluation value (point cloud density evaluation value) that indicates the point cloud generation status is calculated, and a state in which the user is moving the image capture device 1 too fast is detected based on the point cloud density evaluation value. In this case, a warning message is displayed on the screen to urge the user to slow down the speed at which they move the image capture device 1, as work assistance.

[0045] Specifically, the point cloud density evaluation value is defined so that the smaller the value, the lower the point cloud density. Furthermore, a threshold D1 is set for the point cloud density evaluation value. If the point cloud density evaluation value is less than the threshold D1 (point cloud density evaluation value<threshold D1), it is determined that the user is moving the image capture device 1 too fast. In this case, a warning message such as "Point cloud density decreasing (please move slowly)" is displayed on the screen to urge the user to slow down the speed at which the image capture device 1 is moved. It should be noted that the point cloud density may increase by repeatedly capturing images of the same location. Therefore, the point cloud density evaluation value may be calculated from only newly generated point clouds, or may be calculated from not only newly generated point clouds but also previously generated point clouds.

[0046] Furthermore, when an image is captured of an area that includes many areas without distinctive patterns or objects, such as walls, floors, or ceilings, feature point extraction fails, resulting in fewer extracted feature points and a decrease in the accuracy of the point cloud data. Therefore, in this embodiment, an evaluation value (number of feature points) that indicates the feature point extraction status is calculated, and based on the number of feature points, a state in which the user is pointing the camera device 1 in an inappropriate direction is detected. In this case, a warning message is displayed on the screen to prompt the user to point the camera device 1 in an appropriate direction, as work assistance.

[0047] Specifically, a threshold Q1 is set for the number of feature points, and when the number of feature points is less than the threshold Q1 (number of feature points<threshold Q1), it is determined that the direction in which the user points the image capturing device 1 is inappropriate. In this case, a warning message such as "fewer feature points (please set an angle of view that includes distinctive patterns or objects)" is displayed on the screen to prompt the user to point the image capturing device 1 in an appropriate direction. Alternatively, a warning message such as "fewer feature points (please set an angle of view that includes something other than walls and floors)" may be displayed on the screen.

[0048] Furthermore, within the range in which the IMU 23 (see Figure 3) operates normally, position and orientation tracking is less likely to fail due to correction of position and orientation information based on detection data from the IMU 23. Therefore, even when an area including many areas with few features, such as walls, floors, and ceilings, is captured, position and orientation tracking may be able to continue with little error. On the other hand, if the user moves the image capture device 1 too fast during capture, causing the image capture device 1 to move outside the range in which the IMU 23 operates normally, position and orientation tracking is more likely to fail.

[0049] Therefore, when the number of feature points is less than the threshold Q1, it may be determined that the situation is such that tracking of the position and orientation is likely to fail if the user moves the image capture device 1 too fast, and a warning message urging the user to slow down the speed at which the image capture device 1 is moved, such as a warning message stating "fewer feature points (please move slowly)," may be displayed on the screen. This allows the user to slow down the speed at which the image capture device 1 is moved, and moves the image capture device 1 within a range in which the IMU 23 operates normally, thereby stabilizing point cloud generation.

[0050] Furthermore, if the user moves the camera device 1 too quickly while taking a picture, the accuracy of the point cloud data will decrease. However, in addition to blur, point cloud density, and feature points as described above, it is possible to detect whether the user is moving the camera device 1 too quickly based on the acceleration and angular velocity that occur in the camera device 1 itself as the user moves the camera device 1.

[0051] Therefore, in this embodiment, a sudden acceleration / deceleration state in which the acceleration generated in the photographing device 1 falls outside a predetermined range, or a sudden rotation state in which the angular velocity generated in the photographing device 1 falls outside a predetermined range, is detected as a sudden movement by the user, and when the user makes a sudden movement, a warning message is displayed on the screen as work assistance, urging the user to slow down the speed at which the photographing device 1 is moved.

[0052] Specifically, the sudden movement evaluation value (an index based on data detected by the IMU 23) is defined so that the larger the value, the suddener the movement. Furthermore, a threshold value S is set for the sudden movement evaluation value, and if the sudden movement evaluation value is greater than the threshold value S (sudden movement evaluation value > threshold value S), it is determined that the speed at which the user moves the image capture device 1 is too fast. In this case, a warning message urging the user to slow down the speed at which the image capture device 1 is moved is displayed on the screen, for example, a warning message stating "Sudden acceleration / deceleration / sudden rotation has occurred (please move at a slower speed)."

[0053] Furthermore, as shown in Figure 2(B), the support condition table sets, for each item (such as elapsed time) regarding warning messages, the judgment conditions for determining whether or not work support is required through a warning message, and the content of the message to be displayed on the screen.

[0054] As the shooting time becomes longer, the processing load increases, and measurement errors accumulate (cumulate), reducing the accuracy of the point cloud data. Therefore, in this embodiment, an upper limit time is set for shooting, and when the elapsed time from the start of shooting approaches the upper limit time, a warning message is displayed on the screen to alert the user as work support.

[0055] Specifically, a threshold value T is set for the elapsed time, and when the elapsed time is greater than the threshold value T (elapsed time>threshold value T), a warning message such as "The maximum shooting time is approaching" is displayed on the screen. Note that the threshold value T may be calculated by multiplying the maximum time by a predetermined ratio (e.g., 90%).

[0056] Furthermore, as shown in Figure 2(C), the support condition table sets, for each item (blur, point cloud density, feature points, etc.) of notification messages, the judgment conditions for determining whether or not work support is required through a notification message, and the content of the message to be displayed on the screen.

[0057] If the user moves the image capture device 1 too slowly while capturing an image, the efficiency of the capture work will decrease. Therefore, in this embodiment, a state in which the user is moving the image capture device 1 too slowly or a state in which the movement speed does not need to be slowed down is detected based on the blur evaluation value, the point cloud density evaluation value, and the number of feature points. If the user is moving the image capture device 1 too slowly or if the movement speed does not need to be slowed down, a notification message urging the user to move the image capture device 1 faster is displayed on the screen as work assistance.

[0058] Specifically, a threshold value B2 is set for the blur evaluation value, and if the blur evaluation value is greater than the threshold value B2 (blur evaluation value > threshold value B2), it is determined that the user is moving the image capturing device 1 too slowly. A threshold value D2 is set for the point cloud density evaluation value, and if the point cloud density evaluation value is greater than the threshold value D2 (point cloud density evaluation value > threshold value D2), it is determined that the user is moving the image capturing device 1 too slowly. A threshold value Q2 is set for the number of feature points, and if the number of feature points is greater than the threshold value Q2 (number of feature points > threshold value Q2), it is determined that the user does not need to slow down the speed at which they move the image capturing device 1. When it is determined that the user is moving the image capturing device 1 too slowly or that they do not need to slow down the speed at which they move the image capturing device 1, a notification message urging the user to speed up the speed at which they move the image capturing device 1, such as a message saying "Good shooting (movement can be sped up)," is displayed on the screen.

[0059] In addition, in this embodiment, the support condition table is referenced, and based on the priority (priority) of each item, items for which support is to be provided are selected from among the items determined to require support, and the content of the support to be provided, i.e., the message to be displayed as an image (warning message, attention message, notification message), is determined (support content determination process).

[0060] As shown in Figures 2(A), (B), and (C), the support condition table sets priorities for displaying messages. If it is determined that work support is required for multiple items, work support is provided only for the items with the highest priority. In this case, only one item with the highest priority may be selected, or a predetermined number of items may be selected starting from the highest priority. Priorities are set based on the level of urgency, etc.

[0061] If it is determined that work support is required for multiple items without setting priorities, work support (display of a message) may be provided for all of those items. Also, if it is determined that work support is required for multiple items, work support may be provided for all of those items, but messages may be displayed on the screen in order from top to bottom based on priority.

[0062] In addition, in the assistance condition table, the display position of the message on the screen is set for each item of warning messages, attention-call messages, and notification messages.

[0063] Warning messages are displayed in the center of the screen (see FIG. 13). In this example, warning messages regarding blur, point cloud density, feature points, and sudden movements are displayed in the center of the screen. Meanwhile, notification messages and attention messages are displayed at the bottom of the screen (see FIG. 14). In this example, notification messages regarding blur, point cloud density, and feature points, and attention messages regarding elapsed time are displayed at the bottom of the screen.

[0064] The thresholds and priorities registered in the assistance condition table may be fixed values, but in this embodiment, the user can specify them on the setting mode screen (see FIG. 15).

[0065] Next, we will explain the general configuration of the photographing device 1. Fig. 3 is a block diagram showing the general configuration of the photographing device 1. Fig. 4 is a block diagram showing an overview of the processing performed by the processor 16. Fig. 5 is an explanatory diagram showing overlapping regions used in the point cloud density evaluation processing and the feature point evaluation processing.

[0066] As shown in FIG. 3, the photographing device 1 includes, in addition to the sensor unit 12, a display 13 (display unit), an input device 14, a memory 15 (storage unit), and a processor 16 (CPU).

[0067] In addition to the visible camera 21, the sensor unit 12 also includes a depth camera 22 and an IMU 23 (Inertial Measurement Unit).

[0068] The depth camera 22 is a stereo camera that detects infrared light to capture an image of a subject, and outputs depth information (distance image) as the detection result. The distance to the subject can be measured based on the detection result of the depth camera 22. Note that the depth camera 22 may be a sensor other than a stereo camera that can acquire depth information using other methods, such as LiDAR.

[0069] The IMU 23 (detector) detects the motion state of the device itself, specifically, three-dimensional angular velocity and acceleration. Based on the detection results of the IMU 23, the amount of movement and rotation of the image capturing device 1 can be measured.

[0070] The visible camera 21, the depth camera 22, and the IMU 23 may not be integrated into a sensor unit. Alternatively, the depth camera 22 and the IMU 23 may be omitted, and only the visible camera 21 may be provided. Alternatively, the visible camera 21 may be provided in addition to either the depth camera 22 or the IMU 23.

[0071] The display 13 presents various information related to the photographing operation to the user, and displays a photographing mode screen 101 (see FIG. 12) and a setting mode screen 201 (see FIG. 15).

[0072] The input device 14 is used by the user to perform input operations. The input device 14 may be a keyboard, a mouse, a touchpad, a touch panel, etc. If the photographing device 1 is configured as a tablet terminal, a touch panel display is provided in which the touch panel as the input device 14 and the display panel as the display 13 are integrated.

[0073] The memory 15 stores programs and the like to be executed by the processor 16. The memory 15 also stores images captured by the visible camera 21, distance information from the depth camera 22, and detection data from the IMU 23. The memory 15 also stores measurement results (point cloud data) generated by the processor 16.

[0074] The processor 16 performs various processes by executing programs stored in the memory 15. In this embodiment, the processor 16 performs a three-dimensional measurement process P1, a task support process P2, a display information generation process P3, a display process P4, and the like.

[0075] The three-dimensional measurement process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0076] The work support process P2 includes a blur evaluation process P21, a point cloud density evaluation process P22, a feature point evaluation process P23, a sudden movement evaluation process P24, a time passage evaluation process P25, a support condition determination process P26, and a support content determination process P27.

[0077] In this embodiment, the photographing device 1 performs the 3D measurement process P1, the work support process P2, and the display information generation process P3, but all or part of these processes may be performed by a server device (not shown) that can communicate with the photographing device 1.

[0078] As shown in FIG. 4, in the feature extraction process P11, the processor 16 extracts feature information (feature points, etc.) from the image (frame) captured by the visible camera 21.

[0079] In tracking process P12, processor 16 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition related to the position and orientation of the image capture device 1, and updates the position and orientation trajectory data based on the amount of transition. Note that the position and orientation trajectory data is obtained by tracking the position and orientation of the image capture device 1, and includes the tracking results of the position and orientation at the time of capturing each captured image (frame), i.e., information regarding the position and orientation of the image capture device 1 at each time of capturing.

[0080] In the position and orientation correction process P13, the processor 16 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, the position and orientation trajectory data is corrected so as to compensate for the measurement results of locations with few features, such as walls and ceilings.

[0081] In the point cloud generation process P14, the processor 16 generates point cloud data based on the distance information of the depth camera 22 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13.

[0082] In the blur evaluation process P21, the processor 16 performs image analysis on the captured image to obtain an evaluation value (blur evaluation value) that represents the occurrence of blur in the captured image. The blur evaluation value evaluates the occurrence of blur in the captured image, and the smaller the value, the stronger the blur. Note that the image analysis performed in the blur evaluation process P21 can employ various known techniques, and for example, the occurrence of blur is evaluated based on edges contained in the captured image.

[0083] In another pattern of the blur evaluation process P21, the processor 16 acquires an evaluation value (blur evaluation value) that indicates the occurrence of blur in the captured image based on the exposure time of the visible camera 21 and the detection data (acceleration, angular velocity) of the IMU 23. Here, since a long exposure time means that there is a high possibility that blur is occurring strongly, and a large acceleration or angular velocity occurring in the image capture device 1 means that there is a high possibility that blur is occurring strongly, the occurrence of blur can be estimated based on the exposure time of the visible camera 21 and the detection data (acceleration, angular velocity) of the IMU 23.

[0084] The blur evaluation process P21 may be performed based on only either the exposure time of the visible camera 21 or the detection data of the IMU 23. The exposure time of the visible camera 21 itself is not particularly related to the user's photographing operation, but the effect of the motion state of the image capture device 1 on the occurrence of blur varies depending on the length of the exposure time. For example, if the exposure time is long, significant blur will occur unless the acceleration and angular velocity of the image capture device 1 are sufficiently small, and conversely, if the exposure time is short, significant blur will not occur even if the acceleration and angular velocity of the image capture device 1 are somewhat large.

[0085] In the point cloud density evaluation process P22, the processor 16 counts the number of point clouds within the target area based on the point cloud data generated in the point cloud generation process P14, and obtains an evaluation value (point cloud density evaluation value) that indicates the state of point cloud generation. The point cloud density evaluation value indicates whether real-time point cloud generation is being performed appropriately. The smaller the point cloud density evaluation value, the lower the point cloud density.

[0086] The point cloud density evaluation value may be calculated from the number of point clouds generated in a predetermined unit time (for example, one second) (total number of points generated in the unit time) and the size of the shooting range of the visible light camera 21, as shown in the following formula. In this case, if the number of point clouds generated in the entire shooting range of the visible light camera 21 is small, the point cloud density evaluation value will be small. Point cloud density evaluation value = (number of point clouds generated per unit time) / (shooting range)

[0087] The point cloud density evaluation value may also be the number of generated points (the total number of points included in the overlapping area) in the overlapping area (see FIG. 5 ) between the shooting range at the current shooting time t and the shooting range at the previous shooting time t−1. In this case, too, the smaller the point cloud density evaluation value, the lower the point cloud density. Furthermore, not only is there a small number of point clouds generated in the shooting range of the visible camera 21, but the smaller the overlapping area between the shooting range at the current shooting time t and the shooting range at the previous shooting time t−1, the smaller the point cloud density evaluation value. It is also possible that the point cloud density may be improved by repeatedly capturing images of the same location. Therefore, the point cloud density evaluation value may be calculated from only newly generated point clouds, or may be calculated based on not only newly generated point clouds but also previously generated point clouds. In this case, the number of generated point clouds is the total number of all point clouds included in the overlapping area, not just the point cloud generated at the current shooting time t.

[0088] In the feature point evaluation process P23, the processor 16 counts the feature points within the target range based on the extraction result of the feature extraction process P11, and obtains an evaluation value (number of feature points) that indicates the extraction status of the feature points.

[0089] The number of feature points may be the number of feature points extracted in one captured image (one frame), or may be the number of feature points extracted in the overlapping area (see FIG. 5) between the capture range at the current capture time t and the capture range at the previous capture time t-1.

[0090] In the sudden movement evaluation process P24, the processor 16 acquires an evaluation value (sudden movement evaluation value) that indicates the degree of suddenness of the movement of the user moving the image capturing device 1, based on the detection data (acceleration, angular velocity) of the IMU 23. The degree of suddenness of the movement indicates the degree of suddenness of the change in the position and orientation of the image capturing device 1, and specifically indicates the degree of acceleration (including deceleration) that has occurred in the image capturing device 1, or the degree of angular velocity that has occurred in the image capturing device 1.

[0091] In another pattern of the sudden movement evaluation process P24, the processor 16 acquires a sudden movement evaluation value based on the amount of transition related to the position and orientation of the image capture device 1 acquired in the tracking process P12. Here, the amount of transition represents the change in the position and orientation of the image capture device 1, and the magnitude of the amount of transition can be used to evaluate the suddenness of the movement.

[0092] In the time lapse evaluation process P25, the processor 16 measures the time elapsed since the start of shooting and acquires an evaluation value (elapsed time) that indicates the elapsed status of the shooting time. Specifically, when shooting starts, a timer starts counting, and the time measured by the timer at the time of measurement is acquired as the elapsed time.

[0093] In the assistance condition determination process P26, the processor 16 refers to the assistance condition table (see FIG. 2) and determines the necessity and content of work assistance for each item based on the evaluation value for each item (blur, point cloud density, feature points, sudden movements, and elapsed time). The assistance condition table registers thresholds as determination conditions for each item, and the evaluation value for each item is compared with the threshold for each item.

[0094] In the support content determination process P27, the processor 16 refers to the support condition table (see FIG. 2) and selects an item for which work support will be provided from among the items determined to require work support based on the priority of each item, and determines the content of the work support, i.e., the message to be displayed as an image (warning message, attention message, notification message).

[0095] In the display information generation process P3, the processor 16 generates display information for a screen to be displayed on the display 13 based on the image captured by the visible camera 21, the point cloud data generated in the point cloud generation process P14, and the elapsed time acquired in the time passage evaluation process P25. Also, in the display information generation process P3, when work assistance is required, the processor 16 generates display information for a screen including the content of work assistance (warning message, attention message, notification message) determined in the assistance content determination process P27.

[0096] In the display process P4, the processor 16 displays a screen (see FIGS. 12 to 15) on the display 13 based on the display information generated in the display information generation process P3.

[0097] In this embodiment, blur evaluation processing P21, point cloud density evaluation processing P22, feature point evaluation processing P23, sudden movement evaluation processing P24, and time passage evaluation processing P25 are performed as processes to obtain evaluation values ​​for determining whether work assistance is required, but only some of these processes may be performed.

[0098] Next, a description will be given of the procedure of the process performed by the processor 16. FIG.

[0099] First, the processor 16 acquires the image captured by the visible camera 21, the distance information from the depth camera 22, and the detection data from the IMU 23 from the sensor unit 12 (ST101).

[0100] Next, the processor 16 extracts feature points from the image captured by the visible camera 21 (feature extraction process) (ST102).

[0101] Next, the processor 16 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition regarding the position and orientation of the image capturing device 1, and updates the position and orientation trajectory data based on the amount of transition (tracking process) (ST103).

[0102] Next, the processor 16 corrects the position and orientation trajectory data acquired in the tracking process based on the detection data of the IMU 23 (position and orientation correction process) (ST104).

[0103] Next, the processor 16 generates point cloud data based on the distance information of the depth camera 22 and the position and orientation trajectory data acquired in the tracking process and the position and orientation correction process (point cloud generation process) (ST105).

[0104] Next, the processor 16 acquires an evaluation value (blur evaluation value) that indicates the occurrence of blur in the captured image (blur evaluation process) (ST106). At this time, the blur evaluation value is obtained by image analysis of the captured image. The blur evaluation value is also obtained based on the exposure time of the visible light camera 21 and the detection data of the IMU 23.

[0105] Next, processor 16 counts the point clouds within the target range based on the point cloud data generated in the point cloud generation process, and obtains an evaluation value (point cloud density evaluation value) representing the point cloud generation status (point cloud density evaluation process) (ST107).

[0106] Next, processor 16 counts the feature points within the target range based on the extraction results of the feature extraction process, and obtains an evaluation value (number of feature points) that indicates the extraction status of the feature points (feature point evaluation process) (ST108).

[0107] Next, the processor 16 acquires an evaluation value (sudden movement evaluation value) representing the abruptness of the user's movement of the image capturing device 1 (sudden movement evaluation process) (ST109). At this time, the sudden movement evaluation value is calculated based on the detection data of the IMU 23. The sudden movement evaluation value is also calculated based on the amount of transition related to the position and orientation of the image capturing device 1 acquired in the tracking process.

[0108] Next, the processor 16 acquires an evaluation value (elapsed time) that indicates the elapsed state of the photographing time (time elapsed evaluation process) (ST110). At this time, the elapsed time from the start of photographing is measured using a timer.

[0109] The processing in ST106 to ST110 is not limited to the order shown in the figure.

[0110] Next, processor 16 refers to the support condition table (see FIG. 2) and determines the necessity and content of work support for each item based on the evaluation value for each item (blur, point cloud density, feature points, sudden movement, and elapsed time) (support condition determination process) (ST111). The support condition determination process for each item is shown in FIGS. 7 to 11.

[0111] Next, processor 16 determines whether or not it has been determined that work support is required for one or more items (ST112).

[0112] Here, if it is determined that work support is required for one or more items (Yes in ST112), processor 16 then refers to the support condition table (see FIG. 2) and selects items for which work support will be provided from among the items determined to require work support based on the priority of each item, and determines the content of the work support, i.e., the message to be displayed as an image (warning message, attention message, notification message) (support content determination process) (ST113).

[0113] Here, when it is determined that work support is required for multiple items, work support (display of a message) may be provided for all of the items. Furthermore, of the multiple items determined to require work support, only one item with the highest priority may be selected and work support may be provided. Furthermore, of the multiple items determined to require work support, a predetermined number of items may be selected in descending order of priority and work support may be provided.

[0114] Next, the processor 16 generates display information for the screen based on the determined content of the work support (display information generation process), and displays the screen on the display 13 based on the display information (display process) (ST114).

[0115] Next, a description will be given of the assistance condition determination process for blur that is performed by the processor 16. Fig. 7 is a flowchart showing the procedure of the assistance condition determination process for blur.

[0116] Processor 16 determines whether or not work support related to blur is necessary and the details thereof by referring to the support condition table (see FIG. 2). The support condition table stores thresholds B1 and B2, which are used as the determination conditions for warning messages and notification messages related to blur, respectively.

[0117] Specifically, first, processor 16 determines whether the blur evaluation value is less than threshold value B1 (ST201). If the blur evaluation value is less than threshold value B1 (Yes in ST201), processor 16 determines that work assistance regarding blur is required and that the content of the assistance is a warning message (ST202).

[0118] On the other hand, if the blur evaluation value is equal to or greater than threshold B1 (No in ST201), processor 16 then determines whether the blur evaluation value is greater than threshold B2 (ST203). If the blur evaluation value is greater than threshold B2 (Yes in ST203), processor 16 determines that work assistance regarding blur is required and that the content is a notification message (ST204).

[0119] On the other hand, if the blur evaluation value is equal to or less than the threshold value B2 (No in ST203), the processor 16 determines that work support regarding blur is not required (ST205).

[0120] Next, a description will be given of the support condition determination process relating to point cloud density, which is performed by the processor 16. Fig. 8 is a flowchart showing the procedure of the support condition determination process relating to point cloud density.

[0121] The processor 16 determines whether or not work support related to point cloud density is necessary and the details of the support, by referring to the support condition table (see FIG. 2). The support condition table stores thresholds D1 and D2, which are used as the determination conditions for warning messages and notification messages related to point cloud density, respectively.

[0122] Specifically, first, processor 16 determines whether the point cloud density evaluation value is less than threshold D1 (ST301). If the point cloud density evaluation value is less than threshold D1 (Yes in ST301), processor 16 determines that work support related to point cloud density is required and that the content of the support is a warning message (ST302).

[0123] On the other hand, if the point cloud density evaluation value is equal to or greater than threshold D1 (No in ST301), processor 16 then determines whether or not the point cloud density evaluation value is greater than threshold D2 (ST303). Here, if the point cloud density evaluation value is greater than threshold D2 (Yes in ST303), processor 16 determines that work support related to point cloud density is required and that the content is a notification message (ST304).

[0124] On the other hand, if the point cloud density evaluation value is equal to or less than the threshold D2 (No in ST303), the processor 16 determines that task support related to the point cloud density is not required (ST305).

[0125] Next, a description will be given of the assistance condition determination process for feature points, which is performed by the processor 16. Fig. 9 is a flowchart showing the procedure of the assistance condition determination process for feature points.

[0126] Processor 16 determines the necessity and content of task support related to feature points by referring to the support condition table (see FIG. 2). Thresholds Q1 and Q2, which are the determination conditions for each of the warning message and notification message related to feature points, are registered in the support condition table.

[0127] Specifically, first, processor 16 determines whether the number of feature points is less than threshold Q1 (ST401). If the number of feature points is less than threshold Q1 (Yes in ST401), processor 16 determines that task support related to feature points is required and that the content of the support is a warning message (ST402).

[0128] On the other hand, if the number of feature points is equal to or greater than threshold Q1 (No in ST401), processor 16 then determines whether the number of feature points is greater than threshold Q2 (ST403). If the number of feature points is greater than threshold Q2 (Yes in ST403), processor 16 determines that task support related to feature points is required and that the content is a notification message (ST404).

[0129] On the other hand, if the number of feature points is equal to or less than the threshold Q2 (No in ST403), the processor 16 determines that task support related to feature points is not required (ST405).

[0130] Next, a description will be given of the assistance condition determination process for a sudden movement, which is performed by the processor 16. Fig. 10 is a flowchart showing the procedure of the assistance condition determination process for a sudden movement.

[0131] The processor 16 determines whether or not work support is necessary for sudden movements and the content of the support, with reference to the support condition table (see FIG. 2). The support condition table stores a threshold value S that is a determination condition for issuing a warning message for sudden movements.

[0132] Specifically, first, processor 16 determines whether the sudden movement evaluation value is greater than threshold value S (ST501). If the sudden movement evaluation value is greater than threshold value S (Yes in ST501), processor 16 determines that work support for the sudden movement is required and that the content of the support is a warning message (ST502).

[0133] On the other hand, if the sudden movement evaluation value is equal to or less than the threshold value S (No in ST501), the processor 16 determines that work support related to sudden movements is not required (ST503).

[0134] In this embodiment, if the user moves the photographing device 1 too quickly while taking a photograph, a warning message is displayed as work assistance to encourage the user to slow down the speed at which the photographing device 1 is moved.However, if the user's movements are too slow based on the sudden movement evaluation value, a notification message may be displayed as work assistance to improve work efficiency to encourage the user to move the photographing device 1 more quickly.

[0135] In this case, threshold values ​​S1 and S2 are set as judgment conditions for warning messages and notification messages, respectively, and if the sudden movement evaluation value is greater than threshold value S1, the content of the work support is judged to be a warning message, and if the sudden movement evaluation value is less than threshold value S2, the content of the work support is judged to be a notification message.

[0136] Next, a description will be given of the assistance condition determination process relating to elapsed time, which is performed by the processor 16. Fig. 11 is a flowchart showing the procedure of the assistance condition determination process relating to elapsed time.

[0137] The processor 16 determines whether or not work support is necessary and the content of the support with respect to the elapsed time by referring to the support condition table (see FIG. 2). The support condition table stores a threshold value T that is a determination condition for issuing a warning message regarding the elapsed time.

[0138] Specifically, first, processor 16 determines whether the elapsed time is greater than threshold value T (ST601). If the elapsed time is greater than threshold value T (Yes in ST601), processor 16 determines that work support related to the elapsed time is required and that the content of the support is a warning message (ST602).

[0139] On the other hand, if the elapsed time is equal to or less than the threshold value T (No in ST601), the processor 16 determines that task support related to the elapsed time is not required (ST603).

[0140] Next, a description will be given of the image capture mode screen 101 displayed on the display 13. Fig. 12 is an explanatory diagram showing the image capture mode screen 101.

[0141] A shooting mode screen 101 is provided with a main window 102 and a sub-window 103. The main window 102 and the sub-window 103 have different display magnifications, with the main window 102 displaying an enlarged image and the sub-window 103 displaying a reduced image.

[0142] Furthermore, the main window 102 and the sub-window 103 each display a captured image 121 and a map image 122. The captured image 121 is a current captured image, i.e., an image captured in real time and output from the visible camera 21. The map image 122 is an overhead point cloud image, i.e., an image (rendering) of each point of the point cloud data as seen from a viewpoint above the sky.

[0143] Furthermore, the shooting mode screen 101 is provided with a "Start shooting" button 105 and an "End shooting" button 106. When the user operates the "Start shooting" button 105, shooting by the visible camera 21 begins, and the captured image is stored in the memory 15. When the user operates the "End shooting" button 106, shooting by the visible camera 21 ends. Note that, if a depth camera 22 and an IMU 23 are present, distance information from the depth camera 22 and detection data from the IMU 23 are acquired in the same way as shooting by the visible camera 21.

[0144] Furthermore, the image capture mode screen 101 is provided with a "screen switch" button 107 and a check box 108 related to the display of the sub-window 103. When the user operates the "screen switch" button 107, the screen is switched between a state in which the captured image 121 is displayed enlarged in the main window 102 and the map image 122 is displayed reduced in the sub-window 103 (the state shown in FIG. 12), and a state in which the map image 122 is displayed enlarged in the main window 102 and the captured image 121 is displayed reduced in the sub-window 103. When the user operates the check box 108, the sub-window 103 can be switched between a display state and a non-display state.

[0145] Furthermore, the shooting mode screen 101 is provided with a time elapsed status display section 111 and a check box 112 related to displaying the time elapsed status display section 111. The time elapsed status display section 111 displays the elapsed time from the start of shooting. When the user operates the check box 112, the time elapsed status display section 111 can be switched between a displayed state and a hidden state.

[0146] In this example, the time elapsed since the start of shooting is displayed on the time elapsed status display unit 111, but instead of or together with the elapsed time, the remaining time until the upper limit time may be displayed. Furthermore, the ratio of the elapsed time and the remaining time to the upper limit time may be displayed as the time elapsed status. In this case, an image representing the time elapsed status, such as a level indicator whose display changes as time passes, may be displayed.

[0147] Furthermore, the shooting mode screen 101 is provided with a "Shooting" tab 131 and a "Settings" tab 132. The user can switch between the shooting mode and the setting mode by operating the tabs 131 and 132. The shooting mode screen 101 shown in FIG. 12 is when the user operates the "Shooting" tab 131. When the user operates the "Settings" tab 132, the screen transitions to the setting mode screen 201 (see FIG. 15).

[0148] In this embodiment, the photographing mode and the setting mode are switched by operating the tabs 131 and 132, but the photographing mode and the setting mode may be switched by another type of operation input unit such as a button.

[0149] Next, the display status of warning messages, attention-calling messages, and notification messages will be described. Fig. 13 is an explanatory diagram showing the display status of warning messages on the screen 101 in shooting mode. Fig. 14 is an explanatory diagram showing the display status of notification messages on the screen 101 in shooting mode.

[0150] 13, when it is determined that work assistance is necessary and the content of the work assistance is a warning message, a first message display window 141 including the warning message is superimposed in the center of the shooting mode screen 101. In the example shown in Fig. 13, the content of the work assistance is a warning message regarding the blur item, and the warning message "Blur occurring (please move slowly)" is displayed.

[0151] If the content of work assistance is a warning message regarding the point cloud density item, a first message display window 141 containing a warning message saying "Point cloud density decreasing (please move slowly)" is displayed in the center of the screen. If the content of work assistance is a warning message regarding the feature point item, a first message display window 141 containing a warning message saying "Feature points decreasing (please set an angle of view that includes distinctive patterns or objects)" is displayed in the center of the screen. If the content of work assistance is a warning message regarding the sudden movement item, a first message display window 141 containing a warning message saying "Sudden movement occurred (please move slowly)" is displayed in the center of the screen.

[0152] The first message display window 141 continues to be displayed unless the user's photographing work status improves, and disappears when the user's photographing work status improves.

[0153] 14, when it is determined that work assistance is necessary and the content of the work assistance is a notification message, a second message display window 142 including the notification message is superimposed on the bottom of the shooting mode screen 101. In the example shown in Fig. 14, the content of the work assistance is a notification message regarding any one of blur, point cloud density, feature points, and sudden movements, and the notification message "Good shooting (movement speed can be increased)" is displayed.

[0154] Furthermore, if the content of the work support is a warning message regarding the elapsed time item, a second message display window 142 containing a warning message saying "The upper limit of shooting time is approaching" is displayed at the bottom of the screen.

[0155] The second message display window 142 continues to be displayed unless the user's photographing work status improves, and disappears when the user's photographing work status improves. Note that the second message display window 142 may display information that is useful for the user's photographing work or information that calls attention to the user's work, in addition to notification messages that provide work support.

[0156] As described above, in this embodiment, warning messages for each of the items of blur, point cloud density, feature points, and sudden movements are displayed in the center of the screen 101 (see FIG. 13). This allows the user who is taking a photograph to be informed in an easy-to-understand manner of the points that need to be improved immediately in their photographing work. In particular, the warning messages provide written information on the points that need to be improved for each item along with the status and cause, so the user can immediately recognize the points that need to be improved in their photographing work.

[0157] Meanwhile, notification messages regarding the items of blur, point cloud density, and feature points are displayed at the bottom of the screen 101 (see FIG. 14). This allows the user, who is currently taking pictures, to be guided in an easy-to-understand manner about desirable shooting methods when the user's shooting work is good but there is room for improvement. In particular, the notification message, like the warning message, provides written guidance on points that need improvement along with the status and cause, so the user can immediately recognize the points in their shooting work that need improvement.

[0158] Furthermore, a message calling attention to the item of elapsed time is displayed at the bottom of the screen 101 (see FIG. 14). This makes it possible to call the attention of the user who is currently taking a photograph about the elapsed time, which is an item that should be taken into consideration when proceeding with the photographing work.

[0159] In this embodiment, warning messages, attention-calling messages, and notification messages are displayed on the screen 101 as work support, but the work support is not limited to messages displayed on the screen 101. For example, images (icons, marks, color changes, animations, etc.) that represent work support content similar to the warning messages, attention-calling messages, and notification messages may be displayed on the screen 101. Furthermore, work support content similar to the warning messages, attention-calling messages, and notification messages may be expressed by changing the lighting color or blinking of a lamp provided in the photographing device 1. Furthermore, work support content similar to the warning messages, attention-calling messages, and notification messages may be expressed by sounds or audio messages output from a speaker provided in the photographing device 1.

[0160] Next, a description will be given of the setting mode screen 201 (customization screen) displayed on the display 13. Fig. 15 is an explanatory diagram showing the setting mode screen 201.

[0161] The setting mode screen 201 displays a first setting item input section 211 for warning messages, a second setting item input section 212 for attention messages, and a third setting item input section 213 for notification messages.

[0162] The first setting item input section 211 is provided with a check box 231 related to warning messages. By operating the check box 231, the user can simultaneously switch between enabling and disabling the processing of all items related to warning messages. The first setting item input section 211 also has a check box 232 for each item (blur, point cloud density, feature points, sudden movement). The user can individually switch between enabling and disabling the processing of each item by operating the check box 232. The first setting item input section 211 also has a threshold input field 233 for each item. The user can input a number representing the threshold for each item into the input field 233.

[0163] The second setting item input section 212 is provided with a check box 241 related to the alert message. By operating the check box 241, the user can simultaneously switch between enabling and disabling the processing of all items related to the alert message. The second setting item input section 212 is also provided with a check box 242 for each item (elapsed time). By operating the check box 242, the user can individually switch between enabling and disabling the processing of each item. The second setting item input section 212 is also provided with an input field 243 for a threshold value for each item. The user can input a number representing the threshold value for each item into the input field 243. Note that, although only one item (elapsed time) can be set for the alert message in the example shown in FIG. 15, multiple items may be set.

[0164] The third setting item input section 213 is provided with a check box 251 related to the notification message. By operating the check box 251, the user can collectively switch between enabling and disabling the processing of all items related to the notification message. Furthermore, the third setting item input section 213 is provided with a check box 252 for each item (blur, point cloud density, feature points). By operating the check box 252, the user can individually switch between enabling and disabling the processing of each item. Furthermore, the third setting item input section 213 is provided with an input field 253 for a threshold value for each item. The user can input a number representing the threshold value for each item into the input field 253.

[0165] In this example, numbers representing threshold values ​​are input into input fields 233, 243, and 253 as operation input sections for judgment conditions, but the user may also adjust the level of the judgment criteria using an operation input section such as a slide bar or button.

[0166] Furthermore, the setting mode screen 201 is provided with a "Save Settings" button 261 and a "Load Settings" button 262. When the user operates the "Save Settings" button 261, the setting information is saved with the current input contents. When the user operates the "Load Settings" button 262, the previous setting information is loaded and displayed in the setting item input areas 211, 212, and 213.

[0167] As described above, in this embodiment, the user can specify the judgment conditions (thresholds) used to determine whether work assistance is required for each item. This allows the user to set an appropriate threshold value depending on the characteristics of the measurement target location. For example, if the measurement target location is a large space and there are few subjects within a distance at which a point cloud can be generated, it is difficult to generate a point cloud. This can result in the point cloud density evaluation value falling below the threshold, causing frequent warning messages. In this case, if the user finds the warning messages annoying, the user can avoid frequent warning messages by resetting the threshold to a smaller value.

[0168] Furthermore, whether processing for each item is enabled or disabled and the judgment conditions (threshold values) for each item differ depending on the measurement target location. Therefore, setting information may be registered in advance for each type of measurement target location, and setting information suitable for the measurement target location may be loaded when starting a photographing operation. This can reduce the labor required for setting up when starting a photographing operation. Furthermore, by saving setting information customized by the user in association with information about the measurement target location at the end of photographing, the next time the same user photographs the measurement target location, the setting information can be loaded and photographing can be started promptly.

[0169] Next, a description will be given of an operation for changing the priority of each item on the setting mode screen 201. Fig. 16 is an explanatory diagram showing the operation for changing the priority of each item.

[0170] In the first setting item input section 211, a plurality of items (blur, point cloud density, feature points, and sudden movements) are displayed in descending order of priority from the top. Similarly, in the third setting item input section 213, a plurality of items (blur, point cloud density, and feature points) are displayed in descending order of priority from the top. Note that, in the example shown in Fig. 15, only one item (elapsed time) is displayed in the second setting item input section 212, but a plurality of items may be displayed, in which case the plurality of items are displayed in descending order of priority from the top.

[0171] The user can change the priority of each item by performing a predetermined operation in the setting item input sections 211, 212, and 213 (operation input section). For example, if the user performs a drag-and-drop operation on the display portion of each item in the setting item input sections 211, 212, and 213 to rearrange the order of the items, the priority will be changed to the rearranged order.

[0172] Priority-based processing is performed when multiple items are enabled by operating check boxes 231, 241, and 251, and the content of work support, i.e., warning messages, attention messages, and notification messages to be displayed as images, is determined for the multiple enabled items based on the priority.

[0173] Furthermore, the setting item input sections 211, 212, and 213 may be provided with an input field for priority, allowing the user to input the priority numerically.

[0174] Although it is difficult to imagine a warning message and a notification message being displayed at the same time, it is possible that a warning message and a caution message will be displayed at the same time, or that a notification message and a caution message will be displayed at the same time. In this case, if the display positions of the messages displayed simultaneously are different (for example, in the center and at the bottom), they may be displayed simultaneously, but if the display positions are the same (for example, both at the bottom), it is advisable to give priority to one of the messages or to display the messages alternately.

[0175] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0176] The photographing device, photographing work support method, and photographing work support program according to the present invention have the effect of providing a user who is photographing work with clear guidance on areas that need to be improved immediately regarding the user's photographing work, allowing the user to carry out the photographing work in an appropriate manner regardless of the user's level of proficiency, and are useful as a photographing device that photographs each point in a measurement target location in order to perform 3D measurement processing that generates 3D spatial information of the measurement target location based on photographed images of each point in the measurement target location, as well as a photographing work support method and photographing work support program that support the user's photographing work using the photographing device. [Explanation of symbols]

[0177] 1: Imaging device 11: Device body 12: Sensor unit 13: Display 14: Input device 15: Memory 16: Processor 21: Visible camera 22: Depth camera 23:IMU 101: Shooting mode screen 111: Time lapse status display section 121: Photographed image 122: Map image 141: First message display window 142: Second message display window 201: Setting mode screen 211: First setting item input section 212: Second setting item input section 213: Third setting item input section P1: 3D measurement processing P2: Work support processing P3: Display information generation process P4: Display processing P11: Feature extraction processing P12: Tracking processing P13: Position and orientation correction processing P14: Point cloud generation processing P21: Blur evaluation processing P22: Point cloud density evaluation processing P23: Feature point evaluation processing P24: Sudden movement evaluation processing P25: Time-lapse evaluation process P26: Support condition determination process P27: Support content decision process

Claims

1. An imaging device that captures an image of a measurement target location in order to perform a three-dimensional measurement process that generates three-dimensional spatial information of the measurement target location based on a captured image of the measurement target location, a device body held by a user; an imaging unit provided in the device body for imaging a measurement target location; a display unit that displays work support information related to a photographing operation in which a user moves around a measurement target location and causes the photographing unit to photograph the measurement target location; an acquisition unit that acquires an evaluation value representing a state of an action of a user moving the device; a processor that controls the photographing unit, the display unit, and the acquisition unit, The processor: determining whether or not work support is required to improve the state of the photographing work based on the evaluation value acquired by the acquisition unit; When work assistance is required, the work assistance information is displayed on the display unit to prompt the user to take an action to improve the user's work.

2. Further, an inertial measurement unit is provided for detecting the acceleration and angular velocity of the device itself, The processor:

2. The imaging device according to claim 1, wherein the evaluation value is obtained based on a detection result of the inertial measurement unit.

3. The processor: The imaging device according to claim 1 , wherein the evaluation value is obtained from the position and orientation of the imaging device itself acquired in the three-dimensional measurement process.

4. A photographing device that photographs a measurement target location in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on a photographed image of the measurement target location, a device body held by a user; an imaging unit provided in the device body for imaging a measurement target location; a display unit that displays work support information related to a photographing operation in which a user moves around a measurement target location and causes the photographing unit to photograph the measurement target location; a processor that controls the imaging unit and the display unit, The processor: measuring the elapsed time from the start of the photographing, and determining whether or not work assistance is required to improve the state of the photographing work based on an evaluation value that indicates the elapsed state of the photographing time; When work assistance is required, the work assistance information is displayed on the display unit to prompt the user to take an action to improve the user's work.

5. The processor:

5. The photographing apparatus according to claim 4, wherein the information prompting the user to take action is displayed on the display unit as information indicating that the upper limit of the photographing time is approaching.

6. The processor: The photographing device according to claim 1 or claim 4, characterized in that in order to determine whether work assistance is necessary for each of a plurality of items, an evaluation value for each of the plurality of items is obtained and a table in which determination conditions for the evaluation values ​​for each of the plurality of items are registered is referenced.

7. A photography operation support method for performing a process by a processor to support a user's photography operation by causing the photography device to photograph a measurement target location while holding the photography device and moving within the measurement target location, in order to perform a three-dimensional measurement process to generate three-dimensional spatial information of the measurement target location based on a photographed image of the measurement target location, comprising: acquiring an evaluation value representing the state of the user's operation of moving the photographing device, and determining whether or not work assistance is required to improve the state of the photographing work based on the evaluation value; A photographing operation support method characterized in that, when work support is required, information for encouraging the user to take an action to improve the photographing operation is displayed on a display unit as work support information.

8. A photography operation support program that causes a processor to execute a process for supporting a user's photography operation of photographing a measurement target location with an imaging device while moving within the measurement target location, in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on a photographed image of the measurement target location, the program comprising: acquiring an evaluation value representing the state of the user's operation of moving the photographing device, and determining whether or not work assistance is required to improve the state of the photographing work based on the evaluation value; A photography operation support program characterized in that, when work support is required, information for encouraging a user to take an action to improve the photography operation is displayed on a display unit as work support information.

Images