Photographing system, photographing device, and photographing control method

Abstract

To allow a user to easily perform screen operation for instructions or settings on imaging using a touch panel display, while securely holding an imaging apparatus even during the imaging.SOLUTION: An imaging apparatus 1 includes: a sensor unit 11 which includes a camera for imaging a measurement target place; a display input panel unit 12 including a touch panel display 36 for displaying a screen for supporting imaging operation of a user and detecting screen operation by the user; and a support body 13 which supports the sensor unit and the display input panel unit. The support body includes a grip unit 35 to be gripped with one hand of the user. The display input panel unit is arranged near the grip unit. A processor of a control unit 2 for controlling the imaging apparatus 1 displays, on the touch panel display, at least an operation screen to be used by the user for instruction to start imaging. Especially, the processor displays, as the operation screen, an imaging standby screen, an imaging-in-process screen, and an imaging end confirmation screen, so as to be sequentially changed according to an instruction from the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging system, an imaging device, and an imaging control method that include an imaging device that images a measurement target location and a processor that controls the imaging device in order to perform 3D measurement processing that generates 3D spatial information of the measurement target location. [Background technology]

[0002] There is a known 3D measurement technology that generates a point cloud representing a measurement target in a 3D space based on captured images of the measurement target. In recent years, the SLAM (Simultaneous Localization And Mapping) method has attracted attention for this 3D measurement technology. The SLAM method performs self-localization to acquire the position information of the device based on captured images of the measurement target from various directions, held by a mobile object (e.g., a worker), and mapping to generate a point cloud representing the measurement target in a 3D space.

[0003] A known technology related to this SLAM method is to present to the user areas where photography is insufficient by using an image of an arrow that indicates the direction and speed of movement of the imaging device (sensor) held by the user, or an image that distinguishes the direction of measured areas from the direction of unmeasured areas (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6489566 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the conventional technology, a tablet PC (measurement device body) equipped with a distance sensor and a visible camera is used as an imaging device for capturing images of the measurement target location, and the user captures images of the measurement target location by walking through the measurement target location while holding the imaging device with both hands. In addition, the user can use a touch panel display provided on the imaging device to operate the screen for giving instructions and making settings related to the imaging.

[0006] However, there are cases where a user wants to operate the screen to give instructions or make settings related to shooting while shooting. In this case, operating the screen while shooting is cumbersome when the user holds the camera with both hands, as in the conventional technology. In other words, since the user needs to remove one hand from the camera to operate the screen, it becomes difficult to hold the camera stably, and it is difficult to operate the screen.

[0007] Therefore, the main object of the present invention is to provide a photography system, photography device, and photography control method that allows the user to hold the photography device stably while easily operating the screen on a touch panel display to give instructions and make settings related to photography, even during photography. [Means for solving the problem]

[0008] The imaging system of the present invention comprises: A photographing device that photographs the measurement target location, and a control device that is connected to the photographing device and executes 3D measurement processing of the measurement target location. The photographing system includes a sensor unit including a camera for photographing a measurement target location, and a camera for supporting a user's photographing work. Screen used in shooting mode and a touch panel display that displays the screen and detects screen operations by the user. First display and, and a support having a grip portion to be held by a user in one hand and supporting the sensor portion and the first display portion, wherein the control device comprises: a 3D measurement processing portion that performs the 3D measurement processing to generate 3D spatial information of a measurement target location based on imaging data acquired from the imaging device; a second display portion that displays a screen used in a non-imaging mode for performing operations related to management and processing of the imaging data; and a display control portion that controls display information of the screens to be displayed on the first display portion and the second display portion, wherein the display control portion causes the second display portion to display a pause screen and the first display portion to display an operation screen for the user to give at least an instruction to start imaging, in response to a user operation to switch from the non-imaging mode to the imaging mode. The composition is as follows. [Effects of the Invention]

[0011] According to the present invention, When a user operates to transition from non-photography mode to photography mode, the display control unit displays a pause screen on the second display unit of the control device and also displays an operation screen on the first display unit of the photography device, on which the user can give at least an instruction to start photography, so that the transition to photography mode in the photography system can be confirmed; You can easily use your other hand to operate the touch panel display to give instructions and make settings related to shooting. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram showing a situation of a photographing operation performed by a user using the photographing system according to the first embodiment; [Figure 2] An explanatory diagram showing standard shooting conditions and high-altitude shooting conditions [Figure 3] FIG. 1 is a perspective view showing a support for the imaging device; [Figure 4] FIG. 10 is a side view showing the expansion and contraction of the support of the imaging device; [Figure 5] FIG. 1 is a perspective view showing a sensor unit of an imaging device; [Figure 6] A block diagram showing the schematic configuration of an imaging device and a control device. [Figure 7] A block diagram showing an overview of the processing performed by the processor of the control device. [Figure 8] FIG. 10 is an explanatory diagram showing a list of photographed data displayed on the display of the control device. [Figure 9] FIG. 10 is an explanatory diagram showing a detailed image data screen displayed on the display of the control device. [Figure 10] An explanatory diagram showing a screen that pops up on the shooting data details screen [Figure 11] FIG. 10 is an explanatory diagram showing the transition status of the main part of the imaging data details screen. [Figure 12] FIG. 10 is an explanatory diagram showing a setting screen displayed on the display of the control device. [Figure 13] FIG. 10 is an explanatory diagram showing a pause screen displayed on the display of the control device. [Figure 14] FIG. 10 is an explanatory diagram showing a connection failure notification screen displayed on the display of the control device. [Figure 15] FIG. 10 is an explanatory diagram showing a pause screen displayed on a touch panel display of the imaging device. [Figure 16] FIG. 10 is an explanatory diagram showing a shooting standby screen displayed on a touch panel display of the imaging device. [Figure 17] FIG. 10 is an explanatory diagram showing a screen displayed during imaging on a touch panel display of the imaging device. [Figure 18] FIG. 10 is an explanatory diagram showing the transition of the available shooting time guide section on the shooting screen; [Figure 19]An explanatory diagram showing the shooting screen with a message window displayed [Figure 20] An explanatory diagram showing the contents and timing of messages displayed in the message window [Figure 21] FIG. 10 is an explanatory diagram showing a confirmation screen for ending imaging displayed on a touch panel display of the imaging device. [Figure 22] FIG. 10 is an explanatory diagram showing a self-location lost notification screen displayed on a touch panel display of the imaging device. [Figure 23] FIG. 10 is an explanatory diagram showing a recovery screen displayed on a touch panel display of the imaging device. [Figure 24] FIG. 10 is an explanatory diagram showing an enlarged screen of the final position acquisition point displayed on the touch panel display of the imaging device. [Figure 25] FIG. 10 is an explanatory diagram showing a recovery failure notification screen displayed on a touch panel display of the imaging device. [Figure 26] FIG. 10 is an explanatory diagram showing a connection failure notification screen displayed on a touch panel display of the imaging device. [Figure 27] FIG. 10 is an explanatory diagram showing a confirmation screen for ending the imaging mode displayed on the touch panel display of the imaging device. [Figure 28] FIG. 10 is a block diagram showing the schematic configuration of an imaging device and a control device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] The first invention made to solve the above problem is: A photographing device that photographs the measurement target location, and a control device that is connected to the photographing device and executes 3D measurement processing of the measurement target location. The photographing system includes a sensor unit including a camera for photographing a measurement target location, and a camera for supporting a user's photographing work. Screen used in shooting mode and a touch panel display that displays the screen and detects screen operations by the user. First display and, and a support having a grip portion to be held by a user in one hand and supporting the sensor portion and the first display portion, wherein the control device comprises: a 3D measurement processing portion that performs the 3D measurement processing to generate 3D spatial information of a measurement target location based on imaging data acquired from the imaging device; a second display portion that displays a screen used in a non-imaging mode for performing operations related to management and processing of the imaging data; and a display control portion that controls display information of the screens to be displayed on the first display portion and the second display portion, wherein the display control portion causes the second display portion to display a pause screen and the first display portion to display an operation screen for the user to give at least an instruction to start imaging, in response to a user operation to switch from the non-imaging mode to the imaging mode. The composition is as follows.

[0014] According to this, When a user operates to transition from non-photography mode to photography mode, the display control unit displays a pause screen on the second display unit of the control device and also displays an operation screen on the first display unit of the photography device, on which the user can give at least an instruction to start photography, so that the transition to photography mode in the photography system can be confirmed; You can easily use your other hand to operate the touch panel display to give instructions and make settings for shooting. can.

[0015] The second invention is: The display control unit The operation screen may be a shooting standby screen, a shooting in progress screen, and a shooting end confirmation screen, which are displayed on the touch panel display in a sequential manner in response to a user's instruction.

[0016] This allows the user to appropriately make the necessary settings and checks at each stage before, during, and after shooting.

[0017] The third invention is: The display control unit The photographing standby screen is displayed including an operation section for allowing the user to select one of a plurality of photographing modes having different control conditions for the camera.

[0018] According to this, the user can specify the shooting mode according to the characteristics of the measurement location before starting shooting. In this case, either an indoor shooting mode or an outdoor shooting mode may be selected. Furthermore, either of two outdoor shooting modes according to the presence or absence of a neutral density filter may be selected.

[0019] The fourth invention is: The display control unit The screen during shooting is configured to display a first image display frame that displays an enlarged version of either the image captured by the camera or the shooting path image showing the path traveled by the shooting device, a second image display frame that displays a reduced version of the other image, and an operation unit that switches the images to be displayed in the first image display frame and the second image display frame.

[0020] This allows the user to switch between an image to be displayed enlarged and an image to be displayed reduced, as needed. Note that the second image display frame itself may be the operation unit.

[0021] The fifth invention is: The display control unitWhen the image capture is completed, the user is notified that the image capture data has been saved and the image capture standby screen has been restored, and the image capture completion confirmation screen including an operation unit for the user to input confirmation is displayed.

[0022] This allows the user to easily confirm that the image capture has ended.

[0023] The sixth invention is: The display control unit The speed at which the user moves the sensor unit is measured, and the image-taking screen is displayed, including an image that visualizes whether the speed is appropriate.

[0024] This allows the user to easily understand whether the speed at which the sensor unit is moved is appropriate.

[0025] The seventh invention is: The display control unit At the start of photography, the photography-in-progress screen including a predetermined available photography time is displayed, and during photography, at the timing when the remaining photography time obtained by subtracting the elapsed time from the photography start from the available photography time becomes equal to or less than a predetermined value, the photography-in-progress screen including the remaining photography time is displayed.

[0026] This allows the user to know the remaining shooting time when starting shooting, and also allows the user to know the remaining shooting time when the remaining shooting time becomes short.

[0027] The eighth invention is: The display control unit When the tracking process detects that the image capturing device has lost its current position, the device notifies the user of the loss of its position, and also displays a screen on the touch panel display to prompt the user to return to the last position acquisition point.

[0028] This allows the user to quickly return to the last position acquisition point and start photographing again when the user loses his / her own position.

[0029] The ninth invention is: The display control unit the camera and the touch panel display; the control device and a screen notifying the user of the occurrence of the connection failure is displayed on the touch panel display.

[0030] This allows the user to easily understand that the imaging device is not operating normally due to a poor connection.

[0035] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0036] (First embodiment) Fig. 1 is an explanatory diagram showing the situation of a photographing operation performed by a user using the photographing system according to the first embodiment, Fig. 2 is an explanatory diagram showing a standard photographing state and a high-altitude photographing state.

[0037] As shown in FIG. 1, the photography system includes a photography device 1 and a control device 2 (information processing device).

[0038] The photographing device 1 includes a sensor unit 11, a display / input panel unit 12, and a rod-shaped support member 13. The sensor unit 11 and the display / input panel unit 12 of the photographing device 1 are connected via a first cable 21. The display / input panel unit 12 of the photographing device 1 and the control device 2 are connected via a second cable 22.

[0039] The control device 2 is configured as a laptop or tablet PC that can be carried by a user (worker). In the example shown in Fig. 1, the control device 2 can be stored in a shoulder bag and carried by the user, but the manner in which the user carries the control device 2 is not limited to this.

[0040] A user performs a photographing task by walking around a place to be measured while holding the photographing device 1 in his / her hand and making the photographing device 1 take pictures of the place to be measured. At this time, the user can change the position (height) of the sensor unit 11 by moving the arm holding the photographing device 1.

[0041] For example, as shown in Fig. 2(A), by slightly extending the arm holding the photographing device 1 forward, the user can take a photograph with the sensor unit 11 positioned at approximately the same height as the user's eye level (standard photographing state). On the other hand, as shown in Fig. 2(B), by raising the arm holding the photographing device 1, the user can take a photograph with the sensor unit 11 positioned at a higher position than the user's eye level (high altitude photographing state).

[0042] Next, a description will be given of the support 13 of the image capturing device 1. Fig. 3 is a perspective view showing the support 13. Fig. 4 is a side view showing the expansion and contraction state of the support 13.

[0043] 3, the support 13 of the photographing device 1 includes a main body 31, an extendable rod 32, a sensor mounting portion 33, a panel mounting portion 34, and a grip portion 35. The display / input panel 12 includes a touch panel display 36.

[0044] The panel mounting unit 34 is provided on the main body 31. The panel mounting unit 34 includes a joint 41 (ball joint) that connects the display / input panel unit 12 and the main body 31. The joint 41 allows the orientation of the display / input panel unit 12, i.e., the orientation of the display surface of the touch panel display 36, to be changed. Specifically, the display / input panel unit 12 rotates around two axes A1 and A2 that pass through the center of the joint 41 (ball joint), allowing the display / input panel unit 12 to perform a tilting action that changes the orientation of the display / input panel unit 12 up and down, and a panning action that changes the orientation of the display / input panel unit 12 left and right. The panel mounting unit 34 also includes a fixing knob 42. The user can fix the orientation of the display / input panel unit 12 by operating the fixing knob 42. In the example shown in FIG. 3, the joint 41 is a ball joint, so the orientation of the display / input panel unit 12 can be adjusted to any direction within a predetermined angle range.

[0045] In this way, the user can adjust the orientation of the display / input panel unit 12 as appropriate, thereby improving the visibility and operability of the touch panel display 36. For example, when changing the angle of the rod-shaped support 13, such as when photographing at a high altitude, adjusting the orientation of the display / input panel unit 12 in the tilt direction makes it easier to view and operate the screen of the touch panel display 36. Also, for example, when the user holds the photographing device 1 in his or her right hand, adjusting the orientation of the display / input panel unit 12 in the pan direction so that the screen of the touch panel display 36 faces the left side, where the user's face and left hand are located to operate the screen, makes it easier to view and operate the screen of the touch panel display 36.

[0046] The sensor mounting portion 33 is provided at the tip of the extendable rod portion 32. The sensor mounting portion 33 includes a joint 43 (hinge joint) that connects the sensor unit 11 to the tip of the extendable rod portion 32. This joint 43 allows the orientation of the sensor unit 11 to be changed. Specifically, by rotating the sensor unit 11 around the central axis A3 of the joint 43 (hinge joint), the sensor unit 11 can be caused to perform a tilting operation that changes the orientation of the sensor unit 11 up and down. A fixing knob 44 is provided on the sensor mounting portion 33. The user can fix the orientation of the sensor unit 11 by operating the fixing knob 44.

[0047] In the example shown in Figure 3, the display input panel unit 12 can be tilted and panned by joint 41 made of a ball joint in the panel mounting unit 34, but the display input panel unit 12 can also be tilted and panned by combining multiple hinge joints.

[0048] 3, the joint 43 formed of a hinge joint in the sensor mounting section 33 allows only a tilting operation to change the orientation of the sensor section 11 up and down, but the joint 43 may also allow a panning operation to change the orientation of the sensor section 11 left and right. In this case, the joint 43 may be formed of a ball joint or a combination of multiple hinge joints.

[0049] The extensible rod portion 32 is configured to be extensible and retractable in the axial direction. In the example shown in Fig. 4, the extensible rod portion 32 has a structure in which a plurality of sheath tubes with gradually decreasing diameters are slidably connected.

[0050] By extending or retracting the extendable rod portion 32, the user can change the position of the sensor portion 11 relative to the grip portion 35 that is held by hand. That is, as shown in FIG. 4(A), when the extendable rod portion 32 is extended (extended state), the sensor portion 11 is separated from the grip portion 35. This allows the sensor portion 11 to be positioned at a higher position when photographing at high altitudes (see FIG. 2(B)). On the other hand, as shown in FIG. 4(B), when the extendable rod portion 32 is retracted and most of the extendable rod portion 32 is stored in the main body portion 31 (stored state), the overall length of the photographing device 1 is shortened, making it easier to carry and store the photographing device 1 when not in use.

[0051] As described above, in this embodiment, the user can hold the photographing device 1 stably in one hand by grasping the grip portion 35 with the hand. Therefore, the user can take a photograph with the sensor portion 11 facing in any direction by changing the orientation of the arm or body holding the photographing device 1 during photographing. Furthermore, the user can adjust the position of the sensor portion 11 and the orientation of the display input panel portion 12 by extending and retracting the extendable rod even during photographing.

[0052] In this embodiment, the display input panel unit 12 is fixed to the main body unit 31, but the display input panel unit 12 may be fixed to the middle of the extendable rod unit 32 so that the position of the display input panel unit 12 relative to the grip unit 35 can be adjusted. Also, the display input panel unit 12 may be fixed to the main body unit 31 via a movable member (not shown) separate from the extendable rod unit 32 to which the sensor unit 11 is fixed at the tip.

[0053] Next, a description will be given of the sensor unit 11 of the photographing device 1. FIG.

[0054] The sensor section 11 includes a sensor unit 51 and a sensor cover 52 .

[0055] The sensor unit 51 includes a visible camera 53 and a depth sensor 54. The visible camera 53 (color camera) captures color images of the subject. The depth sensor 54 includes an infrared projector 55 and left and right infrared cameras 56, 57 (stereo cameras). The infrared projector 55 irradiates the subject with infrared light. The left and right infrared cameras 56, 57 detect the reflected light of the infrared light irradiated on the subject. Note that the depth sensor 54 may be a sensor capable of acquiring depth information using other methods, such as LiDAR (Light Detection and Ranging), in addition to a stereo camera.

[0056] In the sensor unit 51, a visible camera 53, left and right infrared cameras 56 and 57, and an infrared projector 55 are arranged side by side (substantially horizontally). The housing of the sensor unit 51 has a shape that is long in the horizontal direction.

[0057] The sensor cover 52 includes a cover body 61 and a filter member 62 .

[0058] The cover body 61 covers the sensor unit 51. During shooting, the sensor section 11 may collide with surrounding objects while the photographing device 1 is being moved, and the cover body 61 is provided to protect the sensor unit 51 from the impact of such a collision. The cover body 61 has an opening 66 that exposes the cover glass on the front side of the visible light camera 53, the infrared projector 55, and the infrared cameras 56 and 57.

[0059] The filter member 62 is attached to the cover body 61 so as to be slidable in the longitudinal direction of the sensor unit 51, i.e., in the arrangement direction of the visible light camera 53, the infrared projector 55, and the infrared cameras 56 and 57. The filter member 62 includes a first ring portion 63 and a second ring portion 64. An ND (Neutral Density) filter 65 (neutral density filter) is attached to the first ring portion 63.

[0060] The filter member 62 is provided so as to be slidable between a first position shown in Fig. 5(A) and a second position shown in Fig. 5(B). A user can grip the filter member 62 with their fingers and slide it. In the first position shown in Fig. 5(A), the ND filter 65 does not cover the visible light camera 53. In the second position shown in Fig. 5(B), the ND filter 65 covers the visible light camera 53.

[0061] 5(A), the first ring portion 63 to which the ND filter 65 is attached is located between the visible light camera 53 and one of the infrared cameras 57, and the second ring portion 64 is located at a position surrounding the lens of one of the infrared cameras 57. As a result, none of the visible light camera 53, the infrared projector 55, and the infrared cameras 56 and 57 are covered by the filter member 62.

[0062] 5(B), the first ring portion 63 is positioned at a position corresponding to the visible camera 53, and the second ring portion 64 is positioned at a position between the visible camera 53 and one of the infrared cameras 57. As a result, only the visible camera 53 is covered by the ND filter 65, and neither the infrared projector 55 nor the infrared cameras 56, 57 are covered by the filter member 62.

[0063] When the second ring portion 64 abuts against the inner periphery of the opening of the cover main body 61, the filter member 62 is positioned at the first position shown in Fig. 5(A). Also, a groove (not shown) is formed on the underside of the cover main body 61, extending in the direction of movement of the filter member 62, and when a protrusion provided on the filter member 62 fits into this groove, the movement range of the filter member 62 is defined, and the filter member 62 is positioned at the second position shown in Fig. 5(B).

[0064] When photographing outdoors, the user adjusts the filter member 62 to the second position (see FIG. 5(B)) where the ND filter 65 covers the visible camera 53. On the other hand, when photographing indoors, the user adjusts the filter member 62 to the first position (see FIG. 5(A)) where the ND filter 65 does not cover the visible camera 53. This makes it possible to adjust the brightness of the entire photographed image by reducing the amount of light input to the camera when photographing outdoors. It also makes it possible to avoid the problem of the entire photographed image being tinted purple. In addition, because only the visible camera 53 (color camera) is covered with the ND filter 65, it is possible to avoid the problem of the density of the generated point cloud decreasing when the infrared cameras 56, 57 are also covered with the ND filter 65.

[0065] Next, we will explain the general configuration of the photographing device 1 and the control device 2. Fig. 6 is a block diagram showing the general configuration of the photographing device 1 and the control device 2. Fig. 7 is a block diagram showing an overview of the processing performed by the processor 86 of the control device 2.

[0066] As shown in FIG. 6, the sensor unit 11 of the image capturing device 1 includes a visible camera 53, a depth sensor 54, an IMU 71 (Inertial Measurement Unit), an input / output interface 72, and a connection terminal 73.

[0067] The visible light camera 53 (color camera) takes color images and outputs the color captured images.

[0068] The depth sensor 54 outputs depth information (distance image) as a detection result based on images captured by the left and right infrared cameras 56 and 57 (see FIG. 5).

[0069] The IMU 71 detects the motion state of the device itself, specifically, three-dimensional angular velocity and acceleration. Based on the detection results of the IMU 71, the position, orientation, and velocity of the sensor unit 11 can be detected.

[0070] The input / output interface 72 inputs and outputs data to and from the control device 2 via the display / input panel unit 12. Specifically, the detection data of the visible camera 53, the depth sensor 54, and the IMU 71 are transmitted through the input / output interface 72. The input / output interface 72 may be based on the USB (registered trademark) standard.

[0071] The first cable 21 that connects the sensor unit 11 and the display / input panel unit 12 is detachably coupled to the connection terminal 73 .

[0072] The visible camera 53, depth sensor 54, and IMU 71 may not be integrated into the sensor unit 51 (see FIG. 5). Alternatively, the depth sensor 54 and IMU 71 may be omitted, and only the visible camera 53 may be provided. Alternatively, the visible camera 53 may be provided together with either the depth sensor 54 or the IMU 71.

[0073] The display / input panel unit 12 of the photographing device 1 includes a touch panel display 36 , a repeater 74 , an input / output interface 75 , a first connection terminal 76 , and a second connection terminal 77 .

[0074] The touch panel display 36 displays, under the control of the control device 2, a screen that assists the user in taking photographs.

[0075] The repeater 74 repeats data communication between the control device 2 and the sensor unit 11. The repeater 74 may be a hub based on the USB (registered trademark) standard.

[0076] The input / output interface 75 inputs and outputs data to and from the control device 2. Specifically, it receives display information such as a screen that supports the user's photographing operation from the control device 2. The input / output interface 75 may be based on the USB (registered trademark) standard.

[0077] A first cable 21 that connects the sensor unit 11 and the display input panel unit 12 is detachably coupled to the first connection terminal 76. A second cable 22 that connects the display input panel unit 12 and the control device 2 is detachably coupled to the second connection terminal 87.

[0078] The first cable 21 is used for data communication between the control device 2 and the sensor unit 11 via the display input panel unit 12. In this data communication, for example, shooting data (shooting information) output from the sensor unit 11 is transmitted to the control device 2. The shooting data includes an image captured by the visible camera 53, a detection result (distance information) from the depth sensor 54, a detection result from the IMU 71, and a shooting time. The second cable 22 is used for data communication between the control device 2 and the display input panel unit 12 and data communication between the control device 2 and the sensor unit 11. In the data communication between the control device 2 and the display input panel unit 12, for example, display information of a screen to be displayed on the touch panel display 36 is transmitted from the control device 2, and user operation information detected on the touch panel display 36 is transmitted to the control device 2.

[0079] The control device 2 includes an input / output interface 81 , a display 82 , an input device 83 , a memory 84 , a storage device 85 , a processor 86 , and a connection terminal 87 .

[0080] The input / output interface 81 inputs and outputs data to and from the image capturing device 1. The input / output interface 81 may be based on the USB (registered trademark) standard.

[0081] The display 82 displays a screen for managing previously acquired photographic data, a screen for setting the operating conditions of the photographic device 1, and the like.

[0082] The input device 83 is used by the user to perform input operations. The input device 83 may be a keyboard, a mouse, a touchpad, a touch panel, etc. If the control device 2 is configured as a tablet PC, a touch panel display 36 is provided in which a touch panel as the input device 83 and a display panel as the display 82 are integrated.

[0083] A second cable 22 that connects the control device 2 and the display input panel unit 12 of the image capturing device 1 is detachably coupled to the connection terminal 87 .

[0084] The memory 84 stores programs executed by the processor 86 and the like.

[0085] The storage device 85 stores the shooting data (shooting information) acquired from the image capturing device 1. The shooting data includes the image captured by the visible camera 53, the detection result (distance information) of the depth sensor 54, the detection result of the IMU 71, the shooting time, etc. The storage device 85 also stores point cloud data generated by the processor 86 as a three-dimensional measurement result.

[0086] The processor 86 performs various processes by executing programs stored in the memory 84. In this embodiment, the processor 86 performs a three-dimensional measurement process P1.

[0087] In the three-dimensional measurement process P1, the processor 16 generates point cloud data (environment map) as three-dimensional spatial information about the measurement target location using the SLAM (Simultaneous Localization And Mapping) method based on images captured by the visible camera 53. In the three-dimensional measurement process P1, self-position estimation is performed in addition to the generation of the point cloud data, and the self-position for each measurement, i.e., the position of the imaging point, is acquired.

[0088] As shown in FIG. 7, the three-dimensional measurement process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0089] In the feature extraction process P11, the processor 86 extracts feature information (feature points, etc.) from the image (frame) captured by the visible camera 53.

[0090] In tracking process P12, processor 86 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition related to the position and orientation of the image capture device 1, and updates the position and orientation trajectory data based on the amount of transition. Note that the position and orientation trajectory data is obtained by tracking the position and orientation of the image capture device 1, and includes the tracking results of the position and orientation at the time of capturing each captured image (frame), i.e., information regarding the position and orientation of the image capture device 1 at each time of capturing.

[0091] In the position and orientation correction process P13, the processor 86 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, the position and orientation trajectory data is corrected so as to compensate for the measurement results of locations with few features, such as walls and ceilings.

[0092] In the point cloud generation process P14, the processor 86 generates point cloud data based on the distance information from the depth sensor 54 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13. The point cloud generation process P14 includes a process (standard point cloud generation process) for generating regular point cloud data acquired as a 3D measurement result, and a process (simplified point cloud generation process) for generating simplified point cloud data that allows the user to check the shooting conditions (point cloud generation conditions). The simplified point cloud generation process must be performed in real time during shooting, but the standard point cloud generation process may be performed after shooting.

[0093] As shown in FIG. 6, the processor 86 also performs a drawing generation process P2, a display information generation process P3, and a display process P4.

[0094] In the drawing generation process P2, the processor 86 generates a 3D model of the target location based on the point cloud data of the target location generated by the 3D measurement process P1, and generates a layout drawing of the target location based on the 3D model of the target location. At this time, a 2D layout drawing (plan view) of the target location is generated by projecting the 3D model onto a horizontal plane. In addition, a 3D layout drawing of the target location is generated by projecting the 3D model based on a predetermined line of sight.

[0095] In the display information generation process P3, the processor 86 generates display information for a screen to be displayed on the display 82 of the control device 2. The display 82 displays screens (see FIGS. 8 to 14) on which the user performs operations such as managing the photographed data, issuing instructions for post-photography processing (point cloud generation process P14, drawing generation process P2), and setting processing conditions, etc. The processor 86 also generates display information for a screen to be displayed on the touch panel display 36 of the photographing device 1. The touch panel display 36 displays screens (see FIGS. 15 to 19, 21 to 27) that support the user's photographing work.

[0096] In display processing P4, processor 86 displays a screen (see FIGS. 8 to 14) on display 82 of control device 2 based on the display information generated in display information generation processing P3. Processor 86 also displays a screen (see FIGS. 15 to 19 and 21 to 27) on touch panel display 36 of imaging device 1.

[0097] In this embodiment, a shooting standby screen 311 (see FIG. 16), a shooting in progress screen 331 (see FIG. 17), and shooting end confirmation screens 351 and 361 (see FIG. 21) are displayed on the touch panel display 36 of the photographing device 1 so as to transition sequentially in response to a user's instruction. This allows the user to appropriately make necessary settings and checks at each stage before, during, and at the end of shooting.

[0098] In this embodiment, the non-photographing mode (first mode) and the photographing mode (second mode) are switched in response to a user operation.

[0099] In the non-photographing mode, an operable screen (see Figures 8 to 14) is displayed on the display 82 of the control device 2, which allows the user to give instructions for managing and processing the photographed data, and an inoperable pause screen 301 (see Figure 15) is displayed on the touch panel display 36 of the photographing device 1.

[0100] On the other hand, in the photography mode, photography becomes possible using the photography device 1, and operable screens are displayed on the touch panel display 36 of the photography device 1, such as screens to assist the user in the photography operation (see Figures 15 to 19, 21 to 27), such as screens for instructing the start and end of photography (see Figures 16 and 17), and an inoperable pause screen 221 (see Figure 13) is displayed on the display 82 of the control device 2.

[0101] As shown in FIG. 6, the processor 86 also performs a connection failure detection process P5, a speed presentation process P6, a photographing time presentation process P7, and a self-position lost recovery process P8.

[0102] In the connection failure detection process P5, the processor 86 detects a connection failure on the communication path between the processor 86 and the sensor unit 51 and the touch panel display 36, and notifies the user of the occurrence of the connection failure. Specifically, connection failure notification screens 231, 241 (see FIG. 14) are displayed on the display 82 of the control device 2. In addition, a connection failure notification screen 411 (see FIG. 26) is displayed on the touch panel display 36 of the imaging device 1.

[0103] In the speed presentation process P6, the processor 86 measures the speed at which the user moves the sensor unit 11 based on the detection data (acceleration, angular velocity) of the IMU 71, and visualizes whether the speed is appropriate or not to present to the user. Specifically, a speed bar 334 is displayed on the shooting screen 331 (see FIG. 17).

[0104] In the shooting time presentation process P7, the processor 86 presents the user with a preset available shooting time at the start of shooting. Also, during shooting, the processor 86 presents the user with the remaining shooting time when the remaining shooting time, calculated by subtracting the elapsed time from the start of shooting from the available shooting time, falls below a predetermined time. Specifically, on the shooting-in-progress screen 331 (see FIG. 18), the available shooting time is presented at the start of shooting, and then the remaining shooting time is presented during shooting.

[0105] In the self-position lost return process P8, when the processor 86 detects that the current position of the image capturing device 1 has been lost (tracking lost) in the tracking process P12, the processor 86 notifies the user of the occurrence of the self-position lost, and presents the user with the last position acquisition point to provide return guidance urging the user to return to the last position acquisition point. Specifically, a self-position lost notification screen 371 (see FIG. 22) is displayed, followed by a returning screen 381 (see FIG. 23) including an image captured at the last position acquisition point. Furthermore, when the processor 86 detects that the image capturing device 1 has returned to the last position acquisition point, the normal tracking process P12 is resumed.

[0106] Next, a description will be given of the photographing data list screen 101 displayed on the display 82 of the control device 2. FIG.

[0107] When the photography application is started in the control device 2, a photography data list screen 101 is displayed on the display 82.

[0108] The photography data list screen 101 is provided with a list display section 102. The list display section 102 displays a list of photographed images for each of a plurality of photography data with different measurement locations and photography dates and times. In the example shown in FIG. 8, the photographed images for each photography data are displayed arranged by date. The list display section 102 also displays the photography date and time for each photographed image. If the point cloud generation process P14 has been performed, the list display section 102 displays a check mark 111 indicating this, and if the drawing generation process P2 has been performed, the list display section 102 displays a check mark 112 indicating this.

[0109] When the user selects a piece of photography data by operating one of the photographed images for each piece of photography data displayed in the list display section 102, the screen transitions to a photography data details screen 121 (see FIG. 9) for the selected photography data. At this time, the user can select photography data to be deleted or photography data for which the point cloud generation process P14 and drawing generation process P2 are to be executed.

[0110] If there is no registered photographic data, a message indicating that there is no photographic data is displayed in the list display section 102, such as the text "There is no photographic data."

[0111] Furthermore, the photographing data list screen 101 is provided with a setting button 103. When the user operates the setting button 103, a setting screen 181 (see FIG. 12(A)) is displayed as a pop-up on the photographing data list screen 101.

[0112] The photographing data list screen 101 also has a "Start photographing" button 104. When the user operates the "Start photographing" button 104, the mode switches from non-photographing mode to photographing mode. At this time, the display 82 of the control device 2 transitions to a pause screen 221 (see FIG. 13). Meanwhile, the touch panel display 36 of the photographing device 1 transitions from a pause screen 301 (see FIG. 15) to a photographing standby screen 311 (see FIG. 16).

[0113] Next, the photographing data details screen 121 displayed on the display 82 of the control device 2 will be described. Fig. 9 is an explanatory diagram showing the photographing data details screen 121. Fig. 10 is an explanatory diagram showing a screen that is displayed as a pop-up on the photographing data details screen 121. Fig. 11 is an explanatory diagram showing the transition status of the main parts of the photographing data details screen 121.

[0114] The photographing data details screen 121 is provided with an image display section 122 and a photographing date and time display section 123. The photographed images (key frames) included in the photographing data are played back in order in the image display section 122. The photographing date and time display section 123 displays the photographing date and time of the photographed image displayed in the image display section 122. In the SLAM method, a frame in which a large change in the field of view appears is extracted as a key frame, and the key frame is reflected in the point cloud.

[0115] The photographic data details screen 121 also has a playback operation section 124. The playback operation section 124 has a seek bar 131, a playback stop button 132, a frame forward button 133, and a frame back button 134. The user can arbitrarily specify a playback start time by operating the slider of the seek bar 131. The user can also instruct playback and stop of the photographic data by operating the playback stop button 132. The user can also instruct frame forward and frame back of the photographic data by operating the frame forward button 133 and the frame back button 134.

[0116] The imaging data details screen 121 also has an instruction operation section 125. The instruction operation section 125 has a "generate point cloud" button 141, a "display point cloud" button 142, a "generate drawing" button 143, and a "display drawing" button 144.

[0117] When the user operates the "Generate point cloud" button 141, the point cloud generation process P14 (standard point cloud generation process) is started, and a point cloud generation in progress screen 151 shown in FIG. 10(A) pops up on the shooting data details screen 121. The point cloud generation in progress screen 151 displays a message indicating that the point cloud generation process P14 is being executed. The point cloud generation in progress screen 151 also has a progress bar 152. The progress bar 152 visualizes the progress of the point cloud generation process P14. The point cloud generation in progress screen 151 also has a "Cancel point cloud generation" button 153. When the user operates the "Cancel point cloud generation" button 153, the point cloud generation process P14 is canceled, and the screen returns to the shooting data details screen 121 (see FIG. 9). When the point cloud generation process P14 is completed, the screen returns to the shooting data details screen 121.

[0118] Furthermore, when the user operates the "point cloud display" button 142 on the photographing data details screen 121 shown in FIG. 9, a point cloud display application is started and the point cloud data generated in the point cloud generation process P14 is displayed.

[0119] Furthermore, when the user operates the "Drawing Generation" button 143, a drawing generation in progress screen 161 shown in FIG. 10(B) pops up on the photographing data details screen 121. The drawing generation in progress screen 161 displays a message indicating that the drawing generation process P2 is being executed. The drawing generation in progress screen 161 also has a "Cancel drawing generation" button 162. When the user operates the "Cancel drawing generation" button 162, the drawing generation process P2 is canceled and the screen returns to the photographing data details screen 121 (see FIG. 9). Furthermore, when the drawing generation process P2 is completed, the screen returns to the photographing data details screen 121. The drawing generation in progress screen 161 may also display an animation indicating that the drawing generation process P2 is being executed.

[0120] Furthermore, when the user operates the "Drawing display" button 144 on the photographing data details screen 121 shown in FIG. 9, a drawing display application is started and the drawing generated in the drawing generation process P2 is displayed.

[0121] Furthermore, the photographing data details screen 121 is provided with a "Delete" button 145. When the user operates the "Delete" button 145, a deletion confirmation screen 171 shown in FIG. 10(C) pops up on the photographing data details screen 121. The deletion confirmation screen 171 displays a message for the user to confirm the deletion of the photographing data. Furthermore, the deletion confirmation screen 171 is provided with a "Delete" button 172 and a "Cancel" button 173. When the user operates the "Delete" button 172, the photographing data is deleted, and the screen returns to the photographing data list screen 101 (see FIG. 8). Furthermore, when the user operates the "Cancel" button 173, the screen returns to the photographing data details screen 121 (see FIG. 9).

[0122] Here, as shown in FIG. 11, in the instruction operation unit 125, the states of the "Point Cloud Generation" button 141, the "Point Cloud Display" button 142, the "Drawing Generation" button 143, and the "Drawing Display" button 144 change before and after the point cloud generation process P14 and the drawing generation process P2.

[0123] First, as shown in FIG. 11(A), before the point cloud generation process P14 is executed, only the "Generate Point Cloud" button 141 is operable, and the "Display Point Cloud" button 142, "Generate Drawing" button 143, and "Display Drawing" button 144 are inoperable. When the point cloud generation process P14 is completed, as shown in FIG. 11(B), a check mark 146 is displayed indicating that the point cloud generation process P14 has been executed. Also, in this state, the "Display Point Cloud" button 142 and the "Generate Drawing" button 143 are operable, but the "Display Drawing" button 144 is inoperable. When the drawing generation process P2 is completed, as shown in FIG. 11(C), a check mark 147 is displayed indicating that the drawing generation process P2 has been executed. Also, in this state, the "Display Drawing" button 144 is operable.

[0124] 9, the photographing data details screen 121 is provided with a "return to list" button 126. When the user operates the "return to list" button 126, the screen returns to the photographing data list screen 101 (see FIG. 8).

[0125] Next, a description will be given of the setting screen 181 displayed on the display 82 of the control device 2. FIG.

[0126] When the user operates the setting button 103 on the photographing data list screen 101 (see FIG. 8), a setting screen 181 shown in FIG.

[0127] The setting screen 181 has a save folder setting section 182. The save folder setting section 182 displays a folder path indicating the location where a setting file containing setting information will be saved. The save folder setting section 182 also has a "change" button 191. When the user operates the "change" button 191, a screen (not shown) for specifying a folder pops up and allows the user to specify a folder in which to save the setting file.

[0128] The setting screen 181 also has a camera setting section 183. The camera setting section 183 has input sections 192, 193, and 194 for the resolution of the visible camera 53, the resolution of the depth sensor 54, and the frame rate of the visible camera 53. When the user operates each of the input sections 192, 193, and 194, a pull-down menu (not shown) is displayed. The user can select the resolution of the visible camera 53, the resolution of the depth sensor 54, and the frame rate of the visible camera 53 in the pull-down menu.

[0129] The setting screen 181 also has a device information display section 184. The device information display section 184 displays, as information about the sensor unit 51, registration information about each item of the name (camera name) of the sensor unit 51, the serial number, and the firmware version, as well as the date and time when the previous calibration process was performed. The device information display section 184 also has a "forced restart" button 195. When the user operates the "forced restart" button 195, the sensor unit 51 is forcibly restarted. The device information display section 184 also has an "execute" button 196. When the user operates the "execute" button 196, a device calibration screen 201 shown in FIG. 12(B) is popped up on the setting screen 181.

[0130] Furthermore, a "Done" button 185 is provided on the setting screen 181. When the user operates the "Done" button 185, the setting process is executed based on the input contents on the setting screen 181, and the screen returns to the shooting data list screen 101 (see FIG. 8).

[0131] As shown in FIG. 12(B), the device calibration screen 201 displays a message prompting the user to keep the image capturing device 1 stationary in a stable location for a predetermined time for the calibration process, as well as the remaining time. The device calibration screen 201 also displays a progress bar 202 indicating the progress of the calibration process. The device calibration screen 201 also has a "Cancel" button 203. When the user operates the "Cancel" button 203, the calibration process is canceled and the screen returns to the setting screen 181 (see FIG. 12(A)). When the calibration process is completed, the screen returns to the setting screen 181.

[0132] Next, a description will be given of the pause screen 221 and the poor connection notification screens 231 and 241 displayed on the display 82 of the control device 2. Fig. 13 is an explanatory diagram showing the pause screen 221. Fig. 14 is an explanatory diagram showing the poor connection notification screens 231 and 241.

[0133] When the user operates the "Start shooting" button 104 on the shooting data list screen 101 (see FIG. 8), the non-shooting mode is switched to the shooting mode.

[0134] If the transition to the photographing mode is successful, the screen transitions to a pause screen 221 shown in Fig. 13. The pause screen 221 displays a message indicating that the mode has been switched to the photographing mode, and a message indicating that the screen can be displayed and operated on the display 82 of the control device 2 by performing an operation to end the photographing mode on the screen displayed on the touch panel display 36 of the photographing device 1.

[0135] On the other hand, if the transition to the photographing mode fails, poor connection notification screens 231, 241 (error screens) shown in Fig. 14 pop up on the photographing data list screen 101 depending on the cause of the failure. Here, if a poor connection has occurred in the second cable 22 connecting the display / input panel unit 12 of the photographing device 1 and the control device 2, poor connection notification screen 231 shown in Fig. 14(A) is displayed. On the other hand, if a poor connection has occurred in the first cable 21 connecting the display / input panel unit 12 of the photographing device 1 and the sensor unit 11, poor connection notification screen 241 shown in Fig. 14(B) is displayed.

[0136] The poor connection notification screen 231 shown in FIG. 14(A) displays a message indicating that the image capture mode cannot be entered due to a poor connection between the display / input panel unit 12 of the image capture device 1 and the control device 2, and a message urging the user to check the location of the poor connection. The poor connection notification screen 231 also has a "retry" button 232 and a "cancel" button 233. The user operates the "retry" button 232 after performing work to resolve the poor connection of the second cable 22. This causes the process of entering the image capture mode to be executed again. On the other hand, if the user operates the "cancel" button 233, the screen returns to the image capture data list screen 101 (see FIG. 8).

[0137] 14(B) displays a message indicating that the image capture mode cannot be entered due to a poor connection between the display / input panel unit 12 and the sensor unit 11 of the image capture device 1, and a message urging the user to check the location of the poor connection. The poor connection notification screen 241 also has a "retry" button 242 and a "cancel" button 243. The user operates the "retry" button 242 after performing an operation to resolve the poor connection of the first cable 21 connecting the display / input panel unit 12 and the sensor unit 11 of the image capture device 1. This causes the process of entering the image capture mode to be executed again. On the other hand, if the user operates the "cancel" button 243, the screen returns to the image capture data list screen 101 (see FIG. 8).

[0138] Next, a description will be given of the pause screen 301 displayed on the touch panel display 36 of the photographing device 1. FIG.

[0139] In non-photographing mode, that is, while the user is operating the screen of the control device 2 by displaying the photographing data list screen 101 (see Figure 8), the photographing data details screen 121 (see Figure 9), and the setting screen 181 (see Figure 12) on the display 82, a pause screen 301 is displayed on the touch panel display 36 of the photographing device 1.

[0140] The pause screen 301 displays a message indicating that the device is in non-photographing mode, i.e., that the photographing data list screen 101 (see FIG. 8) or the like is being displayed and operated on the display 82 of the control device 2. The pause screen 301 also displays a message indicating that the device will transition to photographing mode by performing an operation to start photographing on the screen displayed on the display 82 of the control device 2.

[0141] Next, a description will be given of the shooting standby screen 311 displayed on the touch panel display 36 of the photographing device 1. FIG.

[0142] 16(A), an image display section 312 is provided on the shooting standby screen 311. The image display section 312 displays an image captured by the visible camera 53 in real time.

[0143] The shooting standby screen 311 also has a shooting mode designation section 313. When the user operates the shooting mode designation section 313, a pull-down menu 314 is displayed, as shown in FIG. 16(B). The pull-down menu 314 allows the user to select from a plurality of shooting modes with different control conditions (processing conditions for image signal processing) for the visible camera 53. In the example shown in FIG. 16(B), the user can select one of "indoor" (indoor shooting mode), "outdoor" (first outdoor shooting mode), and "outdoor + ND filter" (second outdoor shooting mode) as the shooting mode. Note that each shooting mode uses a different brightness correction table, for example, used in image signal processing (image correction processing).

[0144] The shooting standby screen 311 also has a brightness adjustment bar 315. The user can adjust the brightness of the captured image by operating the slider of the brightness adjustment bar 315. Brightness adjustment can be performed for each shooting mode.

[0145] Furthermore, the shooting standby screen 311 is provided with a shooting start button 316. When the user operates the shooting start button 316, shooting starts and the screen transitions to a shooting in progress screen 331 (see FIG. 17).

[0146] Furthermore, an "End shooting" button 317 is provided on the shooting standby screen 311. When the user operates the "End shooting" button 317, the shooting mode ends, the mode shifts to the non-shooting mode, and the screen transitions to the pause screen 301 (see FIG. 15).

[0147] Next, the during-shooting screen 331 displayed on the touch panel display 36 of the photographing device 1 will be described. Fig. 17 is an explanatory diagram showing the during-shooting screen 331. Fig. 18 is an explanatory diagram showing the transition status of the shooting time guide section 336 on the during-shooting screen 331. Fig. 19 is an explanatory diagram showing the during-shooting screen 331 with a message window 337 displayed. Fig. 20 is an explanatory diagram showing the contents of the message displayed in the message window 337 and the display timing.

[0148] 17(A) and (B), a main window 332 (first image display frame) and a sub-window 333 (second image display frame) are provided on a shooting screen 331. The main window 332 and the sub-window 333 have different display magnifications, with the image displayed in an enlarged manner in the main window 332 and the image displayed in a reduced manner in the sub-window 333.

[0149] The main window 332 and the sub-window 333 also display either a real-time image 341 captured by the visible camera 53 or a photography path image 342 representing the photography path at the measurement target location. The photography path image 342 is obtained by superimposing a line 344 representing the photography path on a point cloud image 343 representing the measurement target location. The point cloud image is obtained by imaging (rendering) each point of the generated point cloud data as seen from a predetermined viewpoint.

[0150] The example shown in Fig. 17(A) is the case of the standard state. In this case, a photographed image 341 is displayed enlarged in the main window 332, and a photographing path image 342 is displayed reduced in the sub-window 333. On the other hand, the example shown in Fig. 17(B) is the case of the photographing path enlarged state. In this case, a photographing path image 342 is displayed enlarged in the main window 332, and a photographed image 341 is displayed reduced in the sub-window 333.

[0151] 17(A) and the expanded shooting path state shown in Fig. 17(B) can be switched between by the user operating the sub-window 333. That is, when the sub-window 333 is operated in the standard state shown in Fig. 17(A), the state transitions to the expanded shooting path state shown in Fig. 17(B), and a shooting path image 342 is enlarged and displayed in the main window 332. When the sub-window 333 is operated in the expanded shooting path state shown in Fig. 17(B), the state transitions to the standard state shown in Fig. 17(A), and a captured image 341 is enlarged and displayed in the main window 332.

[0152] A mesh image may be superimposed within the point cloud generation range on the captured image 341. The mesh image can be obtained by converting the generated point cloud data into mesh data and then performing processing to image the mesh data with the same angle of view as the captured image 341.

[0153] Furthermore, the during-shooting screen 331 is provided with a speed bar 334. The speed bar 334 visualizes and displays the speed at which the user moves the sensor unit 11. Specifically, the speed is expressed by color. For example, the speed bar 334 is drawn with a gradation from green to orange, with the proportion of orange increasing as the speed increases. This allows the user to easily check whether the speed at which the sensor unit 11 is moved is appropriate, specifically, whether the speed at which the sensor unit 11 is moved is not too fast. This makes it possible to prevent the user from moving the sensor unit 11 too fast, which causes blurring and reduces the accuracy of the 3D measurement.

[0154] Furthermore, the during-shooting screen 331 is provided with an elapsed time display section 335. The elapsed time display section 335 displays the time that has elapsed since the start of shooting up to the present.

[0155] 18, a shooting time guide section 336 is displayed on the shooting-in-progress screen 331. While the elapsed time display section 335 is always displayed, the shooting time guide section 336 is displayed only for a predetermined period of time at a predetermined timing. Note that the shooting time guide section 336 is not limited to being displayed only at predetermined timings, but can also be set to be always displayed by a user operation.

[0156] 18(A), when shooting starts, the shooting time guide 336 is displayed for a predetermined time (for example, 5 seconds) from the start of shooting. In this case, the shooting time guide 336 displays the available shooting time (for example, 15 minutes).

[0157] Next, as shown in Fig. 18(B), when the remaining shooting time (the time elapsed since the start of shooting minus the available shooting time) becomes equal to or less than a predetermined time, the shooting time indicator 336 is displayed for a predetermined time (for example, 5 seconds). In this case, the remaining shooting time is displayed in the shooting time indicator 336. The timing at which the shooting time indicator 336 is displayed is not limited to once. For example, the shooting time indicator 336 may be displayed when the remaining shooting time is 3 minutes and when the remaining shooting time is 2 minutes.

[0158] Next, as shown in Fig. 18(C), when the remaining shooting time becomes very short (for example, one minute), the shooting time indicator 336 continues to be displayed until the remaining shooting time runs out. In this case, the remaining shooting time is displayed in the shooting time indicator 336. Also, the shooting time indicator 336 is displayed in a manner different from the state shown in Figs. 18(A) and (B). For example, the shooting time indicator 336 is highlighted by changing the color of the background and frame of the shooting time indicator 336.

[0159] 19, a message window 337 is displayed on the during-shooting screen 331. Messages to notify the user are displayed in the message window 337. Specifically, messages regarding guidance for the shooting operation, various warnings, etc. are displayed.

[0160] The example shown in Fig. 19(A) is an information message. Information messages provide the user with information that will be helpful for the user's photography work or information that will draw attention to the user. The example shown in Fig. 19(B) is a warning message. Warning messages prompt the user to take action to improve the status of photography work that needs to be improved immediately.

[0161] The display position of message window 337 on imaging-in-progress screen 331 may differ depending on the type of message (information provision, warning, etc.) For example, in the example shown in Fig. 19(B), message window 337 for a warning message is displayed at the bottom of imaging-in-progress screen 331, similar to the case of the information message shown in Fig. 19(A), but the warning message may also be displayed in the center of imaging-in-progress screen 331 so that the user can immediately check it.

[0162] 20 shows examples of information messages and warning messages displayed in the message window 337. Information messages include a message indicating successful recovery from a self-position loss, a message indicating completion of position correction processing when loop closing (circular movement) is detected, and a message urging the user to stop the image capture device 1 for position correction processing when loop closing is detected. Warning messages include a message urging the user to stop the image capture device 1 if the user moves the image capture device 1 while loop closing is detected and position correction processing is being performed, a message urging the user to move the image capture device 1 slowly if the user is moving the image capture device 1 too quickly, a message urging the user to change the subject to be captured if few feature points are extracted from the captured image, and a message urging the user to maintain an appropriate distance from the subject if the user is too close to the subject.

[0163] Next, a description will be given of the photographing end confirmation screens 351 and 361 displayed on the touch panel display 36 of the photographing device 1. Fig. 21 is an explanatory diagram showing the photographing end confirmation screens 351 and 361.

[0164] 17, an end of shooting button 339 is provided on the during-shooting screen 331. When the user operates the end of shooting button 339, a process of ending shooting and saving the shot data is executed, and an end of shooting confirmation screen 351 is popped up on the during-shooting screen 331, as shown in FIG.

[0165] The shooting end confirmation screen 351 displays a message that shooting has ended and saving of the shooting data has been completed, and a message that returns to the shooting standby screen 311 (see FIG. 16). The shooting end confirmation screen 351 also has an "OK" button 352. When the user operates the "OK" button 352, the screen transitions to the shooting standby screen 311.

[0166] In this embodiment, if the shooting time (the time elapsed since the start of shooting) exceeds a predetermined available shooting time, the shooting is forcibly ended and a process for saving the shot data is executed. In this case, as shown in Fig. 21(B), a shooting end confirmation screen 361 due to shooting time exceeding is popped up on the shooting-in-progress screen 331.

[0167] The shooting end confirmation screen 361 displays a message that shooting has been forcibly ended due to the time limit being exceeded and that saving of the shooting data has been completed, and a message that the screen will return to the shooting standby screen 311 (see FIG. 16). The shooting end confirmation screen 361 also has an "OK" button 362. When the user operates the "OK" button 362, the screen transitions to the shooting standby screen 311.

[0168] In this embodiment, shooting is forcibly terminated when the shooting time exceeds a predetermined shooting time, but such forced termination may not be performed, or the user may be allowed to select a mode in which forced termination is performed or a mode in which forced termination is not performed.

[0169] Next, a description will be given of the screens displayed on the touch panel display 36 of the image capturing device 1 when the self-position is lost. Fig. 22 is an explanatory diagram showing a self-position lost notification screen 371. Fig. 23 is an explanatory diagram showing a recovering screen 381. Fig. 24 is an explanatory diagram showing a last position acquisition point enlarged screen 391. Fig. 25 is an explanatory diagram showing a recovery failure notification screen 401.

[0170] During shooting, a self-position lost (tracking lost) may occur in the tracking process P12 (self-position estimation process), where the self-position is lost. In this case, it is necessary to return to the last position acquisition point, i.e., the last point where the self-position was correctly estimated, and restart the tracking process P12. Therefore, first, the user is notified that a self-position lost has occurred, and then the user is assisted to return to the last position acquisition point. When it is detected that the user has returned to the last position acquisition point, the tracking process P12 is restarted.

[0171] In this embodiment, when a loss of the subject's position is detected, a subject's position lost notification screen 371 is displayed as a pop-up on the during-shooting screen 331 on the touch panel display 36 of the photographing device 1, as shown in FIG.

[0172] The self-position lost notification screen 371 notifies the user that the self-position has been lost and displays a message urging the user to return to the last position acquisition point. The self-position lost notification screen 371 also has an "OK" button 372. When the user operates the "OK" button 372, the screen transitions to a returning screen 381 shown in Fig. 23. At this time, the processor 86 of the control device 2 starts processing to detect that the image capturing device 1 has returned to the last position acquisition point.

[0173] 23(A) is the case of a standard state, similar to the example shown in Fig. 17(A), in which an enlarged photographing image 341 captured by the visible camera 53 is displayed in the main window 332 (first image display frame), and a reduced photographing path image 342 is displayed in the sub-window 333 (second image display frame). Also, the example shown in Fig. 23(B) is the case of an enlarged photographing path state, similar to the example shown in Fig. 17(B), in which an enlarged photographing path image 342 is displayed in the main window 332, and a reduced photographing image 341 is displayed in the sub-window 333.

[0174] Furthermore, in addition to sub-window 333 (second image display frame) in which photographed image 341 and photographing path image 342 are displayed in reduced size, another sub-window 382 (third image display frame) is provided on returning screen 381. A photographed image of the last position acquisition point is displayed in sub-window 382. This allows the user to easily grasp the last position acquisition point.

[0175] Furthermore, a message window 383 is displayed on the returning screen 381. Similar to the self-position lost notification screen 371 (see FIG. 22), the message window 383 displays a message notifying the user that their own position has been lost and urging them to return to the last position acquisition point.

[0176] When the sub-window 382 displaying the captured image of the last position acquisition point is operated, a last position acquisition point enlarged screen 391 is popped up on the returning screen 381, as shown in FIG. 24. The last position acquisition point enlarged screen 391 displays an enlarged image of the last position acquisition point. An "x" button 392 is also displayed on the last position acquisition point enlarged screen 391. When the user operates the "x" button 392, the last position acquisition point enlarged screen 391 is closed and the screen returns to the returning screen 381.

[0177] Here, when the device returns from losing its own position, that is, when it has successfully returned to the last position acquisition point, the screen changes to the shooting screen 331 (see FIG. 17). Note that, when it is detected that the device has returned to the last position acquisition point, the returning screen 381 may display a message notifying that the device has returned to the last position acquisition point, and then the screen may change to the shooting screen 331.

[0178] On the other hand, if the device is unable to return from its own position being lost even after a predetermined time has elapsed, a return failure notification screen 401 pops up on the returning screen 381, as shown in Fig. 25. The return failure notification screen 401 displays a message indicating that the device has failed to return from its own position being lost ("Tracking return has failed."). The return failure notification screen 401 also has an "OK" button 402. When the user operates the "OK" button 402, the image capture is forcibly terminated, and the screen transitions to the image capture standby screen 311 (see Fig. 16).

[0179] Next, a description will be given of the poor connection notification screen 411 displayed on the touch panel display 36 of the photographing device 1. FIG.

[0180] During shooting, the first cable 21 connecting the display / input panel unit 12 and the sensor unit 11 may become disconnected. In this case, a poor connection notification screen 411 (error screen) shown in FIG. 26 pops up on the shooting-in-progress screen 331.

[0181] The poor connection notification screen 411 displays a message stating that the sensor unit 11, including the visible camera 53, cannot be detected, and a message urging the user to check the condition of the sensor unit 11 and the cable connecting the display input panel unit 12 and the sensor unit 11.

[0182] The poor connection notification screen 411 also has a "retry" button 412 and a "cancel" button 413. The user operates the "retry" button 412 after solving the poor connection of the cable connecting the display / input panel unit 12 and the sensor unit 11 of the image capturing device 1. When the poor connection is solved, the screen transitions to the image capturing standby screen 311 (see FIG. 16). When the user operates the "cancel" button 413, the screen transitions from the image capturing mode to the list mode. At this time, the touch panel display 36 of the image capturing device 1 transitions to the pause screen 301 (see FIG. 15). Meanwhile, the display 82 of the control device 2 transitions from the pause screen 221 (see FIG. 13) to the image capturing data list screen 101 (see FIG. 8).

[0183] There may be a case where the second cable 22 connecting the display input panel unit 12 of the image capturing device 1 and the control device 2 becomes disconnected. In this case, the screen will no longer be displayed on the touch panel display 36 of the image capturing device 1. Meanwhile, on the display 82 of the control device 2, a screen (not shown) notifying the user of the poor connection will pop up on the pause screen 221 (see FIG. 13). The screen notifying the user of the poor connection has an "OK" button. When the user operates the "OK" button, the screen will transition to the image capturing data list screen 101 (see FIG. 8).

[0184] Next, a description will be given of the shooting mode end confirmation screen 421 displayed on the touch panel display 36 of the photographing device 1. FIG.

[0185] When the user operates the "End shooting" button 317 on the shooting standby screen 311 shown in FIG. 16, a shooting mode end confirmation screen 421 shown in FIG.

[0186] The shooting mode termination confirmation screen 421 displays a message inquiring the user as to whether or not to terminate the shooting mode and return to the list mode. The shooting mode termination confirmation screen 421 also has an "OK" button 422 and a "Cancel" button 423. When the user operates the "Cancel" button 423, the screen returns to the shooting standby screen 311 (see FIG. 16). When the user operates the "OK" button 422, the shooting mode is terminated and the screen returns to the list mode. At this time, the touch panel display 36 of the photographing device 1 transitions to the pause screen 301 (see FIG. 15). Meanwhile, the display 82 of the control device 2 transitions from the pause screen 221 (see FIG. 13) to the shooting data list screen 101 (see FIG. 8).

[0187] (Second embodiment) Next, a second embodiment will be described. It should be noted that the points not specifically mentioned here are the same as those of the previous embodiment. Fig. 28 is a block diagram showing the schematic configuration of an image capturing device 1 and a control device 2 according to the second embodiment.

[0188] In the first embodiment, the photographing device 1 and the control device 2 are connected by wire, but in this embodiment, the photographing device 1 and the control device 2 are connected wirelessly. Specifically, the photographing device 1 is equipped with a wireless communication unit 78. Furthermore, the control device 2 (information processing device) is equipped with a wireless communication unit 88. The wireless communication unit 78 of the photographing device 1 and the wireless communication unit 88 of the control device 2 communicate wirelessly using an appropriate wireless communication method such as wireless LAN. In the example shown in FIG. 28, an input / output interface 79 is provided to input and output data to and from a sensor unit 11 having the same configuration as in the first embodiment.

[0189] When the photographing device 1 and the control device 2 are wirelessly connected in this manner, the control device 2 can be left stationary in an appropriate location, such as the location to be measured or nearby, during photographing, eliminating the need for the user to carry the control device 2 around.

[0190] In the first and second embodiments, the photographing system is composed of the photographing device 1 held by the user's hand and the control device 2 carried by the user, but the display / input panel unit 12 of the photographing device 1 may be configured to include all or part of the functions of the control device 2. In this case, the process of generating simple point cloud data (simplified point cloud generation process) for the user to check the photographing situation (point cloud generation situation) requires a relatively small load and needs to be performed in real time during photographing, and therefore may be performed by the photographing device 1. In addition, the process of generating regular point cloud data to be acquired as a 3D measurement result (standard point cloud generation process) requires a relatively large load and only needs to be performed after photographing, and therefore may be performed by a separately provided server device with high processing capabilities.

[0191] In addition, in a configuration in which the image capturing device 1 and the control device 2 are wirelessly connected as in the second embodiment, the control device 2 may be configured on-premise or in the cloud. Also, in a configuration in which the image capturing device 1 having some of the functions of the control device 2 is wirelessly connected to a server device, the server device may be configured on-premise or in the cloud.

[0192] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0193] The photographing system, photographing device, and photographing control method of the present invention have the advantage that the photographing device can be held stably even during photographing, and screen operations for giving instructions and making settings related to photographing on the touch panel display can be easily performed. They are useful as a photographing system, photographing device, and photographing control method that include a photographing device that photographs a measurement target location and a processor that controls the photographing device in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location. [Explanation of symbols]

[0194] 1: Imaging device 2: Control device (information processing device) 11: Sensor section 12: Display and input panel 13:Support 21: First Cable 22: Second Cable 31: Main body 32: Telescopic rod section 33: Sensor mounting part 34: Panel mounting part 35: Grip section 36: Touch panel display 41: Joint 42: Fixed knob 43: Joint 44: Fixed knob 51: Sensor unit 52: Sensor cover 53: Visible camera 54: Depth sensor 61: Cover body 62: Filter member 65:ND filter 101: Shooting data list screen 121: Shooting data details screen 231: Connection failure notification screen 241: Connection failure notification screen 311: Shooting standby screen 331: Shooting screen 335: Elapsed time display 336: Shooting time information section 351: Shooting end confirmation screen 361: Shooting end confirmation screen 371: Self-location lost notification screen 381:Returning screen 411: Poor connection notification screen 421: Shooting mode exit confirmation screen

Claims

1. A photographing system comprising a photographing device that photographs a measurement target location and a control device that is connected to the photographing device and performs three-dimensional measurement processing of the measurement target location, The imaging device is a sensor unit including a camera for capturing an image of a measurement target location; a first display unit including a touch panel display that displays a screen used in a photography mode that assists a user in taking photographs and detects screen operations by the user; a support body having a grip portion that is held by a user with one hand and that supports the sensor portion and the first display portion; The control device a three-dimensional measurement processing unit that performs the three-dimensional measurement processing to generate three-dimensional spatial information of a measurement target location based on the photographing data acquired from the photographing device; a second display unit that displays a screen used in a non-photographing mode for performing operations related to management and processing of the photographed data; a display control unit that controls display information of a screen to be displayed on the first display unit and the second display unit, The display control unit, in response to a user's operation to transition from the non-photography mode to the photography mode, displays a pause screen on the second display unit and an operation screen on the first display unit through which the user can at least instruct the start of photography.

2. The display control unit 2. The photographing system according to claim 1, wherein the operation screens displayed on the touch panel display include a photographing standby screen, a photographing in progress screen, and a photographing end confirmation screen, which are sequentially displayed in response to a user's instruction.

3. The display control unit 3. The photographing system according to claim 2, wherein the photographing standby screen includes an operation section for allowing a user to select one of a plurality of photographing modes having different control conditions for the camera.

4. The display control unit The photographing system according to claim 2, characterized in that the photographing-in-progress screen includes a first image display frame that enlarges and displays either the image photographed by the camera or a photographing path image representing the path traveled by the photographing device, a second image display frame that reduces and displays the other image, and an operation unit that switches the images displayed in the first image display frame and the second image display frame.

5. The display control unit: The photography system according to claim 2, characterized in that, when photography is completed, the user is notified that the photography data has been saved and that the photography standby screen has been returned to, and the photography completion confirmation screen is displayed, which includes an operation unit for the user to input confirmation.

6. The display control unit:

3. The imaging system according to claim 2, wherein the imaging screen includes an image that measures the speed at which the user moves the sensor unit and visualizes whether the speed is appropriate.

7. The display control unit: When starting shooting, the shooting screen including a predetermined available shooting time is displayed; 3. The photographing system according to claim 2, wherein the photographing-in-progress screen including the remaining photographing time is displayed when the remaining photographing time, which is calculated by subtracting the elapsed time from the start of photographing from the available photographing time, becomes equal to or less than a predetermined value during photographing.

8. The display control unit: The photographing system according to claim 1, characterized in that when a self-position loss is detected during tracking processing, in which the photographing device loses track of its current position, the system notifies the user of the occurrence of the self-position loss, presents the user with the last position acquisition point, and displays a screen on the touch panel display prompting the user to return to the last position acquisition point.

9. The display control unit: The photographing system according to claim 1, characterized in that a connection failure on the communication path between the camera and the touch panel display and the control device is detected, and a screen notifying the user of the occurrence of the connection failure is displayed on the touch panel display.

Images