Imaging device

Abstract

To enable a user to easily continue capturing images without causing large burdens on the user, even when the imaging range extends over a wide range, or when imaging at a high elevation is needed.SOLUTION: The present invention comprises: a sensor unit 11 including a camera for capturing an image of the location to be measured; a display input panel unit 12 including a touch panel display 36 for displaying a screen for assisting imaging work of a user and detecting screen operation by the user; and a rod-like support body 13 for supporting the sensor unit and the display input panel unit. The support body includes a grip 35 for a user to grasp with one hand, the sensor unit is fixed to an end of the support body on the side opposite the grip, and the display input panel unit is fixed at a position between the grip of the support body and the sensor unit. The support body, especially, includes a telescopic rod 32 that causes the position of the sensor unit relative to the grip to change. The display input panel unit is secured to the support body so as to be able to adjust the bearing in pan and tilt directions.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a photographing device that photographs a measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location.

Background Art

[0002] There is known a three-dimensional measurement technique for generating a point cloud representing a measurement object in a three-dimensional space based on a photographed image of the measurement object. In this three-dimensional measurement, in recent years, the SLAM (Simultaneous Localization And Mapping) method has attracted attention. In the SLAM method, self-position estimation for acquiring the position information of the own vehicle and mapping for generating a point cloud representing the measurement object in a three-dimensional space are performed based on photographed images obtained by holding a photographing device (e.g., an operator) by a moving body and photographing the measurement object from various directions.

[0003] As a technique related to such a SLAM method, conventionally, an image of an arrow representing the moving direction and moving speed of a photographing device (sensor) held by a user, or an image for discriminating between the direction of a measured area and the direction of an unmeasured area is used to present an area with insufficient photographing to the user (see Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the conventional technique, as a photographing device for photographing a measurement target location, a tablet PC (measurement device main body) equipped with a distance sensor and a visible camera is used, and the user holds the photographing device with both hands and walks inside the measurement target location to photograph the measurement target location.

[0006] Even with such a photographing device, there is little problem when the photographing range is narrow and the photographing can be completed in a relatively short time. However, there are cases where the photographing range extends over a wide area. In such cases, since the burden on the user is large, a configuration that can reduce the user's burden is desired. Also, there are cases where high-angle shooting from a position higher than the perspective of a person is required. In such cases, a configuration that allows the user to easily perform high-angle shooting is desired.

[0007] Therefore, the main object of the present invention is to provide a photographing device that allows a user to easily continue photographing without imposing a large burden on the user even when the photographing range extends over a wide area or when high-angle shooting is required.

Means for Solving the Problems

[0008] The photographing device of the present invention is a photographing device that photographs a measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional space information of the measurement target location, and includes a sensor unit including a camera that photographs the measurement target location, a display input panel unit that displays a screen for assisting the user's photographing operation and detects a screen operation by the user, and a rod-shaped support that supports the sensor unit and the display input panel unit. The support has a grip portion that can be held by the user with one hand, the sensor unit is fixed to an end portion of the support on the side opposite to the grip portion, and the display input panel unit is fixed at a position between the grip portion and the sensor unit on the support.

Effects of the Invention

[0009] According to the present invention, by gripping the grip portion with one hand, the photographing device can be stably held with one hand. Also, since the sensor unit and the display input panel unit are arranged separately, the sensor unit can be arranged at a position away from the user's hand while the display input panel unit is arranged at the user's hand for easy viewing. Thereby, even when the photographing range extends over a wide area or when high-angle shooting is required, the user can easily continue photographing without imposing a large burden on the user.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Embodiments for Carrying Out the Invention

[0011] A first invention made to solve the above problems is a photographing device that performs three-dimensional measurement processing for generating three-dimensional spatial information of a measurement target location, the photographing device including a sensor unit that includes a camera for photographing the measurement target location, a display input panel unit that displays a screen for assisting a user's photographing operation and detects a screen operation by the user, and a rod-shaped support body that supports the sensor unit and the display input panel unit, the support body having a grip portion that can be gripped by a user with one hand, the sensor unit being fixed to an end portion of the support body on a side opposite to the grip portion, and the display input panel unit being fixed at a position between the grip portion and the sensor unit on the support body.

[0012] According to this, by gripping the grip portion with one hand, the photographing device can be stably held with one hand. Further, since the sensor unit and the display input panel unit are separately arranged, the sensor unit can be arranged at a position away from the hand while the display input panel unit is arranged at hand so that it is easy for the user to view. Thereby, even when the photographing range is wide or high-place photographing is required, the user can easily continue photographing without imposing a large burden on the user.

[0013] A second invention is configured such that the support body has a telescopic rod portion that changes the position of the sensor unit with respect to the grip portion.

[0014] According to this, the user can appropriately adjust the position of the sensor unit with respect to the grip portion according to the situation during photographing. Further, since the overall length of the photographing device becomes shorter by retracting the telescopic rod portion, it becomes easy to carry and store the photographing device when not in use.

[0015] A third invention is further configured to include a cable that connects the sensor unit and the display input panel unit.

[0016] According to this, since the sensor unit and the display input panel unit are connected via a cable, the sensor unit and the display input panel unit can be arranged separately.

[0017] Further, in the fourth invention, the display input panel unit is configured to be fixed to the support so that the orientation can be adjusted in at least one of the pan and tilt directions.

[0018] According to this, since the user can appropriately adjust the orientation of the display input panel unit, browsing and operating the screen of the touch panel display become easy.

[0019] Further, in the fifth invention, the display input panel unit includes a first connection terminal to which a first cable connecting the sensor unit and the display input panel unit is detachably coupled, and a second connection terminal to which a second cable connecting a control device for controlling the own device and the display input panel unit is detachably coupled.

[0020] According to this, the handling of the imaging device becomes easy. In this case, the display input panel unit may include a repeater that relays data communication between the control device and the sensor unit, and the second cable may be used for data communication between the control device and the display input unit and for data communication between the control device and the sensor unit.

[0021] Further, the sixth invention further includes a wireless communication unit that performs wireless communication with an information processing device that generates display information for displaying a screen for assisting the user's imaging operation on the touch panel display. The wireless communication unit transmits imaging information including the captured image by the camera and receives the display information from the information processing device.

[0022] According to this, the information processing device is placed and used at an appropriate location. Therefore, the trouble of always carrying the information processing device on the user's shoulder during imaging as in the case where the imaging device is wired-connected to the information processing device is eliminated.

[0023] Further, in the seventh invention, the camera performs color photography, and the sensor unit has a dimming filter that is movably provided between a position covering the lens of the camera and a position not covering the lens of the camera.

[0024] According to this, by suppressing the amount of light input to the camera, it is possible to adjust the brightness of the entire captured image. In addition, it is possible to avoid the problem that coloring occurs in the entire captured image in a bright environment due to the characteristics of the camera.

[0025] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0026] (First Embodiment) FIG. 1 is an explanatory diagram showing the situation of a photographing operation performed by a user using a photographing system according to the first embodiment. FIG. 2 is an explanatory diagram showing a standard photographing state and an elevated photographing state.

[0027] As shown in FIG. 1, the photographing system includes a photographing device 1 and a control device 2 (information processing device).

[0028] The photographing device 1 includes a sensor unit 11, a display input panel unit 12, and a rod-shaped support 13. The sensor unit 11 and the display input panel unit 12 of the photographing device 1 are connected via a first cable 21. The display input panel unit 12 of the photographing device 1 and the control device 2 are connected via a second cable 22.

[0029] The control device 2 is configured as a laptop-type or tablet-type PC that can be carried by a user (operator). In the example shown in FIG. 1, the control device 2 is housed in a shoulder bag and can be carried by the user, but the mode in which the user carries the control device 2 is not limited to this.

[0030] The user holds the photographing device 1 by hand and performs a photographing operation of causing the photographing device 1 to photograph the measurement target location while walking around the measurement target location. At this time, the user can change the position (height) of the sensor unit 11 by moving the arm holding the photographing device 1.

[0031] For example, as shown in FIG. 2(A), the user can perform shooting by slightly extending the arm holding the imaging device 1 to position the sensor unit 11 at approximately the same height as the user's line of sight (standard shooting state). On the other hand, as shown in FIG. 2(B), the user can perform shooting by raising the arm holding the imaging device 1 to position the sensor unit 11 at a position higher than the user's line of sight (high-place shooting state).

[0032] Next, the support 13 of the imaging device 1 will be described. FIG. 3 is a perspective view showing the support 13. FIG. 4 is a side view showing the expansion and contraction state of the support 13.

[0033] As shown in FIG. 3, the support 13 of the imaging device 1 includes a main body portion 31, a telescopic rod portion 32, a sensor mounting portion 33, a panel mounting portion 34, and a grip portion 35. The display input panel portion 12 includes a touch panel display 36.

[0034] The panel mounting portion 34 is provided on the main body portion 31. The panel mounting portion 34 includes a joint 41 (ball joint) that connects the display input panel portion 12 and the main body portion 31. By this joint 41, the orientation of the display input panel portion 12, that is, the orientation of the display surface of the touch panel display 36, can be changed. Specifically, by rotating the display input panel portion 12 about two axes A1 and A2 passing through the center of the joint 41 (ball joint), a tilt operation for changing the orientation of the display input panel portion 12 vertically and a pan operation for changing the orientation of the display input panel portion 12 horizontally can be performed on the display input panel portion 12. Further, a fixing knob 42 is provided on the panel mounting portion 34. By the user operating the fixing knob 42, the orientation of the display input panel portion 12 can be fixed. In the example shown in FIG. 3, since the joint 41 is a ball joint, the orientation of the display input panel portion 12 can be adjusted in an arbitrary direction within a predetermined angle range.

[0035] In this way, since the user can appropriately adjust the orientation of the display input panel unit 12, the visibility and operability of the touch panel display 36 are improved. For example, when changing the angle of the rod-shaped support 13 in the case of high-altitude shooting, etc., if the orientation of the display input panel unit 12 is adjusted in the tilt direction, browsing and operating the screen of the touch panel display 36 become easier. Also, for example, when the user holds the imaging device 1 with the right hand, if the orientation of the display input panel unit 12 is adjusted in the pan direction so that the screen of the touch panel display 36 faces the user's face and the left side where the left hand for screen operation is located, browsing and operating the screen of the touch panel display 36 become easier.

[0036] The sensor attachment portion 33 is provided at the tip of the telescopic rod portion 32. The sensor attachment portion 33 includes a joint 43 (hinge joint) that connects the sensor portion 11 and the tip of the telescopic rod portion 32. By this joint 43, the orientation of the sensor portion 11 can be changed. Specifically, by rotating the sensor portion 11 about the central axis A3 of the joint 43 (hinge joint), a tilt operation for changing the orientation of the sensor portion 11 up and down can be performed on the sensor portion 11. A fixing knob 44 is provided on the sensor attachment portion 33. By the user operating the fixing knob 44, the orientation of the sensor portion 11 can be fixed.

[0037] In the example shown in FIG. 3, in the panel attachment portion 34, the tilt operation and pan operation of the display input panel unit 12 are possible by the joint 41 constituted by a ball joint, but the tilt operation and pan operation of the display input panel unit 12 may be made possible by a combination of a plurality of hinge joints.

[0038] Also, in the example shown in FIG. 3, in the sensor attachment portion 33, only the tilt operation of changing the orientation of the sensor unit 11 up and down is possible by the joint 43 configured by the hinge joint, but the pan operation of changing the orientation of the sensor unit 11 left and right may be enabled. In this case, the joint 43 may be configured by a ball joint or a combination of a plurality of hinge joints.

[0039] The telescopic rod portion 32 is configured to be telescopically extendable in the axial direction. In the example shown in FIG. 4, the telescopic rod portion 32 has a structure in which a plurality of sheath tubes with gradually decreasing diameters are slidably connected.

[0040] The user can change the position of the sensor unit 11 with respect to the grip portion 35 held by hand by extending and retracting the telescopic rod portion 32. That is, as shown in FIG. 4(A), in the state where the telescopic rod portion 32 is extended (extended state), the sensor unit 11 is in a state of being separated from the grip portion 35. For this reason, in high-place shooting (see FIG. 2(B)), the sensor unit 11 can be arranged at a high position. On the other hand, as shown in FIG. 4(B), in the state where the telescopic rod portion 32 is contracted and most of the telescopic rod portion 32 is housed in the main body portion 31 (stored state), the overall length of the imaging device 1 becomes shorter, so that the imaging device 1 can be easily carried and stored when not in use.

[0041] As described above, in this embodiment, the user can stably hold the imaging device 1 with one hand by gripping the grip portion 35 with the hand. For this reason, the user can perform shooting with the sensor unit 11 oriented in an arbitrary direction by changing the orientation of the arm or body holding the imaging device 1 during shooting. Also, the user can adjust the position of the sensor unit 11 by extending and retracting the telescopic rod and can also adjust the orientation of the display input panel portion 12 even during shooting.

[0042] In this embodiment, the display input panel unit 12 is fixed to the main body unit 31. However, the display input panel unit 12 may be fixed in the middle of the telescopic rod unit 32 so that the position of the display input panel unit 12 with respect to the grip unit 35 can be adjusted. Further, the display input panel unit 12 may be fixed to the main body unit 31 via a movable member (not shown) different from the telescopic rod unit 32 to which the sensor unit 11 is fixed at the tip.

[0043] Next, the sensor unit 11 of the imaging device 1 will be described. FIG. 5 is a perspective view showing the sensor unit 11.

[0044] The sensor unit 11 includes a sensor unit 51 and a sensor cover 52.

[0045] The sensor unit 51 includes a visible camera 53 and a depth sensor 54. The visible camera 53 (color camera) captures a color image of the subject. The depth sensor 54 includes an infrared projector 55 and left and right infrared cameras 56 and 57 (stereo cameras). The infrared projector 55 irradiates the subject with infrared light. The left and right infrared cameras 56 and 57 detect the reflected light of the infrared light irradiated on the subject. Note that, in addition to the stereo camera, a sensor capable of acquiring depth information by another method such as LiDAR (Light Detection And Ranging) may be used for the depth sensor 54.

[0046] In the sensor unit 51, the visible camera 53, the left and right infrared cameras 56 and 57, and the infrared projector 55 are arranged side by side in the horizontal direction (substantially horizontal direction). The housing of the sensor unit 51 has a horizontally long shape.

[0047] The sensor cover 52 has a cover body 61 and a filter member 62.

[0048] The cover body 61 covers the sensor unit 51. During shooting, when the imaging device 1 is being moved, the sensor unit 11 may collide with surrounding objects, and the cover body 61 is provided to protect the sensor unit 51 from the impact of such collisions. The cover body 61 has an opening 66 that exposes the cover glass on the front side of the visible camera 53, the infrared projector 55, and the infrared cameras 56 and 57.

[0049] The filter member 62 is slidably attached to the cover body 61 in the longitudinal direction of the sensor unit 51, that is, in the arrangement direction of the visible camera 53, the infrared projector 55, and the infrared cameras 56 and 57. The filter member 62 includes a first ring portion 63 and a second ring portion 64. An ND (Neutral Density) filter 65 (attenuating filter) is attached to the first ring portion 63.

[0050] The filter member 62 is slidably provided between the first position shown in FIG. 5(A) and the second position shown in FIG. 5(B). The user can grip and slide the filter member 62 with a finger. In the first position shown in FIG. 5(A), the ND filter 65 is in a state of not covering the visible camera 53. In the second position shown in FIG. 5(B), the ND filter 65 is in a state of covering the visible camera 53.

[0051] Here, in the first position shown in FIG. 5(A), the first ring portion 63 to which the ND filter 65 is attached is disposed at a position between the visible camera 53 and one of the infrared cameras 57, and the second ring portion 64 is disposed at a position surrounding the lens of one of the infrared cameras 57. As a result, none of the visible camera 53, the infrared projector 55, and the infrared cameras 56 and 57 are covered by the filter member 62.

[0052] On the other hand, at the second position shown in FIG. 5(B), the first ring portion 63 is disposed at a position corresponding to the visible camera 53, and the second ring portion 64 is disposed at a position between the visible camera 53 and one of the infrared cameras 57. As a result, only the visible camera 53 is covered by the ND filter 65, and neither the infrared projector 55 nor the infrared cameras 56 and 57 are covered by the filter member 62.

[0053] Note that when the second ring portion 64 abuts against the inner peripheral portion of the opening of the cover body 61, the filter member 62 is positioned at the first position shown in FIG. 5(A). Further, a groove (not shown) is formed on the lower surface side of the cover body 61 so as to be long in the moving direction of the filter member 62. By fitting the protrusion provided on the filter member 62 into this groove, the moving range of the filter member 62 is defined, and the filter member 62 is positioned at the second position shown in FIG. 5(B).

[0054] In the case of outdoor photography, the user adjusts the filter member 62 to the second position (see FIG. 5(B)) where the ND filter 65 covers the visible camera 53. On the other hand, in the case of indoor photography, the user adjusts the filter member 62 to the first position (see FIG. 5(A)) where the ND filter 65 does not cover the visible camera 53. Thereby, the brightness of the entire captured image can be adjusted by suppressing the amount of light input to the camera in the case of outdoor photography. In addition, it is possible to avoid a problem in which purple coloring occurs in the entire captured image. Further, since only the visible camera 53 (color camera) is covered with the ND filter 65, it is possible to avoid a problem in which the density of the generated point cloud decreases due to the infrared cameras 56 and 57 being covered with the ND filter 65.

[0055] Next, the schematic configurations of the imaging device 1 and the control device 2 will be described. FIG. 6 is a block diagram showing the schematic configurations of the imaging device 1 and the control device 2. FIG. 7 is a block diagram showing an outline of the processing performed by the processor 86 of the control device 2.

[0056] As shown in FIG. 6, the sensor unit 11 of the imaging device 1 includes a visible camera 53, a depth sensor 54, an IMU 71 (Inertial Measurement Unit), an input / output interface 72, and a connection terminal 73.

[0057] The visible camera 53 (color camera) performs color imaging and outputs a color captured image.

[0058] The depth sensor 54 outputs depth information (distance image) as a detection result based on the captured images by the left and right infrared cameras 56, 57 (see FIG. 5).

[0059] The IMU 71 detects the motion state of the device itself, specifically, the three-dimensional angular velocity and acceleration. Based on the detection results of the IMU 71, the position, orientation, and velocity of the sensor unit 11 can be detected.

[0060] The input / output interface 72 performs data input / output with the control device 2 via the display input panel unit 12. Specifically, the detection data of the visible camera 53, the depth sensor 54, and the IMU 71 are transmitted. The input / output interface 72 may be based on the USB (registered trademark) standard.

[0061] A first cable 21 that connects the sensor unit 11 and the display input panel unit 12 is detachably coupled to the connection terminal 73.

[0062] Note that the visible camera 53, the depth sensor 54, and the IMU 71 may not be integrated as a sensor unit 51 (see FIG. 5). Also, a configuration in which the depth sensor 54 and the IMU 71 are omitted and only the visible camera 53 is provided may be used. Further, a configuration in which either the depth sensor 54 or the IMU 71 and the visible camera 53 are provided may be used.

[0063] The display input panel unit 12 of the imaging device 1 includes a touch panel display 36, a repeater 74, an input / output interface 75, a first connection terminal 76, and a second connection terminal 77.

[0064] Based on the control by the control device 2, the touch panel display 36 displays a screen or the like that supports the user's imaging operation.

[0065] The repeater 74 relays data communication between the control device 2 and the sensor unit 11. The repeater 74 may be a hub based on the USB (registered trademark) standard.

[0066] The input / output interface 75 performs data input / output with the control device 2. Specifically, it receives display information such as a screen that supports the user's imaging operation from the control device 2. The input / output interface 75 may be based on the USB (registered trademark) standard.

[0067] A first cable 21 that connects the sensor unit 11 and the display input panel unit 12 is detachably coupled to the first connection terminal 76. A second cable 22 that connects the display input panel unit 12 and the control device 2 is detachably coupled to the second connection terminal 87.

[0068] By the way, the first cable 21 is used for data communication between the control device 2 and the sensor unit 11 via the display input panel unit 12. In this data communication, for example, imaging data (imaging information) output from the sensor unit 11 is transmitted to the control device 2. The imaging data includes a captured image of the visible camera 53, a detection result (distance information) of the depth sensor 54, a detection result of the IMU 71, and the imaging time. The second cable 22 is used for data communication between the control device 2 and the display input panel unit 12 and for data communication between the control device 2 and the sensor unit 11. In the data communication between the control device 2 and the display input panel unit 12, for example, display information of the screen to be displayed on the touch panel display 36 is transmitted from the control device 2, and operation information of the user detected by the touch panel display 36 is transmitted to the control device 2.

[0069] The control device 2 includes an input / output interface 81, a display 82, an input device 83, a memory 84, a storage device 85, a processor 86, and a connection terminal 87.

[0070] The input / output interface 81 performs data input / output with the imaging device 1. The input / output interface 81 may be based on the USB (registered trademark) standard.

[0071] The display 82 displays a screen related to the management of captured data acquired in the past, a screen related to settings such as the operating conditions of the imaging device 1, and the like.

[0072] The input device 83 is for the user to perform input operations. The input device 83 may be a keyboard, a mouse, a touch pad, a touch panel, or the like. When the control device 2 is configured as a tablet PC, a touch panel display 36 in which the touch panel as the input device 83 and the display panel as the display 82 are integrated is provided.

[0073] A second cable 22 for connecting the control device 2 and the display input panel unit 12 of the imaging device 1 is detachably coupled to the connection terminal 87.

[0074] The memory 84 stores programs executed by the processor 86 and the like.

[0075] The storage device 85 stores captured data (imaging information) acquired from the imaging device 1. The captured data includes a captured image of the visible camera 53, a detection result (distance information) of the depth sensor 54, a detection result of the IMU 71, and the imaging time. Further, the storage device 85 stores point cloud data as a three-dimensional measurement result generated by the processor 86.

[0076] The processor 86 performs various processes by executing the programs stored in the memory 84. In the present embodiment, the processor 86 performs three-dimensional measurement processing P1.

[0077] In the three-dimensional measurement process P1, the processor 16 generates point cloud data (environmental map) as three-dimensional space information regarding the measurement target location by using the SLAM (Simultaneous Localization And Mapping) method based on a captured image by the visible camera 53 or the like. Also, in the three-dimensional measurement process P1, self-position estimation is performed in conjunction with the generation of the point cloud data, and the self-position at each time, that is, the position of the shooting point, is acquired.

[0078] As shown in FIG. 7, the three-dimensional measurement process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0079] In the feature extraction process P11, the processor 86 extracts feature information (feature points, etc.) from the captured image (frame) of the visible camera 53.

[0080] In the tracking process P12, the processor 86 compares the feature points extracted this time with the feature points extracted previously, estimates the transition amount regarding the position and orientation of the imaging device 1, and updates the position and orientation trajectory data based on the transition amount. The position and orientation trajectory data is the result of tracking the position and orientation of the imaging device 1 and includes the tracking result of the position and orientation at the time of shooting each captured image (frame), that is, information regarding the position and orientation of the imaging device 1 at each time of shooting.

[0081] In the position and orientation correction process P13, the processor 86 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, for example, the position and orientation trajectory data is corrected to supplement the measurement results in locations with few features such as walls and ceilings.

[0082] In the point cloud generation process P14, the processor 86 generates point cloud data based on the distance information of the depth sensor 54 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13. Further, in the point cloud generation process P14, there are a process of generating normal point cloud data acquired as a three-dimensional measurement result (standard point cloud generation process) and a process of generating simple point cloud data for the user to confirm the shooting situation (point cloud generation situation) (simple point cloud generation process). The simple point cloud generation process needs to be performed in real time during shooting, but the standard point cloud generation process may be performed after shooting.

[0083] Also, as shown in FIG. 6, the processor 86 performs a drawing generation process P2, a display information generation process P3, and a display process P4.

[0084] In the drawing generation process P2, the processor 86 generates a three-dimensional model of the target location based on the point cloud data of the target location generated by the three-dimensional measurement process P1, and generates a layout diagram of the target location based on the three-dimensional model of the target location. At this time, a two-dimensional layout diagram (plan view) of the target location is generated by projecting the three-dimensional model onto the horizontal plane. Also, a three-dimensional layout diagram of the target location is generated by projecting the three-dimensional model based on a predetermined line-of-sight direction.

[0085] In the display information generation process P3, the processor 86 generates display information for the screen to be displayed on the display 82 of the control device 2. On the display 82, a screen (see FIGS. 8 to 14) for the user to perform operations such as management of shooting data, instructions for post-shooting processing (point cloud generation process P14, drawing generation process P2), and settings regarding processing conditions, etc. is displayed. Also, the processor 86 generates display information for the screen to be displayed on the touch panel display 36 of the imaging device 1. On the touch panel display 36, a screen (see FIGS. 15 to 19, FIGS. 21 to 27) for assisting the user's shooting operation is displayed.

[0086] In the display process P4, the processor 86 displays a screen (see FIGS. 8 to 14) on the display 82 of the control device 2 based on the display information generated in the display information generation process P3. Further, the processor 86 displays a screen (see FIGS. 15 to 19 and FIGS. 21 to 27) on the touch panel display 36 of the imaging device 1.

[0087] By the way, in the present embodiment, a shooting standby screen 311 (see FIG. 16), a shooting in - progress screen 331 (see FIG. 17), and shooting end confirmation screens 351, 361 (see FIG. 21) are displayed on the touch panel display 36 of the imaging device 1 so as to sequentially transition according to the user's instructions. Thereby, the user can appropriately perform necessary settings and confirmations at each stage before shooting, during shooting, and after shooting.

[0088] Also, in the present embodiment, a non - shooting mode (first mode) and a shooting mode (second mode) are switched according to the user's operation.

[0089] In the non - shooting mode, as an operable screen on the display 82 of the control device 2, a screen (see FIGS. 8 to 14) for the user to manage and process shooting data and give instructions is displayed, and a non - operable standby screen 301 (see FIG. 15) is displayed on the touch panel display 36 of the imaging device 1.

[0090] On the other hand, in the shooting mode, shooting using the imaging device 1 becomes possible, and as an operable screen on the touch panel display 36 of the imaging device 1, a screen (see FIGS. 15 to 19 and FIGS. 21 to 27) that supports the user's shooting operation, for example, a screen (see FIGS. 16 and 17) for giving instructions to start and end shooting is displayed, and a non - operable standby screen 221 (see FIG. 13) is displayed on the display 82 of the control device 2.

[0091] Also, as shown in FIG. 6, the processor 86 performs a connection failure detection process P5, a speed presentation process P6, a shooting time presentation process P7, and a self - position lost return process P8.

[0092] In the connection failure detection process P5, the processor 86 detects a connection failure on the communication path between the sensor unit 51 and the touch panel display 36 and the processor 86, and notifies the user of the occurrence of the connection failure. Specifically, connection failure notification screens 231 and 241 (see FIG. 14) are displayed on the display 82 of the control device 2. Also, a connection failure notification screen 411 (see FIG. 26) is displayed on the touch panel display 36 of the imaging device 1.

[0093] In the speed presentation process P6, the processor 86 measures the speed at which the user moves the sensor unit 11 based on the detection data (acceleration, angular velocity) of the IMU 71, visualizes whether the speed is appropriate, and presents it to the user. Specifically, a speed bar 334 is displayed on the shooting screen 331 (see FIG. 17).

[0094] In the shooting time presentation process P7, the processor 86 presents the preset shootable time to the user at the start of shooting. Also, during shooting, when the remaining shooting time obtained by subtracting the elapsed time from the start of shooting from the shootable time becomes less than or equal to a predetermined time, the processor 86 presents the remaining shooting time to the user. Specifically, on the shooting screen 331 (see FIG. 18), the shootable time is presented at the start of shooting, and then the remaining shooting time is presented during shooting.

[0095] In the self-position lost return process P8, when the processor 86 detects a self-position loss (tracking loss) in which the current position of the imaging device 1 is lost in the tracking process P12, the processor 86 notifies the user of the occurrence of the self-position loss, presents the final position acquisition point to the user, and provides a return guidance to prompt the user to return to the final position acquisition point. Specifically, a self-position loss notification screen 371 (see FIG. 22) is displayed, and then a return screen 381 (see FIG. 23) including a captured image of the final position acquisition point is displayed. Also, when the processor 86 detects that the imaging device 1 has returned to the final position acquisition point, the normal tracking process P12 is restarted.

[0096] Next, the captured data list screen 101 displayed on the display 82 of the control device 2 will be described. FIG. 8 is an explanatory diagram showing the captured data list screen 101.

[0097] When the imaging application is launched in the control device 2, the imaging data list screen 101 is displayed on the display 82.

[0098] The captured data list screen 101 is provided with a list display section 102. In the list display section 102, captured images are listed for each of a plurality of captured data with different measurement locations and capture dates and times. In the example shown in FIG. 8, the captured images for each captured data are arranged and displayed by date. Also, in the list display section 102, the capture date and time is displayed for each captured image. Further, in the list display section 102, when the point cloud generation process P14 has been performed, a check mark 111 indicating that fact is displayed, and when the drawing generation process P2 has been performed, a check mark 112 indicating that fact is displayed.

[0099] When the user selects captured data by operating any of the captured images for each captured data displayed in the list display section 102, the user transitions to a captured data details screen 121 (see FIG. 9) related to the selected captured data. At this time, the user can select the captured data to be deleted or the captured data for which the point cloud generation process P14 and the drawing generation process P2 are to be executed.

[0100] If there is no registered captured data, a message indicating that there is no captured data, for example, the characters "There is no captured data", is displayed in the list display section 102.

[0101] Also, the captured data list screen 101 is provided with a setting button 103. When the user operates the setting button 103, a setting screen 181 (see FIG. 12(A)) is pop-up displayed on the captured data list screen 101.

[0102] In addition, the shooting data list screen 101 is provided with a button 104 for "starting shooting". When the user operates the button 104 for "starting shooting", the mode shifts from the non-shooting mode to the shooting mode. At this time, on the display 82 of the control device 2, the screen transitions to the standby screen 221 (see FIG. 13). On the other hand, on the touch panel display 36 of the imaging device 1, the screen transitions from the standby screen 301 (see FIG. 15) to the shooting standby screen 311 (see FIG. 16).

[0103] Next, the shooting data details screen 121 displayed on the display 82 of the control device 2 will be described. FIG. 9 is an explanatory diagram showing the shooting data details screen 121. FIG. 10 is an explanatory diagram showing a screen that is pop-up displayed on the shooting data details screen 121. FIG. 11 is an explanatory diagram showing the transition status of the main part of the shooting data details screen 121.

[0104] The shooting data details screen 121 is provided with an image display section 122 and a shooting date and time display section 123. In the image display section 122, the captured images (key frames) included in the shooting data are played back in order. In the shooting date and time display section 123, the shooting date and time of the captured image displayed in the image display section 122 is displayed. In the SLAM method, a frame in which a large change appears in the visual field is extracted as a key frame, and the key frame is reflected in the point cloud.

[0105] In addition, the shooting data details screen 121 is provided with a playback operation section 124. The playback operation section 124 is provided with a seek bar 131, a playback stop button 132, a frame forward button 133, and a frame backward button 134. The user can arbitrarily specify the playback start time by operating the slider of the seek bar 131. Also, the user can instruct the playback and stop of the shooting data by operating the playback stop button 132. Further, the user can instruct the frame forward and frame backward of the shooting data by operating the frame forward button 133 and the frame backward button 134.

[0106] In addition, an instruction operation unit 125 is provided on the captured data details screen 121. The instruction operation unit 125 is provided with a button 141 for "point cloud generation", a button 142 for "point cloud display", a button 143 for "drawing generation", and a button 144 for "drawing display".

[0107] When the user operates the button 141 for "point cloud generation", the point cloud generation process P14 (standard point cloud generation process) is started, and the point cloud generation in progress screen 151 shown in FIG. 10(A) is pop-up displayed on the captured data details screen 121. A message indicating that the point cloud generation process P14 is being executed is displayed on the point cloud generation in progress screen 151. In addition, a progress bar 152 is provided on the point cloud generation in progress screen 151. In the progress bar 152, the progress status of the point cloud generation process P14 is visualized. Also, a button 153 for "abort point cloud generation" is provided on the point cloud generation in progress screen 151. When the user operates the button 153 for "abort point cloud generation", the point cloud generation process P14 is aborted and the process returns to the captured data details screen 121 (see FIG. 9). Also, when the point cloud generation process P14 ends, the process returns to the captured data details screen 121.

[0108] Also, on the captured data details screen 121 shown in FIG. 9, when the user operates the button 142 for "point cloud display", the point cloud display application is launched and the point cloud data generated by the point cloud generation process P14 is displayed.

[0109] When the user operates the button 143 for "drawing generation", the drawing generation in progress screen 161 shown in FIG. 10(B) is pop-up displayed on the captured data details screen 121. A message indicating that the drawing generation process P2 is being executed is displayed on the drawing generation in progress screen 161. Also, a button 162 for "abort drawing generation" is provided on the drawing generation in progress screen 161. When the user operates the button 162 for "abort drawing generation", the drawing generation process P2 is aborted and the process returns to the captured data details screen 121 (see FIG. 9). Also, when the drawing generation process P2 ends, the process returns to the captured data details screen 121. Note that an animation indicating that the drawing generation process P2 is being executed may be displayed on the drawing generation in progress screen 161.

[0110] Also, on the captured data details screen 121 shown in FIG. 9, when the user operates the "drawing display" button 144, the drawing display application is launched, and the drawing generated in the drawing generation process P2 is displayed.

[0111] In addition, the captured data details screen 121 is provided with a "delete" button 145. When the user operates the "delete" button 145, a delete confirmation screen 171 shown in FIG. 10(C) pops up and is displayed on the captured data details screen 121. A message for confirming the deletion of the captured data is displayed on the delete confirmation screen 171. Also, the delete confirmation screen 171 is provided with a "delete" button 172 and a "cancel" button 173. When the user operates the "delete" button 172, the captured data is deleted and the process returns to the captured data list screen 101 (see FIG. 8). When the user operates the "cancel" button 173, the process returns to the captured data details screen 121 (see FIG. 9).

[0112] Here, as shown in FIG. 11, in the instruction operation unit 125, the states of the "point cloud generation" button 141, the "point cloud display" button 142, the "drawing generation" button 143, and the "drawing display" button 144 change before and after the point cloud generation process P14 and the drawing generation process P2.

[0113] First, as shown in FIG. 11(A), before the execution of the point cloud generation process P14, only the "point cloud generation" button 141 is operable, and the "point cloud display" button 142, the "drawing generation" button 143, and the "drawing display" button 144 are inoperable. When the point cloud generation process P14 ends, as shown in FIG. 11(B), a check mark 146 indicating that the point cloud generation process P14 has been executed is displayed. Also, in this state, the "point cloud display" button 142 and the "drawing generation" button 143 are operable, and the "drawing display" button 144 is inoperable. When the drawing generation process P2 ends, as shown in FIG. 11(C), a check mark 147 indicating that the drawing generation process P2 has been executed is displayed. Also, in this state, the "drawing display" button 144 is operable.

[0114] Also, as shown in FIG. 9, the shooting data details screen 121 is provided with a button 126 for "returning to the list". When the user operates the button 126 for "returning to the list", the user returns to the shooting data list screen 101 (see FIG. 8).

[0115] Next, the setting screen 181 displayed on the display 82 of the control device 2 will be described. FIG. 12 is an explanatory diagram showing the setting screen 181.

[0116] On the shooting data list screen 101 (see FIG. 8), when the user operates the setting button 103, the setting screen 181 shown in FIG. 12(A) is pop-up displayed on the shooting data list screen 101.

[0117] The setting screen 181 is provided with a save folder setting section 182. In the save folder setting section 182, a folder path representing the location where the setting file storing the setting information is saved is displayed. Also, the save folder setting section 182 is provided with a button 191 for "changing". When the user operates the button 191 for "changing", a screen for specifying a folder (not shown) is pop-up displayed, and here, the user can specify the folder where the setting file is saved.

[0118] Also, the setting screen 181 is provided with a camera setting section 183. The camera setting section 183 is provided with input sections 192, 193, and 194 for each item of the resolution of the visible camera 53, the resolution of the depth sensor 54, and the frame rate of the visible camera 53. When the user operates each of the input sections 192, 193, and 194, a pull-down menu (not shown) is displayed. The user can select the resolution of the visible camera 53, the resolution of the depth sensor 54, and the frame rate of the visible camera 53 in the pull-down menu.

[0119] In addition, the setting screen 181 is provided with a device information display section 184. In the device information display section 184, as information regarding the sensor unit 51, registration information regarding each item of the name (camera name), serial number, and firmware version of the sensor unit 51, and the date and time when the previous calibration process was performed are displayed. Also, the device information display section 184 is provided with a "forced restart" button 195. When the user operates the "forced restart" button 195, the sensor unit 51 is forcibly restarted. Further, the device information display section 184 is provided with an "execute" button 196. When the user operates the "execute" button 196, the device calibration screen 201 shown in FIG. 12(B) is pop-up displayed on the setting screen 181.

[0120] In addition, the setting screen 181 is provided with a "complete" button 185. When the user operates the "complete" button 185, the setting process is executed with the input content of the setting screen 181, and the process returns to the shooting data list screen 101 (see FIG. 8).

[0121] As shown in FIG. 12(B), on the device calibration screen 201, a message prompting the imaging device 1 to be stationary at a stable location for a predetermined time for the calibration process and the remaining time are displayed. Also, on the device calibration screen 201, a progress bar 202 indicating the progress status of the calibration process is displayed. Further, the device calibration screen 201 is provided with a "cancel" button 203. When the user operates the "cancel" button 203, the calibration process is cancelled and the process returns to the setting screen 181 (see FIG. 12(A)). Also, when the calibration process is completed, the process returns to the setting screen 181.

[0122] Next, the standby screen 221 and the connection failure notification screens 231 and 241 displayed on the display 82 of the control device 2 will be described. FIG. 13 is an explanatory diagram showing the standby screen 221. FIG. 14 is an explanatory diagram showing the connection failure notification screens 231 and 241.

[0123] On the shooting data list screen 101 (see Fig. 8), when the user operates the "Start Shooting" button 104, the mode shifts from the non-shooting mode to the shooting mode.

[0124] Here, when the shift to the shooting mode is successful, the system transitions to the pause screen 221 shown in Fig. 13. The pause screen 221 displays a message indicating that the shooting mode has been entered, and a message indicating that the display and operation of the screen on the display 82 of the control device 2 can be enabled by performing an operation to end the shooting mode on the screen displayed on the touch panel display 36 of the imaging device 1.

[0125] On the other hand, when the shift to the shooting mode fails, depending on the cause of the failure, the connection failure notification screens 231, 241 (error screens) shown in Fig. 14 pop up and are displayed on the shooting data list screen 101. Here, when a connection failure occurs in the second cable 22 that connects the display input panel unit 12 of the imaging device 1 and the control device 2, the connection failure notification screen 231 shown in Fig. 14(A) is displayed. On the other hand, when a connection failure occurs in the first cable 21 that connects the display input panel unit 12 of the imaging device 1 and the sensor unit 11, the connection failure notification screen 241 shown in Fig. 14(B) is displayed.

[0126] The connection failure notification screen 231 shown in Fig. 14(A) displays a message indicating that the shooting mode cannot be entered due to a connection failure between the display input panel unit 12 of the imaging device 1 and the control device 2, and a message prompting the user to check the connection failure location. The connection failure notification screen 231 is also provided with a "Retry" button 232 and a "Cancel" button 233. After the user has resolved the connection failure of the second cable 22, the user operates the "Retry" button 232. As a result, the process of shifting to the shooting mode is executed again. On the other hand, when the user operates the "Cancel" button 233, the system returns to the shooting data list screen 101 (see Fig. 8).

[0127] On the connection failure notification screen 241 shown in FIG. 14(B), a message indicating that the shooting mode cannot be entered due to a connection failure between the display input panel unit 12 and the sensor unit 11 of the imaging device 1, and a message prompting the user to check the connection failure location are displayed. Further, on the connection failure notification screen 241, a button 242 for "Retry" and a button 243 for "Cancel" are provided. After the user has eliminated the connection failure of the first cable 21 that connects the display input panel unit 12 and the sensor unit 11 of the imaging device 1, the user operates the button 242 for "Retry". As a result, the process of shifting to the shooting mode is executed again. On the other hand, when the user operates the button 243 for "Cancel", the screen returns to the shooting data list screen 101 (see FIG. 8).

[0128] Next, the standby screen 301 displayed on the touch panel display 36 of the imaging device 1 will be described. FIG. 15 is an explanatory diagram showing the standby screen 301.

[0129] In the non-shooting mode, that is, while the user is performing screen operations by having the control device 2 display the shooting data list screen 101 (see FIG. 8), the shooting data details screen 121 (see FIG. 9), and the setting screen 181 (see FIG. 12) on the display 82, the standby screen 301 is displayed on the touch panel display 36 of the imaging device 1.

[0130] On the standby screen 301, a message indicating that it is in the non-shooting mode, that is, during the display and operation of the shooting data list screen 101 (see FIG. 8) and the like on the display 82 of the control device 2, is displayed. Further, on the standby screen 301, a message indicating that the shooting mode can be entered by performing an operation to start shooting on the screen displayed on the display 82 of the control device 2 is displayed.

[0131] Next, the shooting standby screen 311 displayed on the touch panel display 36 of the imaging device 1 will be described. FIG. 16 is an explanatory diagram showing the shooting standby screen 311.

[0132] As shown in FIG. 16(A), a shooting standby screen 311 is provided with an image display section 312. A real-time captured image by a visible camera 53 is displayed on the image display section 312.

[0133] In addition, the shooting standby screen 311 is provided with a shooting mode specifying section 313. When the user operates the shooting mode specifying section 313, as shown in FIG. 16(B), a pull-down menu 314 is displayed. In the pull-down menu 314, the user can select a plurality of shooting modes with different control conditions (processing conditions for image signal processing) of the visible camera 53. In the example shown in FIG. 16(B), as the shooting mode, the user can select any one of "Indoor" (indoor shooting mode), "Outdoor" (first outdoor shooting mode), and "Outdoor + ND filter" (second outdoor shooting mode). Note that in each shooting mode, for example, the brightness correction tables used in image signal processing (image correction processing) are different.

[0134] In addition, the shooting standby screen 311 is provided with a brightness adjustment bar 315. The user can adjust the brightness of the captured image by operating the slider of the brightness adjustment bar 315. The brightness can be adjusted for each shooting mode.

[0135] In addition, the shooting standby screen 311 is provided with a shooting start button 316. When the user operates the shooting start button 316, shooting is started and the screen transitions to a shooting-in progress screen 331 (see FIG. 17).

[0136] In addition, the shooting standby screen 311 is provided with a button 317 for "ending shooting". When the user operates the button 317 for "ending shooting", the shooting mode ends and the non-shooting mode is entered, and the screen transitions to a standby screen 301 (see FIG. 15).

[0137] Next, the in - shooting screen 331 displayed on the touch - panel display 36 of the imaging device 1 will be described. FIG. 17 is an explanatory diagram showing the in - shooting screen 331. FIG. 18 is an explanatory diagram showing the transition state of the shooting - time guidance part 336 in the in - shooting screen 331. FIG. 19 is an explanatory diagram showing the in - shooting screen 331 in a state where the message window 337 is opened. FIG. 20 is an explanatory diagram showing the content and display timing of the message that appears in the message window 337.

[0138] As shown in FIGS. 17(A) and (B), the in - shooting screen 331 is provided with a main window 332 (first image display frame) and a sub - window 333 (second image display frame). The display magnifications of the main window 332 and the sub - window 333 are different. In the main window 332, the image is displayed enlarged, and in the sub - window 333, the image is displayed reduced.

[0139] Also, either a real - time captured image 341 by the visible camera 53 or a shooting - path image 342 representing the shooting path at the measurement target location is displayed in the main window 332 and the sub - window 333. The shooting - path image 342 is obtained by superimposing a line 344 representing the shooting path on a point - cloud image 343 representing the measurement target location. The point - cloud image is an image (rendering) of each point of the generated point - cloud data as seen from a predetermined viewpoint.

[0140] The example shown in FIG. 17(A) is the case of the standard state. In this case, the captured image 341 is displayed enlarged in the main window 332, and the shooting - path image 342 is displayed reduced in the sub - window 333. On the other hand, the example shown in FIG. 17(B) is the case of the shooting - path enlarged state. In this case, the shooting - path image 342 is displayed enlarged in the main window 332, and the captured image 341 is displayed reduced in the sub - window 333.

[0141] In addition, the switching between the standard state shown in Fig. 17(A) and the enlarged shooting path state shown in Fig. 17(B) can be performed by the user operating the sub-window 333. That is, in the standard state shown in Fig. 17(A), when the sub-window 333 is operated, it transitions to the enlarged shooting path state shown in Fig. 17(B), and the shooting path image 342 is enlarged and displayed in the main window 332. Also, in the enlarged shooting path state shown in Fig. 17(B), when the sub-window 333 is operated, it transitions to the standard state shown in Fig. 17(A), and the shooting image 341 is enlarged and displayed in the main window 332.

[0142] Note that a mesh image may be superimposed on the shooting image 341 within the point cloud generation range. The mesh image can be obtained by converting the generated point cloud data into mesh data and performing an imaging process on the mesh data at the same field of view as the shooting image 341.

[0143] Also, a speed bar 334 is provided on the shooting screen 331. In the speed bar 334, the speed at which the user moves the sensor unit 11 is visualized and displayed. Specifically, the speed is represented by color. For example, the speed bar 334 is drawn with a gradation from green to orange, and the proportion of orange color increases as the speed increases. Therefore, the user can easily confirm whether the speed at which the sensor unit 11 is moved is appropriate, specifically, whether the speed at which the sensor unit 11 is moved is not too fast. Thereby, it is possible to avoid blurring (defocusing) due to the user moving the sensor unit 11 too fast and a decrease in the accuracy of three-dimensional measurement.

[0144] Also, an elapsed time display section 335 is provided on the shooting screen 331. The elapsed time display section 335 displays the elapsed time from the start of shooting to the present.

[0145] Also, as shown in FIG. 18, during shooting, the shooting time guidance section 336 is displayed on the shooting screen 331. While the elapsed time display section 335 is always displayed, the shooting time guidance section 336 is displayed for a predetermined time at a predetermined timing. Note that the shooting time guidance section 336 can be set to be always displayed by a user operation, not limited to display only at a predetermined timing.

[0146] First, as shown in FIG. 18(A), when shooting starts, the shooting time guidance section 336 is displayed for a predetermined time (for example, 5 seconds) from the start time of shooting. In this case, the available shooting time (for example, 15 minutes) is displayed on the shooting time guidance section 336.

[0147] Next, as shown in FIG. 18(B), the shooting time guidance section 336 is displayed for a predetermined time (for example, 5 seconds) at the timing when the remaining shooting time (the time obtained by subtracting the elapsed time from the start of shooting from the available shooting time) becomes equal to or less than a predetermined time. In this case, the remaining shooting time is displayed on the shooting time guidance section 336. Also, the timing for displaying the shooting time guidance section 336 is not limited to once. For example, the shooting time guidance section 336 may be displayed at the timing when the remaining shooting time becomes 3 minutes and at the timing when the remaining shooting time becomes 2 minutes.

[0148] Next, as shown in FIG. 18(C), the shooting time guidance section 336 is continuously displayed until the remaining shooting time runs out at the timing when the remaining shooting time becomes small (for example, 1 minute). In this case, the remaining shooting time is displayed on the shooting time guidance section 336. Also, the shooting time guidance section 336 is displayed in a manner different from the states shown in FIGS. 18(A) and (B). For example, the shooting time guidance section 336 is highlighted by changing the colors of the background and frame of the shooting time guidance section 336.

[0149] Also, as shown in FIG. 19, a message window 337 is displayed on the shooting screen 331. A message to notify the user is displayed in the message window 337. Specifically, messages regarding guidance on shooting operations and various warnings are displayed.

[0150] The example shown in FIG. 19(A) is for an information message. The information message provides the user with information for reference in the user's shooting operation or information to arouse attention. The example shown in FIG. 19(B) is for a warning message. The warning message prompts the user to take actions to improve the shooting operation state that needs to be improved urgently.

[0151] Note that the message window 337 may have different display positions on the shooting screen 331 according to the type of message (information provision, warning, etc.). For example, in the example shown in FIG. 19(B), the message window 337 for the warning message is displayed at the lower part of the shooting screen 331 as in the case of the information message shown in FIG. 19(A), but it may also be displayed at the center of the shooting screen 331 so that the user can immediately check the warning message.

[0152] FIG. 20 shows an example of an information message and a warning message displayed on the message window 337. The information messages include a message indicating that the return from self-position loss was successful, a message indicating that the position correction process was completed when loop closing (circumferential movement) was detected, a message prompting the user to keep the imaging device 1 stationary for the position correction process when loop closing was detected, and the like. The warning messages include a message prompting the user to keep the imaging device 1 stationary when the user moves the imaging device 1 during the execution of the position correction process after loop closing is detected, a message prompting the user to move the imaging device 1 slowly when the speed at which the user moves the imaging device 1 is too fast, a message prompting the user to change the shooting target when the number of feature points extracted from the captured image is small, a message prompting the user to maintain an appropriate distance from the subject when approaching the subject too closely, and the like.

[0153] Next, the shooting end confirmation screens 351 and 361 displayed on the touch panel display 36 of the imaging device 1 will be described. FIG. 21 is an explanatory diagram showing the shooting end confirmation screens 351 and 361.

[0154] As shown in FIG. 17, a shooting end button 339 is provided on the shooting screen 331 during shooting. When the user operates the shooting end button 339, a process of ending the shooting and saving the shooting data is executed, and as shown in FIG. 21(A), a shooting end confirmation screen 351 is popped up and displayed on the shooting screen 331.

[0155] On the shooting end confirmation screen 351, a message indicating that the shooting has ended and the saving of the shooting data has been completed, and a message indicating returning to the shooting standby screen 311 (see FIG. 16) are displayed. Also, an "OK" button 352 is provided on the shooting end confirmation screen 351. When the user operates the "OK" button 352, the screen transitions to the shooting standby screen 311.

[0156] In addition, in this embodiment, when the shooting time (elapsed time since the start of shooting) exceeds a predetermined shootable time, the shooting is forcibly ended and a process of saving the shooting data is executed. In this case, as shown in FIG. 21(B), a shooting end confirmation screen 361 due to the shooting time exceeding is popped up and displayed on the shooting screen 331.

[0157] On the shooting end confirmation screen 361, a message indicating that the shooting has been forcibly ended due to the time exceeding and the saving of the shooting data has been completed, and a message indicating returning to the shooting standby screen 311 (see FIG. 16) are displayed. Also, an "OK" button 362 is provided on the shooting end confirmation screen 361. When the user operates the "OK" button 362, the screen transitions to the shooting standby screen 311.

[0158] Note that in this embodiment, when the shooting time exceeds a predetermined shootable time, the shooting is forcibly ended, but such forced termination may not be performed, or the user may be able to select between a mode in which forced termination is performed and a mode in which forced termination is not performed.

[0159] Next, the screen displayed on the touch panel display 36 of the imaging device 1 when self-position loss occurs will be described. FIG. 22 is an explanatory diagram showing a self-position loss notification screen 371. FIG. 23 is an explanatory diagram showing a screen during return 381. FIG. 24 is an explanatory diagram showing an enlarged screen of the final position acquisition point 391. FIG. 25 is an explanatory diagram showing a return failure notification screen 401.

[0160] During imaging, a self-position loss (tracking loss) may occur in which the self-position is lost in the tracking process P12 (self-position estimation process). In this case, it is necessary to return to the final position acquisition point, that is, the last point where the normal self-position could be estimated, and restart the tracking process P12. Therefore, first, after notifying the user that self-position loss has occurred, the user is assisted to return to the final position acquisition point, and when it is detected that the user has returned to the final position acquisition point, the tracking process P12 is restarted.

[0161] In the present embodiment, when self-position loss is detected, as shown in FIG. 22, a self-position loss notification screen 371 is pop-up displayed on the imaging screen 331 on the touch panel display 36 of the imaging device 1.

[0162] On the self-position loss notification screen 371, a message is displayed notifying the user that self-position loss has occurred and prompting the user to return to the final position acquisition point. In addition, an "OK" button 372 is provided on the self-position loss notification screen 371. When the user operates the "OK" button 372, the screen transitions to the screen during return 381 shown in FIG. 23. At this time, in the processor 86 of the control device 2, a process for detecting that the imaging device 1 has returned to the final position acquisition point is started.

[0163] Here, the example shown in Fig. 23(A) is the same as the example shown in Fig. 17(A) in the standard state. The captured image 341 by the visible camera 53 is enlarged and displayed in the main window 332 (the first image display frame), and the captured path image 342 is reduced and displayed in the sub-window 333 (the second image display frame). Also, the example shown in Fig. 23(B) is the same as the example shown in Fig. 17(B) in the state where the captured path is enlarged. The captured path image 342 is enlarged and displayed in the main window 332, and the captured image 341 is reduced and displayed in the sub-window 333.

[0164] In addition, on the return screen 381, another sub-window 382 (the third image display frame) is provided separately from the sub-window 333 (the second image display frame) where the captured image 341 and the captured path image 342 are reduced and displayed. The captured image of the final position acquisition point is displayed in the sub-window 382. Thereby, the user can easily grasp the final position acquisition point.

[0165] Also, on the return screen 381, a message window 383 is displayed. In the message window 383, similar to the self-position lost notification screen 371 (see Fig. 22), a message is displayed notifying that a self-position loss has occurred and prompting to return to the final position acquisition point.

[0166] When the sub-window 382 where the captured image of the final position acquisition point is displayed is operated, as shown in Fig. 24, the enlarged screen 391 of the final position acquisition point is pop-up displayed on the return screen 381. In the enlarged screen 391 of the final position acquisition point, the captured image of the final position acquisition point is enlarged and displayed. Also, an "×" button 392 is displayed on the enlarged screen 391 of the final position acquisition point. When the user operates the "×" button 392, the enlarged screen 391 of the final position acquisition point is closed and the screen returns to the return screen 381.

[0167] Here, when the return from the self-position loss, that is, when successfully returning to the final position acquisition point, the transition is made to the shooting screen 331 (see FIG. 17). Note that at the timing when it is detected that the return to the final position acquisition point has been made, after a message notifying that the return to the final position acquisition point is displayed on the return screen 381, the transition may be made to the shooting screen 331.

[0168] On the other hand, when the return from the self-position loss cannot be achieved even after a predetermined time has elapsed, as shown in FIG. 25, the return failure notification screen 401 is pop-up displayed on the return screen 381. On the return failure notification screen 401, a message indicating that the return from the self-position loss has failed ("Tracking return has failed.") is displayed. Further, an "OK" button 402 is provided on the return failure notification screen 401. When the user operates the "OK" button 402, the shooting is forcibly terminated and the transition is made to the shooting standby screen 311 (see FIG. 16).

[0169] Next, the connection failure notification screen 411 displayed on the touch panel display 36 of the imaging device 1 will be described. FIG. 26 is an explanatory diagram showing the connection failure notification screen 411.

[0170] During shooting, the first cable 21 connecting the display input panel unit 12 and the sensor unit 11 may become disconnected. In this case, the connection failure notification screen 411 (error screen) shown in FIG. 26 is pop-up displayed on the shooting screen 331.

[0171] On the connection failure notification screen 411, a message indicating that the sensor unit 11 including the visible camera 53 etc. cannot be detected and a message prompting the user to check the state of the sensor unit 11 and the cable connecting the sensor unit 11 and the display input panel unit 12 are displayed.

[0172] In addition, on the connection failure notification screen 411, a "Retry" button 412 and a "Cancel" button 413 are provided. After the user has resolved the connection failure of the cable connecting the display input panel unit 12 and the sensor unit 11 of the imaging device 1, the user operates the "Retry" button 412. When the connection failure is resolved here, the imaging standby screen 311 (see FIG. 16) is transitioned to. Also, when the user operates the "Cancel" button 413, the mode shifts from the imaging mode to the list mode. At this time, on the touch panel display 36 of the imaging device 1, the standby screen 301 (see FIG. 15) is transitioned to. On the other hand, on the display 82 of the control device 2, the transition is from the standby screen 221 (see FIG. 13) to the imaging data list screen 101 (see FIG. 8).

[0173] Note that the second cable 22 connecting the display input panel unit 12 of the imaging device 1 and the control device 2 may become disconnected. In this case, no screen is displayed on the touch panel display 36 of the imaging device 1. On the other hand, on the display 82 of the control device 2, a screen (not shown) for notifying the user of the connection failure is pop-up displayed on the standby screen 221 (see FIG. 13). An "OK" button is provided on the screen for notifying the user of the connection failure. When the user operates the "OK" button, the imaging data list screen 101 (see FIG. 8) is transitioned to.

[0174] Next, the imaging mode end confirmation screen 421 displayed on the touch panel display 36 of the imaging device 1 will be described. FIG. 27 is an explanatory diagram showing the imaging mode end confirmation screen 421.

[0175] On the imaging standby screen 311 shown in FIG. 16, when the user operates the "End Imaging" button 317, the imaging mode end confirmation screen 421 shown in FIG. 27 is pop-up displayed on the imaging standby screen 311.

[0176] On the shooting mode termination confirmation screen 421, a message is displayed to inquire the user whether it is possible to terminate the shooting mode and return to the list mode. Also, on the shooting mode termination confirmation screen 421, an "OK" button 422 and a "Cancel" button 423 are provided. When the user operates the "Cancel" button 423, the screen returns to the shooting standby screen 311 (see FIG. 16). Also, when the user operates the "OK" button 422, the shooting mode is terminated and the screen returns to the list mode. At this time, on the touch panel display 36 of the imaging device 1, the screen transitions to the standby screen 301 (see FIG. 15). On the other hand, on the display 82 of the control device 2, the screen transitions from the standby screen 221 (see FIG. 13) to the shooting data list screen 101 (see FIG. 8).

[0177] (Second Embodiment) Next, the second embodiment will be described. Note that points not particularly mentioned here are the same as those in the above-described embodiment. FIG. 28 is a block diagram showing a schematic configuration of the imaging device 1 and the control device 2 according to the second embodiment.

[0178] In the first embodiment, the imaging device 1 and the control device 2 are connected by wire, but in this embodiment, the imaging device 1 and the control device 2 are connected wirelessly. Specifically, the imaging device 1 includes a wireless communication unit 78. Also, the control device 2 (information processing device) includes a wireless communication unit 88. The wireless communication unit 78 of the imaging device 1 and the wireless communication unit 88 of the control device 2 perform wireless communication using an appropriate wireless communication method such as a wireless LAN. In the example shown in FIG. 28, an input / output interface 79 is provided to input and output data to and from a sensor unit 11 having the same configuration as that in the first embodiment.

[0179] When the imaging device 1 and the control device 2 are wirelessly connected in this way, during shooting, the control device 2 may be placed at an appropriate location, for example, at or near the measurement target location, and the user does not have to carry the control device 2 around.

[0180] Incidentally, in the first and second embodiments, the imaging system includes the imaging device 1 held by the user's hand and the control device 2 carried by the user. However, the configuration may be such that all or part of the functions of the control device 2 are included in the display input panel unit 12 of the imaging device 1. In this case, the process of generating simple point cloud data (simple point cloud generation process) for the user to confirm the imaging situation (point cloud generation situation) may be performed by the imaging device 1 because the load is relatively small and it needs to be performed in real time during imaging. Also, the process of generating regular point cloud data (standard point cloud generation process) to be acquired as the three-dimensional measurement result may be performed by a separately provided server device with high processing power because the load is relatively large and it may be performed after imaging.

[0181] Also, in the configuration where the imaging device 1 and the control device 2 are wirelessly connected as in the second embodiment, the control device 2 may be configured on-premises or in the cloud. Also, in the configuration where the imaging device 1 having a part of the functions of the control device 2 and the server device are wirelessly connected, the server device may be configured on-premises or in the cloud.

[0182] As described above, the embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited thereto, and can also be applied to embodiments with changes, replacements, additions, omissions, etc. Also, it is possible to form a new embodiment by combining the respective components described in the above embodiments.

Industrial Applicability

[0183] The imaging device according to the present invention has the effect that the user can easily continue imaging without imposing a large burden on the user even when the imaging range is wide or high-altitude imaging is required, and is useful as an imaging device for imaging a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location.

Explanation of Reference Numerals

[0184] 1: Imaging device 2: Control device (information processing device) 11: Sensor unit 12: Display input panel unit 13: Support 21: First cable 22: Second cable 31: Main body unit 32: Telescopic rod unit 33: Sensor mounting part 34: Panel mounting part 35: Grip part 36: Touch panel display 41: Joint 42: Fixed knob 43: Joint 44: Fixed knob 51: Sensor unit 52: Sensor cover 53: Visible camera 54: Depth sensor 61: Cover body 62: Filter member 65: ND filter 101: Shooting data list screen 121: Shooting data detail screen 231: Connection failure notification screen 241: Connection failure notification screen 311: Shooting standby screen 331: Shooting in progress screen 335: Elapsed time display part 336: Shooting time guidance part 351: Shooting end confirmation screen 361: Shooting end confirmation screen 371: Self-position lost notification screen 381: Returning screen 411: Connection failure notification screen 421: Shooting mode end confirmation screen

Claims

1. An imaging device for performing three-dimensional measurement processing to generate three-dimensional space information of a measurement target location, comprising: a sensor unit including a camera for imaging the measurement target location; a display input panel unit including a touch panel display for displaying a screen to assist the user's imaging operation and detecting the user's screen operations; a rod-shaped support for supporting the sensor unit and the display input panel unit; and the support has a grip portion that can be held by the user with one hand; the sensor unit is fixed to an end portion of the support on the side opposite to the grip portion; the display input panel unit is fixed at a position between the grip portion and the sensor unit on the support. An imaging device characterized by this.

2. The imaging device according to claim 1, wherein the support has a telescopic rod portion for changing the position of the sensor unit relative to the grip portion.

3. The imaging device according to claim 1, further comprising a cable connecting the sensor unit and the display input panel unit.

4. The imaging device according to claim 1, wherein the display input panel unit is fixed to the support so that its orientation can be adjusted in at least one of the pan and tilt directions.

5. The imaging device according to claim 1, wherein the display input panel unit includes a first connection terminal to which a first cable connecting the sensor unit and the display input panel unit is detachably coupled, and a second connection terminal to which a second cable connecting a control device for controlling the device itself and the display input panel unit is detachably coupled.

6. The imaging device according to claim 1, further comprising a wireless communication unit for performing wireless communication with an information processing device that generates display information for displaying a screen to assist the user's imaging operation on the touch panel display; the wireless communication unit is characterized by transmitting imaging information including an imaging image captured by the camera and receiving the display information from the information processing device.

7. The camera performs color imaging; the sensor unit of the imaging device according to claim 1 is provided with a neutral density filter that is movable between a position covering the lens of the camera and a position not covering the lens of the camera.

Images