Imaging system, imaging device, and imaging control method

Abstract

[Problem] To enable a user to easily continue imaging without imposing a significant burden on the user, even if the imaging range is extensive, elevated imaging is necessary, or the like. [Solution] This invention comprises: a sensor unit (11) comprising a camera for imaging a location to be measured; a display input panel unit 12 including a touch panel display 36 for displaying a screen for assisting the user in imaging work and also detecting a screen operation by the user; and a rod-shaped support body 13 for supporting the sensor unit and the display input panel unit. The support body comprises a grip section 35 that the user grips with one hand; the sensor unit is fixed to an end of the support body on the side opposite the grip section; and the display input panel unit is fixed to a location on the support body between the grip section and the sensor unit. In particular, the support body comprises a telescopic rod section 32 that changes the position of the sensor unit with respect to the grip section. In addition, the display input panel unit is fixed to the support body so that the orientation can be adjusted in the pan and tilt directions.

Description

Photographing system, photographing device, and photographing control method

[0001] The present disclosure relates to an imaging device that captures an image of a measurement target location in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location.

[0002] A three-dimensional measurement technique is known in which a point cloud representing a measurement target in three-dimensional space is generated based on captured images of the measurement target. In recent years, the SLAM (Simultaneous Localization and Mapping) method has been attracting attention for this three-dimensional measurement technique. The SLAM method performs self-localization, which acquires the position information of a mobile object (e.g., a worker) based on captured images of the measurement target from various directions, and mapping, which generates a point cloud representing the measurement target in three-dimensional space.

[0003] A known technology related to this SLAM method is to present to the user areas where photography is insufficient by using an image of an arrow indicating the direction and speed of movement of the imaging device (sensor) held by the user, or an image that distinguishes between the direction of measured areas and the direction of unmeasured areas (see Patent Document 1).

[0004] Patent No. 6489566

[0005] According to conventional technology, a tablet PC (the measurement device itself) equipped with a distance sensor and a visible camera is used as an imaging device to photograph the measurement target location, and the user holds the imaging device in both hands and walks within the measurement target location to photograph the measurement target location.

[0006] Even with such an image capture device, there is little problem if the capture range is narrow and capture can be completed in a relatively short time. However, there are cases where the capture range is wide, and in such cases, the burden on the user is large, so a configuration that can reduce the burden on the user is desired. Also, there are cases where high-altitude capture is required, where capture is taken from a position higher than the person's viewpoint, and in such cases, a configuration that allows the user to easily perform high-altitude capture is desired.

[0007] Therefore, the main objective of the present disclosure is to provide a photographing device that allows a user to easily continue photographing without imposing a heavy burden on the user, even when the photographing range is wide or when photographing from a high altitude is required.

[0008] The imaging device disclosed herein is an imaging device that images a location to be measured in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the location to be measured, and is equipped with a sensor unit including a camera that images the location to be measured, a display input panel unit that displays a screen to assist the user in their imaging work and includes a touch panel display that detects screen operations by the user, and a rod-shaped support body that supports the sensor unit and the display input panel unit, the support body having a grip portion that the user holds in one hand, the sensor portion fixed to the end of the support body opposite the grip portion, and the display input panel portion fixed to a position on the support body between the grip portion and the sensor portion.

[0009] According to the present disclosure, by gripping the grip portion with one hand, the imaging device can be stably held with one hand. Furthermore, because the sensor unit and the display / input panel unit are separately located, the sensor unit can be placed away from the user's hand while the display / input panel unit remains close at hand for easy viewing. This allows the user to easily continue imaging without imposing a significant burden on the user, even when the imaging range is wide or when imaging from a high altitude is required.

[0010] 1 is a block diagram showing the schematic configuration of the photographing device and a control device; 2 is a block diagram showing an overview of the processing performed by the processor of the control device; 3 is a block diagram showing the photographing data list screen displayed on the display of the control device; 4 is a block diagram showing the photographing data details screen displayed on the display of the control device; 5 is a block diagram showing the transition status of the main parts of the photographing data details screen; 6 is a block diagram showing the setting screen displayed on the display of the control device; 7 is a block diagram showing the pause screen displayed on the display of the control device; 8 is a block diagram showing the transition status of the main parts of the photographing data details screen; 9 is a block diagram showing the setting screen displayed on the display of the control device; 10 is a block diagram showing the pause screen displayed on the display of the control device; FIG. 1 is an explanatory diagram showing a shooting screen displayed on the touch panel display of the device; FIG. 2 is an explanatory diagram showing the transition status of the shooting available time guide section on the shooting screen; FIG. 3 is an explanatory diagram showing a shooting screen with a message window displayed; FIG. 4 is an explanatory diagram showing the content and display timing of the message displayed in the message window; FIG. 5 is an explanatory diagram showing a shooting end confirmation screen displayed on the touch panel display of the shooting device; FIG. 6 is an explanatory diagram showing a self-position lost notification screen displayed on the touch panel display of the shooting device; FIG. 7 is an explanatory diagram showing a recovery screen displayed on the touch panel display of the shooting device;

[0011] The first invention made to solve the above problem is an imaging device that photographs a location to be measured in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured, and is equipped with a sensor unit including a camera that photographs the location to be measured, a display input panel unit that displays a screen to assist the user in their imaging work and includes a touch panel display that detects screen operations by the user, and a rod-shaped support body that supports the sensor unit and the display input panel unit, the support body having a grip portion that the user holds in one hand, the sensor portion being fixed to the end of the support body opposite the grip portion, and the display input panel portion being fixed to a position on the support body between the grip portion and the sensor portion.

[0012] This allows the user to hold the imaging device stably with one hand by grasping the grip with the other. Furthermore, because the sensor unit and the display / input panel unit are separately located, the user can place the sensor unit at a distance from the user's hand while keeping the display / input panel unit close at hand for easy viewing. This allows the user to continue taking images easily without placing a heavy burden on the user, even when the imaging range is wide or when imaging from a high altitude is required.

[0013] In a second aspect of the present invention, the support body has an extendable rod portion that changes the position of the sensor portion relative to the grip portion.

[0014] This allows the user to adjust the position of the sensor unit relative to the grip unit as needed depending on the situation when taking a photograph. In addition, by shortening the telescopic rod unit, the overall length of the photographing device can be shortened, making it easier to carry and store the photographing device when not in use.

[0015] In addition, a third aspect of the present invention is configured to further include a cable connecting the sensor unit and the display / input panel unit.

[0016] According to this, since the sensor unit and the display input panel unit are connected via a cable, the sensor unit and the display input panel unit can be arranged apart from each other.

[0017] In a fourth aspect of the present invention, the display input panel section is fixed to the support body so that its orientation can be adjusted in at least one of pan and tilt directions.

[0018] This allows the user to adjust the orientation of the display / input panel unit as needed, making it easier to view and operate the screen of the touch panel display.

[0019] In addition, in a fifth invention, the display input panel unit is configured to include a first connection terminal to which a first cable connecting the sensor unit and the display input panel unit is detachably connected, and a second connection terminal to which a second cable connecting a control device that controls the device itself and the display input panel unit is detachably connected.

[0020] In this case, the display / input panel unit may include a repeater that relays data communication between the control device and the sensor unit, and the second cable may be used for data communication between the control device and the display / input panel unit and for data communication between the control device and the sensor unit.

[0021] In addition, the sixth invention further includes a wireless communication unit that wirelessly communicates with an information processing device that generates display information for displaying a screen on the touch panel display to assist the user in their photography work, and the wireless communication unit is configured to transmit photography information including images captured by the camera and receive the display information from the information processing device.

[0022] This allows the information processing device to be placed in a suitable location for use, eliminating the need for the user to constantly carry the information processing device, such as by hanging it on their shoulder, while taking pictures, as is the case when the photographing device is connected to the information processing device via a wire.

[0023] In addition, a seventh invention is configured such that the camera takes color photographs, and the sensor unit has a neutral density filter that is movable between a position that covers the lens of the camera and a position that does not cover the lens of the camera.

[0024] This allows the brightness of the entire captured image to be adjusted by reducing the amount of light input to the camera, and also prevents the problem of the entire captured image being colored in a bright environment due to the camera's characteristics.

[0025] The eighth invention made to solve the above problem is a photography system comprising a photography device that photographs the location to be measured and a processor that controls the photography device in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured, wherein the photography device comprises a sensor unit including a camera that photographs the location to be measured, a display input panel unit including a touch panel display that displays a screen to assist the user in their photography work and detects screen operations by the user, and a support body that supports the sensor unit and the display input panel unit, wherein the support body has a grip portion that the user holds in one hand, the display input panel unit is positioned near the grip portion, and the processor is configured to display on the touch panel display an operation screen on which the user can at least instruct the start of photography.

[0026] With this, by holding the grip portion with one hand, the photographing device can be held stably with one hand, and the screen for giving instructions and setting related to photographing on the touch panel display can be easily operated with the other hand. Note that the processor that controls the photographing device may be provided in a control device separate from the photographing device, or may be provided in the photographing device itself.

[0027] In addition, in a ninth invention, the processor is configured to display, as the operation screen, a shooting standby screen, a shooting in progress screen, and a shooting end confirmation screen on the touch panel display in such a manner that the screens transition sequentially in response to user instructions.

[0028] This allows the user to appropriately make the necessary settings and checks at each stage before, during, and after shooting.

[0029] In a tenth aspect of the present invention, the processor displays the shooting standby screen including an operation unit for a user to select one of a plurality of shooting modes having different control conditions for the camera.

[0030] According to this, the user can specify the shooting mode according to the characteristics of the measurement location before starting shooting. In this case, either an indoor shooting mode or an outdoor shooting mode may be selected. Furthermore, either one of two outdoor shooting modes depending on whether a neutral density filter is used or not may be selected.

[0031] In addition, in an eleventh invention, the processor is configured to display the shooting screen, which includes a first image display frame that enlarges and displays either an image captured by the camera or a shooting path image that represents the path traveled by the shooting device, a second image display frame that reduces and displays the other image, and an operation unit that switches the images to be displayed in the first image display frame and the second image display frame.

[0032] This allows the user to switch between an enlarged image and a reduced image as needed. Note that the second image display frame itself may be the operation unit.

[0033] In addition, in a twelfth invention, the processor is configured to notify the user that the shooting data has been saved and that the screen has returned to the shooting standby screen when shooting is completed, and to display the shooting completion confirmation screen including an operation unit for the user to input confirmation.

[0034] This allows the user to easily confirm that the image capture has ended.

[0035] In addition, in a thirteenth aspect of the present invention, the processor is configured to measure the speed at which the user moves the sensor unit and display the shooting screen including an image that visualizes whether the speed is appropriate.

[0036] This allows the user to easily understand whether the speed at which the sensor unit is moved is appropriate.

[0037] In addition, in a fourteenth aspect of the present invention, the processor is configured to display the shooting-in-progress screen including a predetermined available shooting time when shooting starts, and to display the shooting-in-progress screen including the remaining shooting time when the remaining shooting time, calculated by subtracting the elapsed time from the start of shooting from the available shooting time, falls below a predetermined value during shooting.

[0038] This allows the user to know the remaining shooting time when starting shooting, and also allows the user to know the remaining shooting time when the remaining shooting time becomes short.

[0039] In addition, in a fifteenth aspect of the invention, when the processor detects that the current position of the photographing device has been lost during the tracking process, it notifies the user of the occurrence of the loss of its own position, presents the user with the last position acquisition point, and displays a screen on the touch panel display urging the user to return to the last position acquisition point.

[0040] This allows the user to quickly return to the last position acquisition point and start photographing again when the user loses his / her own position.

[0041] In addition, a sixteenth invention is configured such that the processor detects a connection failure on the communication path between the camera and the touch panel display and the processor, and displays a screen on the touch panel display notifying the user of the occurrence of the connection failure.

[0042] This allows the user to easily understand that the imaging device is not operating normally due to a poor connection.

[0043] In addition, a seventeenth invention is an imaging device that photographs a location to be measured in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured, comprising a sensor unit including a camera that photographs the location to be measured, a display input panel unit including a touch panel display that displays a screen to assist the user in their imaging work and detects screen operations by the user, and a support body that supports the sensor unit and the display input panel unit, wherein the support body has a grip portion that the user holds in one hand, the display input panel unit is positioned near the grip portion, and the touch panel display is configured to display an operation screen on which the user at least gives instructions to start imaging.

[0044] As with the eighth invention, this allows the user to hold the photographing device stably with one hand by grasping the grip portion with the other hand, and therefore allows the user to easily use the other hand to operate the screen on the touch panel display to give instructions and make settings regarding photographing.

[0045] In addition, an 18th invention is an imaging control method in which a processor executes a process to control an imaging device that photographs a location to be measured in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured, the method comprising: a sensor unit including a camera that photographs the location to be measured; a display input panel unit including a touch panel display that displays a screen to assist a user in their imaging work and detects screen operations by the user; and a support body that supports the sensor unit and the display input panel unit, the support body having a grip portion that a user holds in one hand, and the display input panel unit being configured to display an operation screen on the touch panel display of the imaging device located near the grip portion, where the user performs at least an operation to instruct the start of imaging.

[0046] As with the eighth invention, this allows the user to hold the photographing device stably with one hand by grasping the grip portion with the other hand, and therefore allows the user to easily use the other hand to operate the screen on the touch panel display to give instructions and make settings regarding photographing.

[0047] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0048] (First embodiment) Fig. 1 is an explanatory diagram showing a situation in which a user performs a photographing operation using a photographing system according to a first embodiment. Fig. 2 is an explanatory diagram showing a standard photographing state and a high-altitude photographing state.

[0049] As shown in FIG. 1, the photographing system includes a photographing device 1 and a control device 2 (information processing device).

[0050] The photographing device 1 includes a sensor unit 11, a display / input panel unit 12, and a rod-shaped support member 13. The sensor unit 11 and the display / input panel unit 12 of the photographing device 1 are connected via a first cable 21. The display / input panel unit 12 of the photographing device 1 and the control device 2 are connected via a second cable 22.

[0051] The control device 2 is configured as a laptop or tablet PC that can be carried by a user (worker). In the example shown in Fig. 1, the control device 2 can be stored in a shoulder bag and carried by the user, but the manner in which the user carries the control device 2 is not limited to this.

[0052] The user performs a photographing operation by walking around a location to be measured while holding the photographing device 1 in his / her hand and causing the photographing device 1 to photograph the location to be measured. At this time, the user can change the position (height) of the sensor unit 11 by moving the arm holding the photographing device 1.

[0053] For example, as shown in Fig. 2(A), the user can take a photograph by slightly extending the arm holding the photographing device 1 forward, placing the sensor unit 11 at a position roughly at the same height as the user's eye level (standard photographing state). On the other hand, as shown in Fig. 2(B), the user can take a photograph by raising the arm holding the photographing device 1, placing the sensor unit 11 at a position higher than the user's eye level (high altitude photographing state).

[0054] Next, a description will be given of the support 13 of the image capturing device 1. Fig. 3 is a perspective view showing the support 13. Fig. 4 is a side view showing the expansion and contraction state of the support 13.

[0055] 3, the support 13 of the imaging device 1 includes a main body 31, an extendable rod 32, a sensor mounting portion 33, a panel mounting portion 34, and a grip portion 35. The display / input panel 12 includes a touch panel display 36.

[0056] The panel mounting unit 34 is provided on the main body 31. The panel mounting unit 34 includes a joint 41 (ball joint) that connects the display input panel unit 12 and the main body 31. This joint 41 allows the orientation of the display input panel unit 12, i.e., the orientation of the display surface of the touch panel display 36, to be changed. Specifically, the display input panel unit 12 rotates around two axes A1 and A2 that pass through the center of the joint 41 (ball joint), allowing the display input panel unit 12 to perform a tilt operation that changes the orientation of the display input panel unit 12 up and down, and a pan operation that changes the orientation of the display input panel unit 12 left and right. The panel mounting unit 34 also includes a fixing knob 42. The user can fix the orientation of the display input panel unit 12 by operating the fixing knob 42. In the example shown in FIG. 3 , because the joint 41 is a ball joint, the orientation of the display input panel unit 12 can be adjusted to any direction within a predetermined angle range.

[0057] In this way, the user can adjust the orientation of the display / input panel unit 12 as appropriate, thereby improving the visibility and operability of the touch panel display 36. For example, when changing the angle of the rod-shaped support body 13, such as when photographing at a high altitude, adjusting the orientation of the display / input panel unit 12 in the tilt direction makes it easier to view and operate the screen of the touch panel display 36. Furthermore, for example, when the user holds the photographing device 1 in his or her right hand, adjusting the orientation of the display / input panel unit 12 in the pan direction so that the screen of the touch panel display 36 faces the left side, where the user's face and left hand, which operates the screen, are located, makes it easier to view and operate the screen of the touch panel display 36.

[0058] The sensor mounting portion 33 is provided at the tip of the extendable rod portion 32. The sensor mounting portion 33 includes a joint 43 (hinge joint) that connects the sensor unit 11 to the tip of the extendable rod portion 32. This joint 43 allows the orientation of the sensor unit 11 to be changed. Specifically, by rotating the sensor unit 11 around the central axis A3 of the joint 43 (hinge joint), the sensor unit 11 can perform a tilting operation that changes the orientation of the sensor unit 11 up and down. The sensor mounting portion 33 is provided with a fixing knob 44. The user can fix the orientation of the sensor unit 11 by operating the fixing knob 44.

[0059] In the example shown in Figure 3, the display input panel unit 12 can be tilted and panned by the joint 41 made of a ball joint in the panel mounting unit 34, but the display input panel unit 12 can also be tilted and panned by combining multiple hinge joints.

[0060] 3, the joint 43 formed of a hinge joint in the sensor mounting section 33 allows only a tilting operation to change the orientation of the sensor unit 11 up and down, but the joint 43 may also allow a panning operation to change the orientation of the sensor unit 11 left and right. In this case, the joint 43 may be formed of a ball joint or a combination of multiple hinge joints.

[0061] The extensible rod portion 32 is configured to be extensible and retractable in the axial direction. In the example shown in Fig. 4, the extensible rod portion 32 has a structure in which a plurality of sheath tubes with gradually decreasing diameters are slidably connected.

[0062] By extending or retracting the extendable rod 32, the user can change the position of the sensor unit 11 relative to the grip unit 35 that is held by hand. That is, as shown in FIG. 4A , when the extendable rod 32 is extended (extended state), the sensor unit 11 is separated from the grip unit 35. This allows the sensor unit 11 to be positioned at a higher position when photographing at high altitudes (see FIG. 2B ). On the other hand, as shown in FIG. 4B , when the extendable rod 32 is retracted and most of the extendable rod 32 is stored in the main body 31 (stored state), the overall length of the photographing device 1 is shortened, making it easier to carry and store the photographing device 1 when not in use.

[0063] As described above, in this embodiment, the user can hold the photographing device 1 stably in one hand by grasping the grip portion 35 with their hand. Therefore, the user can orient the sensor unit 11 in any direction while photographing by changing the orientation of their arm or body holding the photographing device 1. Furthermore, the user can adjust the position of the sensor unit 11 and the orientation of the display / input panel unit 12 by extending or retracting the extendable rod, even while photographing.

[0064] In this embodiment, the display input panel unit 12 is fixed to the main body unit 31, but the display input panel unit 12 may be fixed to the middle of the extendable rod unit 32 so that the position of the display input panel unit 12 relative to the grip unit 35 can be adjusted. Also, the display input panel unit 12 may be fixed to the main body unit 31 via a movable member (not shown) separate from the extendable rod unit 32 to which the sensor unit 11 is fixed at the tip.

[0065] Next, a description will be given of the sensor unit 11 of the image capturing device 1. FIG.

[0066] The sensor section 11 includes a sensor unit 51 and a sensor cover 52 .

[0067] The sensor unit 51 includes a visible camera 53 and a depth sensor 54. The visible camera 53 (color camera) captures color images of the subject. The depth sensor 54 includes an infrared projector 55 and left and right infrared cameras 56 and 57 (stereo cameras). The infrared projector 55 irradiates the subject with infrared light. The left and right infrared cameras 56 and 57 detect reflected light of the infrared light irradiated on the subject. Note that the depth sensor 54 may be a sensor capable of acquiring depth information using other methods, such as LiDAR (Light Detection and Ranging), in addition to a stereo camera.

[0068] In the sensor unit 51, a visible camera 53, left and right infrared cameras 56 and 57, and an infrared projector 55 are arranged side by side (substantially horizontally). The housing of the sensor unit 51 is elongated in the horizontal direction.

[0069] The sensor cover 52 includes a cover body 61 and a filter member 62 .

[0070] The cover body 61 covers the sensor unit 51. During photography, the sensor section 11 may collide with surrounding objects while the photography device 1 is being moved, and the cover body 61 is provided to protect the sensor unit 51 from the impact of such a collision. The cover body 61 has an opening 66 that exposes the cover glass on the front side of the visible camera 53, the infrared projector 55, and the infrared cameras 56 and 57.

[0071] The filter member 62 is attached to the cover body 61 so as to be slidable in the longitudinal direction of the sensor unit 51, i.e., in the arrangement direction of the visible light camera 53, the infrared projector 55, and the infrared cameras 56 and 57. The filter member 62 includes a first ring portion 63 and a second ring portion 64. An ND (Neutral Density) filter 65 (neutral density filter) is attached to the first ring portion 63.

[0072] The filter member 62 is slidably disposed between a first position shown in Fig. 5(A) and a second position shown in Fig. 5(B). A user can grip the filter member 62 with their fingers and slide it. In the first position shown in Fig. 5(A), the ND filter 65 does not cover the visible light camera 53. In the second position shown in Fig. 5(B), the ND filter 65 covers the visible light camera 53.

[0073] 5A, the first ring portion 63 to which the ND filter 65 is attached is located between the visible camera 53 and one of the infrared cameras 57, and the second ring portion 64 is located in a position surrounding the lens of one of the infrared cameras 57. As a result, none of the visible camera 53, the infrared projector 55, and the infrared cameras 56 and 57 are covered by the filter member 62.

[0074] 5(B), the first ring portion 63 is positioned corresponding to the visible camera 53, and the second ring portion 64 is positioned between the visible camera 53 and one of the infrared cameras 57. As a result, only the visible camera 53 is covered by the ND filter 65, and neither the infrared projector 55 nor the infrared cameras 56 and 57 are covered by the filter member 62.

[0075] The filter member 62 is positioned at the first position shown in Fig. 5(A) when the second ring portion 64 abuts against the inner periphery of the opening of the cover body 61. A groove (not shown) is formed on the underside of the cover body 61, extending in the direction of movement of the filter member 62, and a protrusion provided on the filter member 62 fits into this groove, thereby defining the range of movement of the filter member 62 and positioning the filter member 62 at the second position shown in Fig. 5(B).

[0076] When photographing outdoors, the user adjusts the filter member 62 to the second position (see FIG. 5B) where the ND filter 65 covers the visible camera 53. On the other hand, when photographing indoors, the user adjusts the filter member 62 to the first position (see FIG. 5A) where the ND filter 65 does not cover the visible camera 53. This makes it possible to adjust the brightness of the entire captured image by reducing the amount of light input to the camera when photographing outdoors. It also makes it possible to avoid the problem of the entire captured image being tinted purple. In addition, because only the visible camera 53 (color camera) is covered with the ND filter 65, it is possible to avoid the problem of the density of the generated point cloud decreasing when the infrared cameras 56 and 57 are covered with the ND filter 65.

[0077] Next, we will explain the general configuration of the photographing device 1 and the control device 2. Fig. 6 is a block diagram showing the general configuration of the photographing device 1 and the control device 2. Fig. 7 is a block diagram showing an overview of the processing performed by the processor 86 of the control device 2.

[0078] As shown in FIG. 6, the sensor unit 11 of the imaging device 1 includes a visible camera 53 , a depth sensor 54 , an IMU (Inertial Measurement Unit) 71 , an input / output interface 72 , and a connection terminal 73 .

[0079] The visible light camera 53 (color camera) takes color photographs and outputs color photographed images.

[0080] The depth sensor 54 outputs depth information (distance image) as a detection result based on images captured by the left and right infrared cameras 56 and 57 (see FIG. 5).

[0081] The IMU 71 detects the motion state of the device itself, specifically, three-dimensional angular velocity and acceleration. Based on the detection results of the IMU 71, the position, orientation, and velocity of the sensor unit 11 can be detected.

[0082] The input / output interface 72 inputs and outputs data to and from the control device 2 via the display / input panel unit 12. Specifically, the input / output interface 72 transmits detection data from the visible camera 53, the depth sensor 54, and the IMU 71. The input / output interface 72 may be based on the USB (registered trademark) standard.

[0083] The first cable 21 that connects the sensor unit 11 and the display / input panel unit 12 is detachably coupled to the connection terminal 73 .

[0084] The visible camera 53, the depth sensor 54, and the IMU 71 may not be integrated into the sensor unit 51 (see FIG. 5). Alternatively, the depth sensor 54 and the IMU 71 may be omitted, and only the visible camera 53 may be provided. Alternatively, either the depth sensor 54 or the IMU 71 may be provided together with the visible camera 53.

[0085] The display / input panel unit 12 of the photographing device 1 includes a touch panel display 36 , a repeater 74 , an input / output interface 75 , a first connection terminal 76 , and a second connection terminal 77 .

[0086] The touch panel display 36 displays a screen that assists the user in taking photographs, etc., under the control of the control device 2.

[0087] The repeater 74 repeats data communication between the control device 2 and the sensor unit 11. The repeater 74 may be a hub based on the USB (registered trademark) standard.

[0088] The input / output interface 75 inputs and outputs data to and from the control device 2. Specifically, the input / output interface 75 receives display information such as a screen that assists the user in taking photographs from the control device 2. The input / output interface 75 may be based on the USB (registered trademark) standard.

[0089] A first cable 21 that connects the sensor unit 11 and the display / input panel unit 12 is detachably coupled to the first connection terminal 76. A second cable 22 that connects the display / input panel unit 12 and the control device 2 is detachably coupled to the second connection terminal 87.

[0090] The first cable 21 is used for data communication between the control device 2 and the sensor unit 11 via the display input panel unit 12. In this data communication, for example, shooting data (shooting information) output from the sensor unit 11 is transmitted to the control device 2. The shooting data includes an image captured by the visible camera 53, a detection result (distance information) from the depth sensor 54, a detection result from the IMU 71, and a shooting time. The second cable 22 is used for data communication between the control device 2 and the display input panel unit 12 and data communication between the control device 2 and the sensor unit 11. In the data communication between the control device 2 and the display input panel unit 12, for example, display information for a screen to be displayed on the touch panel display 36 is transmitted from the control device 2, and user operation information detected on the touch panel display 36 is transmitted to the control device 2.

[0091] The control device 2 includes an input / output interface 81 , a display 82 , an input device 83 , a memory 84 , a storage device 85 , a processor 86 , and a connection terminal 87 .

[0092] The input / output interface 81 inputs and outputs data to and from the image capturing device 1. The input / output interface 81 may be based on the USB (registered trademark) standard.

[0093] The display 82 displays a screen for managing previously acquired photographic data, a screen for setting the operating conditions of the photographic device 1, and the like.

[0094] The input device 83 is used by the user to perform input operations. The input device 83 may be a keyboard, a mouse, a touchpad, a touch panel, etc. If the control device 2 is configured as a tablet PC, a touch panel display 36 is provided in which the touch panel as the input device 83 and the display panel as the display 82 are integrated.

[0095] A second cable 22 that connects the control device 2 and the display input panel unit 12 of the image capturing device 1 is detachably coupled to the connection terminal 87 .

[0096] The memory 84 stores programs executed by the processor 86 and the like.

[0097] The storage device 85 stores the shooting data (shooting information) acquired from the image capturing device 1. The shooting data includes the image captured by the visible camera 53, the detection results (distance information) of the depth sensor 54, the detection results of the IMU 71, the shooting time, etc. The storage device 85 also stores point cloud data generated by the processor 86 as the three-dimensional measurement results.

[0098] The processor 86 performs various processes by executing programs stored in the memory 84. In this embodiment, the processor 86 performs a three-dimensional measurement process P1.

[0099] In the three-dimensional measurement process P1, the processor 16 generates point cloud data (environment map) as three-dimensional spatial information about the measurement target location using the SLAM (Simultaneous Localization And Mapping) method based on images captured by the visible camera 53. In the three-dimensional measurement process P1, self-position estimation is performed in addition to the generation of the point cloud data, and the self-position for each measurement, i.e., the position of the imaging point, is obtained.

[0100] As shown in FIG. 7, the three-dimensional measurement process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0101] In the feature extraction process P11, the processor 86 extracts feature information (feature points, etc.) from the image (frame) captured by the visible camera 53.

[0102] In tracking processing P12, processor 86 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition related to the position and orientation of the image capture device 1, and updates the position and orientation trajectory data based on the amount of transition. Note that the position and orientation trajectory data is obtained by tracking the position and orientation of the image capture device 1, and includes the tracking results of the position and orientation at the time of capturing each captured image (frame), i.e., information regarding the position and orientation of the image capture device 1 at each time of capturing.

[0103] In the position and orientation correction process P13, the processor 86 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, the position and orientation trajectory data is corrected so as to compensate for the measurement results of locations with few features, such as walls and ceilings.

[0104] In the point cloud generation process P14, the processor 86 generates point cloud data based on the distance information from the depth sensor 54 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13. The point cloud generation process P14 includes a process (standard point cloud generation process) for generating regular point cloud data acquired as a 3D measurement result, and a process (simplified point cloud generation process) for generating simplified point cloud data that allows the user to check the shooting conditions (point cloud generation conditions). The simplified point cloud generation process must be performed in real time during shooting, but the standard point cloud generation process may be performed after shooting.

[0105] As shown in FIG. 6, the processor 86 also performs a drawing generation process P2, a display information generation process P3, and a display process P4.

[0106] In the drawing generation process P2, the processor 86 generates a 3D model of the target location based on the point cloud data of the target location generated in the 3D measurement process P1, and generates a layout drawing of the target location based on the 3D model of the target location. At this time, a 2D layout drawing (plan view) of the target location is generated by projecting the 3D model onto a horizontal plane. In addition, a 3D layout drawing of the target location is generated by projecting the 3D model based on a predetermined line of sight.

[0107] In the display information generation process P3, the processor 86 generates display information for a screen to be displayed on the display 82 of the control device 2. The display 82 displays screens (see FIGS. 8 to 14) on which the user performs operations such as managing the photography data, issuing instructions for post-photography processing (point cloud generation process P14, drawing generation process P2), and setting processing conditions, etc. The processor 86 also generates display information for a screen to be displayed on the touch panel display 36 of the photography device 1. The touch panel display 36 displays screens (see FIGS. 15 to 19 and 21 to 27) that support the user's photography work.

[0108] In display processing P4, the processor 86 displays a screen (see FIGS. 8 to 14) on the display 82 of the control device 2 based on the display information generated in display information generation processing P3. The processor 86 also displays a screen (see FIGS. 15 to 19 and 21 to 27) on the touch panel display 36 of the imaging device 1.

[0109] In this embodiment, a shooting standby screen 311 (see FIG. 16), a shooting in progress screen 331 (see FIG. 17), and shooting end confirmation screens 351 and 361 (see FIG. 21) are displayed on the touch panel display 36 of the photographing device 1 so as to transition sequentially in response to a user's instruction. This allows the user to appropriately make the necessary settings and check the screens at each stage before, during, and after shooting.

[0110] In this embodiment, the non-photographing mode (first mode) and the photographing mode (second mode) are switched in response to a user operation.

[0111] In the non-photography mode, an operable screen (see Figures 8 to 14) is displayed on the display 82 of the control device 2, which allows the user to manage and process the photographed data, and an inoperable pause screen 301 (see Figure 15) is displayed on the touch panel display 36 of the photographing device 1.

[0112] On the other hand, in the photography mode, photography becomes possible using the photography device 1, and the touch panel display 36 of the photography device 1 displays operable screens to assist the user in the photography operation (see Figures 15 to 19, Figures 21 to 27), for example, screens that give instructions to start and end photography (see Figures 16 and 17), and an inoperable pause screen 221 (see Figure 13) is displayed on the display 82 of the control device 2.

[0113] As shown in FIG. 6, the processor 86 also performs a connection failure detection process P5, a speed presentation process P6, a photographing time presentation process P7, and a self-position lost recovery process P8.

[0114] In the connection failure detection process P5, the processor 86 detects a connection failure on the communication path between the processor 86 and the sensor unit 51 and the touch panel display 36, and notifies the user of the occurrence of the connection failure. Specifically, connection failure notification screens 231, 241 (see FIG. 14 ) are displayed on the display 82 of the control device 2. In addition, a connection failure notification screen 411 (see FIG. 26 ) is displayed on the touch panel display 36 of the imaging device 1.

[0115] In the speed presentation process P6, the processor 86 measures the speed at which the user moves the sensor unit 11 based on the detection data (acceleration, angular velocity) of the IMU 71, and visualizes whether the speed is appropriate or not to present to the user. Specifically, a speed bar 334 is displayed on the during-shooting screen 331 (see FIG. 17 ).

[0116] In the shooting time presentation process P7, the processor 86 presents the user with a preset available shooting time at the start of shooting. Furthermore, during shooting, the processor 86 presents the user with the remaining shooting time when the remaining shooting time, calculated by subtracting the elapsed time from the start of shooting from the available shooting time, falls below a predetermined time. Specifically, the available shooting time is presented on the shooting-in-progress screen 331 (see FIG. 18 ) at the start of shooting, and then the remaining shooting time is presented during shooting.

[0117] In the self-position lost return process P8, when the processor 86 detects that the current position of the image capture device 1 has been lost (tracking lost) in the tracking process P12, the processor 86 notifies the user of the occurrence of the self-position lost, and presents the user with the last position acquisition point and provides return guidance urging the user to return to the last position acquisition point. Specifically, a self-position lost notification screen 371 (see FIG. 22) is displayed, followed by a return screen 381 (see FIG. 23) including an image captured at the last position acquisition point. Furthermore, when the processor 86 detects that the image capture device 1 has returned to the last position acquisition point, the normal tracking process P12 is resumed.

[0118] Next, a description will be given of the imaging data list screen 101 displayed on the display 82 of the control device 2. FIG.

[0119] When the photography application is started in the control device 2, a photography data list screen 101 is displayed on the display 82.

[0120] The imaging data list screen 101 has a list display section 102. The list display section 102 displays a list of captured images for each of a plurality of imaging data sets with different measurement locations and imaging dates and times. In the example shown in FIG. 8 , the captured images for each imaging data set are displayed arranged by date. The list display section 102 also displays the imaging date and time for each captured image. If the point cloud generation process P14 has been performed, the list display section 102 displays a check mark 111 indicating this, and if the drawing generation process P2 has been performed, a check mark 112 indicating this.

[0121] When the user selects a piece of photography data by operating one of the photographed images for each piece of photography data displayed in the list display section 102, the screen transitions to a photography data details screen 121 (see FIG. 9 ) for the selected piece of photography data. At this time, the user can select the photography data to be deleted or the photography data for which the point cloud generation process P14 and the drawing generation process P2 are to be executed.

[0122] If there is no registered photographic data, a message indicating that there is no photographic data is displayed in the list display section 102, such as the text "There is no photographic data."

[0123] The imaging data list screen 101 also has a setting button 103. When the user operates the setting button 103, a setting screen 181 (see FIG. 12A) pops up on the imaging data list screen 101.

[0124] The imaging data list screen 101 also has a "Start imaging" button 104. When the user operates the "Start imaging" button 104, the non-imaging mode is switched to the imaging mode. At this time, the display 82 of the control device 2 transitions to a pause screen 221 (see FIG. 13). Meanwhile, the touch panel display 36 of the imaging device 1 transitions from a pause screen 301 (see FIG. 15) to an imaging standby screen 311 (see FIG. 16).

[0125] Next, the photographing data details screen 121 displayed on the display 82 of the control device 2 will be described. Fig. 9 is an explanatory diagram showing the photographing data details screen 121. Fig. 10 is an explanatory diagram showing a screen that is displayed as a pop-up on the photographing data details screen 121. Fig. 11 is an explanatory diagram showing the transition status of the main parts of the photographing data details screen 121.

[0126] The photographic data details screen 121 is provided with an image display section 122 and a photographing date and time display section 123. The photographed images (key frames) included in the photographic data are played back in order in the image display section 122. The photographing date and time display section 123 displays the photographing date and time of the photographed image displayed in the image display section 122. In the SLAM method, a frame in which a significant change in the field of view appears is extracted as a key frame, and the key frame is reflected in the point cloud.

[0127] The image capture data details screen 121 also has a playback operation section 124. The playback operation section 124 has a seek bar 131, a playback stop button 132, a frame advance button 133, and a frame rewind button 134. The user can arbitrarily specify a playback start time by operating the slider of the seek bar 131. The user can also instruct playback and stop of the image capture data by operating the playback stop button 132. The user can also instruct frame advance and frame rewind of the image capture data by operating the frame advance button 133 and the frame rewind button 134.

[0128] The imaging data details screen 121 also has an instruction operation section 125. The instruction operation section 125 has a "generate point cloud" button 141, a "display point cloud" button 142, a "generate drawing" button 143, and a "display drawing" button 144.

[0129] When the user operates the "Generate Point Cloud" button 141, the point cloud generation process P14 (standard point cloud generation process) is started, and a point cloud generation in progress screen 151 shown in FIG. 10A pops up on the imaging data details screen 121. The point cloud generation in progress screen 151 displays a message indicating that the point cloud generation process P14 is being executed. The point cloud generation in progress screen 151 also has a progress bar 152. The progress bar 152 visualizes the progress of the point cloud generation process P14. The point cloud generation in progress screen 151 also has a "Cancel point cloud generation" button 153. When the user operates the "Cancel point cloud generation" button 153, the point cloud generation process P14 is canceled and the screen returns to the imaging data details screen 121 (see FIG. 9 ). When the point cloud generation process P14 is completed, the screen returns to the imaging data details screen 121.

[0130] Furthermore, when the user operates the "point cloud display" button 142 on the photographing data details screen 121 shown in FIG. 9, a point cloud display application is launched and the point cloud data generated in the point cloud generation process P14 is displayed.

[0131] Furthermore, when the user operates the "Drawing Generation" button 143, a drawing generation in progress screen 161 shown in FIG. 10(B) pops up on the photographing data details screen 121. The drawing generation in progress screen 161 displays a message indicating that the drawing generation process P2 is being executed. The drawing generation in progress screen 161 also has a "Cancel drawing generation" button 162. When the user operates the "Cancel drawing generation" button 162, the drawing generation process P2 is canceled and the screen returns to the photographing data details screen 121 (see FIG. 9). Furthermore, when the drawing generation process P2 is completed, the screen returns to the photographing data details screen 121. The drawing generation in progress screen 161 may also display an animation indicating that the drawing generation process P2 is being executed.

[0132] Furthermore, when the user operates the "Drawing display" button 144 on the photographing data details screen 121 shown in FIG. 9, a drawing display application is started and the drawing generated in the drawing generation process P2 is displayed.

[0133] The photographing data details screen 121 also has a "Delete" button 145. When the user operates the "Delete" button 145, a deletion confirmation screen 171 shown in FIG. 10(C) pops up on the photographing data details screen 121. The deletion confirmation screen 171 displays a message prompting the user to confirm the deletion of the photographing data. The deletion confirmation screen 171 also has a "Delete" button 172 and a "Cancel" button 173. When the user operates the "Delete" button 172, the photographing data is deleted and the screen returns to the photographing data list screen 101 (see FIG. 8). When the user operates the "Cancel" button 173, the screen returns to the photographing data details screen 121 (see FIG. 9).

[0134] Here, as shown in FIG. 11, in the instruction operation unit 125, the states of the "Point Cloud Generation" button 141, the "Point Cloud Display" button 142, the "Drawing Generation" button 143, and the "Drawing Display" button 144 change before and after the point cloud generation process P14 and the drawing generation process P2.

[0135] First, as shown in FIG. 11A, before the point cloud generation process P14 is executed, only the "Generate Point Cloud" button 141 is operable, and the "Display Point Cloud" button 142, "Generate Drawing" button 143, and "Display Drawing" button 144 are inoperable. When the point cloud generation process P14 is completed, as shown in FIG. 11B, a check mark 146 is displayed indicating that the point cloud generation process P14 has been executed. Also, in this state, the "Display Point Cloud" button 142 and the "Generate Drawing" button 143 are operable, but the "Display Drawing" button 144 is inoperable. When the drawing generation process P2 is completed, as shown in FIG. 11C, a check mark 147 is displayed indicating that the drawing generation process P2 has been executed. Also, in this state, the "Display Drawing" button 144 is operable.

[0136] 9, the photographing data detail screen 121 is provided with a "return to list" button 126. When the user operates the "return to list" button 126, the screen returns to the photographing data list screen 101 (see FIG. 8).

[0137] Next, a description will be given of the setting screen 181 displayed on the display 82 of the control device 2. FIG.

[0138] When the user operates the setting button 103 on the photographing data list screen 101 (see FIG. 8), a setting screen 181 shown in FIG. 12A pops up on the photographing data list screen 101.

[0139] The setting screen 181 has a save folder setting section 182. The save folder setting section 182 displays a folder path indicating the location where a setting file containing setting information will be saved. The save folder setting section 182 also has a "Change" button 191. When the user operates the "Change" button 191, a screen (not shown) for specifying a folder pops up and allows the user to specify a folder in which to save the setting file.

[0140] The setting screen 181 also has a camera setting section 183. The camera setting section 183 has input sections 192, 193, and 194 for the resolution of the visible camera 53, the resolution of the depth sensor 54, and the frame rate of the visible camera 53. When the user operates each of the input sections 192, 193, and 194, a pull-down menu (not shown) is displayed. The user can select the resolution of the visible camera 53, the resolution of the depth sensor 54, and the frame rate of the visible camera 53 from the pull-down menu.

[0141] The setting screen 181 also has a device information display section 184. The device information display section 184 displays information about the sensor unit 51, such as the name (camera name), serial number, and firmware version of the sensor unit 51, as well as the date and time when the previous calibration process was performed. The device information display section 184 also has a "forced restart" button 195. When the user operates the "forced restart" button 195, the sensor unit 51 is forcibly restarted. The device information display section 184 also has an "execute" button 196. When the user operates the "execute" button 196, a device calibration screen 201 shown in FIG. 12(B) pops up on the setting screen 181.

[0142] The setting screen 181 also has a "Done" button 185. When the user operates the "Done" button 185, the setting process is executed based on the input contents on the setting screen 181, and the screen returns to the shooting data list screen 101 (see FIG. 8).

[0143] As shown in Fig. 12(B), the device calibration screen 201 displays a message prompting the user to keep the image capture device 1 stationary in a stable location for a predetermined time for the calibration process, as well as the remaining time. The device calibration screen 201 also displays a progress bar 202 indicating the progress of the calibration process. The device calibration screen 201 also has a "Cancel" button 203. When the user operates the "Cancel" button 203, the calibration process is canceled and the screen returns to the setting screen 181 (see Fig. 12(A)). When the calibration process is completed, the screen returns to the setting screen 181.

[0144] Next, a description will be given of the pause screen 221 and the poor connection notification screens 231 and 241 displayed on the display 82 of the control device 2. Fig. 13 is an explanatory diagram showing the pause screen 221. Fig. 14 is an explanatory diagram showing the poor connection notification screens 231 and 241.

[0145] When the user operates the "Start shooting" button 104 on the shooting data list screen 101 (see FIG. 8), the non-shooting mode is switched to the shooting mode.

[0146] If the transition to the photographing mode is successful, the screen transitions to a pause screen 221 shown in Fig. 13. The pause screen 221 displays a message indicating that the mode has been switched to the photographing mode, and a message indicating that the screen can be displayed and operated on the display 82 of the control device 2 by performing an operation to end the photographing mode on the screen displayed on the touch panel display 36 of the photographing device 1.

[0147] On the other hand, if the transition to the photographing mode fails, depending on the cause of the failure, a poor connection notification screen 231, 241 (error screen) shown in Fig. 14 pops up on the photographing data list screen 101. Here, if a poor connection has occurred in the second cable 22 connecting the display / input panel unit 12 of the photographing device 1 and the control device 2, the poor connection notification screen 231 shown in Fig. 14(A) is displayed. On the other hand, if a poor connection has occurred in the first cable 21 connecting the display / input panel unit 12 of the photographing device 1 and the sensor unit 11, the poor connection notification screen 241 shown in Fig. 14(B) is displayed.

[0148] The poor connection notification screen 231 shown in Fig. 14(A) displays a message indicating that the image capture mode cannot be entered due to a poor connection between the display / input panel unit 12 of the image capture device 1 and the control device 2, and a message urging the user to check the location of the poor connection. The poor connection notification screen 231 also has a "retry" button 232 and a "cancel" button 233. The user operates the "retry" button 232 after performing work to resolve the poor connection of the second cable 22. This causes the process of entering the image capture mode to be executed again. On the other hand, if the user operates the "cancel" button 233, the screen returns to the image capture data list screen 101 (see Fig. 8).

[0149] The poor connection notification screen 241 shown in FIG. 14(B) displays a message indicating that the image capture mode cannot be entered due to a poor connection between the display / input panel unit 12 and the sensor unit 11 of the image capture device 1, and a message prompting the user to check the location of the poor connection. The poor connection notification screen 241 also has a "retry" button 242 and a "cancel" button 243. The user operates the "retry" button 242 after correcting the poor connection of the first cable 21 connecting the display / input panel unit 12 and the sensor unit 11 of the image capture device 1. This causes the process of entering the image capture mode to be executed again. On the other hand, if the user operates the "cancel" button 243, the image capture data list screen 101 (see FIG. 8) is displayed again.

[0150] Next, a description will be given of the pause screen 301 displayed on the touch panel display 36 of the photographing device 1. FIG.

[0151] In non-photography mode, i.e., while the user is operating the screen by displaying the photography data list screen 101 (see Figure 8), the photography data details screen 121 (see Figure 9), and the settings screen 181 (see Figure 12) on the display 82 of the control device 2, a pause screen 301 is displayed on the touch panel display 36 of the photography device 1.

[0152] The pause screen 301 displays a message indicating that the device is in non-photographing mode, i.e., that the photographing data list screen 101 (see FIG. 8 ) or the like is being displayed and operated on the display 82 of the control device 2. The pause screen 301 also displays a message indicating that the device will transition to photographing mode by performing an operation to start photographing on the screen displayed on the display 82 of the control device 2.

[0153] Next, a description will be given of the shooting standby screen 311 displayed on the touch panel display 36 of the photographing device 1. FIG.

[0154] 16A, an image display section 312 is provided on the image capture standby screen 311. The image display section 312 displays an image captured by the visible camera 53 in real time.

[0155] The shooting standby screen 311 also has a shooting mode designation section 313. When the user operates the shooting mode designation section 313, a pull-down menu 314 is displayed, as shown in FIG. 16(B). The pull-down menu 314 allows the user to select from a plurality of shooting modes with different control conditions (processing conditions for image signal processing) for the visible camera 53. In the example shown in FIG. 16(B), the user can select one of the shooting modes: "indoor" (indoor shooting mode), "outdoor" (first outdoor shooting mode), or "outdoor + ND filter" (second outdoor shooting mode). Note that each shooting mode uses a different brightness correction table, for example, used in image signal processing (image correction processing).

[0156] The shooting standby screen 311 also has a brightness adjustment bar 315. The user can adjust the brightness of the captured image by operating the slider of the brightness adjustment bar 315. Brightness adjustment can be performed for each shooting mode.

[0157] The shooting standby screen 311 also has a shooting start button 316. When the user operates the shooting start button 316, shooting starts and the screen transitions to a shooting in progress screen 331 (see FIG. 17).

[0158] Furthermore, the shooting standby screen 311 is provided with an "End shooting" button 317. When the user operates the "End shooting" button 317, the shooting mode is ended, the mode shifts to the non-shooting mode, and the screen transitions to the pause screen 301 (see FIG. 15).

[0159] Next, the during-shooting screen 331 displayed on the touch panel display 36 of the photographing device 1 will be described. Fig. 17 is an explanatory diagram showing the during-shooting screen 331. Fig. 18 is an explanatory diagram showing the transition status of the shooting time guide section 336 on the during-shooting screen 331. Fig. 19 is an explanatory diagram showing the during-shooting screen 331 with a message window 337 displayed. Fig. 20 is an explanatory diagram showing the contents of the message displayed in the message window 337 and the display timing.

[0160] 17A and 17B, a main window 332 (first image display frame) and a sub-window 333 (second image display frame) are provided on a shooting screen 331. The main window 332 and the sub-window 333 have different display magnifications, with the main window 332 displaying an enlarged image and the sub-window 333 displaying a reduced image.

[0161] The main window 332 and the sub-window 333 also display either a real-time image 341 captured by the visible camera 53 or a photography path image 342 representing the photography path at the measurement target location. The photography path image 342 is obtained by superimposing a line 344 representing the photography path on a point cloud image 343 representing the measurement target location. The point cloud image is obtained by imaging (rendering) each point of the generated point cloud data as seen from a predetermined viewpoint.

[0162] The example shown in Fig. 17(A) is the case of the standard state. In this case, a photographed image 341 is displayed enlarged in the main window 332, and a photographed path image 342 is displayed reduced in the sub-window 333. On the other hand, the example shown in Fig. 17(B) is the case of the photographed path enlarged state. In this case, a photographed path image 342 is displayed enlarged in the main window 332, and a photographed image 341 is displayed reduced in the sub-window 333.

[0163] Furthermore, the user can switch between the standard state shown in Fig. 17(A) and the expanded shooting path state shown in Fig. 17(B) by operating the sub-window 333. That is, when the sub-window 333 is operated in the standard state shown in Fig. 17(A), the state transitions to the expanded shooting path state shown in Fig. 17(B), and a shooting path image 342 is enlarged and displayed in the main window 332. When the sub-window 333 is operated in the expanded shooting path state shown in Fig. 17(B), the state transitions to the standard state shown in Fig. 17(A), and a photographed image 341 is enlarged and displayed in the main window 332.

[0164] A mesh image may be superimposed within the point cloud generation range of the photographed image 341. The mesh image can be obtained by converting the generated point cloud data into mesh data and then performing processing to image the mesh data with the same angle of view as the photographed image 341.

[0165] The shooting screen 331 also has a speed bar 334. The speed bar 334 visualizes and displays the speed at which the user moves the sensor unit 11. Specifically, the speed is represented by color. For example, the speed bar 334 is drawn with a gradation from green to orange, with the proportion of orange increasing as the speed increases. This allows the user to easily check whether the speed at which the sensor unit 11 is moved is appropriate, specifically, whether the speed at which the sensor unit 11 is moved is not too fast. This makes it possible to prevent the user from moving the sensor unit 11 too fast, which can cause blurring and reduce the accuracy of the 3D measurement.

[0166] The image capturing screen 331 also has an elapsed time display section 335. The elapsed time display section 335 displays the time that has elapsed since the start of image capturing.

[0167] 18, a shooting time guide section 336 is displayed on the shooting-in-progress screen 331. While the elapsed time display section 335 is always displayed, the shooting time guide section 336 is displayed only for a predetermined period of time at a predetermined timing. Note that the shooting time guide section 336 is not limited to being displayed only at predetermined timings, but can also be set to be always displayed by a user operation.

[0168] 18A, when shooting starts, a shooting time guide section 336 is displayed for a predetermined time (e.g., 5 seconds) from the start of shooting. In this case, the shooting time guide section 336 displays the remaining shooting time (e.g., 15 minutes).

[0169] Next, as shown in FIG. 18(B), when the remaining shooting time (the time elapsed since the start of shooting minus the available shooting time) falls below a predetermined time, the shooting time indicator 336 is displayed for a predetermined time (for example, 5 seconds). In this case, the remaining shooting time is displayed in the shooting time indicator 336. The timing at which the shooting time indicator 336 is displayed is not limited to one time. For example, the shooting time indicator 336 may be displayed when the remaining shooting time is 3 minutes and when the remaining shooting time is 2 minutes.

[0170] Next, as shown in Fig. 18(C), when the remaining shooting time becomes very short (for example, one minute), the shooting time indicator 336 continues to be displayed until the remaining shooting time runs out. In this case, the remaining shooting time is displayed in the shooting time indicator 336. The shooting time indicator 336 is also displayed in a manner different from that shown in Figs. 18(A) and 18(B). For example, the shooting time indicator 336 is highlighted by changing the color of the background and frame of the shooting time indicator 336.

[0171] 19, a message window 337 is displayed on the during-shooting screen 331. Messages to notify the user are displayed in the message window 337. Specifically, messages regarding guidance for the shooting operation, various warnings, etc. are displayed.

[0172] The example shown in Fig. 19(A) is an information message. Information messages provide the user with information that will be helpful for the user's photography work or information that calls attention to the user's attention. The example shown in Fig. 19(B) is a warning message. Warning messages prompt the user to take action to improve the status of photography work that needs to be improved immediately.

[0173] The display position of the message window 337 on the during-shooting screen 331 may be varied depending on the type of message (information provision, warning, etc.) For example, in the example shown in Fig. 19(B), the message window 337 for a warning message is displayed at the bottom of the during-shooting screen 331, similar to the case of the information message shown in Fig. 19(A), but the warning message may also be displayed in the center of the during-shooting screen 331 so that the user can immediately check it.

[0174] 20 shows examples of information messages and warning messages displayed in the message window 337. Information messages include a message indicating successful recovery from a self-position loss, a message indicating completion of position correction processing when loop closing (circular movement) is detected, and a message urging the user to stop the image capture device 1 for position correction processing when loop closing is detected. Warning messages include a message urging the user to stop the image capture device 1 if the user moves the image capture device 1 while loop closing is detected and position correction processing is being performed, a message urging the user to move the image capture device 1 slowly if the user is moving the image capture device 1 too quickly, a message urging the user to change the subject to be captured if few feature points are extracted from the captured image, and a message urging the user to maintain an appropriate distance from the subject if the user gets too close.

[0175] Next, a description will be given of the end-of-photography confirmation screens 351 and 361 displayed on the touch panel display 36 of the photographing device 1. FIG.

[0176] 17, an end of shooting button 339 is provided on the during-shooting screen 331. When the user operates the end of shooting button 339, a process of ending shooting and saving the shot data is executed, and an end of shooting confirmation screen 351 is popped up on the during-shooting screen 331, as shown in FIG.

[0177] The shooting end confirmation screen 351 displays a message that shooting has ended and the saving of the shooting data has been completed, and a message that the screen will return to the shooting standby screen 311 (see FIG. 16 ). The shooting end confirmation screen 351 also has an "OK" button 352. When the user operates the "OK" button 352, the screen transitions to the shooting standby screen 311.

[0178] In this embodiment, if the shooting time (the time elapsed since the start of shooting) exceeds a predetermined available shooting time, the shooting is forcibly ended and processing to save the shot data is executed. In this case, as shown in FIG. 21(B), a shooting end confirmation screen 361 due to shooting time exceeding is displayed as a pop-up on the shooting-in-progress screen 331.

[0179] The shooting end confirmation screen 361 displays a message that shooting has been forcibly ended due to the time limit being exceeded and that saving of the shooting data has been completed, and a message that the screen will return to the shooting standby screen 311 (see FIG. 16 ). The shooting end confirmation screen 361 also has an "OK" button 362. When the user operates the "OK" button 362, the screen transitions to the shooting standby screen 311.

[0180] In this embodiment, shooting is forcibly terminated when the shooting time exceeds a predetermined shooting time, but such forced termination may not be performed, or the user may be allowed to select a mode in which forced termination is performed or a mode in which forced termination is not performed.

[0181] Next, a description will be given of screens displayed on the touch panel display 36 of the image capturing device 1 when the self-position is lost. Fig. 22 is an explanatory diagram showing a self-position lost notification screen 371. Fig. 23 is an explanatory diagram showing a recovery screen 381. Fig. 24 is an explanatory diagram showing a last position acquisition point enlarged screen 391. Fig. 25 is an explanatory diagram showing a recovery failure notification screen 401.

[0182] During shooting, a self-position lost (tracking lost) may occur in the tracking process P12 (self-position estimation process), where the self-position is lost. In this case, it is necessary to return to the last position acquisition point, i.e., the last point where the self-position was correctly estimated, and restart the tracking process P12. Therefore, first, the user is notified that a self-position lost has occurred, and then the user is assisted to return to the last position acquisition point. When it is detected that the user has returned to the last position acquisition point, the tracking process P12 is restarted.

[0183] In this embodiment, when a loss of the self-position is detected, a self-position lost notification screen 371 is displayed as a pop-up on the in-photographing screen 331 on the touch panel display 36 of the photographing device 1, as shown in FIG.

[0184] The self-location lost notification screen 371 notifies the user that the self-location has been lost and displays a message urging the user to return to the last location acquisition point. The self-location lost notification screen 371 also has an "OK" button 372. When the user operates the "OK" button 372, the screen transitions to a returning screen 381 shown in Fig. 23. At this time, the processor 86 of the control device 2 starts processing to detect that the image capturing device 1 has returned to the last location acquisition point.

[0185] 23(A) is the case of a standard state, similar to the example shown in Fig. 17(A), in which an enlarged photographing image 341 captured by the visible camera 53 is displayed in the main window 332 (first image display frame), and a reduced photographing path image 342 is displayed in the sub-window 333 (second image display frame). Also, the example shown in Fig. 23(B) is the case of an enlarged photographing path state, similar to the example shown in Fig. 17(B), in which an enlarged photographing path image 342 is displayed in the main window 332, and a reduced photographing image 341 is displayed in the sub-window 333.

[0186] In addition to the sub-window 333 (second image display frame) in which the photographed image 341 and the photographing path image 342 are displayed in reduced size, the return screen 381 also has another sub-window 382 (third image display frame). The sub-window 382 displays a photographed image of the final location acquisition point. This allows the user to easily grasp the final location acquisition point.

[0187] Furthermore, a message window 383 is displayed on the returning screen 381. Similar to the self-position lost notification screen 371 (see FIG. 22 ), the message window 383 displays a message notifying the user that their own position has been lost and urging them to return to the last position acquisition point.

[0188] When the subwindow 382 displaying the captured image of the last position acquisition point is operated, a last position acquisition point enlarged screen 391 is popped up on the returning screen 381, as shown in FIG. 24 . The last position acquisition point enlarged screen 391 displays an enlarged image of the last position acquisition point. An "x" button 392 is also displayed on the last position acquisition point enlarged screen 391. When the user operates the "x" button 392, the last position acquisition point enlarged screen 391 is closed and the screen returns to the returning screen 381.

[0189] Here, when the device returns from losing its own position, i.e., when it has successfully returned to the last position acquisition point, the screen transitions to the image capturing screen 331 (see FIG. 17 ). Note that, when it is detected that the device has returned to the last position acquisition point, a message notifying the device of the return screen 381 that the device has returned to the last position acquisition point may be displayed on the return screen 381, and then the screen transitions to the image capturing screen 331.

[0190] On the other hand, if the device is unable to recover from its own position being lost even after a predetermined time has elapsed, a recovery failure notification screen 401 pops up on the recovering screen 381, as shown in Fig. 25. The recovery failure notification screen 401 displays a message indicating that recovery from its own position being lost has failed ("Tracking recovery has failed."). The recovery failure notification screen 401 also has an "OK" button 402. When the user operates the "OK" button 402, the image capture is forcibly terminated, and the screen transitions to the image capture standby screen 311 (see Fig. 16).

[0191] Next, a description will be given of the poor connection notification screen 411 displayed on the touch panel display 36 of the photographing device 1. FIG.

[0192] During imaging, the first cable 21 connecting the display / input panel unit 12 and the sensor unit 11 may become disconnected. In this case, a poor connection notification screen 411 (error screen) shown in FIG. 26 pops up on the imaging-in progress screen 331.

[0193] The poor connection notification screen 411 displays a message stating that the sensor unit 11, including the visible camera 53, cannot be detected, and a message urging the user to check the condition of the sensor unit 11 and the cable connecting the display input panel unit 12 to the sensor unit 11.

[0194] The poor connection notification screen 411 also has a "retry" button 412 and a "cancel" button 413. The user operates the "retry" button 412 after correcting the poor connection in the cable connecting the display / input panel unit 12 and the sensor unit 11 of the image capture device 1. When the poor connection is corrected, the screen transitions to the image capture standby screen 311 (see FIG. 16). When the user operates the "cancel" button 413, the screen transitions from image capture mode to list mode. At this time, the touch panel display 36 of the image capture device 1 transitions to the pause screen 301 (see FIG. 15). Meanwhile, the display 82 of the control device 2 transitions from the pause screen 221 (see FIG. 13) to the image capture data list screen 101 (see FIG. 8).

[0195] It should be noted that the second cable 22 connecting the display input panel unit 12 of the image capture device 1 and the control device 2 may become disconnected. In this case, the screen will no longer be displayed on the touch panel display 36 of the image capture device 1. Meanwhile, on the display 82 of the control device 2, a screen (not shown) notifying the user of the poor connection pops up on the pause screen 221 (see FIG. 13). The screen notifying the user of the poor connection has an "OK" button. When the user operates the "OK" button, the screen transitions to the image capture data list screen 101 (see FIG. 8).

[0196] Next, a description will be given of the photographing mode termination confirmation screen 421 displayed on the touch panel display 36 of the photographing device 1. FIG.

[0197] When the user operates the "End shooting" button 317 on the shooting standby screen 311 shown in FIG. 16, a shooting mode end confirmation screen 421 shown in FIG. 27 pops up on the shooting standby screen 311.

[0198] The image capture mode termination confirmation screen 421 displays a message inquiring the user as to whether or not to terminate the image capture mode and return to the list mode. The image capture mode termination confirmation screen 421 also has an "OK" button 422 and a "Cancel" button 423. When the user operates the "Cancel" button 423, the screen returns to the image capture standby screen 311 (see FIG. 16). When the user operates the "OK" button 422, the image capture mode is terminated and the screen returns to the list mode. At this time, the touch panel display 36 of the image capture device 1 transitions to the pause screen 301 (see FIG. 15). Meanwhile, the display 82 of the control device 2 transitions from the pause screen 221 (see FIG. 13) to the image capture data list screen 101 (see FIG. 8).

[0199] Second Embodiment Next, a second embodiment will be described. It should be noted that the points not specifically mentioned here are the same as those of the previous embodiment. Fig. 28 is a block diagram showing the schematic configuration of an image capturing device 1 and a control device 2 according to the second embodiment.

[0200] In the first embodiment, the photographing device 1 and the control device 2 are connected by wire, but in this embodiment, the photographing device 1 and the control device 2 are connected wirelessly. Specifically, the photographing device 1 includes a wireless communication unit 78. Furthermore, the control device 2 (information processing device) includes a wireless communication unit 88. The wireless communication unit 78 of the photographing device 1 and the wireless communication unit 88 of the control device 2 communicate wirelessly using an appropriate wireless communication method such as wireless LAN. In the example shown in FIG. 28 , an input / output interface 79 is provided to input and output data to and from a sensor unit 11 having a configuration similar to that of the first embodiment.

[0201] When the photographing device 1 and the control device 2 are wirelessly connected in this manner, the control device 2 can be left stationary in an appropriate location, such as the location to be measured or nearby, during photographing, eliminating the need for the user to carry the control device 2 around.

[0202] In the first and second embodiments, the photography system is composed of the photography device 1 held by the user's hand and the control device 2 carried by the user, but the display / input panel unit 12 of the photography device 1 may include all or part of the functions of the control device 2. In this case, the process of generating simple point cloud data (simplified point cloud generation process) for the user to check the photography situation (point cloud generation situation) requires a relatively small load and needs to be performed in real time during photography, and therefore may be performed by the photography device 1. Furthermore, the process of generating regular point cloud data to be acquired as a 3D measurement result (standard point cloud generation process) requires a relatively large load and only needs to be performed after photography, and therefore may be performed by a separately provided server device with high processing capabilities.

[0203] In the second embodiment, in which the image capturing device 1 and the control device 2 are wirelessly connected, the control device 2 may be configured on-premise or in the cloud. In the second embodiment, in which the image capturing device 1 having some of the functions of the control device 2 is wirelessly connected to a server device, the server device may be configured on-premise or in the cloud.

[0204] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments.

[0205] The imaging device disclosed herein has the advantage of allowing a user to easily continue taking images without placing a heavy burden on the user, even when the imaging range is wide or when imaging from a high altitude is required, and is useful as an imaging device for photographing a measurement target location in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location.

[0206] 1: Photographing device 2: Control device (information processing device) 11: Sensor section 12: Display input panel section 13: Support 21: First cable 22: Second cable 31: Main body section 32: Telescopic rod section 33: Sensor mounting section 34: Panel mounting section 35: Grip section 36: Touch panel display 41: Joint 42: Fixing knob 43: Joint 44: Fixing knob 51: Sensor unit 52: Sensor cover 53: Visible camera 54: Depth sensor 61: Cover body 62: Filter member 65: ND filter 101: Photographing data list screen 121: Photographing data details screen 231: Connection failure notification screen 241: Connection failure notification screen 311: Photographing standby screen 331: Photographing in progress screen 335: Elapsed time display section 336: Photographing time guide section 351: Photographing end confirmation screen 361: Photographing end confirmation screen 371: Self-location lost notification screen 381: Recovery in progress screen 411: Connection failure notification screen 421: Shooting mode end confirmation screen

Claims

1. An imaging device that performs three-dimensional measurement processing for generating three-dimensional spatial information of a measurement target location, the imaging device including: a sensor unit including a camera that images the measurement target location; a display input panel unit that displays a screen for assisting a user's imaging operation and detects a screen operation by the user; and a rod-shaped support that supports the sensor unit and the display input panel unit, wherein the support has a grip portion that can be held by the user with one hand, the sensor unit is fixed to an end portion of the support on a side opposite to the grip portion, and the display input panel unit is fixed to a position between the grip portion and the sensor unit on the support.

2. The imaging device according to claim 1, wherein the support has a telescopic rod portion that changes a position of the sensor unit with respect to the grip portion.

3. The imaging device according to claim 1, further comprising a cable that connects the sensor unit and the display input panel unit.

4. The imaging device according to claim 1, wherein the display input panel unit is fixed to the support so that its orientation can be adjusted in at least one of the pan and tilt directions.

5. The imaging device according to claim 1, wherein the display input panel unit includes a first connection terminal to which a first cable connecting the sensor unit and the display input panel unit is detachably coupled, and a second connection terminal to which a second cable connecting a control device that controls the device itself and the display input panel unit is detachably coupled.

6. The imaging device according to claim 1, further comprising a wireless communication unit that performs wireless communication with an information processing device that generates display information for causing a screen for assisting a user's imaging operation to be displayed on the touch panel display, wherein the wireless communication unit transmits imaging information including an imaging image captured by the camera and receives the display information from the information processing device.

7. The imaging device according to claim 1, wherein the camera performs color imaging, and the sensor unit has a neutral density filter that is movably provided between a position covering the lens of the camera and a position not covering the lens of the camera.

8. A photographing system for performing three-dimensional measurement processing for generating three-dimensional space information of a measurement target location, the system comprising a photographing device for photographing the measurement target location and a processor for controlling the photographing device, wherein the photographing device includes a sensor unit including a camera for photographing the measurement target location, a display input panel unit including a touch panel display for displaying a screen for assisting a user's photographing operation and detecting a screen operation by the user, and a support for supporting the sensor unit and the display input panel unit, the support having a grip portion that can be held by the user with one hand, the display input panel unit being disposed in the vicinity of the grip portion, and the processor being configured to display, on the touch panel display, an operation screen for which at least an instruction to start photographing is performed by the user.

9. The photographing system according to claim 8, wherein the processor is configured to display, on the touch panel display, a photographing standby screen, a photographing-in-progress screen, and a photographing completion confirmation screen as the operation screen so as to sequentially transition according to a user's instruction.

10. The photographing system according to claim 9, wherein the processor is configured to display the photographing standby screen including an operation unit for the user to select any one of a plurality of photographing modes having different control conditions of the camera.

11. The photographing system according to claim 9, wherein the processor is configured to display the photographing-in-progress screen including a first image display frame for enlarging and displaying either one of a photographed image by the camera and a photographing path image representing a path along which the photographing device has moved, a second image display frame for reducing and displaying the other image, and an operation unit for switching the images to be displayed in the first image display frame and the second image display frame.

12. The photographing system according to claim 9, wherein the processor is configured to display the photographing completion confirmation screen including an operation unit for notifying the user of saving of photographing data and return to the photographing standby screen at the end of photographing and for inputting that the user has confirmed it at the end of photographing.

13. The photographing system according to claim 9, wherein the processor is configured to display the photographing-in-progress screen including an image visualizing whether or not a speed at which the user moves the sensor unit is appropriate by measuring the speed.

14. The processor causes the in - shooting screen including a predetermined shootable time to be displayed at the start of shooting, and causes the in - shooting screen including the remaining shooting time to be displayed at a timing when the remaining shooting time obtained by subtracting the elapsed time from the start of shooting from the shootable time becomes equal to or less than a predetermined value during shooting. The imaging system according to claim 9, characterized in that.

15. When the processor detects a self - position loss in which the current position of the imaging device is lost in the tracking process, it notifies the user of the occurrence of the self - position loss, presents the final position acquisition point to the user, and causes the touch - panel display to display a screen prompting the user to return to the final position acquisition point. The imaging system according to claim 8, characterized in that.

16. The processor detects a connection failure on the communication path between the camera, the touch - panel display, and the processor, and causes the touch - panel display to display a screen notifying the user of the occurrence of the connection failure. The imaging system according to claim 8, characterized in that.

17. An imaging device for performing a three - dimensional measurement process for generating three - dimensional spatial information of a measurement target location, the imaging device including: a sensor unit including a camera for imaging the measurement target location; a display input panel unit including a touch - panel display for displaying a screen assisting the user's imaging operation and detecting a screen operation by the user; and a support for supporting the sensor unit and the display input panel unit, the support having a grip portion that can be held by the user with one hand, the display input panel unit being disposed in the vicinity of the grip portion, and the touch - panel display displaying at least an operation screen for the user to give an instruction to start shooting. The imaging device is characterized in that.

18. A shooting control method in which a processor executes a process of controlling a shooting device that shoots a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location, the method comprising: a sensor unit including a camera that shoots the measurement target location; a display input panel unit including a touch panel display that displays a screen for assisting a user's shooting operation and detects a screen operation by the user; and a support that supports the sensor unit and the display input panel unit, wherein the support has a grip portion that can be held by the user with one hand, and the display input panel unit causes the touch panel display in the shooting device disposed in the vicinity of the grip portion to display an operation screen on which the user performs an operation for instructing at least the start of shooting.

Images