Imaging system, imaging apparatus, and imaging control method

Abstract

To allow a user to easily perform screen operation for instructions or settings on imaging using a touch panel display, while securely holding an imaging apparatus even during the imaging.SOLUTION: An imaging apparatus 1 includes: a sensor unit 11 which includes a camera for imaging a measurement target place; a display input panel unit 12 including a touch panel display 36 for displaying a screen for supporting imaging operation of a user and detecting screen operation by the user; and a support body 13 which supports the sensor unit and the display input panel unit. The support body includes a grip unit 35 to be gripped with one hand of the user. The display input panel unit is arranged near the grip unit. A processor of a control unit 2 for controlling the imaging apparatus 1 displays, on the touch panel display, at least an operation screen to be used by the user for instruction to start imaging. Especially, the processor displays, as the operation screen, an imaging standby screen, an imaging-in-process screen, and an imaging end confirmation screen, so as to be sequentially changed according to an instruction from the user.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging system, an imaging device, and an imaging control method, which include an imaging device that captures an object to be measured and a processor that controls the imaging device, for performing three-dimensional measurement processing for generating three-dimensional spatial information of the object to be measured.

Background Art

[0002] There is known a three-dimensional measurement technique for generating a point cloud representing an object to be measured in a three-dimensional space based on a captured image of the object to be measured. In this three-dimensional measurement, in recent years, the SLAM (Simultaneous Localization And Mapping) method has attracted attention. In the SLAM method, a moving object (for example, an operator) holds an imaging device, and based on captured images of the object to be measured taken from various directions, self-position estimation for acquiring the position information of the own vehicle and mapping for generating a point cloud representing the object to be measured in a three-dimensional space are performed.

[0003] As a technique related to such a SLAM method, conventionally, an image of an arrow representing the moving direction and moving speed of an imaging device (sensor) held by a user, or an image for discriminating between the direction of a measured area and the direction of an unmeasured area is used to present an area with insufficient imaging to the user (see Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the prior art, as a photographing device for photographing a measurement target location, a tablet PC (measurement device main body) equipped with a distance sensor and a visible camera is used, and the user holds the photographing device with both hands and walks inside the measurement target location to photograph the measurement target location. Also, using the touch panel display provided in the photographing device, the user can perform screen operations for instructions and settings related to photographing.

[0006] However, there may be a case where, during photographing, it is desired to perform screen operations for instructions and settings related to photographing. In this case, in a configuration where the user holds the photographing device with both hands as in the prior art, screen operations during photographing are troublesome. That is, since it is necessary to release one hand that is holding the photographing device for the screen operation, the photographing device cannot be stably held, and the screen operation cannot be easily performed.

[0007] Therefore, a main object of the present invention is to provide a photographing system, a photographing device, and a photographing control method that can easily perform screen operations for instructions and settings related to photographing on a touch panel display while stably holding the photographing device even during photographing.

Means for Solving the Problem

[0008] The photographing system of the present invention is a photographing system including a photographing device that photographs a measurement target location and a processor that controls the photographing device in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location. The photographing device includes a sensor unit including a camera that photographs the measurement target location, a display input panel unit including a touch panel display that displays a screen for assisting the user's photographing work and detects a screen operation by the user, and a support that supports the sensor unit and the display input panel unit. The support has a grip portion that can be held by the user with one hand, the display input panel unit is disposed in the vicinity of the grip portion, and the processor is configured to display, on the touch panel display, at least an operation screen for the user to give an instruction to start photographing.

[0009] In addition, the imaging device of the present invention is an imaging device that images a measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location, and includes a sensor unit including a camera that images the measurement target location, a display input panel unit including a touch panel display that displays a screen for assisting a user's imaging operation and detects a screen operation by the user, and a support that supports the sensor unit and the display input panel unit. The support has a grip portion that can be held by the user with one hand, the display input panel unit is disposed in the vicinity of the grip portion, and the touch panel display is configured to display at least an operation screen for the user to give an instruction to start imaging.

[0010] In addition, the imaging control method of the present invention is an imaging control method in which a processor executes a process of controlling an imaging device that images a measurement target location in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location. The imaging device includes a sensor unit including a camera that images the measurement target location, a display input panel unit including a touch panel display that displays a screen for assisting a user's imaging operation and detects a screen operation by the user, and a support that supports the sensor unit and the display input panel unit. The support has a grip portion that can be held by the user with one hand, and the display input panel unit is configured to display an operation screen for the user to perform at least an operation instructing the start of imaging on the touch panel display disposed in the vicinity of the grip portion of the imaging device.

Advantages of the Invention

[0011] According to the present invention, by gripping the grip portion with one hand, the imaging device can be stably held with one hand, so that a screen operation for instructions and settings related to imaging on the touch panel display can be easily performed with the other hand.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Embodiments for Carrying Out the Invention

[0013] A first invention made to solve the above problems is an imaging system including an imaging device that images a measurement target location and a processor that controls the imaging device in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location. The imaging device includes a sensor unit including a camera that images the measurement target location, a display input panel unit including a touch panel display that displays a screen for assisting the user's imaging operation and detects a screen operation by the user, and a support that supports the sensor unit and the display input panel unit. The support has a grip portion that can be held by the user with one hand, the display input panel unit is disposed in the vicinity of the grip portion, and the processor is configured to display, on the touch panel display, at least an operation screen for the user to give an instruction to start imaging.

[0014] According to this, by gripping the grip portion with one hand, the imaging device can be stably held with one hand, so that the screen operation for instructions and settings regarding imaging on the touch panel display can be easily performed with the other hand. Note that the processor that controls the imaging device may be provided in a control device separate from the imaging device, or may be provided in the imaging device itself.

[0015] Further, in the second invention, the processor is configured to cause the touch panel display to sequentially display a shooting standby screen, a shooting in-progress screen, and a shooting completion confirmation screen as the operation screen in response to a user's instruction.

[0016] According to this, the user can appropriately perform the necessary settings and confirmations at each stage before shooting, during shooting, and after shooting.

[0017] Further, in the third invention, the processor is configured to display the shooting standby screen including an operation unit for the user to select any one of a plurality of shooting modes with different control conditions of the camera.

[0018] According to this, before starting shooting, the user can specify a shooting mode according to the characteristics of the measurement target location. In this case, either an indoor shooting mode or an outdoor shooting mode may be selectable. Further, either one of two outdoor shooting modes according to the presence or absence of a neutral density filter may be selectable.

[0019] Further, in the fourth invention, the processor includes a first image display frame that enlarges and displays either one of the captured image by the camera and the shooting path image representing the path along which the imaging device has moved, a second image display frame that reduces and displays the other image, and an operation unit that switches the images to be displayed in the first image display frame and the second image display frame, and is configured to display the shooting in-progress screen.

[0020] According to this, the user can switch between an image to be enlarged and displayed as needed and an image to be reduced and displayed. Note that the second image display frame itself may be an operation unit.

[0021] Further, in the fifth invention, when the shooting is finished, the processor notifies the user of the saving of the shooting data and the return to the shooting standby screen, and displays the shooting end confirmation screen including an operation unit for the user to input that the user has confirmed.

[0022] According to this, the user can easily confirm that the shooting has ended.

[0023] Further, in the sixth invention, the processor measures the speed at which the user moves the sensor unit, and displays the shooting in - progress screen including an image visualizing whether or not the speed is appropriate.

[0024] According to this, the user can easily grasp whether or not the speed at which the sensor unit is moved is appropriate.

[0025] Further, in the seventh invention, at the start of shooting, the processor displays the shooting in - progress screen including a predetermined shooting - available time, and during shooting, at the timing when the remaining shooting time obtained by subtracting the elapsed time from the start of shooting from the shooting - available time becomes equal to or less than a predetermined value, the processor displays the shooting in - progress screen including the remaining shooting time.

[0026] According to this, the user can grasp the shooting - available time at the start of shooting, and can also grasp the remaining shooting time at the timing when the remaining shooting time decreases.

[0027] Further, in the eighth invention, when the processor detects a self - position loss in which the current position of the imaging device is lost in the tracking process, the processor notifies the user of the occurrence of the self - position loss, presents the final position acquisition point to the user, and displays on the touch - panel display a screen prompting the user to return to the final position acquisition point.

[0028] According to this, when self-position loss occurs, the user can quickly return to the final position acquisition point and reshoot.

[0029] Also, in the ninth invention, the processor detects a connection failure on the communication path between the camera, the touch panel display, and the processor, and causes the touch panel display to display a screen notifying the user of the occurrence of the connection failure.

[0030] According to this, the user can easily grasp that the imaging device does not operate normally due to a connection failure.

[0031] Also, the tenth invention is an imaging device that performs three-dimensional measurement processing for generating three-dimensional space information of a measurement target location, including a sensor unit including a camera that images the measurement target location, a display input panel unit that displays a screen for assisting the user's imaging operation and detects a screen operation by the user, and a support that supports the sensor unit and the display input panel unit, wherein the support has a grip portion that can be held by the user with one hand, the display input panel unit is disposed in the vicinity of the grip portion, and the touch panel display is configured to display at least an operation screen for the user to give an instruction to start imaging.

[0032] According to this, similar to the first invention, by holding the grip portion with one hand, the imaging device can be stably held with one hand, so that screen operations for instructions and settings related to imaging on the touch panel display can be easily performed with the other hand.

[0033] Further, the eleventh invention is a shooting control method in which a processor executes a process of controlling a shooting device that shoots a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location. The shooting control method includes a sensor unit including a camera that shoots the measurement target location, a display input panel unit including a touch panel display that displays a screen for assisting the user's shooting operation and detects a screen operation by the user, and a support that supports the sensor unit and the display input panel unit. The support has a grip portion that can be held by the user with one hand, and the display input panel unit is configured to display an operation screen on which the user performs an operation for instructing at least the start of shooting with respect to the touch panel display in the shooting device disposed in the vicinity of the grip portion.

[0034] According to this, as in the first invention, by holding the grip portion with one hand, the shooting device can be stably held with one hand, so that the screen operation for instructions and settings related to shooting on the touch panel display can be easily performed with the other hand.

[0035] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0036] (First Embodiment) FIG. 1 is an explanatory diagram showing a situation of a shooting operation performed by a user using a shooting system according to the first embodiment. FIG. 2 is an explanatory diagram showing a standard shooting state and an elevated shooting state.

[0037] As shown in FIG. 1, the shooting system includes a shooting device 1 and a control device 2 (information processing device).

[0038] The shooting device 1 includes a sensor unit 11, a display input panel unit 12, and a rod-shaped support 13. The sensor unit 11 and the display input panel unit 12 of the shooting device 1 are connected via a first cable 21. The display input panel unit 12 of the shooting device 1 and the control device 2 are connected via a second cable 22.

[0039] The control device 2 is composed of a laptop or tablet PC that can be carried by a user (operator). In the example shown in FIG. 1, the control device 2 is stored in a shoulder bag and can be carried by the user, but the mode in which the user carries the control device 2 is not limited to this.

[0040] The user performs a photographing operation of holding the photographing device 1 by hand and making the photographing device 1 photograph the measurement target location while walking around the measurement target location. At this time, the user can change the position (height) of the sensor unit 11 by moving the arm holding the photographing device 1.

[0041] For example, as shown in FIG. 2(A), the user can perform photographing by arranging the sensor unit 11 at approximately the same position as the height of the user's line of sight by slightly extending the arm holding the photographing device 1 (standard photographing state). On the other hand, as shown in FIG. 2(B), the user can perform photographing by arranging the sensor unit 11 at a position higher than the user's line of sight by raising the arm holding the photographing device 1 (high-place photographing state).

[0042] Next, the support 13 of the photographing device 1 will be described. FIG. 3 is a perspective view showing the support 13. FIG. 4 is a side view showing the expansion and contraction state of the support 13.

[0043] As shown in FIG. 3, the support 13 of the photographing device 1 includes a main body portion 31, a telescopic rod portion 32, a sensor mounting portion 33, a panel mounting portion 34, and a grip portion 35. The display input panel portion 12 includes a touch panel display 36.

[0044] The panel mounting portion 34 is provided on the main body portion 31. The panel mounting portion 34 includes a joint 41 (ball joint) that connects the display input panel portion 12 and the main body portion 31. By this joint 41, the orientation of the display input panel portion 12, that is, the orientation of the display surface of the touch panel display 36 can be changed. Specifically, the display input panel portion 12 rotates about two axes A1 and A2 passing through the center of the joint 41 (ball joint), thereby causing the display input panel portion 12 to perform a tilt operation of changing the orientation vertically and a pan operation of changing the orientation horizontally. Also, a fixing knob 42 is provided on the panel mounting portion 34. By the user operating the fixing knob 42, the orientation of the display input panel portion 12 can be fixed. In the example shown in FIG. 3, since the joint 41 is a ball joint, the orientation of the display input panel portion 12 can be adjusted in an arbitrary direction within a predetermined angle range.

[0045] In this way, since the user can appropriately adjust the orientation of the display input panel portion 12, the visibility and operability of the touch panel display 36 are improved. For example, when changing the angle of the rod-shaped support 13 in the case of high-altitude shooting, if the orientation of the display input panel portion 12 is adjusted in the tilt direction, browsing and operating the screen of the touch panel display 36 become easier. Also, for example, when the user holds the photographing device 1 with the right hand, if the orientation of the display input panel portion 12 is adjusted in the pan direction so that the screen of the touch panel display 36 faces the user's face or the left side where the left hand for screen operation is located, browsing and operating the screen of the touch panel display 36 become easier.

[0046] The sensor attachment part 33 is provided at the tip of the telescopic rod part 32. The sensor attachment part 33 includes a joint 43 (hinge joint) that connects the sensor part 11 and the tip of the telescopic rod part 32. By this joint 43, the orientation of the sensor part 11 can be changed. Specifically, by rotating the sensor part 11 about the central axis A3 of the joint 43 (hinge joint), the sensor part 11 can be made to perform a tilt operation that changes the orientation of the sensor part 11 up and down. A fixing knob 44 is provided on the sensor attachment part 33. By the user operating the fixing knob 44, the orientation of the sensor part 11 can be fixed.

[0047] In the example shown in FIG. 3, in the panel attachment part 34, the display input panel part 12 can perform a tilt operation and a pan operation by a joint 41 composed of a ball joint, but the display input panel part 12 may be made capable of performing a tilt operation and a pan operation by a combination of a plurality of hinge joints.

[0048] Also, in the example shown in FIG. 3, in the sensor attachment part 33, only a tilt operation that changes the orientation of the sensor part 11 up and down is possible by a joint 43 composed of a hinge joint, but a pan operation that changes the orientation of the sensor part 11 left and right may be made possible. In this case, the joint 43 may be composed of a ball joint or a combination of a plurality of hinge joints.

[0049] The telescopic rod part 32 is configured to be telescopable in the axial direction. In the example shown in FIG. 4, the telescopic rod part 32 has a structure in which a plurality of sheath tubes with gradually decreasing diameters are slidably connected.

[0050] The user can change the position of the sensor unit 11 relative to the grip portion 35 held by hand by extending and retracting the telescopic rod portion 32. That is, as shown in FIG. 4(A), in the state where the telescopic rod portion 32 is extended (extended state), the sensor unit 11 is separated from the grip portion 35. For this reason, in high-place shooting (see FIG. 2(B)), the sensor unit 11 can be arranged at a high position. On the other hand, as shown in FIG. 4(B), in the state where the telescopic rod portion 32 is contracted and most of the telescopic rod portion 32 is housed in the main body portion 31 (stored state), the overall length of the imaging device 1 becomes shorter, so that it becomes easier to carry and store the imaging device 1 when not in use.

[0051] As described above, in this embodiment, the user can stably hold the imaging device 1 with one hand by gripping the grip portion 35 with the hand. For this reason, the user can perform shooting with the sensor unit 11 directed in an arbitrary direction by changing the orientation of the arm or body that holds the imaging device 1 during shooting. Also, during shooting, the user can extend and retract the telescopic rod to adjust the position of the sensor unit 11, and can also adjust the orientation of the display input panel portion 12.

[0052] In this embodiment, the display input panel portion 12 is fixed to the main body portion 31, but a configuration may also be adopted in which the display input panel portion 12 is fixed in the middle of the telescopic rod portion 32 so that the position of the display input panel portion 12 relative to the grip portion 35 can be adjusted. Further, the display input panel portion 12 may be fixed to the main body portion 31 via a movable member (not shown) different from the telescopic rod portion 32 to which the sensor unit 11 is fixed at the tip.

[0053] Next, the sensor unit 11 of the imaging device 1 will be described. FIG. 5 is a perspective view showing the sensor unit 11.

[0054] The sensor unit 11 includes a sensor unit 51 and a sensor cover 52.

[0055] The sensor unit 51 includes a visible camera 53 and a depth sensor 54. The visible camera 53 (color camera) captures a subject in color. The depth sensor 54 includes an infrared projector 55 and left and right infrared cameras 56, 57 (stereo cameras). The infrared projector 55 irradiates the subject with infrared light. The left and right infrared cameras 56, 57 detect the reflected light of the infrared light irradiated on the subject. Note that, in addition to the stereo camera, a sensor capable of acquiring depth information of other methods such as LiDAR (Light Detection And Ranging) may be used for the depth sensor 54.

[0056] In the sensor unit 51, the visible camera 53, the left and right infrared cameras 56, 57, and the infrared projector 55 are arranged side by side in the lateral direction (substantially horizontal direction). The housing of the sensor unit 51 has a long shape in the lateral direction.

[0057] The sensor cover 52 has a cover body 61 and a filter member 62.

[0058] The cover body 61 covers the sensor unit 51. During shooting, when the imaging device 1 is being moved, the sensor unit 11 may collide with surrounding objects, and the cover body 61 is provided to protect the sensor unit 51 from the impact of this collision. The cover body 61 has an opening 66 that opens the cover glass on the front side of the visible camera 53, the infrared projector 55, and the infrared cameras 56, 57.

[0059] The filter member 62 is slidably attached to the cover body 61 in the longitudinal direction of the sensor unit 51, that is, in the arrangement direction of the visible camera 53, the infrared projector 55, and the infrared cameras 56, 57. The filter member 62 includes a first ring portion 63 and a second ring portion 64. An ND (Neutral Density) filter 65 (attenuation filter) is attached to the first ring portion 63.

[0060] The filter member 62 is provided slidably between a first position shown in Fig. 5(A) and a second position shown in Fig. 5(B). The user can grip and slide the filter member 62 with a finger. In the first position shown in Fig. 5(A), the ND filter 65 does not cover the visible camera 53. In the second position shown in Fig. 5(B), the ND filter 65 covers the visible camera 53.

[0061] Here, in the first position shown in Fig. 5(A), the first ring portion 63 to which the ND filter 65 is attached is disposed at a position between the visible camera 53 and one of the infrared cameras 57, and the second ring portion 64 is disposed at a position surrounding the lens of one of the infrared cameras 57. Thereby, none of the visible camera 53, the infrared projector 55, and the infrared cameras 56, 57 is covered by the filter member 62.

[0062] On the other hand, in the second position shown in Fig. 5(B), the first ring portion 63 is disposed at a position corresponding to the visible camera 53, and the second ring portion 64 is disposed at a position between the visible camera 53 and one of the infrared cameras 57. Thereby, only the visible camera 53 is covered by the ND filter 65, and none of the infrared projector 55 and the infrared cameras 56, 57 is covered by the filter member 62.

[0063] Note that the second ring portion 64 abuts against the inner peripheral portion of the opening of the cover body 61, whereby the filter member 62 is positioned at the first position shown in Fig. 5(A). Further, a groove (not shown) is formed long in the moving direction of the filter member 62 on the lower surface side of the cover body 61, and the protrusion provided on the filter member 62 fits into this groove, whereby the moving range of the filter member 62 is defined and the filter member 62 is positioned at the second position shown in Fig. 5(B).

[0064] In the case of outdoor photography, the user adjusts the filter member 62 to the second position (see FIG. 5(B)) where the ND filter 65 covers the visible camera 53. On the other hand, in the case of indoor photography, the user adjusts the filter member 62 to the first position (see FIG. 5(A)) where the ND filter 65 does not cover the visible camera 53. Thereby, the brightness of the entire captured image can be adjusted by suppressing the amount of light input to the camera in the case of outdoor photography. Also, it is possible to avoid the problem that purple coloring occurs in the entire captured image. Further, since only the visible camera 53 (color camera) is covered with the ND filter 65, it is possible to avoid the problem that the density of the generated point cloud decreases due to the infrared cameras 56 and 57 being covered with the ND filter 65.

[0065] Next, the schematic configurations of the imaging device 1 and the control device 2 will be described. FIG. 6 is a block diagram showing the schematic configurations of the imaging device 1 and the control device 2. FIG. 7 is a block diagram showing an outline of the processing performed by the processor 86 of the control device 2.

[0066] As shown in FIG. 6, the sensor unit 11 of the imaging device 1 includes a visible camera 53, a depth sensor 54, an IMU 71 (Inertial Measurement Unit), an input / output interface 72, and a connection terminal 73.

[0067] The visible camera 53 (color camera) performs color photography and outputs a color captured image.

[0068] The depth sensor 54 outputs depth information (distance image) as a detection result based on the captured images by the left and right infrared cameras 56 and 57 (see FIG. 5).

[0069] The IMU 71 detects the motion state of the own device, specifically, three-dimensional angular velocity and acceleration. Based on the detection result of the IMU 71, the position, orientation, and velocity of the sensor unit 11 can be detected.

[0070] The input / output interface 72 performs data input / output with the control device 2 via the display input panel unit 12. Specifically, the detection data of the visible camera 53, the depth sensor 54, and the IMU 71 are transmitted. The input / output interface 72 may be based on the USB (registered trademark) standard.

[0071] A first cable 21 that connects the sensor unit 11 and the display input panel unit 12 is detachably coupled to the connection terminal 73.

[0072] Note that the visible camera 53, the depth sensor 54, and the IMU 71 may not be integrated as a sensor unit 51 (see FIG. 5). Also, a configuration in which the depth sensor 54 and the IMU 71 are omitted and only the visible camera 53 is provided may be used. Further, a configuration in which either the depth sensor 54 or the IMU 71 and the visible camera 53 are provided may be used.

[0073] The display input panel unit 12 of the imaging device 1 includes a touch panel display 36, a repeater 74, an input / output interface 75, a first connection terminal 76, and a second connection terminal 77.

[0074] The touch panel display 36 displays a screen or the like that supports the user's imaging operation based on the control by the control device 2.

[0075] The repeater 74 relays data communication between the control device 2 and the sensor unit 11. The repeater 74 may be a hub based on the USB (registered trademark) standard.

[0076] The input / output interface 75 performs data input / output with the control device 2. Specifically, it receives display information such as a screen that supports the user's imaging operation from the control device 2. The input / output interface 75 may be based on the USB (registered trademark) standard.

[0077] A first cable 21 for connecting the sensor unit 11 and the display input panel unit 12 is detachably coupled to the first connection terminal 76. A second cable 22 for connecting the display input panel unit 12 and the control device 2 is detachably coupled to the second connection terminal 87.

[0078] Incidentally, the first cable 21 is used for data communication between the control device 2 and the sensor unit 11 via the display input panel unit 12. In this data communication, for example, shooting data (shooting information) output from the sensor unit 11 is transmitted to the control device 2. The shooting data includes a captured image of the visible camera 53, a detection result (distance information) of the depth sensor 54, a detection result of the IMU 71, and a shooting time. The second cable 22 is used for data communication between the control device 2 and the display input panel unit 12 and data communication between the control device 2 and the sensor unit 11. In the data communication between the control device 2 and the display input panel unit 12, for example, display information of a screen to be displayed on the touch panel display 36 is transmitted from the control device 2, and operation information of the user detected by the touch panel display 36 is transmitted to the control device 2.

[0079] The control device 2 includes an input / output interface 81, a display 82, an input device 83, a memory 84, a storage device 85, a processor 86, and a connection terminal 87.

[0080] The input / output interface 81 inputs and outputs data to and from the imaging device 1. The input / output interface 81 may be based on the USB (registered trademark) standard.

[0081] The display 82 displays a screen related to the management of shooting data acquired in the past, a screen related to the setting of the operation conditions of the imaging device 1, and the like.

[0082] The input device 83 is for the user to perform input operations. The input device 83 may be a keyboard, a mouse, a touch pad, a touch panel, or the like. When the control device 2 is configured as a tablet PC, a touch panel display 36 in which a touch panel as the input device 83 and a display panel as the display 82 are integrated is provided.

[0083] A second cable 22 for connecting the control device 2 and the display input panel unit 12 of the imaging device 1 is detachably coupled to the connection terminal 87.

[0084] The memory 84 stores programs executed by the processor 86 and the like.

[0085] The storage device 85 stores the captured data (imaging information) acquired from the imaging device 1. The captured data includes a captured image of the visible camera 53, a detection result (distance information) of the depth sensor 54, a detection result of the IMU 71, and the imaging time. In addition, the storage device 85 stores point cloud data as a three-dimensional measurement result generated by the processor 86.

[0086] The processor 86 performs various processes by executing the programs stored in the memory 84. In this embodiment, the processor 86 performs three-dimensional measurement processing P1.

[0087] In the three-dimensional measurement process P1, the processor 16 generates point cloud data (environmental map) as three-dimensional space information regarding the measurement target location using the SLAM (Simultaneous Localization And Mapping) method based on a captured image by the visible camera 53 or the like. In addition, in the three-dimensional measurement process P1, self-position estimation is performed in conjunction with the generation of the point cloud data, and the self-position at each time, that is, the position of the imaging point, is acquired.

[0088] As shown in FIG. 7, the three-dimensional measurement process P1 includes a feature extraction process P11, a tracking process P12, a position and attitude correction process P13, and a point cloud generation process P14.

[0089] In the feature extraction process P11, the processor 86 extracts feature information (such as feature points) from the captured image (frame) of the visible camera 53.

[0090] In the tracking process P12, the processor 86 compares the feature points extracted this time with the feature points extracted previously, estimates the transition amount regarding the position and orientation of the imaging device 1, and updates the position and orientation trajectory data based on the transition amount. The position and orientation trajectory data is the result of tracking the position and orientation of the imaging device 1, and includes the tracking results of the position and orientation at the time of capturing each captured image (frame), that is, information regarding the position and orientation of the imaging device 1 at each time of imaging.

[0091] In the position and orientation correction process P13, the processor 86 corrects the position and orientation trajectory data obtained in the tracking process P12 based on the detection data of the IMU 23. Here, for example, the position and orientation trajectory data is corrected to supplement the measurement results in locations with few features such as walls and ceilings.

[0092] In the point cloud generation process P14, the processor 86 generates point cloud data based on the distance information of the depth sensor 54 and the position and orientation trajectory data obtained in the tracking process P12 and the position and orientation correction process P13. In addition, the point cloud generation process P14 includes a process of generating normal point cloud data to be obtained as a three-dimensional measurement result (standard point cloud generation process) and a process of generating simple point cloud data for the user to confirm the imaging situation (point cloud generation situation) (simple point cloud generation process). The simple point cloud generation process needs to be performed in real time during imaging, but the standard point cloud generation process may be performed after imaging.

[0093] Also, as shown in FIG. 6, the processor 86 performs the drawing generation process P2, the display information generation process P3, and the display process P4.

[0094] In the drawing generation process P2, the processor 86 generates a three-dimensional model of the target location based on the point cloud data of the target location generated by the three-dimensional measurement process P1, and generates a layout drawing of the target location based on the three-dimensional model of the target location. At this time, a two-dimensional layout drawing (plan view) of the target location is generated by projecting the three-dimensional model onto the horizontal plane. In addition, a three-dimensional layout drawing of the target location is generated by projecting the three-dimensional model based on a predetermined line-of-sight direction.

[0095] In the display information generation process P3, the processor 86 generates the display information of the screen to be displayed on the display 82 of the control device 2. On the display 82, a screen (see FIGS. 8 to 14) for the user to perform operations such as management of the shooting data, instructions for post-shooting processing (point cloud generation process P14, drawing generation process P2), and settings regarding processing conditions, etc. is displayed. In addition, the processor 86 generates the display information of the screen to be displayed on the touch panel display 36 of the imaging device 1. On the touch panel display 36, a screen (see FIGS. 15 to 19, FIGS. 21 to 27) for assisting the user's shooting operation is displayed.

[0096] In the display process P4, the processor 86 displays a screen (see FIGS. 8 to 14) on the display 82 of the control device 2 based on the display information generated in the display information generation process P3. In addition, the processor 86 displays a screen (see FIGS. 15 to 19, FIGS. 21 to 27) on the touch panel display 36 of the imaging device 1.

[0097] Incidentally, in the present embodiment, a shooting standby screen 311 (see FIG. 16), a shooting in-progress screen 331 (see FIG. 17), and shooting end confirmation screens 351, 361 (see FIG. 21) are displayed on the touch panel display 36 of the imaging device 1 so as to sequentially transition according to the user's instructions. Thereby, the user can appropriately perform necessary settings and confirmations at each stage before shooting, during shooting, and after shooting.

[0098] In addition, in the present embodiment, a non-shooting mode (first mode) and a shooting mode (second mode) are switched according to the user's operation.

[0099] In the non-photographing mode, as an operable screen on the display 82 of the control device 2, a screen (see FIGS. 8 to 14) for the user to manage and process photographic data and give instructions is displayed, and a non-operable standby screen 301 (see FIG. 15) is displayed on the touch panel display 36 of the photographing device 1.

[0100] On the other hand, in the photographing mode, photographing using the photographing device 1 becomes possible, and as an operable screen on the touch panel display 36 of the photographing device 1, a screen (see FIGS. 15 to 19, FIGS. 21 to 27) for assisting the user's photographing work, for example, a screen (see FIGS. 16 and 17) for giving instructions to start and end photographing is displayed, and a non-operable standby screen 221 (see FIG. 13) is displayed on the display 82 of the control device 2.

[0101] Also, as shown in FIG. 6, the processor 86 performs a connection failure detection process P5, a speed presentation process P6, a photographing time presentation process P7, and a self-position loss recovery process P8.

[0102] In the connection failure detection process P5, the processor 86 detects a connection failure on the communication path between the sensor unit 51 and the touch panel display 36 and the processor 86, and notifies the user of the occurrence of the connection failure. Specifically, connection failure notification screens 231, 241 (see FIG. 14) are displayed on the display 82 of the control device 2. Also, a connection failure notification screen 411 (see FIG. 26) is displayed on the touch panel display 36 of the photographing device 1.

[0103] In the speed presentation process P6, the processor 86 measures the speed at which the user moves the sensor unit 11 based on the detection data (acceleration, angular velocity) of the IMU 71, visualizes whether the speed is appropriate, and presents it to the user. Specifically, a speed bar 334 is displayed on the photographing screen 331 (see FIG. 17).

[0104] In the shooting time prompt process P7, the processor 86 prompts the user with a preset shootable time at the start of shooting. Also, during shooting, when the remaining shooting time obtained by subtracting the elapsed time from the start of shooting from the shootable time reaches a predetermined time or less, the processor 86 prompts the user with the remaining shooting time. Specifically, on the shooting screen 331 (see FIG. 18) during shooting, the shootable time is prompted at the start of shooting, and then the remaining shooting time is prompted during shooting.

[0105] In the self-position lost return process P8, when the processor 86 detects a self-position loss (tracking loss) in which the current position of the imaging device 1 is lost in the tracking process P12, it notifies the user of the occurrence of the self-position loss and prompts the user with the final position acquisition point to guide the user to return to the final position acquisition point. Specifically, a self-position loss notification screen 371 (see FIG. 22) is displayed, and then a returning screen 381 (see FIG. 23) including a captured image of the final position acquisition point is displayed. Also, when the processor 86 detects that the imaging device 1 has returned to the final position acquisition point, the normal tracking process P12 is restarted.

[0106] Next, the captured data list screen 101 displayed on the display 82 of the control device 2 will be described. FIG. 8 is an explanatory diagram showing the captured data list screen 101.

[0107] When the shooting application is launched in the control device 2, the captured data list screen 101 is displayed on the display 82.

[0108] On the captured data list screen 101, a list display section 102 is provided. In the list display section 102, captured images are listed for each of a plurality of captured data with different measurement locations and capture dates and times. In the example shown in FIG. 8, the captured images for each captured data are arranged and displayed by date. Also, in the list display section 102, the capture date and time is displayed for each captured image. Further, in the list display section 102, when the point cloud generation process P14 has been completed, a check mark 111 indicating that fact is displayed, and when the drawing generation process P2 has been completed, a check mark 112 indicating that fact is displayed.

[0109] When the user selects captured data by operating any one of the captured images for each captured data displayed in the list display section 102, the user transitions to a captured data details screen 121 (see FIG. 9) related to the selected captured data. At this time, the user can select the captured data to be deleted or the captured data for which the point cloud generation process P14 and the drawing generation process P2 are to be executed.

[0110] In addition, when there is no registered captured data, a message indicating that there is no captured data, for example, the characters "There is no captured data", is displayed in the list display section 102.

[0111] Also, on the captured data list screen 101, a setting button 103 is provided. When the user operates the setting button 103, a setting screen 181 (see FIG. 12(A)) is pop-up displayed on the captured data list screen 101.

[0112] Also, on the captured data list screen 101, a button 104 for "starting shooting" is provided. When the user operates the button 104 for "starting shooting", the mode transitions from the non-shooting mode to the shooting mode. At this time, on the display 82 of the control device 2, the screen transitions to a standby screen 221 (see FIG. 13). On the other hand, on the touch panel display 36 of the imaging device 1, the screen transitions from a standby screen 301 (see FIG. 15) to a shooting standby screen 311 (see FIG. 16).

[0113] Next, the captured data details screen 121 displayed on the display 82 of the control device 2 will be described. FIG. 9 is an explanatory diagram showing the captured data details screen 121. FIG. 10 is an explanatory diagram showing a screen that pops up on the captured data details screen 121. FIG. 11 is an explanatory diagram showing the transition status of the main part of the captured data details screen 121.

[0114] The captured data details screen 121 is provided with an image display section 122 and a shooting date and time display section 123. In the image display section 122, the captured images (key frames) included in the captured data are played back in order. The shooting date and time of the captured image displayed on the image display section 122 is displayed on the shooting date and time display section 123. In the SLAM method, a frame in which a large change appears in the visual field is extracted as a key frame, and the key frame is reflected in the point cloud.

[0115] In addition, the captured data details screen 121 is provided with a playback operation section 124. The playback operation section 124 is provided with a seek bar 131, a playback stop button 132, a frame advance button 133, and a frame rewind button 134. The user can arbitrarily specify the playback start time by operating the slider of the seek bar 131. Also, the user can instruct the playback and stop of the captured data by operating the playback stop button 132. Further, the user can instruct the frame advance and frame rewind of the captured data by operating the frame advance button 133 and the frame rewind button 134.

[0116] In addition, the captured data details screen 121 is provided with an instruction operation section 125. The instruction operation section 125 is provided with a "point cloud generation" button 141, a "point cloud display" button 142, a "drawing generation" button 143, and a "drawing display" button 144.

[0117] When the user operates the "Point Cloud Generation" button 141, the point cloud generation process P14 (standard point cloud generation process) is started, and the point cloud generation in-progress screen 151 shown in FIG. 10(A) is pop-up displayed on the captured data details screen 121. A message indicating that the point cloud generation process P14 is in progress is displayed on the point cloud generation in-progress screen 151. Also, a progress bar 152 is provided on the point cloud generation in-progress screen 151. In the progress bar 152, the progress of the point cloud generation process P14 is visualized. Also, a button 153 for "Abort Point Cloud Generation" is provided on the point cloud generation in-progress screen 151. When the user operates the button 153 for "Abort Point Cloud Generation", the point cloud generation process P14 is aborted and returns to the captured data details screen 121 (see FIG. 9). Also, when the point cloud generation process P14 ends, it returns to the captured data details screen 121.

[0118] Also, on the captured data details screen 121 shown in FIG. 9, when the user operates the "Point Cloud Display" button 142, the point cloud display application is launched, and the point cloud data generated by the point cloud generation process P14 is displayed.

[0119] Also, when the user operates the "Drawing Generation" button 143, the drawing generation in-progress screen 161 shown in FIG. 10(B) is pop-up displayed on the captured data details screen 121. A message indicating that the drawing generation process P2 is in progress is displayed on the drawing generation in-progress screen 161. Also, a button 162 for "Abort Drawing Generation" is provided on the drawing generation in-progress screen 161. When the user operates the button 162 for "Abort Drawing Generation", the drawing generation process P2 is aborted and returns to the captured data details screen 121 (see FIG. 9). Also, when the drawing generation process P2 ends, it returns to the captured data details screen 121. Note that an animation indicating that the drawing generation process P2 is in progress may be displayed on the drawing generation in-progress screen 161.

[0120] Also, on the captured data details screen 121 shown in FIG. 9, when the user operates the "Drawing Display" button 144, the drawing display application is launched, and the drawing generated by the drawing generation process P2 is displayed.

[0121] In addition, on the captured data details screen 121, a "Delete" button 145 is provided. When the user operates the "Delete" button 145, a deletion confirmation screen 171 shown in FIG. 10(C) pops up and is displayed on the captured data details screen 121. A message for confirming the deletion of the captured data is displayed on the deletion confirmation screen 171. Also, on the deletion confirmation screen 171, a "Delete" button 172 and a "Cancel" button 173 are provided. When the user operates the "Delete" button 172, the captured data is deleted and the process returns to the captured data list screen 101 (see FIG. 8). Also, when the user operates the "Cancel" button 173, the process returns to the captured data details screen 121 (see FIG. 9).

[0122] Here, as shown in FIG. 11, in the instruction operation unit 125, the states of the "Point cloud generation" button 141, the "Point cloud display" button 142, the "Drawing generation" button 143, and the "Drawing display" button 144 change before and after the point cloud generation process P14 and the drawing generation process P2.

[0123] First, as shown in FIG. 11(A), before the execution of the point cloud generation process P14, only the "Point cloud generation" button 141 is operable, and the "Point cloud display" button 142, the "Drawing generation" button 143, and the "Drawing display" button 144 are inoperable. When the point cloud generation process P14 ends, as shown in FIG. 11(B), a check mark 146 indicating that the point cloud generation process P14 has been executed is displayed. Also, in this state, the "Point cloud display" button 142 and the "Drawing generation" button 143 are operable, and the "Drawing display" button 144 is inoperable. When the drawing generation process P2 ends, as shown in FIG. 11(C), a check mark 147 indicating that the drawing generation process P2 has been executed is displayed. Also, in this state, the "Drawing display" button 144 is operable.

[0124] In addition, as shown in FIG. 9, on the captured data details screen 121, a "Return to list" button 126 is provided. When the user operates the "Return to list" button 126, the process returns to the captured data list screen 101 (see FIG. 8).

[0125] Next, the setting screen 181 displayed on the display 82 of the control device 2 will be described. FIG. 12 is an explanatory diagram showing the setting screen 181.

[0126] In the captured data list screen 101 (see FIG. 8), when the user operates the setting button 103, the setting screen 181 shown in FIG. 12(A) is pop-up displayed on the captured data list screen 101.

[0127] The setting screen 181 is provided with a save folder setting section 182. In the save folder setting section 182, a folder path indicating the location where the setting file storing the setting information is saved is displayed. Further, the save folder setting section 182 is provided with a "change" button 191. When the user operates the "change" button 191, a screen for specifying a folder (not shown) is pop-up displayed, and here, the user can specify the folder where the setting file is to be saved.

[0128] Also, the setting screen 181 is provided with a camera setting section 183. The camera setting section 183 is provided with input sections 192, 193, and 194 for each item of the resolution of the visible camera 53, the resolution of the depth sensor 54, and the frame rate of the visible camera 53. When the user operates each of the input sections 192, 193, and 194, a pull-down menu (not shown) is displayed. The user can select the resolution of the visible camera 53, the resolution of the depth sensor 54, and the frame rate of the visible camera 53 in the pull-down menu.

[0129] In addition, the setting screen 181 is provided with a device information display section 184. In the device information display section 184, as information regarding the sensor unit 51, registration information regarding each item of the name (camera name), serial number, and firmware version of the sensor unit 51, and the date and time when the previous calibration process was performed are displayed. Further, the device information display section 184 is provided with a "forced restart" button 195. When the user operates the "forced restart" button 195, the sensor unit 51 is forcibly restarted. Also, the device information display section 184 is provided with an "execute" button 196. When the user operates the "execute" button 196, the device calibration screen 201 shown in FIG. 12(B) is pop-up displayed on the setting screen 181.

[0130] In addition, the setting screen 181 is provided with a "complete" button 185. When the user operates the "complete" button 185, the setting process is executed with the input content of the setting screen 181, and the user returns to the shooting data list screen 101 (see FIG. 8).

[0131] As shown in FIG. 12(B), on the device calibration screen 201, a message prompting the imaging device 1 to be stationary at a stable location for a predetermined time for the calibration process and the remaining time are displayed. Also, on the device calibration screen 201, a progress bar 202 indicating the progress status of the calibration process is displayed. Further, the device calibration screen 201 is provided with a "cancel" button 203. When the user operates the "cancel" button 203, the calibration process is cancelled and the user returns to the setting screen 181 (see FIG. 12(A)). Also, when the calibration process is completed, the user returns to the setting screen 181.

[0132] Next, the standby screen 221 and the connection failure notification screens 231 and 241 displayed on the display 82 of the control device 2 will be described. FIG. 13 is an explanatory diagram showing the standby screen 221. FIG. 14 is an explanatory diagram showing the connection failure notification screens 231 and 241.

[0133] On the shooting data list screen 101 (see Fig. 8), when the user operates the "Start Shooting" button 104, the mode shifts from the non-shooting mode to the shooting mode.

[0134] Here, when the transition to the shooting mode is successful, the system transitions to the pause screen 221 shown in Fig. 13. The pause screen 221 displays a message indicating the transition to the shooting mode and a message indicating that the display and operation of the screen on the display 82 of the control device 2 can be enabled by performing an operation to end the shooting mode on the screen displayed on the touch panel display 36 of the imaging device 1.

[0135] On the other hand, when the transition to the shooting mode fails, depending on the cause of the failure, connection failure notification screens 231, 241 (error screens) shown in Fig. 14 are pop-up displayed on the shooting data list screen 101. Here, when a connection failure occurs in the second cable 22 connecting the display input panel unit 12 of the imaging device 1 and the control device 2, the connection failure notification screen 231 shown in Fig. 14(A) is displayed. On the other hand, when a connection failure occurs in the first cable 21 connecting the display input panel unit 12 of the imaging device 1 and the sensor unit 11, the connection failure notification screen 241 shown in Fig. 14(B) is displayed.

[0136] The connection failure notification screen 231 shown in Fig. 14(A) displays a message indicating that the shooting mode cannot be entered due to a connection failure between the display input panel unit 12 of the imaging device 1 and the control device 2, and a message prompting the user to check the connection failure location. The connection failure notification screen 231 is also provided with a "Retry" button 232 and a "Cancel" button 233. After the user has resolved the connection failure of the second cable 22, the user operates the "Retry" button 232. As a result, the process of transitioning to the shooting mode is executed again. On the other hand, when the user operates the "Cancel" button 233, the system returns to the shooting data list screen 101 (see Fig. 8).

[0137] On the connection failure notification screen 241 shown in FIG. 14(B), a message indicating that the shooting mode cannot be entered due to a connection failure between the display input panel unit 12 and the sensor unit 11 of the imaging device 1, and a message prompting the user to check the connection failure location are displayed. Further, on the connection failure notification screen 241, a "Retry" button 242 and a "Cancel" button 243 are provided. After the user performs an operation to eliminate the connection failure of the first cable 21 that connects the display input panel unit 12 and the sensor unit 11 of the imaging device 1, the user operates the "Retry" button 242. Thereby, the process of shifting to the shooting mode is executed again. On the other hand, when the user operates the "Cancel" button 243, the screen returns to the shooting data list screen 101 (see FIG. 8).

[0138] Next, the standby screen 301 displayed on the touch panel display 36 of the imaging device 1 will be described. FIG. 15 is an explanatory diagram showing the standby screen 301.

[0139] In the non-shooting mode, that is, while the user is operating the screen by displaying the shooting data list screen 101 (see FIG. 8), the shooting data details screen 121 (see FIG. 9), and the setting screen 181 (see FIG. 12) on the display 82 of the control device 2, the standby screen 301 is displayed on the touch panel display 36 of the imaging device 1.

[0140] On the standby screen 301, a message indicating that it is in the non-shooting mode, that is, in the middle of the display and operation of the shooting data list screen 101 (see FIG. 8) and the like on the display 82 of the control device 2 is displayed. Further, on the standby screen 301, a message indicating that the shooting mode can be entered by performing an operation to start shooting on the screen displayed on the display 82 of the control device 2 is displayed.

[0141] Next, the shooting standby screen 311 displayed on the touch panel display 36 of the imaging device 1 will be described. FIG. 16 is an explanatory diagram showing the shooting standby screen 311.

[0142] As shown in FIG. 16(A), a shooting standby screen 311 is provided with an image display unit 312. A real-time captured image by the visible camera 53 is displayed on the image display unit 312.

[0143] Also, the shooting standby screen 311 is provided with a shooting mode designation unit 313. When the user operates the shooting mode designation unit 313, as shown in FIG. 16(B), a pull-down menu 314 is displayed. In the pull-down menu 314, the user can select a plurality of shooting modes with different control conditions (processing conditions of image signal processing) of the visible camera 53. In the example shown in FIG. 16(B), as the shooting mode, the user can select any one of "Indoor" (indoor shooting mode), "Outdoor" (first outdoor shooting mode), and "Outdoor + ND filter" (second outdoor shooting mode). Note that in each shooting mode, for example, the brightness correction table used in image signal processing (image correction processing) is different.

[0144] Also, the shooting standby screen 311 is provided with a brightness adjustment bar 315. The user can adjust the brightness of the captured image by operating the slider of the brightness adjustment bar 315. The brightness can be adjusted for each shooting mode.

[0145] Also, the shooting standby screen 311 is provided with a shooting start button 316. When the user operates the shooting start button 316, shooting is started and the screen transitions to a shooting-in progress screen 331 (see FIG. 17).

[0146] Also, the shooting standby screen 311 is provided with a button 317 for "ending shooting". When the user operates the button 317 for "ending shooting", the shooting mode is ended and the non-shooting mode is entered, and the screen transitions to a standby screen 301 (see FIG. 15).

[0147] Next, the in - shooting screen 331 displayed on the touch panel display 36 of the imaging device 1 will be described. FIG. 17 is an explanatory diagram showing the in - shooting screen 331. FIG. 18 is an explanatory diagram showing the transition status of the shooting time guide part 336 in the in - shooting screen 331. FIG. 19 is an explanatory diagram showing the in - shooting screen 331 with the message window 337 opened. FIG. 20 is an explanatory diagram showing the content and display timing of the message displayed in the message window 337.

[0148] As shown in FIGS. 17(A) and (B), the in - shooting screen 331 is provided with a main window 332 (first image display frame) and a sub - window 333 (second image display frame). The display magnifications of the main window 332 and the sub - window 333 are different. In the main window 332, the image is displayed enlarged, and in the sub - window 333, the image is displayed reduced.

[0149] Also, either the real - time captured image 341 by the visible camera 53 or the shooting path image 342 representing the shooting path at the measurement target location is displayed in the main window 332 and the sub - window 333. The shooting path image 342 is obtained by superimposing a line 344 representing the shooting path on the point - cloud image 343 representing the measurement target location. The point - cloud image is an image (rendering) of each point of the generated point - cloud data as seen from a predetermined viewpoint.

[0150] The example shown in FIG. 17(A) is the case of the standard state. In this case, the captured image 341 is displayed enlarged in the main window 332, and the shooting path image 342 is displayed reduced in the sub - window 333. On the other hand, the example shown in FIG. 17(B) is the case of the shooting - path enlarged state. In this case, the shooting path image 342 is displayed enlarged in the main window 332, and the captured image 341 is displayed reduced in the sub - window 333.

[0151] In addition, the switching between the standard state shown in FIG. 17(A) and the enlarged shooting path state shown in FIG. 17(B) can be performed by the user operating the sub-window 333. That is, in the standard state shown in FIG. 17(A), when the sub-window 333 is operated, the state transitions to the enlarged shooting path state shown in FIG. 17(B), and the shooting path image 342 is enlarged and displayed in the main window 332. Also, in the enlarged shooting path state shown in FIG. 17(B), when the sub-window 333 is operated, the state transitions to the standard state shown in FIG. 17(A), and the shooting image 341 is enlarged and displayed in the main window 332.

[0152] Note that a mesh image may be superimposed on the shooting image 341 within the point cloud generation range. The mesh image can be obtained by converting the generated point cloud data into mesh data and performing an imaging process on the mesh data at the same field of view as the shooting image 341.

[0153] Also, a speed bar 334 is provided on the shooting screen 331 during shooting. In the speed bar 334, the speed at which the user moves the sensor unit 11 is visualized and displayed. Specifically, the speed is represented by color. For example, the speed bar 334 is drawn with a gradation from green to orange, and the proportion of orange color increases as the speed increases. Therefore, the user can easily confirm whether the speed at which the sensor unit 11 is moved is appropriate, specifically, whether the speed at which the sensor unit 11 is moved is not too fast. This can avoid the occurrence of blur (blurring) due to the user moving the sensor unit 11 too fast and a decrease in the accuracy of three-dimensional measurement.

[0154] Also, an elapsed time display section 335 is provided on the shooting screen 331 during shooting. The elapsed time display section 335 displays the elapsed time from the start of shooting to the present.

[0155] Also, as shown in FIG. 18, during shooting, the shooting time guide portion 336 is displayed on the shooting screen 331. The elapsed time display portion 335 is always displayed, while the shooting time guide portion 336 is displayed for a predetermined time at a predetermined timing. Note that the shooting time guide portion 336 can be set to be always displayed by a user operation, not limited to being displayed only at a predetermined timing.

[0156] First, as shown in FIG. 18(A), when shooting starts, the shooting time guide portion 336 is displayed for a predetermined time (e.g., 5 seconds) from the start time of shooting. In this case, the available shooting time (e.g., 15 minutes) is displayed on the shooting time guide portion 336.

[0157] Next, as shown in FIG. 18(B), at the timing when the remaining shooting time (the time obtained by subtracting the elapsed time from the start of shooting from the available shooting time) becomes equal to or less than a predetermined time, the shooting time guide portion 336 is displayed for a predetermined time (e.g., 5 seconds). In this case, the remaining shooting time is displayed on the shooting time guide portion 336. Also, the timing for displaying the shooting time guide portion 336 is not limited to once. For example, the shooting time guide portion 336 may be displayed at the timing when the remaining shooting time becomes 3 minutes and at the timing when the remaining shooting time becomes 2 minutes.

[0158] Next, as shown in FIG. 18(C), at the timing when the remaining shooting time becomes very short (e.g., 1 minute), the shooting time guide portion 336 is continuously displayed until the remaining shooting time runs out. In this case, the remaining shooting time is displayed on the shooting time guide portion 336. Also, the shooting time guide portion 336 is displayed in a manner different from the states shown in FIGS. 18(A) and (B). For example, the shooting time guide portion 336 is emphasized by changing the colors of the background and frame of the shooting time guide portion 336.

[0159] Also, as shown in FIG. 19, a message window 337 is displayed on the shooting screen 331. A message to be notified to the user is displayed in the message window 337. Specifically, messages regarding guidance on shooting operations and various warnings are displayed.

[0160] The example shown in Fig. 19(A) is for an information message. The information message provides the user with information for reference in the user's shooting operation or information that calls attention. The example shown in Fig. 19(B) is for a warning message. The warning message prompts the user to take actions to improve the shooting operation state that needs to be improved urgently.

[0161] Note that the message window 337 may be configured such that the display position on the shooting screen 331 differs according to the type of message (information provision, warning, etc.). For example, in the example shown in Fig. 19(B), the message window 337 for the warning message is displayed at the lower part of the shooting screen 331 as in the case of the information message shown in Fig. 19(A), but it may be displayed at the center of the shooting screen 331 so that the user can immediately check the warning message.

[0162] Fig. 20 shows an example of an information message and a warning message displayed on the message window 337. Examples of the information message include a message indicating that the return from the self-position loss was successful, a message indicating that the position correction process was completed when loop closing (circumferential movement) was detected, and a message prompting the user to keep the imaging device 1 stationary for the position correction process when loop closing was detected. Examples of the warning message include a message prompting the user to keep the imaging device 1 stationary when the user moves the imaging device 1 during the execution of the position correction process when loop closing is detected, a message prompting the user to move the imaging device 1 slowly when the speed at which the user moves the imaging device 1 is too fast, a message prompting the user to change the shooting target when the number of feature points extracted from the captured image is small, and a message prompting the user to maintain an appropriate distance from the subject when approaching the subject too closely.

[0163] Next, the shooting end confirmation screens 351 and 361 displayed on the touch panel display 36 of the imaging device 1 will be described. Fig. 21 is an explanatory diagram showing the shooting end confirmation screens 351 and 361.

[0164] As shown in FIG. 17, a shooting end button 339 is provided on the shooting screen 331 during shooting. When the user operates the shooting end button 339, the process of ending the shooting and saving the shooting data is executed, and as shown in FIG. 21(A), a shooting end confirmation screen 351 is pop-up displayed on the shooting screen 331.

[0165] On the shooting end confirmation screen 351, a message indicating that the shooting has ended and the saving of the shooting data has been completed, and a message indicating returning to the shooting standby screen 311 (see FIG. 16) are displayed. Also, an "OK" button 352 is provided on the shooting end confirmation screen 351. When the user operates the "OK" button 352, the screen transitions to the shooting standby screen 311.

[0166] Also, in this embodiment, when the shooting time (elapsed time since the start of shooting) exceeds a predetermined shootable time, the shooting is forcibly ended, and the process of saving the shooting data is executed. In this case, as shown in FIG. 21(B), a shooting end confirmation screen 361 due to exceeding the shooting time is pop-up displayed on the shooting screen 331.

[0167] On the shooting end confirmation screen 361, a message indicating that the shooting has been forcibly ended due to exceeding the time and the saving of the shooting data has been completed, and a message indicating returning to the shooting standby screen 311 (see FIG. 16) are displayed. Also, an "OK" button 362 is provided on the shooting end confirmation screen 361. When the user operates the "OK" button 362, the screen transitions to the shooting standby screen 311.

[0168] Note that in this embodiment, when the shooting time exceeds a predetermined shootable time, the shooting is forcibly ended, but such forced termination may not be performed, or the user may be able to select a mode in which forced termination is performed and a mode in which forced termination is not performed.

[0169] Next, the screen displayed on the touch panel display 36 of the imaging device 1 when self-position loss occurs will be described. FIG. 22 is an explanatory diagram showing a self-position loss notification screen 371. FIG. 23 is an explanatory diagram showing a screen during return 381. FIG. 24 is an explanatory diagram showing an enlarged screen of the final position acquisition point 391. FIG. 25 is an explanatory diagram showing a return failure notification screen 401.

[0170] During imaging, a self-position loss (tracking loss) in which the self-position is lost may occur in the tracking process P12 (self-position estimation process). In this case, it is necessary to return to the final position acquisition point, that is, the last point where the normal self-position could be estimated, and restart the tracking process P12. Therefore, first, after notifying the user that self-position loss has occurred, assist the user to return to the final position acquisition point, and when it is detected that the user has returned to the final position acquisition point, restart the tracking process P12.

[0171] In the present embodiment, when self-position loss is detected, on the touch panel display 36 of the imaging device 1, as shown in FIG. 22, a self-position loss notification screen 371 is pop-up displayed on the imaging screen 331.

[0172] On the self-position loss notification screen 371, a message is displayed notifying the user that self-position loss has occurred and prompting the user to return to the final position acquisition point. In addition, an "OK" button 372 is provided on the self-position loss notification screen 371. When the user operates the "OK" button 372, the screen transitions to the screen during return 381 shown in FIG. 23. At this time, in the processor 86 of the control device 2, a process for detecting that the imaging device 1 has returned to the final position acquisition point is started.

[0173] Here, the example shown in Fig. 23(A) is the same as the example shown in Fig. 17(A) in the standard state. The captured image 341 by the visible camera 53 is enlarged and displayed in the main window 332 (the first image display frame), and the captured path image 342 is reduced and displayed in the sub-window 333 (the second image display frame). Also, the example shown in Fig. 23(B) is the same as the example shown in Fig. 17(B) in the state where the captured path is enlarged. The captured path image 342 is enlarged and displayed in the main window 332, and the captured image 341 is reduced and displayed in the sub-window 333.

[0174] In addition, on the returning screen 381, another sub-window 382 (the third image display frame) is provided separately from the sub-window 333 (the second image display frame) where the captured image 341 and the captured path image 342 are reduced and displayed. The captured image at the final position acquisition point is displayed in the sub-window 382. Thereby, the user can easily grasp the final position acquisition point.

[0175] Also, on the returning screen 381, a message window 383 is displayed. In the message window 383, similar to the self-position lost notification screen 371 (see Fig. 22), a message is displayed notifying that the self-position has been lost and prompting to return to the final position acquisition point.

[0176] When the sub-window 382 where the captured image at the final position acquisition point is displayed is operated, as shown in Fig. 24, the enlarged screen 391 of the final position acquisition point is pop-up displayed on the returning screen 381. In the enlarged screen 391 of the final position acquisition point, the captured image at the final position acquisition point is enlarged and displayed. Also, an "×" button 392 is displayed on the enlarged screen 391 of the final position acquisition point. When the user operates the "×" button 392, the enlarged screen 391 of the final position acquisition point is closed and returns to the returning screen 381.

[0177] Here, when a return from the self-position loss, that is, a successful return to the final position acquisition point is achieved, a transition is made to the shooting screen 331 (see FIG. 17). Note that at the timing when it is detected that the return to the final position acquisition point has been made, after a message notifying that the return to the final position acquisition point is displayed on the return screen 381, a transition may be made to the shooting screen 331.

[0178] On the other hand, when the return from the self-position loss cannot be achieved even after a predetermined time has elapsed, as shown in FIG. 25, a return failure notification screen 401 is pop-up displayed on the return screen 381. A message indicating that the return from the self-position loss has failed ("Tracking return has failed.") is displayed on the return failure notification screen 401. Also, an "OK" button 402 is provided on the return failure notification screen 401. When the user operates the "OK" button 402, shooting is forcibly terminated and a transition is made to the shooting standby screen 311 (see FIG. 16).

[0179] Next, the connection failure notification screen 411 displayed on the touch panel display 36 of the imaging device 1 will be described. FIG. 26 is an explanatory diagram showing the connection failure notification screen 411.

[0180] During shooting, the first cable 21 connecting the display input panel unit 12 and the sensor unit 11 may become disconnected. In this case, the connection failure notification screen 411 (error screen) shown in FIG. 26 is pop-up displayed on the shooting screen 331.

[0181] A message indicating that the sensor unit 11 including the visible camera 53 etc. cannot be detected and a message prompting the user to check the state of the cable connecting the sensor unit 11 and the cable connecting the display input panel unit 12 and the sensor unit 11 are displayed on the connection failure notification screen 411.

[0182] In addition, the connection failure notification screen 411 is provided with a "Retry" button 412 and a "Cancel" button 413. After the user has resolved the connection failure of the cable connecting the display input panel unit 12 of the imaging device 1 and the sensor unit 11, the user operates the "Retry" button 412. When the connection failure is resolved here, the imaging standby screen 311 (see FIG. 16) is transitioned to. Also, when the user operates the "Cancel" button 413, the mode shifts from the imaging mode to the list mode. At this time, on the touch panel display 36 of the imaging device 1, the standby screen 301 (see FIG. 15) is transitioned to. On the other hand, on the display 82 of the control device 2, the standby screen 221 (see FIG. 13) is transitioned to the imaging data list screen 101 (see FIG. 8).

[0183] Note that the second cable 22 connecting the display input panel unit 12 of the imaging device 1 and the control device 2 may become disconnected. In this case, no screen is displayed on the touch panel display 36 of the imaging device 1. On the other hand, on the display 82 of the control device 2, a screen (not shown) for notifying the user of the connection failure is pop-up displayed on the standby screen 221 (see FIG. 13). The screen for notifying the user of the connection failure is provided with an "OK" button. When the user operates the "OK" button, the imaging data list screen 101 (see FIG. 8) is transitioned to.

[0184] Next, the imaging mode end confirmation screen 421 displayed on the touch panel display 36 of the imaging device 1 will be described. FIG. 27 is an explanatory diagram showing the imaging mode end confirmation screen 421.

[0185] In the imaging standby screen 311 shown in FIG. 16, when the user operates the "End Imaging" button 317, the imaging mode end confirmation screen 421 shown in FIG. 27 is pop-up displayed on the imaging standby screen 311.

[0186] On the shooting mode end confirmation screen 421, a message is displayed to inquire the user whether it is possible to end the shooting mode and return to the list mode. Also, on the shooting mode end confirmation screen 421, an "OK" button 422 and a "Cancel" button 423 are provided. When the user operates the "Cancel" button 423, the screen returns to the shooting standby screen 311 (see Fig. 16). Also, when the user operates the "OK" button 422, the shooting mode ends and the screen returns to the list mode. At this time, on the touch panel display 36 of the imaging device 1, the screen transitions to the standby screen 301 (see Fig. 15). On the other hand, on the display 82 of the control device 2, the screen transitions from the standby screen 221 (see Fig. 13) to the shooting data list screen 101 (see Fig. 8).

[0187] (Second Embodiment) Next, the second embodiment will be described. Note that points not particularly mentioned here are the same as those in the above embodiment. Fig. 28 is a block diagram showing the schematic configuration of the imaging device 1 and the control device 2 according to the second embodiment.

[0188] In the first embodiment, the imaging device 1 and the control device 2 are connected by wire, but in this embodiment, the imaging device 1 and the control device 2 are connected wirelessly. Specifically, the imaging device 1 includes a wireless communication unit 78. Also, the control device 2 (information processing device) includes a wireless communication unit 88. The wireless communication unit 78 of the imaging device 1 and the wireless communication unit 88 of the control device 2 perform wireless communication using an appropriate wireless communication method such as a wireless LAN. In the example shown in Fig. 28, an input / output interface 79 is provided to perform data input / output with the sensor unit 11 having the same configuration as that in the first embodiment.

[0189] When the imaging device 1 and the control device 2 are wirelessly connected in this way, during shooting, the control device 2 can be placed stationary at an appropriate location, for example, at or near the measurement target location, and the user does not have to carry the control device 2 around.

[0190] Incidentally, in the first and second embodiments, the imaging system includes the imaging device 1 held by the user's hand and the control device 2 carried by the user. However, the display input panel unit 12 of the imaging device 1 may include all or part of the functions of the control device 2. In this case, the process of generating simple point cloud data (simple point cloud generation process) for the user to confirm the imaging situation (point cloud generation situation) has a relatively small load and needs to be performed in real time during imaging, so it may be performed by the imaging device 1. Also, the process of generating normal point cloud data (standard point cloud generation process) to be acquired as the three-dimensional measurement result has a relatively large load and may be performed after imaging, so it may be performed by a separately provided server device with high processing power.

[0191] Also, in the configuration where the imaging device 1 and the control device 2 are wirelessly connected as in the second embodiment, in addition to being configured on-premises, the control device 2 may be configured in the cloud. Also, in the configuration where the imaging device 1 having part of the functions of the control device 2 and the server device are wirelessly connected, the server device may be configured on-premises or in the cloud.

[0192] As described above, the embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments with changes, replacements, additions, omissions, etc. Also, it is possible to combine the respective components described in the above embodiments to form a new embodiment.

Industrial Applicability

[0193] The imaging system, imaging device, and imaging control method according to the present invention have the effect of being able to easily perform screen operations for instructions and settings related to imaging on the touch panel display while stably holding the imaging device even during imaging, and are useful as an imaging system, imaging device, and imaging control method including an imaging device that captures a measurement target location and a processor that controls the imaging device in order to perform three-dimensional measurement processing for generating three-dimensional spatial information of the measurement target location.

Explanation of Reference Numerals

[0194] 1: Imaging device 2: Control device (information processing device) 11: Sensor unit 12: Display input panel unit 13: Support 21: First cable 22: Second cable 31: Main body part 32: Telescopic rod part 33: Sensor mounting part 34: Panel mounting part 35: Grip part 36: Touch panel display 41: Joint 42: Fixed knob 43: Joint 44: Fixed knob 51: Sensor unit 52: Sensor cover 53: Visible camera 54: Depth sensor 61: Cover body 62: Filter member 65: ND filter 101: Shooting data list screen 121: Shooting data detail screen 231: Connection failure notification screen 241: Connection failure notification screen 311: Shooting standby screen 331: Shooting in progress screen 335: Elapsed time display part 336: Shooting time guidance part 351: Shooting end confirmation screen 361: Shooting end confirmation screen 371: Self-position lost notification screen 381: Returning screen 411: Connection failure notification screen 421: Shooting mode end confirmation screen

Claims

1. A photographing system for performing three-dimensional measurement processing for generating three-dimensional space information of a measurement target location, comprising: a photographing device that photographs the measurement target location; and a processor that controls the photographing device, wherein the photographing device comprises a sensor unit including a camera that photographs the measurement target location, a display input panel unit including a touch panel display that displays a screen for assisting a user's photographing operation and detects a screen operation by the user, and a support that supports the sensor unit and the display input panel unit, wherein the support has a grip portion that can be held by the user with one hand, wherein the display input panel unit is disposed in the vicinity of the grip portion, and wherein the processor causes the touch panel display to display at least an operation screen for the user to give an instruction to start photographing, characterized in that it is a photographing system.

2. The processor causes the touch panel display to display, as the operation screen, a photographing standby screen, a photographing screen, and a photographing end confirmation screen to sequentially transition according to a user's instruction, characterized in that it is the photographing system according to claim 1.

3. The processor causes the touch panel display to display the photographing standby screen including an operation unit for the user to select any one of a plurality of photographing modes having different control conditions of the camera, characterized in that it is the photographing system according to claim 2.

4. The processor causes the touch panel display to display the photographing screen including a first image display frame for enlarging and displaying either one of the photographed image by the camera and a photographing path image representing the path along which the photographing device has moved, a second image display frame for reducing and displaying the other image, and an operation unit for switching the images to be displayed in the first image display frame and the second image display frame, characterized in that it is the photographing system according to claim 2.

5. The processor at the end of photographing, notifies the user of the saving of the photographing data and the return to the photographing standby screen, and causes the touch panel display to display the photographing end confirmation screen including an operation unit for the user to input that the user has confirmed, characterized in that it is the photographing system according to claim 2.

6. The processor measures the speed at which the user moves the sensor unit and causes the touch panel display to display the photographing screen including an image visualizing whether or not the speed is appropriate, characterized in that it is the photographing system according to claim 2.

7. The processor At the start of shooting, display the in-shooting screen including a predetermined shootable time. During shooting, at the timing when the remaining shooting time obtained by subtracting the elapsed time from the start of shooting from the shootable time becomes equal to or less than a predetermined value, display the in-shooting screen including the remaining shooting time. The imaging system according to claim 2, characterized in that.

8. The processor When detecting a self-position loss in which the current position of the imaging device is lost in the tracking process, notifies the user of the occurrence of the self-position loss, presents the final position acquisition point to the user, and causes the touch panel display to display a screen prompting the user to return to the final position acquisition point. The imaging system according to claim 1, characterized in that.

9. The processor Detects a connection failure on the communication path between the camera, the touch panel display, and the processor, and causes the touch panel display to display a screen notifying the user of the occurrence of the connection failure. The imaging system according to claim 1, characterized in that.

10. An imaging device for imaging a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location, A sensor unit including a camera for imaging a measurement target location; A display input panel unit including a touch panel display for displaying a screen assisting the user's imaging operation and detecting a screen operation by the user; A support for supporting the sensor unit and the display input panel unit, The support has a grip portion that can be held by the user with one hand, The display input panel unit is disposed in the vicinity of the grip portion, The touch panel display is characterized in that it displays at least an operation screen for the user to give an instruction to start shooting. An imaging device.

11. An imaging control method for executing, by a processor, a process of controlling an imaging device that images a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location, A sensor unit including a camera for photographing a measurement target location, a display input panel unit including a touch panel display for displaying a screen for assisting a user's photographing operation and detecting a screen operation by the user, and a support for supporting the sensor unit and the display input panel unit, wherein the support has a grip portion that can be held by a user with one hand, and the display input panel unit is configured to A photographing control method characterized by displaying an operation screen for allowing a user to perform an operation for instructing at least the start of photographing.

Images