Three-dimensional capture device and three-dimensional capture system

Abstract

To provide a 3D capturing device capable of 3D capturing a moving large structure with high precision.SOLUTION: The 3D capturing device 2 includes: a stereo line sensor unit 11 in which multiple line sensors 111L and 111R are arranged on the same straight line in a longitudinal direction; and a stereo line sensor unit 12 in which multiple line sensors 121L and 121R are arranged on the same straight line in a longitudinal direction. The stereo line sensor units 11 and 12 are placed so that the longitudinal direction of the line sensors 111L and 111R of the stereo line sensor unit 11 and the longitudinal direction of the line sensors 121L and 121R of the stereo line sensor unit 12 cross each other.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a three-dimensional capture device and a three-dimensional capture system.

Background Art

[0002] In the inspection of large and complex structures such as automobiles during shipment inspection or quality inspection of used products, inspectors are inspecting each part. However, there are parts that are difficult to detect visually, such as the underside of a vehicle body in the case of an automobile. Recently, the types of automobiles have become diversified, and the positions of parts to be inspected are different from each other, so it has become difficult to inspect each part of such a structure.

[0003] Therefore, it has been proposed to three-dimensionally represent such a large and complex structure in a virtual reality space so that an inspector can inspect it with an image at a desired position and orientation.

[0004] Patent Document 1 describes an invention of a three-dimensional coordinate acquisition device capable of calibrating a stereo camera without requiring a special device.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] When measuring a large structure, taking a picture of it results in a huge amount of imaging data, and it takes a huge amount of time to process the huge amount of imaging data. Especially in the method using a stereo area sensor, the number of consecutive images generated is large. And the processing time for stereo matching of a large number of images also becomes huge. In the imaging method using a stereo area sensor, since the object of stereo matching is two axes (two-dimensional), it is difficult to achieve a precision where the seams of the images are not noticeable in three-dimensional synthesis.

[0007] In the imaging method using one stereo line sensor, compared with the imaging method using a stereo line sensor, the amount of data can be significantly reduced, but the three-dimensional information in the vertical direction of the line sensor cannot be obtained, and the three-dimensional information of the structure appearance cannot be obtained sufficiently. Furthermore, in the imaging method using one stereo line sensor, when the moving speed of the subject changes, incorrect three-dimensional information will be obtained.

[0008] Therefore, an object of the present invention is to three-dimensionally capture a large-sized moving structure with high precision.

Means for Solving the Problem

[0009] To solve the above-mentioned problems, the three-dimensional capture device of the present invention includes a plurality of imaging units in which a plurality of line sensors are arranged on the same straight line in the longitudinal direction, and the On the ground plane movement amount measurement unit that measures the movement amount of the subject, and the longitudinal directions of the line sensors of each of the imaging units intersect and the line sensor of each of the imaging units is present on a plane parallel to the ground plane and each of the imaging units is installed so as to intersect, which is characterized in that. The three-dimensional capture device of the present invention includes three or more imaging units in which a plurality of line sensors are arranged on the same straight line in the longitudinal direction, and the On the ground plane movement amount estimation unit that estimates the movement amount of the subject from the imaging data captured by each of the imaging units, and the longitudinal directions of the line sensors of each of the imaging units intersect and the line sensor of each of the imaging units is present on a plane parallel to the ground plane and each of the imaging units is installed so as to intersect, which is characterized in that 。

[0010] The three-dimensional capture system of the present invention includes a plurality of photographing units in which a plurality of line sensors are arranged on the same straight line in the longitudinal direction, and the On the ground plane amount-of-movement measurement unit that measures the amount of movement of the subject, and the longitudinal direction of the line sensors of each of the photographing units intersects and the line sensor of each of the imaging units is present on a plane parallel to the ground plane such that each of the photographing units is installed. This is the gist of the invention. The three-dimensional capture system of the present invention includes three or more photographing units in which a plurality of line sensors are arranged on the same straight line in the longitudinal direction, and the On the ground plane amount-of-movement estimation unit that estimates the amount of movement of the subject from the photographed data photographed by each of the photographing units, and the longitudinal direction of the line sensors of each of the photographing units intersects and the line sensor of each of the imaging units is present on a plane parallel to the ground plane such that each of the photographing units is installed. This is the gist of the invention 。 There Other means will be described in the mode for carrying out the invention.

Effect of the Invention

[0011] According to the present invention, it becomes possible to three-dimensionally capture a large-sized structure that moves with high accuracy.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8A

Figure 8B

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22A

Figure 22B

Figure 23

Figure 24

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the respective drawings. Hereinafter, the "three-dimensional information" means a set of coordinate values on an orthogonal coordinate system of a plurality of points of interest on the appearance of a subject. The three-dimensional information approximately represents the appearance of a structure in a shape connected by sides connecting two vertices of one or more triangular planes (polygons). In the three-dimensional information, color information is assigned to each vertex in order to represent the color on the surface of the structure.

[0014] "Occlusion" means a state in which, when obtaining three-dimensional information, a certain area is in the shadow of another object and does not appear in a stereo sensor (line sensor, area sensor), or the information necessary for obtaining three-dimensional information is not complete. The occluded area is treated as a hollow in the three-dimensional information.

[0015] 《First Embodiment》 In the three-dimensional capture device of the first embodiment, two stereo line sensor devices are arranged in a crossed manner to photograph a subject. The three-dimensional capture device generates combined images (two in total), respectively, and performs stereo matching between the two combined images to generate disparity information and obtain three-dimensional information. The three-dimensional capture device further synthesizes the three-dimensional information obtained respectively to complement each other the three-dimensional information in directions that cannot be acquired by a single stereo line sensor device. In this embodiment, instead of an area sensor, a line sensor is used so that the data does not become huge and three-dimensional capture with less occlusion can be performed.

[0016] FIG. 1 is a configuration diagram of a three-dimensional capture device 2 according to the first embodiment. The three-dimensional capture device 2 includes stereo line sensor devices 11 and 12 and a movement amount sensor 3, and is configured by connecting a keyboard 241, a mouse 242, and a display 231. This three-dimensional capture device 2 captures three-dimensional information of a subject, and is realized, for example, by installing a three-dimensional capture program in a general-purpose computer. Note that the present invention is not limited to the three-dimensional capture device 2 configured by a single computer, and may be realized as a three-dimensional capture system in which a plurality of devices cooperate with each other.

[0017] The stereo line sensor device 11 is configured to include two line sensor devices 11L and 11R. The line sensor device 11L includes a line sensor 111L and an optical system 112L including a lens. The line sensor device 11R includes a line sensor 111R and an optical system 112R including a lens. The line sensors 111L and 111R are one-dimensional imaging elements configured by a CCD (Charge Coupling Device) or a CMOS (Complementary-MOS). The optical system 112L forms an optical image on the line sensor 111L. The optical system 112R forms an optical image on the line sensor 111R.

[0018] The line sensor 111L and the line sensor 111R are arranged so as to be separated by a distance L and their longitudinal directions are located on the same straight line. That is, the stereo line sensor device 11 is a photographing unit in which a plurality of line sensors 111L and 111R are arranged on the same straight line in the longitudinal direction. And the optical axis of the optical system 112L and the optical axis of the optical system 112R are parallel.

[0019] The stereo line sensor device 12 is configured to include two line sensor devices 12L and 12R. The line sensor device 12L is composed of a line sensor 121L and an optical system 122L including a lens. The line sensor device 12R is composed of a line sensor 121R and an optical system 122R including a lens. The line sensors 121L and 121R are one-dimensional imaging elements composed of CCD or CMOS. The optical system 122L forms an optical image on the line sensor 121L. The optical system 122R forms an optical image on the line sensor 121R.

[0020] The two line sensors 121L and 121R are arranged so as to be separated by a distance L and their longitudinal directions are located on the same straight line. That is, the stereo line sensor device 12 is a photographing unit in which a plurality of line sensors 121L and 121R are arranged on the same straight line in the longitudinal direction. And the optical axis of the optical system 122L and the optical axis of the optical system 122R are parallel.

[0021] The movement amount sensor 3 is provided close to the stereo line sensor devices 11 and 12, and is a movement amount measurement unit that measures the movement amount and movement direction of the subject in the optical axis direction of these.

[0022] The three-dimensional capture device 2 further includes a line sensor interface 25, an MPU (Micro Processing Unit) 21, a memory 22, a graphic interface 23, and an input interface 24.

[0023] The line sensor interface 25 is an interface for transmitting and receiving data between the stereo line sensor devices 11 and 12. The MPU 21 is a processing device that comprehensively controls the three-dimensional capture device 2, and uses the memory 22 as a work area. The memory 22 is a volatile memory or a non-volatile memory, and is a storage unit that stores information.

[0024] The graphic interface 23 is, for example, a GPU (Graphic Processing Unit) and generates an image to be displayed on the display 231. The input interface 24 is an interface for receiving signals from the mouse 242 and the keyboard 241, and is, for example, a serial signal controller or a USB (registered trademark) (Universal Serial BUS) host controller, etc.

[0025] The three-dimensional capture device 2 of this embodiment processes the captured image data by the one-dimensional line sensors 111L, 111R, 121L, 121R. As a result, it can be handled as a relatively small image (data volume) compared to the case of adopting an area sensor, and the stereo matching process is also lightened. Further, compared to the case of adopting an area sensor, the factors causing errors can be reduced.

[0026] FIG. 2A is a plan view showing an arrangement example of the stereo line sensor devices 11, 12. Here, the stereo line sensor device 11 is arranged such that the one-dimensional line sensors 111L, 111R are separated by a distance L and their longitudinal directions are located on the same straight line 115.

[0027] The stereo line sensor device 12 is arranged such that the one-dimensional line sensors 121L, 121R are separated by a distance L and their longitudinal directions are located on the same straight line 125. The stereo line sensor devices 11, 12 are installed such that the longitudinal direction of the line sensors 111L, 111R intersects with the longitudinal direction of the line sensors 121L, 121R.

[0028] The intersection point of the straight line 115 and the straight line 125 is equidistant from the line sensors 111L, 111R and is also equidistant from the line sensors 121L, 121R. That is, the line sensors 111L, 111R and the line sensors 121L, 121R are arranged on concentric circles.

[0029] FIG. 2B is a perspective view showing an arrangement example of the stereo line sensor devices 11, 12. Above the line sensor 111L, an optical system 112L is arranged. Above the line sensor 111R, an optical system 112R is arranged. The optical axis 113L of the optical system 112L and the optical axis 113R of the optical system 112R are parallel. That is, the stereo line sensor device 11 includes optical systems 112L and 112R having the same direction as the optical axis. The visual field 114 on the subject 4 by the stereo line sensor device 11 is obtained by magnifying the line sensor 111L at the magnification of the optical system 112L and projecting it in the direction of the optical axis 113L, and magnifying the line sensor 111R at the magnification of the optical system 112R and projecting it in the direction of the optical axis 113R.

[0030] Above the line sensor 121L, an optical system 122L is arranged. Above the line sensor 121R, an optical system 122R is arranged. The optical axis 123L of the optical system 122L and the optical axis 123R of the optical system 112R are parallel. That is, the stereo line sensor device 12 includes optical systems 122L and 122R having the same direction as the optical axis. The visual field 124 on the subject 4 by the stereo line sensor device 11 is obtained by magnifying the line sensor 121L at the magnification of the optical system 122L and projecting it in the direction of the optical axis 123L, and magnifying the line sensor 121R at the magnification of the optical system 122R and projecting it in the direction of the optical axis 123R.

[0031] In this configuration, the occlusion caused by adopting the line sensor can be mutually complemented by the two stereo line sensor devices 11 and 12. It is desirable that the optical axes 113L, 113R and the optical axes 123L, 123R are parallel.

[0032] Figure 3A is a plan view showing an arrangement example of the stereo line sensor devices 11 and 12 of the modified example. The stereo line sensor devices 11 and 12 intersect. The intersection point of the straight line 115 and the straight line 125 is equidistant from the line sensors 121R and 111L, and is also equidistant from the line sensors 121L and 111R. That is, the line sensors 121R and 111L are arranged on the first concentric circle, and the line sensors 121L and 111R are arranged on the second concentric circle.

[0033] Figure 3B is a perspective view showing an arrangement example of the stereo line sensor devices 11 and 12 of the modified example. An optical system 112L is disposed above the line sensor 111L. An optical system 112R is disposed above the line sensor 111R. The optical axis 113L of the optical system 112L and the optical axis 113R of the optical system 112R are parallel. The field of view 114 on the subject 4 by the stereo line sensor device 11 is obtained by magnifying the line sensor 111L at the magnification of the optical system 112L and projecting it in the direction of the optical axis 113L, and magnifying the line sensor 111R at the magnification of the optical system 112R and projecting it in the direction of the optical axis 113R.

[0034] An optical system 122L is disposed above the line sensor 121L. An optical system 122R is disposed above the line sensor 121R. The optical axis 123L of the optical system 122L and the optical axis 123R of the optical system 112R are parallel. The field of view 124 on the subject 4 by the stereo line sensor device 11 is obtained by magnifying the line sensor 121L at the magnification of the optical system 122L and projecting it in the direction of the optical axis 123L, and magnifying the line sensor 121R at the magnification of the optical system 122R and projecting it in the direction of the optical axis 123R.

[0035] Also in the configuration of the modified example, occlusion due to the adoption of the line sensor can be mutually complementary. It is desirable that the optical axes 113L, 113R and the optical axes 123L, 123R are parallel.

[0036] Figure 4 is a block diagram of the three-dimensional capture device 2. The MPU 21 (see FIG. 1) of the three-dimensional capture device 2 implements each part of the imaging unit 51, the speed change correction unit 52, the disparity map generation unit 53, the point cloud data generation unit 54, the point cloud data synthesis unit 55, the three-dimensional mesh conversion unit 56, and the three-dimensional mesh display unit 57 by executing a control program (not shown).

[0037] The imaging unit 51 controls the stereo line sensor devices 11 and 12 to acquire imaging data. The speed change correction unit 52 corrects the imaging data according to the speed change of the subject. The speed change correction unit 52 is a correction unit that corrects the imaging data captured by each imaging unit based on the movement amount of the subject.

[0038] The disparity map generation unit 53 generates disparity map data from the left and right stereo images of the stereo line sensor device 11, and generates disparity map data from the left and right stereo images of the stereo line sensor device 12. The point cloud data generation unit 54 generates point cloud data from the two pieces of disparity map data respectively. The point cloud data synthesis unit 55 synthesizes the two pieces of point cloud data. Thereby, the disparity map generation unit 53 can complement occlusion.

[0039] The three-dimensional mesh conversion unit 56 converts the point cloud data synthesized by the point cloud data synthesis unit 55 into three-dimensional mesh data. The three-dimensional mesh display unit 57 converts the three-dimensional mesh data into a display image from a desired viewpoint and displays it on the screen.

[0040] FIG. 5 is a conceptual diagram showing various data generated by the three-dimensional capture device 2. The various data generated by the three-dimensional capture device 2 are stored in the memory 22. The left imaging data 611L is the data captured by the imaging unit 51 using the line sensor 111L. The right imaging data 611R is the data captured by the imaging unit 51 using the line sensor 111R. The left imaging data 611L and the right imaging data 611R constitute a stereo image.

[0041] The left imaging data 612L is the data captured by the imaging unit 51 using the line sensor 121L. The right imaging data 612R is the data captured by the imaging unit 51 using the line sensor 121R. The left imaging data 612L and the right imaging data 612R constitute a stereo image.

[0042] The movement amount log data 64 is data obtained by the movement amount sensor 3 measuring the movement amount of the subject at each imaging timing. The corrected left imaging data 621L is obtained by correcting the left imaging data 611L based on the movement amount log data 64. The corrected right imaging data 621R is obtained by correcting the right imaging data 611R based on the movement amount log data 64.

[0043] The corrected left imaging data 622L is obtained by correcting the left imaging data 612L based on the movement amount log data 64. The corrected right imaging data 622R is obtained by correcting the right imaging data 612R based on the movement amount log data 64.

[0044] The corrected disparity map data 631 is data of the disparity map based on the corrected left imaging data 621L and the corrected right imaging data 621R. The corrected disparity map data 632 is data of the disparity map based on the corrected left imaging data 622L and the corrected right imaging data 622R.

[0045] The three-dimensional point cloud data 651 is obtained by converting the corrected disparity map data 631 into three-dimensional point cloud data. The three-dimensional point cloud data 652 is obtained by converting the corrected disparity map data 632 into three-dimensional point cloud data. The synthesized three-dimensional point cloud data 66 is obtained by synthesizing the three-dimensional point cloud data 651 and the three-dimensional point cloud data 652. The three-dimensional mesh data 67 is obtained by converting the synthesized three-dimensional point cloud data 66 into three-dimensional mesh data.

[0046] FIG. 6 is a diagram showing the operation of the stereo line sensor device 11. Note that the stereo line sensor device 12 operates in the same manner. The line sensors 111R and 111L of the present embodiment image in synchronization with an external shutter synchronization signal, similar to many industrial sensors. Note that the line sensors 111R and 111L perform imaging using an electronic shutter that electronically controls the imaging element, without relying on a physical shutter mechanism.

[0047] Note that the line sensors 111R and 111L of this embodiment use a single cable to both receive the synchronization signal and transmit the captured data. Note that this is not the only way, and they may have independent cables for receiving the synchronization signal and transmitting the captured data. Also, the synchronization signal may be generated by an external device instead of being generated by the three-dimensional capture device 2.

[0048] Upon receiving an instruction to start shooting from the imaging unit 51 of the three-dimensional capture device 2, the line sensors 111R and 111L start shooting and transmit the captured image data to the imaging unit 51. Then, upon receiving an instruction to end shooting from the imaging unit 51 of the three-dimensional capture device 2, the line sensors 111R and 111L end shooting.

[0049] The line sensors 111R and 111L perform imaging simultaneously upon receiving the synchronization signal. The line sensors 111R and 111L each have a flat field of view with a field angle greater than 0 degrees by means of optical systems 112R and 112L having lenses.

[0050] The line sensors 111R and 111L are arranged in a straight line with a predetermined distance (baseline length) between them and in the same direction. There is an area where the field of view 126 of the line sensor 111R and the field of view 116 of the line sensor 111L overlap. This overlapping area is the range in which the three-dimensional information of the subject 4 can be obtained. The method of obtaining the three-dimensional information of each part of the subject 4 is mainly based on the principle of triangulation using two line sensors.

[0051] Figs. 7A and 7B are diagrams showing the operation of photographing an automobile 71 with the stereo line sensor devices 11 and 12. Fig. 7A is a side view of the automobile 71. Fig. 7B is a rear view of the automobile 71. As shown in Figs. 7A and 7B, the stereo line sensor devices 11 and 12 are arranged below the vehicle body and inside the left and right tires.

[0052] Figs. 8A and 8B are diagrams showing the operation of photographing a train 72 with the stereo line sensor devices 11 and 12. As shown in FIGS. 8A and 8B, the stereo line sensor devices 11 and 12 are arranged below the car body of the train 72 and inside the left and right wheels.

[0053] FIG. 9 is a diagram showing the operation of the three-dimensional capture device 2. The speed change correction unit 52 generates corrected left captured data 621L obtained by correcting the left captured data 611L based on the movement amount log data 64. The speed change correction unit 52 generates corrected right captured data 621R obtained by correcting the right captured data 611R based on the movement amount log data 64.

[0054] The speed change correction unit 52 generates corrected left captured data 622L obtained by correcting the left captured data 612L based on the movement amount log data 64. The speed change correction unit 52 generates corrected right captured data 622R obtained by correcting the right captured data 612R based on the movement amount log data 64.

[0055] The disparity map generation unit 53 calculates the disparity between the corrected left captured data 621L and the corrected right captured data 621R, and generates corrected disparity map data 631. The disparity map generation unit 53 calculates the disparity between the corrected left captured data 622L and the corrected right captured data 622R, and generates corrected disparity map data 632.

[0056] The point cloud data generation unit 54 generates three-dimensional point cloud data 651 from the corrected disparity map data 631, and generates three-dimensional point cloud data 652 from the corrected disparity map data 632. The point cloud data synthesis unit 55 generates synthesized three-dimensional point cloud data 66 from the three-dimensional point cloud data 651 and the three-dimensional point cloud data 652.

[0057] The three-dimensional mesh conversion unit 56 generates three-dimensional mesh data 67 from the synthesized three-dimensional point cloud data 66.

[0058] FIG. 10 is a flowchart showing the processing of the three-dimensional capture device 2. When the process starts, the imaging unit 51 measures the amount of movement of the subject and, in parallel, executes imaging by each stereo line sensor device 11, 12 (step S10). The speed change correction unit 52 corrects the speed change based on the measured amount of movement (step S11).

[0059] The disparity map generation unit 53 generates a disparity map from the left and right imaging data of each stereo line sensor device 11, 12 (step S12). The point cloud data generation unit 54 generates each point cloud data from each disparity map (step S13). The point cloud data synthesis unit 55 synthesizes each point cloud data (step S14).

[0060] The three-dimensional mesh conversion unit 56 converts the synthesized point cloud data into a three-dimensional mesh (step S15). When the three-dimensional mesh display unit 57 displays the three-dimensional mesh (step S16), the process of FIG. 10 ends.

[0061] FIG. 11 is a flowchart showing the imaging process. This imaging process corresponds to the process of step S10 in FIG. 10. The imaging unit 51 instructs the movement amount sensor 3 to start measuring the movement amount and starts recording the movement amount log data 64 in the memory 22 (step S20). The measurement result of the movement amount is output as at least a two-dimensional vector value on the ground plane. However, a device that outputs a three-dimensional vector value including the vertical direction may also be used.

[0062] The imaging unit 51 starts outputting a shutter synchronization signal to the stereo line sensor devices 11, 12 via the line sensor interface 25 and transmits an imaging start instruction (step S21). The imaging unit 51 receives imaging data from each stereo line sensor device 11, 12 via the line sensor interface 25 (step S22).

[0063] When saving, the imaging unit 51 fills the storage area in accordance with the orientation of each line sensor (step S23). That is, the imaging unit 51 saves the data from the line sensor 111L as left captured data 611L. The imaging unit 51 saves the data from the line sensor 111R as right captured data 611R. The imaging unit 51 saves the data from the line sensor 121L as left captured data 612L. The imaging unit 51 saves the data from the line sensor 121R as right captured data 612R. The imaging unit 51 fills in blank or overlapping pixels by bilinear interpolation or the like (step S24).

[0064] The imaging unit 51 sends an imaging end instruction to each line sensor via the line sensor interface 25 and stops the output of the shutter synchronization signal (step S25). Then, the imaging unit 51 instructs the movement amount sensor 3 to end the measurement (step S26).

[0065] FIG. 12 is a flowchart showing the correction process of each image data. This correction process corresponds to the process of step S11 in FIG. 10.

[0066] Based on the movement amount log data 64, the speed change correction unit 52 corrects the left captured data 611L, the right captured data 611R, the left captured data 612L, and the right captured data 612R to a state where they are captured at a uniform movement speed (step S30). When correcting, the speed change correction unit 52 fills in the values of the blank areas and overlapping areas by bilinear interpolation or the like (step S31). Then, the speed change correction unit 52 saves the result of correcting each image data as corrected image data (step S32).

[0067] The speed change correction unit 52 saves the result of correcting the image data of the left captured data 611L to the corrected left captured data 621L, saves the result of correcting the image data of the right captured data 611R to the corrected right captured data 621R, saves the result of correcting the image data of the left captured data 612L to the corrected left captured data 622L, and saves the result of correcting the image data of the right captured data 612R to the corrected right captured data 622R.

[0068] As described above, in the left captured data 611L, the right captured data 611R, the left captured data 612L, and the right captured data 612R, pixels are arranged in a pitch proportional to the distance both vertically and horizontally.

[0069] FIG. 13 is a flowchart showing the parallax map generation process. This process corresponds to the process of step S12 in FIG. 10. The parallax map generation unit 53 performs a stereo matching process on the stereo images after movement amount correction to obtain parallax map data (step S40), and stores each parallax map data in the memory 22 (step S41).

[0070] Here, the parallax map generation unit 53 obtains parallax information for each part based on two images from the stereo line sensor device 11. Then, the parallax map generation unit 53 obtains parallax information for each part based on two images from the stereo line sensor device 12.

[0071] This method generally searches for mutually similar parts between two images, and takes the difference in the positions of the similar parts as the parallax. In this method, based on the parallax of the images and the displacement in the longitudinal direction of the two line sensors in the stereo line sensor device, the relative position from the line sensor is obtained as a three-dimensional coordinate value in the manner of triangulation.

[0072] FIG. 14 is a flowchart showing the point cloud data generation process. This imaging process corresponds to the process of step S13 in FIG. 10. The point cloud data generation unit 54 generates point cloud data respectively by triangulation calculation based on each parallax map data and the interval between the left and right line sensors (step S50), stores the generated point cloud data in the memory 22 (step S51), and ends the process of FIG. 14.

[0073] FIG. 15 is a flowchart showing the composite point cloud data generation process. This imaging process corresponds to the process of step S14 in FIG. 10. The point cloud data synthesis unit 55 merges a plurality of three-dimensional point cloud data 651 and 652 (step S60). This merging means synthesizing the point cloud data possessed by both point cloud data. When the point cloud data synthesis unit 55 stores the merged synthesized three-dimensional point cloud data 66 in the memory 22 (step S61), it ends the process of FIG. 15.

[0074] FIG. 16 is a flowchart showing the three-dimensional mesh data generation process. This process corresponds to the process of step S15 in FIG. 10. The three-dimensional mesh conversion unit 56 generates three-dimensional mesh data 67 including them based on the synthesized three-dimensional point cloud data 66 (step S70). In order to obtain a mesh from the point cloud data, the three-dimensional mesh conversion unit 56 forms triangles from nearby vertices and forms triangles sharing one side using nearby vertices. Repeating in the same way below, three-dimensional mesh data 67, which is a set of triangles (mesh) simulating the shape of the structure, can be obtained. Then, when the three-dimensional mesh conversion unit 56 stores the three-dimensional mesh data 67 in the memory 22 (step S71), it ends the process of FIG. 16.

[0075] FIG. 17 is a flowchart showing the display process of the three-dimensional mesh data. This process corresponds to the process of step S16 in FIG. 10. The three-dimensional mesh display unit 57 determines the viewpoint coordinates by the input of the keyboard 241 or the mouse 242 (step S80). Then, the three-dimensional mesh display unit 57 displays, on the display in perspective display, the portion of the three-dimensional mesh data 67 that falls within the visual field of the viewpoint coordinates and the attention point coordinates (step S81).

[0076] As a method of obtaining a perspective display from the three-dimensional mesh data 67, it is common to use three-dimensional drawing functions mounted on most of the graphic interfaces 23, such as OpenGL (registered trademark), Valkan, and Metal.

[0077] 《Effects of the First Embodiment》 According to the first embodiment, since images from two viewpoints can be obtained for one point of the photographed structure, three-dimensionalization by stereo matching becomes possible, and reduction of image data and shortening of processing time can be expected.

[0078] Furthermore, by stereo matching of two line sensors, an improvement in accuracy can be expected because the matching target only requires one axis (one dimension). With one stereo line sensor, occlusion occurs in the direction perpendicular to the line sensor, but with two crossed stereo line sensors, occlusion can be reduced by mutual complementation.

[0079] Also, in this embodiment, an image is obtained by a line sensor instead of an area sensor, so the data does not become huge, and three-dimensional capture with less occlusion can be performed.

[0080] <<Second Embodiment>> In the second embodiment, three or more stereo line sensors are arranged in different directions, and the speed change is estimated from the photographed images of the line sensors, making it possible to correct the image data.

[0081] FIG. 18 is a plan view showing an arrangement example of the stereo line sensor devices 11, 12, and 13 of the second embodiment. The stereo line sensor devices 11, 12, and 13 of the second embodiment intersect at an angle of 60 degrees.

[0082] The stereo line sensor device 11 is arranged such that the one-dimensional line sensors 111L and 111R are separated by a distance L and their longitudinal directions are located on the same straight line 115. The stereo line sensor device 12 is arranged such that the one-dimensional line sensors 121L and 121R are separated by a distance L and their longitudinal directions are located on the same straight line 125. The stereo line sensor device 13 is arranged such that the one-dimensional line sensors 131L and 131R are separated by a distance L and their longitudinal directions are located on the same straight line 135.

[0083] The intersections of these straight lines 115, 125, and 135 are equidistant from the line sensors 111L, 111R, equidistant from the line sensors 121L, 121R, and equidistant from the line sensors 131L, 131R. That is, the line sensors 111L, 111R, the line sensors 121L, 121R, and the line sensors 131L, 131R are arranged on concentric circles.

[0084] FIG. 19 is a plan view showing an arrangement example of the stereo line sensor devices 11, 12, and 13 of the modified example. The stereo line sensor devices 11, 12, the stereo line sensor devices 12, 13, and the stereo line sensor devices 13, 11 intersect each other.

[0085] The intersection of the straight line 115 and the straight line 125, the intersection of the straight line 125 and the straight line 135, and the intersection of the straight line 135 and the straight line 115 are located at the vertices of an equilateral triangle. As a result, the line sensors 111L, 111R, the line sensors 121L, 121R, and the line sensors 131L, 131R are arranged on concentric circles. With this arrangement, no matter what the moving direction of the subject is, any two of the stereo line sensor devices 11, 12, and 13 can scan in a direction different from the moving direction. Therefore, with two of the stereo line sensor devices 11, 12, and 13, it is possible to measure the moving amount without a moving amount sensor, and three-dimensional capture is also possible.

[0086] In the two stereo line sensor devices 11 and 12 shown in FIGS. 2A and 3A, one of them may be parallel to the moving direction of the subject. At this time, the stereo line sensor devices 11 and 12 may not be able to perform correct three-dimensional capture and may have a blind spot. By configuring the three-dimensional capture system with three or more stereo line sensor devices 11, 12, and 13 arranged at different angles, this blind spot can be eliminated.

[0087] FIG. 20 is a block diagram of the three-dimensional capture device 2 according to the second embodiment. The MPU 21 (see FIG. 1) of the three-dimensional capture device 2 implements each part of the imaging unit 51, speed change correction unit 52, disparity map generation unit 53, point cloud data generation unit 54, point cloud data synthesis unit 55, three-dimensional mesh conversion unit 56, three-dimensional mesh display unit 57, and the movement amount estimation unit 58, which are the same as those in the first embodiment, by executing a control program (not shown).

[0088] Based on the captured data, the movement amount estimation unit 58 estimates the movement amount of the subject and generates movement amount log data 641. In this embodiment, since the movement amount is estimated from the image data of two of the stereo line sensor devices 11, 12, and 13, the movement amount sensor 3 is not required, and cost reduction can be expected.

[0089] FIG. 21 is a conceptual diagram showing various data generated by the three-dimensional capture device 2. In addition to the same data as in the first embodiment, the three-dimensional capture device 2 generates disparity map raw data 681, 682 and orthographic projection raw image data 691, 692.

[0090] 《Explanation of Movement Amount Estimation》 Here, it is assumed that the images of the stereo line sensor devices 11 and 12 are used among the three stereo line sensor devices 11, 12, and 13 for explanation.

[0091] The movement amount estimation unit 58 generates and stores the disparity map raw data 681, 682 from the raw image data of the stereo line sensor devices 11 and 12. At this time, the disparity map raw data 681, 682 is an image with the horizontal axis representing distance and the vertical axis representing time. The movement amount estimation unit 58 may use either of the two stereo line sensor devices 11 and 12 as a reference. FIGS. 22A and 22B illustrate the process of estimating the movement amount of the subject based on the field of view 124 of the stereo line sensor device 12 with respect to the field of view 114 of the first stereo line sensor device 11. Here, the white rectangle is a feature part of the subject.

[0092] FIG. 22A is a plan view for explaining a method of measuring the movement amount of a subject. FIG. 22B is a perspective view for explaining a method of measuring the movement amount of a subject. The first stereo line sensor device 11 detects a feature portion of the subject at a certain time, indicating that the stereo line sensor device 12 has detected this feature portion after a lapse of a predetermined time. Since the viewing angle 114 of the stereo line sensor device 11, the viewing angle 124 of the stereo line sensor device 12, and their intersection angle are known, by substituting these into Equation (1) based on the detection of the feature portion of the subject, the movement distance L of the subject i,j can be calculated.

[0093]

Equation

[0094] Here, α L , α U , β L , β U are coefficients determined from the installation positions and orientations of the first stereo line sensor device 11 and the second stereo line sensor device 12.

[0095] FIG. 23 is a diagram for explaining a method of measuring the movement amount of a subject. For example, paying attention to a prescribed number of pixel blocks in the ortho-generated image data 691 converted into an orthographic display, a block having a high correlation with the pixel block is searched for in the ortho-generated image data 692. By searching the ortho-generated image data 691 and the ortho-generated image data 692 at any time, the movement time and movement distance of the feature portion can be obtained from the positional relationship between the two stereo line sensor devices 11 and 12 and the in-image position of the feature portion.

[0096] Here, t L i is the time when two feature portions 602L and 602R are detected in the ortho-generated image data 692. x L i,J is the coordinate at which the feature portion 602R is detected. x L i,J+1is the coordinate at which the feature portion 602L was detected.

[0097] t U i,j is the time when the feature portion 601R on the right side was detected in the orthoimage raw data 692. Note that the feature portion 601R corresponds to the feature portion 602R of the orthoimage raw data 692. x U i,J is the coordinate at which the feature portion 601R was detected.

[0098] t U i,j+1 is the time when the feature portion 601L on the left side was detected. Note that the feature portion 601L corresponds to the feature portion 602L of the orthoimage raw data 692. x U i,J+1 is the coordinate at which the feature portion 601L was detected.

[0099] By substituting this distance, the coordinates of the feature portion, and the detection time into Equation (2), the moving speed V i,j corresponding to this coordinate can be measured. Note that it is desirable to take the average or weighted average for each line of the moving speed.

Equation

[0100] From the speed of each line, the moving amount V i of the subject is obtained according to Equation (3).

Equation

[0101] This moving amount V i of the subject is saved to the movement amount log data 641. Thereby, the movement amount sensor 3 can be made unnecessary.

[0102] FIG. 24 is a flowchart showing the measurement process of the moving amount of the subject. The parallax map generation unit 53 performs stereo matching processing on each left and right captured data to generate parallax map data for each stereo line sensor device (step S90). Then, the parallax map generation unit 53 generates an orthographic image for each stereo line sensor device from the generated parallax map data (step S91), and saves the generated orthographic image as the raw parallax map data for each stereo line sensor device (step S92).

[0103] The movement amount estimation unit 58 focuses on a specified number of pixel blocks in one of the raw parallax map data, searches for pixel blocks with high correlation to that pixel block within the other raw parallax map data, and obtains the distance within the image (step S93). Then, the movement amount estimation unit 58 obtains the required time between the pixel blocks from the distance within the image of each image block and the positional relationship of the fields of view of the two stereo line sensor devices (step S94).

[0104] The movement amount estimation unit 58 divides the known distance for each pixel of each line by the required time to obtain the speed (step S95), obtains the average of the speeds calculated from each pixel as the speed of each line (step S96). Then, when the movement amount estimation unit 58 obtains the movement amount from the speed of each line (step S97), the process of FIG. 24 ends.

[0105] Also, a modification example is described below. In the three-dimensional synthesis process, the point cloud data generated from the two stereo line sensor devices is orthographically projected in the optical axis direction and converted into two-dimensional images respectively. There is a positional deviation between the two stereo line sensor devices. The deviation is obtained by taking the correlation between the two two-dimensional images and reflected in the coordinate values of one of the point cloud data to eliminate the positional deviation. After eliminating the positional deviation, the two point cloud data are saved in the memory 22 so that they can be treated as one point cloud data. Even in the process of this modification example, the movement amount can be measured without a movement amount sensor.

[0106] According to the second embodiment, a dedicated speed measurement device is not required, and cost reduction can be expected.

[0107] The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. It is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0108] Each of the above configurations, functions, processing units, processing means, etc. may be realized by hardware such as an integrated circuit, for example, for a part or all of them. Each of the above configurations, functions, etc. may also be realized by software by a processor interpreting and executing a program for realizing each function. Information such as a program, table, file, etc. for realizing each function can be placed in a recording device such as a memory, hard disk, SSD (Solid State Drive), or a recording medium such as a flash memory card, DVD (Digital Versatile Disk).

[0109] In each embodiment, the control lines and information lines show those considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. In reality, it may be considered that almost all the configurations are interconnected.

Explanation of Reference Numerals

[0110] 11, 12, 13 Stereo line sensor device 11L, 11R, 12L, 12R, 13L, 13R Line sensor device 111L, 111R, 121L, 121R, 131L, 131R Line sensor 112L, 112R, 122L, 122R Optical system 113L, 113R, 123L, 123R Optical axis 114, 116, 124, 126 Field of view 115, 125, 135 Straight line 2 Three-dimensional capture device 21 MPU 22 Memory 23 Graphic interface 24 Input interface 241 Keyboard 242 Mouse 231 Display 25 Line sensor interface 3 Movement sensor 4 Subject 41 Target 51 Photographing unit 52 Speed change correction unit 53 Disparity map generation unit 54 Point cloud data generation unit 55 Point cloud data synthesis unit 56 Three-dimensional mesh conversion unit 57 Three-dimensional mesh display unit 58 Movement amount estimation unit 611L Left photographing data 611R Right photographing data 621L Corrected left photographing data 621R Corrected right photographing data 612L Left photographing data 612R Right photographing data 622L Corrected left photographing data 622R Corrected right photographing data 631,632 Corrected disparity map data 681,682 Disparity map raw data 691,692 Orthographic projection raw image data 64,641 Movement amount log data 651,652 Three-dimensional point cloud data 66 Synthesized three-dimensional point cloud data 67 Three-dimensional mesh data 7 Subject 71 Automobile 72 Train

Claims

1. A plurality of photographing units in which a plurality of line sensors are arranged on the same straight line in the longitudinal direction, a movement amount measuring unit that measures the movement amount of the subject on the ground plane, and each of the photographing units is installed such that the longitudinal directions of the line sensors of each photographing unit intersect and the line sensors of each photographing unit exist on a plane parallel to the ground plane, A three-dimensional capture device characterized by this.

2. Further comprising a correction unit that corrects the captured data captured by each of the photographing units based on the movement amount of the subject on the ground plane, The three-dimensional capture device according to claim 1, characterized by this.

3. Three or more photographing units in which a plurality of line sensors are arranged on the same straight line in the longitudinal direction, a movement amount estimation unit that estimates the movement amount of the subject on the ground plane from the captured data captured by each of the photographing units, and each of the photographing units is installed such that the longitudinal directions of the line sensors of each photographing unit intersect and the line sensors of each photographing unit exist on a plane parallel to the ground plane, A three-dimensional capture device characterized by this.

4. Further comprising a correction unit that corrects the captured data captured by each of the photographing units based on the movement amount of the subject on the ground plane, The three-dimensional capture device according to claim 3, characterized by this.

5. A parallax map generation unit that generates parallax map data from the captured data captured by each of the photographing units, a point cloud data generation unit that generates three-dimensional point cloud data from each of the parallax map data generated by the parallax map generation unit, and a point cloud data synthesis unit that synthesizes each of the three-dimensional point cloud data created by the point cloud data generation unit. The three-dimensional capture device according to claim 3, characterized by this.

6. A three-dimensional mesh conversion unit that converts the synthesized three-dimensional point cloud data synthesized by the point cloud data synthesis unit into three-dimensional mesh data, The three-dimensional capture device according to claim 5, further comprising this and characterized by this.

7. A three-dimensional mesh display unit that converts the three-dimensional mesh data into a display image from a desired viewpoint, The three-dimensional capture device according to claim 6, further comprising this and characterized by this.

8. Each of the photographing units includes a plurality of optical systems having the same direction as the optical axis. The three-dimensional capture device according to claim 1, characterized by this.

9. The optical axes of the plurality of optical systems included in at least two of each of the photographing units are parallel. The three-dimensional capture device according to claim 8, characterized in that...

10. A plurality of imaging units in which a plurality of line sensors are arranged on the same straight line in the longitudinal direction, and a movement amount measurement unit that measures the movement amount of the subject on the ground plane, wherein the longitudinal directions of the line sensors of each of the imaging units intersect, and each of the imaging units is installed such that the line sensors of each of the imaging units exist on a plane parallel to the ground plane. A three-dimensional capture system, characterized in that...

11. Three or more imaging units in which a plurality of line sensors are arranged on the same straight line in the longitudinal direction, and a movement amount estimation unit that estimates the movement amount of the subject on the ground plane from the imaging data captured by each of the imaging units, wherein the longitudinal directions of the line sensors of each of the imaging units intersect, and each of the imaging units is installed such that the line sensors of each of the imaging units exist on a plane parallel to the ground plane. A three-dimensional capture system, characterized in that...

12. A parallax map generation unit that generates parallax map data respectively from the imaging data captured by each of the imaging units, a point cloud data generation unit that generates three-dimensional point cloud data respectively from each of the parallax map data generated by the parallax map generation unit, and a point cloud data synthesis unit that synthesizes each of the three-dimensional point cloud data created by the point cloud data generation unit. The three-dimensional capture system according to claim 11, characterized in that...

Images