Point cloud data generating apparatus

Abstract

To collect point group data capable of measuring an absolute distance only by a camera.SOLUTION: The point-cloud-data generating device 10 includes an input device 11 that inputs a plurality of images simultaneously captured from different viewpoints using a plurality of cameras 200A and 200B in a known positional relationship, a point cloud generator 12 that generates point cloud from the plurality of images and estimates positions of the plurality of cameras 200A and 200B on the point cloud, and a corrector 13 that calculates a ratio of an interval between the plurality of cameras 200A and 200B in a real space to an interval between the plurality of cameras 200A and 200B on the point cloud and corrects a scale of the point cloud using the calculated ratio.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a point cloud data generation device. [Background technology]

[0002] The Mobile Mapping System (MMS) is a system that measures roads and surrounding structures in 3D while driving a vehicle equipped with equipment such as a Global Navigation Satellite System (GNSS), laser scanner, and camera. The laser scanner makes it possible to acquire point cloud data around the road, and the camera makes it possible to identify road signs and other objects.

[0003] When applying MMS to the maintenance and inspection of utility poles, a vehicle equipped with various devices drives through the inspection area and uses the MMS to acquire 3D data and images of the inspection area. The 3D data of the utility pole can be used to detect the inclination of the utility pole and the deflection of the wires, and the images can be used to detect cracks in the utility pole. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "MMS (Mobile Mapping System)", NTT Access Service Systems Laboratories, Internet〈 URL: https: / / www.rd.ntt / as / mms / 〉 [Non-patent document 2] "Structure from Motion Overview", MathWorks, Internet <URL: https: / / jp.mathworks.com / help / vision / ug / structure-from-motion.html> Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional MMS are equipped with laser scanners to acquire 3D point cloud data of the road area. However, because laser scanners are expensive, there was a need for a method to acquire 3D data using only a camera to reduce costs.

[0006] Structure from Motion (SfM) is a technology that estimates 3D structures from multiple 2D images taken from different viewpoints. By analyzing multiple images, point cloud data that represents the shape of an object can be obtained. However, the coordinates of the obtained point cloud data are relative values, and there was a problem in that absolute values ​​of length could not be obtained from the point cloud data.

[0007] The present disclosure has been made in view of the above, and aims to collect point cloud data that allows absolute distances to be measured using only a camera. [Means for solving the problem]

[0008] A point cloud data generation device according to one aspect of the present disclosure includes an input unit that inputs multiple images captured simultaneously from different viewpoints using multiple cameras in known positional relationships; a generation unit that generates point cloud data from the multiple images and estimates the positional relationships of the multiple cameras on the point cloud data; and a correction unit that calculates the ratio of the spacing between the multiple cameras in real space to the spacing between the multiple cameras on the point cloud data, and corrects the scale of the point cloud data using the calculated ratio. [Effects of the Invention]

[0009] According to the present disclosure, point cloud data capable of measuring absolute distances can be collected using only a camera. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a vehicle equipped with a measuring device. [Figure 2] FIG. 2 is a flowchart showing an example of a process for capturing images for generating point cloud data. [Figure 3]FIG. 3 is a diagram illustrating an example of the configuration of a point cloud data generating device. [Figure 4] FIG. 4 is a flowchart illustrating an example of a process for generating point cloud data. [Figure 5] FIG. 5 is a diagram showing an example of converting the scale of point cloud data into the scale of real space. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of a point cloud data generating device. DETAILED DESCRIPTION OF THE INVENTION

[0011] An example of a vehicle equipped with a measuring device will be described with reference to FIG.

[0012] The vehicle 100 shown in the figure has two cameras 200A and 200B installed on the roof as measuring devices. The cameras 200A and 200B are installed so that their optical axes are parallel to each other and point forward of the vehicle (the direction of travel of the vehicle). The cameras 200A and 200B are fixed at a predetermined distance d apart in a direction perpendicular to the direction of travel of the vehicle and parallel to the ground.

[0013] When a trigger signal is input to each of the cameras 200A and 200B at the same timing while the vehicle 100 is traveling, the cameras 200A and 200B capture images. In other words, the cameras 200A and 200B capture images synchronously and output the images captured at the same timing. The trigger signal may be input each time the vehicle 100 travels a predetermined distance, or may be input each time a predetermined time elapses. The trigger signal may also be input when the vehicle 100 approaches a measurement target.

[0014] 1, cameras 200A and 200B are oriented in the direction of travel of the vehicle, but cameras 200A and 200B may be arranged facing rearward of vehicle 100, toward the side of vehicle 100, or diagonally forward or diagonally backward from the direction of travel of the vehicle. If necessary, the optical axes of cameras 200A and 200B may be tilted upward or downward. In any case, the relationship between the installation positions of cameras 200A and 200B in real space is known.

[0015] Three or more cameras may be installed on the vehicle 100. Even when three or more cameras are installed, all of the cameras are synchronized to capture images.

[0016] If the relative positional relationship between cameras 200A and 200B can be fixed, the present invention can also be applied to vehicles other than automobiles.

[0017] An example of a process for capturing images for generating point cloud data of a measurement object will be described with reference to the flowchart in Fig. 2. The measurement object is, for example, a utility pole installed along a road. The travel route of the vehicle 100 is determined in advance based on the position of the measurement object. It is assumed that the vehicle 100 travels along the route.

[0018] 2 will be described as being executed by a device such as a personal computer mounted on the vehicle 100, but the present invention is not limited to this. Each process may be executed by a plurality of devices.

[0019] In step S11, the device inputs a trigger signal to the cameras 200A and 200B, and the cameras 200A and 200B simultaneously capture images of the surroundings of the vehicle 100.

[0020] In step S12, the device assigns a timestamp at the time of shooting and location information of the shooting location to the images captured by cameras 200A and 200B, and stores the simultaneously captured images as a pair in a storage device provided in the device. The device may have a function for estimating its own position (e.g., GNSS), or may input the position of vehicle 100 from another device. The timestamp may be generated based on a clock built into the device. The timestamp and location information of the shooting location may use metadata assigned to the captured images by cameras 200A and 200B. The captured images may be stored in a storage device provided in cameras 200A and 200B, and after driving, a pair of captured images captured at the same time may be obtained from the storage device.

[0021] In step S13, the device determines whether or not all of the measurement objects have been photographed. For example, the device determines that all of the measurement objects have been photographed when the vehicle 100 has reached the end of the travel route. If all of the measurement objects have been photographed, the process ends.

[0022] If the driving route remains, in step S14, the device determines whether or not it has moved a predetermined distance in order to capture images of the surroundings of the vehicle 100 at predetermined intervals. If it has moved the predetermined distance, the device returns to step S11 and captures images of the surroundings of the vehicle 100.

[0023] The above process results in multiple sets of images taken simultaneously along the travel route, each set containing two images taken at the same time.

[0024] An example of the configuration of the point cloud data generating device 10 will be described with reference to FIG.

[0025] The point cloud data generating device 10 shown in the figure includes an input unit 11, a point cloud generating unit 12, and a correcting unit 13.

[0026] The input unit 11 inputs a plurality of captured images (also referred to as a set of captured images) simultaneously captured by a plurality of cameras 200A and 200B whose positional relationship in real space is known. The input unit 11 may input position information and timestamps of the captured images.

[0027] The point cloud generation unit 12 generates point cloud data from multiple captured images and estimates the capture positions of each camera 200A, 200B on the point cloud data. SfM can be used to generate the point cloud data. In SfM, feature points (e.g., corners and edges) are extracted from each of the 2D images taken from different viewpoints, and correspondence between the feature points is matched between the 2D images taken from different viewpoints. Based on the matched feature point information, the position and orientation of the camera that captured each image is estimated. Based on the estimated camera position and orientation, the 3D coordinates of the feature points are calculated, and point cloud data of the target object is generated.

[0028] Correction unit 13 corrects the scale of the point cloud data to absolute values ​​in real space. Because the scale in real space cannot be determined from the point cloud data generated by point cloud generation unit 12, correction unit 13 calculates the ratio of the distance d between cameras 200A and 200B in real space to the distance between cameras 200A and 200B in the coordinate system of the point cloud data, and corrects the coordinates of the point cloud data to the scale in real space using the calculated ratio.

[0029] For example, if camera 200B is placed perpendicular to the optical axis of camera 200A, the coordinate system of the point cloud data is defined as a coordinate system in which the position of camera 200A is the origin of the point cloud data, the optical axis direction of camera 200A is the z-axis direction, the upward direction of camera 200A is the y-axis direction, and the direction from camera 200A to camera 200B is the x-axis direction, with the x-coordinate of camera 200B being 1 unit. Because the distance d (e.g., 1 m) between cameras 200A and 200B in real space is known, the position of each point in real space with camera 200A as the origin can be identified by multiplying the coordinates of each point in the point cloud data by d.

[0030] An example of the processing of the point cloud data generating device 10 will be described with reference to the flowchart of FIG.

[0031] In step S21, the point cloud data generation device 10 inputs images captured within the travel route range. When two cameras are used, the images are read from multiple sets of time-series images captured at the same time on the travel route, and one set of images includes two images.

[0032] In step S22, the point cloud data generation device 10 generates point cloud data of the entire driving route from the read photographed images and estimates the self-position of the photographing point (camera) on the point cloud data. If the same feature points are contained in photographed images photographed at different times, point cloud data can also be generated from the photographed images photographed at different times. Therefore, the point cloud data generation device 10 processes each set of photographed images in time series to generate point cloud data of the entire driving route.

[0033] In step S23, the point cloud data generation device 10 calculates the ratio of the distance between the cameras from the distance between the cameras in real space to the distance between the shooting points (camera positions) on the point cloud data. If the distance between the cameras in real space is d and the distance between the cameras in the coordinate system of the point cloud data is y, the ratio K of the distance d between the cameras in real space to the distance y between the cameras in the coordinate system of the point cloud data can be calculated as K=d / y.

[0034] In step S24, the point cloud data generation device 10 corrects the coordinates of the point cloud to absolute values ​​based on the ratio calculated in step S23. As shown in Fig. 5, the position in real space can be identified by multiplying the coordinates in the coordinate system of the point cloud data by K. In other words, multiplying the scale of the point cloud data by K can convert it to the scale of real space.

[0035] By converting the scale of the point cloud data into the scale of real space and then adding the absolute coordinate information of the shooting position, it is possible to calculate the absolute coordinates in real space of each point of the point cloud data.

[0036] When point cloud data is generated from each set of images taken by multiple vehicles, the scale of the point cloud data can be converted to the scale of real space even if the camera spacing between the vehicles is not uniform, so the point cloud data generation device 10 can synthesize point cloud data converted to the scale of real space.

[0037] As described above, the point cloud data generation device 10 of this embodiment includes an input unit 11 that inputs multiple images captured simultaneously from different viewpoints using multiple cameras 200A, 200B in a known positional relationship, a point cloud generation unit 12 that generates point cloud data from the multiple images and estimates the positions of the multiple cameras 200A, 200B on the point cloud data, and a correction unit 13 that calculates the ratio of the spacing between the multiple cameras 200A, 200B in real space to the spacing between the multiple cameras 200A, 200B on the point cloud data and corrects the scale of the point cloud data using the calculated ratio. This makes it possible to generate point cloud data that allows distances in real space to be measured from images captured by the cameras.

[0038] The point cloud data generation device 10 described above can be, for example, a general-purpose computer system including a central processing unit (CPU) 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in Fig. 6. In this computer system, the CPU 901 executes a predetermined program loaded onto the memory 902, thereby realizing the point cloud data generation device 10. This program can be recorded on a computer-readable non-transitory recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be distributed via a network. [Explanation of symbols]

[0039] 10 Point cloud data generator 11 Input section 12 Point cloud generator 13 Correction unit 100 vehicles 200A, 200B Camera

Claims

1. an input unit that inputs a plurality of images simultaneously captured from different viewpoints using a plurality of cameras in known positional relationships; a generation unit that generates point cloud data from the plurality of images and estimates positions of the plurality of cameras on the point cloud data; a correction unit that calculates a ratio of an interval between the plurality of cameras in real space to an interval between the plurality of cameras on the point cloud data, and corrects a scale of the point cloud data using the calculated ratio. Point cloud data generator.

2. 2. The point cloud data generating device according to claim 1, the plurality of cameras are fixed to a vehicle, and the plurality of images are captured by synchronizing the capture timings of the plurality of cameras while the vehicle is traveling; Point cloud data generator.

3. 3. The point cloud data generating device according to claim 2, the input unit inputs the plurality of images in time series taken along a travel route of the vehicle; The generation unit generates point cloud data of the entire traveling route. Point cloud data generator.

4. 3. The point cloud data generating device according to claim 2, the input unit inputs a plurality of images taken using different vehicles; Synthesize multiple point cloud data converted to the scale of real space. Point cloud data generator.

Images