Road data three-dimensional reconstruction method and device, computing server, system and storage medium

Abstract

The invention provides a road data three-dimensional reconstruction method and device, a computing server, a system and a storage medium. The road data three-dimensional reconstruction method comprises an original data acquisition step, a dense point cloud acquisition step, a surface reconstruction step, a grid optimization step and a texture mapping step. Wherein the dense point cloud acquisition step comprises a binocular point cloud generation sub-step, a monocular point cloud generation sub-step and a depth map fusion sub-step. According to the invention, more real scene reconstruction is realized, the cost is reduced, and the precision is higher.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of autonomous driving technology, and more specifically, to a method, device, computing server, system and storage medium for three-dimensional reconstruction of road data. Background Art

[0002] In the process of developing autonomous driving, data collection is a prerequisite for algorithm development, and the quality of data collection determines the performance of the algorithm. Therefore, autonomous driving companies often collect a large amount of road data and build data sets by manually annotating target information. However, the amount of data required for actual algorithm training far exceeds the amount of manually annotated data, and obtaining data sets only through manual annotation cannot meet the needs of autonomous driving development.

[0003] Currently, in the public domain, there are several technical routes for 3D reconstruction of road environments:

[0004] The topological 3D reconstruction of an object is performed by scanning the point cloud with a laser radar. This solution is to collect road environment information through several rotating laser radars, and then infer the 3D contour of the object after point cloud registration. The topological 3D reconstruction of an object is performed by scanning the point cloud with a laser radar. Since the laser radar can only collect sparse point clouds and cannot obtain surface pixel information, it can generally only restore the shape of the object and cannot obtain the color information of the object;

[0005] Through the 3D reconstruction of the fusion of LiDAR and camera, this solution reconstructs the shape through LiDAR and uses the color information of the camera to complete the reconstructed texture to obtain a more realistic 3D model. Through the 3D reconstruction of the fusion of LiDAR and camera, this solution can accurately obtain the contour surface and pixel information of the object, with high reconstruction accuracy, but it requires the use of expensive LiDAR, which is costly.

[0006] Pure visual 3D reconstruction relies only on cameras. The specific algorithms can be further divided into two categories: the first is traditional algorithms, which generally use multi-view stereo vision technology to infer depth information and perform 3D reconstruction; the other is to use deep learning algorithms to directly generate 3D grids by inputting multiple frames of images. In pure visual 3D reconstruction, since the camera posture needs to be solved through inter-frame motion estimation, this step will bring errors to the entire system, and the errors are propagable. If the posture with errors is used for reconstruction, the reconstruction accuracy will further decrease. At the same time, 3D reconstruction based on traditional algorithms has the problem of scale uncertainty; reconstruction based on deep learning algorithms depends on the data set, has poor generalization ability, and is not very accurate. Summary of the invention

[0007] In order to solve the above-mentioned problems in the prior art, in a first aspect, an embodiment of the present invention provides a method for three-dimensional reconstruction of road data, the method comprising: a raw data acquisition step: acquiring forward binocular raw data, left monocular raw data, right monocular raw data, rear monocular raw data, vehicle fusion positioning information and sensor parameters; a dense point cloud acquisition step, including a binocular point cloud generation sub-step, a monocular point cloud generation sub-step and a depth map fusion sub-step, wherein the binocular point cloud generation sub-step: based on the forward binocular raw data, generating a dense 3D point cloud of the binocular data through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery and point cloud normal vector estimation; the monocular point cloud generation sub-step: performing monocular instance segmentation and predefined target filtering on multiple frames of the left monocular raw data, the right monocular raw data and the rear monocular raw data to obtain multi-frame data results, and performing multi-frame data results on the vehicle The vehicle fusion positioning information and the sensor parameters are used to perform multi-view stereo vision processing to generate a dense 3D point cloud restored by multi-frame monocular data; the depth map fusion sub-step: according to the vehicle fusion positioning information and the sensor parameters, the dense 3D point cloud of the binocular data and the dense 3D point cloud restored by the multi-frame monocular data are fused in a coordinate system to obtain fused dense 3D point cloud data; the surface reconstruction step: according to the fused dense 3D point cloud data, by constructing a signed distance field, the triangular facets of the surface are calculated and generated; the mesh optimization step: according to the triangular facets, the positions of the mesh vertices of the triangular facets are optimized based on the photometric consistency assumption to obtain the optimized triangular facets; the texture mapping step: according to the optimized triangular facets, through perspective selection, texture coordinate calculation, global color optimization and local color optimization processing, the two-dimensional pixels in the original image are projected into the optimized triangular facets by mapping to obtain a reconstructed three-dimensional model of road data.

[0008] In some embodiments, the binocular point cloud generation sub-step includes: performing distortion correction and stereo correction on the single-frame forward-looking binocular raw data, performing binocular stereo matching on the left and right target pixels, and obtaining a disparity map; performing monocular instance segmentation after performing distortion correction on the monocular data of the first camera in the forward-looking binocular raw data, so as to segment out the predefined target in the monocular data of the first camera, and obtain a first semantic map, wherein the first semantic map contains the category information of each pixel in the monocular data of the first camera; in the disparity map, according to the semantic According to the category information in the figure, the disparity value of the predefined target is set to zero, and the disparity map with the predefined target filtered out is output; according to the disparity map with the predefined target filtered out, the 3D point cloud in the binocular camera coordinate system at the current moment is restored through the geometric relationship between the disparity value and the depth value; the neighboring points of each point in the 3D point cloud in the binocular camera coordinate system at the current moment are found, and the principal component analysis is performed on each point in the 3D point cloud and its neighboring points, and the normal vector of each point in the 3D point cloud is estimated, so as to obtain a dense 3D point cloud with the predefined target filtered out and the normal vector of each point in the binocular camera coordinate system.

[0009] In some embodiments, performing monocular instance segmentation and predefined target filtering on multiple frames of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain a multi-frame data result includes: using the sensor distortion coefficient in the sensor parameters to perform distortion correction on a single frame of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain the distortion-corrected monocular data; performing monocular instance segmentation on the distortion-corrected monocular data to segment out the predefined targets in the distortion-corrected monocular data to obtain a second semantic map; setting the pixel values ​​of the predefined targets in the distortion-corrected monocular data to zero; performing distortion correction, monocular instance segmentation, and predefined target zeroing operations on each frame of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain the multi-frame data result.

[0010] In some embodiments, performing multi-view stereoscopic vision processing on the multi-frame data results, the vehicle fusion positioning information and the sensor parameters includes: aligning the vehicle fusion positioning posture to the timestamp of the frame list according to the frame list and the vehicle fusion positioning information list in the multi-frame data results, and using the sensor parameters to obtain the camera posture at all frame times with the 0th frame as the origin through coordinate transformation; extracting feature points and calculating descriptors on the frame list in the multi-frame data results to obtain a feature point list; using descriptors to perform incremental feature point matching on each perspective according to the feature point list of all frames, and outputting a matched feature point list; selecting the feature point list with large parallax according to the matched feature point list. The perspective uses the camera posture to triangulate all feature points to obtain a preprocessed 3D point cloud; the preprocessed 3D point cloud is globally bundled and adjusted to output a globally optimized 3D point cloud; the optimized feature points are used as seed points, and pixel blocks of a predetermined size around them are used as seed image blocks, and the seed image blocks are regionally grown and expanded to reconstruct pixel blocks in the neighborhood, and local nonlinear depth optimization is performed on each seed point and its neighborhood pixels to obtain 3D points corresponding to the neighborhood pixels, and this step is traversed until all neighborhood pixels are reconstructed to generate a reconstructed 3D point cloud; the reconstructed 3D point cloud is globally optimized in depth and normal vector to obtain a dense 3D point cloud restored by the multi-frame monocular data.

[0011] In some embodiments, the depth map fusion sub-step further includes: filtering out unreasonable points in the dense 3D point cloud of the binocular data and the dense 3D point cloud restored from the multi-frame monocular data through consistency constraints and visibility constraints.

[0012] In some embodiments, the mesh optimization step includes: according to the triangular facets generated in the surface reconstruction step, based on the photometric consistency assumption, taking the accumulated photometric difference after reprojection as the objective function, the vertex coordinates as the optimization variables, and the triangular facet vertices as the initial values, optimizing the mesh vertex positions of the triangular facets through global nonlinear optimization to obtain the optimized triangular facets.

[0013] In a second aspect, an embodiment of the present invention proposes a three-dimensional reconstruction device for road data, the device comprising: a raw data acquisition module, configured to: acquire forward binocular raw data, left monocular raw data, right monocular raw data, rear monocular raw data, vehicle fusion positioning information and sensor parameters; a dense point cloud acquisition module comprising a binocular point cloud generation submodule, a monocular point cloud generation submodule and a depth map fusion submodule, wherein the binocular point cloud generation submodule is configured to: generate a dense 3D point cloud of binocular data according to the forward binocular raw data through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery and point cloud normal vector estimation; the monocular point cloud generation submodule is configured to: perform monocular instance segmentation and predefined target filtering on multiple frames of the left monocular raw data, the right monocular raw data and the rear monocular raw data to obtain multi-frame data results, and perform multi-frame data results on the vehicle fusion positioning information and the sensor parameters. The sensor parameters perform multi-view stereo vision processing to generate a dense 3D point cloud restored by multi-frame monocular data; the depth map fusion submodule is configured to: fuse the dense 3D point cloud of the binocular data and the dense 3D point cloud restored by the multi-frame monocular data in a coordinate system according to the vehicle fusion positioning information and the sensor parameters, so as to obtain the fused dense 3D point cloud data; the surface reconstruction module is configured to: calculate and generate the triangular facets of the surface according to the fused dense 3D point cloud data by constructing a signed distance field; the mesh optimization module is configured to: optimize the positions of the mesh vertices of the triangular facets based on the photometric consistency assumption according to the triangular facets, so as to obtain the optimized triangular facets; the texture mapping module is configured to: project the two-dimensional pixels in the original image into the optimized triangular facets through mapping according to the optimized triangular facets through perspective selection, texture coordinate calculation, global color optimization and local color optimization processing, so as to obtain the reconstructed three-dimensional model of the road data.

[0014] In a third aspect, an embodiment of the present invention proposes a computing server, which includes a memory and a processor, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the three-dimensional reconstruction method of road data described in any of the above embodiments is implemented.

[0015] In a fourth aspect, an embodiment of the present invention proposes a road data three-dimensional reconstruction system, the system comprising: a wire-controlled vehicle, a computing platform, a front-view binocular camera, a left-view monocular camera, a right-view monocular camera, a rear-view monocular camera, a gateway domain controller, a vehicle networking communication terminal, a communication antenna, and a computing server described in the above embodiment. Among them, the computing platform, the front-view binocular camera, the left-view monocular camera, the right-view monocular camera, the rear-view monocular camera, the gateway domain controller, the vehicle networking communication terminal, and the communication antenna are loaded on the wire-controlled vehicle. The computing platform is used to: after detecting that a pre-defined trigger condition is met, send data information within a period of time including the trigger condition moment to the gateway domain controller in a package, wherein the data information at least includes: the raw data of the front-view binocular camera, the left-view monocular camera, the right-view monocular camera, and the rear-view monocular camera, vehicle fusion positioning information, and sensor parameters. The gateway domain controller is used to: forward the data information received from the computing platform to the vehicle networking communication terminal. The vehicle networking communication terminal is used to: send the data information to the computing server through the communication antenna.

[0016] In a fifth aspect, an embodiment of the present invention provides a storage medium storing computer-readable instructions, which, when executed by a processor, executes the method for three-dimensional reconstruction of road data described in any of the above embodiments.

[0017] The embodiments of the present invention disclose a three-dimensional reconstruction method and system for autonomous driving road collection data. The method is mainly used to perform three-dimensional reconstruction of static environment scenes based on data collected from mass-produced vehicles to reproduce the environment around the real vehicle.

[0018] The 3D reconstruction system for collecting data from the entire road collects the entire road environment data through cameras in four directions on the vehicle, and at the same time transmits the data together with the vehicle positioning data and sensor parameters back to the computing server for 3D reconstruction of the entire system.

[0019] The entire 3D reconstruction algorithm process of the road collection data recovers and fuses the camera data in four directions into a dense point cloud of the road environment in a coordinate system, and performs surface reconstruction, mesh optimization and texture mapping on the point cloud.

[0020] In the binocular point cloud generation submodule and the monocular point cloud generation submodule, the processing flow of filtering out predefined targets through the instance segmentation algorithm results.

[0021] Compared with the existing topological 3D reconstruction of objects by scanning point clouds through LiDAR, the present invention simultaneously restores the 3D shape and color information of surrounding objects through surface reconstruction and texture mapping, which is more realistic in scene reconstruction.

[0022] Compared with the existing three-dimensional reconstruction by fusing laser radar and camera, the present invention only uses the vehicle-mounted camera, which is lower in cost.

[0023] Compared with existing pure visual 3D reconstruction, the present invention uses the vehicle-mounted fusion positioning information and sensor parameters to solve the camera pose, which has higher accuracy than the camera pose obtained by inter-frame motion estimation in pure visual 3D reconstruction. At the same time, the scale of the fusion positioning information is fixed, and the scale of the 3D reconstruction is consistent with the real scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0025] Figure 1 A flowchart showing a method for three-dimensional reconstruction of road data according to an embodiment of the present invention is shown;

[0026] Figure 2 A schematic diagram showing a dense point cloud acquisition process according to an embodiment of the present invention is shown;

[0027] Figure 3 A schematic diagram showing a binocular point cloud generation process according to an embodiment of the present invention is shown;

[0028] Figure 4 A schematic diagram showing a monocular point cloud generation process according to an embodiment of the present invention is shown;

[0029] Figure 5 A schematic diagram showing the installation positions of various parts in a road data three-dimensional reconstruction system according to an embodiment of the present invention is shown;

[0030] Figure 6 A connection schematic diagram of a road data 3D reconstruction system according to an embodiment of the present invention is shown.

[0031] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0032] The principle and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way.

[0033] In one aspect, an embodiment of the present invention provides a method for three-dimensional reconstruction of road data. Figure 1, which shows a flow chart of a method for three-dimensional reconstruction of road data according to an embodiment of the present invention. The method comprises: an original data acquisition step, a dense point cloud acquisition step, a surface reconstruction step, a mesh optimization step and a texture mapping step.

[0034] In the raw data acquisition step, the front view binocular raw data, the left view monocular raw data, the right view monocular raw data, the rear view monocular raw data, the vehicle fusion positioning information and the sensor parameters are acquired. For example, these data can be acquired from an on-board computing platform mounted on the vehicle.

[0035] The dense point cloud acquisition step may include a binocular point cloud generation sub-step, a monocular point cloud generation sub-step and a depth map fusion sub-step. Figure 2 A schematic diagram of a dense point cloud acquisition process according to an embodiment of the present invention is shown. In the dense point cloud acquisition step, a segment of N frames of forward binocular raw data, left monocular camera raw data, right monocular camera raw data, rear monocular camera raw data, vehicle fusion positioning information, and sensor parameters are input, and after being processed by the binocular point cloud generation submodule, the monocular point cloud generation submodule, and the depth map fusion, dense 3D point cloud data of the static environment around the vehicle is obtained.

[0036] In the binocular point cloud generation sub-step, based on the forward-looking binocular raw data, a dense 3D point cloud of the binocular data is generated through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery, and point cloud normal vector estimation.

[0037] In the monocular point cloud generation sub-step, monocular instance segmentation and predefined target filtering are performed on multiple frames of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain multi-frame data results. Multi-view stereo vision processing is performed on the multi-frame data results, the vehicle fusion positioning information, and the sensor parameters to generate a dense 3D point cloud restored by the multi-frame monocular data.

[0038] In the depth map fusion sub-step, according to the vehicle fusion positioning information and sensor parameters, the dense 3D point cloud of the binocular data and the dense 3D point cloud restored from multi-frame monocular data are fused in a coordinate system to obtain the fused dense 3D point cloud data.

[0039] In the surface reconstruction step, the triangular facets of the generated surface are calculated by constructing the signed distance field based on the fused dense 3D point cloud data.

[0040] In the mesh optimization step, the positions of the mesh vertices of the triangles are optimized based on the photometric consistency assumption to obtain optimized triangles.

[0041] In the texture mapping step, according to the optimized triangular facets, through perspective selection, texture coordinate calculation, global color optimization and local color optimization processing, the two-dimensional pixels in the original image are projected onto the optimized triangular facets through mapping to obtain a reconstructed three-dimensional model of the road data.

[0042] Therefore, the three-dimensional reconstruction method of road data proposed in the embodiment of the present invention restores the three-dimensional mesh surface information and color information of the static environment in the road collection data.

[0043] refer to Figure 3 , which shows a schematic diagram of a binocular point cloud generation process according to an embodiment of the present invention. As an embodiment of the present invention, the binocular point cloud generation sub-step may include:

[0044] Perform distortion correction and stereo correction on the single-frame forward-looking binocular raw data, perform binocular stereo matching on the left and right target pixels, and obtain a disparity map;

[0045] Performing monocular instance segmentation after performing distortion correction on the monocular data of the first camera in the forward-looking binocular raw data, so as to segment out a predefined target in the monocular data of the first camera, and obtain a first semantic map, wherein the first semantic map includes category information of each pixel in the monocular data of the first camera;

[0046] In the disparity map, according to the category information in the semantic map, the disparity value of the predefined target is set to zero, and the disparity map with the predefined target filtered out is output;

[0047] According to the disparity map after filtering out the predefined target, the 3D point cloud in the binocular camera coordinate system at the current moment is restored through the geometric relationship between the disparity value and the depth value;

[0048] Find the neighboring points of each point in the 3D point cloud in the binocular camera coordinate system at the current moment, perform principal component analysis on each point in the 3D point cloud and its neighboring points, estimate the normal vector of each point in the 3D point cloud, and obtain a dense 3D point cloud and the normal vector of each point that filter out the predefined targets in the binocular camera coordinate system.

[0049] exist Figure 3 In the specific example of the binocular point cloud generation sub-step shown, a single frame of binocular raw data is input, and after binocular stereo matching, monocular instance segmentation, predefined target filtering, single frame binocular point cloud recovery and point cloud normal vector estimation, a dense 3D point cloud of a single frame of binocular data is obtained. The main steps are as follows:

[0050] a) passing a single frame of binocular raw data to the binocular stereo matching module, and passing a pre-selected camera M1 data in the binocular data to the monocular instance segmentation module;

[0051] b) In the binocular stereo matching step, the data is firstly subjected to distortion correction and stereo correction, and then the left and right target pixels are matched using a stereo matching algorithm based on deep learning to obtain a disparity map;

[0052] c) In the monocular instance segmentation step, after the monocular data is distorted and corrected, the predefined targets (including vehicles, pedestrians, bicycles, etc.) in the image are segmented through the convolutional neural network instance segmentation algorithm, and a semantic map is output; the semantic map contains the category information of each pixel, and its function is to filter out the predefined targets according to the category information in the next step;

[0053] d) In the predefined target filtering step, a disparity map and a semantic map are input, and in the disparity map, the disparity value of the predefined target is set to zero according to the category information in the semantic map, and the disparity map with the predefined target filtered out is output; the disparity points set to zero will not be restored to 3D points in the next step;

[0054] e) In the single-frame binocular point cloud restoration step, the disparity map after filtering out the predefined target is input, and the 3D point cloud in the binocular camera coordinate system at the current moment is restored through the geometric relationship between the disparity value and the depth value;

[0055] f) In the point cloud normal vector estimation, the nearest neighbor is used to find the N nearest neighbor points of each point (for example, an octree can be used for space partitioning and a KD-Tree algorithm can be used for nearest neighbor search), and then PCA (principal component analysis) is performed to estimate the normal vector of the point;

[0056] g) The binocular point cloud generation sub-step finally outputs a dense 3D point cloud and the normal vector of each point in the binocular camera coordinate system after filtering out the predefined target. The output of the i-th point of the 3D point cloud is (x i ,y i 、z i , α i , β i , γ i ), where (x i ,y i 、z i ) is the three-dimensional space coordinate of the point, (α i , β i , γ i ) is the normal vector of the point.

[0057] As an embodiment of the present invention, performing monocular instance segmentation and predefined target filtering on multiple frames in the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain the multi-frame data results may include:

[0058] Using the sensor distortion coefficient in the sensor parameters, distortion correction is performed on the single-frame left-view monocular raw data, right-view monocular raw data, and rear-view monocular raw data to obtain the distortion-corrected monocular data;

[0059] Performing monocular instance segmentation on the distortion-corrected monocular data, segmenting a predefined target in the distortion-corrected monocular data, and obtaining a second semantic map;

[0060] Setting the pixel values ​​of predefined targets in the distortion-corrected monocular data to zero;

[0061] Distortion correction, monocular instance segmentation and predefined target zeroing operations are performed on each frame of the left-view monocular raw data, the right-view monocular raw data and the rear-view monocular raw data to obtain multi-frame data results.

[0062] As an embodiment of the present invention, performing multi-view stereo vision processing on multi-frame data results, vehicle fusion positioning information and sensor parameters may include:

[0063] According to the frame list and vehicle fusion positioning information list in the multi-frame data results, align the vehicle fusion positioning posture to the timestamp of the frame list, and use the sensor parameters to obtain the camera posture at all frame times with the 0th frame as the origin through coordinate transformation;

[0064] Perform feature point extraction and descriptor calculation on the frame list in the multi-frame data result to obtain a feature point list;

[0065] According to the feature point lists of all frames, use the descriptor to perform incremental feature point matching for each view and output the matched feature point list;

[0066] According to the matched feature point list, select the view with large parallax and use the camera pose to triangulate all feature points to obtain the preprocessed 3D point cloud;

[0067] Perform global bundling adjustment on the pre-processed 3D point cloud and output the globally optimized 3D point cloud;

[0068] The optimized feature points are used as seed points, and the pixel blocks of predetermined size around them are used as seed image blocks. The seed image blocks are expanded by regional growth to reconstruct the pixel blocks in the neighborhood. Local nonlinear depth optimization is performed on each seed point and its neighborhood pixels to obtain the 3D points corresponding to the neighborhood pixels. This step is repeated until all the neighborhood pixels are reconstructed to generate a reconstructed 3D point cloud.

[0069] The reconstructed 3D point cloud is globally optimized in depth and normal vector to obtain a dense 3D point cloud restored from multi-frame monocular data.

[0070] refer to Figure 4, which shows a schematic diagram of the monocular point cloud generation process according to an embodiment of the present invention. In the monocular point cloud generation sub-step, a segment of N frames of monocular raw data is input, and each frame is subjected to monocular instance segmentation and predefined target filtering. The multi-frame results are input into the multi-view stereo vision together with the vehicle fusion positioning information, sensor distortion coefficients and external parameters for processing to obtain a dense 3D point cloud restored by multi-frame monocular data. The main steps are as follows:

[0071] a) Input a single frame of monocular raw data and sensor distortion coefficients (k1, k2, k3, p1, p2), perform distortion correction, and output the camera image after distortion correction;

[0072] b) Input the distortion-corrected image for monocular instance segmentation, segment the predefined targets (including vehicles, pedestrians, bicycles, etc.) in the image through the convolutional neural network instance segmentation algorithm, and output the semantic map;

[0073] c) setting the predefined target pixel value to zero in the image outputted in step a);

[0074] d) Repeat the process from a) to c) for each frame output until all frames have completed the predefined target filtering;

[0075] e) In the multi-view stereo vision step, input a list of all frames that complete the predefined target filtering (f t0 、f t1 …f tn ), where f tn Represents the frame at time tn; vehicle fusion positioning information list (p τ0 、p τ1 …p τk ), where p τk represents the vehicle fusion positioning information (x τk ,y τk 、z τk ,θ τk ), x τk ,y τk 、z τk is the position coordinate of the vehicle, θ τk is the yaw angle of the vehicle; sensor external parameters (rotation matrix and translation vector R, t), the specific processing steps are as follows:

[0076] i. Input the list of frames to be filtered out by the predefined target (f t0 、f t1 …f tn ) and vehicle fusion positioning information list (p τ0 、p τ1 …p τk), align the vehicle fusion positioning posture to the timestamp of the frame list according to the timestamp linear difference, and use the sensor external parameters (rotation matrix and translation vector R cb ,t cb ), and the camera position (rotation matrix and translation vector R) at all frames with frame 0 as the origin is obtained through coordinate transformation cb0 ,t cb0 Here, since the posture of the vehicle fusion positioning is derived from the result of multi-source sensor fusion positioning, the information of GNSS satellite positioning, IMU inertial navigation sensor, wheel speed odometer, visual odometer, etc. is simultaneously considered, and its positioning posture accuracy is much higher than the posture accuracy of the three-dimensional reconstruction estimation in the known method, which ensures the accuracy of subsequent three-dimensional reconstruction.

[0077] ii. Input a list of all frames that complete the predefined target filtering (f t0 、f t1 …f tn ), extract feature points and calculate descriptors, and output a list of feature points (fpl0, fpl1…fpl n ), where fpl n is the feature point list of the nth frame, with fpl n ={[fp0, fp1…fp m ]、[ds0、ds1…ds m ]}, ;fp m is the mth feature point in this frame, ds m is the descriptor of the feature point.

[0078] iii. Input the feature point list of all frames (fpl0, fpl1…fpl n ), use the descriptor to perform incremental feature point matching for each view, and output a list of matched feature points (fpla0, fpla1…fpla n ), where the feature point list after matching the nth frame is fpla n ={[fp0, fp1…fp m ]、[ds0、ds1…ds m |, [ma0, ma1…ma m}, The mth feature point matching result is an N×1 vector. The i-th element of the vector is j, indicating that the feature point matches the j-th feature point in the i-th frame.

[0079] iv. Input the matched feature point list (fpla0, fpla1…fpla n ), select the viewing angle with large parallax and use the camera pose to triangulate all feature points, and output the preprocessed 3D point cloud;

[0080] y. Input the preprocessed 3D point cloud, perform global bundling adjustment on it, and output the globally optimized 3D point cloud;

[0081] Vi. Take the optimized feature point as the seed point and an N*N pixel block around it as the seed image block (patch);

[0082] Vii. Perform regional growth expansion on the seed image block to reconstruct the pixel block in the neighborhood. This step performs local nonlinear depth optimization on each seed point and its neighborhood pixels, outputs the 3D points corresponding to the neighborhood pixels, and traverses this step until all the neighborhood pixels are reconstructed;

[0083] viii. Input all the reconstructed 3D point clouds in the previous step and perform global depth and normal vector optimization.

[0084] As an embodiment of the present invention, the depth map fusion sub-step may further include: filtering out unreasonable points in the dense 3D point cloud of binocular data and the dense 3D point cloud restored from multi-frame monocular data through consistency constraints and visibility constraints.

[0085] As a specific example of the depth map fusion sub-step, the vehicle positioning information list (p τ0 、p τ1 …p τk ) and sensor extrinsic information (rotation matrix and translation vector R, t), the dense 3D point cloud obtained by all cameras is integrated into one coordinate system through coordinate transformation, and unreasonable points are filtered out through consistency constraints and visibility constraints. Among them, the consistency constraint means that in three-dimensional space, the Euclidean distance between points must be greater than a certain threshold. If there are two points less than the threshold, they should be the same point, and the close points should be merged. The visibility constraint means that there can only be one point in the ray passing through the optical center of the camera. If there are more than two points on the same optical center ray, due to occlusion, only the point closest to the optical center can be seen, and the distant points should be filtered out.

[0086] As a specific example, in the surface reconstruction step, the fused 3D point cloud and normal vector are input to calculate and generate the triangular facets of the surface (vertex coordinates Ver_Loc, triangle vertex index matrix Id_triangle). In the implementation process, the PSR (Poisson surface reconstruction) method is preferably used for surface reconstruction.

[0087] As an embodiment of the present invention, the mesh optimization step may include: according to the triangular facets generated in the surface reconstruction step, based on the photometric consistency assumption, taking the cumulative photometric difference after reprojection as the objective function, the vertex coordinates as the optimization variables, and the triangular facet vertices as the initial values, optimizing the mesh vertex positions of the triangular facets through global nonlinear optimization to obtain optimized triangular facets.

[0088] As a specific example, in the mesh optimization step, the triangles calculated by surface reconstruction are input, and based on the photometric consistency assumption, the cumulative photometric difference after reprojection is used as the objective function, the vertex coordinates Ver_Loc are the optimization variables, and the input triangle vertex is used as the initial value. Through global nonlinear optimization, the mesh vertex positions of the triangles are optimized, and the optimized triangles (vertex coordinates Ver_Loc, triangle vertex index matrix Id_triangle) are output. The photometric consistency assumption means that the photometric (RGB value) of the same point should be consistent between two different frames.

[0089] As a specific example, in the texture mapping step, the optimized triangle face (vertex coordinates Ver_Loc, triangle vertex index matrix Id_triangle) is input, and the perspective selection (selecting the perspective closest to the normal vector of the triangle face), texture coordinate calculation, global color optimization (optimizing the color adjustment rate g of each face) is performed. i , so that several faces with common vertices have the same luminosity value at the vertex, establish a matrix equation and solve it using the least squares method) and local color optimization (Poisson image editing is preferred), project the two-dimensional pixels in the original image onto the 3D mesh facets by mapping, and output the reconstructed three-dimensional model.

[0090] Note that for clarity purposes, Figure 2-Figure 4 In the present invention, steps and sub-steps are respectively shown as modules, sub-modules, etc., which are interchangeable with the expressions of steps and sub-steps.

[0091] On the other hand, an embodiment of the present invention proposes a three-dimensional reconstruction device for road data, which may include an original data acquisition module, a dense point cloud acquisition module, a surface reconstruction module, a mesh optimization module and a texture mapping module.

[0092] The raw data acquisition module can be configured to: acquire forward binocular raw data, left monocular raw data, right monocular raw data, rearward monocular raw data, vehicle fusion positioning information and sensor parameters.

[0093] The dense point cloud acquisition module may include a binocular point cloud generation submodule, a monocular point cloud generation submodule and a depth map fusion submodule.

[0094] The stereo point cloud generation submodule can be configured to generate a dense 3D point cloud of the stereo data based on the forward-looking stereo raw data through stereo stereo matching, monocular instance segmentation, predefined target filtering, single-frame stereo point cloud recovery, and point cloud normal vector estimation.

[0095] The monocular point cloud generation submodule can be configured to: perform monocular instance segmentation and predefined target filtering on multiple frames in the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain multi-frame data results, perform multi-view stereo vision processing on the multi-frame data results, vehicle fusion positioning information and sensor parameters, and generate a dense 3D point cloud restored by multi-frame monocular data.

[0096] The depth map fusion submodule can be configured to: fuse the dense 3D point cloud of binocular data and the dense 3D point cloud restored by multi-frame monocular data in one coordinate system according to the vehicle fusion positioning information and sensor parameters, and obtain fused dense 3D point cloud data.

[0097] The surface reconstruction module can be configured to calculate and generate triangular facets of the surface by constructing a signed distance field according to the fused dense 3D point cloud data.

[0098] The mesh optimization module can be configured to: optimize the positions of mesh vertices of the triangular facets based on the photometric consistency assumption according to the triangular facets to obtain optimized triangular facets.

[0099] The texture mapping module can be configured to: project the two-dimensional pixels in the original image onto the optimized triangular facets through mapping according to the optimized triangular facets through perspective selection, texture coordinate calculation, global color optimization and local color optimization processing, so as to obtain a reconstructed three-dimensional model of road data.

[0100] It should be noted that the functions implemented by each module in the road data three-dimensional reconstruction device proposed in the embodiment of the present invention correspond one-to-one to the various steps of the road data three-dimensional reconstruction method described above. For its specific implementation methods, examples and beneficial effects, please refer to the above description of the method.

[0101] In another aspect, an embodiment of the present invention provides a computing server, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the three-dimensional reconstruction method of road data described in any of the above embodiments is implemented.

[0102] On the other hand, an embodiment of the present invention proposes a three-dimensional reconstruction system for road data, including: a wire-controlled vehicle, a computing platform, a front-view binocular camera, a left-view monocular camera, a right-view monocular camera, a rear-view monocular camera, a gateway domain controller, an Internet of Vehicles communication terminal, a communication antenna, and a computing server described according to the above embodiment.

[0103] refer to Figure 5-Figure 6 ,in Figure 5 It shows a schematic diagram of the installation positions of various parts in the road data three-dimensional reconstruction system according to an embodiment of the present invention, Figure 6The connection diagram of the road data 3D reconstruction system according to an embodiment of the present invention is shown. The computing platform, the front binocular camera, the left monocular camera, the right monocular camera, the rear monocular camera, the gateway domain controller, the Internet of Vehicles communication terminal and the communication antenna are installed on the wire-controlled vehicle.

[0104] The computing platform can be used to: after detecting that a pre-defined trigger condition is met, package and send data information within a period of time including the trigger condition moment to the gateway domain controller, where the data information at least includes: the original data of the front-view binocular camera, the left-view monocular camera, the right-view monocular camera, the rear-view monocular camera, the vehicle fusion positioning information and sensor parameters.

[0105] The gateway domain controller can be used to forward the data information received from the computing platform to the Internet of Vehicles communication terminal.

[0106] The Internet of Vehicles communication terminal can be used to send data information to the computing server via the communication antenna.

[0107] In another aspect, an embodiment of the present invention provides a storage medium storing computer-readable instructions, which, when executed by a processor, executes the method for three-dimensional reconstruction of road data described in any of the above embodiments.

[0108] The technical solution for 3D reconstruction of road data proposed in the embodiment of the present invention adopts a "shadow mode" data collection method. During the driving process of mass-produced vehicles, according to some trigger conditions, the vehicle data for a period of time before and after the trigger conditions are recorded and sent back to the background for scene reproduction for iterative software development. This solves the problem of insufficient data acquisition in the prior art.

[0109] The embodiments of the present invention disclose a three-dimensional reconstruction method and system for autonomous driving road collection data. The method is mainly used to perform three-dimensional reconstruction of static environment scenes based on data collected from mass-produced vehicles to reproduce the environment around the real vehicle.

[0110] The 3D reconstruction system for collecting data from the entire road collects the entire road environment data through cameras in four directions on the vehicle, and at the same time transmits the data together with the vehicle positioning data and sensor parameters back to the computing server for 3D reconstruction of the entire system.

[0111] The entire 3D reconstruction algorithm process of the road collection data recovers and fuses the camera data in four directions into a dense point cloud of the road environment in a coordinate system, and performs surface reconstruction, mesh optimization and texture mapping on the point cloud.

[0112] In the binocular point cloud generation submodule and the monocular point cloud generation submodule, the processing flow of filtering out predefined targets through the instance segmentation algorithm results.

[0113] Compared with the existing topological 3D reconstruction of objects by scanning point clouds through LiDAR, the present invention simultaneously restores the 3D shape and color information of surrounding objects through surface reconstruction and texture mapping, which is more realistic in scene reconstruction.

[0114] Compared with the existing three-dimensional reconstruction by fusing laser radar and camera, the present invention only uses the vehicle-mounted camera, which is lower in cost.

[0115] Compared with existing pure visual 3D reconstruction, the present invention uses the vehicle-mounted fusion positioning information and sensor parameters to solve the camera pose, which has higher accuracy than the camera pose obtained by inter-frame motion estimation in pure visual 3D reconstruction. At the same time, the scale of the fusion positioning information is fixed, and the scale of the 3D reconstruction is consistent with the real scene.

[0116] For the purpose of illustration, the foregoing description of the embodiments of the present invention has been given, which is not exhaustive nor intended to limit the present invention to the disclosed exact form. It will be appreciated by those skilled in the art that various changes can be made without departing from the scope of the present invention, and that the elements therein can be replaced with equivalents. In addition, without departing from the basic scope of the present invention, many modifications can be made so that specific situations or materials are adapted to the teachings of the present invention. Therefore, the present invention is not intended to be limited to the specific embodiments disclosed as the best mode for realizing the present invention, and the present invention will include all embodiments falling within the scope of the appended claims.

Claims

1. A method for three-dimensional reconstruction of road data, characterized in that: The method comprises: Raw data acquisition steps: acquiring forward binocular raw data, left monocular raw data, right monocular raw data, rear monocular raw data, vehicle fusion positioning information and sensor parameters; The dense point cloud acquisition step includes a binocular point cloud generation sub-step, a monocular point cloud generation sub-step and a depth map fusion sub-step, wherein: The binocular point cloud generation sub-step: based on the forward-looking binocular original data, through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery and point cloud normal vector estimation, Generate dense 3D point cloud of stereo data; The monocular point cloud generation sub-step comprises: performing monocular instance segmentation and predefined target filtering on multiple frames of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain a multi-frame data result, performing multi-view stereo vision processing on the multi-frame data result, the vehicle fusion positioning information, and the sensor parameters to generate a dense 3D point cloud restored by the multi-frame monocular data; The depth map fusion sub-step: according to the vehicle fusion positioning information and the sensor parameters, the dense 3D point cloud of the binocular data and the dense 3D point cloud restored by the multi-frame monocular data are fused in a coordinate system to obtain fused dense 3D point cloud data; Surface reconstruction step: According to the fused dense 3D point cloud data, by constructing a signed distance field, calculating and generating triangular facets of the surface; Mesh optimization step: according to the triangular facet, optimizing the positions of the mesh vertices of the triangular facet based on the photometric consistency assumption to obtain an optimized triangular facet; Texture mapping step: According to the optimized triangular facets, through perspective selection, texture coordinate calculation, global color optimization and local color optimization processing, the two-dimensional pixels in the original image are projected onto the optimized triangular facets by mapping to obtain a reconstructed three-dimensional model of road data.

2. The method for three-dimensional reconstruction of road data according to claim 1, characterized in that: The binocular point cloud generation sub-step comprises: Performing distortion correction and stereo correction on the single-frame forward-looking binocular raw data, and performing binocular stereo matching on the left and right target pixels to obtain a disparity map; Performing monocular instance segmentation after performing distortion correction on the monocular data of the first camera in the forward-looking binocular raw data, so as to segment out a predefined target in the monocular data of the first camera, and obtain a first semantic map, wherein the first semantic map includes category information of each pixel in the monocular data of the first camera; In the disparity map, according to the category information in the semantic map, the disparity value of the predefined target is set to zero, and the disparity map with the predefined target filtered out is output; According to the disparity map after filtering out the predefined target, the 3D point cloud in the binocular camera coordinate system at the current moment is restored through the geometric relationship between the disparity value and the depth value; Find the neighboring points of each point in the 3D point cloud in the binocular camera coordinate system at the current moment, perform principal component analysis on each point in the 3D point cloud and its neighboring points, estimate the normal vector of each point in the 3D point cloud, and obtain a dense 3D point cloud and the normal vector of each point in the binocular camera coordinate system that filters out predefined targets.

3. The method for three-dimensional reconstruction of road data according to claim 1, characterized in that: Performing monocular instance segmentation and predefined target filtering on multiple frames of the left-view monocular original data, the right-view monocular original data, and the rear-view monocular original data to obtain a multi-frame data result includes: Using the sensor distortion coefficient in the sensor parameters, distortion correction is performed on the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data of a single frame to obtain distortion-corrected monocular data; Performing monocular instance segmentation on the distortion-corrected monocular data to segment a predefined target in the distortion-corrected monocular data to obtain a second semantic map; Setting the pixel value of the predefined target in the distortion-corrected monocular data to zero; Distortion correction, monocular instance segmentation, and predefined target zeroing operations are performed on each frame of the left-view monocular original data, the right-view monocular original data, and the rear-view monocular original data to obtain the multi-frame data result.

4. The method for three-dimensional reconstruction of road data according to claim 1, characterized in that: Performing multi-view stereo vision processing on the multi-frame data results, the vehicle fusion positioning information and the sensor parameters includes: According to the frame list and the vehicle fusion positioning information list in the multi-frame data result, align the vehicle fusion positioning posture to the timestamp of the frame list, and use the sensor parameters to obtain the camera posture at all frame times with the 0th frame as the origin through coordinate transformation; Performing feature point extraction and descriptor calculation on the frame list in the multi-frame data result to obtain a feature point list; According to the feature point lists of all frames, use the descriptor to perform incremental feature point matching for each view and output the matched feature point list; According to the matched feature point list, a viewing angle with a large parallax is selected to triangulate all feature points using the camera pose to obtain a preprocessed 3D point cloud; Performing global bundling adjustment on the preprocessed 3D point cloud, and outputting a globally optimized 3D point cloud; The optimized feature points are used as seed points, and the pixel blocks of predetermined size around them are used as seed image blocks. The seed image blocks are expanded by regional growth to reconstruct the pixel blocks in the neighborhood. Local nonlinear depth optimization is performed on each seed point and its neighborhood pixels to obtain the 3D points corresponding to the neighborhood pixels. This step is repeated until all the neighborhood pixels are reconstructed to generate a reconstructed 3D point cloud. The reconstructed 3D point cloud is globally optimized in depth and normal vector to obtain a dense 3D point cloud restored by the multi-frame monocular data.

5. The method for three-dimensional reconstruction of road data according to any one of claims 1 to 4, characterized in that: The depth map fusion sub-step also includes: filtering out unreasonable points in the dense 3D point cloud of the binocular data and the dense 3D point cloud restored from the multi-frame monocular data through consistency constraints and visibility constraints.

6. The method for three-dimensional reconstruction of road data according to any one of claims 1 to 4, characterized in that: The grid optimization step comprises: According to the triangular facets generated in the surface reconstruction step, based on the photometric consistency assumption, the accumulated photometric difference after reprojection is used as the objective function, the vertex coordinates are used as the optimization variables, and the triangular facet vertices are used as the initial values. Through global nonlinear optimization, the mesh vertex positions of the triangular facets are optimized to obtain the optimized triangular facets.

7. A three-dimensional reconstruction device for road data, characterized in that: The device comprises: A raw data acquisition module is configured to: acquire forward binocular raw data, left monocular raw data, right monocular raw data, rear monocular raw data, vehicle fusion positioning information and sensor parameters; The dense point cloud acquisition module includes a binocular point cloud generation submodule, a monocular point cloud generation submodule and a depth map fusion submodule, among which: The binocular point cloud generation submodule is configured to: generate a dense 3D point cloud of binocular data according to the forward-looking binocular raw data through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery, and point cloud normal vector estimation; The monocular point cloud generation submodule is configured to: perform monocular instance segmentation and predefined target filtering on multiple frames of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain multi-frame data results, perform multi-view stereo vision processing on the multi-frame data results, the vehicle fusion positioning information, and the sensor parameters, and generate a dense 3D point cloud restored by the multi-frame monocular data; The depth map fusion submodule is configured to: fuse the dense 3D point cloud of the binocular data and the dense 3D point cloud restored by the multi-frame monocular data in a coordinate system according to the vehicle fusion positioning information and the sensor parameters, to obtain fused dense 3D point cloud data; A surface reconstruction module is configured to: calculate and generate triangular facets of the surface by constructing a signed distance field according to the fused dense 3D point cloud data; A mesh optimization module is configured to: optimize the positions of mesh vertices of the triangular facets based on the photometric consistency assumption according to the triangular facets to obtain an optimized triangular facet; The texture mapping module is configured to: project the two-dimensional pixels in the original image onto the optimized triangular facets through mapping according to the optimized triangular facets through perspective selection, texture coordinate calculation, global color optimization and local color optimization processing, so as to obtain a reconstructed three-dimensional model of road data.

8. A computing server, characterized in that: The computing server includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method for three-dimensional reconstruction of road data according to any one of claims 1 to 6 is implemented.

9. A road data three-dimensional reconstruction system, characterized in that: The system comprises: a wire-controlled vehicle, a computing platform, a front-view binocular camera, a left-view monocular camera, a right-view monocular camera, a rear-view monocular camera, a gateway domain controller, a vehicle networking communication terminal, a communication antenna, and a computing server according to claim 8, wherein: The computing platform, the front-view binocular camera, the left-view monocular camera, the right-view monocular camera, the rear-view monocular camera, the gateway domain controller, the Internet of Vehicles communication terminal and the communication antenna are mounted on the wire-controlled vehicle. The computing platform is used to: after detecting that a predefined trigger condition is met, package and send data information within a period of time including the trigger condition moment to the gateway domain controller, wherein the data information at least includes: the original data of the front-view binocular camera, the left-view monocular camera, the right-view monocular camera, and the rear-view monocular camera, vehicle fusion positioning information, and sensor parameters; The gateway domain controller is used to: forward the data information received from the computing platform to the Internet of Vehicles communication terminal; The Internet of Vehicles communication terminal is used to send the data information to the computing server via the communication antenna.

10. A storage medium, characterized in that: Computer-readable instructions are stored, and when the instructions are executed by a processor, the method according to any one of claims 1 to 6 is executed.

Images