Road data three-dimensional reconstruction method, device, computing server, system and storage medium

Abstract

The present invention provides a method, device, computing server, system, and storage medium for 3D road data reconstruction. The method includes: raw data acquisition, dense point cloud acquisition, surface reconstruction, mesh optimization, and texture mapping. The dense point cloud acquisition step includes binocular point cloud generation, monocular point cloud generation, and depth map fusion. This method achieves more realistic scene reconstruction, reduces costs, and achieves higher accuracy.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of autonomous driving technology, and more specifically, to a method, device, computing server, system, and storage medium for three-dimensional reconstruction of road data. Background Art

[0002] In the development of autonomous driving, data collection is a prerequisite for algorithm development, and the quality of data collection determines the performance of the algorithm. Therefore, autonomous driving companies often collect large amounts of road data and construct datasets by manually annotating target information. However, the amount of data required for actual algorithm training far exceeds the amount of manually annotated data, and datasets obtained solely through manual annotation cannot meet the needs of autonomous driving development.

[0003] Currently, in the public domain, there are several technical approaches for 3D reconstruction of road environments:

[0004] Using LiDAR to scan point clouds for topological 3D reconstruction of objects. This approach uses several rotating LiDARs to collect road environment information, then infers the object's 3D outline through point cloud registration. Because LiDAR can only collect sparse point clouds and cannot obtain surface pixel information, it can generally only restore the object's shape, not its color.

[0005] This solution uses LiDAR and camera fusion for 3D reconstruction. This solution uses LiDAR to reconstruct shape and uses the camera's color information to complement the reconstructed texture, resulting in a more realistic 3D model. This LiDAR and camera fusion 3D reconstruction solution accurately captures the object's contour surface and pixel information, achieving high reconstruction accuracy. However, this requires expensive LiDAR, which is a relatively high cost.

[0006] Purely visual 3D reconstruction relies solely on cameras, and the specific algorithms can be further divided into two categories: the first is traditional algorithms, which generally use multi-view stereo vision technology to infer depth information for 3D reconstruction; the other uses deep learning algorithms, which directly generate 3D meshes from multiple frames of image input. Purely visual 3D reconstruction requires inter-frame motion estimation to determine the camera's pose, which introduces errors into the entire system. Errors are propagative, and using poses with errors for reconstruction further degrades reconstruction accuracy. Furthermore, 3D reconstruction based on traditional algorithms suffers from scale uncertainty; reconstruction based on deep learning algorithms relies on datasets, has poor generalization capabilities, and is also not very accurate. Summary of the Invention

[0007] In order to solve the above-mentioned problems in the prior art, in a first aspect, an embodiment of the present invention provides a method for three-dimensional reconstruction of road data, the method comprising: a raw data acquisition step: acquiring forward binocular raw data, left monocular raw data, right monocular raw data, rear monocular raw data, vehicle fusion positioning information and sensor parameters; a dense point cloud acquisition step, comprising a binocular point cloud generation sub-step, a monocular point cloud generation sub-step and a depth map fusion sub-step, wherein the binocular point cloud generation sub-step: based on the forward binocular raw data, generating a dense 3D point cloud of the binocular data through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery and point cloud normal vector estimation; the monocular point cloud generation sub-step: performing monocular instance segmentation and predefined target filtering on multiple frames of the left monocular raw data, the right monocular raw data and the rear monocular raw data to obtain multi-frame data results, and performing multi-frame data matching on the multi-frame data results, the vehicle The vehicle fusion positioning information and the sensor parameters are used to perform multi-view stereo vision processing to generate a dense 3D point cloud restored by multi-frame monocular data; the depth map fusion sub-step: according to the vehicle fusion positioning information and the sensor parameters, the dense 3D point cloud of the binocular data and the dense 3D point cloud restored by the multi-frame monocular data are fused in a coordinate system to obtain fused dense 3D point cloud data; the surface reconstruction step: according to the fused dense 3D point cloud data, by constructing a signed distance field, the triangular facets of the surface are calculated and generated; the mesh optimization step: according to the triangular facet, the positions of the mesh vertices of the triangular facet are optimized based on the photometric consistency assumption to obtain the optimized triangular facet; the texture mapping step: according to the optimized triangular facet, through perspective selection, texture coordinate calculation, global color optimization and local color optimization processing, the two-dimensional pixels in the original image are projected onto the optimized triangular facet by mapping to obtain a reconstructed three-dimensional model of the road data.

[0008] In some embodiments, the binocular point cloud generation sub-step includes: performing distortion correction and stereo correction on the single-frame forward-looking binocular raw data, performing binocular stereo matching on the left and right target pixels, and obtaining a disparity map; performing monocular instance segmentation on the monocular data of the first camera in the forward-looking binocular raw data after distortion correction to segment the predefined target in the monocular data of the first camera, and obtaining a first semantic map, wherein the first semantic map contains the category information of each pixel in the monocular data of the first camera; in the disparity map, according to the semantic The category information in the figure is used to set the disparity value of the predefined target to zero, and output the disparity map with the predefined target filtered out; according to the disparity map with the predefined target filtered out, the 3D point cloud in the binocular camera coordinate system at the current moment is restored through the geometric relationship between the disparity value and the depth value; the neighboring points of each point in the 3D point cloud in the binocular camera coordinate system at the current moment are found, and principal component analysis is performed on each point in the 3D point cloud and its neighboring points, and the normal vector of each point in the 3D point cloud is estimated to obtain a dense 3D point cloud with the predefined target filtered out in the binocular camera coordinate system and the normal vector of each point.

[0009] In some embodiments, performing monocular instance segmentation and predefined target filtering on multiple frames of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain multi-frame data results includes: utilizing the sensor distortion coefficient in the sensor parameters to perform distortion correction on the single-frame left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain distortion-corrected monocular data; performing monocular instance segmentation on the distortion-corrected monocular data to segment out the predefined targets in the distortion-corrected monocular data to obtain a second semantic map; setting the pixel values ​​of the predefined targets in the distortion-corrected monocular data to zero; performing distortion correction, monocular instance segmentation, and predefined target zeroing operations on each frame of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain the multi-frame data results.

[0010] In some embodiments, performing multi-view stereo vision processing on the multi-frame data results, the vehicle fusion positioning information and the sensor parameters includes: aligning the vehicle fusion positioning posture to the timestamp of the frame list according to the frame list and the vehicle fusion positioning information list in the multi-frame data results, and using the sensor parameters to obtain the camera posture at all frame moments with the 0th frame as the origin through coordinate transformation; performing feature point extraction and descriptor calculation on the frame list in the multi-frame data results to obtain a feature point list; using the descriptor to perform incremental feature point matching on each perspective according to the feature point list of all frames, and outputting a matched feature point list; selecting the feature point with the largest parallax according to the matched feature point list. The perspective uses the camera pose to triangulate all feature points to obtain a preprocessed 3D point cloud; the preprocessed 3D point cloud is globally bundled and adjusted to output a globally optimized 3D point cloud; the optimized feature points are used as seed points, and pixel blocks of a predetermined size around them are used as seed image blocks. The seed image blocks are subjected to regional growth expansion to reconstruct the pixel blocks in the neighborhood, and local nonlinear depth optimization is performed on each seed point and its neighborhood pixels to obtain the 3D points corresponding to the neighborhood pixels. This step is traversed until all neighborhood pixels are reconstructed to generate a reconstructed 3D point cloud; the reconstructed 3D point cloud is globally optimized for depth and normal vectors to obtain a dense 3D point cloud restored from the multi-frame monocular data.

[0011] In some embodiments, the depth map fusion sub-step further includes: filtering out unreasonable points in the dense 3D point cloud of the binocular data and the dense 3D point cloud restored from the multi-frame monocular data through consistency constraints and visibility constraints.

[0012] In some embodiments, the mesh optimization step includes: according to the triangular facets generated in the surface reconstruction step, based on the photometric consistency assumption, taking the cumulative photometric difference after reprojection as the objective function, the vertex coordinates as the optimization variables, and the triangular facet vertices as the initial values, optimizing the mesh vertex positions of the triangular facets through global nonlinear optimization to obtain the optimized triangular facets.

[0013] In a second aspect, an embodiment of the present invention proposes a three-dimensional reconstruction device for road data, the device comprising: a raw data acquisition module, configured to: acquire forward binocular raw data, left monocular raw data, right monocular raw data, rear monocular raw data, vehicle fusion positioning information and sensor parameters; a dense point cloud acquisition module comprising a binocular point cloud generation submodule, a monocular point cloud generation submodule and a depth map fusion submodule, wherein the binocular point cloud generation submodule is configured to: generate a dense 3D point cloud of binocular data based on the forward binocular raw data through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery and point cloud normal vector estimation; the monocular point cloud generation submodule is configured to: perform monocular instance segmentation and predefined target filtering on multiple frames of the left monocular raw data, the right monocular raw data and the rear monocular raw data to obtain multi-frame data results, and perform multi-frame data matching on the multi-frame data results, the vehicle fusion positioning information and the sensor parameters. The sensor parameters perform multi-view stereo vision processing to generate a dense 3D point cloud restored by multi-frame monocular data; the depth map fusion submodule is configured to: fuse the dense 3D point cloud of the binocular data and the dense 3D point cloud restored by the multi-frame monocular data in a coordinate system according to the vehicle fusion positioning information and the sensor parameters, so as to obtain the fused dense 3D point cloud data; the surface reconstruction module is configured to: calculate and generate the triangular facets of the surface according to the fused dense 3D point cloud data by constructing a signed distance field; the mesh optimization module is configured to: optimize the positions of the mesh vertices of the triangular facets based on the photometric consistency assumption according to the triangular facets, so as to obtain the optimized triangular facets; the texture mapping module is configured to: project the two-dimensional pixels in the original image into the optimized triangular facets by mapping according to the optimized triangular facets through perspective selection, texture coordinate calculation, global color optimization and local color optimization processing, so as to obtain the reconstructed three-dimensional model of the road data.

[0014] In a third aspect, an embodiment of the present invention proposes a computing server, which includes a memory and a processor, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the three-dimensional reconstruction method of road data described in any of the above embodiments is implemented.

[0015] In a fourth aspect, embodiments of the present invention provide a three-dimensional road data reconstruction system, comprising: a controlled-by-wire vehicle, a computing platform, a front binocular camera, a left monocular camera, a right monocular camera, a rear monocular camera, a gateway domain controller, a vehicle-to-vehicle communication terminal, a communication antenna, and a computing server according to the above embodiments. The computing platform, the front binocular camera, the left monocular camera, the right monocular camera, the rear monocular camera, the gateway domain controller, the vehicle-to-vehicle communication terminal, and the communication antenna are mounted on the controlled-by-wire vehicle. The computing platform is configured to, upon detecting that a predefined trigger condition has been met, package and send data information for a period of time including the triggering moment to the gateway domain controller, wherein the data information includes at least raw data from the front binocular camera, the left monocular camera, the right monocular camera, and the rear monocular camera, vehicle fusion positioning information, and sensor parameters. The gateway domain controller is configured to forward the data information received from the computing platform to the vehicle-to-vehicle communication terminal. The vehicle-to-vehicle communication terminal is configured to transmit the data information to the computing server via the communication antenna.

[0016] In a fifth aspect, an embodiment of the present invention provides a storage medium storing computer-readable instructions. When the instructions are executed by a processor, the method for three-dimensional reconstruction of road data described in any of the above embodiments is executed.

[0017] The embodiments of the present invention disclose a three-dimensional reconstruction method and system for autonomous driving road data collection. The method is mainly used to perform three-dimensional reconstruction of static environmental scenes based on data collected from mass-produced vehicles to reproduce the environment around the real vehicle.

[0018] The 3D reconstruction system for collecting data from the entire road uses cameras in four directions on the vehicle to collect data on the entire road environment. At the same time, this data, along with vehicle positioning data and sensor parameters, is transmitted back to the computing server for 3D reconstruction of the entire system.

[0019] The entire 3D reconstruction algorithm process of the road collection data recovers and fuses the camera data in four directions into a dense point cloud of the road environment in a coordinate system, and then performs surface reconstruction, mesh optimization and texture mapping on the point cloud.

[0020] In the binocular point cloud generation submodule and the monocular point cloud generation submodule, the processing flow of filtering out predefined targets through the results of the instance segmentation algorithm.

[0021] Compared with the existing topological three-dimensional reconstruction of objects through laser radar scanning point clouds, the present invention simultaneously restores the three-dimensional shape and color information of surrounding objects through surface reconstruction and texture mapping, which is more realistic in scene reconstruction.

[0022] Compared with the existing three-dimensional reconstruction through the fusion of laser radar and camera, the present invention only uses the vehicle-mounted camera, which is lower in cost.

[0023] Compared to existing purely visual 3D reconstruction methods, this method uses fused vehicle-based positioning information and sensor parameters to calculate the camera pose. This method achieves higher accuracy than purely visual 3D reconstruction, which relies on inter-frame motion estimation. Furthermore, the scale of the fused positioning information is fixed, ensuring that the scale of the 3D reconstruction matches the real scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which:

[0025] Figure 1 A flowchart of a method for three-dimensional reconstruction of road data according to an embodiment of the present invention is shown;

[0026] Figure 2 A schematic diagram showing a dense point cloud acquisition process according to an embodiment of the present invention is shown;

[0027] Figure 3 A schematic diagram showing a binocular point cloud generation process according to an embodiment of the present invention is shown;

[0028] Figure 4 A schematic diagram showing a monocular point cloud generation process according to an embodiment of the present invention is shown;

[0029] Figure 5 A schematic diagram showing the installation positions of various parts in a road data 3D reconstruction system according to an embodiment of the present invention is shown;

[0030] Figure 6 FIG. 1 shows a connection diagram of a road data 3D reconstruction system according to an embodiment of the present invention.

[0031] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0032] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way.

[0033] In one aspect, an embodiment of the present invention provides a method for 3D reconstruction of road data. Figure 1, which shows a flow chart of a method for 3D reconstruction of road data according to an embodiment of the present invention. The method comprises: a raw data acquisition step, a dense point cloud acquisition step, a surface reconstruction step, a mesh optimization step, and a texture mapping step.

[0034] In the raw data acquisition step, the forward binocular raw data, the left monocular raw data, the right monocular raw data, the rearward monocular raw data, the vehicle fusion positioning information, and the sensor parameters are acquired. For example, this data can be acquired from an onboard computing platform mounted on the vehicle.

[0035] The dense point cloud acquisition step may include a binocular point cloud generation sub-step, a monocular point cloud generation sub-step, and a depth map fusion sub-step. Figure 2 A schematic diagram illustrates the dense point cloud acquisition process according to an embodiment of the present invention. In this step, N frames of forward binocular raw data, left monocular raw data, right monocular raw data, and rear monocular raw data, along with vehicle fusion positioning information and sensor parameters, are input. After processing by the binocular point cloud generation submodule, the monocular point cloud generation submodule, and depth map fusion, dense 3D point cloud data of the static environment surrounding the vehicle is obtained.

[0036] In the binocular point cloud generation sub-step, based on the forward-looking binocular raw data, a dense 3D point cloud of the binocular data is generated through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery, and point cloud normal vector estimation.

[0037] In the monocular point cloud generation sub-step, monocular instance segmentation and predefined target filtering are performed on multiple frames of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain multi-frame data results. Multi-view stereo vision processing is performed on the multi-frame data results, the vehicle fusion positioning information, and the sensor parameters to generate a dense 3D point cloud restored from the multi-frame monocular data.

[0038] In the depth map fusion sub-step, based on the vehicle fusion positioning information and sensor parameters, the dense 3D point cloud of binocular data and the dense 3D point cloud restored from multi-frame monocular data are fused in a coordinate system to obtain the fused dense 3D point cloud data.

[0039] In the surface reconstruction step, the triangular facets of the generated surface are calculated by constructing the signed distance field based on the fused dense 3D point cloud data.

[0040] In the mesh optimization step, the positions of the mesh vertices of the triangles are optimized based on the photometric consistency assumption to obtain optimized triangles.

[0041] In the texture mapping step, based on the optimized triangular facets, through perspective selection, texture coordinate calculation, global color optimization and local color optimization processing, the two-dimensional pixels in the original image are projected onto the optimized triangular facets through mapping to obtain a reconstructed three-dimensional model of the road data.

[0042] Therefore, the three-dimensional reconstruction method of road data proposed in the embodiment of the present invention restores the three-dimensional mesh surface information and color information of the static environment in the road collection data.

[0043] refer to Figure 3 , which shows a schematic diagram of a binocular point cloud generation process according to an embodiment of the present invention. As an embodiment of the present invention, the binocular point cloud generation sub-step may include:

[0044] Perform distortion correction and stereo correction on the single-frame forward-looking binocular raw data, perform binocular stereo matching on the left and right target pixels, and obtain a disparity map;

[0045] Performing monocular instance segmentation after performing distortion correction on the monocular data of the first camera in the forward-looking binocular raw data to segment a predefined target in the monocular data of the first camera, thereby obtaining a first semantic map, wherein the first semantic map includes category information of each pixel in the monocular data of the first camera;

[0046] In the disparity map, the disparity value of the predefined target is set to zero according to the category information in the semantic map, and the disparity map with the predefined target filtered out is output;

[0047] According to the disparity map after filtering out the predefined target, the 3D point cloud in the current binocular camera coordinate system is restored through the geometric relationship between the disparity value and the depth value;

[0048] Find the nearest neighbor points of each point in the 3D point cloud in the current binocular camera coordinate system, perform principal component analysis on each point in the 3D point cloud and its nearest neighbor points, estimate the normal vector of each point in the 3D point cloud, and obtain a dense 3D point cloud and the normal vector of each point in the binocular camera coordinate system after filtering out the predefined targets.

[0049] exist Figure 3 In the specific example of the binocular point cloud generation sub-step shown, a single frame of binocular raw data is input, and after binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery, and point cloud normal vector estimation, a dense 3D point cloud of the single-frame binocular data is obtained. The main steps are as follows:

[0050] a) passing the single-frame binocular raw data into the binocular stereo matching module, and at the same time passing the data of a pre-selected camera M1 in the binocular data into the monocular instance segmentation module;

[0051] b) In the binocular stereo matching step, the data is first subjected to distortion correction and stereo calibration, and then the left and right target pixels are matched using a deep learning-based stereo matching algorithm to obtain a disparity map;

[0052] c) In the monocular instance segmentation step, after distortion correction is performed on the monocular data, a convolutional neural network instance segmentation algorithm is used to segment the predefined objects in the image (including vehicles, pedestrians, bicycles, etc.), and a semantic map is output. The semantic map contains the category information of each pixel and is used to filter out the predefined objects based on the category information in the next step.

[0053] d) In the predefined target filtering step, the disparity map and the semantic map are input, the disparity values ​​of the predefined targets in the disparity map are reset to zero according to the category information in the semantic map, and the disparity map with the predefined targets filtered out is output; the disparity points that are reset to zero will not be restored to 3D points in the next step;

[0054] e) In the single-frame binocular point cloud recovery step, the disparity map after filtering out the predefined target is input, and the 3D point cloud in the binocular camera coordinate system at the current moment is recovered through the geometric relationship between the disparity value and the depth value;

[0055] f) In point cloud normal estimation, use the nearest neighbor algorithm to find the N nearest neighbor points of each point (for example, an octree can be used for space partitioning and a KD-Tree algorithm can be used for nearest neighbor search), and then perform PCA (principal component analysis) to estimate the normal vector of the point;

[0056] g) The binocular point cloud generation sub-step finally outputs a dense 3D point cloud and the normal vector of each point in the binocular camera coordinate system after filtering out the predefined target. The output of the i-th point in the 3D point cloud is (x i 、y i 、z i , α i , β i , γ i ), where (x i 、y i 、z i ) is the three-dimensional space coordinate of the point, (α i , β i , γ i ) is the normal vector of the point.

[0057] As an embodiment of the present invention, performing monocular instance segmentation and predefined target filtering on multiple frames of left-view monocular raw data, right-view monocular raw data, and rear-view monocular raw data to obtain multi-frame data results may include:

[0058] Using the sensor distortion coefficient in the sensor parameters, the distortion correction is performed on the single-frame left-view monocular raw data, right-view monocular raw data, and rear-view monocular raw data to obtain the distortion-corrected monocular data;

[0059] Performing monocular instance segmentation on the distortion-corrected monocular data to segment predefined targets in the distortion-corrected monocular data and obtain a second semantic map;

[0060] Set the pixel values ​​of predefined targets in the distortion-corrected monocular data to zero;

[0061] Distortion correction, monocular instance segmentation, and predefined target zeroing operations are performed on each frame of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain multi-frame data results.

[0062] As an embodiment of the present invention, performing multi-view stereo vision processing on multi-frame data results, vehicle fusion positioning information and sensor parameters may include:

[0063] According to the frame list and vehicle fusion positioning information list in the multi-frame data results, the vehicle fusion positioning pose is aligned to the timestamp of the frame list, and the sensor parameters are used to obtain the camera pose at all frame times with the 0th frame as the origin through coordinate transformation;

[0064] Perform feature point extraction and descriptor calculation on the frame list in the multi-frame data result to obtain a feature point list;

[0065] Based on the feature point lists of all frames, use the descriptor to perform incremental feature point matching for each view and output the matched feature point list;

[0066] According to the matched feature point list, select the view with large parallax and use the camera pose to triangulate all feature points to obtain the pre-processed 3D point cloud;

[0067] Perform global bundle adjustment on the pre-processed 3D point cloud and output the globally optimized 3D point cloud;

[0068] The optimized feature point is used as the seed point, and the pixel block of predetermined size around it is used as the seed image block. The seed image block is expanded by region growing to reconstruct the pixel blocks in the neighborhood. Local nonlinear depth optimization is performed on each seed point and its neighborhood pixels to obtain the 3D points corresponding to the neighborhood pixels. This step is repeated until all the neighborhood pixels are reconstructed to generate the reconstructed 3D point cloud.

[0069] The reconstructed 3D point cloud is globally optimized for depth and normal vectors to obtain a dense 3D point cloud restored from multi-frame monocular data.

[0070] refer to Figure 4, which shows a schematic diagram of the monocular point cloud generation process according to an embodiment of the present invention. In the monocular point cloud generation sub-step, a segment of N frames of monocular raw data is input. After each frame undergoes monocular instance segmentation and predefined target filtering, the multi-frame results are input into a multi-view stereo vision system along with vehicle fusion positioning information, sensor distortion coefficients, and external parameters to obtain a dense 3D point cloud restored from the multi-frame monocular data. The main steps are as follows:

[0071] a) Input a single frame of monocular raw data and sensor distortion coefficients (k1, k2, k3, p1, p2), perform distortion correction, and output the distortion-corrected camera image;

[0072] b) Input the distortion-corrected image for monocular instance segmentation. The convolutional neural network instance segmentation algorithm segments the predefined objects in the image (including vehicles, pedestrians, bicycles, etc.) and outputs a semantic map.

[0073] c) setting a predefined target pixel value to zero in the image outputted in step a);

[0074] d) Repeat the process from a) to c) for each frame output until all frames have completed the predefined target filtering;

[0075] e) In the multi-view stereo vision step, input a list of all frames that have completed predefined target filtering (f t0 、f t1 …f tn ), where f tn Indicates the frame at time tn; vehicle fusion positioning information list (p τ0 、p τ1 …p τk ), where p τk Represents the vehicle fusion positioning information (x τk 、y τk 、z τk ,θ τk ), x τk 、y τk 、z τk is the position coordinate of the vehicle, θ τk is the vehicle's yaw angle; the sensor external parameters (rotation matrix and translation vector R, t), the specific processing steps are as follows:

[0076] i. Input the list of frames to be filtered out by predefined targets (f t0 、f t1 …f tn ) and vehicle fusion positioning information list (p τ0 、p τ1 …p τk), align the vehicle fusion positioning posture to the timestamp of the frame list according to the timestamp linear difference, and use the sensor external parameters (rotation matrix and translation vector R cb , t cb ), the camera pose (rotation matrix and translation vector R) at all frames with the 0th frame as the origin is obtained through coordinate transformation cb0 , t cb0 Here, because the vehicle's fusion positioning pose is derived from the results of multi-source sensor fusion positioning, it simultaneously considers information such as GNSS satellite positioning, IMU inertial navigation sensors, wheel speed odometers, and visual odometry. Its positioning pose accuracy is much higher than the pose accuracy estimated by 3D reconstruction in known methods, ensuring the accuracy of subsequent 3D reconstruction.

[0077] ii. Input a list of all frames that complete the predefined target filtering (f t0 、f t1 …f tn ), extract feature points and calculate descriptors, and output a list of feature points (fpl0, fpl1…fpl n ), where fpl n is the feature point list of the nth frame, with fpl n ={[fp0, fp1…fp m ]、[ds0、ds1…ds m ]}, ;fp m is the mth feature point in the frame, ds m is the descriptor of the feature point.

[0078] iii. Input the feature point list of all frames (fpl0, fpl1…fpl n ), use the descriptor to perform incremental feature point matching for each view, and output the matched feature point list (fpla0, fpla1…fpla n ), where the feature point list after matching the nth frame is fpla n ={[fp0, fp1…fp m ]、[ds0、ds1…ds m |, [ma0, ma1…ma m}, The result of the mth feature point matching is an N×1 vector. The i-th element of the vector is j, indicating that the feature point matches the j-th feature point in the i-th frame.

[0079] iv. Input the matched feature point list (fpla0, fpla1…fpla n ), select a view with large parallax and triangulate all feature points using the camera pose, and output the preprocessed 3D point cloud;

[0080] y. Input the preprocessed 3D point cloud, perform global bundle adjustment on it, and output the globally optimized 3D point cloud;

[0081] Vi. Use the optimized feature point as the seed point and an N*N pixel block around it as the seed image block (patch);

[0082] Vii. Perform region growing and expansion on the seed image block to reconstruct the pixel blocks in the neighborhood. This step performs local nonlinear depth optimization on each seed point and its neighborhood pixels, outputting the 3D points corresponding to the neighborhood pixels. This step is repeated until all neighborhood pixels are reconstructed.

[0083] viii. Input all the reconstructed 3D point clouds from the previous step and perform global depth and normal vector optimization.

[0084] As an embodiment of the present invention, the depth map fusion sub-step may further include: filtering out unreasonable points in the dense 3D point cloud of binocular data and the dense 3D point cloud restored from multi-frame monocular data through consistency constraints and visibility constraints.

[0085] As a specific example of the depth map fusion sub-step, the vehicle positioning information list (p τ0 、p τ1 …p τk ) and sensor extrinsic parameter information (rotation matrix and translation vector R, t), the dense 3D point cloud obtained by all cameras is integrated into one coordinate system through coordinate transformation, and unreasonable points are filtered out through consistency constraints and visibility constraints. Among them, the consistency constraint means that in three-dimensional space, the Euclidean distance between points must be greater than a certain threshold. If there are two points less than the threshold, they should be the same point, and the close points should be merged. The visibility constraint means that there can only be one point in the ray passing through the optical center of the camera. If there are more than two points on the same optical center ray, due to occlusion, only the point closest to the optical center can be seen, and the distant points should be filtered out.

[0086] As a specific example, in the surface reconstruction step, the fused 3D point cloud and normal vector are input to calculate the triangular facets of the generated surface (vertex coordinates Ver_Loc, triangle vertex index matrix Id_triangle). In the implementation process, the PSR (Poisson surface reconstruction) method is preferably used for surface reconstruction.

[0087] As an embodiment of the present invention, the mesh optimization step may include: according to the triangular facets generated in the surface reconstruction step, based on the photometric consistency assumption, taking the cumulative photometric difference after reprojection as the objective function, the vertex coordinates as the optimization variables, and the triangular facet vertices as the initial values, through global nonlinear optimization, optimizing the mesh vertex positions of the triangular facets to obtain optimized triangular facets.

[0088] As a specific example, in the mesh optimization step, the input triangles obtained from surface reconstruction are used. Based on the photometric consistency assumption, the cumulative photometric difference after reprojection is used as the objective function, the vertex coordinates Ver_Loc are used as the optimization variable, and the input triangle vertices are used as the initial values. Through global nonlinear optimization, the mesh vertex positions of the triangles are optimized, and the optimized triangles (vertex coordinates Ver_Loc, triangle vertex index matrix Id_triangle) are output. The photometric consistency assumption states that the photometric value (RGB value) of the same point should be consistent between two different frames.

[0089] As a specific example, in the texture mapping step, the optimized triangle face (vertex coordinates Ver_Loc, triangle vertex index matrix Id_triangle) is input, and the perspective selection (selecting the perspective closest to the normal vector of the triangle face), texture coordinate calculation, global color optimization (optimizing the color adjustment rate g of each face) is performed. i , so that several faces with a common vertex have the same luminosity value at the vertex, establish a matrix equation and solve it using the least squares method) and local color optimization (preferably Poisson image editing), project the two-dimensional pixels in the original image into the 3D mesh facets by mapping, and output the reconstructed three-dimensional model.

[0090] Note that for clarity purposes, Figure 2-Figure 4 In the present invention, steps and sub-steps are respectively shown as modules, sub-modules, etc., which are interchangeable with the expressions of steps and sub-steps.

[0091] On the other hand, an embodiment of the present invention proposes a three-dimensional reconstruction device for road data, which may include a raw data acquisition module, a dense point cloud acquisition module, a surface reconstruction module, a mesh optimization module, and a texture mapping module.

[0092] The raw data acquisition module can be configured to: acquire forward-looking binocular raw data, left-looking monocular raw data, right-looking monocular raw data, rear-looking monocular raw data, vehicle fusion positioning information and sensor parameters.

[0093] The dense point cloud acquisition module may include a binocular point cloud generation submodule, a monocular point cloud generation submodule and a depth map fusion submodule.

[0094] The binocular point cloud generation submodule can be configured to generate a dense 3D point cloud of binocular data based on the forward-looking binocular raw data through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery, and point cloud normal vector estimation.

[0095] The monocular point cloud generation submodule can be configured to: perform monocular instance segmentation and predefined target filtering on multiple frames of left-view monocular raw data, right-view monocular raw data, and rear-view monocular raw data to obtain multi-frame data results, perform multi-view stereo vision processing on the multi-frame data results, vehicle fusion positioning information and sensor parameters, and generate a dense 3D point cloud restored from the multi-frame monocular data.

[0096] The depth map fusion submodule can be configured to: fuse the dense 3D point cloud of binocular data and the dense 3D point cloud restored from multi-frame monocular data into a coordinate system according to the vehicle fusion positioning information and sensor parameters, to obtain fused dense 3D point cloud data.

[0097] The surface reconstruction module can be configured to calculate and generate triangular facets of the surface based on the fused dense 3D point cloud data by constructing a signed distance field.

[0098] The mesh optimization module can be configured to: optimize the positions of mesh vertices of the triangular facets based on the photometric consistency assumption according to the triangular facets to obtain optimized triangular facets.

[0099] The texture mapping module can be configured to: project the two-dimensional pixels in the original image onto the optimized triangular facets through viewing angle selection, texture coordinate calculation, global color optimization and local color optimization processing according to the optimized triangular facets, and obtain a reconstructed three-dimensional model of the road data.

[0100] It should be noted that the functions implemented by each module in the road data three-dimensional reconstruction device proposed in the embodiment of the present invention correspond one-to-one to the various steps of the road data three-dimensional reconstruction method described above. For its specific implementation methods, examples and beneficial effects, please refer to the above description of the method.

[0101] In another aspect, an embodiment of the present invention provides a computing server, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the three-dimensional reconstruction method of road data described in any of the above embodiments is implemented.

[0102] On the other hand, an embodiment of the present invention proposes a three-dimensional reconstruction system for road data, including: a wire-controlled vehicle, a computing platform, a front-view binocular camera, a left-view monocular camera, a right-view monocular camera, a rear-view monocular camera, a gateway domain controller, a vehicle network communication terminal, a communication antenna, and a computing server described according to the above embodiment.

[0103] refer to Figure 5-Figure 6 ,in Figure 5 It shows a schematic diagram of the installation positions of various parts in the road data 3D reconstruction system according to an embodiment of the present invention. Figure 6A schematic diagram shows the connections for a 3D road data reconstruction system according to an embodiment of the present invention. The computing platform, forward-facing binocular camera, left-facing monocular camera, right-facing monocular camera, rear-facing monocular camera, gateway domain controller, IoV communication terminal, and communication antenna are installed on a wire-controlled vehicle.

[0104] The computing platform can be used to: after detecting that a pre-defined trigger condition is met, package and send data information within a period of time including the trigger condition moment to the gateway domain controller, where the data information includes at least: the raw data of the front-view binocular camera, the left-view monocular camera, the right-view monocular camera, the rear-view monocular camera, the vehicle fusion positioning information and sensor parameters.

[0105] The gateway domain controller can be used to forward the data information received from the computing platform to the Internet of Vehicles communication terminal.

[0106] The Internet of Vehicles communication terminal can be used to send data information to the computing server through the communication antenna.

[0107] In another aspect, an embodiment of the present invention provides a storage medium storing computer-readable instructions. When the instructions are executed by a processor, the method for three-dimensional reconstruction of road data described in any of the above embodiments is executed.

[0108] The 3D road data reconstruction technology proposed in this embodiment uses a "shadow mode" data collection method. During the driving of a mass-produced vehicle, based on certain trigger conditions, vehicle data is recorded for a period of time before and after the trigger condition. This data is then transmitted back to the backend for scene reconstruction and used for iterative software development. This solves the problem of insufficient data acquisition in existing technologies.

[0109] The embodiments of the present invention disclose a three-dimensional reconstruction method and system for autonomous driving road data collection. The method is mainly used to perform three-dimensional reconstruction of static environmental scenes based on data collected from mass-produced vehicles to reproduce the environment around the real vehicle.

[0110] The 3D reconstruction system for collecting data from the entire road uses cameras in four directions on the vehicle to collect data on the entire road environment. At the same time, this data, along with vehicle positioning data and sensor parameters, is transmitted back to the computing server for 3D reconstruction of the entire system.

[0111] The entire 3D reconstruction algorithm process of the road collection data recovers and fuses the camera data in four directions into a dense point cloud of the road environment in a coordinate system, and then performs surface reconstruction, mesh optimization and texture mapping on the point cloud.

[0112] In the binocular point cloud generation submodule and the monocular point cloud generation submodule, the processing flow of filtering out predefined targets through the results of the instance segmentation algorithm.

[0113] Compared with the existing topological three-dimensional reconstruction of objects through laser radar scanning point clouds, the present invention simultaneously restores the three-dimensional shape and color information of surrounding objects through surface reconstruction and texture mapping, which is more realistic in scene reconstruction.

[0114] Compared with the existing three-dimensional reconstruction through the fusion of laser radar and camera, the present invention only uses the vehicle-mounted camera, which is lower in cost.

[0115] Compared to existing purely visual 3D reconstruction methods, this method uses fused vehicle-based positioning information and sensor parameters to calculate the camera pose. This method achieves higher accuracy than purely visual 3D reconstruction, which relies on inter-frame motion estimation. Furthermore, the scale of the fused positioning information is fixed, ensuring that the scale of the 3D reconstruction matches the real scene.

[0116] For illustrative purposes, the foregoing description of the embodiments of the present invention has been given, which is not exhaustive nor intended to limit the present invention to disclosed exact forms. It will be appreciated by those skilled in the art that various changes may be made without departing from the scope of the present invention, and that elements therein may be replaced with equivalents. In addition, without departing from the basic scope of the present invention, many modifications may be made so that specific situations or materials are adapted to the teachings of the present invention. Therefore, the present invention is not intended to be limited to the specific embodiments disclosed as the best mode for realizing the present invention, and the present invention will include all embodiments within the scope of the appended claims.

Claims

1. A method for three-dimensional reconstruction of road data, characterized in that: The method comprises: Raw data acquisition steps: Acquire forward binocular raw data, left monocular raw data, right monocular raw data, rear monocular raw data, vehicle fusion positioning information and sensor parameters; The dense point cloud acquisition step includes a binocular point cloud generation sub-step, a monocular point cloud generation sub-step, and a depth map fusion sub-step, wherein: The binocular point cloud generation sub-step: based on the forward-looking binocular original data, through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery and point cloud normal vector estimation, Generate dense 3D point cloud of binocular data; The monocular point cloud generation sub-step comprises: performing monocular instance segmentation and predefined target filtering on multiple frames of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain multi-frame data results; performing multi-view stereo vision processing on the multi-frame data results, the vehicle fusion positioning information, and the sensor parameters to generate a dense 3D point cloud restored from the multi-frame monocular data; The depth map fusion sub-step: according to the vehicle fusion positioning information and the sensor parameters, fusing the dense 3D point cloud of the binocular data and the dense 3D point cloud restored by the multi-frame monocular data in a coordinate system to obtain fused dense 3D point cloud data; Surface reconstruction step: Based on the fused dense 3D point cloud data, a signed distance field is constructed to calculate and generate triangular facets of the surface; Mesh optimization step: optimizing the positions of mesh vertices of the triangles based on the photometric consistency assumption to obtain optimized triangles; Texture mapping step: Based on the optimized triangular facets, through perspective selection, texture coordinate calculation, global color optimization and local color optimization processing, the two-dimensional pixels in the original image are projected onto the optimized triangular facets by mapping to obtain a reconstructed three-dimensional model of the road data.

2. The method for 3D reconstruction of road data according to claim 1, characterized in that: The binocular point cloud generation sub-step includes: Performing distortion correction and stereo correction on the single-frame forward-looking binocular raw data, performing binocular stereo matching on the left and right target pixels, and obtaining a disparity map; Performing monocular instance segmentation after performing distortion correction on the monocular data of the first camera in the forward-looking binocular raw data to segment a predefined target in the monocular data of the first camera, thereby obtaining a first semantic map, wherein the first semantic map includes category information of each pixel in the monocular data of the first camera; In the disparity map, according to the category information in the semantic map, the disparity value of the predefined target is set to zero, and the disparity map with the predefined target filtered out is output; According to the disparity map with the predefined target filtered out, the 3D point cloud in the binocular camera coordinate system at the current moment is restored through the geometric relationship between the disparity value and the depth value; Find the neighboring points of each point in the 3D point cloud in the binocular camera coordinate system at the current moment, perform principal component analysis on each point in the 3D point cloud and its neighboring points, estimate the normal vector of each point in the 3D point cloud, and obtain a dense 3D point cloud and the normal vector of each point in the binocular camera coordinate system after filtering out the predefined target.

3. The method for 3D reconstruction of road data according to claim 1, characterized in that: Performing monocular instance segmentation and predefined target filtering on multiple frames of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain a multi-frame data result includes: Using the sensor distortion coefficient in the sensor parameters, performing distortion correction on the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data of a single frame to obtain distortion-corrected monocular data; performing monocular instance segmentation on the distortion-corrected monocular data to segment predefined targets in the distortion-corrected monocular data to obtain a second semantic map; Setting the pixel values ​​of the predefined targets in the distortion-corrected monocular data to zero; Distortion correction, monocular instance segmentation, and predefined target zeroing operations are performed on each frame of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain the multi-frame data result.

4. The method for 3D reconstruction of road data according to claim 1, characterized in that: Performing multi-view stereo vision processing on the multi-frame data results, the vehicle fusion positioning information, and the sensor parameters includes: According to the frame list and vehicle fusion positioning information list in the multi-frame data result, align the vehicle fusion positioning posture to the timestamp of the frame list, and use the sensor parameters to obtain the camera posture at all frame times with the 0th frame as the origin through coordinate transformation; Performing feature point extraction and descriptor calculation on the frame list in the multi-frame data result to obtain a feature point list; Based on the feature point lists of all frames, use the descriptor to perform incremental feature point matching for each view and output the matched feature point list; According to the matched feature point list, a view angle with large parallax is selected to triangulate all feature points using the camera pose to obtain a pre-processed 3D point cloud; Performing global bundle adjustment on the preprocessed 3D point cloud and outputting a globally optimized 3D point cloud; The optimized feature point is used as the seed point, and the pixel block of predetermined size around it is used as the seed image block. The seed image block is expanded by region growing to reconstruct the pixel blocks in the neighborhood. Local nonlinear depth optimization is performed on each seed point and its neighborhood pixels to obtain the 3D points corresponding to the neighborhood pixels. This step is repeated until all the neighborhood pixels are reconstructed to generate the reconstructed 3D point cloud. Global depth and normal vector optimization is performed on the reconstructed 3D point cloud to obtain a dense 3D point cloud restored from the multi-frame monocular data.

5. The method for three-dimensional reconstruction of road data according to any one of claims 1 to 4, characterized in that: The depth map fusion sub-step further includes: filtering out unreasonable points in the dense 3D point cloud of the binocular data and the dense 3D point cloud restored from the multi-frame monocular data through consistency constraints and visibility constraints.

6. The method for three-dimensional reconstruction of road data according to any one of claims 1 to 4, characterized in that: The grid optimization step comprises: According to the triangular facets generated in the surface reconstruction step, based on the photometric consistency assumption, the cumulative photometric difference after reprojection is used as the objective function, the vertex coordinates are used as the optimization variables, and the triangular facet vertices are used as the initial values. Through global nonlinear optimization, the mesh vertex positions of the triangular facets are optimized to obtain the optimized triangular facets.

7. A three-dimensional reconstruction device for road data, characterized in that: The device comprises: A raw data acquisition module is configured to acquire forward binocular raw data, left monocular raw data, right monocular raw data, rearward monocular raw data, vehicle fusion positioning information, and sensor parameters; The dense point cloud acquisition module includes a binocular point cloud generation submodule, a monocular point cloud generation submodule and a depth map fusion submodule, among which, The binocular point cloud generation submodule is configured to: generate a dense 3D point cloud of binocular data based on the forward-looking binocular raw data through binocular stereo matching, monocular instance segmentation, predefined target filtering, single-frame binocular point cloud recovery, and point cloud normal vector estimation; The monocular point cloud generation submodule is configured to: perform monocular instance segmentation and predefined target filtering on multiple frames of the left-view monocular raw data, the right-view monocular raw data, and the rear-view monocular raw data to obtain multi-frame data results; perform multi-view stereo vision processing on the multi-frame data results, the vehicle fusion positioning information, and the sensor parameters to generate a dense 3D point cloud restored from the multi-frame monocular data; The depth map fusion submodule is configured to: fuse the dense 3D point cloud of the binocular data and the dense 3D point cloud restored from the multi-frame monocular data into a coordinate system according to the vehicle fusion positioning information and the sensor parameters, to obtain fused dense 3D point cloud data; A surface reconstruction module is configured to: calculate and generate triangular facets of the surface by constructing a signed distance field based on the fused dense 3D point cloud data; A mesh optimization module configured to: optimize the positions of mesh vertices of the triangular facets based on the photometric consistency assumption according to the triangular facets to obtain optimized triangular facets; The texture mapping module is configured to: project the two-dimensional pixels in the original image onto the optimized triangular facets through viewing angle selection, texture coordinate calculation, global color optimization and local color optimization processing according to the optimized triangular facets, so as to obtain a reconstructed three-dimensional model of the road data.

8. A computing server, characterized in that: The computing server includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method for three-dimensional reconstruction of road data according to any one of claims 1 to 6 is implemented.

9. A road data 3D reconstruction system, characterized in that: The system includes: a wire-controlled vehicle, a computing platform, a front-view binocular camera, a left-view monocular camera, a right-view monocular camera, a rear-view monocular camera, a gateway domain controller, a vehicle network communication terminal, a communication antenna, and a computing server according to claim 8, wherein: The computing platform, the front-view binocular camera, the left-view monocular camera, the right-view monocular camera, the rear-view monocular camera, the gateway domain controller, the Internet of Vehicles communication terminal and the communication antenna are installed on the wire-controlled vehicle. The computing platform is configured to: after detecting that a predefined trigger condition is met, package and send data information within a period of time including the trigger condition moment to the gateway domain controller, wherein the data information at least includes: raw data of the front-view binocular camera, the left-view monocular camera, the right-view monocular camera, and the rear-view monocular camera, vehicle fusion positioning information, and sensor parameters; The gateway domain controller is used to forward the data information received from the computing platform to the Internet of Vehicles communication terminal; The Internet of Vehicles communication terminal is used to send the data information to the computing server via the communication antenna.

10. A storage medium, characterized in that: Computer-readable instructions are stored, and when the instructions are executed by a processor, the method according to any one of claims 1 to 6 is executed.

Images