Deep learning 3D reconstruction method based on FMCW lidar and vision fusion

Abstract

The present invention belongs to the field of indoor and outdoor robot application scene reconstruction, specifically, a deep learning 3D reconstruction method based on FMCW laser radar and visual fusion, comprising the following steps: 1) setting multiple laser radars and cameras to ensure time synchronization; 2) calibrating multiple laser radars and cameras; 3) real-time collection of FMCW laser radar data and camera image data in each scene and angle, corresponding to timestamps; 4) processing the collected data as a training data set; 5) the data are input into the deep learning model network for training to obtain a 3D Gaussian model of dynamic and static objects in the scene; 6) after coordinate conversion and velocity calculation of the collected FMCW laser radar data, combined with the collected camera image data, input into the 3D Gaussian model, and the present invention obtains scene reconstruction information. The present invention uses the native velocity information of the FMCW laser radar, combined with visual information, to better characterize the characteristics of dynamic objects and achieve native dynamic and static distinction.

Description

Technical Field

[0001] The present invention belongs to the field of indoor and outdoor robot application scene reconstruction, and specifically is a deep learning 3D reconstruction method based on FMCW laser radar and vision fusion. Background Art

[0002] In current indoor and outdoor robot application scenarios such as warehousing, logistics, and industrial manufacturing, 3D reconstruction technology is widely used to reconstruct, extract models, and simulate the scenes in which the robots are located.

[0003] At present, there are solutions in the industry that use laser radar and camera visual information fusion, but the existing technical solutions cannot obtain native speed information from the sensor. As a result, dynamic objects, such as obstacles such as people and cars in the scene, or other moving robots and manipulators, cannot be accurately distinguished from static scenes. In this way, dynamic objects often affect the reconstruction accuracy of static scenes; it is also impossible to accurately reconstruct dynamic objects, and there are often problems such as overlap and ghosting.

[0004] In order to improve accuracy and solve the problem of the influence of dynamic objects in reconstruction, existing algorithms often introduce more sensors such as IMU and RTK to perform complex posture calculations, making 3D reconstruction a complex system that is inconvenient to develop and debug, as well as to decouple various modules. Summary of the invention

[0005] The purpose of the present invention is to provide a deep learning 3D reconstruction method based on FMCW laser radar and visual fusion. The native speed information of FMCW laser radar combined with visual information can better characterize the characteristics of dynamic objects, realize native dynamic and static distinction, make 3D reconstruction more robust, and also make it possible to accurately reconstruct dynamic objects. At the same time, the use of deep learning for model training avoids the complexity of complex system development, debugging and maintenance.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a deep learning 3D reconstruction method based on FMCW laser radar and visual fusion, comprising the following steps:

[0007] 1) Set up multiple FMCW laser radars and cameras on the AGV and ensure time synchronization between the FMCW laser radars and cameras;

[0008] 2) Calibrate the installation positions of multiple FMCW lidars and cameras;

[0009] 3) The acquisition equipment installed on the AGV drives in the scene to be reconstructed, collects FMCW lidar data and camera image data from all angles of each scene in real time, stores them synchronously, and corresponds them with timestamps for synchronous playback;

[0010] 4) Process the collected FMCW lidar data and camera image data to form key frames for use as training data sets;

[0011] 5) Input the FMCW lidar data and camera image data into the deep learning model network for training, adjust the model parameters, optimize the training results, and obtain the 3D Gaussian model of the dynamic and static objects in the scene;

[0012] 6) The data set information of the collected FMCW lidar data after coordinate transformation and velocity solution, as well as the collected camera image data, are input into step 5) to obtain the 3D Gaussian model of the dynamic and static objects in the scene, and rendering is performed at a specific time and a specific observation angle to obtain the scene reconstruction information.

[0013] The step 1) is specifically:

[0014] The corresponding FMCW laser radars and cameras are set with the same field of view and in the same coordinate system to achieve a common field of view. Multiple FMCW laser radars and cameras are arranged around the AGV body to cover a 360° field of view. At the same time, the time synchronization between sensors is achieved through the main controller of the AGV or by controlling the external trigger devices of multiple FMCW laser radars and cameras.

[0015] The installation positions of the multiple FMCW laser radars and cameras are calibrated as follows:

[0016] 1-1) Determine the calibration target, that is, determine the specific object or reference point that the lidar and camera observe together;

[0017] 1-2) Collect data from the installed LiDAR and camera through AGV, so that both the camera and LiDAR can observe the calibration target and record the corresponding data;

[0018] 1-3) Data preprocessing: preprocess the collected data, including denoising, point cloud registration, and image dedistortion operations to ensure the accuracy of subsequent calibration;

[0019] 1-4) Calibration parameters: After data preprocessing is completed, calibrate the internal and external parameters between the camera and the lidar, that is, calibrate the internal parameters of the camera and the external parameters of the transformation matrix from the camera to the lidar;

[0020] 1-5) Matching feature points: Extract common feature points from camera images and lidar point clouds through image processing and point cloud processing techniques;

[0021] 1-6) Solve the transformation relationship: Use the extracted feature points to obtain the relative posture relationship between each lidar and the camera, and the posture relationship with the vehicle body through the least squares method or optimization algorithm; thus obtaining the position of the radar point cloud in the 3D world and the corresponding projection relationship with the camera pixels.

[0022] In step 3), the laser radar data and camera image data in each scene and angle are collected in real time, stored synchronously, and corresponded with timestamps, specifically:

[0023] 2-1) Assume that the camera acquisition time is t, and the latest time of the laser radar acquisition before and after time t is t1 and t2 respectively;

[0024] 2-2) Match the corresponding points of the point clouds of the previous and next two frames through the iterative closest point algorithm;

[0025] 2-3) Interpolate the coordinates and intensity values ​​of each point pair. Assuming that the point p1 at time t1 matches the point p2 at time t2, the interpolation of the lidar point cloud data at time t is:

[0026]

[0027] The step 4) is specifically:

[0028] 3-1) Data cleaning: For the laser radar data and camera image data with synchronized timestamps, remove abnormal data or noise data based on the statistical characteristics and thresholds of the data;

[0029] 3-2) Data slicing: Divide the laser radar data and camera image data after synchronization timestamp into continuous time segments Frame;

[0030] 3-3) Extracting regions of interest: Based on target detection or semantic segmentation, the YOLO deep learning model is used to extract regions of interest (ROIs) from lidar data and camera image data.

[0031] 3-4) Extract key frames and annotate data: Extract key frames from the frames of continuous time segments Frame through inter-frame similarity; and annotate the key frames with object category, position, and posture information to facilitate supervised learning;

[0032] 3-5) Dataset construction: Integrate into a training data set according to steps 3-1) to 3-5), wherein the training data set includes: key frame images, lidar data and annotation information;

[0033] 3-6) Data enhancement and segmentation: Perform data enhancement by rotating, scaling, and translating the dataset to increase data diversity; at the same time, divide the dataset into training set, validation set, and test set for model training and evaluation in step 5).

[0034] The step 6) is specifically:

[0035] 4-1) 3D Gaussians initialization:

[0036] Initialize the 3D Gaussian model to represent the spatial information of the LiDAR point cloud and convert the point cloud data into a 3D Gaussian distribution;

[0037] 4-2) 3D Gaussians projection:

[0038] Project the 3D Gaussian model onto the camera image plane, map the point cloud data into the image space, and align it with the camera image data;

[0039] 4-3) Rasterized image generation:

[0040] The mapping result of the 3D Gaussian distribution is converted into a rasterized image, a depth information map corresponding to the camera image is generated, and the depth information map is fused with the camera image to obtain a pseudo observation image so as to be input into the deep learning model together with the camera image data;

[0041] 4-4) Construct a loss function and calculate the difference between the pseudo-observation image output by the model and the real image;

[0042] 4-5) Back propagation and adaptive density control:

[0043] The loss function is used as the optimization target, and the model parameters are updated through the back propagation algorithm to control the density of the 3D Gaussian distribution to adapt to the characteristics of different scenes and targets and continuously optimize the model; in the back propagation, the cross attention mechanism is introduced to extract the different time features of the lidar point cloud and the camera image, and return to step 4-2).

[0044] The step 4-3) is specifically:

[0045] a. Divide the 3D image space into N grid cells; map each Gaussian distribution of the 3D Gaussian distribution to a corresponding position in the rasterized image, and calculate the weight or intensity of each Gaussian distribution in the grid;

[0046] b. For each grid cell, according to its intersection with the Gaussian distribution, weighted processing is performed according to the weight distribution of the Gaussian distribution in the grid to calculate the depth value of the grid cell;

[0047] c. Depth value mapping:

[0048] Map the calculated rasterized depth value to the corresponding position in the camera image coordinate system;

[0049] d. Deep information extraction:

[0050] According to the position of the depth value in the camera image, the depth information is extracted to generate a depth information map corresponding to the camera image;

[0051] e. The depth information map is processed by edge smoothing and depth value filtering and then fused with the camera image to obtain a pseudo observation image containing depth information.

[0052] The step 4-4) is specifically:

[0053] According to the cosine similarity principle, the loss function of the real image is:

[0054]

[0055] Among them, X i , Y i are the pixel colors of the pseudo observed image and the real image, respectively.

[0056] The scene reconstruction information includes: the position, speed, color, and semantic information of each dynamic and static object at the same time.

[0057] A deep learning 3D reconstruction system based on FMCW laser radar and vision fusion includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement a deep learning 3D reconstruction method based on FMCW laser radar and vision fusion when executing the computer program.

[0058] The present invention has the following beneficial effects and advantages:

[0059] 1. By using the method of the present invention for 3D reconstruction, static objects can be reconstructed more robustly, and static objects can also be reconstructed, and a rich and stable robot application scenario model can be obtained. This method avoids the current situation of complex system and module coupling of existing methods, solves the problem through a unified deep learning model, and makes the system logic more concise, clear and reliable.

[0060] 2. The present invention uses the native velocity information of the FMCW lidar, combined with visual information, to better characterize the features of dynamic objects, achieve native dynamic and static distinction, make 3D reconstruction more robust, and also make it possible to accurately reconstruct dynamic objects.

[0061] 3. The present invention uses deep learning for model training, avoiding the complexity of complex system development, debugging and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is an overall flow chart of the method of the present invention;

[0063] Figure 2This is a flow chart of the deep learning model of the present invention;

[0064] Figure 3 An example of an image collected in a preferred embodiment of the present invention;

[0065] Figure 4 An example of a radar point cloud collected in a preferred embodiment of the present invention;

[0066] Figure 5 The following are examples of renderings obtained from other angles by reconstruction in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0067] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0068] like Figure 1-2 As shown, it is an overall flow chart of the method of the present invention. The deep learning 3D reconstruction method based on FMCW laser radar and visual fusion of the present invention comprises the following steps:

[0069] 1) Set up multiple FMCW laser radars and cameras on the AGV and ensure time synchronization between the FMCW laser radars and cameras;

[0070] The corresponding FMCW laser radars and cameras are set with the same field of view and in the same coordinate system to achieve a common field of view. Multiple FMCW laser radars and cameras are arranged around the AGV body to cover a 360° field of view. At the same time, the time synchronization between sensors is achieved through the main controller of the AGV or by controlling the external trigger devices of multiple FMCW laser radars and cameras.

[0071] 2) Calibrate the installation positions of multiple FMCW lidars and cameras;

[0072] 2-1) Determine the calibration target, that is, determine the specific object or reference point that the lidar and camera observe together;

[0073] 2-2) Collect data from the installed laser radar and camera through AGV, so that both the camera and the laser radar can observe the calibration target and record the corresponding data;

[0074] 2-3) Data preprocessing: preprocess the collected data, including denoising, point cloud registration, and image dedistortion operations to ensure the accuracy of subsequent calibration;

[0075] 2-4) Calibration parameters: After data preprocessing is completed, calibrate the internal and external parameters between the camera and the lidar, that is, calibrate the internal parameters of the camera and the external parameters of the transformation matrix from the camera to the lidar;

[0076] 2-5) Matching feature points: Extract common feature points from camera images and lidar point clouds through image processing and point cloud processing techniques;

[0077] 2-6) Solve the transformation relationship: Use the extracted feature points to obtain the relative posture relationship between each lidar and the camera, and the posture relationship with the vehicle body through the least squares method or optimization algorithm; thus obtaining the position of the radar point cloud in the 3D world and the corresponding projection relationship with the camera pixels.

[0078] 3) The acquisition equipment installed on the AGV drives in the scene to be reconstructed, collects FMCW lidar data and camera image data from all angles of each scene in real time, stores them synchronously, and corresponds them with timestamps for synchronous playback;

[0079] 3-1) Assume that the camera acquisition time is t, and the latest time of the laser radar acquisition before and after time t is t1 and t2 respectively;

[0080] 3-2) Match the corresponding points of the point clouds of the previous and next two frames through the iterative closest point algorithm;

[0081] 3-3) Interpolate the coordinates and intensity values ​​of each point pair. Assuming that the point p1 at time t1 matches the point p2 at time t2, the interpolation of the lidar point cloud data at time t is:

[0082]

[0083] 4) Process the collected FMCW lidar data and camera image data to form key frames for use as training data sets;

[0084] 4-1) Data cleaning: For the laser radar data and camera image data with synchronized timestamps, remove abnormal data or noise data based on the statistical characteristics and thresholds of the data;

[0085] 4-2) Data slicing: Divide the laser radar data and camera image data after synchronization timestamp into continuous time segments Frame;

[0086] 4-3) Extracting regions of interest: Based on target detection or semantic segmentation, the YOLO deep learning model is used to extract regions of interest (ROIs) from lidar data and camera image data.

[0087] 4-4) Extract key frames and annotate data: Extract key frames from the frames of continuous time segments Frame through inter-frame similarity; and annotate the key frames with object category, position, and posture information to facilitate supervised learning;

[0088] 4-5) Dataset construction: Integrate into a training data set according to steps 4-1) to 4-5), wherein the training data set includes: key frame images, lidar data and annotation information;

[0089] 4-6) Data enhancement and segmentation: The data set is rotated, scaled, and translated to enhance the data and increase the diversity of the data. At the same time, the data set is divided into a training set, a validation set, and a test set for model training and evaluation in step 5).

[0090] 5) Input the FMCW lidar data and camera image data into the deep learning model network for training, adjust the model parameters, optimize the training results, and obtain the 3D Gaussian model of the dynamic and static objects in the scene;

[0091] 6) The data set information of the collected FMCW lidar data after coordinate transformation and velocity solution, as well as the collected camera image data, are input into step 5) to obtain the 3D Gaussian model of the dynamic and static objects in the scene, and rendering is performed at a specific time and a specific observation angle to obtain the scene reconstruction information.

[0092] Scene reconstruction information, including: the position, speed, color, and semantic information of each dynamic and static object at the same time.

[0093] Among them, step 6) is specifically:

[0094] like Figure 2 As shown, it is a flow chart of the deep learning model of the present invention; the present invention proposes a new deep learning multi-sensor fusion 3D reconstruction model. The information of the FMCW lidar point cloud after coordinate transformation and velocity solution is input into the deep learning model together with the camera image for reasoning. The model steps mainly include 3D Gaussians initialization, 3D Gaussians projection, adaptive density control, cross attention of different time features, rasterized image generation and other processes. Finally, the loss function with the real image is calculated, back propagation is performed, and network training is performed.

[0095] 6-1) 3D Gaussians initialization:

[0096] Initialize the 3D Gaussian model to represent the spatial information of the LiDAR point cloud and convert the point cloud data into a 3D Gaussian distribution;

[0097] 6-2) 3D Gaussians projection:

[0098] Project the 3D Gaussian model onto the camera image plane, map the point cloud data into the image space, and align it with the camera image data;

[0099] 6-3) Rasterized image generation:

[0100] The mapping result of the 3D Gaussian distribution is converted into a rasterized image, a depth information map corresponding to the camera image is generated, and the depth information map is fused with the camera image to obtain a pseudo observation image so as to be input into the deep learning model together with the camera image data;

[0101] Specifically, this step includes the following steps:

[0102] a. Divide the 3D image space into N grid cells; map each Gaussian distribution of the 3D Gaussian distribution to a corresponding position in the rasterized image, and calculate the weight or intensity of each Gaussian distribution in the grid;

[0103] b. For each grid cell, according to its intersection with the Gaussian distribution, weighted processing is performed according to the weight distribution of the Gaussian distribution in the grid to calculate the depth value of the grid cell;

[0104] c. Depth value mapping:

[0105] Map the calculated rasterized depth value to the corresponding position in the camera image coordinate system;

[0106] d. Deep information extraction:

[0107] According to the position of the depth value in the camera image, the depth information is extracted to generate a depth information map corresponding to the camera image;

[0108] The principle of 3DGaussians is as follows:

[0109] The final color of the pixel is:

[0110]

[0111] where c i is the learned color, the final opacity α i 'Learn the opacity α i The product with Gauss is:

[0112]

[0113] e. The depth information map is processed by edge smoothing and depth value filtering and then fused with the camera image to obtain a pseudo observation image containing depth information.

[0114] 6-4) Construct a loss function and calculate the difference between the pseudo-observation image output by the model and the real image;

[0115] According to the cosine similarity principle, the loss function of the real image is:

[0116]

[0117] Among them, X i , Y i are the pixel colors of the pseudo observed image and the real image, respectively.

[0118] 6-5) Back propagation and adaptive density control:

[0119] Take the loss function as the optimization target, update the model parameters through the back propagation algorithm, control the density of the 3D Gaussian distribution to adapt to the characteristics of different scenes and targets, and continuously optimize the model; in the back propagation, introduce the cross attention mechanism to extract the different time features of the lidar point cloud and the camera image, and return to step 6-2).

[0120] Embodiment 1:

[0121] This embodiment proposes a new multi-sensor fusion 3D reconstruction framework. The FMCW laser radar and camera visual information fusion solution is used to simultaneously reconstruct dynamic and static objects. The input is the point cloud of the FMCW laser radar and the image of the camera, and the output is the position, speed, color, semantics and other information of static and dynamic objects at the same time. The point cloud information of the FMCW laser radar includes the position and speed of the point. The input radar point clouds are transformed in coordinates and solved for speed relative to the vehicle, and reasoned through the deep learning model to obtain reconstruction information. The overall system flow chart is as follows: Figure 1 shown.

[0122] 3D reconstruction using FMCW lidar and camera images is performed in the following steps:

[0123] 1. Install and deploy sensors.

[0124] First, install sensors on the equipment used for data collection, such as robots. FMCW laser radar and cameras need to have a common field of view. Sensors can be deployed around the robot as needed to achieve a 360-degree field of view coverage. It is necessary to ensure time synchronization between sensors through a controller or external trigger device. , to ensure sensor time synchronization.

[0125] 2. Sensor calibration

[0126] Calibrate the sensor installation position to obtain the relative posture relationship between each laser radar and the camera, and the posture relationship with the vehicle body. In this way, the position of the radar point cloud in the 3D world and the corresponding projection relationship with the camera pixel are obtained.

[0127] 3. Data Collection

[0128] The acquisition device is driven in the scene to be reconstructed to collect the lidar and camera image data from all angles of each scene. The data is collected in real time and stored synchronously with timestamps for synchronous playback. Figure 3-4 shown.

[0129] 4. Data Processing

[0130] The collected data is processed, cleaned, sliced, and the regions of interest are extracted to form key frames as training data sets.

[0131] 5. Model Training

[0132] Input radar and image data into the deep learning model network for training. Adjust model parameters and optimize training results. Obtain a stable and usable 3D Gaussian model of dynamic and static objects in the scene.

[0133] 6. Model Reasoning

[0134] For any time and observation angle of interest, the trained model is used to infer the 3D Gaussian model of the dynamic and static scenes at that time, and then rendered at a specific observation angle to obtain a reconstructed image of the scene. Figure 5 shown.

[0135] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0136] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A deep learning 3D reconstruction method based on FMCW lidar and vision fusion, characterized in that: The following steps are involved: 1) Set up multiple FMCW laser radars and cameras on the AGV and ensure time synchronization between the FMCW laser radars and cameras; 2) Calibrate the installation positions of multiple FMCW lidars and cameras; 3) The acquisition equipment installed on the AGV drives in the scene to be reconstructed, collects FMCW lidar data and camera image data from all angles of each scene in real time, stores them synchronously, and corresponds them with timestamps for synchronous playback; 4) Process the collected FMCW lidar data and camera image data to form key frames for use as training data sets; 5) Input the FMCW lidar data and camera image data into the deep learning model network for training, adjust the model parameters, optimize the training results, and obtain the 3D Gaussian model of the dynamic and static objects in the scene; 6) The data set information of the collected FMCW lidar data after coordinate transformation and velocity solution, as well as the collected camera image data, are input into step 5) to obtain the 3D Gaussian model of the dynamic and static objects in the scene, and rendering is performed at a specific time and a specific observation angle to obtain the scene reconstruction information.

2. The deep learning 3D reconstruction method based on FMCW laser radar and vision fusion according to claim 1 is characterized in that: The step 1) is specifically: The corresponding FMCW laser radars and cameras are set with the same field of view and in the same coordinate system to achieve a common field of view. Multiple FMCW laser radars and cameras are arranged around the AGV body to cover a 360° field of view. At the same time, the time synchronization between sensors is achieved through the main controller of the AGV or by controlling the external trigger devices of multiple FMCW laser radars and cameras.

3. The deep learning 3D reconstruction method based on FMCW laser radar and vision fusion according to claim 1 is characterized in that: The installation positions of the multiple FMCW laser radars and cameras are calibrated as follows: 1-1) Determine the calibration target, that is, determine the specific object or reference point that the lidar and camera observe together; 1-2) Collect data from the installed LiDAR and camera through AGV, so that both the camera and LiDAR can observe the calibration target and record the corresponding data; 1-3) Data preprocessing: preprocess the collected data, including denoising, point cloud registration, and image dedistortion operations to ensure the accuracy of subsequent calibration; 1-4) Calibration parameters: After data preprocessing is completed, calibrate the internal and external parameters between the camera and the lidar, that is, calibrate the internal parameters of the camera and the external parameters of the transformation matrix from the camera to the lidar; 1-5) Matching feature points: Extract common feature points from camera images and lidar point clouds through image processing and point cloud processing techniques; 1-6) Solve the transformation relationship: Use the extracted feature points to obtain the relative posture relationship between each lidar and the camera, and the posture relationship with the vehicle body through the least squares method or optimization algorithm; thus obtaining the position of the radar point cloud in the 3D world and the corresponding projection relationship with the camera pixels.

4. The deep learning 3D reconstruction method based on FMCW laser radar and vision fusion according to claim 1 is characterized in that: In step 3), the laser radar data and camera image data in each scene and angle are collected in real time, stored synchronously, and corresponded with timestamps, specifically: 2-1) Assume that the camera acquisition time is t, and the latest time of the laser radar acquisition before and after time t is t1 and t2 respectively; 2-2) Match the corresponding points of the point clouds of the previous and next two frames through the iterative closest point algorithm; 2-3) Interpolate the coordinates and intensity values ​​of each point pair. Assuming that the point p1 at time t1 matches the point p2 at time t2, the interpolation of the lidar point cloud data at time t is:

5. The deep learning 3D reconstruction method based on FMCW laser radar and vision fusion according to claim 1 is characterized in that: The step 4) is specifically: 3-1) Data cleaning: For the laser radar data and camera image data with synchronized timestamps, remove abnormal data or noise data based on the statistical characteristics and thresholds of the data; 3-2) Data slicing: Divide the laser radar data and camera image data after synchronization timestamp into continuous time segments Frame; 3-3) Extracting regions of interest: Based on target detection or semantic segmentation, the YOLO deep learning model is used to extract regions of interest (ROIs) from lidar data and camera image data. 3-4) Extract key frames and annotate data: extract key frames from the frames of continuous time segments through inter-frame similarity; and annotate the key frames with object category, position, and posture information to facilitate supervised learning; 3-5) Dataset construction: Integrate into a training data set according to steps 3-1) to 3-5), wherein the training data set includes: key frame images, lidar data and annotation information; 3-6) Data enhancement and segmentation: Perform data enhancement by rotating, scaling, and translating the dataset to increase data diversity; at the same time, divide the dataset into training set, validation set, and test set for model training and evaluation in step 5).

6. The deep learning 3D reconstruction method based on FMCW laser radar and vision fusion according to claim 1 is characterized in that: The step 6) is specifically: 4-1) 3D Gaussians initialization: Initialize the 3D Gaussian model to represent the spatial information of the LiDAR point cloud and convert the point cloud data into a 3D Gaussian distribution; 4-2) 3D Gaussians projection: Project the 3D Gaussian model onto the camera image plane, map the point cloud data into the image space, and align it with the camera image data; 4-3) Rasterized image generation: The mapping result of the 3D Gaussian distribution is converted into a rasterized image, a depth information map corresponding to the camera image is generated, and the depth information map is fused with the camera image to obtain a pseudo observation image so as to be input into the deep learning model together with the camera image data; 4-4) Construct a loss function and calculate the difference between the pseudo-observation image output by the model and the real image; 4-5) Back propagation and adaptive density control: The loss function is used as the optimization target, and the model parameters are updated through the back propagation algorithm to control the density of the 3D Gaussian distribution to adapt to the characteristics of different scenes and targets and continuously optimize the model; in the back propagation, the cross attention mechanism is introduced to extract the different time features of the lidar point cloud and the camera image, and return to step 4-2).

7. The deep learning 3D reconstruction method based on FMCW laser radar and vision fusion according to claim 6 is characterized in that: The step 4-3) is specifically: a. Divide the 3D image space into N grid cells; map each Gaussian distribution of the 3D Gaussian distribution to a corresponding position in the rasterized image, and calculate the weight or intensity of each Gaussian distribution in the grid; b. For each grid cell, according to its intersection with the Gaussian distribution, weighted processing is performed according to the weight distribution of the Gaussian distribution in the grid to calculate the depth value of the grid cell; c. Depth value mapping: Map the calculated rasterized depth value to the corresponding position in the camera image coordinate system; d. Deep information extraction: According to the position of the depth value in the camera image, the depth information is extracted to generate a depth information map corresponding to the camera image; e. The depth information map is processed by edge smoothing and depth value filtering and then fused with the camera image to obtain a pseudo observation image containing depth information.

8. The deep learning 3D reconstruction method based on FMCW laser radar and vision fusion according to claim 6 is characterized in that: The step 4-4) is specifically: According to the cosine similarity principle, the loss function of the real image is: Among them, X i , Y i are the pixel colors of the pseudo observed image and the real image, respectively.

9. The deep learning 3D reconstruction method based on FMCW laser radar and vision fusion according to claim 1 is characterized in that: The scene reconstruction information includes: the position, speed, color, and semantic information of each dynamic and static object at the same time.

10. The deep learning 3D reconstruction system based on FMCW laser radar and vision fusion according to claim 1, characterized in that: It comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement a deep learning 3D reconstruction method based on FMCW laser radar and vision fusion as described in any one of claims 1 to 9 when executing the computer program.

Images