A Method and System for Enhancing LiDAR Point Cloud Data in Severe Weather Scenarios

Abstract

The embodiments of this application relate to the field of autonomous driving simulation technology, and particularly to a method and system for enhancing lidar point cloud data in harsh weather scenarios. The method includes the following steps: First, a physical model and a lidar model established based on the splashing phenomenon in rainy days are constructed in the simulator, and a simulation data set is generated based on the physical model and the lidar model. Then, a point cloud intensity predictor based on weather information is constructed and trained. The collected multi-modal simulation data is input into the trained point cloud intensity predictor to generate the intensity features of the point cloud data, completing the generation of the simulation point cloud and realizing the enhancement of lidar point cloud data in rainy days. The embodiments of this application solve the problem of poor authenticity of the point cloud intensity of the simulation data in the simulator by establishing a splashing emitter model to simulate the splashing phenomenon in rainy days and a point cloud intensity predictor based on weather information.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of autonomous driving simulation, and particularly to a method and system for enhancing lidar point cloud data in adverse weather scenarios. Background Art

[0002] In the field of autonomous driving perception research, adverse weather conditions such as rainfall will seriously affect the perception performance of autonomous driving sensors. An important reason for the low detection accuracy of lidar perception models in adverse weather is the lack of large-scale point cloud data sets under different weather conditions.

[0003] Most of the existing technologies apply simulation algorithms to process the lidar point cloud data collected in sunny weather. This method cannot expand or edit autonomous driving scenarios and has poor flexibility. And for the dynamic phenomena in rainy days, the existing simulation methods use post-processing methods such as reconstruction or sampling, lack the intensity characteristics of the simulated point cloud, and are difficult to be embedded into the existing autonomous driving simulators, and cannot generate point cloud data of rainy day dynamic phenomena in real time. Summary of the Invention

[0004] The embodiments of the present application provide a method and system for enhancing lidar point cloud data in adverse weather scenarios, aiming to propose a simulator-based simulation scheme for rainy day lidar point cloud data generation, establish a splash emitter model to simulate splash phenomena in rainy days, and a point cloud intensity predictor based on weather information to solve the problem of poor authenticity of the point cloud intensity of simulation data in the simulator.

[0005] To solve the above technical problems, in a first aspect, the embodiments of the present application provide a method for enhancing lidar point cloud data in adverse weather scenarios, including the following steps: First, build a physical model and a lidar model based on rainy day splash phenomena in a simulator, and generate a simulation data set based on the physical model and the lidar model; then, build a point cloud intensity predictor based on weather information and train the point cloud intensity predictor; input the collected multi-modal data into the trained point cloud intensity predictor, and input the collected multi-modal simulation data into the trained point cloud intensity predictor to generate the intensity characteristics of the point cloud data, complete the generation of the simulated point cloud, and realize the enhancement of rainy day lidar point cloud data.

[0006] In some exemplary embodiments, a physical model based on the rain splash phenomenon is constructed in the simulator, including: constructing a physical model based on the rain splash phenomenon according to the generation mechanisms of two splash phenomena, namely tread pick-up and tire side wave; performing simulation calculations based on the physical model; the simulation calculations include: by respectively calculating the number of droplet generations and the initial generation positions of the droplets of the two generation mechanisms, and controlling the force on the droplets in real time during the particle flight process, simulating the flight trajectories of the splash droplets.

[0007] In some exemplary embodiments, the volume flow rate formula is used to respectively calculate the number of droplet generations of the two generation mechanisms, as shown in Equation (1):

[0008] VR TP = Kbvh groove

[0009] VR SD = 0.5bv(WD - Kh groove -(1 - K)h film )

[0010] WD = 6e -4 T 0.09 (LI) 0.6 S -0.33 (1)

[0011] Wherein, VR TP and VR SD respectively represent the volume flow rates of tread pick-up and tire side wave; K is the tire groove width ratio; h groove is the texture depth of the tire tread, h film is the water depth picked up due to capillary action every time the tire rotates; WD is the road surface water depth; T is the road surface texture depth; L is the drainage length; I represents the rainfall rate; S represents the slope.

[0012] In some exemplary embodiments, during the simulation calculation process, the survival time of the droplets is constrained to truly simulate the annihilation process of the splash phenomenon, reducing the computing resources in the simulation process while ensuring the simulation authenticity.

[0013] In some exemplary embodiments, the simulator is an autonomous driving simulator; in the simulation environment, a splash emitter and the collected point cloud data with weather effects are used to simulate the splash phenomenon.

[0014] In some exemplary embodiments, based on the physical model and the lidar model, a simulation data set is generated, including: by obtaining the weather information and vehicle driving information in the simulator, dynamically generating splash droplets, and simulating the flight trajectories of the droplets according to the physical model to achieve the splash phenomenon simulation in the rainy day scenario.

[0015] In some exemplary embodiments, a neural network based on the U-net structure is adopted to learn the point cloud intensity from lidar data, and a point cloud intensity predictor based on weather information is constructed. The Waymo dataset is used as the training dataset, and the RGB camera projection image, depth information, lidar semantic label, and weather information are represented as a multi-channel image in the lidar spherical coordinate system as the input, and based on the trained point cloud intensity predictor, the intensity value of the point cloud is predicted.

[0016] In some exemplary embodiments, training the point cloud intensity predictor includes: introducing an RGB projection mask and a mask for lidar echo loss during training, and supervising and training the point cloud intensity predictor by calculating the point cloud intensity L2 loss function with the mask.

[0017] In some exemplary embodiments, after generating the intensity feature of the point cloud data, it further includes: based on the Waymo dataset, evaluating the influence of different weather information as input and different width images as input on the point cloud intensity predictor respectively; the evaluation includes: calculating the point cloud intensity error using the root mean square error of intensity; based on the generated rainy-day lidar simulation point cloud data and the Waymo dataset, testing the quality of the simulation data in the 3D object detection task.

[0018] In a second aspect, an embodiment of the present application further provides a lidar point cloud data enhancement system for bad weather scenarios, including: a model construction module, a point cloud intensity prediction module, and a data enhancement module connected in sequence; the model construction module is used to construct a physical model and a lidar model based on the rainy-day splashing phenomenon in the simulator, and generate a simulation dataset based on the physical model and the lidar model; the point cloud intensity prediction module is used to construct a point cloud intensity predictor based on weather information and train the point cloud intensity predictor; the data enhancement module is used to input the collected multi-modal simulation data into the trained point cloud intensity predictor, generate the intensity feature of the point cloud data, complete the generation of the simulation point cloud, and realize the enhancement of the rainy-day lidar point cloud data.

[0019] The technical solution provided by the embodiment of the present application has at least the following advantages:

[0020] An embodiment of the present application provides a method and system for enhancing lidar point cloud data in bad weather scenarios. The method includes the following steps: First, construct a physical model and a lidar model based on the rainy-day splashing phenomenon in the simulator, and generate a simulation dataset based on the physical model and the lidar model; then, construct a point cloud intensity predictor based on weather information and train the point cloud intensity predictor; input the collected multi-modal simulation data into the trained point cloud intensity predictor, generate the intensity feature of the point cloud data, complete the generation of the simulation point cloud, and realize the enhancement of the rainy-day lidar point cloud data.

[0021] An embodiment of the present application provides a method for enhancing lidar point cloud data in a severe weather scenario. By proposing a splash simulator model based on a simulator and a "point cloud intensity predictor" based on weather information, a new method for simulating and generating rainy-day point cloud data is improved, solving the problems of weak real-time performance and poor flexibility in existing methods. In the present application, a physical splash phenomenon model and a lidar model are established in an autonomous driving simulator, and by reconstructing the traffic scenario under rain, a lidar simulation data set under real rainfall conditions is generated. At the same time, the present application also develops a "point cloud intensity predictor" based on weather information, adds realistic point cloud intensity features, predicts the point cloud intensity through additional weather information, and combines lidar, camera, etc. as multi-modal inputs, uses the point cloud intensity ground truth as supervision, and establishes the association between weather and point cloud intensity distribution to achieve the generation of highly realistic simulated point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] One or more embodiments are exemplarily illustrated by the pictures in the corresponding drawings. These exemplary illustrations do not constitute a limitation on the embodiments. Unless otherwise stated, the figures in the drawings do not constitute a scale limitation.

[0023] Figure 1 It is a schematic flowchart of a method for enhancing lidar point cloud data in a severe weather scenario provided by an embodiment of the present application;

[0024] Figure 2 It is a schematic diagram of generating rainy-day lidar point cloud simulation data by establishing a physical model and a lidar model in an autonomous driving simulator provided by an embodiment of the present application;

[0025] Figure 3 、 Figure 4 It is a schematic diagram of a physical model established based on the rainy-day splash phenomenon provided by an embodiment of the present application;

[0026] Figure 5 It is a schematic diagram of the force on a splash droplet provided by an embodiment of the present application;

[0027] Figure 6 It is a schematic diagram of the force trajectory of a splash droplet provided by an embodiment of the present application;

[0028] Figure 7 It is a schematic flowchart of a "point cloud intensity predictor" based on weather information predicting the point cloud intensity provided by an embodiment of the present application;

[0029] Figure 8 It is a schematic diagram of the structure of a system for enhancing lidar point cloud data in a severe weather scenario provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] As can be seen from the background art, there are technical problems in the prior art that the autonomous driving scenarios cannot be expanded or edited, and the flexibility is poor. Moreover, when dealing with the dynamic phenomena in rainy days, the existing simulation methods use the post-processing methods of reconstruction or sampling, lack the intensity characteristics of the simulated point cloud, and are difficult to be embedded into the existing autonomous driving simulators, and cannot generate the point cloud data of the dynamic phenomena in rainy days in real time.

[0031] Most of the existing rainy-day lidar simulation methods are based on the physical model of the lidar receiving power. According to the rainfall rate, additional noise point clouds are added to the existing point clouds through the Monte Carlo method or statistical method. For the splashing phenomenon in rainy days, the existing methods adopt a data-driven approach to reconstruct or sample the noise point clouds of the splashing phenomenon, and can perform particle trajectory simulation based on the physical model of the splashing phenomenon. The simulated splashing particles are respectively passed through the spatial and temporal similarity matching algorithms to make it closer to the real splashing phenomenon. Finally, the lidar point cloud data with splashing effect noise is synthesized based on the lidar data under clear weather conditions.

[0032] Therefore, the existing rainy-day lidar simulation methods have the following defects: First, the existing methods for generating point cloud data in bad weather can only generate data based on the existing data, cannot generate data in any scenario, and lack the ability to edit the scenario. Second, the existing bad-weather point cloud simulation methods cannot reproduce the dynamic phenomena related to specific weather conditions, such as the splashing effect behind a vehicle driving at high speed in rainy days. At the same time, the existing intensity prediction algorithms for simulated point clouds based on simulators lack consideration of weather conditions.

[0033] To solve the above technical problems, the embodiments of the present application provide a method and system for enhancing lidar point cloud data in bad weather scenarios. The method includes the following steps: First, a physical model and a lidar model based on the splashing phenomenon in rainy days are constructed in the simulator, and a simulation data set is generated based on the physical model and the lidar model; then, a point cloud intensity predictor based on weather information is constructed, and the point cloud intensity predictor is trained; the collected multi-modal simulation data is input into the trained point cloud intensity predictor to generate the intensity characteristics of the point cloud data, complete the generation of the simulated point cloud, and realize the enhancement of the rainy-day lidar point cloud data. By providing a method and system for enhancing lidar point cloud data in bad weather scenarios, the embodiments of the present application aim to propose a simulator-based rainy-day autonomous driving scenario simulation scheme for the simulation generation of rainy-day lidar point cloud data, establish a splashing emitter model to simulate the splashing phenomenon in rainy days, and a point cloud intensity predictor based on weather information to solve the problem of poor authenticity of the point cloud intensity of the simulation data in the simulator.

[0034] The following will elaborate on the embodiments of the present application in conjunction with the accompanying drawings. However, those of ordinary skill in the art can understand that in the embodiments of the present application, many technical details are provided to help readers better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented.

[0035] Referring to Figure 1 , an embodiment of the present application provides a method for enhancing lidar point cloud data in a severe weather scenario, including the following steps:

[0036] Step S1: Construct a physical model and a lidar model based on the rain splash phenomenon in a simulator, and generate a simulation dataset based on the physical model and the lidar model.

[0037] Step S2: Construct a point cloud intensity predictor based on weather information and train the point cloud intensity predictor.

[0038] Step S3: Input the collected multi-modal simulation data into the trained point cloud intensity predictor to generate the intensity features of the point cloud data, complete the generation of the simulated point cloud, and achieve the enhancement of lidar point cloud data in rainy days.

[0039] Most of the existing technologies process lidar point cloud data collected in clear weather using simulation algorithms. This method cannot expand or edit the autonomous driving scenario, and has poor flexibility. And for the dynamic phenomena in rainy days, the existing simulation methods use post-processing methods such as reconstruction or sampling, lack the intensity features of the simulated point cloud, and are difficult to be embedded into the existing autonomous driving simulator, and cannot generate point cloud data of rainy day dynamic phenomena in real time. Therefore, the present application proposes a simulation scheme for rainy day autonomous driving scenarios based on a simulator, by establishing a physical model (splash emitter model) to simulate the splash phenomenon in rainy days, and a "point cloud intensity predictor" based on weather information, to solve the problem of poor authenticity of the point cloud intensity of the simulation data in the simulator.

[0040] In the simulation environment of the present application, the present application uses a splash emitter and the collected point cloud data with weather effects to simulate the splash phenomenon. The present application has established a data collection pipeline for a rainy highway scenario and uses an intensity predictor to enhance the point cloud features.

[0041] This patent relates to the technical field of autonomous driving simulation. Specifically, it is a simulation generation scheme for lidar point cloud data in rainy days. The purpose of this patent invention is to propose a new method based on a splash simulator model in a simulator and a "point cloud intensity predictor" based on weather information to improve the simulation generation of rainy day point cloud data, and solve the problems of weak real-time performance and poor flexibility existing in the existing methods. As Figure 2As shown, in this application, a physical model of splashing phenomenon and a lidar model are established in an autonomous driving simulator, and by reconstructing the traffic scene under rainy days, a lidar simulation dataset under real rainfall conditions is generated. At the same time, this application also develops a "point cloud intensity predictor" based on weather information to add realistic point cloud intensity features.

[0042] In some embodiments, step S1 constructs a physical model based on the splashing phenomenon in rainy days in the simulator, including: constructing a physical model based on the splashing phenomenon in rainy days according to the generation mechanisms of two splashing phenomena, namely tread pick-up and tire side wave; based on the physical model, performing simulation calculations; the simulation calculations include: by respectively calculating the droplet generation quantity and the initial droplet generation position of the two generation mechanisms, and in real time controlling the force condition of the droplets during the particle flight process, simulating the flight trajectory of the splashing droplets.

[0043] Specifically, step S1 is mainly based on the rainy-day lidar simulation process in the simulator. This application proposes a rainy-day lidar data simulation scheme, by establishing a physical model based on the splashing phenomenon in rainy days in the simulator, as Figure 3 、 Figure 4 shown, which consists of the generation mechanisms of two splashing phenomena, namely tread pick-up and tire side wave. In the simulation calculations, this application uses the volumetric flow rate formula to calculate the droplet generation quantity of the two mechanisms respectively.

[0044] In some embodiments, using the volumetric flow rate formula, the droplet generation quantity and the initial droplet generation position of the two generation mechanisms are respectively calculated, as shown in Equation (1):

[0045] VR TP =Kbvh groove

[0046] VR SD =0.5bv(WD-Kh groove -(1-K)h film )

[0047] WD=6e -4 T 0.09 (LI) 0.6 S -0.33 (1)

[0048] Among them, VR TP and VR SD respectively represent the volumetric flow rates of tread pick-up and tire side wave; K is the tire groove width ratio; h groove is the texture depth of the tire tread, h filmThe water depth picked up by capillary action each time the tire rotates; WD is the road surface water depth; T is the road surface texture depth; L is the drainage length; I represents the rainfall rate; S represents the slope.

[0049] It should be noted that the road surface water depth can be calculated by an empirical formula based on the rainfall rate.

[0050] In some embodiments, the simulator is an autonomous driving simulator; in the simulation environment, a splash emitter and the collected point cloud data with weather effects are used to simulate the splash phenomenon.

[0051] In some embodiments, based on the physical model and the lidar model, a simulation data set is generated, including: obtaining weather information and vehicle driving information in the simulator, dynamically generating splash droplets, and simulating the flight trajectory of the droplets according to the physical model to realize the simulation of the splash phenomenon in rainy scenarios.

[0052] In the simulation calculation, the present application also controls the force on the droplets in real time during the flight of the particles, such as Figure 5 、 Figure 6 As shown, the present application simulates the flight trajectory of the splash droplets.

[0053] In some embodiments, during the simulation calculation process, the survival time of the droplets is restricted to truly simulate the annihilation process of the splash phenomenon, ensuring the authenticity of the simulation while reducing the computing resources in the simulation process.

[0054] In order to truly simulate the annihilation process of the splash phenomenon, the present application restricts the survival time of the droplets, ensuring the authenticity of the simulation while reducing the computing resources in the simulation process.

[0055] Step S2 is mainly a process of constructing a point cloud intensity predictor based on weather information and training the point cloud intensity predictor. In some embodiments, a neural network based on the U-net structure is used to learn the point cloud intensity from lidar data to construct a point cloud intensity predictor based on weather information; the Waymo data set is used as the training data set, and the RGB camera projection image, depth information, lidar semantic label, and weather information are represented as a multi-channel image in the lidar spherical coordinate system as the input, and based on the trained point cloud intensity predictor, the intensity value of the point cloud is predicted.

[0056] Specifically, to improve the authenticity of the simulated point cloud, the present application trains a point cloud intensity predictor based on weather, such as Figure 7As shown, a neural network using a U-net structure learns the point cloud intensity from lidar data. In this application, the Waymo dataset is used to create a training dataset. The RGB camera projection image, lidar semantic label, depth, and additionally added weather information are represented as a multi-channel image in the lidar spherical coordinate system as input to predict the intensity value of the point cloud.

[0057] In some embodiments, the point cloud intensity predictor is trained, including: introducing an RGB projection mask and a mask for lidar echo loss during training, and performing supervised training on the point cloud intensity predictor by calculating the point cloud intensity L2 loss function with the mask.

[0058] Next, step S3 is executed: inputting the collected multi-modal simulation data into the trained point cloud intensity predictor to generate the intensity feature of the point cloud data, completing the generation of the simulated point cloud, and realizing the enhancement of the rainy-day lidar point cloud data. On the CARLA autonomous driving simulator, the embodiments of this application apply the simulation scheme proposed in this application, and use the OpenCDA data acquisition framework to collect lidar data, semantic lidar data, RGB camera images, and weather and vehicle label information in the traffic scene at the same time. Inputting the collected multi-modal data into the point cloud intensity predictor to obtain the intensity feature of the point cloud data, and completing the generation of the rainy-day lidar simulated point cloud.

[0059] On the one hand, this application proposes a simulation algorithm for splashing phenomena based on a physical model. According to the two mechanisms of the splashing phenomenon, this application establishes a physical model in the simulator, dynamically generates splashing droplets by obtaining the weather information and vehicle driving information in the simulator, and simulates the flight of the droplets according to the model to realize the simulation of the splashing phenomenon in the rainy-day scene. On the other hand, this application also proposes a point cloud intensity predictor based on weather information. This application proposes the structure and training method of the point cloud intensity predictor, predicts the point cloud intensity through additional weather information, combines lidar, camera, etc. as multi-modal inputs, and uses the point cloud intensity ground truth as supervision to establish the association between the weather and the point cloud intensity distribution to realize the generation of highly realistic simulated point clouds.

[0060] Compared with the prior art, the advantages of the method for enhancing lidar point cloud data in harsh weather scenarios provided by this application are as follows:

[0061] (1) The prior art solutions can only simulate weather phenomena on existing lidar data in clear weather, it is difficult to replace and edit the scene, and it is difficult to embed the simulation of dynamic effects into the simulator. This application can directly generate rainy-day lidar data based on the simulator, which can be used for enhancing rainy-day lidar data for autonomous driving and improving the perception model ability.

[0062] (2) In the existing technical solutions, the influence of weather conditions is generally not considered in the prediction method of point cloud intensity. The perception predictor proposed in this application can use different weather information as input, and then more accurately predict the intensity characteristics of the point cloud.

[0063] In some embodiments, after generating the intensity characteristics of the point cloud data, it further includes: based on the Waymo dataset, evaluating the influence of using different weather information as input and different-width images as input on the point cloud intensity predictor respectively; the evaluation includes: calculating the point cloud intensity error using the root mean square error of intensity; based on the generated rainy-day lidar simulation point cloud data and the Waymo dataset, testing the quality of the simulation data in the 3D object detection task.

[0064] This application has been proven feasible through experiments, simulations, and use.

[0065] Based on the Waymo dataset, this application evaluated the influence of additional input weather information channels and input images of different widths on the point cloud intensity predictor, and the experimental results are shown in Table 1.

[0066] Table 1 Influence of input weather information channels and input images of different widths on the point cloud intensity predictor

[0067]

[0068] It can be seen from Table 1 that: this application uses the root mean square error of intensity to calculate the point cloud intensity error, which is the average result of all weather conditions including sunny, light rain, and heavy rain. The error of the point cloud intensity predictor with additional weather information as input in this application is significantly smaller.

[0069] Based on the rainy-day lidar simulation data and the Waymo dataset generated by this application, this application tested the quality of the simulation data in the 3D object detection task. This application mixed the real rainy-day dataset and the rainy-day dataset of the simulation data with the real sunny weather dataset respectively as the training set of the experiment, and evaluated the model metrics on the rainy-day test set. The experimental results are shown in Table 2 below.

[0070] Table 2 Evaluation model metrics of the training dataset on the rainy-day test set

[0071]

[0072] It can be seen from Table 2 that: the performance of the simulation mixed dataset on the rainy-day test set is better than that of the model trained with real data, and it also maintains basically the same accuracy as the model trained with real data on the sunny-day test set.

[0073] See Figure 8, the embodiment of the present application also provides a lidar point cloud data enhancement system for severe weather scenarios, including: a model construction module 101, a point cloud intensity prediction module 102, and a data enhancement module 103 connected in sequence; the model construction module 101 is used to construct a physical model and a lidar model based on the rain splash phenomenon in a simulator, and generate a simulation data set based on the physical model and the lidar model; the point cloud intensity prediction module 102 is used to construct a point cloud intensity predictor based on weather information and train the point cloud intensity predictor; the data enhancement module 103 is used to input the collected multi-modal simulation data into the trained point cloud intensity predictor, generate the intensity features of the point cloud data, complete the generation of the simulation point cloud, and realize the enhancement of the lidar point cloud data in rainy days.

[0074] With the above technical solutions, the embodiment of the present application provides a method and system for enhancing lidar point cloud data in severe weather scenarios. The method includes the following steps: First, construct a physical model and a lidar model based on the rain splash phenomenon in a simulator, and generate a simulation data set based on the physical model and the lidar model; then, construct a point cloud intensity predictor based on weather information and train the point cloud intensity predictor; input the collected multi-modal simulation data into the trained point cloud intensity predictor, generate the intensity features of the point cloud data, complete the generation of the simulation point cloud, and realize the enhancement of the lidar point cloud data in rainy days.

[0075] The embodiment of the present application provides a method for enhancing lidar point cloud data in severe weather scenarios. By proposing a new method for simulating and generating rainy-day point cloud data by using a splash simulator model based on a simulator and a "point cloud intensity predictor" based on weather information, the problems of weak real-time performance and poor flexibility existing in the existing methods are solved. The present application establishes a physical splash phenomenon model and a lidar model in an autonomous driving simulator, and generates a lidar simulation data set under real rainfall conditions by reconstructing the traffic scene in rainy days. At the same time, the present application also develops a "point cloud intensity predictor" based on weather information, adds realistic point cloud intensity features, predicts the point cloud intensity through additional weather information, and combines lidar, camera, etc. as multi-modal inputs, uses the point cloud intensity ground truth as supervision, and establishes the association between weather and point cloud intensity distribution to achieve the generation of highly realistic simulation point clouds.

[0076] Those of ordinary skill in the art can understand that the above embodiments are specific embodiments for implementing the present application. In actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application. Any person skilled in the art can make their own changes and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application should be determined by the scope defined in the claims.

Claims

1. A method for enhancing lidar point cloud data in adverse weather scenarios, characterized in that, it includes the following steps: Build a physical model and a lidar model based on rain splash phenomena in a simulator, and generate a simulation dataset based on the physical model and the lidar model; Build a point cloud intensity predictor based on weather information and train the point cloud intensity predictor; Input the collected multi-modal simulation data into the trained point cloud intensity predictor to generate the intensity characteristics of the point cloud data, complete the generation of the simulated point cloud, and achieve the enhancement of lidar point cloud data in rainy days; The step of building a physical model based on rain splash phenomena in the simulator includes: Build a physical model based on rain splash phenomena according to the generation mechanisms of two splash phenomena, namely tread pickup and tire side waves; Based on the physical model, perform simulation calculations; The simulation calculations include: by calculating the number of droplet generations and the initial generation positions of droplets for the two generation mechanisms respectively, and controlling the force on the droplets in real time during the particle flight process, simulate the flight trajectories of the splashed droplets.

2. The method for enhancing lidar point cloud data in adverse weather scenarios according to claim 1, characterized in that, During the simulation calculation process, constrain the survival time of the droplets to truly simulate the annihilation process of the splash phenomenon, ensuring simulation authenticity while reducing the computing resources in the simulation process.

3. The method for enhancing lidar point cloud data in adverse weather scenarios according to claim 1, characterized in that, The simulator is an autonomous driving simulator; In the simulated environment, use a splash emitter and the collected point cloud data with weather effects to simulate the splash phenomenon.

4. The method for enhancing lidar point cloud data in adverse weather scenarios according to claim 1, characterized in that, Generating a simulation dataset based on the physical model and the lidar model includes: Obtain the weather information and vehicle driving information in the simulator, dynamically generate splashed droplets, and simulate the flight trajectories of the droplets according to the physical model to achieve the simulation of the splash phenomenon in rainy scenarios.

5. The method for enhancing lidar point cloud data in adverse weather scenarios according to claim 1, characterized in that, Adopt a neural network based on the U-net structure to learn the point cloud intensity from lidar data and build a point cloud intensity predictor based on weather information; Use the Waymo dataset as the training dataset, represent the RGB camera projection image, depth information, lidar semantic label, and weather information as a multi-channel image in the lidar spherical coordinate system as the input, and predict the intensity value of the point cloud based on the trained point cloud intensity predictor.

6. The method for enhancing lidar point cloud data in adverse weather scenarios according to claim 1, characterized in that, Training the point cloud intensity predictor includes: introducing an RGB projection mask and a mask for lidar echo loss during training, and supervising and training the point cloud intensity predictor by calculating the L2 loss function of the point cloud intensity with the mask.

7. The method for enhancing lidar point cloud data in adverse weather scenarios according to claim 1, It is characterized in that after generating the intensity feature of the point cloud data, it further includes: Based on the Waymo dataset, evaluate the influence of different weather information as input and different width images as input on the point cloud intensity predictor respectively; The evaluation includes: calculating the point cloud intensity error by using the root mean square error of intensity; based on the generated rainy-day lidar simulation point cloud data and the Waymo dataset, testing the quality of the simulation data in the 3D object detection task.

8. A lidar point cloud data enhancement system for bad weather scenarios It is characterized in that it includes: A model construction module, a point cloud intensity prediction module, and a data enhancement module connected in sequence; The model construction module is used to construct a physical model and a lidar model based on the rainy-day splashing phenomenon in the simulator, and generate a simulation dataset based on the physical model and the lidar model; The point cloud intensity prediction module is used to construct a point cloud intensity predictor based on weather information and train the point cloud intensity predictor; The data enhancement module is used to input the collected multi-modal simulation data into the trained point cloud intensity predictor, generate the intensity feature of the point cloud data, complete the generation of the simulation point cloud, and realize the enhancement of the rainy-day lidar point cloud data; Constructing the physical model based on the rainy-day splashing phenomenon in the simulator includes: According to the generation mechanisms of the two splashing phenomena of tread pickup and tire side wave, construct a physical model based on the rainy-day splashing phenomenon; Based on the physical model, perform simulation calculations; The simulation calculation includes: by calculating the number of droplet generations and the initial generation positions of the droplets of the two generation mechanisms respectively, and controlling the force on the droplets in real time during the particle flight process, simulating the flight trajectory of the splashing droplets.

Images