Multi-weather laser radar point cloud generation method based on physical model

Abstract

The invention discloses a multi-weather laser radar point cloud generation method based on a physical model, and the method comprises the steps: employing a weather simulator based on the physical model, and simulating laser radar point cloud data after different days for training; carrying out multi-step denoising through a denoising-diffusion network which comprises a designed spider-black man-bar denoising network; in the de-noising process, a weather signal controller is designed, and the specific day to be generated can be appointed in a unified multi-weather generation framework. According to the invention, multiple laser radar point cloud data in different weather states can be generated in a unified framework, the weather conditions are controlled accurately and efficiently, and the generated laser radar point cloud data in different weathers have excellent fidelity and diversity.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving, and specifically, is a multi-weather lidar point cloud generation method based on a physical model. Background Art

[0002] LiDAR point cloud generation aims to generate extremely large amounts of diverse, high-fidelity LiDAR point cloud data, which is of great significance in areas such as autonomous driving. Due to the high price of LiDAR point cloud data acquisition equipment, complex road conditions, and huge labor costs, LiDAR point cloud data in the field of autonomous driving is still in short supply. Data in bad weather is particularly scarce, because in addition to the high equipment and labor costs mentioned above, data collection in bad weather also faces more complex road conditions and a higher risk of traffic accidents. When data in bad weather is scarce, performing autonomous driving tasks in bad weather will generate more uncontrollable risks. It is an urgent task to obtain data in bad weather and use it for training or fine-tuning existing models.

[0003] There are usually two ways to generate lidar point clouds. The first one uses more traditional generation models, such as autoencoders and adversarial generation models. Representative methods include LG3D, DGLiDAR, and SnowflakeNet. Their training process is very unstable, and the generated lidar point clouds are very rough and have only fixed sizes, which greatly limits the diversity of the generated results. The second method uses a diffusion-denoising model. Representative methods include LiDARGen and R2DM. They all use Unet as a denoising network and can only extract features in a fixed frame. For lidar point clouds, they cannot capture the crucial long-range dependencies and have great performance limitations. It should be pointed out that they cannot generate lidar point cloud data under bad weather. To address this problem, existing methods generally choose to use simulators for simulation. Representative methods include LSS and FSRL. However, they can only simulate a limited number of single weather data and cannot generate diversified lidar point cloud data with specified weather. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-weather lidar point cloud generation method based on a physical model to solve the problems existing in the existing methods.

[0005] The technical solution to achieve the purpose of the present invention is: a multi-weather lidar point cloud generation method based on a physical model, comprising the following steps:

[0006] Step 1: Build a weather simulator based on a physical model, use the laser propagation mechanism under different weather conditions to simulate different weather conditions for normal weather lidar point cloud data, and provide lidar point cloud training data under different weather conditions;

[0007] Step 2: Construct a Spider-Mamba denoising network, convert the lidar point cloud training data under different weather conditions into isometric projection images, and feed them into the constructed Spider-Mamba denoising network. Use the Spider-Mamba mechanism to model the global features of the isometric projection images in a low-computational-cost manner.

[0008] Step 3: Build a weather signal controller and use the information bottleneck mechanism to generate weather control signals with higher knowledge density, so as to achieve precise control of the specified weather in a unified generation framework;

[0009] Step 4: construct a weather signal constraint function that uses the information bottleneck mechanism, and use it together with the two-norm constraint function as the final loss function to constrain the end-to-end training process of the Spider-Black Mamba denoising network. Compared with the prior art, the present invention has the following significant advantages:

[0010] (1) The present invention designs a weather simulator based on a physical model. From the perspective of laser imaging, it can simulate laser radar point cloud data of various weather conditions and use it for model training.

[0011] (2) The present invention designs a Spider-Mamba denoising network, which includes a designed Spider-Mamba mechanism, which can well capture long-range dependencies, better model sparse isometric image features, and has linear complexity. Compared with previous methods, the present invention achieves the capture of long-range dependencies through the Spider-Mamba mechanism, which is beneficial to improving the generation effect of LiDAR point clouds.

[0012] (3) The present invention constructs a Tianhou signal controller that can generate control signals with high knowledge density and rich semantic information, thereby enabling precise control of designated weather conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the overall framework diagram of the method of the present invention.

[0014] Figure 1 (a) is the structural diagram of the weather simulator based on the physical model, (b) is the structural diagram of the Spider-Mamba denoising network, (c) is the structural diagram of the Spider-Mamba mechanism, and (d) is the structural diagram of the weather signal controller.

[0015] Figure 2 This is a comparison chart of the generated results with other methods.

[0016] Figure 3This is a comparison chart between the generated results and the real data. DETAILED DESCRIPTION

[0017] The present invention provides a method for generating multi-weather LiDAR point cloud based on a physical model. The method constructs a weather simulator based on a physical model to simulate LiDAR point cloud data of different weather conditions for training; then a denoising-diffusion network is used, which includes a designed spider-black mamba denoising network, to perform multi-step denoising; during the denoising process, the present invention designs a weather signal controller, which can specify the specific weather conditions to be generated in a unified multi-weather generation framework. The present invention can realize the generation of LiDAR point cloud data under multiple different weather conditions in a unified framework, the control of weather conditions is accurate and efficient, and the generated LiDAR point cloud data of different weather conditions has excellent fidelity and diversity.

[0018] The specific steps are:

[0019] Step 1: The model receives P n As input, we first use the weather simulator based on the physical model to simulate the laser propagation principle in different weather conditions to create sufficient training data. The input is the normal weather point cloud data P n , the weather simulator based on the physical model randomly selects a weather type for simulation and obtains the weather status data [P s , P r , P rf , P f ] and converted to an equidistant projection image After adding random noise, we get And sent to the Spider-Mamba denoising network, snowy weather status data The specific steps of simulation are:

[0020]

[0021] in, Represents the simulation steps for snowy days in a physics-based weather simulator. Represents normal weather point cloud data P n The isometric projection image form, r s is a coefficient used to measure the severity of weather conditions, f drop represents the Bernoulli distribution function for random dropout, with dropout rate and r s Related, and It is composed of 0-1 values ​​and is used to simulate the attenuation of laser with distance. d is the distance value from the laser point to the laser emitter.

[0022] Rainy day status data The specific steps of simulation are:

[0023]

[0024] Among them, p z is the z-height of the laser spot.

[0025] Fog status data The specific steps of the simulation are the same as those for snowy days, except that the fog density parameter r f different.

[0026] Flood status data The specific steps of simulation are:

[0027]

[0028] Step 2: Use the Spider-Mamba denoising network, a structure that denoises in latent space, to perform iterative denoising. In order to reduce the amount of calculation, the present invention splits the features into four parts according to the channel. Input After passing through the encoder, we get Next, cut v into feature blocks Where J is the number of feature blocks and P is the size of the feature blocks. Then it is split into 4 sub-parts by channel Then, The feature embedding vector is projected into D dimensions. The steps are:

[0029]

[0030] in, yes The jth block of is a learnable projection matrix. After that, T l-1 Send to the first level of Spider-Mamba to get T l Repeat this process until T is obtained. L The present invention further processes it to obtain And splice them according to the channel dimension to obtain The steps are:

[0031]

[0032] Among them, f SM represents the Spider-Mamba mechanism, f mlp represents the multi-layer perceptron processing step, f norm represents the standardization operation, f cat Represents a concatenation operation along the channel dimension.

[0033] The final output uses the two-norm loss function L SMDN To constrain, the steps are:

[0034]

[0035] Among them, f SMDN represents the Spider-Mamba denoising network, t represents the current denoising step, and θ represents the learnable parameters of the network.

[0036] Step 3: Using the Weather Signal Controller To learn control signals with compact semantic information. The present invention utilizes the text encoder of the CLIP model and preset weather description words to provide semantic vector c i As semantic anchoring, the steps are:

[0037]

[0038] Among them, c j The weather vector represented by The weather conditions are kept consistent. As described above, the present invention uses the information bottleneck IB to establish the loss function L WCL To constrain, the steps are:

[0039]

[0040] Where I represents mutual information, β is a weight hyperparameter, set to 0.2, is a learnable parameter in the network. The optimization goal is to minimize To minimize w relative to c i≠j Redundant information to ensure The most relevant information is retained. At the same time, maximizing To help w learn more compact semantic information and guide the diffusion-denoising network to generate more stylized data.

[0041] Step 4: In the end-to-end training process designed by the present invention, supervised training is performed using the following loss function:

[0042] L=L SMDN +L WCL #(8)

[0043] The present invention is further described below in conjunction with the accompanying drawings and embodiments of the specification. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] The overall framework of the method is as follows Figure 1As shown in FIG. 1 , a method for generating laser radar point cloud based on an isometric projection transformer is shown in FIG. 1 , and the specific steps of its training are as follows:

[0045] Step 1: Figure 1 Part (a) shows the weather simulator based on the physical model. The model receives P n As input, we first use the weather simulator based on the physical model to simulate the laser propagation principle in different weather conditions to create sufficient training data. The input is the normal weather point cloud data P n , the weather simulator based on the physical model randomly selects a weather type for simulation and obtains the weather status data [P s , P r , P rf , P f ] and converted to an isometric projection image After adding random noise, we get And sent to the Spider-Mamba denoising network, snowy weather status data The specific steps of simulation are:

[0046]

[0047] in, Represents the simulation steps for snowy days in a physics-based weather simulator. Represents normal weather point cloud data P n The isometric projection image form, r s is a coefficient used to measure the severity of weather conditions, f drop represents the Bernoulli distribution function for random dropout, with dropout rate and r s Related, and It is composed of 0-1 values ​​and is used to simulate the attenuation of laser with distance. d is the distance value from the laser point to the laser emitter.

[0048] Rainy day status data The specific steps of simulation are:

[0049]

[0050] Among them, p z is the z-height of the laser spot.

[0051] Fog status data The specific steps of the simulation are the same as those for snowy days, except that the fog density parameter r f different.

[0052] Flood status data The specific steps of simulation are:

[0053]

[0054] Step 2: Figure 1 Part (b) shows the structure of the Spider Mamba denoising network. Figure 1 Part (c) of the paper shows the details of the Spider-Mamba mechanism. The present invention uses the Spider-Mamba denoising network, a structure that performs denoising in latent space, to perform iterative denoising. In order to reduce the amount of calculation, the present invention splits the features into four parts according to the channel. Input After passing through the encoder, we get Next, Cut to feature block Where J is the number of feature blocks and P is the size of the feature blocks. Then it is split into 4 sub-parts by channel Then, The feature embedding vector is projected into D dimensions. The steps are:

[0055]

[0056] in, yes The jth block of is a learnable projection matrix. After that, T l-1 Send to the first level of Spider-Mamba to get T l Repeat this process until T is obtained. L The present invention further processes it to obtain And splice them according to the channel dimension to obtain The steps are:

[0057]

[0058] Among them, f SM represents the Spider-Mamba mechanism, f mlp represents the multi-layer perceptron processing step, f norm represents the standardization operation, f cat Represents a concatenation operation along the channel dimension.

[0059] The final output uses the two-norm loss function L SMDN To constrain, the steps are:

[0060]

[0061] Among them, f SMDN represents the Spider-Mamba denoising network, t represents the current denoising step, and θ represents the learnable parameters of the network.

[0062] Step 3: Use the weather signal controller w to learn the control signal with compact semantic information. The present invention uses the text encoder C of the CLIP model and the preset weather description words to provide the semantic vector c i As semantic anchoring, the steps are:

[0063]

[0064] Among them, c j The weather vector represented by The weather conditions are kept consistent. As described above, the present invention uses the information bottleneck IB to establish the loss function L WCL To constrain, the steps are:

[0065]

[0066] Where I represents mutual information, β is a weight hyperparameter, set to 0.2, is a learnable parameter in the network. The optimization goal is to minimize To minimize w relative to c i≠j Redundant information to ensure The most relevant information is retained. At the same time, maximizing To help w learn more compact semantic information and guide the diffusion-denoising network to generate more stylized data.

[0067] Step 4: In the end-to-end training process designed by the present invention, supervised training is performed using the following loss function:

[0068] L=L SMDN +L WCL #(8)

[0069] In order to further verify the feasibility and effectiveness of the method of the present invention, the technical effect of the present invention is further described in combination with experiments. The hardware platform used in the experiment of the present invention is: Intel (R) Xeon (R) Gold 6230CPU@2.10GHz×32, memory is 160G, 4 RTX 3090 GPUs with 24G video memory, and the software platform used in the experiment of the present invention is: Ubuntu18.04.6 LTS operating system and Python3.8.17, Pytorch2.1.0 deep learning framework. The KITTI360 dataset and the Seeing through fog dataset are used to analyze the laser radar point cloud generation effect obtained by this method. Here, the four indicators of Frechet range distance (FRD), Frechet point cloud distance (FPD), Jensen-Shannon divergence (JSD), and minimum matching distance (MMD) are used to evaluate the performance of laser radar point cloud generation.

[0070] The method of the present invention is used to train on the KITTI360 dataset. During generation, randomly sampled Gaussian noise is used as input to feed the model to obtain diversified lidar point cloud generation results. In order to test the performance of the method of the present invention, the proposed lidar point cloud generation method based on isometric projection transformer is compared with several existing methods, including: LiDARGAN, a method based on generative adversarial network proposed by Caccia, L et al. in their paper “Deep generative modeling of lidar data. In: 2019 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 5034-5040. 2019.”, LiDARVAE, a method based on variational autoencoder proposed by Caccia, L et al. in their paper “Deep generative modeling of lidar data. IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 5034-5040. 2019.”, and Sauer, A. et al. in their paper “Projected gansconverge faster. Advances in Neural Information Processing Systems (NeurIPS). pp: 17480-174922021." ProjectedGAN, a diffusion-denoising model-based method LiDARGen proposed by Zyrianov, V. et al. in their published paper "Learning to generate realistic lidar point clouds. European Conference on Computer Vision (ECCV). pp. 17-35.2022.", and R2DM proposed by Nakashima, K. et al. in their published paper "Lidar data synthesis with denoising diffusion probabilistic models. arXiv preprint arXiv: 2309.09256 (2023)". The quantitative results are listed in Table 1. The intersection-union ratio of the above-mentioned evaluation indicators is used as the evaluation criterion, and the values ​​of all evaluation indicators are as small as possible.

[0071] Table 1 Quantitative test results on KITTI360 using the method of the present invention and the prior art

[0072] method FPD FRD MMD JSD LiDARGAN - 3003.8 30.60 - LiDARVAE - 2261.5 10.00 16.10 ProjectedGAN - 2117.2 3.47 8.50 LiDAR Gen 90.29 579.39 7.39 7.38 R2DM 6.24 149.66 1.91 3.05 Method of the present invention 6.15 140.44 0.61 2.66

[0073] From the results in Table 1, it can be seen that the method of the present invention has achieved the best results in all four indicators, indicating that the lidar point cloud data generated by the present invention has the best authenticity and diversity. The qualitative test results compared with other methods are as follows Figure 2 As shown in Figure 2, the comparison with the real-world data is Figure 3 As shown in the figure, the laser radar point cloud data generated by the method of the present invention is more detailed than the existing method, and the target is generated more clearly.

[0074] The above embodiments should be regarded as illustrative and non-restrictive, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the invention and within the spirit and principles of the present invention should be covered by the protection scope of the present invention.

Claims

1. A multi-weather lidar point cloud generation method based on a physical model, characterized in that: The following steps are involved: Step 1: Build a weather simulator based on a physical model, use the laser propagation mechanism under different weather conditions to simulate different weather conditions for normal weather lidar point cloud data, and provide lidar point cloud training data under different weather conditions; Step 2: Construct a Spider-Mamba denoising network, convert the lidar point cloud training data under different weather conditions into isometric projection images, and feed them into the constructed Spider-Mamba denoising network. Use the Spider-Mamba mechanism to model the global features of the isometric projection images in a low-computational-cost manner. Step 3: Build a weather signal controller and use the information bottleneck mechanism to generate weather control signals with higher knowledge density, so as to achieve precise control of the specified weather in a unified generation framework; Step 4: Construct a weather signal constraint function that utilizes the information bottleneck mechanism, and use it together with the two-norm constraint function as the final loss function to constrain the end-to-end training process of the Spider-Black Mamba denoising network.

2. The method for generating multi-weather laser radar point cloud based on physical model according to claim 1, characterized in that: In the physical model-based weather simulator described in step 1, different simulation processes are designed according to the laser propagation mechanism in different weather conditions: First, the input of the model is the normal weather point cloud data P n , the weather simulator based on the physical model randomly selects a weather type for simulation and obtains the weather status data [P s ,P r ,P rf ,P f ], where P s Represents the point cloud data under snow conditions, P r Represents the point cloud data under rainy conditions, P rf Represents point cloud data under flood conditions, P f Represents the point cloud data under foggy conditions; then, the corresponding point cloud data is converted to an equidistant projection image under the corresponding weather conditions After adding random noise, we get the noisy equidistant projection image. Snowy weather isometric projection image The specific steps of simulation are: in, Represents the simulation steps for snowy days in a physics-based weather simulator. Represents normal weather point cloud data P n The isometric projection image form, r s is a coefficient used to measure the severity of weather conditions, f drop represents the Bernoulli distribution function for random dropout, with dropout rate and r s Related, and It consists of a value of 0-1, which is used to simulate the attenuation of the laser with distance, d is the distance value from the laser point to the laser emitter; Rainy day status data The specific steps of simulation are: Among them, p z is the z-axis height of the laser spot; Foggy weather isometric projection image The specific steps of simulation are: Among them, r f It is a fog density parameter used to measure the severity of weather; Flood status isometric projection image The specific steps of simulation are:

3. The method for generating multi-weather laser radar point cloud based on physical model according to claim 1, characterized in that: Step 2 uses the state space equation to construct the Spider-Mamba denoising network, including; First, the noisy equidistant projection image Send it to an encoder consisting of four layers of convolution to obtain denoising features Next, Cut to feature block Where J is the number of feature blocks and P is the size of the feature blocks; then, Then it is split into 4 sub-parts by channel final, The feature embedding vector is projected into D dimensions. The steps are: in, yes The jth block of is a learnable projection matrix; after that, T l-1 Send to the first level of Spider-Mamba to get T l Repeat this process until T is obtained. L ; further processing is performed to obtain And splice them according to the channel dimension to obtain The steps are: Among them, f SM represents the Spider-Mamba mechanism, f mlp represents the multi-layer perceptron processing step, f norm represents the standardization operation, f cat Represents the splicing operation by channel dimension; The final output uses the two-norm loss function L SMDN To make constraints, the steps are: Among them, f SMDN represents the Spider-Mamba denoising network, t represents the current denoising step, and θ represents the learnable parameters of the network.

4. The method for generating multi-weather laser radar point cloud based on physical model according to claim 1, characterized in that: Step 3: Build the weather signal controller To learn control signals with compact semantic information; text encoder using CLIP model and preset weather description words to provide semantic vector c i As a semantic anchor, its implementation steps are: Among them, c j The weather vector represented by Keep the weather conditions consistent and use the information bottleneck IB to establish the loss function L WCL To constrain, the steps are: Where I represents mutual information, β is a weight hyperparameter, is a learnable parameter in the network.

5. The method for generating multi-weather lidar point cloud based on physical model according to any one of claims 1 to 4, characterized in that: In the end-to-end training process of step 4, the loss function used is: L=L SMDN +L WCL #(8)

Images