A multi-weather laser radar point cloud generation method based on a physical model

Abstract

The application discloses a kind of multi-weather laser radar point cloud generation methods based on physical model, contain a weather simulator based on physical model, simulate different weather after laser radar point cloud data for training;Again through denoising-diffusion network, it contains a designed spider-black manba denoising network, multi-step denoising is carried out;In the denoising process, the application designs a weather signal controller, can in unified multi-weather generation framework, specify the specific weather after to be generated.The application can be realized in a unified framework to generate multi different weather state under laser radar point cloud data, weather condition control is accurate and efficient, and the laser radar point cloud data of different weather generated has excellent fidelity and diversity.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving, specifically, it is a method for generating multi-weather lidar point clouds based on a physical model. Background Technology

[0002] LiDAR point cloud generation aims to produce massive amounts of diverse and highly realistic LiDAR point cloud data, which is of great significance in fields such as autonomous driving. However, due to the high cost of LiDAR point cloud data acquisition equipment, complex road conditions, and significant labor costs, LiDAR point cloud data for autonomous driving remains extremely scarce. Data collected in adverse weather conditions is particularly rare, as in addition to the aforementioned high equipment and labor costs, data collection in adverse weather conditions faces even more complex road conditions and a higher risk of traffic accidents. When data in adverse weather conditions is scarce, autonomous driving tasks performed in such conditions introduce more uncontrollable risks. Acquiring data in adverse weather conditions and using it for training or fine-tuning existing models is an urgent task.

[0003] Current LiDAR point cloud generation algorithms typically fall into two categories. The first utilizes more traditional generative models, such as autoencoders and adversarial generative models. Representative methods include LG3D, DGLiDAR, and SnowflakeNet. However, their training processes are highly unstable, resulting in coarse-grained LiDAR point clouds with fixed sizes, severely limiting the diversity of the generated results. The second approach employs diffusion-denoising models, with representative methods including LiDARGen and R2DM. Both utilize Unet as the denoising network, which can only extract features from fixed bounding boxes. For LiDAR point clouds, this approach fails to capture crucial long-range dependencies, exhibiting significant performance limitations. It's worth noting that neither approach can generate LiDAR point cloud data under adverse weather conditions. To address this issue, existing methods generally employ simulators, with representative methods including LSS and FSRL. However, these can only simulate a limited amount of single-weather data, failing to generate diverse, weather-specific LiDAR point cloud data. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-weather lidar point cloud generation method based on a physical model, in order to solve the problems existing in the existing methods.

[0005] The technical solution to achieve the purpose of this invention is: a method for generating multi-weather lidar point clouds based on a physical model, comprising the following steps:

[0006] Step 1: Construct a weather simulator based on a physical model. Utilize the laser propagation mechanism under different weather conditions to simulate different weather conditions from normal weather lidar point cloud data, so as to provide lidar point cloud training data under different weather conditions.

[0007] Step 2: Construct a Spider-Black Mamba denoising network. Convert the LiDAR point cloud training data under different weather conditions into isometric projection images and feed them into the constructed Spider-Black Mamba denoising network. Utilize the Spider-Mamba mechanism to model the global features of the isometric projection images in a low-computational-cost manner.

[0008] Step 3: Construct a weather signal controller and use the information bottleneck mechanism to generate weather control signals with higher knowledge density, so as to achieve precise control of a specified weather within a unified generation framework.

[0009] Step four involves constructing a weather signal constraint function that utilizes the information bottleneck mechanism, and using it, along with the L2 constraint function, as the final loss function to constrain the end-to-end training process of the Spider-Black Mamba denoising network. Compared with existing technologies, this invention has the following significant advantages:

[0010] (1) The present invention designs a weather simulator based on a physical model. From the perspective of laser imaging, it can simulate laser radar point cloud data of various weather conditions and use it for model training.

[0011] (2) This invention designs a Spider-Mamba denoising network, which includes a designed Spider-Mamba mechanism that can effectively capture long-range dependencies, better model sparse isometric projection image features, and has linear complexity. Compared with previous methods, this invention achieves the capture of long-range dependencies through the Spider-Mamba mechanism, which is beneficial to improving the generation effect of LiDAR point clouds.

[0012] (3) The present invention constructs a weather signal controller that can generate control signals with high knowledge density and rich semantic information, thereby enabling precise control of a specified weather. Attached Figure Description

[0013] Figure 1 This is an overall framework diagram of the method of the present invention.

[0014] Figure 1 (a) is a structural diagram of a weather simulator based on a physical model, (b) is a structural diagram of a spider-mamba denoising network, (c) is a structural diagram of a spider-mamba mechanism, and (d) is a structural diagram of a weather signal controller.

[0015] Figure 2 This is a comparison chart of the generated results with other methods.

[0016] Figure 3It is a comparison chart of the generated results and the actual data. Detailed Implementation

[0017] This invention provides a method for generating multi-weather lidar point clouds based on a physical model. The method constructs a weather simulator based on a physical model to simulate lidar point cloud data under different weather conditions for training. Then, a denoising-diffusion network, which includes a designed spider-black mamba denoising network, is used for multi-step denoising. During the denoising process, this invention designs a weather signal controller, which allows specifying the specific weather conditions to be generated within a unified multi-weather generation framework. This invention can generate lidar point cloud data under multiple different weather conditions within a unified framework, with accurate and efficient control of weather conditions. The generated lidar point cloud data under different weather conditions exhibits excellent fidelity and diversity.

[0018] The specific steps are as follows:

[0019] Step 1: The model receives P n As input, a physical model-based weather simulator is first used to perform targeted simulations of laser propagation principles under different weather conditions, generating sufficient training data. The input is normal weather point cloud data P. n A weather simulator based on a physical model randomly selects a weather condition for simulation to obtain weather state data [P]. s ,P r ,P rf ,P f One of them, and convert it to an isometric projection image. After adding random noise, the result is... The data is then fed into a Spider-Mamba denoising network to process the snowy weather conditions. The specific steps of the simulation are as follows:

[0020]

[0021] in, This represents the simulation steps for snowy weather in a physics-based weather simulator. P represents normal weather point cloud data n The isometric projection image form, r s It is a coefficient used to measure the severity of weather, f drop Represents the Bernoulli distribution function used for random dropping, with the dropping rate being related to r. s Related, and Composed of values ​​from 0 to 1, used to simulate laser attenuation with distance, d is the distance from the laser point to the laser emitter.

[0022] Rainy day status data The specific steps of the simulation are as follows:

[0023]

[0024] Where, p z It is the z-axis height of the laser point.

[0025] Fog weather status data The specific steps of the simulation are the same as those for a snowy day, except for the fog concentration parameter r. f different.

[0026] Flood status data The specific steps of the simulation are as follows:

[0027]

[0028] Step 2: Iterative denoising is performed using a Spider-Mamba denoising network, which denoises in the latent space. To reduce computation, the features are split into four parts according to channels. Input After passing through the encoder, it will obtain Next, Cut into feature blocks Where J is the number of feature blocks and P is the size of the feature blocks. It was then divided into 4 sub-parts according to the channel. Then, The projection into a D-dimensional feature embedding vector is achieved through the following steps:

[0029]

[0030] in, yes The j-th block, It is a learnable projection matrix. After this, T... l-1 Send it to the lth layer of the Spider-Mamba to get T l Repeat this process until T is obtained. L The present invention further processes it to obtain... And then concatenate them according to the channel dimension to obtain The steps are as follows:

[0031]

[0032] Among them, f SM Representing the Spider-Mamba mechanism, f mlp f represents the processing steps of a multilayer perceptron. norm Represents standardized operation, f cat This represents a splicing operation based on channel dimensions.

[0033] The final output utilizes the L2-norm loss function L SMDNThe steps to apply constraints are as follows:

[0034]

[0035] Among them, f SMDN θ represents the Spider-Mamba denoising network, t represents the current denoising step, and θ represents the learnable parameters of the network.

[0036] Step 3: Utilize the weather signal controller This invention utilizes a CLIP model text encoder to learn control signals with compact semantic information. and preset weather descriptors to provide semantic vector c i As a semantic anchor, the steps are as follows:

[0037]

[0038] Among them, c j The weather vector represented by The weather conditions remain consistent, as described above. In this invention, the information bottleneck IB is used to establish a loss function L. WCL To impose constraints, the steps are as follows:

[0039]

[0040] Where I represents mutual information, and β is the weight hyperparameter, set to 0.2. These are the learnable parameters in the network. The optimization goal is to minimize them. To minimize w relative to c i≠j Redundant information, ensuring consistency with The most relevant information is retained. At the same time, maximizing... This helps w learn more compact semantic information, guiding the diffusion-denoising network to generate more stylistic data.

[0041] Step 4: The end-to-end training process designed in this invention utilizes the following loss function for supervised training:

[0042] L = L SMDN +L WCl (9)

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments. The described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0044] The overall framework of the method is as follows Figure 1As shown, a method for generating lidar point clouds based on an equidistant projection transformer includes the following specific training steps:

[0045] Step 1: Figure 1 Part (a) shows the weather simulator based on a physical model, where the model receives P n As input, a physical model-based weather simulator is first used to perform targeted simulations of laser propagation principles under different weather conditions, generating sufficient training data. The input is normal weather point cloud data P. n A weather simulator based on a physical model randomly selects a weather condition for simulation to obtain weather state data [P]. s ,P r ,P rf ,P f One of them, and convert it to an isometric projection image. After adding random noise, the result is... The data is then fed into a Spider-Mamba denoising network to process the snowy weather conditions. The specific steps of the simulation are as follows:

[0046]

[0047] in, This represents the simulation steps for snowy weather in a physics-based weather simulator. P represents normal weather point cloud data n The isometric projection image form, r s It is a coefficient used to measure the severity of weather, f drop Represents the Bernoulli distribution function used for random dropping, with the dropping rate being related to r. s Related, and Composed of values ​​from 0 to 1, used to simulate laser attenuation with distance, d is the distance from the laser point to the laser emitter.

[0048] Rainy day status data The specific steps of the simulation are as follows:

[0049]

[0050] Where, p z It is the z-axis height of the laser point.

[0051] Fog weather status data The specific steps of the simulation are the same as those for a snowy day, except for the fog concentration parameter r. f different.

[0052] Flood status data The specific steps of the simulation are as follows:

[0053]

[0054] Step 2: Figure 1 Part (b) shows the structure of the Spider Mamba denoising network. Figure 1 Part (c) illustrates the specific details of the Spider-Mamba mechanism. This invention utilizes the Spider-Mamba denoising network, a structure that performs denoising in the latent space, for iterative denoising. To reduce computational cost, this invention splits the features into four parts according to channels. Input After passing through the encoder, it will obtain Next, Cut into feature blocks Where J is the number of feature blocks and P is the size of the feature blocks. It was then divided into 4 sub-parts according to the channel. Then, The projection into a D-dimensional feature embedding vector is achieved through the following steps:

[0055]

[0056] in, yes The j-th block, It is a learnable projection matrix. After this, T... l-1 Send it to the lth layer of the Spider-Mamba to get T l Repeat this process until T is obtained. L The present invention further processes it to obtain... And then concatenate them according to the channel dimension to obtain The steps are as follows:

[0057]

[0058] Among them, f SM Representing the Spider-Mamba mechanism, f mlp f represents the processing steps of a multilayer perceptron. norm Represents standardized operation, f cat This represents a splicing operation based on channel dimensions.

[0059] The final output utilizes the L2-norm loss function L SMDN The steps to apply constraints are as follows:

[0060]

[0061] Among them, f SMDN θ represents the Spider-Mamba denoising network, t represents the current denoising step, and θ represents the learnable parameters of the network.

[0062] Step 3: Utilize the weather signal controller This invention utilizes a CLIP model text encoder to learn control signals with compact semantic information. and preset weather descriptors to provide semantic vector c i As a semantic anchor, the steps are as follows:

[0063]

[0064] Among them, c j The weather vector represented by The weather conditions remain consistent, as described above. In this invention, the information bottleneck IB is used to establish a loss function L. WCL To impose constraints, the steps are as follows:

[0065]

[0066] Where I represents mutual information, and β is the weight hyperparameter, set to 0.2. These are the learnable parameters in the network. The optimization goal is to minimize them. To minimize w relative to c i≠j Redundant information, ensuring consistency with The most relevant information is retained. At the same time, maximizing... This helps w learn more compact semantic information, guiding the diffusion-denoising network to generate more stylistic data.

[0067] Step 4: The end-to-end training process designed in this invention utilizes the following loss function for supervised training:

[0068] L = L SMDN +L WCL (9)

[0069] To further verify the feasibility and effectiveness of the method of this invention, the technical effects of this invention are further described in conjunction with experiments. The hardware platform used in the experiments of this invention is: Intel(R) Xeon(R) Gold 6230 CPU@2.10GHz×32, 160G of memory, and four RTX 3090 GPUs with 24G of video memory each. The software platform used in the experiments of this invention is: Ubuntu 18.04.6LTS operating system and Python 3.8.17 and PyTorch 2.1.0 deep learning framework. The KITTI360 dataset and the Seeing through fog dataset are used to analyze the LiDAR point cloud generation effect obtained by this method. Here, the performance of LiDAR point cloud generation is evaluated using four indicators: Frechet range distance (FRD), Frechet point cloud distance (FPD), Jensen–Shannon divergence (JSD), and minimum matching distance (MMD).

[0070] The method of this invention is used to train on the KITTI360 dataset. During generation, Gaussian noise is randomly sampled as input to feed the model, resulting in diverse LiDAR point cloud generation results. To test the performance of the method of this invention, the proposed lidar point cloud generation method based on equidistant projection transformer is compared with several existing methods. The comparison methods include: LiDARGAN, a generative adversarial network-based method proposed by Caccia, L. et al. in their paper "Deep generative modeling of lidar data. In: 2019 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 5034–5040. 2019."; LiDARVAE, a variational autoencoder-based method proposed by Caccia, L. et al. in their paper "Deep generative modeling of lidar data. IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 5034–5040. 2019."; and Sauer, A. et al. in their paper "Projected gans converge faster. Advances in Neural Information Processing Systems (NeurIPS). pp: 17480–17492". The methods proposed in 2021 include ProjectedGAN (based on generative adversarial networks), LiDARGen (based on diffusion-denoising probabilistic models), and R2DM (based on diffusion probabilistic models), as proposed by Zyrianov, V. et al. in their paper "Learning to generate realistic lidar point clouds. European Conference on Computer Vision (ECCV). pp. 17–35. 2022.", and Nakashima, K. et al. in their paper "Lidar data synthesis with denoising diffusion probabilistic models. arXiv preprint arXiv:2309.09256 (2023)". The quantitative results are shown in Table 1. Using the intersection-union ratio (IUU) as the evaluation criterion, a smaller value for all evaluation metrics is better.

[0071] Table 1. Quantitative test results on the KITTI 360 using the method of the present invention and existing technologies.

[0072] method FPD FRD MMD JSD LiDARGAN - 3003.8 30.60 - LiDARVAE - 2261.5 10.00 16.10 ProjectedGAN - 2117.2 3.47 8.50 LiDARGen 90.29 579.39 7.39 7.38 R2DM 6.24 149.66 1.91 3.05 Method of the present invention 6.15 140.44 0.61 2.66

[0073] As can be seen from the results in Table 1, the method of this invention achieved the best results in all four indicators, indicating that the lidar point cloud data generated by this invention possesses the best realism and diversity. Qualitative test results comparing it with other methods are as follows: Figure 2 As shown, the comparison with data collected from the real world is as follows: Figure 3 As shown, the lidar point cloud data generated by the method of this invention is more detailed than that of existing methods, and the target generation is clearer.

[0074] The above embodiments should be considered as exemplary and non-limiting, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for multi-weather lidar point cloud generation based on physical model, characterized in that, Comprising the following steps: Step one, constructing a weather simulator based on a physical model, using the laser propagation mechanism under different weather conditions, simulating normal weather lidar point cloud data under different weather conditions to provide lidar point cloud training data under different weather conditions; Step two, constructing a spider-black mamba denoising network, converting the lidar point cloud training data under different weather conditions into an isometric projection image, and sending it into the constructed spider-black mamba denoising network, using the spider-mamba mechanism to model the global features of the isometric projection image in a low computational overhead manner; Step three, constructing a weather signal controller, using an information bottleneck mechanism to generate a weather control signal with higher knowledge density to achieve precise control of the specified weather in a unified generation framework; Step four, constructing a weather signal constraint function using the information bottleneck mechanism, and using it together with the two-norm constraint function as the final loss function to constrain the end-to-end training process of the spider-black mamba denoising network.

2. The physics model based multi-weather lidar point cloud generation method of claim 1, wherein: In step one, the construction of the weather simulator based on the physical model, for the laser propagation mechanism under different weather conditions, the following different simulation processes are designed: Firstly, the input of the model is normal weather point cloud data P n , a weather simulator based on a physical model randomly selects a kind of weather for simulation, and obtains one of weather state data [P s , P r , P rf , P f ], wherein P s represents point cloud data in snow weather, P r represents point cloud data in rain weather, P rf represents point cloud data in flood weather, and P f represents point cloud data in fog weather; then, the corresponding point cloud data is converted into an equirectangular projection image After adding random noise, a noisy equirectangular projection image is obtained Snowy day state equirectangular projected image The specific steps of the simulation are: wherein, represents a simulation step for snow weather in a physical model based weather simulator, represents normal weather point cloud data P n in the form of an equirectangular projection image, r s is a coefficient for measuring the degree of weather severity, f drop represents a Bernoulli distribution function for random discarding, the discarding rate being related to r s , and consists of values from 0-1, for simulating laser attenuation with distance, d being the distance value of the laser point to the laser emitter; Rainy day status data The specific steps of the simulation are: where p z is the z-axis height of the laser spot; Foggy state equirectangular projected image The specific steps of the simulation are: wherein r f is a fog concentration parameter for measuring the severity of the fog; Flood condition equirectangular projected image The specific steps of the simulation are:

3. The physics model based multi-weather lidar point cloud generation method of claim 2, wherein: Step two, constructing a spider-mamba denoising network using state space equations, including: First, the noisy equidistant projection image is fed into an encoder composed of four convolutional layers to get the denoised features Then, the denoised features are cropped into feature patches where J is the number of feature patches and P is the size of the feature patches; then, are split into 4 sub-parts along the channel Finally, are projected into D-dimensional feature embedding vectors by the following steps:​ wherein, is the j-th block of is a learnable projection matrix; after which, T l-1 is fed into the l-th layer of Spider-Mamba to obtain T l , and this process is repeated until T L is obtained; which is further processed to obtain and concatenated along the channel dimension to obtain The steps are: wherein f SM represents a spider-mamba mechanism, f mlp represents a multi-layer perceptron processing step, f norm represents a standardization operation, f cat represents a channel-wise concatenation operation; The final output utilizes a two-norm loss function L SMDN Constraints are imposed, the steps of which are: where f SMDN denotes the spider-mamba denoising network, t denotes the current denoising step, and denotes the learnable parameters of the network.

4. The physics model based multi-weather lidar point cloud generation method of claim 3, wherein: Building weather signal controller in step three to learn control signals with compact semantic information; using the text encoder of the CLIP model and the preset weather descriptor to provide a semantic vector c i As a semantic anchor, the implementation steps are: Wherein, c j The weather vector represented is consistent with The weather state in which the information bottleneck IB is set up loss function L WCL Is constrained, and the steps are: where I represents mutual information, β is a weight hyperparameter, are learnable parameters in the network.

5. The physics model based multi-weather lidar point cloud generation method of claim 4, wherein, In the end-to-end training process of step four, the loss function used is: L = L SMDN + L WCL (10).

Images