Laser radar target detection method in rainy and snowy weather based on deep learning

Abstract

The invention provides a deep learning-based laser radar target detection method in rainy and snowy weather, and provides a denoising solution for accurately filtering medium and large rain and snow scenes so as to improve the real-time performance of ground target detection and solve the technical difficulty of laser radar on-board deployment. Comprising the following implementation steps: step 1) data acquisition and training environment preparation; 2) reading data and extracting original characteristics of the environment; step 3), image processing; step 4), feature extraction; 5) extracting medium and large rain and snow point cloud noisy points; sixthly, rain and snow noisy points are removed; and step 7), inputting a set of the filtered normal point clouds into a centerpoint target detection network to realize sensing of surrounding obstacles in complex weather.

Description

Technical Field

[0001] The present application relates to a target detection method for filtering laser radar noise in rainy and snowy environments, and belongs to the field of unmanned driving and automatic navigation. Background Art

[0002] To ensure the safety of the vehicle itself, road facilities, and pedestrians, unmanned vehicles must have a high level of environmental perception. Environmental perception is the basis for obtaining information about the surrounding environment for autonomous decision-making and navigation control, and non-ground data detection is an important part of environmental perception. The existing obstacle detection methods for unmanned vehicles at home and abroad mainly include laser radar, because it has the advantages of being less affected by light and higher accuracy.

[0003] LiDAR sensors are susceptible to adverse weather conditions such as snow, fog and rain in autonomous driving, which causes unnecessary noise factors and seriously reduces the cognitive performance of the perception system. The following previously published domestic patent applications, for example, the publication number is CN112213735A, and the name is a laser point cloud denoising method for rainy and snowy weather. The method disclosed in the patent application converts the point cloud into a depth map and then judges the rain and snow noise points by the number of connections in the eight-connected domain. It has a positive effect on light rain and light snow; however, it is not effective for medium and heavy rain and snow, and more rain and snow noise points will be retained. In addition, the calculation takes a long time and the real-time performance is poor. Another example is the publication number CN116660934A, which is named a vehicle, rain / snow weather identification method, vehicle control method and system. Its solution can only identify rain and snow weather, but cannot solve the impact of rain and snow noise points on perception.

[0004] The existing technology is generally based on the neighboring point cloud search of the current point cloud, which requires a traversal search of the surrounding point clouds of each point, which has a large amount of calculation and poor real-time performance. Assuming a 32-line laser radar, it will generate about 57W points per second. If the surrounding point cloud is acquired and the density is calculated for each point, it will take a considerable time. Moreover, after calculating the neighboring point cloud, it is necessary to perform point count and neighbor distance calculation through a preset threshold. Since the noise distribution of medium and heavy rain and snow is completely different from that of light rain and snow, a single set of thresholds can usually only solve some noise points in some light rain and snow scenes, and the filtering effect is limited; in the face of medium and heavy rain and snow scenes, the noise filtering is poor, and if the filtering threshold is too strict, the surrounding actual environment point cloud may be mistakenly deleted, affecting the perception effect and causing missed detection.

[0005] In view of this, this patent application is proposed. Summary of the invention

[0006] The deep learning-based LiDAR target detection method in rainy and snowy weather conditions described in this application aims to solve the problems existing in the above-mentioned prior art and propose a denoising solution for accurate filtering in heavy rain and snow scenes, in order to improve the real-time performance of ground target detection and solve the technical difficulties of LiDAR on-vehicle deployment.

[0007] To achieve the above design objectives, the deep learning-based lidar target detection method in rainy and snowy weather includes the following implementation steps:

[0008] Step 1), data collection and training environment preparation;

[0009] Step 2), read the data and extract the original characteristics of the environment;

[0010] Extract the PCD of the point cloud in the rainy environment and obtain the point cloud features (x, y, z, i, diff) from it; where x is the X-axis coordinate value of the point cloud, y is the Y-axis coordinate value of the point cloud, z is the Z-axis coordinate value, i is the intensity value of the point cloud, and diff is the echo value of the point cloud;

[0011] For each frame of input point cloud, the average echo value diff_average is calculated and compared with the preset threshold T to confirm whether it is light rain and snow weather or moderate to heavy rain and snow weather; T is the average echo value obtained by different laser radars based on the proportion of rain and snow noise points in the actual point cloud;

[0012] The Euclidean distance, reflection intensity, and echo value of the input point cloud are converted into depth map, intensity map, and echo map respectively, and projected to the corresponding pixel position of the image;

[0013] Step 3), image processing;

[0014] Smoothly fill the blank values ​​in the depth map, intensity map and echo map;

[0015] Step 4), feature extraction;

[0016] The smoothed feature map is sent to the convolutional neural network for feature extraction, and the point cloud spatial sparse feature map is extracted through multi-layer downsampling modules and multi-layer upsampling modules;

[0017] According to the judgment result of step 2), if it is light rain and snow, jump to step 6) for execution; if it is moderate to heavy rain and snow, continue to execute the following step 5);

[0018] Step 5) Extraction of noise points in medium and heavy rain and snow point clouds;

[0019] The sparse distribution feature map of the point cloud space of the previous frame is fused with the current frame through a multi-head temporal attention mechanism. That is, by passing the time difference between the spatial information feature map of the point cloud of the previous frame and the two frames, the deformable attention mechanism is used to extract the rain and snow noise features in the sparse distribution map of the point cloud space of the previous frame.

[0020] Step 6) Remove rain and snow noise;

[0021] The sparse distribution map of the current frame point cloud is obtained, and the rain and snow noise results are predicted by the detection head, and the three-channel rain and snow noise result map of depth, intensity and echo is output;

[0022] Combine the point cloud height, intensity value, echo value and the model's prediction results to filter out noise points in rainy and snowy days;

[0023] Step 7) Input the filtered normal point cloud set into the centerpoint target detection network to realize the perception of surrounding obstacles in complex weather conditions.

[0024] Furthermore, the step 1) includes the following steps:

[0025] Install the LiDAR on the vehicle and collect data in rainy and snowy environments; convert the data collected by the LiDAR into frames of point cloud PCD and save them with time corresponding information; build a training environment and construct a training network architecture including pre-processing, single-frame network part, timing network part, post-processing and loss function code; prepare to extract point cloud data, conduct training, and output the model.

[0026] Further, in the step 2), each pixel position of the depth map, intensity map and echo map is derived from the angle conversion of the point cloud relative to the laser radar coordinate system; wherein the height H of the image resolution is determined by the laser radar beam, and the width W of the image resolution is determined by the maximum number of points of the horizontal scanning of the laser radar; the maximum angle range of the vertical angle of the laser radar determines the height direction α H The angle range of β in the width direction is determined according to the maximum angle range of the left and right angles of the laser radar. W The angle range of the image is set according to the number of laser radar beams and the maximum number of points in the horizontal scan. The angle value α of the current point cloud point in the height direction is calculated by the arc sine of the z value of the current point cloud and the distance d from the current point to the center of the radar. α accounts for the angle α in the height direction. H The product of the ratio and the total height is the row coordinate r of the current point in the image. The calculation formula is as follows: α = arcsin (z, d), r = α / α H *H; The angle value β of the current point cloud point in the width direction is determined based on the angle of the point cloud from the X axis of the laser radar coordinate system on the (X, Y) plane. β accounts for the angle β in the width direction.W The product of the ratio and the total width is the column coordinate c of the current point in the image. The calculation formula is as follows: β = arctan (y, x), c = β / β W …W.

[0027] Furthermore, the step 3) uses a Savitzky-Golay filter for smooth filling, sets the length m of the window M to 5, and retains more high-frequency information while focusing on the smoothing effect; at the same time, the left and right boundaries of the image are filled with reflect, and the blanks in the three-channel image input vector (3*H*W) are filtered and smoothed along the width direction, and the original values ​​are still maintained where there are values, thereby improving data accuracy without changing the signal trend; and finally, a feature map (3*H*W) of the three channels of depth, intensity and echo after smooth filling is obtained;

[0028] The smoothing function expression of the Savitzky-Golay filter is as follows:

[0029]

[0030] Among them, i is the index of the channel, j is the index in the height direction, k is the index in the width direction, and M n is the weight value within the filter window, and m is the total length of the window.

[0031] Further, in the step 4), the downsampling module adopts a combination of wavelet transform layer, convolution layer, Batchnorm layer, RELU nonlinear activation layer and Dropout layer; each downsampling doubles the number of channels and reduces the length and width of the feature map by one half; the upsampling module adopts a residual network to accept the features from the downsampling, and the specific processing layer adopts a combination of convolution layer, Batchnorm layer, RELU nonlinear activation layer, Dropout layer and inverse wavelet transform layer; each upsampling reduces the number of channels by one half and increases the length and width of the feature map by one half; through convolution, discrete wavelet transform downsampling and upsampling, all sub-bands are used as input after each transformation, and have a stronger ability to model spatial context and inter-sub-band dependencies; finally, the point cloud spatial distribution feature map (3*H*W) of the current frame is obtained;

[0032] Furthermore, in the step five), for the point cloud spatial distribution feature map (3*H*W) obtained in step four), a learnable bias parameter (N*2*H*W) and a learnable weight coefficient parameter (N*H*W) are configured for the feature map of each channel; wherein the learnable bias parameter includes the relative offset result of each pixel of the feature map of this frame to the N (x_pixel, y_pixel) surrounding the same pixel position of the previous frame; 2 represents the x_pixel pixel value offset in the pixel width direction and the y_pixel pixel value offset in the height direction.

[0033] Furthermore, the step six) predicts the rain and snow noise results based on a detection head represented by a 1x1 three-channel two-dimensional convolutional layer.

[0034] Furthermore, the step six) comprises the following steps:

[0035] 6.1) Set all the original point clouds of this frame as P0;

[0036] All ground points whose point cloud z values ​​are lower than the ground plane z_ground are first excluded and set as non-rain and snow noise ground points P1, leaving the remaining unclassified point cloud P2;

[0037] 6.2) Determine the point cloud category in the remaining unclassified point cloud P2. According to the prediction results of the three-channel rain and snow noise result map, process the results of the three channels in each pixel to predict the point cloud category in this pixel; among them, for each pixel's depth result d_result, intensity result i_result and echo result diff_result, determine whether the result of the point cloud in the remaining unclassified point cloud P2 is greater than the threshold δ, that is, the following expression:

[0038] d_result*i_result 2 …diff_result>δ

[0039] If the result of the point cloud in the remaining unclassified point cloud P2 is less than the threshold δ, these points are judged to be normal point clouds P3 excluded by the model; if the result of the point cloud in the remaining unclassified point cloud P2 is greater than the threshold δ, these points are judged to be candidate rain and snow noise points P4;

[0040] 6.3) Further distinguish the candidate rain and snow noise points in the candidate rain and snow noise point P4;

[0041] The points with intensity values ​​between 4 and 255 and echo values ​​between 0 and 3 cm are set as the normal point cloud P5 detected by mistake in the model, and the remaining part of P4 is set as the rain and snow noise point P6 after verification in this frame;

[0042] 6.4) Summarize and obtain a set of normal point clouds;

[0043] The normal point cloud set includes the ground point cloud P1, the normal point cloud P3 excluded by the model, and the normal point cloud P5 misdetected by the model.

[0044] In summary, the advantages and beneficial effects of this application are as follows:

[0045] 1. It can effectively filter out point cloud noise in light, medium and heavy rain and snow weather, and provide a better solution for point cloud applications in harsh environments;

[0046] 2. The use of deep neural networks can accelerate point cloud data calculations through hardware such as GPUs, significantly improving the operating speed;

[0047] 3. The noise points in the moderate to heavy rain and snow scenes can be accurately identified through the self-supervision algorithm, and there is no need to label the rain and snow point clouds in large quantities, so the calculation amount is small and the accuracy is high;

[0048] 4. This application can be trained through self-generated point cloud noise to further reduce the detection cost of ground targets in subsequent real-time scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The following specifically describes the implementation scheme in conjunction with the accompanying drawings;

[0050] Figure 1 This is a flow chart of the deep learning-based laser radar target detection method in rainy and snowy weather described in this application;

[0051] Figure 2 It is a schematic diagram of the coordinate transformation from point cloud to image;

[0052] Figure 3 This is a schematic diagram of the network for extracting rain and snow noise features;

[0053] Figure 4 It is a schematic diagram of the downsampling structure;

[0054] Figure 5 It is a schematic diagram of the upsampling structure;

[0055] Figure 6 This is a schematic diagram of the multi-head temporal attention mechanism; DETAILED DESCRIPTION

[0056] Embodiment 1, as Figures 1 to 6 As shown, the present application proposes a lidar target detection method in rainy and snowy weather based on deep learning. The method converts a frame of point cloud into a depth map, an intensity map and a dual echo map, and inputs the spatial information and temporal information of rain and snow noise into a neural network to extract noise in medium and heavy rain and snow scenes, predict the points caused by rain and snow particles, and delete this part of the noise from the point cloud to form a point cloud that does not include rain and snow noise.

[0057] Spatial information refers to the sparsity of target signals in real environments, while rain and snow noise points are non-sparse in space. Therefore, after converting rain and snow noise from the time domain to the frequency domain, it will have a larger bandwidth. The convolutional neural network (CNN) is used to extract rain and snow noise points with a larger bandwidth in the frequency domain, and the rain and snow point cloud is filtered out under the premise of maximizing the sparsity of the background real environment point cloud;

[0058] However, it is difficult to filter out all the noise points by relying on spatial information alone. On the basis of the above noise filtering based on spatial information, the acquisition of temporal information is added for medium and heavy rain and snow days to strengthen the sparse information of the current frame, and then the noise points of medium and heavy rain and snow days are filtered out based on the prediction. Since the noise points of rain and snow are scattered and random and easily affected by airflow, the noise points reflected by rain and snow particles are unpredictable. Reflected in the time series, the noise points that existed at the previous moment are most likely not in the same position at the current moment. In this regard, the spatial information obtained from the point cloud of the previous frame is retained, and then the elements in each feature map of the current frame are searched for elements in several positions around the element in the previous frame through a multi-head temporal attention mechanism for more accurate feature fusion.

[0059] Specifically, the process steps include:

[0060] Step 1) Data collection and training environment preparation

[0061] Install the LiDAR on the vehicle to collect data in rainy and snowy environments;

[0062] Convert the data collected by the LiDAR into point cloud PCD frames and save them with time correspondence information;

[0063] Build a training environment and construct a training network architecture including pre-processing, single-frame network part, temporal network part, post-processing and loss function code;

[0064] Prepare to extract point cloud data, perform training, and output the model.

[0065] Step 2), read the data and extract the original characteristics of the environment;

[0066] Extract the PCD of the point cloud in the rainy environment and obtain the point cloud features (x, y, z, i, diff) from it, where x is the X-axis coordinate value of the point cloud, y is the Y-axis coordinate value of the point cloud, z is the Z-axis coordinate value, i is the intensity value of the point cloud, and diff is the echo value of the point cloud;

[0067] The average echo value diff_average of the point cloud input in each frame is calculated, and the average echo value diff_average is compared with the preset threshold T to determine whether it is light rain and snow weather or moderate to heavy rain and snow weather;

[0068] T is the average echo value obtained by different lidars based on whether the proportion of rain and snow noise points in the actual point cloud is above 3 to 5%;

[0069] In this embodiment, if the number of rain and snow noise points is less than 3% of the current frame point cloud, it is considered as light rain and snow weather. On the contrary, if the number of rain and snow noise points is greater than 3%, it is considered as medium to heavy rain and snow weather.

[0070] The calculation formula for the average echo value diff_average is Where N is the number of points in the current frame;

[0071] If the average echo value diff_average is less than or equal to the threshold value T, the current frame is considered to be a point cloud of light rain and snow weather; if the average echo value diff_average is greater than the threshold value T, the current frame is considered to be a point cloud of medium to heavy rain and snow weather, and time series information needs to be added for filtering;

[0072] like Figure 2 As shown, the Euclidean distance, reflection intensity, and echo value of the input point cloud are converted into a depth map, an intensity map, and an echo map, respectively, and projected to the corresponding pixel position of the image;

[0073] The position of each pixel in the above three images comes from the angle conversion of the point cloud relative to the laser radar coordinate system; the height H of the image resolution is determined by the laser radar beam, and the width W of the image resolution is determined by the maximum number of points in the horizontal scanning of the laser radar;

[0074] The maximum angle range of the up and down angles of the laser radar determines the α in the height direction. H The angle range of β in the width direction is determined according to the maximum angle range of the left and right angles of the laser radar. W Angular range;

[0075] Set the image resolution based on the number of LiDAR beams and the maximum number of points in the horizontal scan;

[0076] Among them, the angle value α of the current point cloud point in the height direction is calculated by the arc sine of the z value of the current point cloud and the distance d value of the current point from the center of the radar;

[0077] α is the angle α in the height direction H The product of the ratio and the total height is the row coordinate r of the current point in the image. The calculation formula is as follows: α = arcsin (z, d), r = α / α H *H;

[0078] The angle value β of the current point cloud point in the width direction is determined according to the angle of the point cloud in the (X, Y) plane from the X axis of the laser radar coordinate system. β accounts for the angle β in the width direction. WThe product of the ratio and the total width is the column coordinate c of the current point in the image. The calculation formula is as follows: β = arctan (y, x), c = β / β W *W;

[0079] Step 3), image processing;

[0080] Smoothly fill the blank values ​​in the depth map, intensity map and echo map;

[0081] The Savitzky-Golay filter is used, where the length of the window M is set to 5, which retains more high-frequency information while focusing on the smoothing effect; at the same time, the left and right boundaries of the image are filled with reflect, and the blanks in the three-channel image input vector (3*H*W) are filtered and smoothed along the width direction. The original values ​​are still maintained in the places where there are values, and the data accuracy is improved without changing the signal trend; finally, the characteristic map of the depth, intensity and echo three channels after smooth filling (3*H*W) is obtained;

[0082] The smoothing function expression of the Savitzky-Golay filter is as follows:

[0083]

[0084] Among them, i is the index of the channel, j is the index in the height direction, k is the index in the width direction, and M n is the weight value in the filter window, and m is the total length of the window;

[0085] Step 4), feature extraction;

[0086] The smoothed feature map is sent to Figure 3 The convolutional neural network (CNN) shown in FIG. 1 is used for feature extraction, and a sparse feature map of the point cloud space is extracted through a multi-layer downsampling module and a multi-layer upsampling module;

[0087] Convolutional neural networks (CNNs) usually use pooling to expand the receptive field, which has the advantage of low computational complexity. However, pooling may lead to information loss, which has an adverse effect on subsequent operations such as feature extraction and analysis. Normal point clouds are sparse in space, while rain and snow noise will affect the sparsity of point clouds. After discrete wavelet transform, noise will have a larger bandwidth in the frequency domain, so it is more suitable for analyzing and extracting signals with larger bandwidth.

[0088] Therefore, this application adopts a network form that combines wavelet transform and convolutional neural network to extract the characteristics of non-sparse rain and snow noise, that is, wavelet transform plus CNN downsampling, and wavelet inverse transform combined with CNN upsampling, to better balance the receptive field size and computational efficiency of rain and snow noise.

[0089] Specifically, the downsampling module adopts a combination of wavelet transform layer, convolution layer, Batchnorm layer, RELU nonlinear activation layer and Dropout layer, such as Figure 4 As shown in the figure, each downsampling doubles the number of channels and halves the length and width of the feature map.

[0090] The upsampling module uses a residual network to accept features from downsampling. The specific processing layer uses a combination of convolutional layer, Batchnorm layer, RELU nonlinear activation layer, Dropout layer and inverse wavelet transform layer, such as Figure 5 As shown in the figure, each upsampling reduces the number of channels by half and doubles the length and width of the feature map.

[0091] Through the above series of convolution, discrete wavelet transform downsampling and upsampling, all sub-bands are used as input after each transformation, and it has a stronger ability to model spatial context and dependencies between sub-bands; finally, the point cloud spatial distribution feature map (3*H*W) of the current frame is obtained;

[0092] According to the judgment result of step 2), if it is light rain and snow, jump to step 6) for execution; if it is moderate to heavy rain and snow, continue to execute the following step 5);

[0093] Step 5) Extraction of noise points in medium and heavy rain and snow point clouds;

[0094] like Figure 6 As shown in the figure, the spatial sparse distribution feature map of the point cloud of the previous frame is fused with the current frame through a multi-head temporal attention mechanism, that is, by passing the spatial information feature map of the point cloud of the previous frame and the time difference between the two frames, the deformable attention mechanism is used to extract the rain and snow noise features in the spatial sparse distribution map of the point cloud of the previous frame;

[0095] For the point cloud spatial distribution feature map (3*H*W) obtained in step 4), configure a (N*2*H*W) learnable bias parameter and a (N*H*W) learnable weight coefficient parameter for the feature map of each channel; wherein the learnable bias parameter includes the relative offset result of each pixel of the feature map of this frame to the N (x_pixel, y_pixel) surrounding the same pixel position of the previous frame; 2 represents the x_pixel pixel value offset in the pixel width direction and the y_pixel pixel value offset in the height direction.

[0096] The learnable weight coefficient includes the specific contribution ratio of each pixel in the feature map of this frame to the N pixels around the same pixel in the previous frame, so as to enhance the sparsity of the features of the heavy rain and snow noise in the current frame through the network self-attention mechanism;

[0097] Step 6) Remove rain and snow noise;

[0098] The obtained sparse distribution map of the current frame point cloud is used to predict the rain and snow noise results through a detection head represented by a 1x1 three-channel two-dimensional convolution layer, and a three-channel rain and snow noise result map of depth, intensity and echo is output;

[0099] Combine the point cloud height, intensity value, echo value and the model's prediction results to filter out noise points in rainy and snowy days;

[0100] The steps include:

[0101] 6.1) Set all the original point clouds of this frame as P0 (original input points); exclude all ground points whose point cloud z values ​​are lower than the ground plane z_ground, set them as non-rain and snow noise points P1 (ground points), and leave the remaining unclassified point cloud P2 (i.e., P2 is the remaining unclassified points after excluding the ground points);

[0102] 6.2) Determine the point cloud category in the remaining unclassified point cloud P2. According to the prediction results of the three-channel rain and snow noise result map, process the results of the three channels in each pixel to predict the point cloud category in this pixel; among them, for each pixel's depth result d_result (cm), intensity result i_result and echo result diff_result (cm), determine whether the result of the point cloud in the remaining unclassified point cloud P2 is greater than the threshold δ, that is, the following expression:

[0103] d_result*i_result 2 *diff_result>δ

[0104] If the result of the point cloud in the remaining unclassified point cloud P2 is less than the threshold δ, these points are judged to be non-rain and snow noise points P3 (normal point cloud excluded by the model); if the result of the point cloud in the remaining unclassified point cloud P2 is greater than the threshold δ, these points are judged to be candidate rain and snow noise points P4 (candidate rain and snow noise points after excluding the ground);

[0105] 6.4) Further distinguish the candidate rain and snow noise points in the candidate rain and snow noise point P4;

[0106] Points with intensity values ​​between 4 and 255 and echo values ​​between 0 and 3 cm are set as non-rain and snow noise points P5 (i.e., false detections in the model, which are normal point clouds; the reason is that the rain and snow noise points have very low intensity and very high echo values ​​due to refraction and echo effects), and the rest of P4 is set as the rain and snow noise points P6 of this frame (the rain and snow noise points after verification);

[0107] 6.4) Summarize the normal point cloud collection

[0108] The normal point cloud set includes the ground point cloud P1, the normal point cloud P3 excluded by the model, and the normal point cloud P5 misdetected by the model;

[0109] Step 7) Input the filtered normal point cloud set into the centerpoint target detection network to realize the perception of surrounding obstacles in complex weather conditions.

[0110] The above contents and the embodiments given in the accompanying drawings are only preferred solutions for achieving the purpose of the present invention. Those skilled in the art can be inspired by this and directly deduce other alternative structures that conform to the design concept of the present invention. Other structural features obtained thereby should also belong to the scope of the solution described in the present invention.

Claims

1. A deep learning-based laser radar target detection method in rainy and snowy weather, characterized by: It includes the following process steps: Step 1), data collection and training environment preparation; Step 2), read the data and extract the original characteristics of the environment; Extract the PCD of the point cloud in the rainy environment and obtain the point cloud features (x, y, z, i, diff) from it; where x is the X-axis coordinate value of the point cloud, y is the Y-axis coordinate value of the point cloud, z is the Z-axis coordinate value, i is the intensity value of the point cloud, and diff is the echo value of the point cloud; For each frame of input point cloud, the average echo value diff_average is calculated and compared with the preset threshold T to confirm whether it is light rain and snow weather or moderate to heavy rain and snow weather; T is the average echo value obtained by different laser radars based on the proportion of rain and snow noise points in the actual point cloud; The Euclidean distance, reflection intensity, and echo value of the input point cloud are converted into depth map, intensity map, and echo map respectively, and projected to the corresponding pixel position of the image; Step 3), image processing; Smoothly fill the blank values ​​in the depth map, intensity map and echo map; Step 4), feature extraction; The smoothed feature map is sent to the convolutional neural network for feature extraction, and the point cloud spatial sparse feature map is extracted through multi-layer downsampling modules and multi-layer upsampling modules; According to the judgment result of step 2), if it is light rain and snow, jump to step 6) for execution; if it is moderate to heavy rain and snow, continue to execute the following step 5); Step 5) Extraction of noise points in medium and heavy rain and snow point clouds; The sparse distribution feature map of the point cloud space of the previous frame is fused with the current frame through a multi-head temporal attention mechanism. That is, by passing the time difference between the spatial information feature map of the point cloud of the previous frame and the two frames, the deformable attention mechanism is used to extract the rain and snow noise features in the sparse distribution map of the point cloud space of the previous frame. Step 6) Remove rain and snow noise; The sparse distribution map of the current frame point cloud is obtained, and the rain and snow noise results are predicted by the detection head, and the three-channel rain and snow noise result map of depth, intensity and echo is output; Combine the point cloud height, intensity value, echo value and the model's prediction results to filter out noise points in rainy and snowy days; Step 7) Input the filtered normal point cloud set into the centerpoint target detection network to realize the perception of surrounding obstacles in complex weather conditions.

2. The deep learning-based laser radar target detection method in rainy and snowy weather according to claim 1, characterized in that: The step 1) comprises the following steps, Install the LiDAR on the vehicle to collect data in rainy and snowy environments; Convert the data collected by the LiDAR into point cloud PCD frames and save them with time correspondence information; Build a training environment and construct a training network architecture including pre-processing, single-frame network part, temporal network part, post-processing and loss function code; Prepare to extract point cloud data, perform training, and output the model.

3. The deep learning-based laser radar target detection method in rainy and snowy weather according to claim 1, characterized in that: In the step 2), each pixel position of the depth map, intensity map and echo map is derived from the angle conversion of the point cloud relative to the laser radar coordinate system; wherein the height H of the image resolution is determined by the laser radar beam, and the width W of the image resolution is determined by the maximum number of points of the horizontal scan of the laser radar; The maximum angle range of the up and down angles of the laser radar is used to determine the α in the height direction. H The angle range of β in the width direction is determined according to the maximum angle range of the left and right angles of the laser radar. W Angular range; Set the image resolution based on the number of LiDAR beams and the maximum number of points in the horizontal scan; Among them, the angle value α of the current point cloud point in the height direction is calculated by the arc sine of the z value of the current point cloud and the distance d value of the current point from the center of the radar; α is the angle α in the height direction H The product of the ratio and the total height is the row coordinate r of the current point in the image. The calculation formula is as follows: α = arcsin (z, d), r = α / α H *H; The angle value β of the current point cloud point in the width direction is determined according to the angle of the point cloud in the (X, Y) plane from the X axis of the laser radar coordinate system. β accounts for the angle β in the width direction. W The product of the ratio and the total width is the column coordinate c of the current point in the image. The calculation formula is as follows: β = arctan (y, x), c = β / β W …W.

4. The method for detecting targets by laser radar in rainy and snowy weather based on deep learning according to claim 1, characterized in that: The step three) uses a Savitzky-Golay filter for smooth filling, sets the length m of the window M to 5, and retains more high-frequency information while focusing on the smoothing effect; at the same time, the left and right boundaries of the image are filled with reflect, and the blanks in the three-channel image input vector (3*H*W) are filtered and smoothed along the width direction, and the original values ​​are still maintained where there are values, thereby improving data accuracy without changing the signal trend; finally, a feature map (3*H*W) of the three channels of depth, intensity and echo after smooth filling is obtained; The smoothing function expression of the Savitzky-Golay filter is as follows: Among them, i is the index of the channel, j is the index in the height direction, k is the index in the width direction, and M n is the weight value within the filter window, and m is the total length of the window.

5. The method for detecting target by laser radar in rainy and snowy weather based on deep learning according to claim 1, characterized in that: In the step 4), the downsampling module adopts a combination of a wavelet transform layer, a convolution layer, a Batchnorm layer, a RELU nonlinear activation layer and a Dropout layer; each downsampling doubles the number of channels and reduces the length and width of the feature map by half; The upsampling module uses a residual network to accept features from downsampling. The specific processing layer uses a combination of convolutional layer, Batchnorm layer, RELU nonlinear activation layer, Dropout layer and inverse wavelet transform layer. Each upsampling reduces the number of channels by half and doubles the length and width of the feature map. Through convolution, discrete wavelet transform downsampling and upsampling, all sub-bands are used as input after each transformation, and it has a stronger ability to model spatial context and dependencies between sub-bands; finally, the point cloud spatial distribution feature map (3*H*W) of the current frame is obtained.

6. The deep learning-based laser radar target detection method in rainy and snowy weather according to claim 1, characterized in that: In the step 5), for the point cloud spatial distribution feature map (3*H*W) obtained in step 4), a learnable bias parameter (N*2*H*W) and a learnable weight coefficient parameter (N*H*W) are configured for the feature map of each channel; wherein the learnable bias parameter includes the relative offset result of each pixel of the feature map of this frame to the N (x_pixel, y_pixel) surrounding the same pixel position of the previous frame; 2 represents the x_pixel pixel value offset in the pixel width direction and the y_pixel pixel value offset in the height direction.

7. The deep learning-based laser radar target detection method in rainy and snowy weather according to claim 1, characterized in that: The step six) predicts the rain and snow noise results based on a detection head represented by a 1x1 three-channel two-dimensional convolutional layer.

8. The deep learning-based laser radar target detection method in rainy and snowy weather according to claim 1, characterized in that: The step six) comprises the following steps, 6.1) Set all the original point clouds of this frame as P0; All ground points whose point cloud z values ​​are lower than the ground plane z_ground are first excluded and set as non-rain and snow noise ground points P1, leaving the remaining unclassified point cloud P2; 6.2) Determine the point cloud category in the remaining unclassified point cloud P2. According to the prediction results of the three-channel rain and snow noise result map, process the results of the three channels in each pixel to predict the point cloud category in this pixel; among them, for each pixel's depth result d_result, intensity result i_result and echo result diff_result, determine whether the result of the point cloud in the remaining unclassified point cloud P2 is greater than the threshold δ, that is, the following expression: d_result…i_result 2 …diff_result>δ If the result of the point cloud in the remaining unclassified point cloud P2 is less than the threshold δ, these points are judged to be normal point clouds P3 excluded by the model; if the result of the point cloud in the remaining unclassified point cloud P2 is greater than the threshold δ, these points are judged to be candidate rain and snow noise points P4; 6.3) Further distinguish the candidate rain and snow noise points in the candidate rain and snow noise point P4; The points with intensity values ​​between 4 and 255 and echo values ​​between 0 and 3 cm are set as the normal point cloud P5 detected by mistake in the model, and the remaining part of P4 is set as the rain and snow noise point P6 after verification in this frame; 6.4) Summarize and obtain a set of normal point clouds; The normal point cloud set includes the ground point cloud P1, the normal point cloud P3 excluded by the model, and the normal point cloud P5 misdetected by the model.

Images