A radar camera fusion method based on autonomous driving in rainy weather

Abstract

The present invention provides a radar camera fusion method based on rainy day automatic driving, including: obtaining street sample camera data and performing raindrop simulation to form a training data set, combining the deep unfolding network with the proximal gradient descent algorithm to build a multi-stage iterative deraining network architecture, and using the training data set to train to obtain a deraining network model; obtaining the rainy day camera data of the street to be tested, inputting it into the deraining network model, and obtaining deraining image data; obtaining the corresponding radar point cloud data and preprocessing it to obtain effective point cloud data; building a radar camera fusion network architecture based on the centernet network and the bottleneck attention mechanism, and training to obtain a radar camera fusion network model; inputting the deraining image data and the corresponding effective point cloud data into the radar camera fusion model to obtain the target detection result. The present invention can improve the target detection accuracy in rainy day environments, so that the accuracy and robustness of the automatic driving perception ability are also improved.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and in particular to a radar camera fusion method based on autonomous driving in rainy weather. Background Art

[0002] As people's quality of life improves, their demands for daily travel are also increasing, so the development of autonomous driving technology is also changing with each passing day. Autonomous driving perception, as a safety guarantee for autonomous driving, is the valve for making correct autonomous driving decisions. However, the degree of environmental perception of single-modal data is limited, and multiple sensors are required for information fusion when performing target detection. The camera's image data brings rich semantics to the road, but it cannot provide depth information and cannot handle occlusion problems. Millimeter-wave radar data can provide scene depth through the high-penetration data of radar point clouds, which complements the camera's 2D environmental perception capabilities. How to integrate millimeter-wave radar and cameras for highly robust target detection is currently a major research hotspot.

[0003] In the existing detection methods of fusion of millimeter-wave radar point cloud data and camera, due to the redundancy of point cloud data, after filtering out the redundant point cloud and fusing it with the camera image data, the millimeter-wave radar point cloud information cannot be fully utilized, and the main detection standard is still based on the semantic information of the camera. At the same time, the real street scenes are complicated, and the existing related fusion research mainly focuses on street detection tasks under normal weather conditions. However, for bad weather, such as rainy days, when the semantic information of the camera cannot be trusted, related research is still relatively scarce, and street target detection tasks cannot be performed well.

[0004] Therefore, there is an urgent need for a radar-camera fusion method that can fully utilize the high spatial resolution information of millimeter-wave radar in harsh environments and minimize the impact of the environment on camera semantics when performing street target detection under rainy weather conditions. Summary of the invention

[0005] Based on this, it is necessary to provide a radar camera fusion method based on rainy day autonomous driving to address the above technical problems.

[0006] A radar camera fusion method based on rainy day automatic driving includes the following steps: obtaining street sample camera data, performing raindrop simulation on the sample camera data to form a training data set, building a multi-stage iterative deraining network architecture by combining a deep unfolding network with a proximal gradient descent algorithm, and using the training data set to train the deraining network architecture to obtain a deraining network model; obtaining street rainy day camera data to be tested, inputting the deraining network architecture into the deraining network model to obtain deraining image data; obtaining radar point cloud data corresponding to the street rainy day camera data to be tested and performing preprocessing to obtain valid point cloud data; building a radar camera fusion network architecture based on a CenterNet network and a bottleneck attention mechanism, and training to obtain a radar camera fusion network model; inputting the deraining image data and the corresponding valid point cloud data into the radar camera fusion model, performing two regression tests, and fusing the regression information of the two regression tests to obtain a target detection result.

[0007] In one of the embodiments, the step of obtaining street sample camera data and performing raindrop simulation on the sample camera data to form a training data set includes: obtaining street sample camera data, generating random noise for the sample camera data using Gaussian distribution, and simulating rainy day conditions using affine transformation to obtain a noisy data set of simulated rainy day streets; and making a one-to-one correspondence between the sample camera data and images in the noisy data set to form noisy and non-noise image pairs to obtain a training data set for training the deraining network architecture.

[0008] In one embodiment, the multi-stage iterative rain removal network architecture is constructed by combining the deep unfolding network with the proximal gradient descent algorithm, including: w g(w)+h(w), where g(w) and h(w) correspond to differentiable and non-differentiable functions respectively; the gradient descent process is calculated for the differentiable function g(w), that is, the gradient descent result for the variable w with a step length of t For a non-differentiable function h(w), the proximal operator prox of the variable z is calculated with a step size of t through the gradient descent result z th(·) (z); the output proximal operator prox th(·) (z) is used as the variable input and the non-differentiable function is repeatedly calculated iteratively until the minimum value of the optimization function is obtained.

[0009] In one embodiment, the deraining network model includes an iterative network of seven stages, and the iterative network of each stage includes a connected gradient descent module, an internal fusion module and an inter-segment fusion module to form an encoder-decoder network; wherein the gradient descent module is used to simulate the gradient descent process of the deraining stage; the internal fusion module is used to fuse the feature information within the stage; and the inter-segment fusion module is used to fuse the feature information of the current stage and the previous stage.

[0010] In one embodiment, the encoder-decoder network has pyramid features of three scale features inside, and the input features are transmitted to the channel attention mechanism module to obtain the global features of the current stage; the three scale features are transmitted to the inter-segment fusion module respectively to fuse the feature information of the current stage and the previous stage; and the global features of the last two scale features and the first scale feature are transmitted to the internal fusion module for intra-stage feature fusion.

[0011] In one of the embodiments, the internal fusion module includes: passing the encoding features of the current scale to the residual module, the residual module is used to extract the current encoding features through void convolution; fusing the output features with the acquired global features to obtain the fusion features of the features of each scale in the current stage.

[0012] In one of the embodiments, the radar point cloud data corresponding to the camera data of the street to be tested in rainy days is obtained and preprocessed to obtain valid point cloud data, including: obtaining the radar point cloud data corresponding to the camera data of the street to be tested in rainy days, and expanding the radar point cloud data using a point cloud expansion algorithm; traversing the expanded radar point cloud data to perform coordinate transformation, and filtering the point cloud according to the coordinate transformation result; traversing the filtered point cloud data, generating a cylinder for each point cloud according to a preset size, and moving the cylinder coordinates to the center of the point cloud to obtain the expanded point cloud data; using the centernet network to perform target detection on the expanded point cloud data, and generating a truncated cone for the detected target according to the detection frame, matching and filtering the spatial point cloud information, and outputting the valid point cloud data.

[0013] In one of the embodiments, the traversed and expanded radar point cloud data is subjected to coordinate transformation, and point cloud filtering is performed according to the coordinate transformation result, including: transferring the point cloud to the camera coordinate system through rigid body transformation through rotation and translation parameters composed of millimeter-wave radar extrinsic parameters, camera extrinsic parameters, and posture information of the sensor and the vehicle body; projecting the radar point cloud data to the image plane according to the camera intrinsic parameters, and transferring it to the pixel coordinate system through coordinate transformation; performing point cloud filtering according to a preset maximum point cloud distance, aligning the filtered point cloud image with the corresponding derained image data, filtering the point cloud projected outside the camera image pixel plane, and obtaining filtered point cloud data.

[0014] In one of the embodiments, the method uses a centernet network to perform target detection on point cloud data, generates a frustum for the detected target according to a detection box, matches and filters the spatial point cloud information, and outputs valid point cloud data, including: performing target detection on point cloud data through a centernet network to obtain a detected target, wherein the levels of hierarchical depth aggregation of the centernet network are set to 1, 1, 1, 2, 2 and 1 respectively; in the training phase and the test phase, a frustum is generated for the detected target using a detection box, wherein in the test phase, a depth compensation parameter is set; traversing the point cloud data, filtering the point cloud information outside the frustum, and outputting valid point cloud data.

[0015] In one of the embodiments, the derained image data and the corresponding valid point cloud data are input into the radar camera fusion model, and two regression tests are performed. The regression information of the two regression tests is fused to obtain the target detection result, including: performing target detection on the derained image data based on the centernet network to obtain the first regression information, and the first regression information includes depth, heat map, width, height, offset, angle and dimension information; generating heat map information according to the valid point cloud data, and the heat map information includes radar channel target RCS information, depth, speed and speed component; splicing the heat map information with the heat map of the derained image data, matching the spatial attention and channel attention through the bottleneck attention mechanism, and obtaining the output feature; performing a second regression on the output feature through the centernet network to obtain the second regression information, and the second regression information includes attributes, angle, depth and speed; adjusting the detection frame based on the first regression information and the second regression information, and obtaining the target detection result through the adjusted detection frame.

[0016] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: by acquiring street sample camera data and performing raindrop simulation on it to form a training data set, a multi-stage iterative rain removal network architecture is built by combining a deep unfolding network and a proximal gradient descent algorithm, a rain removal network model is obtained by training the training data set, and high-resolution semantic information of the camera is obtained through multi-stage iterative rain removal; rainy day camera data of the street to be tested is acquired, rain-removed image data is obtained through the rain removal network model, and rich camera semantic information of the street to be tested is obtained; radar point cloud data corresponding to the rainy day camera data of the street to be tested is acquired and pre-processed to obtain effective point cloud data, so as to achieve more effective and full utilization of radar point cloud information; based on the CenterNet network and the bottleneck attention mechanism, a radar camera fusion network architecture is built, and a radar camera fusion model is obtained by training, and two regression tests are performed on the input matching rain-removed image data and effective point cloud data, and the regression information of the two tests is fused to obtain the target detection result, which effectively combines the fully utilized radar point cloud information and the camera's high-resolution semantic information, improves the target detection accuracy in rainy environments, and improves the accuracy and robustness of the autonomous driving perception capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a radar camera fusion method based on rainy day autonomous driving in one embodiment;

[0018] Figure 2 It is a flowchart of a radar camera fusion method based on rainy day autonomous driving in one embodiment;

[0019] Figure 3 A schematic diagram of the structure of a rain removal network model in one embodiment;

[0020] Figure 4 A schematic diagram of street rain removal results in one embodiment;

[0021] Figure 5 A schematic diagram of the structure of a radar camera fusion model in one embodiment;

[0022] Figure 6 A schematic diagram of a 3D detection image result after fusing a millimeter-wave radar point cloud and a camera image in one embodiment;

[0023] Figure 7 Schematic diagram of target detection results from the BEV perspective in an embodiment. DETAILED DESCRIPTION

[0024] Before describing the specific embodiments of the present invention, the overall concept of the present invention is described as follows:

[0025] The present invention is mainly developed for the process of autonomous driving. Currently, target detection in rainy days during autonomous driving is affected by the environment and the target detection accuracy is low.

[0026] Therefore, the present invention proposes a radar camera fusion method based on rainy day autonomous driving, which forms a training data set by acquiring street sample camera data and simulating raindrops thereon, and builds a multi-stage iterative deraining network architecture by combining a deep unfolding network and a proximal gradient descent algorithm, and uses the training data set to train a deraining network model, and obtains high-resolution semantic information of the camera through multi-stage iterative deraining; acquires rainy day camera data of the street to be tested, obtains derained image data through the deraining network model, and obtains rich camera semantic information of the street to be tested; acquires radar point cloud data corresponding to the rainy day camera data of the street to be tested and pre-processes it to obtain effective point cloud data, so as to achieve more effective and full utilization of radar point cloud information; builds a radar camera fusion network architecture based on a centernet network and a bottleneck attention mechanism, trains to obtain a radar camera fusion model, performs two regression tests on its input-matched derained image data and effective point cloud data, and fuses the regression information of the two tests to obtain a target detection result, which effectively combines the fully utilized radar point cloud information and the camera's high-resolution semantic information, improves the target detection accuracy in a rainy day environment, and improves the accuracy and robustness of the autonomous driving perception capability.

[0027] After introducing the overall concept of the present invention, in order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail by specific implementation methods in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0028] In one embodiment, Figure 1-2 As shown, a radar camera fusion method based on rainy day autonomous driving is provided, comprising the following steps:

[0029] Step S101, obtain street sample camera data, simulate raindrops on the sample camera data to form a training data set, combine the deep unfolding network with the proximal gradient descent algorithm to build a multi-stage iterative rain removal network architecture, and use the training data set to train the rain removal network architecture to obtain a rain removal network model.

[0030] Specifically, sample camera data of the street is obtained, and artificial rain is added to the sample camera data through raindrop simulation, so as to obtain training data sets of rainy and non-rainy streets. A multi-stage iterative deraining network architecture is constructed by combining the deep unfolding network and the proximal gradient descent algorithm. The constructed deraining network architecture is trained with the training data set to obtain a deraining network model. Through multi-stage gradient descent, multiple iterative deraining operations can be performed to reduce the adverse effects of rainy days on target detection, effectively solve the problems of image semantic information occlusion and blur under rainy conditions, and improve the accuracy of target detection.

[0031] The steps for obtaining the training data set are as follows: obtain street sample camera data, use Gaussian distribution to generate random noise for the sample camera data, and use affine transformation to simulate rainy conditions to obtain a noisy data set for simulated rainy days on the street; make a one-to-one correspondence between the sample camera data and the images in the noisy data set to form a pair of noisy and non-noisy images to obtain a training data set for training the deraining network architecture.

[0032] Specifically, before the data fusion stage, deraining processing for camera images was introduced. Gaussian distribution was used to generate random noise for street sample camera data, and affine transformation was used to simulate rainy conditions to generate rain streaks of different lengths, thicknesses, and offset angles, thereby achieving artificial raining on street images and obtaining a noisy dataset of simulated rainy streets. The sample camera data were matched one-to-one with the noisy dataset to form noisy and non-noise image pairs, and a training dataset was obtained for training the deraining network architecture to obtain a deraining network model.

[0033] The steps of building an iterative rain removal network architecture include: w g(w)+h(w), where g(w) and h(w) correspond to differentiable and non-differentiable functions respectively; the gradient descent process is calculated for the differentiable function g(w), that is, the gradient descent result for the variable w with a step length of t For a non-differentiable function h(w), the proximal operator prox of the variable z is calculated with a step size of t through the gradient descent result z th(·) (z); the output proximal operator prox th(·) (z) is used as the variable input and the non-differentiable function is repeatedly calculated iteratively until the minimum value of the optimization function is obtained.

[0034] Specifically, in order to better build the deraining network architecture, the deraining process is regarded as an optimization function, including two parts: a differentiable function and a non-differentiable function. The gradient descent process of the differentiable function is calculated by the proximal gradient descent algorithm to obtain the gradient descent result; for the non-differentiable function, the proximal operator is calculated based on the gradient descent result, and the proximal operator is used as a variable input. The calculation of the non-differentiable function is iterated repeatedly until the minimum value of the optimization function is obtained. The minimum value of the optimization function corresponds to the best deraining effect. The deraining network architecture is as follows: Figure 3 shown.

[0035] The deraining network model includes an iterative network of seven stages, and the iterative network of each stage includes a connected gradient descent module, an internal fusion module and an inter-segment fusion module to form an encoder-decoder network; the gradient descent module is used to simulate the gradient descent process of the deraining stage; the internal fusion module is used to fuse the feature information within the stage; and the inter-segment fusion module is used to fuse the feature information of the current stage and the previous stage.

[0036] Specifically, the trained deraining network model includes an iterative network with seven stages. Through multi-stage iterative deraining, the problems of reduced camera resolution and insufficient semantics under harsh rainy conditions are effectively improved. The iterative network of each stage includes interconnected gradient descent modules, internal fusion modules and inter-segment fusion modules. The three modules together constitute an encoder-decoder network. Among them, the gradient descent module is used to simulate the gradient descent process of the deraining stage; the internal fusion module and the inter-segment fusion module together constitute the proximal mapping, which are used to perform feature information within the stage and feature information of the stage parts, respectively, to avoid feature disappearance during the network iteration process; in each stage, the fusion of information within the stage and the fusion of information between segments are introduced to avoid gradient disappearance or gradient explosion while improving the interactive ability of information in each stage.

[0037] Among them, the encoder-decoder network includes pyramid features of three scale features. The input features are transmitted to the channel attention module to obtain the global features of the current stage; the three scale features are transmitted to the inter-segment fusion module respectively to fuse the feature information of the current stage and the previous stage; and the last two scale features and the global features of the first scale feature are passed to the internal fusion module for intra-stage feature fusion.

[0038] Specifically, the encoder-decoder network is composed of pyramid features of three scale features. The input features are first transmitted to the channel attention mechanism module to obtain the global features of the current stage τ Each scale feature will be processed by the inter-segment fusion module to fuse the feature information of the current stage and the previous stage; at the same time, the last two scale features will be fused with the global features of the first scale feature through the internal fusion module to achieve intra-stage feature fusion, so as to extract feature information of each stage, avoid omission of feature information during network iteration, and improve the interactive ability of information at each stage.

[0039] Among them, the working steps of the internal fusion module include: transmitting the encoding features of the current scale to the residual module, and the residual module is used to extract the current encoding features through void convolution; fusing the output features with the acquired global features to obtain the fusion features of the features of each scale in the current stage.

[0040] Specifically, when the internal feature information of the stage is fused, the encoding feature of the current scale is transmitted to the residual module used to extract the current encoding feature. The module uses a dilated convolution to enhance the receptive field. The output feature is combined with the obtained global feature. They are jointly passed into the feature fusion module used to fuse the features of each scale in the current stage, so as to achieve the fusion of features of each scale in the stage and obtain the fused features.

[0041] In one embodiment, the rainy day image y is passed into the gradient descent module as the initial W0 input feature to obtain the gradient descent result Where t represents the gradient descent step size, It means to derive the differentiable part of W0, which is realized by simulating the degradation matrix using the residual block.

[0042] After gradient descent, the data Z1 is passed into the proximal mapping module composed of the encoder-decoder network to generate the proximal operator prox th(·) (Z1) completes the first stage of processing, where the calculation of the proximal operator depends on the cooperation of the internal fusion module and the inter-segment fusion module. The encoder-decoder network of each stage is composed of three pyramid networks with different scale features. The incoming features will first pass through the channel attention mechanism module to obtain the global features of the current stage τ Each scale feature will be passed to the inter-segment fusion module for processing, and the last two scale features will be passed to the internal fusion module with the global feature of the first scale for intra-stage feature fusion.

[0043] Proximal operator prox th(·) (Z1) is used as the network input W1, and the above gradient descent, feature fusion and proximal operator input are repeated until the seven-stage processing is completed, and the deraining result W7 is finally output.

[0044] Step S102, obtaining camera data of the street to be tested in rainy days, and inputting it into the rain removal network model to obtain rain removal image data.

[0045] Specifically, the rain removal network architecture is trained with the training data set until the network converges to obtain the rain removal network model, and the rainy camera data of the street to be tested is obtained and input into the rain removal network model for rain removal processing to obtain the rain removal image data. The obtained rain removal image is as follows: Figure 4 shown.

[0046] Step S103, obtaining radar point cloud data corresponding to the camera data of the street to be tested in rainy days and preprocessing it to obtain valid point cloud data.

[0047] Specifically, the millimeter-wave radar is used to obtain radar point cloud data corresponding to the camera data of the street under test in rainy weather, and the radar point cloud data is preprocessed, including point cloud expansion, point cloud dilation, point cloud projection, point cloud filtering and other processing operations, so as to make full use of the radar point cloud information and obtain effective point cloud data, so as to improve the accuracy of information such as depth and speed information obtained based on the point cloud data.

[0048] Among them, step S103 includes: obtaining radar point cloud data corresponding to the camera data of the street to be tested in rainy days, and expanding the radar point cloud data using a point cloud expansion algorithm; traversing the expanded radar point cloud data to perform coordinate transformation, and filtering the point cloud according to the coordinate transformation result; traversing the filtered point cloud data, generating a cylinder for each point cloud according to a preset size, and moving the cylinder coordinates to the center of the point cloud to obtain the expanded point cloud data; using the centernet network to perform target detection on the expanded point cloud data, and generating a truncated cone for the detected target according to the detection frame, matching and filtering the spatial point cloud information, and outputting valid point cloud data.

[0049] Specifically, radar point cloud data corresponding to the camera data of the street to be tested in rainy days is obtained, and the point cloud expansion algorithm is used to expand the radar point cloud data in the width, height and time dimensions of the image coordinate system. In the time dimension, multi-frame millimeter-wave radar point cloud images are used for data enhancement to achieve the expansion of the time dimension. In the width and height dimensions, the width and height of the target are expanded on the pixel plane. The radar data is enhanced by point cloud expansion, and expansion in space and time can make more comprehensive use of point cloud information and reduce subsequent filtering of valid point clouds.

[0050] The expanded radar point cloud data is traversed to perform coordinate transformation, and then the point cloud information that does not meet the requirements is filtered out; the filtered point cloud data is traversed, and a cylinder is generated for each point cloud according to the preset size. The length, width and height of the preset cylinder size are 0.2, 0.2 and 1 respectively, and the cylinder coordinates are moved to the center of the point cloud to obtain the expanded point cloud data and realize point cloud expansion; the centernet network is used to perform target detection on the expanded point cloud image, and the detection box is used to generate 3DROI frustum for the detected targets one by one, and the spatial point cloud information is filtered, and the valid point cloud data obtained after processing is output, so as to make full use of the radar point cloud data and improve the accuracy of target detection.

[0051] Among them, the steps of point cloud filtering are: transfer the rigid body transformation to the camera coordinate system through the rotation and translation parameters composed of millimeter-wave radar extrinsic parameters, camera extrinsic parameters, and the posture information of the sensor and the vehicle body; project the radar point cloud data to the image plane according to the camera intrinsic parameters, and transfer it to the pixel coordinate system through coordinate transformation; filter the point cloud according to the preset maximum point cloud distance, align the filtered point cloud image with the corresponding derained image data, filter the point cloud projected outside the camera image pixel plane, and obtain the filtered point cloud data.

[0052] Specifically, when performing point cloud filtering, the rotation and translation parameters composed of the millimeter-wave radar extrinsic parameters, the camera extrinsic parameters, and the posture information of the sensor and the vehicle body are used to perform rigid body transformation and transfer to the camera coordinate system; the information is projected to the image plane according to the camera intrinsic parameters, and then transferred to the pixel coordinate system through coordinate transformation; the preset maximum point cloud distance is 60cm, and the point cloud with a depth less than 60cm is filtered; the transferred point cloud image is aligned with the corresponding derained image data, and the point cloud projected outside the camera image pixel plane is filtered to obtain the filtered point cloud data, so as to obtain valid point cloud data.

[0053] Among them, the steps of outputting valid point cloud data are: performing target detection on point cloud data through the CenterNet network to obtain detected targets, and the levels of hierarchical depth aggregation of the CenterNet network are set to 1, 1, 1, 2, 2 and 1 respectively; in the training stage and the test stage, a detection box is used to generate a frustum for the detected target, wherein in the test stage, a depth compensation parameter is set; traversing the point cloud data, filtering the point cloud information outside the frustum, and outputting valid point cloud data.

[0054] Specifically, the first target detection is performed using the centernet network with DLA-34 as the main architecture, in which the levels of hierarchical depth aggregation of DLAseg are set to 1, 1, 1, 2, 2, 1 respectively; according to the training and testing phases, the ROI frustum is generated using the image detection box, wherein in the testing phase, since the depth of the target is determined only based on the features of the image, the estimated depth deviates slightly, so the compensation parameter δ can be set to compensate for the depth, thereby reducing the depth estimation error; the point cloud data is traversed, the redundant point cloud information outside the frustum is filtered out one by one, and the processed valid point cloud data is output, so that the radar point cloud information can be fully utilized, which helps to improve the target detection accuracy.

[0055] Step S104: Based on the CenterNet network and the bottleneck attention mechanism, a radar camera fusion network architecture is built, and a radar camera fusion network model is obtained by training.

[0056] Specifically, the radar camera fusion network architecture is built by combining the CenterNet network and the bottleneck attention mechanism. The CenterNet network can be used for street object perception and recognition, and the bottleneck attention mechanism is used for interactive learning in space and channels when splicing image features with point cloud features. The radar camera fusion network architecture is trained through a sample data set, and the following is obtained: Figure 5 The radar camera fusion network model shown can perform multimodal fusion processing on the derained image data and the corresponding radar point cloud data, thereby realizing the effective combination of camera information and radar information.

[0057] In step S105, the derained image data and the corresponding valid point cloud data are input into the radar camera fusion model, two regression tests are performed, and the regression information of the two regression tests is fused to obtain the target detection result.

[0058] Specifically, the rain-free image data of the street to be tested and the corresponding valid point cloud data are input into the radar camera fusion model, and the image is subjected to two regression tests through the model, and the regression information obtained from the two regression tests is fused, such as Figure 6 As shown in the figure, the 3D target detection result on the image after fusing the millimeter wave radar point cloud and the camera image is output. The final target detection result is as follows Figure 7 As shown in the figure, the deraining network model provides rich camera semantic information, which, combined with the fully utilized point cloud data, improves the accuracy of target detection in rainy days, and further improves the accuracy and robustness of autonomous driving perception capabilities.

[0059] Among them, step S105 includes: performing target detection on the derained image data based on the centernet network to obtain the first regression information, the first regression information includes depth, heat map, width, height, offset, angle and dimension information; generating heat map information according to the effective point cloud data, the heat map information includes radar channel target RCS information, depth, speed and speed component; splicing the heat map information with the heat map of the derained image data, matching the spatial attention and channel attention through the bottleneck attention mechanism, and obtaining the output feature; performing secondary regression on the output feature through the centernet network to obtain secondary regression information, the secondary regression information includes attributes, angle, depth and speed; adjusting the detection frame in combination with the first regression information and the secondary regression information, and obtaining the target detection result through the adjusted detection frame.

[0060] Specifically, 3D target detection is performed on the derained image data based on the CenterNet network to achieve the first regression and obtain multi-attribute regression data including depth, heat map, width, height, offset, angle, and dimension information; at the same time, effective point cloud data is used to generate heat map information, including radar channel target RCS information ρ, depth d, velocity V and its velocity component v on the x and y axes. x ,v y , a total of five types of heat maps, for each detected object, the heat map is expressed as generating a two-dimensional box with the same center and the same length and width as its own two-dimensional bounding box, and the size of the ratio is controlled by the parameter α. The specific heat value is expressed as follows:

[0061]

[0062] Among them, M i is the normalization factor for each channel, f i is the eigenvalue of the corresponding channel, is the center coordinate of the j-th 2D bounding box that matches it, w j ,h j are the width and height of the j-th 2D bounding box.

[0063] At the same time, a heat map is generated based on the derained image data, and the heat map information of the radar is combined to splice the two. Through the calm attention mechanism, the spatial attention and channel attention are matched to obtain the output features.

[0064] The output features are transmitted to the centernet network for secondary regression to obtain regression information about attributes, angles, depths, and speeds. The 3D detection frame is adjusted based on the two regression information, and the target's movement information is displayed in the point cloud BEV (Bird's Eye View) view to obtain the final target detection result.

[0065] In this embodiment, a training data set is formed by acquiring street sample camera data and performing raindrop simulation on it, a multi-stage iterative rain removal network architecture is built by combining a deep unfolding network and a proximal gradient descent algorithm, a rain removal network model is obtained by training the training data set, and high-resolution semantic information of the camera is obtained through multi-stage iterative rain removal; rainy day camera data of the street to be tested is acquired, rain removal image data is obtained through the rain removal network model, and rich camera semantic information of the street to be tested is obtained; radar point cloud data corresponding to the rainy day camera data of the street to be tested is acquired and pre-processed to obtain effective point cloud data, so as to achieve more effective and full utilization of radar point cloud information; based on the CenterNet network and the bottleneck attention mechanism, a radar camera fusion network architecture is built, and a radar camera fusion model is obtained by training, and two regression tests are performed on its input matching rain removal image data and effective point cloud data, and the regression information of the two tests is fused to obtain the target detection result, which effectively combines the fully utilized radar point cloud information and the camera high-resolution semantic information, improves the target detection accuracy in rainy environments, and improves the accuracy and robustness of the autonomous driving perception capability.

[0066] Obviously, those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a computer storage medium (ROM / RAM, magnetic disk, optical disk) and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Therefore, the present invention is not limited to any specific combination of hardware and software.

[0067] The above contents are further detailed descriptions of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A radar camera fusion method based on rainy day autonomous driving, characterized in that: The following steps are involved: Acquire street sample camera data, perform raindrop simulation on the sample camera data, and form a training data set, including: acquire street sample camera data, generate random noise on the sample camera data using Gaussian distribution, and simulate rainy day conditions using affine transformation to obtain a noisy data set of simulated rainy days on the street; make a one-to-one correspondence between the sample camera data and images in the noisy data set to form noisy and non-noise image pairs, and obtain a training data set for training a rain removal network architecture; Combining the deep unfolding network with the proximal gradient descent algorithm to build a multi-stage iterative rain removal network architecture, including: w g(w)+h(w), where g(w) and h(w) correspond to differentiable and non-differentiable functions respectively; the gradient descent process is calculated for the differentiable function g(w), that is, the gradient descent result for the variable w with a step length of t For a non-differentiable function h(w), the proximal operator prox of the variable z is calculated with a step size of t through the gradient descent result z th(·) (z); the output proximal operator prox th(·) (z) as a variable input, repeatedly iteratively calculating the non-differentiable function until the minimum value of the optimization function is obtained; and using the training data set to train the deraining network architecture to obtain a deraining network model; Obtaining rainy day camera data of the street to be tested, and inputting it into the rain removal network model to obtain rain removal image data; Obtain radar point cloud data corresponding to the camera data of the street to be tested in rainy days and perform preprocessing to obtain valid point cloud data; Based on the centemet network and bottleneck attention mechanism, a radar camera fusion network architecture is built, and the radar camera fusion network model is trained; The derained image data and the corresponding valid point cloud data are input into the radar camera fusion model, and two regression tests are performed. The regression information of the two regression tests is fused to obtain the target detection result.

2. The radar camera fusion method based on rainy day autonomous driving according to claim 1, characterized in that: The deraining network model includes an iterative network of seven stages, and the iterative network of each stage includes a connected gradient descent module, an internal fusion module and an inter-segment fusion module to form an encoder-decoder network; wherein the gradient descent module is used to simulate the gradient descent process of the deraining stage; the internal fusion module is used to fuse the feature information within the stage; and the inter-segment fusion module is used to fuse the feature information of the current stage and the previous stage.

3. The radar camera fusion method based on rainy day automatic driving according to claim 2 is characterized in that: The encoder-decoder network has pyramid features of three scale features inside, and the input features are transmitted to the channel attention mechanism module to obtain the global features of the current stage; the three scale features are transmitted to the inter-segment fusion module respectively to fuse the feature information of the current stage and the previous stage; and the global features of the last two scale features and the first scale feature are transmitted to the internal fusion module for intra-stage feature fusion.

4. The radar camera fusion method based on rainy day autonomous driving according to claim 3 is characterized in that: The internal fusion module includes: The encoding features of the current scale are passed to the residual module, and the residual module is used to extract the current encoding features through the dilated convolution; The output features are fused with the acquired global features to obtain the fused features of the features at each scale in the current stage.

5. The radar camera fusion method based on rainy day autonomous driving according to claim 1, characterized in that: The step of obtaining radar point cloud data corresponding to the camera data of the street to be tested in rainy weather and performing preprocessing to obtain valid point cloud data includes: Acquire radar point cloud data corresponding to the camera data of the street under test in rainy weather, and expand the radar point cloud data using a point cloud expansion algorithm; Traverse the expanded radar point cloud data to perform coordinate transformation, and perform point cloud filtering based on the coordinate transformation results; Traverse the filtered point cloud data, generate a cylinder for each point cloud according to the preset size, and move the cylinder coordinates to the center of the point cloud to obtain the expanded point cloud data; The centernet network is used to perform target detection on the expanded point cloud data, and a frustum is generated for the detected target according to the detection frame, the spatial point cloud information is matched and filtered, and the valid point cloud data is output.

6. The radar camera fusion method based on rainy day autonomous driving according to claim 5 is characterized in that: The traversed expanded radar point cloud data is subjected to coordinate transformation, and point cloud filtering is performed according to the coordinate transformation result, including: The rotation and translation parameters composed of millimeter-wave radar external parameters, camera external parameters, and the posture information of the sensor and the vehicle body are used to perform rigid body transformation and transfer to the camera coordinate system; Projecting the radar point cloud data onto an image plane according to camera intrinsic parameters, and transferring it to a pixel coordinate system through coordinate transformation; Point cloud filtering is performed according to a preset maximum point cloud distance, and the filtered point cloud image is aligned with the corresponding derained image data, and the point cloud projected outside the camera image pixel plane is filtered to obtain filtered point cloud data.

7. The radar camera fusion method based on rainy day autonomous driving according to claim 5 is characterized in that: The centernet network is used to detect the target on the point cloud data, and a frustum is generated for the detected target according to the detection frame, and the spatial point cloud information is matched and filtered to output the valid point cloud data, including: Performing target detection on the point cloud data through a CenterNet network to obtain detected targets, wherein the levels of hierarchical depth aggregation of the CenterNet network are set to 1, 1, 1, 2, 2, and 1 respectively; In the training stage and the testing stage, a detection frame is used to generate a frustum for the detected target, wherein in the testing stage, a depth compensation parameter is set; The point cloud data is traversed, the point cloud information outside the frustum is filtered, and valid point cloud data is outputted.

8. The radar camera fusion method based on rainy day autonomous driving according to claim 1, characterized in that: The derained image data and the corresponding valid point cloud data are input into the radar camera fusion model, two regression tests are performed, and the regression information of the two regression tests is fused to obtain the target detection result, including: Performing target detection on the derained image data based on the CenterNet network to obtain first regression information, wherein the first regression information includes depth, heat map, width, height, offset, angle and dimension information; Generate heat map information according to the valid point cloud data, wherein the heat map information includes radar channel target RCS information, depth, speed and speed component; Splicing the heat map information with the heat map of the derained image data, matching the spatial attention and the channel attention through the bottleneck attention mechanism, and obtaining the output features; Performing secondary regression on the output features through a centernet network to obtain secondary regression information, wherein the secondary regression information includes attributes, angles, depths, and speeds; The detection frame is adjusted based on the first regression information and the second regression information, and the target detection result is obtained through the adjusted detection frame.

Images