Radar feature generation method and device, equipment and storage medium

Abstract

The invention discloses a radar feature generation method, device and equipment and a storage medium, and relates to the technical field of automatic driving, and the radar feature generation method comprises the steps: carrying out the feature extraction of radar point cloud data, and obtaining a target radar disturbance feature; determining target image disturbance characteristics according to the current image data and a thermodynamic diagram guide calculation strategy; and determining target dense radar features according to the target radar disturbance features and the target image disturbance features. Under the condition that millimeter-wave radar point cloud data is sparse, a large kernel convolution technology is adopted to capture a long-distance dependency relationship, convolution kernel parameters are optimized through Gaussian noise disturbance, an overfitting problem is avoided, and in the dense feature generation process, thermal mask guidance and a multi-modal fusion technology are utilized to generate dense features. A high-quality dense millimeter-wave radar feature map is generated, the perception performance is improved, and the millimeter-wave radar feature extraction efficiency is further improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to radar feature generation methods, devices, equipment and storage media. Background Art

[0002] Visual feature extraction tasks play an important role in autonomous driving systems and are the basis for downstream tasks such as detection, tracking, decision-making, and planning. In recent years, thanks to the low cost of camera sensors, 3D perception technology based on camera multi-view has developed rapidly. However, camera data lacks depth information and 3D geometric clues, and cannot achieve high-precision and robust feature representation of targets in 3D space. Therefore, lidar sensors are integrated into the system to fuse the two modal data of point cloud and image to achieve more accurate perception feature extraction. However, the performance of lidar will degrade in adverse weather conditions, including fog, rain, and snow, and its wide application is limited due to its high cost. As an alternative, millimeter-wave radar has received increasing attention due to its robust performance in adverse conditions and its ability to provide 3D spatial and velocity information. At the same time, it is cheaper than lidar, so combining multi-view cameras with economical millimeter-wave radar sensors to achieve more comprehensive and reliable multimodal visual perception is a feasible and effective solution. Therefore, how to efficiently extract millimeter-wave radar features and use 2D image features to densify millimeter-wave radar feature generation is a problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of this application is to provide a radar feature generation method, device, equipment and storage medium, aiming to solve the technical problem of low efficiency of existing methods for extracting millimeter-wave radar features.

[0004] To achieve the above purpose, the present application proposes a radar feature generation method, which includes:

[0005] Extract features from radar point cloud data to obtain target radar disturbance features;

[0006] Determine the target image perturbation characteristics based on the current image data and the heat map guided calculation strategy;

[0007] A target dense radar feature is determined according to the target radar disturbance feature and the target image disturbance feature.

[0008] In one embodiment, the step of extracting features from radar point cloud data to obtain target radar disturbance features includes:

[0009] Convert the format of radar point cloud data to obtain radar voxel data;

[0010] Inputting the radar voxel data into a target convolution encoder to obtain an initial radar disturbance feature;

[0011] Performing view conversion on the initial radar disturbance feature to obtain a target radar disturbance feature.

[0012] In one embodiment, the step of inputting the radar voxel data into a target convolution encoder to obtain an initial radar disturbance feature includes:

[0013] Inputting the radar voxel data into a target convolution encoder for feature processing to obtain target radar sparse features;

[0014] An initial radar disturbance feature is obtained according to the target radar sparse feature and Gaussian distribution noise.

[0015] In one embodiment, the step of obtaining the initial radar disturbance feature according to the target radar sparse feature and Gaussian distribution noise includes:

[0016] Determine the noise weight matrix according to the Gaussian distribution noise and the initial convolution kernel weight;

[0017] Performing model training according to the noise weight matrix to obtain a target noise feature model;

[0018] An initial radar disturbance feature is determined according to the target radar sparse feature and the target noise feature model.

[0019] In one embodiment, the step of determining the target image disturbance feature according to the current image data and the heat map guided calculation strategy includes:

[0020] Perform feature extraction and downsampling on the current image data according to the preset residual network to obtain the initial image features;

[0021] Determining thermal map image features according to the thermal map guided calculation strategy, the radar point cloud data, and the initial image features;

[0022] Performing view conversion on the thermal map image features to obtain target image perturbation features.

[0023] In one embodiment, the step of determining the thermal map image features according to the thermal map guided calculation strategy, the radar point cloud data and the initial image features comprises:

[0024] Determine the current generated thermal map based on the thermal map guided calculation strategy and radar point cloud data;

[0025] Determine the current decoding heat map according to the preset image decoding model and the image bird's-eye view features;

[0026] Perform loss calculation according to the currently generated thermogram and the currently decoded thermogram to obtain a target image thermogram;

[0027] A thermal map image feature is obtained according to the target image thermal map and the initial image feature.

[0028] In one embodiment, the step of determining the target dense radar feature according to the target radar disturbance feature and the target image disturbance feature includes:

[0029] Performing feature splicing according to the target radar disturbance feature and the target image disturbance feature to obtain a target splicing feature;

[0030] The target dense radar feature is determined according to the target splicing feature, a preset self-attention strategy and the number of feature channels.

[0031] In addition, to achieve the above purpose, the present application also proposes a radar feature generation device, the radar feature generation device comprising:

[0032] A processing module is used to extract features from radar point cloud data to obtain target radar disturbance features;

[0033] The processing module is further used to determine the target image disturbance characteristics according to the current image data and the heat map guided calculation strategy;

[0034] A generating module is used to determine a target dense radar feature according to the target radar disturbance feature and the target image disturbance feature.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a radar feature generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the radar feature generation method described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the radar feature generation method described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the radar feature generation method described above are implemented.

[0038] This application obtains the target radar disturbance feature by extracting features from radar point cloud data; determines the target image disturbance feature based on the current image data and the heat map guidance calculation strategy; and determines the target dense radar feature based on the target radar disturbance feature and the target image disturbance feature. In the case of sparse millimeter-wave radar point cloud data, the large kernel convolution technology is used to capture long-distance dependencies, and the convolution kernel parameters are optimized through Gaussian noise perturbation to avoid overfitting problems. In the process of generating dense features, the thermal mask guidance and multimodal fusion technology are used to generate high-quality dense millimeter-wave radar feature maps, improve perception performance, and thus improve the efficiency of extracting millimeter-wave radar features. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A schematic diagram of a flow chart provided for Embodiment 1 of the radar signature generation method of the present application;

[0042] Figure 2 A schematic diagram of a heat map generation method provided in Example 1 of the radar feature generation method of the present application;

[0043] Figure 3 A schematic diagram of a flow chart provided for Embodiment 2 of the radar signature generation method of the present application;

[0044] Figure 4 A schematic diagram of a brief flow chart of a radar feature generation method provided in Example 1 of the present application;

[0045] Figure 5 This is a schematic diagram of the module structure of the radar feature generation device according to an embodiment of the present application;

[0046] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the radar feature generation method in the embodiment of the present application.

[0047] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0049] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0050] The main solution of the embodiment of the present application is: extracting features from radar point cloud data to obtain target radar disturbance features; determining target image disturbance features based on current image data and heat map guided calculation strategy; determining target dense radar features based on the target radar disturbance features and the target image disturbance features.

[0051] Visual feature extraction tasks play an important role in autonomous driving systems and are the basis for downstream tasks such as detection, tracking, decision-making, and planning. In recent years, thanks to the low cost of camera sensors, 3D perception technology based on camera multi-view has developed rapidly. However, camera data lacks depth information and 3D geometric clues, and cannot achieve high-precision and robust feature representation of targets in 3D space. Therefore, lidar sensors are integrated into the system to fuse the two modal data of point cloud and image to achieve more accurate perception feature extraction. However, the performance of lidar will degrade in adverse weather conditions, including fog, rain, and snow, and its wide application is limited due to its high cost. As an alternative, millimeter-wave radar has received increasing attention due to its robust performance in adverse conditions and its ability to provide 3D spatial and velocity information. At the same time, it is cheaper than lidar, so combining multi-view cameras with economical millimeter-wave radar sensors to achieve more comprehensive and reliable multimodal visual perception is a feasible and effective solution. Therefore, how to efficiently extract millimeter-wave radar features and use 2D image features to densify millimeter-wave radar feature generation is a problem that needs to be solved urgently.

[0052] This application obtains the target radar disturbance feature by extracting features from radar point cloud data; determines the target image disturbance feature based on the current image data and the heat map guidance calculation strategy; and determines the target dense radar feature based on the target radar disturbance feature and the target image disturbance feature. In the case of sparse millimeter-wave radar point cloud data, the large kernel convolution technology is used to capture long-distance dependencies, and the convolution kernel parameters are optimized through Gaussian noise perturbation to avoid overfitting problems. In the process of generating dense features, the thermal mask guidance and multimodal fusion technology are used to generate high-quality dense millimeter-wave radar feature maps, improve perception performance, and thus improve the efficiency of extracting millimeter-wave radar features.

[0053] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a radar signature generation device capable of realizing the above functions, etc. The following takes the radar signature generation device as an execution subject as an example to illustrate this embodiment and the following embodiments.

[0054] Based on this, the embodiment of the present application provides a radar feature generation method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the radar signature generation method of the present application.

[0055] In this embodiment, the radar signature generation method includes steps S10 to S30:

[0056] Step S10, extracting features from radar point cloud data to obtain target radar disturbance features;

[0057] It should be noted that this embodiment attempts to fuse millimeter-wave radar and images at the feature level, most of which follow the BEVFusion paradigm (fusing data from multiple sensors into a unified bird's-eye view coordinate system) and projecting multi-view image features and radar features into a bird's eye view (BEV). However, the current millimeter-wave radar-camera fusion method still uses encoders designed for lidar sensors (such as PointPillars) to process millimeter-wave radar data, ignoring the difference between millimeter-wave radar and lidar, that is, compared with lidar point clouds, millimeter-wave point cloud data is sparser, so directly using existing lidar point cloud encoders to process millimeter-wave radar data is suboptimal. Therefore, this embodiment solves the problem of millimeter-wave radar point cloud feature expression from two aspects: feature encoding based on large convolution kernel perturbation and feature densification. When millimeter-wave radar point cloud data is sparse, large-kernel convolution technology is used to capture long-distance dependencies, and Gaussian noise perturbation is used to optimize the convolution kernel parameters to avoid overfitting problems. In the process of dense feature generation, thermal mask guidance and multimodal fusion technology are used to generate high-quality dense millimeter-wave radar feature maps to improve perception performance.

[0058] It can be understood that radar point cloud data refers to a collection of target position information in the surrounding environment obtained by a millimeter-wave radar sensor, and the target radar disturbance feature refers to the millimeter-wave radar bird's-eye view disturbance feature (Radar-BEV disturbance feature).

[0059] In the specific implementation, the set of target position information in the surrounding environment obtained by the millimeter-wave radar sensor is processed by a large-core convolution encoder, and then combined with Gaussian noise, and finally the millimeter-wave radar bird's-eye view disturbance feature, that is, the target radar disturbance feature, is obtained. In this embodiment, the large receptive field characteristic of the large-core convolution is used to effectively model the dependency relationship of long-distance point clouds, thereby improving the expression ability of the global features of the scene. At the same time, by adding Gaussian noise to the large-core convolution parameter optimization, the convolution kernel is prevented from overfitting sparse features, thereby improving the robustness and generalization ability of feature extraction.

[0060] Step S20, determining the target image disturbance characteristics according to the current image data and the heat map guided calculation strategy;

[0061] It can be understood that the current image data refers to the image data collected by the vehicle-mounted camera, the heat map guided calculation strategy refers to the image feature generation strategy based on the heat map guidance, and the target image perturbation feature refers to the image bird's-eye view feature (Image-BEV feature).

[0062] In the specific implementation, the image data collected by the vehicle camera is processed according to the image feature generation strategy guided by the heat map, and then the image bird's-eye view features, that is, the target image disturbance features, are obtained. Among them, the image feature generation guided by the heat map uses the precise position information of the millimeter-wave radar to guide the regional image features, improve the image feature's representation ability for the BEV scene information, and is used for the final densification of the millimeter-wave radar features.

[0063] In a feasible implementation, step S20 may include steps A21 to A23:

[0064] Step A21, extracting features from the current image data and downsampling according to a preset residual network to obtain initial image features;

[0065] It should be noted that the preset residual network refers to a pre-set residual network (Residual Network, ResNet), and the initial image features refer to image features extracted by the residual network.

[0066] In the specific implementation, the ResNet algorithm is used to process the image data collected by the camera, extract image features, and output a 16-fold downsampled image feature map for subsequent heat map guidance, that is, to obtain the initial image features.

[0067] Step A22, determining the thermal map image features according to the thermal map guided calculation strategy, the radar point cloud data and the initial image features;

[0068] It can be understood that the heat map image features refer to image features guided by the heat map.

[0069] In the specific implementation, according to the image feature generation strategy guided by the heat map, the set of target position information in the surrounding environment obtained by the millimeter-wave radar sensor is mapped to the image plane through the sensor calibration matrix. In order to ensure that the initial image feature coordinates remain consistent, the mapped pixel coordinates are also scaled by 16 times, and then the radius size is controlled by the standard deviation of the Gaussian distribution to draw the heat map, and the image features guided by the heat map, namely the thermal map image features, are obtained.

[0070] In a feasible implementation, step A22 may include steps B221 to B224:

[0071] Step B221, determining the currently generated heat map according to the heat map guided calculation strategy and the radar point cloud data;

[0072] It can be understood that the currently generated heat map refers to a heat map drawn by controlling the radius size through the standard deviation of the Gaussian distribution.

[0073] In the specific implementation, the millimeter-wave radar point cloud position information is mapped to the image plane through the sensor calibration matrix, and then the mapped pixel coordinates are also scaled 16 times to obtain the corresponding pixel coordinates, and taken as the center point. Finally, the radius size is controlled by the standard deviation of the Gaussian distribution to draw the heat map and obtain the currently generated heat map.

[0074] Step B222, determining a current decoding heat map according to a preset image decoding model and image bird's-eye view features;

[0075] It can be understood that the preset image decoding model refers to a model of a pre-set multi-layer convolution structure, the image bird's-eye view feature refers to the features of the image from a bird's-eye view (i.e., the perspective of looking vertically downward from above), and the current decoding heat map refers to the heat map generated by the model of a multi-layer convolution structure.

[0076] In a specific implementation, the features representing the image from a bird's-eye view are decoded according to a pre-set multi-layer convolutional structure model, and then a thermal mask estimated by the multi-layer convolutional structure model is obtained, that is, the current decoded thermal map.

[0077] Step B223, performing loss calculation according to the currently generated thermogram and the currently decoded thermogram to obtain a target image thermogram;

[0078] It can be understood that the target image heat map refers to the heat map corresponding to the final image features.

[0079] In the specific implementation, the MSE loss calculation is performed based on the heat map drawn by controlling the radius size through the standard deviation of the Gaussian distribution and the heat map generated by the model of the multi-layer convolution structure to obtain the heat map corresponding to the final image feature. Among them, MSE (Mean Squared Error) is a commonly used loss function, especially suitable for regression problems. It measures the average square difference between the predicted value and the true value.

[0080] Step B224, obtaining the thermal map image features according to the target image thermal map and the initial image features.

[0081] It can be understood that, by multiplying the heat map corresponding to the final image feature and the image feature extracted by the residual network element by element, an image feature guided by the heat map, namely, a heat map image feature, is obtained.

[0082] Step A23, performing view conversion on the thermal map image features to obtain target image disturbance features.

[0083] In a specific implementation, this embodiment uses the LSS algorithm to perform feature conversion on the image features guided by the heat map, and then obtains the Image-BEV features, wherein the LSS (Lift-Splat-Shoot) algorithm is a method for converting image features into BEV (Bird's Eye View) features.

[0084] It should be noted that this embodiment uses the ResNet algorithm to process the image data collected by the camera, extracts image features, and outputs a 16-fold downsampled image feature map for subsequent heat map guidance; the millimeter-wave radar point cloud position information is mapped to the image plane through the sensor calibration matrix. To ensure consistency with the feature map coordinates, the mapped pixel coordinates are also scaled 16 times to obtain the corresponding pixel coordinates (x, y), and the heat map is drawn with the radius controlled by the standard deviation σ of the Gaussian distribution. The specific calculation formula is:

[0085]

[0086] in is the pixel coordinate, the generated heat map Y xyc ∈[0, 1], the closer to the center point (x, y), the smaller the Y xycThe closer it is to 1. Subsequently, the heat map is element-by-element multiplied by the output image features to obtain the image features guided by the heat map. Among them, with the help of the thermal mask calculated by the above formula, a thermal distribution map of the object corresponding to the BEV size of the scene can be obtained. For the thermal mask, it cannot be directly obtained in the actual model reasoning process because there is no way to get the marked bounding box. Therefore, a multi-layer convolutional structure model is used in the training process to decode the image BEV features to obtain an estimated thermal mask. This mask data is calculated with the thermal mask generated above for MSE loss. After this calculation, the estimated thermal mask can be obtained in the network reasoning process. Finally, the LSS algorithm is used to convert the obtained image features guided by the heat map into Image-BEV features (in this embodiment, the size of the BEV space is defined as 128×128), with a size of H×W×C, where H and W are 128, and C represents 64, which is consistent with the Radar-BEV perturbation features in 1.2. Among them, the areas associated with millimeter-wave radar points in the image features guided by the heat map are enhanced, thereby invisibly guiding the better generation of subsequent depth values ​​and semantic features and reducing the impact of depth noise.

[0087] It should be noted that the specific heat map generation method requires the use of the 3D BoundingBox annotated in the data set, that is, the 3D annotation box. Figure 2 As shown. After calculation by the above formula, the original Bounding Box in the BEV space is converted into a radius circle with Gaussian characteristics.

[0088] Step S30: determining a target dense radar feature according to the target radar disturbance feature and the target image disturbance feature.

[0089] It can be understood that the target dense radar signature refers to a dense millimeter-wave radar signature.

[0090] In the specific implementation, this embodiment enhances the extraction of millimeter-wave radar point cloud features and the generation of image features guided by heat maps through a large-kernel convolutional encoder, and obtains Radar-BEV (Radar BEV features) perturbation features and Image-BEV features (camera BEV features), respectively, and then splices and fuses the Radar-BEV perturbation features and Image-BEV features to achieve the densification of millimeter-wave radar features, that is, splices and fuses the Radar-BEV perturbation feature map and the Image-BEV feature map to output dense millimeter-wave radar features, which can be used for downstream tasks of autonomous driving.

[0091] In a feasible implementation, step S30 may include steps A31 to A32:

[0092] Step A31, performing feature splicing according to the target radar disturbance feature and the target image disturbance feature to obtain a target splicing feature;

[0093] It can be understood that the target splicing feature refers to the feature obtained by splicing the Radar-BEV feature and the Image-BEV feature.

[0094] In a specific implementation, in order to achieve the densification of millimeter-wave radar features, the Radar-BEV features and the Image-BEV features are spliced ​​to obtain features spliced ​​from the Radar-BEV features and the Image-BEV features, namely, target splicing features.

[0095] Step A32, determining the target dense radar feature according to the target splicing feature, the preset self-attention strategy and the number of feature channels.

[0096] It can be understood that the preset self-attention strategy refers to a preset multi-head self-attention strategy, and the number of feature channels refers to the number of feature channel dimensions.

[0097] In the specific implementation, after obtaining the splicing of the image BEV features and the Radar BEV features, the multi-head self-attention method is used to complete the splicing of the image BEV features and the Radar BEV features to complete the densification calculation. The specific calculation method is expressed as:

[0098]

[0099] Among them, D BEV represents the final dense BEV feature. MHSA stands for MultiHeadSelfAttention. concat(B cam , B rad ) represents the camera BEVB cam and Radar BEV B rad C represents B cam , B rad The number of feature channel dimensions, in this embodiment, the size of C can be 64, 256, 512, 1024.

[0100] This embodiment obtains the target radar disturbance feature by extracting features from radar point cloud data; determines the target image disturbance feature according to the current image data and the heat map guidance calculation strategy; and determines the target dense radar feature according to the target radar disturbance feature and the target image disturbance feature. When the millimeter-wave radar point cloud data is sparse, the large kernel convolution technology is used to capture long-distance dependencies, and the convolution kernel parameters are optimized by Gaussian noise perturbation to avoid overfitting problems. In the process of generating dense features, the thermal mask guidance and multimodal fusion technology are used to generate high-quality dense millimeter-wave radar feature maps, improve perception performance, and thus improve the efficiency of extracting millimeter-wave radar features.

[0101] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , step S10 in the radar feature generation method further includes steps S11 to S13:

[0102] Step S11, converting the format of radar point cloud data to obtain radar voxel data;

[0103] It can be understood that radar voxel data refers to millimeter-wave radar point cloud data represented by voxels.

[0104] In a specific implementation, the point cloud data acquired by the millimeter-wave radar is converted into voxel representation, that is, the three-dimensional space is divided into small cubic units, each unit contains one or more point cloud points, and the millimeter-wave radar point cloud data represented by voxels, namely radar voxel data, is obtained.

[0105] Step S12, inputting the radar voxel data into a target convolution encoder to obtain an initial radar disturbance feature;

[0106] It can be understood that the target convolution encoder refers to a large-core convolution encoder, wherein, in this embodiment, the size of each sub-flow type convolution kernel in the four stages is expanded to 7×7×7, the down-sampled sparse convolution block remains unchanged, and the initial radar disturbance feature refers to the millimeter-wave radar disturbance feature with Gaussian noise added.

[0107] In the specific implementation, the millimeter-wave radar point cloud is input into the large-core convolution encoder for feature processing, and Gaussian noise is added during the training process to generate disturbance features. Then, the millimeter-wave radar disturbance features with added Gaussian noise are obtained according to the processing results of the large-core convolution encoder, that is, the initial radar disturbance features.

[0108] In a feasible implementation, step S12 may include steps A121 to A122:

[0109] Step A121, inputting the radar voxel data into a target convolution encoder for feature processing to obtain target radar sparse features;

[0110] It can be understood that the target radar sparse features refer to the millimeter-wave radar sparse features output by the large-kernel convolution encoder.

[0111] In the specific implementation, the millimeter wave radar point cloud is input into the large core convolution encoder for feature processing. The large core convolution encoder consists of four stages, each of which includes a downsampling sparse convolution block and a sub-stream sparse convolution block. Through resolution reduction and channel increase, the feature expression capability is gradually enhanced, and the millimeter wave radar sparse features output by the large core convolution encoder are obtained, that is, the target radar sparse features.

[0112] Step A122, obtaining initial radar disturbance characteristics according to the target radar sparse characteristics and Gaussian distribution noise.

[0113] It can be understood that Gaussian distribution noise refers to normally distributed noise. In this embodiment, the convolution kernel parameters are optimized by Gaussian noise perturbation to avoid overfitting problems.

[0114] In a specific implementation, this embodiment adds Gaussian noise to each large kernel weight in the four stages during the training process, encodes and generates millimeter-wave radar disturbance features to enhance the generalization of the model, and finally obtains the millimeter-wave radar disturbance features with added Gaussian noise, that is, the initial radar disturbance features.

[0115] In a feasible implementation, step A122 may include steps B1221 to B1223:

[0116] Step B1221, determining a noise weight matrix according to Gaussian distribution noise and initial convolution kernel weights;

[0117] It can be understood that the noise weight matrix refers to the weight matrix after adding noise, and the initial convolution kernel weight refers to the weight of the original convolution kernel.

[0118] In a specific implementation, the weights of the original convolution kernel are combined with Gaussian distribution noise with a mean of 0 and a standard deviation of σ to obtain a weight matrix after adding noise, namely, a noise weight matrix.

[0119] Step B1222, performing model training according to the noise weight matrix to obtain a target noise feature model;

[0120] In the specific implementation, the target noise feature model refers to the disturbance feature generation model after adding noise. By adding Gaussian noise to each large kernel weight in the four stages during the training process, the disturbance feature generation model after adding noise, that is, the target noise feature model, is obtained.

[0121] Step B1223, determining the initial radar disturbance characteristics according to the target radar sparse characteristics and the target noise characteristic model.

[0122] In the specific implementation, the millimeter-wave radar sparse features output by the large-core convolution encoder are subjected to noise addition processing by using a disturbance feature generation model after adding noise, that is, Gaussian noise is added to each large-core weight W in the four stages to encode and generate millimeter-wave radar disturbance features, that is, millimeter-wave radar disturbance features with added Gaussian noise are obtained.

[0123] Step S13, performing view conversion on the initial radar disturbance feature to obtain a target radar disturbance feature.

[0124] In the specific implementation, the millimeter-wave radar perturbation feature with added Gaussian noise is subjected to view conversion, that is, the discrete millimeter-wave radar feature is assigned to the final BEV view space to obtain the Radar-BEV perturbation feature, that is, the target radar perturbation feature.

[0125] It should be noted that the large-core convolution encoder in this embodiment improves and enhances the sparse characteristics of Radar. This is also because Radar data is more sparse. By using a large-core convolution enhancement model to enhance the modeling ability between long-distance point clouds, feature extraction is more efficient. The millimeter-wave radar point cloud is input into the large-core convolution encoder for feature processing. The encoder consists of four stages, each of which includes a downsampling sparse convolution block and a sub-stream sparse convolution block. The feature expression capability is gradually enhanced through resolution reduction and channel increase. Since the millimeter-wave radar point cloud is sparser than the lidar point cloud, and the sub-stream convolution kernel commonly used to process the lidar point cloud is 3×3×3, it cannot effectively capture the dependencies between millimeter-wave point clouds. Therefore, this solution chooses to expand the size of each sub-stream convolution kernel in the four stages to 7×7×7, and the downsampling sparse convolution block remains unchanged.

[0126] In this embodiment, Gaussian noise is added during the training process to generate disturbance features. Increasing the convolution kernel will undoubtedly bring more parameters to learn, and the existing 3D data sets available for learning may only contain thousands of scenes, which cannot be compared with 2D image data sets. In addition, 3D point clouds or voxels are sparse, not dense images. Therefore, optimizing the large kernel convolution parameters may be insufficient, leading to overfitting problems. This embodiment chooses to add Gaussian noise to each large kernel weight W in the 4 stages during the training process, and encodes and generates millimeter wave radar disturbance features to enhance the generalization of the model. The specific calculation formula is:

[0127]

[0128] W noisy =W+ε

[0129] ε~N(0,σ 2 )

[0130] Where f(p) represents the perturbation eigenvalue at the output position p, k is the offset distance at position p, and X p+k is the sparse feature in the kernel space K, W is the original convolution kernel weight, W noisy is the weight matrix after adding noise, and ε represents Gaussian distribution noise with a mean of 0 and a standard deviation of σ. Finally, the discrete millimeter-wave radar features are assigned to the final BEV view space (in this embodiment, the size of the BEV space is defined as 128×128), and the final Radar-BEV perturbation feature is H×W×C, where H and W are 128 and C represents 64.

[0131] This embodiment converts the format of radar point cloud data to obtain radar voxel data; inputs the radar voxel data into the target convolution encoder to obtain the initial radar perturbation feature; and performs view conversion on the initial radar perturbation feature to obtain the target radar perturbation feature. By utilizing the large receptive field characteristics of large kernel convolution, the dependency relationship of long-distance point clouds is effectively modeled, and the expression ability of the global features of the scene is improved. At the same time, Gaussian noise is added in the optimization of large kernel convolution parameters to prevent the convolution kernel from overfitting sparse features, thereby improving the robustness and generalization ability of feature extraction.

[0132] For example, to help understand the implementation process of the radar feature generation method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 4 , Figure 4 A brief flowchart of a radar feature generation method is provided. Specifically, when millimeter-wave radar point cloud data is sparse, the millimeter-wave radar point cloud data is processed at a faster speed. The large kernel convolution technology is used to capture long-distance dependencies on voxelized millimeter-wave radar point cloud data. The convolution kernel parameters are optimized by Gaussian noise perturbation to avoid overfitting problems, and Radar-BEV perturbation features are obtained. The image features guided by thermal maps are obtained according to the millimeter-wave radar point cloud data combined with the extracted features of the image. The view is then transformed based on LSS to finally obtain Image-BEV features. In the process of generating dense features, the thermal mask guidance and multimodal fusion technology are used to splice and fuse the Radar-BEV perturbation features and the Image-BEV features to generate high-quality dense millimeter-wave radar feature maps, thereby improving perception performance.

[0133] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the radar feature generation method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0134] This application also provides a radar feature generation device, please refer to Figure 5 , the radar feature generating device comprises:

[0135] The processing module 10 is used to extract features from the radar point cloud data to obtain target radar disturbance features;

[0136] The processing module 10 is also used to determine the target image disturbance characteristics according to the current image data and the heat map guided calculation strategy;

[0137] The generating module 20 is used to determine the target dense radar feature according to the target radar disturbance feature and the target image disturbance feature.

[0138] Optionally, the processing module 10 is further used for:

[0139] Convert the format of radar point cloud data to obtain radar voxel data;

[0140] Inputting the radar voxel data into a target convolution encoder to obtain an initial radar disturbance feature;

[0141] Performing view conversion on the initial radar disturbance feature to obtain a target radar disturbance feature.

[0142] Optionally, the processing module 10 is further used for:

[0143] Inputting the radar voxel data into a target convolution encoder for feature processing to obtain target radar sparse features;

[0144] An initial radar disturbance feature is obtained according to the target radar sparse feature and Gaussian distribution noise.

[0145] Optionally, the processing module 10 is further used for:

[0146] Determine the noise weight matrix according to the Gaussian distribution noise and the initial convolution kernel weight;

[0147] Performing model training according to the noise weight matrix to obtain a target noise feature model;

[0148] An initial radar disturbance feature is determined according to the target radar sparse feature and the target noise feature model.

[0149] Optionally, the processing module 10 is further used for:

[0150] Perform feature extraction and downsampling on the current image data according to the preset residual network to obtain the initial image features;

[0151] Determining thermal map image features according to the thermal map guided calculation strategy, the radar point cloud data, and the initial image features;

[0152] Performing view conversion on the thermal map image features to obtain target image perturbation features.

[0153] Optionally, the processing module 10 is further used for:

[0154] Determine the current generated thermal map based on the thermal map guided calculation strategy and radar point cloud data;

[0155] Determine the current decoding heat map according to the preset image decoding model and the image bird's-eye view features;

[0156] Perform loss calculation according to the currently generated thermogram and the currently decoded thermogram to obtain a target image thermogram;

[0157] A thermal map image feature is obtained according to the target image thermal map and the initial image feature.

[0158] Optionally, the processing module 10 is further used for:

[0159] Performing feature splicing according to the target radar disturbance feature and the target image disturbance feature to obtain a target splicing feature;

[0160] The target dense radar feature is determined according to the target splicing feature, a preset self-attention strategy and the number of feature channels.

[0161] The radar feature generation device provided by the present application adopts the radar feature generation method in the above embodiment, which can solve the technical problem of low efficiency of the existing method for extracting millimeter wave radar features. Compared with the prior art, the beneficial effects of the radar feature generation device provided by the present application are the same as the beneficial effects of the radar feature generation method provided by the above embodiment, and the other technical features in the radar feature generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0162] The present application provides a radar signature generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the radar signature generation method in the above-mentioned embodiment 1.

[0163] Reference below Figure 6, which shows a schematic diagram of the structure of a radar signature generation device suitable for implementing the embodiment of the present application. The radar signature generation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The radar signature generating device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0164] like Figure 6 As shown, the radar signature generation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the radar signature generation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the radar signature generation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a radar signature generation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0165] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0166] The radar feature generation device provided by the present application adopts the radar feature generation method in the above embodiment, which can solve the technical problem of low efficiency of the existing method for extracting millimeter wave radar features. Compared with the prior art, the beneficial effects of the radar feature generation device provided by the present application are the same as the beneficial effects of the radar feature generation method provided by the above embodiment, and the other technical features in the radar feature generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0167] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0168] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0169] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the radar signature generation method in the above-mentioned embodiment.

[0170] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0171] The above-mentioned computer-readable storage medium may be included in the radar signature generation device; or it may exist independently without being assembled into the radar signature generation device.

[0172] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the radar feature generation device, the radar feature generation device: extracts features from radar point cloud data to obtain target radar disturbance features; determines target image disturbance features based on current image data and a heat map-guided calculation strategy; and determines target dense radar features based on the target radar disturbance features and the target image disturbance features.

[0173] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0175] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0176] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned radar feature generation method, and can solve the technical problem of low efficiency of the existing method for extracting millimeter-wave radar features. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the radar feature generation method provided by the above-mentioned embodiment, and will not be repeated here.

[0177] The present application also provides a computer program product, including a computer program, which implements the steps of the radar signature generation method as described above when the computer program is executed by a processor.

[0178] The computer program product provided by the present application can solve the technical problem of low efficiency of the existing method for extracting millimeter wave radar features. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the radar feature generation method provided by the above embodiment, which will not be repeated here.

[0179] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A radar feature generation method, characterized in that: The radar signature generation method comprises: Extract features from radar point cloud data to obtain target radar disturbance features; Determine the target image perturbation characteristics based on the current image data and the heat map guided calculation strategy; A target dense radar feature is determined according to the target radar disturbance feature and the target image disturbance feature.

2. The method according to claim 1, characterized in that The step of extracting features from radar point cloud data to obtain target radar disturbance features includes: Convert the format of radar point cloud data to obtain radar voxel data; Inputting the radar voxel data into a target convolution encoder to obtain an initial radar disturbance feature; Performing view conversion on the initial radar disturbance feature to obtain a target radar disturbance feature.

3. The method according to claim 2, characterized in that The step of inputting the radar voxel data into a target convolution encoder to obtain an initial radar disturbance feature comprises: Inputting the radar voxel data into a target convolution encoder for feature processing to obtain target radar sparse features; An initial radar disturbance feature is obtained according to the target radar sparse feature and Gaussian distribution noise.

4. The method according to claim 3, characterized in that The step of obtaining the initial radar disturbance feature according to the target radar sparse feature and Gaussian distribution noise comprises: Determine the noise weight matrix according to the Gaussian distribution noise and the initial convolution kernel weight; Performing model training according to the noise weight matrix to obtain a target noise feature model; An initial radar disturbance feature is determined according to the target radar sparse feature and the target noise feature model.

5. The method according to claim 1, characterized in that The step of determining the target image disturbance feature according to the current image data and the heat map guided calculation strategy includes: Perform feature extraction and downsampling on the current image data according to the preset residual network to obtain the initial image features; Determining thermal map image features according to the thermal map guided calculation strategy, the radar point cloud data, and the initial image features; Performing view conversion on the thermal map image features to obtain target image perturbation features.

6. The method according to claim 5, characterized in that The step of determining the thermal map image features according to the thermal map guided calculation strategy, the radar point cloud data and the initial image features comprises: Determine the current generated thermal map based on the thermal map guided calculation strategy and radar point cloud data; Determine the current decoding heat map according to the preset image decoding model and the image bird's-eye view features; Perform loss calculation according to the currently generated thermogram and the currently decoded thermogram to obtain a target image thermogram; A thermal map image feature is obtained according to the target image thermal map and the initial image feature.

7. The method according to claim 1, characterized in that The step of determining the target dense radar feature according to the target radar disturbance feature and the target image disturbance feature comprises: Performing feature splicing according to the target radar disturbance feature and the target image disturbance feature to obtain a target splicing feature; The target dense radar feature is determined according to the target splicing feature, a preset self-attention strategy and the number of feature channels.

8. A radar signature generating device, characterized in that: The device comprises: A processing module is used to extract features from radar point cloud data to obtain target radar disturbance features; The processing module is further used to determine the target image disturbance characteristics according to the current image data and the heat map guided calculation strategy; A generating module is used to determine a target dense radar feature according to the target radar disturbance feature and the target image disturbance feature.

9. A radar signature generating device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the radar signature generation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the radar signature generation method according to any one of claims 1 to 7 are implemented.

Images