Attention-based 4d millimeter wave radar and vision fusion method

Abstract

The application discloses a kind of fusion methods of 4D millimeter wave radar and vision based on attention, comprising the following steps: using the format of voxel to carry out radar backbone feature extraction to 4D millimeter wave radar point cloud data, obtain the feature of radar in BEV space;Image data is extracted to image backbone feature, and the feature of 2D of image is obtained;The 2D feature of image is projected by view projection module, and the depth is densely predicted by classification, and the feature of image in BEV space is obtained according to the predicted image depth and camera external parameter;Finally, the weight of millimeter wave radar and visual feature is reasonably distributed by using attention mechanism for fusion in feature layer through the designed fusion module.The application solves the problem of mutual dependence and weight distribution difficulty when 4D millimeter wave radar and vision are fused.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of radar and vision fusion, and particularly relates to a 4D millimeter wave radar and vision fusion method based on attention. BACKGROUND

[0002] Millimeter wave radar and computer vision technology are widely used in automatic driving, security, intelligent transportation and other fields. Millimeter wave radar has strong penetration ability and the advantage of not being affected by light, rain and snow, but it cannot provide high-precision target recognition and tracking information. In contrast, computer vision technology can provide more rich target information, but it is greatly affected by light, weather and other factors.

[0003] Generally, the fusion strategy of radar and vision is divided into three types: decision layer fusion (usually we call it post fusion), feature layer fusion (middle layer fusion) and data layer fusion (pre fusion). The decision layer fusion is to fuse the final results output by the radar-based model, such as 3D BoudingBox, and the results output by the vision detection, such as 2D BoudingBox, through filtering algorithm; the feature layer fusion is to project the final results output by one modality to the deep learning feature layer of another modality, and then use a subsequent fusion network for information fusion; the data layer fusion is to directly fuse the original data of two modalities, and then use a neural network to directly output the final result.

[0004] These fusion strategies have their own advantages and disadvantages, but the post fusion is widely used in the industry because it is more flexible and has better robustness. The output results of different modalities are integrated through artificially designed algorithms and rules, and different modalities have different usage priorities in different situations, so the influence of single sensor failure on the system can be better handled. However, the post fusion has many disadvantages, one is that the information utilization is not very sufficient, two is that the system link becomes more complex, the longer the link, the more likely to have problems, three is that when the rules are stacked more and more, the maintenance cost will be high. The pre fusion scheme is currently more popular in the academic field, which can better utilize the end-to-end characteristics of neural networks. However, the pre fusion scheme is rarely directly used in vehicles, and the reason is that the robustness of the current pre fusion scheme does not meet the actual requirements, especially when the radar signal is problematic, the current pre fusion scheme is almost unable to handle it.

[0005] In the actual environment, the following problems will be encountered:

[0006] (1) Inaccurate external parameters of radar and camera: due to calibration problems or vehicle running jolt, the external parameters will be inaccurate, causing deviation in the direct projection of point cloud and image.

[0007] (2) Camera noise: such as lens dirt block, frame stuck, or even a camera damage, etc. which leads to the point cloud projection to the image cannot find the corresponding feature or get the wrong feature.

[0008] (3) Radar noise: in addition to the dirt block problem; for some low-reflective objects, the radar itself characteristics lead to the return point missing.

[0009] For problems (1) and (2), some methods have provided some compatibility capabilities, such as DeepFusion, but for problem (3) radar noise caused by point cloud missing, all are helpless. Because this kind of method all need to query the image characteristics through the point cloud coordinates, once the point cloud is missing, all means cannot be carried out. SUMMARY

[0010] In order to solve the problems existing in the prior art, the purpose of the present application is to provide a 4D millimeter wave radar and vision fusion perception method based on attention.

[0011] The specific implementation steps of the present application are as follows: a fusion method of 4D millimeter wave radar and vision based on attention, comprising the following steps:

[0012] Step 1: radar feature extraction: adopting the format of voxel to extract the radar backbone of 4D millimeter wave radar data, to obtain the BEV feature of radar data;

[0013] Step 2: image feature extraction: image backbone feature extraction is performed on image data to obtain 2D feature of image, and each picture feature pixel is projected back to 3D space through projection to form image BEV feature;

[0014] Step 3: feature fusion: by introducing attention mechanism, the radar and vision data obtained in steps 1 and 2 are fused in BEV space through attention coding to obtain comprehensive target information;

[0015] Step 4: target detection: using the fused BEV feature information for target detection.

[0016] Preferably, the step of adopting the format of voxel to extract the radar backbone of 4D millimeter wave radar data in step 1 to obtain the radar feature data comprises:

[0017] The voxel is selected as the form of the point cloud BEV feature feature extraction network, the BEV feature feature extraction network takes VoxelNet as the backbone network, and then adds a feature pyramid network, wherein the backbone network divides the three-dimensional point cloud into a certain number of voxels, after random sampling and normalization of the points, local feature extraction is performed on each non-empty voxel using a plurality of voxel feature encoding layers to obtain voxel-level features, and then a 3D convolution module is used to further abstract the features (increase the receptive field and learn the geometric spatial representation) to obtain the BEV feature of the point cloud; the BEV feature is further refined by the feature pyramid network, the features are extracted from the BEV feature by using a bottom-up path, and the features are combined and refined by using a top-down path, so that feature maps of different resolutions can be obtained, and they all contain the semantic information of the original deepest feature map.

[0018] Preferably, the specific steps of performing image backbone feature extraction on the image data in step 2 to obtain the 2d feature of the image include:

[0019] Swin Transformer is used as the backbone network, and a feature pyramid network layer is added to obtain the 2d feature of the image. The depth information of the image is estimated by the view projection module, and the construction of the image BEV view is completed, wherein the whole model of the backbone network adopts a hierarchical structure, and there are a total of 4 stages. The resolution of the input feature map is reduced in each stage, and the receptive field is gradually expanded. At the beginning of the input, patch embedding is performed to divide the image into several small blocks and embed them into the embedding. Each stage includes two parts, namely patch merging and Swin Transformer module.

[0020] Preferably, the specific process of projecting each picture feature pixel back into the 3D space to form the image BEV feature is as follows:

[0021] A set of discrete depth values is generated for each pixel of the image. The method of generating depth values for pixels is as follows: in the frustum at a distance of 1-60 meters from the camera, there is an optional depth value every 1 meter, N points are sampled on this straight line, the depth information of the feature points is predicted, and a D-dimensional vector is used to represent the depth information through softmax, D represents the number of distances with an interval of 1 meter in the range of 1-60 meters, and the depth information obtained by each pixel point is weighted with the image features at the same position to generate a pseudo-point cloud similar to a flat pyramid. The coordinates of the picture in the 3D space are obtained by using the camera extrinsic and intrinsic parameters and the Frustum Feature obtained in the foregoing, and then the picture is flattened. The specific process is as follows: the range of the BEV view angle is limited, the size of each grid is determined, and the features projected onto the corresponding grid are collected into a grid; in the top view, there may be multiple features in the same grid, the image point cloud is quantized along the x and y dimensions using a fixed step, the BEV pooling operation is used to aggregate features in each BEV grid, and the features are unfolded along the z axis.

[0022] Preferably, the radar and image features are fused in step 3, and the fusion steps include:

[0023] In space, the features of the two are first stacked in channels, then respectively pass through global maximum pooling and global average pooling, then respectively pass through a 3*3 convolution kernel, and then pass through a sigmoid activation operation to generate a final spatial attention feature map, and each pixel of the BEV features of the radar and the image is weighted through the feature map;

[0024] In channels, the radar features and the image features subjected to spatial attention extraction are stacked in channels, then pass through average pooling, a 3*3 convolution kernel and a sigmoid operation to obtain channel weights, and the stacked features are multiplied through the weights to obtain the final fused features.

[0025] Compared with the prior art, the present application has the following advantages: the present application unifies multi-modal features in a shared bird's eye view (BEV) representation space, well preserves geometric and semantic information, solves the mutual dependence and weight distribution problem in 4D millimeter wave radar and vision fusion. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The flow of the present application.

[0027] Figure 2 The schematic diagram of the image feature encoding module of the present application.

[0028] Figure 3 The schematic diagram of the feature fusion module of the present application. DETAILED DESCRIPTION

[0029] The application will be described in further detail below with reference to the drawings. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the application. These are all within the scope of protection of the application.

[0030] As shown in the figure, a 4D millimeter wave radar and vision fusion method based on attention, the specific steps are: Figure 1

[0031] Step 1: Radar feature extraction: 4D millimeter wave radar data is extracted using voxel format to obtain BEV features of radar data;

[0032] When performing radar backbone feature extraction, the point cloud of the millimeter wave radar is relatively sparse, and it is difficult to directly use the feature extraction method of dense point cloud. In addition, most point-level feature extraction methods can only fuse local information features, and the correlation with overall information is not strong. Therefore, in the field of autonomous driving, point-level features cannot be directly used for 3D target detection tasks. For 4D millimeter wave radar, its point cloud is also a three-dimensional point cloud, so the feature extraction method of laser radar point cloud can also be applied to 4D millimeter wave radar, and voxel-based feature expression: Voxel is selected. This part of the network design is mainly based on VoxelNet as the foundation, plus an FPN (Feature Pyramid Network). The backbone network divides the three-dimensional point cloud into a certain number of voxels, after random sampling and normalization, a certain number of voxel feature coding layers are used for local feature extraction for each non-empty voxel, to obtain voxel-level features, and then a 3D convolution module is used to further abstract the features (increase the receptive field and learn the geometric space representation) to obtain the BEV features of the point cloud; the BEV features are further refined by the feature pyramid network, the features are extracted from the BEV features by using a bottom-up path, and the features are combined and refined by using a top-down path, so that feature maps of different resolutions can be obtained, and they all contain semantic information of the original deepest feature map.

[0033] Through actual testing, this method can effectively extract radar point features.

[0034] Step 2: Image feature extraction: image backbone feature extraction is performed on image data to obtain 2D features of the image, and each picture feature pixel is projected back to the 3D space by projection to form image BEV features;

[0035] ​When extracting image backbone features, for high-resolution images with many pixels, both Transformers and CNNs (Convolutional Neural Networks) require a lot of computation, and the late-stage computing power requirement is high. Therefore, a Swin Transformer containing a sliding window operation and having a hierarchical design is used. This network design is mainly based on the Swin Transformer, plus an FPN layer, which can effectively extract 2D features of the image. In order to obtain the features of the image in the BEV space, a view projection module is designed, as shown in Figure 2 The module explicitly estimates the depth information of the image to complete the construction of the BEV view of the picture. For the obtained 2D image features, each feature pixel is dispersed to D discrete points along the camera light line, the depth probability distribution of each pixel in the image is predicted, and the related features are scaled according to the corresponding depth probability to obtain the image feature point cloud. Then, the image point cloud is quantized along the x, y dimensions using a fixed step, the features are aggregated in each BEV grid using the BEV pooling operation, and then the features are unfolded along the z axis.

[0036] The backbone network adopts a hierarchical structure, and there are a total of 4 stages, each of which reduces the resolution of the input feature map and gradually expands the receptive field, playing a role in reducing the amount of calculation. At the beginning of the input, PatchEmbedding is performed to divide the image into several small blocks and embed them into Embedding. Each stage contains two parts, namely Patch merging (except the first block is a linear layer) and Swin Transformer module; for the view projection module, first generate a bunch of discrete depth values for each pixel point of the image, so that the network can choose the appropriate depth during model training. The method of generating depth values for pixels is as follows: in the view frustum from 1 meter to 60 meters from the camera, there is a selectable depth value every 1 meter (so each pixel has 61 selectable discrete depth values), so N points can be sampled on this straight line, then the network needs to predict the depth information (distribution over depth) of this feature point, which is represented by a D-dimensional vector after softmax, D represents the distance in the range of 1-60 meters with an interval of 1 meter, that is, D = 61, so each position on D represents the probability value of the pixel being in this depth range. The depth information obtained by each pixel point is weighted (scale) with the image features at the same position to generate a pseudo point cloud shaped like a flat pyramid (frustum), and the camera extrinsic and intrinsic parameters and the pyramid features obtained before are used to convert the coordinates of the image in the 3D space, and then "flatten", the specific process is: by limiting the range of BEV view angle, the size of each grid is determined, and the features projected into the corresponding grid are collected into a grid. In the top view, there may be multiple features in the same grid, and a fixed step is used to quantize the image point cloud along the x and y dimensions, and a BEV pooling operation is used to aggregate features in each BEV grid, and the features are unfolded along the z axis.

[0037] Step 3: Feature fusion: by introducing an attention mechanism, the radar and visual data obtained in steps 1 and 2 are attentionally encoded and fused in the BEV space to obtain comprehensive target information.

[0038] When performing feature fusion, as Figure 3As shown, in order to more effectively extract important features from the point cloud features and the picture features, the idea of spatial attention is applied, and the generated attention map can be obtained by selectively combining the features according to the relevance of the feature mapping and the target detection task, so as to reflect the relative importance of the image and the point cloud data, and more important point cloud features and picture features can be extracted according to the attention map. In order to effectively fuse the BEV features of the camera and the radar, the channel attention extraction method is used. For two features with different channel numbers, they are first stacked and connected, and then they are fused with learnable static weights, and important channels can be selected through the channel attention extraction method, so that more important fused features can be obtained.

[0039] Step 4: target detection: target detection is performed by using the fused BEV feature information.

[0040] Finally, the fused features are input into the detection head based on the transformer to obtain the final target detection result.

[0041] The application is a BEV feature layer fusion method based on attention, which applies a modal-specific backbone network to extract features corresponding to an image and a 4D millimeter wave radar respectively, then converts the features into unified BEV representation, fuses them through an attention-based mechanism, and finally outputs the target detection result through a detection head.

[0042] In the application, the processing of the radar point cloud and the processing of the image are independent, are encoded by using a neural network, are projected into a unified BEV space, and then are fused in the BEV space. In this case, the radar and the vision are not dependent on each other, so that the flexibility of approximate post-fusion can be realized: a single modality can complete the task independently, when multiple modalities are added, the performance will be greatly improved, but when a certain modality is missing or produces noise, it will not have a destructive result on the whole. The method also realizes adaptive fusion of radar and vision data by introducing an attention mechanism, and improves the accuracy of perception tasks such as target tracking and recognition.

[0043] The application will be further described in detail below in combination with specific embodiments.

[0044] Embodiment 1

[0045] Radar feature extraction: the voxel format is used to extract the radar backbone of the 4D millimeter wave radar data, and the BEV feature of the radar data is obtained.

[0046] Image feature extraction: image backbone feature extraction is performed on the image data to obtain 2D features of the image, and then each pixel in the image is predicted to have a depth probability distribution, each feature pixel is scattered to D discrete points along the camera light line, and the related features are scaled according to the corresponding depth probability, the image feature point cloud is obtained, then the image point cloud is quantized along the x, y dimensions, the BEV pooling operation is used to aggregate the features in each BEV grid, and then the features are unfolded along the z axis to obtain the features of the image in the BEV space.

[0047] Feature fusion: first, the importance of radar and picture features is weighed by weighting each pixel of the BEV features of the radar and the picture in space. Then, the extracted radar features and picture features are stacked in the channel, and the channel attention is learned to extract attention in the fused channel, and the weights of the features of the two sensors are reasonably distributed to further improve the detection accuracy.

[0048] Target detection: the fused features are input into the 3D target detection head to obtain the 3D target detection result.

[0049] The parts or structures not specifically described in the present application can adopt the prior art or existing products, which will not be described here. The above description is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for fusing attention-based 4D millimeter-wave radar and vision, characterized in that, Includes the following steps: Step 1: Radar Feature Extraction: The 4D millimeter-wave radar data is extracted using a voxel format to obtain the BEV features of the radar data, including: The input to the point cloud BEV feature extraction network is selected in the form of voxels. The BEV feature extraction network uses VoxelNet as the backbone network and adds a feature pyramid network. The backbone network divides the 3D point cloud into a certain number of voxels. After random sampling and normalization of the points, several voxel feature encoding layers are used for local feature extraction on each non-empty voxel to obtain voxel-level features. Then, the features are further abstracted by the 3D convolution module to obtain the BEV features of the point cloud. The BEV features are further refined by the feature pyramid network. Features are extracted from the BEV features by using a bottom-up path and combined and refined by a top-down path to obtain feature maps of different resolutions, all of which contain the semantic information of the original deepest feature map. Step 2: Image Feature Extraction: Extract the image backbone features from the image data to obtain the 2D features of the image. Project each image feature pixel back into 3D space to form the image BEV features, including: Using the Swin Transformer as the backbone network and adding a feature pyramid network layer, 2D features of the image are obtained. The depth information of the image is explicitly estimated through the view projection module to complete the construction of the image's BEV view. The backbone network model adopts a hierarchical structure with a total of 4 stages. Each stage reduces the resolution of the input feature map and gradually expands the receptive field. At the beginning of the input, Patch Embedding is performed to segment the image into several small blocks and embed them into the embedding. Each stage contains two parts: Patch merging and the Swin Transformer module. Step 3: Feature Fusion: By introducing an attention mechanism, the radar and visual data obtained in Step 1 and Step 2 are fused through attention encoding in the BEV space to obtain comprehensive target information; Step 4: Target detection: Target detection is performed using the fused BEV feature information.

2. The attention-based fusion method for 4D millimeter-wave radar and vision according to claim 1, characterized in that, The specific process of projecting each image feature pixel back into 3D space to form the BEV feature of the image is as follows: To generate a set of discrete depth values ​​for each pixel in the image, the method is as follows: within the view frustum of the camera, from 1 meter to 60 meters away, there is an optional depth value every 1 meter. N points are sampled along this line to predict the depth information of the feature points. This is represented by a D-dimensional vector through softmax, where D represents the number of distances in 1-meter intervals within the 1-60 meter range. The depth information obtained for each pixel is weighted with the image features at the same location to generate a pseudo-point cloud with a shape similar to a flat-topped pyramid. The camera's extrinsic and intrinsic parameters are used to transform the image with the previously obtained Frustum Feature to obtain the coordinates of the image in 3D space. Then, the image is flattened. The specific process is as follows: by limiting the range of the BEV viewpoint, the size of each grid is defined, and the features projected onto the corresponding grid are aggregated into one grid. In the top view, there may be multiple features in the same grid. The image point cloud is quantized along the x and y dimensions using a fixed step size. The BEV pooling operation is used to aggregate the features in each BEV grid, and the features are expanded along the z-axis.

3. The attention-based fusion method for 4D millimeter-wave radar and vision according to claim 1, characterized in that, Step 3 involves fusing radar and image features. The fusing steps include: In space, the features of the two are first superimposed, then global max pooling and global average pooling are applied respectively, followed by 3×3 convolution kernels, and then sigmoid activation operation to generate the final spatial attention feature map. This feature map is used to weight each pixel of the BEV features of radar and image. In each channel, the radar features and image features that have undergone spatial attention extraction are superimposed. Then, the channel weights are obtained through average pooling, 3×3 convolution kernels, and Sigmoid operations. The superimposed features are multiplied by these weights to obtain the final fused features.

Images