Cross multi-scale feature fusion millimeter wave radar target detection method combined with deformable convolution

Abstract

The invention discloses a cross multi-scale feature fusion millimeter wave radar target detection method in combination with deformable convolution. The traditional jump connection mainly carries out feature fusion at the same resolution level, on the basis, the scheme provides a cross-resolution feature map fusion method of an encoder-decoder structure, up-sampling jump connection is constructed between a deep encoder and a shallow decoder, and the method aims to strengthen the supplement of deep semantics to shallow details and improve the fusion efficiency of the deep encoder and the shallow decoder. And the detection capability of a shallow decoder on different scale targets (such as pedestrians and vehicles) is enhanced. And meanwhile, aiming at the characteristic that a radar image sequence contains rich time sequence characteristics, time sequence deformable convolution adopting unbounded dynamic weight is introduced into the deepest layer of the network, so that the motion modes of different targets can be better captured, and the expression ability of the deep characteristics is enhanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of millimeter wave radar data processing, and particularly relates to a millimeter wave radar target detection method combining cross multi-scale feature fusion and deformable convolution. BACKGROUND

[0002] Target detection aims to detect the categories and locations of different targets, and is a crucial task for applications such as intelligent driving, intelligent security, and unmanned aerial vehicle technology. In the early stage, many camera-based target detectors were widely used in intelligent driving systems. Although the camera sensor is conducive to better understanding of the semantic information of the visual scene, the quality of the collected images may be greatly reduced under some adverse conditions (such as fog, too strong or too weak light). Laser radar is another optical sensor applied in the automatic driving technology, and the radar point cloud data can be used to realize accurate target detection and positioning tasks. However, the laser radar has the disadvantages of high price and being easily affected by the environment. Compared with the first two sensors, the millimeter wave radar has the following advantages: (1) millimeter wave has strong penetration ability for non-metallic obstacles such as fog, smoke and dust, and can better cope with adverse environments; (2) the price is relatively low, and the volume and weight are small, which is more suitable for vehicle-mounted radar; (3) the millimeter wave radar has large bandwidth and high working frequency, and has good ranging and speed measuring capability. When the millimeter wave radar is used to realize the target detection task, there is no way to directly generate specific visual images. One method is to perform a series of preprocessing on the original ADC data to extract the distance, speed and angle information of the target, so as to generate range-azimuth angle maps (RAMaps) and range-Doppler maps (RDMaps). Unlike traditional radar point cloud-based detection, this method converts radar signals into a two-dimensional format similar to visual images, where each pixel corresponds to a position in the range-azimuth angle coordinate system (or range-Doppler coordinate system), and the pixel value represents the intensity of the radar reflection signal, which is similar to the pixel grid of a general visual image. In the target detection task on the radar image, the essence is to classify and regress each pixel, which is very similar to the pixel-level task in the image processing field. Therefore, the key to the millimeter wave radar target detection task is how to effectively extract and fuse the target features (such as shape size, motion pattern, etc.) of the radar image using a suitable method to achieve good detection effect.

[0003] Some researches have achieved outstanding results in radar target detection. RADDet, based on the improved ResNet (RadarResNet) combined with the double detection head of the YOLO anchor mechanism, directly predicts the 3D position, speed and category of dynamic targets through radar Range-Azimuth-Doppler (RAD) data for the first time, collects a RAD tensor radar data set, and uses an instance-level automatic labeling technology to label the data set. RODNet directly detects targets in radar images based on a 3D convolution stacking hourglass network supervised across modalities, and collects a data set CRUW of synchronized RGB images and radar image sequences in different scenes. However, the existing methods still have missed detection and false detection in complex scenes such as target overlap, night, and extreme weather, resulting in reduced detection accuracy.

[0004] To enhance the feature fusion effect and further improve the detection performance of the model in complex scenes, a cross multi-scale feature fusion millimeter wave radar target detection method combined with deformable convolution is disclosed. The traditional skip connection mainly performs feature fusion at the same resolution level, and on this basis, a cross-resolution feature map fusion method of an encoder-decoder structure is proposed in the scheme, an up-sampling skip connection is constructed between the deep encoder and the shallow decoder, aiming to strengthen the supplement of deep semantics to shallow details and enhance the detection ability of the shallow decoder to different scale targets (such as pedestrians and vehicles). Meanwhile, according to the characteristics that the radar image sequence contains rich temporal features, a temporal deformable convolution with unbounded dynamic weight is introduced in the deepest layer of the network, so as to better capture the motion patterns of different targets and enhance the expression ability of deep features. SUMMARY

[0005] The purpose of the present application is to improve the detection performance of radar in complex scenes. First, the radar image sequence is input into the designed feature fusion network, the local features of the target are extracted by the shallow encoder, the temporal motion patterns are captured by the deformable convolution of the deep encoder, the deep features are fused into the shallow decoder through the cross-resolution up-sampling skip connection, the multi-scale feature fusion is enhanced, the high-resolution feature map is generated, the confidence map is output by the decoder, the target category and position are predicted pixel by pixel, and finally the post-processing method based on the dynamic threshold non-maximum suppression of the target position is used to extract the final detection result.

[0006] The technical solution of the present application is:

[0007] A cross multi-scale feature fusion millimeter wave radar target detection method combined with deformable convolution, comprising:

[0008] Step 1: After pre-processing the millimeter wave radar data, a radar range-azimuth image sequence is obtained, which is input into the designed feature fusion network, and the local signal intensity, target edge and size and other features of the radar image are extracted by the shallow layer encoder through step-by-step downsampling; a time-varying convolution with unbounded dynamic weight is introduced in the deepest layer of the network to capture the time sequence motion features of the target;

[0009] Step 2: The shallow and deep features are fused by combining the dense skip connection of the same resolution level and the up-sampling skip connection across the resolution, to generate a high-resolution feature map, and the decoder outputs a confidence map to predict the target category and position probability pixel by pixel;

[0010] Step 3: The dynamic threshold non-maximum suppression based on the target position is applied to the confidence map, and the peak value is extracted as the detection result, each peak value corresponds to the category, position and confidence of a target.

[0011] Beneficial effects:

[0012] The application discloses a cross multi-scale feature fusion millimeter wave radar target detection method combined with deformable convolution, proposes a cross-resolution feature map fusion method of an encoder-decoder structure, and introduces a time-varying convolution with unbounded dynamic weight to enhance the deep feature expression, and simultaneously adopts a dynamic threshold as an evaluation standard in the post-processing link, so that the generalization ability of detection can be effectively improved.

[0013] (1) The traditional skip connection mainly performs feature fusion at the same resolution level, and the cross-resolution feature map fusion method of the encoder-decoder structure is proposed on the basis, and the up-sampling skip connection is constructed between the deep encoder and the shallow decoder, aiming at strengthening the supplement of the deep semantic to the shallow details and enhancing the detection ability of the shallow decoder to different scale targets (such as pedestrians and vehicles).

[0014] (2) According to the characteristics that the radar image sequence contains rich time sequence features, a time-varying convolution with unbounded dynamic weight is introduced in the deepest layer of the network, so as to better capture the motion mode of different targets and enhance the expression ability of the deep features.

[0015] (3) The dynamic threshold non-maximum suppression method based on the target position is adopted to post-process the confidence map, and the detection result is extracted, which can effectively reduce the missed detection caused by target overlap and too small target, so as to improve the generalization ability of detection. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 , the specific structure diagram of the cross multi-scale feature fusion network provided by the application;

[0017] Figure 2, a specific structure schematic diagram of a time sequence deformable convolution module provided by the application and adopting unbounded dynamic weights is provided;

[0018] Figure 3 , a whole flow chart of a cross multi-scale feature fusion millimeter wave radar target detection method combining deformable convolution is provided. DETAILED DESCRIPTION

[0019] The application will be described in detail below with reference to the drawings and examples.

[0020] The application discloses a cross multi-scale feature fusion millimeter wave radar target detection method combining deformable convolution, a designed feature fusion network extracts local features of a target through a shallow encoder, and a time sequence deformable convolution with unbounded dynamic weights is introduced in a deep encoder to capture a time sequence motion mode of the target, cross-resolution up-sampling skip connections fuse deep features to a shallow decoder, enhance multi-scale feature fusion, generate a high-resolution feature map, a decoder outputs a confidence map, pixel-by-pixel prediction of a target category and position, and finally, a post-processing method based on a dynamic threshold non-maximum suppression of the confidence map is used to extract a final detection result.

[0021] Step 0: acquisition of a radar image

[0022] After receiving and preprocessing a radar echo signal, the radar echo signal is subjected to FFT transformation in a distance dimension and a receiving antenna dimension in sequence, distance and azimuth information of a target is estimated, and finally, a range-azimuth map (RAMaps) is generated.

[0023] Step 1: after preprocessing millimeter wave radar data, a radar range-azimuth image sequence is obtained, and is input into a designed feature fusion network, a shallow encoder extracts local signal strength, target edge and size and the like of a radar image through step-by-step down-sampling; a time sequence deformable convolution with unbounded dynamic weights is introduced in the deepest layer of the network to capture time sequence motion features of a target.

[0024] The RAMaps sequence is input into the designed feature fusion network, and the size of each frame of radar image is 128*128. Figure 1 X in the formula i,j represents an encoder or decoder node, wherein i represents down-sampling depth, j represents a convolution layer index at the depth, X 0,0The encoder extracts features such as local signal strength and target edges from radar RAMaps through progressive downsampling with a pooling kernel size of 2×2×2. The convolution kernel of traditional convolution operations is regular. Due to the relative motion of the target, the position of the object in the radar range-azimuth coordinates in the temporal dimension will shift, and traditional convolution has difficulty fully capturing this temporal variation. Deformable convolution adjusts the shape of the convolution kernel by introducing a learnable offset, allowing the network to adaptively focus on specific areas of the input feature map. This paper extends deformable convolution to 3D convolution, and in the deepest layer of the network, X 4,0 The encoder introduces a temporal deformable convolution module TDCN with unbounded dynamic weights to capture the temporal motion characteristics of the target. The specific operation of this module is as follows:

[0025] Taking a 3D convolution kernel of size 3×3×3 (time-distance-azimuth) as an example, the conventional receptive field can be defined as

[0026] R={(-1,-1,-1),(-1,0,0),…,(0,1,1),(1,1,1)}

[0027] For each convolution kernel center p0 on the output feature map y, the traditional 3D convolution can be expressed as follows:

[0028]

[0029] Among them, w is the convolution kernel weight, x represents the trilinear interpolation sampling on the input feature map, and p n Each position of the defined receptive field R is enumerated.

[0030] like Figure 2 As shown, in order to dynamically adjust the sampling position in the temporal and spatial dimensions, so as to flexibly capture the temporal characteristics of the target, for each sampling point p n Generate 3D offset And use unbounded dynamic weight m gn Replace the original convolution kernel weight w(p n ), improve the training convergence speed, enhance the dynamics and expressiveness of the operator. A sub-network (1×1×1 convolution) is used to adjust the offset Δp gn and unbounded dynamic weight m gn Make a prediction, dynamically predict the input features, and predict the offset Δp gn and unbounded dynamic weight m gn Used to calculate the output feature map, the formula is as follows:

[0031]

[0032] where g represents the gth group, g = 1, 2,..., G, the input features are divided into G groups, and each group processes C in / G channels, the size of the single-group input feature map is C1x16x128x128, the channel number C1=C in / G, the size of the single-group output feature map is C2x16x128x128, the channel number C2=C out / G, the same group of channels share the offset and unbounded dynamic weight, and the memory access is reduced. After the grouped feature extraction, the final output is spliced by channel dimension, and then 1x1x1 fusion convolution is used to enhance the feature expression ability.

[0033] Step 2: Combine the dense skip connection at the same resolution level and the up-sampling skip connection across the resolution, fuse the shallow and deep features, generate high-resolution feature maps, and the decoder outputs the confidence map, which predicts the target class and position probability pixel by pixel.

[0034] As shown in Figure 1 , dense skip connections are used between the same resolution level encoder and decoder, and up-sampling skip connections across the resolution are constructed, up-sampling skip connections are constructed between the encoder node X 2,0 (X 3,0 , X 4,0 ) and the decoder node X 0,1 (X 1,1 , X 2,1 ), the shallow and deep features are fused, and high-resolution feature maps are generated. The specific operation of the up-sampling skip connection is introduced as follows:

[0035] 3D transpose convolution is used for up-sampling, taking the input size as D in xH in xW in , the convolution kernel size as K d xK h xK w , and the stride as (S d , S h , S w ) for example. First, insert S d -1, S h -1, S w -1 zeros between every two adjacent volume pixels in the depth, height, and width dimensions to obtain an intermediate quantity with a size of D'xH'xW', where

[0036] D' = D in +(D in -1) x (S d -1),

[0037] H' = Hin +(H in -1)×(S h -1),

[0038] W′=W in +(W in -1)×(S w -1).

[0039] Then, according to the required output size, zeros are filled in the depth, height, and width dimensions to ensure that the desired size can be obtained after convolution; finally, 3D convolution is performed on the intermediate amount after zero padding to obtain the upsampling result.

[0040] Take encoder node X 2,0 and decoder node X 0,1 For example, the radar image is generated by encoder X 0,0 After two downsampling, it reaches the encoder X 2,0 , by upsampling the convolution kernel with a size of 4×4×4 and a stride of (4, 4, 4) so ​​that the node X 2,0 Output feature map Resolution and Node X 0,1 Consistent, upsampled feature map Then with X 0,0 The feature map Conf 0,0 and X 1,0 The result of upsampling Splicing along the channel dimension to obtain the final fused feature map can be expressed as

[0041]

[0042] Here, Upsample represents the upsampling operation, and H(·) represents the concatenation operation along the channel dimension.

[0043] The fused feature map is passed through the decoder to output the confidence map Predict object category and location probabilities pixel by pixel.

[0044] Step 3: Apply dynamic threshold non-maximum suppression based on target position to the confidence map and extract peaks as detection results. Each peak corresponds to the category, position and confidence of a target.

[0045] like Figure 3In the last step, a post-processing method of non-maximum suppression based on dynamic threshold of target position is used to extract peaks of the predicted confidence map as detection results. Each peak corresponds to the category, position and confidence of a target. In conventional image target detection, non-maximum suppression is used to eliminate overlapping bounding boxes and retain the bounding box with the highest confidence. However, there is no clear definition of bounding boxes in radar images, and it is more suitable to use the center point to locate the target. Therefore, non-maximum suppression based on target position can be used to identify and filter radar detection results, and the similarity index is modified from the intersection-over-union (IoU) between bounding boxes to the target position similarity (OLS). The conventional operation is to manually set a fixed OLS threshold to evaluate the similarity of two points, which will result in poor generalization ability of detection in different scenarios. Therefore, the present invention proposes a non-maximum suppression using a dynamic OLS threshold. The specific operation is as follows:

[0046] (1) Use a 3×3 sliding window to detect the local maximum of the confidence map of each channel to form a candidate peak set

[0047] (2) Filter out the peak q with the highest confidence * And remove it from the set; then calculate q according to the following formula * and each of the remaining q i The OLS value of

[0048]

[0049] Where d is the distance between two detection points in the radar image, s is the distance from the target to the sensor, and k cls is the error tolerance for a specific target class, which represents the typical deviation in position detection for that class of targets.

[0050] (3) If OLS is greater than the threshold T OLS , indicating that the two points are too close and may belong to the same target, then delete q i Otherwise, keep q i In order to improve the generalization ability of detection, a dynamic OLS threshold T is set OLS , the formula is as follows,

[0051]

[0052] Among them, T0 is the basic threshold, which is set to 0.5; c is the confidence of the current peak, ranging from 0 to 1; d i is the distance between the target and the sensor; d max is the maximum possible detection distance, used for normalization; α and β are adjustment coefficients, α controls the impact of confidence on the threshold, and β controls the impact of distance on the threshold, ranging from 0 to 1.

[0053] (4) repeat steps (2) and (3) until the candidate peak set Q is empty.

[0054] To sum up, the above is only the preferred embodiment of the present application, not for limiting the scope of protection of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A millimeter-wave radar target detection method combining cross-multi-scale feature fusion with deformable convolution, characterized in that: The following steps are involved: Step 1: Preprocess the millimeter-wave radar data to obtain a radar range-azimuth image sequence, which is then fed into the designed feature fusion network. The shallow encoder extracts radar image features such as local signal strength, target edges, and size through step-by-step downsampling. A temporal deformable convolution with unbounded dynamic weights is introduced in the deepest layer of the network to capture the temporal motion characteristics of the target; Step 2: Combine dense skip connections at the same resolution level and upsampling skip connections across resolutions to fuse shallow and deep features and generate high-resolution feature maps. The decoder outputs confidence maps and predicts the object category and location probability pixel by pixel. Step 3: Apply dynamic threshold non-maximum suppression based on target position to the confidence map and extract peaks as detection results. Each peak corresponds to the category, position and confidence of a target.

2. The cross-multi-scale feature fusion millimeter-wave radar target detection method combined with deformable convolution as claimed in claim 1 is characterized in that: In step 1, the deformable convolution is extended to 3D convolution, and the deepest layer X of the network 4,0 The encoder introduces a temporal deformable convolution module TDCN with unbounded dynamic weights to capture the temporal motion characteristics of the target. The specific operation of this module is as follows: The conventional receptive field is defined as R = {(-1,-1,-1), (-1,0,0),…, (0,1,1), (1,1,1)} For each convolution kernel center p0 on the output feature map y, the traditional 3D convolution is expressed as Among them, w is the convolution kernel weight, x represents the trilinear interpolation sampling on the input feature map, and p n Each position of the defined receptive field R is enumerated.

3. The cross-multi-scale feature fusion millimeter-wave radar target detection method combined with deformable convolution as claimed in claim 1 is characterized in that: In step 1, for each sampling point p n Generate 3D offset And use unbounded dynamic weight m gn Replace the original convolution kernel weight w(p n ).

4. The method for millimeter-wave radar target detection using cross-multi-scale feature fusion combined with deformable convolution as claimed in claim 1, characterized in that: In step 1, a sub-network (1×1×1 convolution) is used to offset Δp gn and unbounded dynamic weight m gn Make a prediction, dynamically predict the input features, and predict the offset Δp gn and unbounded dynamic weight m gn Used to calculate the output feature map, the formula is as follows: Among them, g represents the g-th group, g=1,2,...,G, the input features are divided into G groups, and each group processes C in / G channels, the size of a single set of input feature maps is C1×16×128×128, the number of channels C1=C in / G, the size of a single group of output feature maps is C2×16×128×128, and the number of channels C2=C out / G, the same group of channels share offsets and unbounded dynamic weights to reduce memory access; after group feature extraction, the final output is concatenated between groups. The features are concatenated according to the channel dimension and then enhanced through 1×1×1 fusion convolution.

5. The cross-multi-scale feature fusion millimeter-wave radar target detection method combined with deformable convolution as claimed in claim 1 is characterized in that: In step 2, dense skip connections are used between the encoder and decoder at the same resolution level, and upsampling skip connections across resolutions are constructed. 2,0 (X 3,0 , X 4,0 ) and decoder node X 0,1 (X 1 ,1 , X 2,1 ) to build an upsampling skip connection between the layers, fuse shallow and deep features, and generate a high-resolution feature map. The specific operations are as follows: 3D transposed convolution is used for upsampling, with input size D in ×H in ×W in , the convolution kernel size is K d ×K h ×K w , the stride is (S d ,S h ,S w ) as an example, first, in the depth, height, and width dimensions, S is inserted between every two adjacent volume pixels. d -1, S h -1, S w -1 zero value, resulting in an intermediate quantity with dimensions D′×H′×W′, where D′=D in +(D in -1)×(S d -1), H′=H in +(H in -1)×(S h -1), W′=W in +(W in -1)×(S w -1). Then, according to the required output size, zeros are filled in the depth, height, and width dimensions to ensure that the desired size can be obtained after convolution; finally, 3D convolution is performed on the intermediate amount after zero padding to obtain the upsampling result.

6. The method for millimeter-wave radar target detection using cross-multi-scale feature fusion combined with deformable convolution as claimed in claim 1, characterized in that: In step 2, the radar image is generated by encoder X 0,0 After two downsampling, it reaches the encoder X 2,0 , by upsampling the convolution kernel with a size of 4×4×4 and a stride of (4, 4, 4) so ​​that the node X 2,0 Output feature map Resolution and Node X 0,1 Consistent, upsampled feature map Then with X 0,0 The feature map Conf 0,0 and X 1,0 The result of upsampling Splicing along the channel dimension to obtain the final fused feature map, which can be expressed as Here, Upsample represents the upsampling operation, and H(·) represents the concatenation operation along the channel dimension.

7. The method for millimeter-wave radar target detection using cross-multi-scale feature fusion combined with deformable convolution as claimed in claim 1, characterized in that: In step 2, the fused feature map is passed through the decoder to output the confidence map Predict object category and location probabilities pixel by pixel.

8. The method for millimeter-wave radar target detection using cross-multi-scale feature fusion combined with deformable convolution as claimed in claim 1, characterized in that: In step 3, a post-processing method based on dynamic threshold non-maximum suppression of the target position is used to extract peaks from the predicted confidence map as detection results. Each peak corresponds to the category, position and confidence of a target.

9. The method for millimeter-wave radar target detection using cross-multi-scale feature fusion combined with deformable convolution as claimed in claim 1, characterized in that: In step 3, a non-maximum suppression method using dynamic OLS threshold is proposed. The specific operation is as follows: (1) Use a 3×3 sliding window to detect the local maximum of the confidence map of each channel to form a candidate peak set (2) Filter out the peak q with the highest confidence * And remove it from the set; then calculate q according to the following formula * and each of the remaining q i The OLS value of Where d is the distance between two detection points in the radar image, s is the distance from the target to the sensor, and k cls is the error tolerance of a specific target class, which represents the typical deviation of the position detection of this type of target; (3) If OLS is greater than the threshold T OLS , indicating that the two points are too close and may belong to the same target, then delete q i Otherwise, keep q i ; To improve the generalization ability of detection, a dynamic OLS threshold T is set OLS , the formula is as follows, Among them, T0 is the basic threshold, which is set to 0.5; c is the confidence of the current peak, ranging from 0 to 1; d i is the distance between the target and the sensor; d max is the maximum possible detection distance, used for normalization; α and β are adjustment coefficients, α controls the impact of confidence on the threshold, and β controls the impact of distance on the threshold, ranging from 0 to 1; (4) Repeat steps (2) and (3) until the candidate peak set Q is empty.

Images