Transformer-based millimeter-wave radar target detection method

By using a Transformer-based bounding box and center key point detection model, the problems of insufficient feature extraction capability and low angular resolution in millimeter-wave radar target detection are solved, achieving more efficient target detection.

CN118506147BActive Publication Date: 2026-11-17XIDIAN UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311856480.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-29
Publication Date
2026-11-17
Estimated Expiration
2043-12-29

AI Technical Summary

Technical Problem

In existing millimeter-wave radar target detection methods, convolutional neural networks have limited ability to extract features from radar spectra, especially when background clutter interference is severe, which affects their performance. In addition, insufficient angular resolution leads to numerous false alarms and inaccurate target detection.

Method used

We employ a bounding box detection model and a center key point detection model based on the Transformer architecture. We use Fourier transform and position encoding of the range-azimuth-Doppler millimeter-wave radar spectrum, combined with Transformer Encoder and Decoder blocks for feature extraction and decoding. We use complex linear layers to replace the Fourier transform and optimize the model parameters.

Benefits of technology

It improves the feature extraction capability of radar target detection, reduces the false alarm rate, increases angular resolution, and enhances detection performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118506147B_ABST
    Figure CN118506147B_ABST
Patent Text Reader

Abstract

This invention provides a Transformer-based millimeter-wave radar target detection method to address the technical problems of limited feature extraction capabilities of convolutional neural networks for radar spectra and insufficient angular resolution in existing millimeter-wave radar target detection methods. The Transformer-based millimeter-wave radar target detection method provided by this invention establishes a bounding box detection model or a center keypoint detection model based on the Transformer architecture, and inputs the millimeter-wave radar spectra into the corresponding model for training to better learn global features and obtain better feature extraction capabilities. Then, the preprocessed millimeter-wave radar spectra to be detected are input into the trained bounding box detection model to obtain the model's prediction results. Post-processing of the model's prediction results yields the final detection result, completing the millimeter-wave radar target detection. The Transformer-based millimeter-wave radar target detection method provided by this invention effectively improves the model's detection performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to radar target detection technology, and more particularly to a millimeter-wave radar target detection method based on Transformer. Background Technology

[0002] The primary objective of radar target detection is to obtain information such as the target's position, velocity, and azimuth by analyzing the energy of the target's echo. Traditional radar target detection mainly relies on conventional pulse radar and lidar, using statistical signal processing or deep learning methods to detect and identify targets based on extracted radar target point cloud data. In recent years, with the rapid development of millimeter-wave radar technology, millimeter-wave radar sensors have received increasing attention due to their advantages such as all-weather operation and greater robustness to adverse weather and lighting conditions.

[0003] With the improvement of millimeter-wave radar imaging capabilities and the rapid development of deep learning technology, target detection methods based on millimeter-wave radar have also undergone significant changes. More and more research is employing deep learning methods to directly detect targets on range-Doppler (RD), range-azimuth (RA), or range-Azimuth-Doppler (RAD) spectra. Compared with traditional millimeter-wave radar data target detection and recognition methods based on statistical signal processing and point clouds, spectra-based methods can effectively avoid problems such as manually designing constant false alarm rate detectors, loss of point cloud data information, and incorrect clustering of point cloud data. This helps to fully utilize radar echo information, obtain better target representations through learning, and achieve accurate spatial positioning and classification of targets.

[0004] The rapid development of deep learning technology in recent years has also driven the transformation of millimeter-wave radar data analysis methods from traditional multi-stage processing (constant false alarm rate detection - direction of arrival estimation - point cloud clustering - target recognition) to end-to-end deep learning frameworks. The RADDet dataset is the first millimeter-wave radar dataset to use bounding box annotation. Current detection algorithms for this dataset mainly treat the Doppler dimension as a channel in the range-angle-Doppler spectrum, using a 2D residual convolutional neural network for feature extraction. This is followed by two branches: one branch uses a 3D detector to detect the bounding box in the range-azimuth-Doppler spectrum; the other branch is followed by a coordinate transformation layer from polar coordinates to Cartesian coordinates, and then uses a 2D detector to detect the bounding box in the Cartesian coordinate bird's-eye view. While this algorithm simultaneously handles 3D and 2D target detection tasks, using the Doppler dimension as a channel in 3D tasks is clearly not ideal. The velocity represented by the Doppler dimension is an important attribute of the target, and its features obviously need to be extracted. On the other hand, the characteristic that radar echo signal intensity varies with factors such as distance means that convolutional neural networks, which have comparable performance for visible light images, may not be well applied to radar spectral maps.

[0005] Currently, most detection algorithms on the CRUW dataset focus on detecting key points at the center of radar targets on the range-azimuth spectrum. These algorithms primarily involve feature extraction and decoding using convolutional neural networks. Due to poor detection performance on single-frame range-azimuth spectra, subsequent algorithms incorporated a temporal dimension, performing detection across multiple frames of range-azimuth spectra to improve performance. However, insufficient angular resolution remains a significant weakness of millimeter-wave radar, hindering its ability to distinguish adjacent targets and acquire fine-grained target structure information. This is the primary cause of target misclassification and inaccurate localization, fundamentally limiting the performance of radar target detection.

[0006] In summary, existing millimeter-wave radar target detection methods have the following main shortcomings:

[0007] (1) Due to the characteristics of radar spectrum data, the convolutional neural network used in existing radar target detection methods has limited ability to extract features from radar spectrum, especially on radar spectrum with severe background clutter interference, the performance of feature extraction is more severely affected, and there may be more false alarms.

[0008] (2) Due to insufficient angular resolution, the performance of target detection tasks on radar spectra is limited. Under fixed hardware conditions, how to further improve the angular resolution of radar spectra data through algorithms is an important issue for improving the target detection capability of millimeter-wave radar. Summary of the Invention

[0009] The purpose of this invention is to solve the technical problems of limited feature extraction capability of convolutional neural networks for radar spectrum and insufficient angular resolution in existing millimeter-wave radar target detection methods, and to provide a millimeter-wave radar target detection method based on Transformer.

[0010] To achieve the above objectives, the technical solution provided by this invention is as follows:

[0011] A Transformer-based millimeter-wave radar target detection method, characterized by the following steps:

[0012] Step 1: Perform range-azimuth-Doppler Fourier transform, Doppler Fourier transform, azimuth Fourier transform, and normalization on the raw analog-to-digital conversion data of the millimeter-wave radar in the RADDet dataset to obtain range-azimuth-Doppler millimeter-wave radar spectral data.

[0013] Step 2: Establish a bounding box detection model based on the Transformer architecture. The bounding box detection model includes a Transformer Encoder block with an M-layer hierarchical structure, used for feature extraction. Convolutional layers are used between the Transformer Encoder blocks in each hierarchical structure for downsampling operations. The output of the M-th layer Transformer Encoder block is connected to the bounding box detection head. Wherein, M ≥ 3.

[0014] Step 3: Training the bounding box detection model;

[0015] 3.1) Input the range-azimuth-Doppler millimeter-wave radar spectrum obtained in step 1 into the bounding box detection model established in step 2;

[0016] 3.2) Perform Patch Embedding operation on the input range-azimuth-Doppler millimeter-wave radar spectrum, divide it into vector-represented blocks, and perform position encoding on all vector-represented blocks; feed the feature map of each block and the corresponding position encoding into the first-level Transformer Encoder block, and construct the relationship between the sequences through the self-attention mechanism of the Transformer Encoder block;

[0017] 3.3) After the Transformer Encoder blocks of the first to M-1 hierarchical structures, downsampling operations are performed by convolutional layers. At the same time, the feature maps in the Transformer Encoder blocks of each hierarchical structure are processed according to the method in step 3.2 until the feature map in the Transformer Encoder block of the M-1 hierarchical structure is divided into blocks and position encoded. The feature map after adding the sequence of each block and the corresponding position encoding is sent to the Transformer Encoder block of the M hierarchical structure. The feature map received in the Transformer Encoder block of the M hierarchical structure is sent to the bounding box detection head.

[0018] 3.4) By analyzing the feature map data fed into the bounding box detection head, the regression and classification results are analyzed, and then the loss function is calculated; the loss function is passed to each layer of the hierarchical structure through the backpropagation algorithm to update the bounding box detection model parameters;

[0019] Step 4: Process the analog-to-digital conversion data of the millimeter-wave radar spectrum to be detected using the method in Step 1, and input the processed range-azimuth-Doppler millimeter-wave radar spectrum into the bounding box detection model trained in Step 3 to obtain the model prediction results. Perform post-processing on the model prediction results to obtain the final detection results and complete the millimeter-wave radar target detection.

[0020] Furthermore, in step 2, M = 3;

[0021] In step 3.2, each Transformer Encoder block in the hierarchical structure includes two sub-blocks. The first sub-block uses a 3D window attention mechanism, and the second sub-block uses a 3D moving window attention mechanism.

[0022] In step 3.2, during each downsampling operation, the feature map is downsampled to half its original size;

[0023] In step 4, the prediction results are post-processed to perform non-maximum suppression.

[0024] Furthermore, in step 1, the raw analog-to-digital conversion data of the millimeter-wave radar in the RADDet dataset undergoes range-dimensional Fourier transform, Doppler-dimensional Fourier transform, azimuth-dimensional Fourier transform, and normalization processing, specifically as follows:

[0025] Set up a complex linear layer for azimuth-dimensional Fourier transform; first, directly perform range-dimensional Fourier transform and Doppler-dimensional Fourier transform on the original analog-to-digital conversion data of millimeter-wave radar in the RADDet dataset, then perform azimuth-dimensional Fourier transform through the complex linear layer, and add a normalization layer after the complex linear layer to normalize the data.

[0026] Furthermore, in step 3.4, the error is calculated based on the output of the complex linear layer and the expected standard Fourier transform result, and the obtained error is added to the final loss function of the bounding box detection model so that the weights of the complex linear layer approximate the weights of the Fourier transform.

[0027] Alternatively, in step 3.1, the weights of the complex linear layer are initialized to Fourier transform weights so that the bounding box detection model can automatically optimize the weights during training.

[0028] Furthermore, in step 3.2, the position code PE is implemented using the following formula:

[0029]

[0030]

[0031] Where pos is the position of the current block in the sequence, 2i is the dimension index of the block represented by the even-numbered position vector, 2i+1 is the dimension index of the block represented by the odd-numbered position vector, and d model The dimension of the current block.

[0032] This invention also provides a millimeter-wave radar target detection method based on Transformer, which is characterized by including the following steps:

[0033] Step 1: Perform range-azimuth Fourier transform, azimuth Fourier transform and normalization on the raw analog-to-digital conversion data of millimeter-wave radar in the CRUW dataset to obtain the range-azimuth millimeter-wave radar spectrum.

[0034] Step 2: Establish a center keypoint detection model based on the Transformer architecture. This model includes a Transformer Encoder block with a P-layer hierarchical structure and a Transformer Decoder block with a Q-layer hierarchical structure. The Transformer Encoder block performs encoding, and convolutional layers are used between its hierarchical Transformer Encoder blocks for downsampling. The Transformer Decoder block performs decoding, and deconvolutional layers are used between its hierarchical Transformer Decoder blocks for upsampling. The output of the P-th layer Transformer Encoder block is connected to the input of the first layer Transformer Decoder block, and the output of the Q-th layer Transformer Decoder block is connected to the center keypoint detection head. Where P ≥ 3, Q = P - 1.

[0035] Step 3: Training the center key point detection model;

[0036] 3.1) Input the range-azimuth millimeter-wave radar spectrum obtained in step 1 into the center key point detection model established in step 2;

[0037] 3.2) Perform Patch Embedding operation on the input range-azimuth millimeter-wave radar spectrum, divide it into vector-represented blocks, and perform position encoding on all vector-represented blocks; feed the feature map of each block, which is the sum of the sequence and the corresponding position encoding, into the Transformer Encoder block of the first-level hierarchical structure, and construct the relationship between the sequences through the self-attention mechanism of the Transformer Encoder block;

[0038] 3.3) After the Transformer Encoder blocks of the first to P-1 layer hierarchical structures, the convolutional layers perform downsampling operations. At the same time, the feature maps in the Transformer Encoder blocks of each layer hierarchical structure are processed according to the method in step 3.2 until the feature map in the Transformer Encoder block of the P-1 layer hierarchical structure is divided into blocks and position encoded. Then, the feature map after adding the sequence of each block and the corresponding position encoding is sent into the Transformer Encoder block of the P layer hierarchical structure.

[0039] 3.4) Perform PatchEmbedding operation on the feature map output from the Transformer Encoder block of the P-th layer structure, divide it into vector-represented blocks, and perform positional encoding on all vector-represented blocks; feed the feature map after adding the sequence of each block and the corresponding positional encoding into the Transformer Decoder block of the first layer structure, and at the same time, build the relationship between the sequences through the self-attention mechanism of the Transformer Encoder block, and build the relationship between the Transformer Encoder block and the Transformer Decoder block through the cross-attention of the Transformer Decoder block, so as to perform decoding operation;

[0040] 3.5) Upsampling is performed on the Transformer Decoder blocks from the first to the Qth layer using deconvolution layers. Simultaneously, the feature maps in each Transformer Decoder block are processed according to step 3.4 until the feature map in the (Q-1)th layer's Transformer Decoder block is segmented and its position encoded. The feature map obtained by adding the sequence of each block and its corresponding position encoding is then fed into the Qth layer's Transformer Decoder block. The feature map received from the Qth layer's Transformer Decoder block is then fed into the center keypoint detection head.

[0041] 3.6) By analyzing the feature map data fed into the central keypoint detection head, the regression and classification results are analyzed, and then the loss function is calculated; the loss function is passed to each layer of the hierarchical structure through the backpropagation algorithm to update the parameters of the central keypoint detection model;

[0042] Step 4: Process the analog-to-digital conversion data of the millimeter-wave radar spectrum to be detected using the method in Step 1, and input the processed range-azimuth millimeter-wave radar spectrum into the center key point detection model trained in Step 3 to obtain the model prediction results. Perform post-processing on the model prediction results to obtain the final detection results and complete the millimeter-wave radar target detection.

[0043] Furthermore, in step 2, P = 3, Q = 2; and linear layers are set between the Transformer Decoder blocks of each hierarchical structure.

[0044] In step 3.2, the Transformer Encoder block and the Transformer Decoder block of each hierarchical structure are divided into two sub-blocks. The first sub-block uses the three-dimensional window attention mechanism, and the second sub-block uses the three-dimensional moving window attention mechanism.

[0045] In step 3.3, during each downsampling operation, the feature map is downsampled to half its original size;

[0046] In step 3.5, during each upsampling operation, the feature map is upsampled to twice its original size;

[0047] In step 4, the prediction results are post-processed by extracting the eight-neighbor peak of the thermal value and performing location-based non-maximum suppression.

[0048] Furthermore, in step 1, the raw analog-to-digital conversion data of the millimeter-wave radar in the CRUW dataset undergoes range-dimensional Fourier transform, azimuth-dimensional Fourier transform, and normalization processing, specifically as follows:

[0049] Set up a complex linear layer for azimuth-dimensional Fourier transform; first, directly perform range-dimensional Fourier transform on the original analog-to-digital conversion data of millimeter-wave radar in the CRUW dataset, then perform azimuth-dimensional Fourier transform through the complex linear layer, and add a normalization layer after the complex linear layer to normalize the data.

[0050] In step 3.6, the error is calculated based on the output of the complex linear layer and the expected standard Fourier transform result, and the obtained error is added to the final loss function of the center key point detection model so that the weights of the complex linear layer approximate the weights of the Fourier transform.

[0051] Alternatively, in step 3.1, the weights of the complex linear layer are initialized to Fourier transform weights so that the center key point detection model can automatically optimize the weights during training.

[0052] Furthermore, in step 3, the position code PE is implemented using the following formula:

[0053]

[0054]

[0055] Where pos is the position of the current block in the sequence, 2i is the dimension index of the block represented by the even-numbered position vector, 2i+1 is the dimension index of the block represented by the odd-numbered position vector, and d model The dimension of the current block.

[0056] The advantages of this invention compared to the prior art are as follows:

[0057] 1. This invention addresses the problem that background clutter, caused by the characteristics of millimeter-wave radar spectral data, significantly interferes with detection performance and easily leads to false alarms. It provides two Transformer-based millimeter-wave radar target detection methods, namely, a bounding box detection model and a center key point detection model based on the Transformer architecture, and inputs the millimeter-wave radar spectral data into the corresponding models for training. This allows for better learning of global features, resulting in better feature extraction capabilities and effectively improving the detection performance of the models.

[0058] 2. Under fixed hardware conditions, this invention uses a complex linear layer (neural network) instead of a Fourier transform, which enables the model to learn a better transform, effectively improving angular resolution and further enhancing model performance. Attached Figure Description

[0059] Figure 1 This is a flowchart of an embodiment of a millimeter-wave radar target detection method based on Transformer;

[0060] Figure 2 This is a flowchart illustrating the preprocessing of raw analog-to-digital conversion data from a millimeter-wave radar in Implementation Example 1 of a Transformer-based millimeter-wave radar target detection method.

[0061] Figure 3 This is a flowchart of a second embodiment of a millimeter-wave radar target detection method based on Transformer. Detailed Implementation

[0062] To make the advantages and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] Example 1

[0064] A Transformer-based millimeter-wave radar target detection method employs bounding box detection, such as... Figure 1 As shown, the specific steps include:

[0065] Step 1: Preprocess the raw analog-to-digital conversion data of the millimeter-wave radar.

[0066] Bounding box detection typically uses the RADDet dataset. The raw analog-to-digital conversion data of millimeter-wave radar in the RADDet dataset is subjected to range-dimensional Fourier transform, Doppler-dimensional Fourier transform, azimuth-dimensional Fourier transform, and normalization processing to obtain the range-azimuth-Doppler millimeter-wave radar spectrum.

[0067] Since changes in azimuth angle have a significant impact on the model during millimeter-wave radar target detection, this embodiment incorporates a complex linear layer simulating the azimuth angle Fourier transform to ensure azimuth accuracy and reduce model response time. For example... Figure 2 As shown, in this embodiment, when transforming the analog-to-digital conversion data in the RADDet dataset, a distance-dimensional Fourier transform and a Doppler-dimensional Fourier transform are performed directly first. Then, a complex linear layer simulates the azimuth-dimensional Fourier transform, and a normalization layer is added after the complex linear layer to normalize the data. This allows the model to learn a better azimuth-dimensional Fourier transform by replacing the traditional azimuth-dimensional Fourier transform with a complex linear layer, i.e., by using a neural network instead of the traditional azimuth-dimensional Fourier transform, under fixed hardware conditions. This effectively improves the angle resolution capability and enhances model performance. Even if the model performance is not improved, the difference in the dimensionality of the angle dimension before and after the Fourier transform effectively reduces the storage size of the model input data, which is valuable for subsequent applications on the platform.

[0068] In other embodiments of the present invention, a complex linear layer simulating azimuth-dimensional Fourier transform and distance-dimensional Fourier transform can also be set, or a complex linear layer simulating azimuth-dimensional Fourier transform and Doppler-dimensional Fourier transform can be set, or a complex linear layer simulating azimuth-dimensional Fourier transform, Doppler-dimensional Fourier transform, and distance-dimensional Fourier transform can be set. Although this setting can also enable the model to learn a better Fourier transform, the complexity of the model will be significantly increased, thereby increasing the model response time. Alternatively, in other embodiments of the present invention, a complex linear layer can be omitted, and the analog-to-digital conversion data in the RADDet dataset can be directly subjected to distance-dimensional Fourier transform, Doppler-dimensional Fourier transform, and azimuth-dimensional Fourier transform. After the transformation, the input data is normalized using the mean and variance of the overall data. However, this setting cannot enable model learning, and the accuracy of the model's prediction results may be affected to some extent.

[0069] This embodiment uses raw analog-to-digital conversion data from the RADDet dataset for millimeter-wave radar. Each frame has a size of 256*64*8, where 256 represents the number of sampling points per period and 64 represents the number of periods (64 periods constitute one frame). The millimeter-wave radar sensor has two transmitting antennas and four receiving antennas, so the original angular dimension is 8. First, a range-dimensional Fourier transform is performed, i.e., a 256-point Fast Fourier Transform within each period. Then, a Doppler-dimensional Fourier Transform is performed, i.e., a 64-point Fast Fourier Transform within each period. The data format is now 8 range-Doppler spectra. Next, the data is input into a complex linear layer to simulate an azimuth-dimensional Fourier transform. Since the number of points in the azimuth-dimensional Fourier transform and the dimension of the data annotation in the azimuth dimension of the original dataset are both 256, the dimension of the complex linear layer in this embodiment is 256.

[0070] Step 2: Establish a bounding box detection model based on the Transformer architecture.

[0071] The bounding box detection model in this invention includes a Transformer Encoder block with an M-layer hierarchical structure for feature extraction. This embodiment uses a Transformer Encoder block with a 3-layer hierarchical structure. Convolutional layers are used between the Transformer Encoder blocks for downsampling operations, and the output of the third-layer Transformer Encoder block is connected to the bounding box detection head.

[0072] In this embodiment, the bounding box detection head consists of three stacked 3D convolutional layers. The last 3D convolutional layer is used to perform bounding box regression and target category classification. The structure of the bounding box detection head in this invention is not limited to the above form. In other embodiments of this invention, the bounding box detection head can also be configured as independent regression and classification heads, each composed of three stacked convolutional layers.

[0073] Step 3: Training the bounding box detection model.

[0074] 3.1) Input the range-azimuth-Doppler millimeter-wave radar spectrum obtained in step 1 into the bounding box detection model established in step 2.

[0075] 3.2) Before inputting the range-azimuth-Doppler millimeter-wave radar spectrum into the TransformerEncoder block of the M-layer hierarchical structure, a Patch Embedding operation is performed on the range-azimuth-Doppler millimeter-wave radar spectrum to divide it into vector-represented blocks (tokens). That is, the input features of the millimeter-wave radar spectrum are divided into x*x*x blocks, where x represents the size of each dimension of the block.

[0076] Since each dimension of a millimeter-wave radar spectrum has a specific physical meaning, and the target's position within the spectrum represents different physical attributes and exhibits different characteristics, it is necessary to perform positional encoding on each vector-represented block to characterize the positional relationships between blocks. This allows the Transformer Encoder block to recognize the positional order of radar spectrum features. In this embodiment, a linear transformation of trigonometric sin and cosine functions is used to perform positional encoding on each vector-represented block. In other embodiments of this invention, adaptive positional encoding can also be used, enabling the model to autonomously learn complex positional representations from the data.

[0077] The positional encoding of each vector block is performed using linear transformations of the trigonometric sin and cosine functions. These vectors are distinguished based on the parity of their positions. Therefore, the positional encoding PE is represented as follows:

[0078]

[0079]

[0080] Where pos is the position of the current block in the sequence, 2i is the dimension index of the even-positioned block, 2i+1 is the dimension index of the odd-positioned block, and d model The dimension of the current block.

[0081] Then, the feature map obtained by adding the sequence of each block and the corresponding position encoding is fed into the Transformer Encoder block of the first layer of the hierarchical structure, and the relationship between the sequences is constructed through the self-attention mechanism of the Transformer Encoder block; at the same time, the output sequence is expanded to the size of the input.

[0082] 3.3) After the first and second layer Transformer Encoder blocks in the convolutional layers, downsampling operations are performed, with each downsampling operation reducing the feature map to half its original size. In this invention, Patch Embedding is performed on the feature map obtained from each downsampling operation, dividing its input features into x*x*x blocks. Since the size of the output feature map changes with each layer, each block requires positional encoding. After Patch Embedding and positional encoding, the feature map is fed into the next layer's Transformer Encoder block. The Transformer Encoder block's self-attention mechanism builds relationships between sequences until Patch Embedding and positional encoding are performed on the feature map in the second layer's Transformer Encoder block. The feature map resulting from the sum of the sequence and the corresponding positional encoding for each block is then fed into the third layer's Transformer Encoder block. Finally, the feature map received from the third layer's Transformer Encoder block is fed into the bounding box detection head.

[0083] 3.4) By inputting feature map data into the bounding box detection head, the regression and classification results are analyzed, and the loss function is calculated. The loss function is then passed to each hierarchical structure through the backpropagation algorithm to update the bounding box detection model parameters and achieve the training of the bounding box detection model.

[0084] Because calculating self-attention on the entire feature map is too complex, in this embodiment, each layer of the Transformer Encoder block includes two sub-blocks. The first sub-block uses a 3D window attention mechanism, and the second sub-block uses a 3D moving window attention mechanism. The 3D window attention mechanism calculates the self-attention of each window, and the 3D moving window attention mechanism calculates the self-attention of the entire image. These two mechanisms significantly reduce the computational complexity of self-attention.

[0085] Meanwhile, to enable the complex linear layer to learn an effect similar to the Fourier transform, this embodiment initializes the weights of the complex linear layer to Fourier transform weights during training. This allows the bounding box detection model to automatically optimize the weights during training, obtaining a better Fourier transform. In other embodiments of the invention, the error can be calculated based on the output of the complex linear layer and the desired standard Fourier transform result, and the obtained error can be added to the final loss function of the bounding box detection model to make the weights of the complex linear layer approximate the Fourier transform weights.

[0086] Step 4: Preprocess the analog-to-digital conversion data of the millimeter-wave radar spectrum to be detected using the method in Step 1. Input the preprocessed millimeter-wave radar spectrum into the bounding box detection model trained in Step 3 to obtain the model prediction results. Perform post-processing such as non-maximum suppression on the model prediction results to remove some prediction results with low confidence, and obtain the final detection results, thus completing the millimeter-wave radar target detection.

[0087] The effectiveness of the millimeter-wave radar target detection method based on Transformer bounding box detection in this invention is verified by several sets of experimental data in Tables 1 and 2.

[0088] Table 1 shows the average accuracy of different categories under different cross-union ratio thresholds in this embodiment.

[0089]

[0090]

[0091] Table 2 Comparison of this embodiment with other detection algorithms

[0092]

[0093] As shown in Table 1, this embodiment effectively improves the feature extraction capability of the model on the RADDet dataset, and achieves high detection accuracy for each category of targets. Although the bounding box used as the ground truth is the bounding box of the spectrum after Fourier transform, the model's performance is not bad. This reflects that using complex linear layers instead of Fourier transform and allowing the network to learn a better transform on its own further improves the angular resolution.

[0094] As shown in Table 2, the average accuracy of the present invention across all categories is improved compared to the previous detection algorithm.

[0095] Example 2

[0096] This invention also provides a millimeter-wave radar target detection method based on Transformer, such as... Figure 3 As shown, center key point detection is used, which specifically includes the following steps:

[0097] Step 1: Preprocess the raw analog-to-digital conversion data of the millimeter-wave radar.

[0098] Center key point detection typically uses the CRUW dataset. Since the CRUW dataset only includes the target's range and azimuth, it only detects the target's position on the range-azimuth spectrum. This involves performing range-dimensional Fourier transform, azimuth-dimensional Fourier transform, and normalization on the analog-to-digital conversion data in the CRUW dataset to obtain the range-azimuth millimeter-wave radar spectrum.

[0099] This embodiment includes a complex linear layer simulating the azimuth-dimensional Fourier transform. When transforming the analog-to-digital conversion data in the CRUW dataset, a distance-dimensional Fourier transform is first performed directly, followed by a complex linear layer simulating the azimuth-dimensional Fourier transform. A normalization layer is then added after the complex linear layer to normalize the data. This allows the model to learn a better azimuth-dimensional Fourier transform by replacing the traditional azimuth-dimensional Fourier transform with a complex linear layer, i.e., by using a neural network instead of the traditional azimuth-dimensional Fourier transform, under fixed hardware conditions. This effectively improves the angle resolution and enhances model performance. Even if model performance is not improved, the difference in the dimensionality of the angle dimension before and after the Fourier transform effectively reduces the storage size of the model input data, which is valuable for subsequent applications on the platform. In other embodiments of this invention, complex linear layers simulating both the azimuth-dimensional and distance-dimensional Fourier transforms can be included, or the complex linear layer can be omitted, and the distance-dimensional and azimuth-dimensional Fourier transforms can be performed directly on the analog-to-digital conversion data in the RADDet dataset.

[0100] In this embodiment, the single input data is 16 frames of distance-azimuth spectrum. Time information is added to the 16 frames of distance-azimuth spectrum to improve detection performance and then normalized.

[0101] Step 2: Establish a center key point detection model based on the Transformer architecture.

[0102] The center keypoint detection model in this invention includes a Transformer Encoder block with a P-layer hierarchical structure and a Transformer Decoder block with a Q-layer hierarchical structure. The Transformer Encoder block is used for encoding, with convolutional layers used for downsampling between its layers. The Transformer Decoder block is used for decoding, with deconvolutional layers used for upsampling between its layers. Where P ≥ 3, Q = P - 1, in this embodiment P = 3, Q = 2, the Transformer Encoder block has a three-layer hierarchical structure (three encoding layers), and the Transformer Decoder block has a two-layer hierarchical structure (two decoding layers). Simultaneously, linear layers are set between each layer of the Transformer Decoder block to compress the feature map to a size that can be received by the next layer's Transformer Decoder block.

[0103] The output of the Transformer Encoder block in the third-level hierarchy is connected to the input of the Transformer Decoder block in the first-level hierarchy, and the output of the Transformer Decoder block in the second-level hierarchy is connected to the center key point detection head.

[0104] In this embodiment, the central key point detection head is composed of three convolutional blocks stacked together. The number of channels in the last convolutional block is the number of categories, and the channel in which the final prediction result is located is the category to which it belongs.

[0105] Step 3: Training the center key point detection model;

[0106] 3.1) Input the range-azimuth millimeter-wave radar spectrum obtained in step 1 into the center key point detection model established in step 2.

[0107] 3.2) Before inputting the range-azimuth millimeter-wave radar spectrum into the three-layer hierarchical TransformerEncoder block, a Patch Embedding operation is performed on the range-azimuth millimeter-wave radar spectrum to divide it into vector-represented blocks (tokens). That is, the input features of the millimeter-wave radar spectrum are divided into x*x*x blocks, where x represents the size of each dimension of the block.

[0108] Since each dimension of a millimeter-wave radar spectrum has a specific physical meaning, and the target's position within the spectrum represents different physical attributes and exhibits different characteristics, it is necessary to perform positional encoding on each vector-represented block to characterize the positional relationships between blocks. This allows the Transformer Encoder block to recognize the positional order of radar spectrum features. In this embodiment, a linear transformation of trigonometric sin and cosine functions is used to perform positional encoding on each vector-represented block. In other embodiments of this invention, adaptive positional encoding can also be used, enabling the model to autonomously learn complex positional representations from the data.

[0109] The positional encoding of each vector-represented block is specifically performed using a linear transformation of the trigonometric sin and cosine functions:

[0110]

[0111]

[0112] Where pos is the position of the current block in the sequence, 2i is the dimension index of the block represented by the even-numbered position vector, 2i+1 is the dimension index of the block represented by the odd-numbered position vector, and d model The dimension of the current block.

[0113] Then, the feature map obtained by adding the sequence of each block and the corresponding position encoding is fed into the Transformer Encoder block of the first layer of the hierarchical structure. The relationship between the sequences is constructed through the self-attention mechanism of the Transformer Encoder block, and the output sequence is expanded to the size of the input.

[0114] 3.3) Downsampling operations are performed on the feature maps after the first and second layer Transformer Encoder blocks via convolutional layers. Each downsampling operation downsamples the feature map to half its original size. In this invention, a Patch Embedding operation is performed on the feature maps obtained from each downsampling operation, dividing the input features into x*x*x blocks. Since the size of the output feature maps changes with each layer, positional encoding is required for each block after division. After Patch Embedding and positional encoding, the blocks are fed into the next layer's Transformer Encoder block. The self-attention mechanism of the Transformer Encoder block is used to build relationships between sequences until Patch Embedding and positional encoding are performed on the feature maps in the second layer's Transformer Encoder block. Finally, the feature map resulting from the sum of the sequence and the corresponding positional encoding of each block is fed into the third layer's Transformer Encoder block.

[0115] 3.4) Perform PatchEmbedding operation on the feature map output from the Transformer Encoder block of the third-level structure, divide it into vector-represented blocks, and perform positional encoding on all vector-represented blocks; feed the feature map after adding the sequence of each block and the corresponding positional encoding into the Transformer Decoder block of the first-level structure, and at the same time, build the relationship between the sequences through the self-attention mechanism of the Transformer Encoder block, and build the relationship between the Transformer Encoder block and the Transformer Decoder block through the cross-attention of the Transformer Decoder block, so as to perform the decoding operation.

[0116] 3.5) Upsampling operations are performed by deconvolutional layers before the Transformer Decoder blocks in the first and second hierarchical structures. Each upsampling operation upsamples the feature map to twice its original size. The upsampling operation before the Transformer Decoder block in the second hierarchical structure restores the feature map to its original size. After performing patch embedding and positional encoding on the feature map obtained from each upsampling operation, it is fed into the next level Transformer Decoder block. At the same time, the relationship between sequences is constructed through the self-attention mechanism of the Transformer Encoder block, and the relationship between the Transformer Encoder block and the Transformer Decoder block is constructed through the cross-attention mechanism of the Transformer Decoder block for decoding operations. This continues until the feature map in the Transformer Decoder block of the Q-1 hierarchical structure is divided into blocks and positionally encoded. The feature map obtained by adding the sequence of each block and the corresponding positional encoding is then fed into the Transformer Decoder block of the Q hierarchical structure. In this embodiment, it is fed into the Transformer Decoder block of the second hierarchical structure. At this point, the feature map received from the TransformerDecoder block of the second-level structure is sent to the center key point detection head.

[0117] 3.6) By inputting the feature map data of the center keypoint detection head, the regression and classification results are analyzed, and then the loss function is calculated; the loss function is passed to each layer of the hierarchical structure through the backpropagation algorithm to update the parameters of the center keypoint detection model and realize the training of the center keypoint detection model.

[0118] Because calculating self-attention on the entire feature map is computationally too complex, this embodiment divides each layer of the Transformer Encoder block and each layer of the Transformer Decoder block into two sub-blocks. The first sub-block uses a 3D window attention mechanism, and the second sub-block uses a 3D moving window attention mechanism. The 3D window attention mechanism calculates the self-attention of each window, and the 3D moving window attention mechanism calculates the self-attention of the entire image. These two mechanisms significantly reduce the computational complexity of self-attention.

[0119] Meanwhile, to enable the complex linear layer to learn an effect similar to the Fourier transform, when simulating the Fourier transform in the azimuth dimension using the complex linear layer, this embodiment initializes the weights of the complex linear layer to Fourier transform weights during training. This allows the center keypoint detection model to automatically optimize the weights during training, obtaining a better Fourier transform. In other embodiments of the invention, the error can be calculated based on the output of the complex linear layer and the desired standard Fourier transform result, and the obtained error can be added to the final loss function of the center keypoint detection model, so that the weights of the complex linear layer approximate the weights of the Fourier transform.

[0120] Step 4: Preprocess the millimeter-wave radar spectrum to be detected using the method in Step 1. Input the preprocessed millimeter-wave radar spectrum into the center key point detection model trained in Step 3 to obtain the model prediction results. Since the prediction results of the center key point detection head are in the form of a heatmap, it is necessary to perform eight-neighbor peak extraction of the heatmap value and position-based non-maximum suppression processing based on the model prediction results to obtain the final detection results, thus completing the millimeter-wave radar target detection.

[0121] The following experimental data in Table 3 verify the effectiveness of the millimeter-wave radar target detection method based on Transformer center key point detection of the present invention.

[0122] Table 3 shows the detection results for each category in this embodiment and a comparison of the average accuracy and recall with other detection algorithms.

[0123]

[0124] As shown in Table 3, this embodiment achieves an average precision of 0.818 and an average recall of 0.863 across all categories on the CRUW millimeter-wave radar general dataset. Compared to previous millimeter-wave radar target detection algorithms based on center key point detection, this embodiment demonstrates improvements in both average precision and recall.

[0125] This invention addresses the issues of insufficient angular resolution and characteristics of radar spectral data. Based on a hierarchical Transformer, it enables better learning of millimeter-wave radar features. Simultaneously, it uses complex linear layers to replace the Fourier transform in preprocessing, allowing the neural network to learn potentially better transforms through self-optimization, thereby improving angular resolution and further enhancing the model's detection performance.

[0126] The above description is only used to illustrate the technical solutions of the present invention, and is not intended to limit them. For those skilled in the art, modifications can be made to the specific technical solutions described in the above embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions protected by the present invention.

Claims

1. A method for target detection of a millimeter wave radar based on a Transformer, characterized in that, Includes the following steps: Step 1: Perform range-azimuth-Doppler Fourier transform, Doppler Fourier transform, azimuth Fourier transform, and normalization on the raw analog-to-digital conversion data of the millimeter-wave radar in the RADDet dataset to obtain range-azimuth-Doppler millimeter-wave radar spectral data. Step 2: Establish a bounding box detection model based on the Transformer architecture. The bounding box detection model includes a Transformer Encoder block with an M-layer hierarchical structure, used for feature extraction. Convolutional layers are used between the Transformer Encoder blocks in each hierarchical structure for downsampling operations. The output of the M-th layer Transformer Encoder block is connected to the bounding box detection head. Wherein, M ≥ 3. Step 3: Training the bounding box detection model; 3.1) Input the range-azimuth-Doppler millimeter-wave radar spectrum obtained in step 1 into the bounding box detection model established in step 2; 3.2) Perform Patch Embedding operation on the input range-azimuth-Doppler millimeter-wave radar spectrum, divide it into vector-represented blocks, and perform position encoding on all vector-represented blocks; feed the feature map of each block and the corresponding position encoding into the first-level Transformer Encoder block, and construct the relationship between the sequences through the self-attention mechanism of the Transformer Encoder block; 3.3) After the Transformer Encoder blocks of the first to M-1 hierarchical structures, downsampling operations are performed by convolutional layers. At the same time, the feature maps in the Transformer Encoder blocks of each hierarchical structure are processed according to the method in step 3.2 until the feature map in the Transformer Encoder block of the M-1 hierarchical structure is divided into blocks and position encoded. The feature map after adding the sequence of each block and the corresponding position encoding is sent to the Transformer Encoder block of the M hierarchical structure. The feature map received in the Transformer Encoder block of the M hierarchical structure is sent to the bounding box detection head. 3.4) By analyzing the feature map data fed into the bounding box detection head, the regression and classification results are analyzed, and then the loss function is calculated; the loss function is passed to each layer of the hierarchical structure through the backpropagation algorithm to update the bounding box detection model parameters; Step 4: Process the analog-to-digital conversion data of the millimeter-wave radar spectrum to be detected using the method in Step 1, and input the processed range-azimuth-Doppler millimeter-wave radar spectrum into the bounding box detection model trained in Step 3 to obtain the model prediction results. Perform post-processing on the model prediction results to obtain the final detection results and complete the millimeter-wave radar target detection.

2. The millimeter-wave radar target detection method based on Transformer according to claim 1, characterized in that: In step 2, M = 3; In step 3.2, each Transformer Encoder block in the hierarchical structure includes two sub-blocks. The first sub-block uses a 3D window attention mechanism, and the second sub-block uses a 3D moving window attention mechanism. In step 3.3, during each downsampling operation, the feature map is downsampled to half its original size; In step 4, the prediction results are post-processed to perform non-maximum suppression.

3. The millimeter-wave radar target detection method based on Transformer according to claim 1 or 2, characterized in that: In step 1, the raw analog-to-digital conversion data of millimeter-wave radar in the RADDet dataset undergoes range-dimensional Fourier transform, Doppler-dimensional Fourier transform, azimuth-dimensional Fourier transform, and normalization processing, specifically as follows: Set up a complex linear layer for azimuth-dimensional Fourier transform; first, directly perform range-dimensional Fourier transform and Doppler-dimensional Fourier transform on the original analog-to-digital conversion data of millimeter-wave radar in the RADDet dataset, then perform azimuth-dimensional Fourier transform through the complex linear layer, and add a normalization layer after the complex linear layer to normalize the data.

4. The millimeter-wave radar target detection method based on Transformer according to claim 3, characterized in that: In step 3.4, the error is calculated based on the output of the complex linear layer and the expected standard Fourier transform result, and the obtained error is added to the final loss function of the bounding box detection model so that the weights of the complex linear layer approximate the weights of the Fourier transform. Alternatively, in step 3.1, the weights of the complex linear layer are initialized to Fourier transform weights so that the bounding box detection model can automatically optimize the weights during training.

5. The millimeter-wave radar target detection method based on Transformer according to claim 4, characterized in that: In step 3.2, the position code PE is implemented using the following formula: Where pos is the position of the current block in the sequence, 2i is the dimension index of the block represented by the even-numbered position vector, 2i+1 is the dimension index of the block represented by the odd-numbered position vector, and d model The dimension of the current block. 6.A method for target detection of a millimeter wave radar based on a Transformer, characterized in that, Includes the following steps: Step 1: Perform range-azimuth Fourier transform, azimuth Fourier transform and normalization on the raw analog-to-digital conversion data of millimeter-wave radar in the CRUW dataset to obtain the range-azimuth millimeter-wave radar spectrum. Step 2: Establish a center keypoint detection model based on the Transformer architecture. This model includes a Transformer Encoder block with a P-layer hierarchical structure and a Transformer Decoder block with a Q-layer hierarchical structure. The Transformer Encoder block performs encoding, and convolutional layers are used between its hierarchical Transformer Encoder blocks for downsampling. The Transformer Decoder block performs decoding, and deconvolutional layers are used between its hierarchical Transformer Decoder blocks for upsampling. The output of the P-th layer Transformer Encoder block is connected to the input of the first layer Transformer Decoder block, and the output of the Q-th layer Transformer Decoder block is connected to the center keypoint detection head. Where P ≥ 3, Q = P - 1. Step 3: Training the center key point detection model; 3.1) Input the range-azimuth millimeter-wave radar spectrum obtained in step 1 into the center key point detection model established in step 2; 3.2) Perform Patch Embedding operation on the input range-azimuth millimeter-wave radar spectrum, divide it into vector-represented blocks, and perform position encoding on all vector-represented blocks; feed the feature map of each block and the corresponding position encoding into the first-level Transformer Encoder block, and construct the relationship between the sequences through the self-attention mechanism of the Transformer Encoder block; 3.3) After the Transformer Encoder blocks of the first to P-1 layer hierarchical structures, the convolutional layers perform downsampling operations. At the same time, the feature maps in the Transformer Encoder blocks of each layer hierarchical structure are processed according to the method in step 3.2 until the feature map in the Transformer Encoder block of the P-1 layer hierarchical structure is divided into blocks and position encoded. Then, the feature map after adding the sequence of each block and the corresponding position encoding is sent into the Transformer Encoder block of the P layer hierarchical structure. 3.4) Perform PatchEmbedding operation on the feature map output from the Transformer Encoder block of the P-th layer structure, divide it into vector-represented blocks, and perform positional encoding on all vector-represented blocks; feed the feature map after adding the sequence of each block and the corresponding positional encoding into the Transformer Decoder block of the first layer structure, and at the same time, build the relationship between the sequences through the self-attention mechanism of the Transformer Encoder block, and build the relationship between the Transformer Encoder block and the Transformer Decoder block through the cross-attention of the Transformer Decoder block, so as to perform decoding operation; 3.5) Upsampling is performed on the Transformer Decoder blocks from the first to the Qth layer using deconvolution layers. Simultaneously, the feature maps in each Transformer Decoder block are processed according to step 3.4 until the feature map in the (Q-1)th layer's Transformer Decoder block is segmented and its position encoded. The feature map obtained by adding the sequence of each block and its corresponding position encoding is then fed into the Qth layer's Transformer Decoder block. The feature map received from the Qth layer's Transformer Decoder block is then fed into the center keypoint detection head. 3.6) By analyzing the feature map data fed into the central keypoint detection head, the regression and classification results are analyzed, and then the loss function is calculated; the loss function is passed to each layer of the hierarchical structure through the backpropagation algorithm to update the parameters of the central keypoint detection model; Step 4: Process the analog-to-digital conversion data of the millimeter-wave radar spectrum to be detected using the method in Step 1, and input the processed range-azimuth millimeter-wave radar spectrum into the center key point detection model trained in Step 3 to obtain the model prediction results. Perform post-processing on the model prediction results to obtain the final detection results and complete the millimeter-wave radar target detection.

7. The millimeter-wave radar target detection method based on Transformer according to claim 6, characterized in that: In step 2, P = 3 and Q = 2; linear layers are set between the Transformer Decoder blocks of each hierarchical structure; In step 3.2, the Transformer Encoder block and the Transformer Decoder block of each hierarchical structure are divided into two sub-blocks. The first sub-block uses the three-dimensional window attention mechanism, and the second sub-block uses the three-dimensional moving window attention mechanism. In step 3.3, during each downsampling operation, the feature map is downsampled to half its original size; In step 3.5, during each upsampling operation, the feature map is upsampled to twice its original size; In step 4, the prediction results are post-processed by extracting the eight-neighbor peak of the thermal value and performing location-based non-maximum suppression.

8. The millimeter-wave radar target detection method based on Transformer according to claim 6 or 7, characterized in that: In step 1, the raw analog-to-digital conversion data of millimeter-wave radar in the CRUW dataset undergoes range-dimensional Fourier transform, azimuth-dimensional Fourier transform, and normalization processing, specifically as follows: Set up a complex linear layer for azimuth-dimensional Fourier transform; first, directly perform range-dimensional Fourier transform on the original analog-to-digital conversion data of millimeter-wave radar in the CRUW dataset, then perform azimuth-dimensional Fourier transform through the complex linear layer, and add a normalization layer after the complex linear layer to normalize the data.

9. The millimeter-wave radar target detection method based on Transformer according to claim 8, characterized in that: In step 3.6, the error is calculated based on the output of the complex linear layer and the expected standard Fourier transform result, and the obtained error is added to the final loss function of the center key point detection model so that the weights of the complex linear layer approximate the weights of the Fourier transform. Alternatively, in step 3.1, the weights of the complex linear layer are initialized to Fourier transform weights so that the center key point detection model can automatically optimize the weights during training.

10. The millimeter-wave radar target detection method based on Transformer according to claim 9, characterized in that: In step 3, the position code PE is implemented using the following formula: Where pos is the position of the current block in the sequence, 2i is the dimension index of the block represented by the even-numbered position vector, 2i+1 is the dimension index of the block represented by the odd-numbered position vector, and d model The dimension of the current block.

Citation Information

Patent Citations

  • Radar radio frequency image target detection method

    CN114842196A

  • Multi-sensor target detection method based on visual self-attention

    CN115880551A