Method for target detection based on millimeter wave radar

Abstract

The application discloses a target detection method based on a millimeter wave radar, acquires no-label radar heat map data and labeled radar heat map data; an encoder network is constructed, the encoder network is trained on the no-label radar heat map data in a contrast learning mode to obtain encoder network parameters, and the encoder network parameters are set as unlearnable; a decoder network is constructed, the decoder network is connected with the trained encoder network to obtain a coding and decoding network, and the coding and decoding network is trained by using the labeled radar heat map data to obtain decoder network parameters; the trained coding and decoding network is copied to obtain a teacher network and a student network, and the teacher network and the student network are trained by using the no-label radar heat map data and the labeled radar heat map data, so that the trained student network is used for target detection; and thus reliable radar target detection can be realized without relying on a large number of labeled data samples.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a target detection method based on millimeter-wave radar, a computer-readable storage medium, and a computer device. Background Technology

[0002] Millimeter-wave radar-based target detection technology is of significant practical importance for the realization of autonomous driving technology. It is unaffected by adverse weather conditions, possessing all-weather, all-day environmental perception capabilities. This technology can reliably detect vehicles, pedestrians, and other obstacles on the road under adverse conditions such as nighttime, rain, snow, and heavy fog, contributing to improved safety of autonomous driving systems. However, achieving accurate target detection often relies on a large amount of labeled radar data, which is inherently difficult to obtain. First, radar heatmap data itself makes it difficult to intuitively identify the type, shape, and location of targets, making manual annotation complex and time-consuming. Second, using cameras as supervision to automatically generate annotation information is often inaccurate. Camera identification of target type and location heavily depends on the performance of visual algorithms; furthermore, cameras are susceptible to target occlusion, which significantly affects detection results—a problem absent in millimeter-wave radar heatmap data, easily leading to mislabeling and omissions. Therefore, how to achieve reliable target detection based on a small amount of labeled radar data and a large amount of unlabeled radar data is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. Therefore, one objective of this invention is to propose a target detection method based on millimeter-wave radar that combines self-supervised learning, supervised learning, and semi-supervised learning, thereby achieving reliable radar target detection without relying on a large number of labeled data samples.

[0004] A second objective of this invention is to provide a computer-readable storage medium.

[0005] The third objective of this invention is to provide a computer device.

[0006] To achieve the above objectives, a first aspect of the present invention proposes a target detection method based on millimeter-wave radar, comprising the following steps: acquiring unlabeled radar heatmap data and labeled radar heatmap data; constructing an encoder network, training the encoder network on the unlabeled radar heatmap data using a contrastive learning approach to obtain the encoder network parameters, and setting the encoder network parameters to be unlearnable; constructing a decoder network, connecting the decoder network to the trained encoder network to obtain an encoder-decoder network, and training the encoder-decoder network using the labeled radar heatmap data to obtain the decoder network parameters; replicating the trained encoder-decoder network to obtain a teacher network and a student network, and training the teacher network and student network using the unlabeled radar heatmap data and the labeled radar heatmap data, so as to use the trained student network for target detection.

[0007] According to an embodiment of the present invention, a target detection method based on millimeter-wave radar first acquires unlabeled and labeled radar heatmap data. Then, an encoder network is constructed and trained on the unlabeled radar heatmap data using contrastive learning to obtain encoder network parameters, which are then set to be non-learnable. Next, a decoder network is constructed and connected to the trained encoder network to obtain an encoder-decoder network. The encoder-decoder network is then trained using labeled radar heatmap data to obtain decoder network parameters. Finally, the trained encoder-decoder network is replicated to obtain a teacher network and a student network. Both the teacher and student networks are trained using unlabeled and labeled radar heatmap data, so that the trained student network can be used for target detection. Thus, by combining self-supervised learning, supervised learning, and semi-supervised learning, reliable radar target detection can be achieved without relying on a large number of labeled data samples.

[0008] In addition, the target detection method based on millimeter-wave radar proposed in the above embodiments of the present invention may also have the following additional technical features:

[0009] Optionally, acquiring unlabeled and labeled radar heatmap data includes: acquiring raw echo data collected by a millimeter-wave radar sensor; performing range-domain Fast Fourier Transform (FFT) processing on the raw echo data; performing low-pass filtering on the FFT-processed echo data to remove high-frequency noise; performing angle-domain FFT processing on the low-pass-filtered echo data to obtain the unlabeled radar heatmap data; and selecting from published labeled radar heatmap datasets to collect radar heatmap data segments with accurate labeling information to obtain the labeled radar heatmap data.

[0010] Optionally, the encoder network is trained on the unlabeled radar heatmap data using a contrastive learning approach, including: performing a first data augmentation on the unlabeled radar heatmap data to obtain a first data sequence, and performing a second data augmentation on the unlabeled radar heatmap data to obtain a second data sequence; passing the first data sequence through the encoder network and a projection MLP head to obtain a first projection feature, and passing the second data sequence through the encoder network and a projection MLP head to obtain a second projection feature; passing the first projection feature through a prediction MLP head to obtain a first prediction feature, and passing the second projection feature through a prediction MLP head to obtain a second prediction feature; and calculating the loss function of the contrastive learning based on the first prediction feature, the first projection feature, the second prediction feature, and the second projection feature.

[0011] Optionally, the loss function for the contrastive learning is obtained according to the following formula:

[0012]

[0013]

[0014]

[0015] Among them, z u1 Indicates the first projective feature; z u2 Indicates the second projective feature; p u1 p represents the first predictive feature. u2 represents the second predicted feature; ||·||2 represents the L2 norm; (·) represents the dot product of vectors; stopgrad represents stopping the gradient operation.

[0016] Optionally, training the codec network using the labeled radar heatmap data includes: predicting the labeled radar heatmap data through the codec network to obtain a predicted sequence; calculating the cross-entropy loss function between the predicted sequence and the real labeled sequence; and updating the decoder network parameters through the backpropagation process of the gradient.

[0017] Alternatively, the cross-entropy loss function can be obtained according to the following formula:

[0018]

[0019] Among them, y l (cls,i,j) represents the predicted probability that category cls appears in the i-th row and j-th column of the radar heatmap; p l (cls,i,j) represents the probability that category cls appears in the i-th row and j-th column of the labeled radar heatmap.

[0020] Optionally, the teacher network and student network are trained using the unlabeled radar heatmap data and the labeled radar heatmap data, including: performing a first data augmentation on the unlabeled radar heatmap data to obtain a first data sequence, and performing a second data augmentation on the unlabeled radar heatmap data to obtain a second data sequence; forward propagating the first data sequence through the teacher network to generate a first pseudo-label, and forward propagating the second data sequence through the student network to generate a first predicted value; calculating the mean squared error loss between the first predicted value and the first pseudo-label, and updating the student network parameters through backpropagation of gradients, while using an exponential moving average. The teacher network parameters are updated using the following methods: The labeled radar heatmap data undergoes a third data augmentation to obtain a third data sequence, and a fourth data augmentation is performed to obtain a fourth data sequence; the third data sequence is forward-propagated through the teacher network to generate a second pseudo-label, and the fourth data sequence is forward-propagated through the student network to generate a second predicted value; the weighted sum of the cross-entropy loss between the second predicted value and the true label, and the mean squared error loss between the second predicted value and the second pseudo-label is calculated, and the student network parameters are updated through backpropagation of the gradient, while the teacher network parameters are updated using an exponential moving average method.

[0021] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium storing a target detection program based on millimeter-wave radar, which, when executed by a processor, implements the target detection method based on millimeter-wave radar as described above.

[0022] To achieve the above objectives, a third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the target detection method based on millimeter-wave radar as described above. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a target detection method based on millimeter-wave radar according to an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of a self-supervised contrastive learning process according to an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of a supervised learning process according to an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of a semi-supervised learning process according to an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the architecture of a codec network according to an embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram of the architecture of a short-time feature fusion module according to an embodiment of the present invention;

[0029] Figure 7 This is a schematic diagram of the architecture of a feature extraction module according to an embodiment of the present invention. Detailed Implementation

[0030] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0031] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0032] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0033] Figure 1 This is a flowchart illustrating a target detection method based on millimeter-wave radar according to an embodiment of the present invention, as shown below. Figure 1 As shown, the target detection method based on millimeter-wave radar includes the following steps:

[0034] S101, acquire unlabeled radar heat map data and labeled radar heat map data.

[0035] As one embodiment, acquiring unlabeled and labeled radar heatmap data includes: acquiring raw echo data collected by a millimeter-wave radar sensor; performing range-domain Fast Fourier Transform (FFT) processing on the raw echo data; performing low-pass filtering on the FFT-processed echo data to remove high-frequency noise; performing angle-domain FFT processing on the low-pass-filtered echo data to obtain unlabeled radar heatmap data; and selecting from published labeled radar heatmap datasets to collect radar heatmap data segments with accurate labeling information to obtain labeled radar heatmap data.

[0036] It should be noted that the amount of unlabeled radar heatmap data acquired is much larger than that of labeled radar heatmap data, while the amount of labeled radar heatmap data acquired is much smaller than that of unlabeled radar heatmap data.

[0037] In other words, a large amount of raw echo data is collected using millimeter-wave radar sensors in various real-world environments. The collected raw echo data is then subjected to a range-domain Fast Fourier Transform (FFT) and low-pass filtering to remove high-frequency noise. Subsequently, an angle-domain FFT is performed to obtain a large amount of unlabeled radar heatmap data. At the same time, existing published labeled radar heatmap datasets are selected, and radar heatmap data fragments with accurate labeling information are collected as a small amount of labeled radar heatmap data.

[0038] S102, Construct an encoder network. Use contrastive learning to train the encoder network on unlabeled radar heatmap data to obtain encoder network parameters, and set the encoder network parameters to be non-learnable.

[0039] In other words, a large-scale encoder network is trained on a large amount of unlabeled radar heatmap data using a contrastive learning method.

[0040] As one embodiment, an encoder network is trained on unlabeled radar heatmap data using a contrastive learning approach, including: performing a first data augmentation on the unlabeled radar heatmap data to obtain a first data sequence, and performing a second data augmentation on the unlabeled radar heatmap data to obtain a second data sequence; passing the first data sequence through the encoder network and a projection MLP head to obtain a first projected feature, and passing the second data sequence through the encoder network and a projection MLP head to obtain a second projected feature; passing the first projected feature through a prediction MLP head to obtain a first predicted feature, and passing the second projected feature through a prediction MLP head to obtain a second predicted feature; and calculating a loss function for contrastive learning based on the first predicted feature, the relationship between the first projected features, and the relationship between the second predicted feature and the second projected feature.

[0041] In other words, firstly, the unlabeled data sequence x u Performing two different data augmentation operations yields a processed pair of data sequences x. u1 x u2 , where x u x u1 x u2 ∈R B×C×T×chirp×H×WB represents the batch size, C represents the number of channels in the radar heatmap, T represents the number of radar heatmap frame sequences, chirp represents the number of chirps in a frame, and H and W represent the height and width of the radar heatmap, respectively. The data augmentation method adopts random scale cropping and enlarging back to the original size, random horizontal flipping, and random addition of Gaussian noise. The cropping scale is uniformly sampled in the interval [s, 1]. The variance of the Gaussian noise is taken as g times the variance of the current batch of samples, and the mean is taken as the mean of the current batch of samples.

[0042] Then, x u1 x u2 This pair of data sequences is used to obtain the projected feature z through an encoder network and a projection MLP head. u1 , z u2 Projecting feature z u1 , z u2 The predicted feature p is obtained through a predictive MLP head. u1 p u2 Among them, z u1 , z u2 p u1 p u2 ∈R B×d Where B represents the batch size and d represents the feature dimension.

[0043] Next, the cosine similarity between the predicted features and the projected features is calculated as the loss function for contrastive learning. The formula for the loss function is as follows:

[0044]

[0045]

[0046]

[0047] Among them, z u1 Indicates the first projective feature; z u2 Indicates the second projective feature; p u1 p represents the first predictive feature. u2 represents the second predicted feature; ||·||2 represents the L2 norm; (·) represents the dot product of vectors; stopgrad represents stopping the gradient operation.

[0048] Finally, this process is used to train a large-scale encoder for a certain number of epochs on the collected unlabeled dataset, and the final encoder network parameters are saved.

[0049] S103, Construct a decoder network, connect the decoder network with the trained encoder network to obtain an encoder-decoder network, and train the encoder-decoder network using labeled radar heatmap data to obtain the decoder network parameters.

[0050] In other words, the encoder network parameters are frozen and a smaller decoder network is added afterward to build a complete encoder-decoder network, and the encoder-decoder network is trained in a supervised manner using a small amount of labeled radar heatmap data.

[0051] As an example, the codec network is trained using labeled radar heatmap data, including: predicting the labeled radar heatmap data through the codec network to obtain a predicted sequence; calculating the cross-entropy loss function between the predicted sequence and the real labeled sequence; and updating the decoder network parameters through the backpropagation process of the gradient.

[0052] In other words, firstly, the encoder network parameters are set to be non-learnable to ensure that their parameters are not updated during gradient backpropagation, while a smaller-scale decoder network is constructed. The encoder and decoder networks are then connected to form an encoder-decoder architecture network.

[0053] Next, a small number of labeled data sequences x l The predicted sequence y is obtained by using this encoding / decoding network. l Calculate the predicted sequence y l With the real label sequence p l The cross-entropy loss is calculated, and the decoder network parameters are updated through the backpropagation process of the gradient. Where x... l ∈R B×C×T×chirp×H×W y l p l ∈R B×Cls×T×H×W B represents the batch size, C represents the number of channels in the radar heatmap, T represents the number of radar heatmap frame sequences, Chirps represent the number of chirps within a frame, H and W represent the height and width of the radar heatmap, respectively, and Cls represents the number of channels for the predicted class. Cross-entropy loss is used as the loss function to supervise the training process, and its calculation formula can be expressed as:

[0054]

[0055] Among them, y l (cls,i,j) represents the predicted probability that category cls appears in the i-th row and j-th column of the radar heatmap; p l (cls,i,j) represents the probability that category cls appears in the i-th row and j-th column of the labeled radar heatmap.

[0056] Finally, a supervised training process for the encoder-decoder network was completed using a small amount of labeled radar heatmap data for a certain number of rounds, and the final decoder network parameters were saved.

[0057] S104, the trained encoder-decoder network is copied to obtain the teacher network and student network, and the teacher network and student network are trained using unlabeled radar heatmap data and labeled radar heatmap data, so that the trained student network can be used for target detection.

[0058] As one embodiment, the teacher network and student network are trained using unlabeled radar heatmap data and labeled radar heatmap data, including: performing a first data augmentation on the unlabeled radar heatmap data to obtain a first data sequence, and performing a second data augmentation on the unlabeled radar heatmap data to obtain a second data sequence; forward propagating the first data sequence through the teacher network to generate a first pseudo-label, and forward propagating the second data sequence through the student network to generate a first predicted value; calculating the mean squared error loss between the first predicted value and the first pseudo-label, and updating the student network parameters through backpropagation of gradients, while using exponential moving averages. The teacher network parameters are updated using an averaging method; a third data augmentation is performed on the labeled radar heatmap data to obtain a third data sequence, and a fourth data augmentation is performed on the labeled radar heatmap data to obtain a fourth data sequence; the third data sequence is forward-propagated through the teacher network to generate a second pseudo-label, and the fourth data sequence is forward-propagated through the student network to generate a second predicted value; the weighted sum of the cross-entropy loss between the second predicted value and the true label and the mean squared error loss between the second predicted value and the second pseudo-label is calculated, and the student network parameters are updated through backpropagation of gradients, while the teacher network parameters are updated using an exponential moving average method.

[0059] In other words, a small batch of data sequence x∈ is randomly selected from both the unlabeled and labeled datasets.

[0060] R B×C×T×chirp×H×W Where B represents the batch size, C represents the number of channels in the radar heatmap, T represents the number of radar heatmap frame sequences, Chirps represent the number of chirps within a frame, and H and W represent the height and width of the radar heatmap, respectively. This small batch of data x is augmented twice to obtain the data sequences x1, x2∈R. B×C×T×chirp×H×W Data augmentation is performed by adding Gaussian noise, and during training, the outputs of each layer of the model are regularized using dropout. x1 is forward-propagated through the teacher network to generate pseudo-labels p. t ∈R B×Cls×T×H×W x2 propagates forward through the student network to generate the predicted value y. s ∈R B×Cls×T×H×W Calculate the predicted value y s With real label p l Cross-entropy loss and predicted value y s With pseudo-tag p tThe weighted sum of the mean squared error losses, the loss function can be expressed as:

[0061]

[0062] Where β represents the predicted value y s Compared with the true value p l The cross-entropy loss is controlled by β, which is 0 when the mini-batch data belongs to the unlabeled dataset and 1 when the mini-batch data belongs to the labeled dataset; γ(T) represents the weighting coefficient, which is a quantity that increases linearly with the number of training epochs, and can be specifically expressed as: Where T represents the current training round number, N0 represents the custom warm-up round number, and N represents the required final training round number. Then, the student network parameters θ are updated through backpropagation of gradients. s Simultaneously, the teacher network parameter θ is updated using an exponential moving average method. T The calculation formula for the teacher network parameter update process can be expressed as: θ T (t)=αθ T (t-1)+(1-α)θ s (t), where θ s (t) represents the model parameters of the student network at time t, θ T (t-1) represents the model parameters of the teacher network at time t-1, θ T (t) represents the model parameters of the teacher network at time t.

[0063] Iterate the above process continuously until the student network performance reaches its optimal level, and save the final student network parameters θ. s It was then used in actual millimeter-wave radar target detection tasks.

[0064] Therefore, this application adopts a strategy combining self-supervised learning, supervised learning, and semi-supervised learning. It does not rely on a large number of labeled data samples, but only on a small amount of labeled and a large amount of unlabeled millimeter-wave radar data to achieve reliable radar target detection, overcoming the problem of difficult radar data labeling. Furthermore, currently, there are no pre-trained models for target detection on large-scale millimeter-wave radar heatmap datasets. Therefore, compared with the general teacher-student architecture semi-supervised learning method, this application provides better initialization parameters for the teacher network and student network, which can effectively improve the final performance of the student network, and at the same time effectively avoid the confirmation bias problem in false labels, thus achieving a more reliable radar target detection task.

[0065] As a specific implementation method, such as Figure 2 As shown, a contrastive learning method is used to train a large-scale encoder:

[0066] 1) The large amount of unlabeled millimeter-wave radar data acquired is processed into sequence x u Provides unlabeled data sequences x u Performing two different data augmentation operations yields a processed pair of data sequences x. u1 x u2 ,in

[0067] x u x u1 x u2 ∈R B×C×T×chirp×H×W B represents the batch size, set to 4; C represents the number of channels in the radar heatmap, defaulting to 2; T represents the number of radar heatmap frame sequences, set to 16; Chirps represent the number of chirps in a frame, set to 4; H and W represent the height and width of the radar heatmap, respectively, defaulting to 128. Data augmentation methods include random-scale cropping and resizing to the original size, random horizontal flipping, and random addition of Gaussian noise. The cropping scale is uniformly sampled within the range [0.5, 1]. The variance of the Gaussian noise is taken from 0.01 times the variance of the current batch of samples, and the mean is taken from the mean of the current batch of samples.

[0068] 2) x u1 x u2 This pair of data sequences is used to obtain the projected feature z through an encoder network and a projection MLP head. u1 , z u2 Projection feature z u1 The predicted feature p is obtained after predicting the MLP head. u1 Used to predict projection feature z u2 That is, p u1 With z u2 The similarity between features z should be as close as possible. Similarly, feature z... u2 The predicted feature p is obtained after predicting the MLP head. u2 Used to predict projection feature z u1 Among them, z u1 , z u2 p u1 p u2 ∈R B×d B represents the batch size, and d represents the feature dimension, which is set to 2048.

[0069] 3) Calculate the cosine similarity between the predicted features and the projected features as the loss function for contrastive learning. The formula for calculating the loss function is:

[0070]

[0071]

[0072]

[0073] Among them, z u1 Indicates the first projective feature; z u2 Indicates the second projective feature; p u1 p represents the first predictive feature. u2 represents the second predicted feature; ||·||2 represents the L2 norm; (·) represents the dot product of vectors; stopgrad represents stopping the gradient operation.

[0074] 4) Use the AdamW optimizer to train, set the initial learning rate to 0.0001, train for 50 epochs, stop training, and save the encoder network parameters.

[0075] Encoder network such as Figure 5 As shown, it includes an SSF short-term feature fusion module, a Stem backbone network module, a first feature extraction module ConvFormerBlock (×6), a first downsampling module DownSample, a second feature extraction module ConvFormerBlock (×12), a second downsampling module DownSample, and a third feature extraction module ConvFormerBlock (×6).

[0076] Among them, the SSF short-time feature fusion module is as follows: Figure 6 As shown, this method is used to extract and fuse features from multiple chirp data within a single frame of millimeter-wave radar data, obtaining a single fused feature to represent the features of multiple chirp data within a single frame. The left branch is used to extract the target's velocity features, employing a 4×1×1 3D convolutional kernel with a stride of 1 and zero padding. The right branch is used to extract the target's spatial features, employing a 1×3×3 3D convolutional kernel with a stride of 1 and zero padding of 0×1×1. C represents the number of channels, set to 64. Finally, the fused feature f0∈R is obtained. B×32×16×128×128 .

[0077] The Setm backbone network module employs a 3D convolutional kernel with a kernel size of 9×5×5, a stride of 2, and padding of 4×2×2. This is followed by a BatchNorm layer and a GELU activation function, used to perform feature extraction and downsampling operations on the frame feature sequence composed of single fused features extracted by the short-term feature fusion module, obtaining the initial coding feature f at the first scale. a1 ∈R B ×64×16×64×64 ;

[0078] The first feature extraction module consists of six stacked ConvFormerBlock modules, as shown in the diagram. Figure 7 As shown, this is used to extract three-dimensional spatiotemporal features from the initial encoded features, where C represents the number of channels in the input data, ultimately obtaining the first-scale encoded features f. b1∈R B×64×16×64×64 ;

[0079] The first downsampling module employs average pooling downsampling with a kernel size of 2 and a stride of 2 to downsample the first-scale encoded features. This is followed by a BatchNorm layer, a Linear layer, and a GELU activation function. The Linear layer increases the dimensionality of the input data from C channels to 2C, ultimately obtaining the initial second-scale encoded features f. a2 ∈R B ×128×16×32×32 ;

[0080] The second feature extraction module consists of 12 stacked ConvFormerBlock modules, as shown in the example below. Figure 7 As shown, this is used to extract three-dimensional spatiotemporal features from the initial encoded features, where C represents the number of channels in the input data, ultimately obtaining the second-scale encoded features f. b2 ∈R B×128×16×32×32 ;

[0081] The second downsampling module employs average pooling downsampling with a kernel size of 2 and a stride of 2 to downsample the first-scale encoded features. This is followed by a BatchNorm layer, a Linear layer, and a GELU activation function. The Linear layer increases the number of channels C of the input data to 2C, ultimately obtaining the initial encoded features f for the third scale. a3 ∈R B ×256×16×16×16 ;

[0082] The third feature extraction module consists of six stacked ConvFormerBlock modules, as shown in the example below. Figure 7 As shown, this is used to extract three-dimensional spatiotemporal features from the initial encoded features, where C represents the number of channels in the input data, ultimately obtaining the third-scale encoded features f. b3 ∈R B×256×16×16×16 ;

[0083] In summary, small batch data x u After being input into the encoder network, the encoded features f are finally obtained. b3 ∈R B ×256×16×16×16 We then perform average pooling downsampling on the feature, with a kernel size of 8 and a stride of 8, and then expand it to obtain the final encoded feature f∈R. B×2048 .

[0084] The projection MLP head includes:

[0085] The first layer is a linear layer with an input dimension of 2048 and an output dimension of 2048. It has no bias term. After that, it is processed by a batch normalization layer and a ReLU activation function.

[0086] The second linear layer has an input dimension of 2048 and an output dimension of 2048, with no bias term. It is then processed by a batch normalization layer and a ReLU activation function.

[0087] The third linear layer has an input dimension of 2048 and an output dimension of 2048, with no bias term, and is then processed through a batch normalization layer.

[0088] Predicting MLP headers includes:

[0089] The first layer is a linear layer with an input dimension of 2048 and an output dimension of 512. It has no bias term. After that, it is processed by a batch normalization layer and a ReLU activation function.

[0090] The second layer is a linear layer with an input dimension of 512 and an output dimension of 2048, and has no bias term.

[0091] like Figure 3 As shown, the encoder network parameters are frozen and a smaller decoder network is added afterward to construct the complete encoder-decoder network, which handles a small number of labeled data sequences x. l The predicted sequence y is obtained by using this encoding / decoding network. l Calculate the predicted sequence y l With the real label sequence p l The cross-entropy loss is calculated, and the decoder network parameters are updated through the backpropagation process of the gradient. Where x... l ∈R B×C×T×chirp×H×W y l p l ∈R B×Cls×T×H×W B represents the batch size, set to 4; C represents the number of channels in the radar heatmap, defaulting to 2; T represents the number of radar heatmap frame sequences, set to 16; Chirps represent the number of chirps within a frame, set to 4; H and W represent the height and width of the radar heatmap, respectively, set to 128; Cls represents the number of channels for the predicted class, set to 3. Cross-entropy loss is used as the loss function to supervise the training process, and its calculation formula can be expressed as:

[0092]

[0093] Among them, y l (cls,i,j) represents the predicted probability that category cls appears in the i-th row and j-th column of the radar heatmap; p l (cls,i,j) represents the probability that category cls appears in the i-th row and j-th column of the labeled radar heatmap.

[0094] The AdamW optimizer was used for training. The initial learning rate was set to 0.0001. After 20 training epochs, training was stopped and the decoder network parameters were saved.

[0095] The added smaller-scale decoder network, such as Figure 5 As shown, it includes a third upsampling module UpSample, a second linear feature fusion module Linear, a second feature extraction module ConvFormerBlock (×2), a second upsampling module UpSample, a first linear feature fusion module Linear, a first feature extraction module ConvFormerBlock (×2), a first upsampling module UpSample, and a classification layer module ClassesLayer:

[0096] The third upsampling module, UpSample, performs a double trilinear interpolation upsampling operation on the third-scale encoded features. This is followed by a BatchNorm layer, a Linear layer, and a GELU activation function. The Linear layer reduces the dimensionality of the input data's channel count C to [value missing]. Finally, the initial features f at the second scale are obtained. c2 ∈R B×128×16×32×32 .

[0097] The second linear feature fusion module is a simple concatenation of a linear layer and a GELU activation function. The linear layer has 2C input channels and C output channels, where C represents the number of channels of the input data fed into the linear feature fusion module. This allows for feature fusion of the concatenated features to obtain the initial decoding features f at the second scale. d2 ∈R B×128×16×32×32 ;

[0098] The second feature extraction module consists of two stacked ConvFormerBlock modules, as shown in the example below. Figure 7 As shown, this is used to extract three-dimensional spatiotemporal features from the initial encoded features, where C represents the number of channels in the input data, ultimately obtaining the second-scale decoding features f. e2 ∈R B×128×16×32×32 ;

[0099] The second upsampling module, UpSample, performs a double trilinear interpolation upsampling operation on the second-scale decoded features. This is followed by a BatchNorm layer, a Linear layer, and a GELU activation function. The Linear layer reduces the dimensionality of the input data's channel count C to [value missing]. Finally, the initial feature f at the first scale is obtained. c1 ∈R B×64×16×64×64 .

[0100] The first linear feature fusion module is a simple concatenation of a Linear layer and a GELU activation function. The Linear layer has 2C input channels and C output channels, where C represents the number of input data channels fed into the linear feature fusion module. This allows for feature fusion of the concatenated features to obtain the initial decoded features f at the first scale. d1 ∈R B×64×16×64×64 ;

[0101] The first feature extraction module consists of two stacked ConvFormerBlock modules, as shown in the diagram. Figure 7 As shown, this is used to extract three-dimensional spatiotemporal features from the initial encoded features, where C represents the number of channels in the input data, ultimately obtaining the first-scale decoding features f. e1 ∈R B×64×16×64×64 ;

[0102] The first upsampling module, UpSample, performs a double trilinear interpolation upsampling operation on the decoded features at the first scale. This is followed by a BatchNorm layer, a Linear layer, and a GELU activation function. The Linear layer reduces the dimensionality of the input data's channel count C to [value missing]. Finally, the original scale decoding features f are obtained. c0 ∈R B×32×16×128×128 .

[0103] The classification layer module, ClassesLayer, is a simple concatenation of a Linear layer and a Sigmoid activation function. The Linear layer has 32 input channels and 3 output channels, representing the prediction of three target categories. It performs pixel-level classification prediction on the original scale decoded features to obtain the final predicted value y∈R of the encoding / decoding network. B×32×16×128×128 .

[0104] like Figure 4 As shown, the supervised training-obtained encoder-decoder network is duplicated twice and used to initialize the teacher and student networks. A mini-batch data sequence x∈R is randomly selected from both the unlabeled and labeled datasets. B ×C×T×chirp×H×W Where B represents the batch size, set to 4; C represents the number of channels in the radar heatmap, defaulting to 2; T represents the number of radar heatmap frame sequences, set to 16; Chirps represent the number of chirps within a frame, set to 4; and H and W represent the height and width of the radar heatmap, respectively, defaulting to 128. This small batch of data x is then augmented twice to obtain the data sequences x1, x2 ∈

[0105] R B×C×T×chirp×H×WData augmentation was performed by adding Gaussian noise. The variance of the Gaussian noise was taken as 0.01 times the variance of the current batch of samples, and the mean was taken as the mean of the current batch of samples. Simultaneously, during training, the outputs of each layer of the model were regularized using dropout with a dropout rate of 0.5. x1 was forward-propagated through the teacher network to generate pseudo-labels p. t ∈R B×Cls×T×H×W x2 propagates forward through the student network to generate the predicted value y. s ∈R B×Cls×T×H×W Calculate the predicted value y s With real label p l Cross-entropy loss and predicted value y s With pseudo-tag p t The weighted sum of the mean squared error losses, the loss function can be expressed as:

[0106]

[0107] Where β represents the predicted value y s Compared with the true value p l The cross-entropy loss is controlled by β, which is 0 when the mini-batch data belongs to the unlabeled dataset and 1 when the mini-batch data belongs to the labeled dataset; γ(T) represents the weighting coefficient, which is a quantity that increases linearly with the number of training epochs, and can be specifically expressed as: Where T represents the current training round number, N0 represents the custom warm-up round number, set to 20, N represents the required final training round number, set to 50, and γ is set to 2.

[0108] The student network parameters θ are updated through backpropagation of the gradient. s Simultaneously, the teacher network parameter θ is updated using an exponential moving average method. T The calculation formula for the teacher network parameter update process can be expressed as θ T (t)=αθ T (t-1)+(1-α)θ s (t), where θ s (t) represents the model parameters of the student network at time t, θ T (t-1) represents the model parameters of the teacher network at time t-1, θ T (t) represents the model parameters of the teacher network at time t.

[0109] The model was trained for 50 epochs using an early stopping strategy: training was stopped if the predictive loss of the student network did not decrease in the most recent 5 epochs. Ultimately, a high-performing student network was obtained and applied to a practical millimeter-wave radar target detection task, achieving better results and stronger robustness than general supervised models.

[0110] In summary, the target detection method based on millimeter-wave radar according to embodiments of the present invention first acquires unlabeled and labeled radar heatmap data; then, an encoder network is constructed and trained on the unlabeled radar heatmap data using contrastive learning to obtain encoder network parameters, which are then set to be non-learnable; next, a decoder network is constructed and connected to the trained encoder network to obtain an encoder-decoder network, which is then trained on labeled radar heatmap data to obtain decoder network parameters; finally, the trained encoder-decoder network is replicated to obtain a teacher network and a student network, which are then trained on both unlabeled and labeled radar heatmap data to perform target detection using the trained student network; thus, with limited labeled data support, the effectiveness and accuracy of millimeter-wave radar target detection are achieved by fully utilizing a large amount of unlabeled data.

[0111] To implement the above embodiments, this invention proposes a computer-readable storage medium storing a target detection program based on millimeter-wave radar. When executed by a processor, the millimeter-wave radar-based target detection program implements the target detection method based on millimeter-wave radar as described above.

[0112] According to embodiments of the present invention, a computer-readable storage medium stores a target detection program based on millimeter-wave radar, enabling a processor to implement the target detection method based on millimeter-wave radar as described above when executing the program. Thus, by combining self-supervised learning, supervised learning, and semi-supervised learning, reliable radar target detection can be achieved without relying on a large number of labeled data samples.

[0113] To implement the above embodiments, this invention proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the target detection method based on millimeter-wave radar as described above.

[0114] According to the computer device of the present invention, a target detection program based on millimeter-wave radar is stored in a memory, so that when the processor executes the target detection program based on millimeter-wave radar, the target detection method based on millimeter-wave radar as described above is implemented. Thus, by combining self-supervised learning, supervised learning and semi-supervised learning, reliable radar target detection can be achieved without relying on a large number of labeled data samples.

[0115] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0119] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0120] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0121] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0122] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0123] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0124] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0125] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0126] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A target detection method based on millimeter-wave radar, characterized in that, Includes the following steps: Acquire both unlabeled and labeled radar heatmap data; An encoder network is constructed and trained on the unlabeled radar heatmap data using a contrastive learning approach to obtain the encoder network parameters, and the encoder network parameters are set to be non-learnable. A decoder network is constructed, and the decoder network is connected to the trained encoder network to obtain an encoder-decoder network. The encoder-decoder network is then trained using the labeled radar heatmap data to obtain the decoder network parameters. The trained encoder-decoder network is copied to obtain a teacher network and a student network. The teacher network and the student network are then trained using the unlabeled radar heatmap data and the labeled radar heatmap data, so that the trained student network can be used for target detection. The encoder network is trained on the unlabeled radar heatmap data using a contrastive learning approach, including: The labelless radar heatmap data is subjected to a first data enhancement to obtain a first data sequence, and the labelless radar heatmap data is subjected to a second data enhancement to obtain a second data sequence. The first data sequence is passed through the encoder network and the projection MLP head to obtain the first projection feature, and the second data sequence is passed through the encoder network and the projection MLP head to obtain the second projection feature; The first projected feature is processed through the prediction MLP head to obtain the first predicted feature, and the second projected feature is processed through the prediction MLP head to obtain the second predicted feature. The loss function for the contrastive learning is calculated based on the first predicted feature, the first projected feature, the second predicted feature, and the second projected feature; The loss function for the contrastive learning is obtained according to the following formula: ， in, Indicates the first projective feature; Indicates the second projective feature; Indicates the first predictive feature; Indicates the second predictive feature; Represents the 2-norm; ) represents the dot product of vectors, and stopgrad means to stop the gradient operation.

2. The target detection method based on millimeter-wave radar as described in claim 1, characterized in that, Acquire both unlabeled and labeled radar heatmap data, including: Acquire raw echo data collected by millimeter-wave radar sensors; The original echo data is processed by a fast Fourier transform in the range domain. Low-pass filtering is applied to the echo data after Fast Fourier Transform processing to remove high-frequency noise. The echo data after low-pass filtering is processed by Fast Fourier Transform in the angle domain to obtain the labelless radar heat map data. The published labeled radar heatmap datasets are selected to collect radar heatmap data segments with accurate labeling information in order to obtain the labeled radar heatmap data.

3. The target detection method based on millimeter-wave radar as described in claim 2, characterized in that, Training the encoding / decoding network using the labeled radar heatmap data includes: The labeled radar heatmap data is used to predict the sequence through the encoding / decoding network. The cross-entropy loss function between the predicted sequence and the true label sequence is calculated, and the decoder network parameters are updated through the backpropagation process of the gradient.

4. The target detection method based on millimeter-wave radar as described in claim 3, characterized in that, The cross-entropy loss function is obtained according to the following formula: in, This represents the probability that category cls appears in the i-th row and j-th column of the predicted radar heatmap; This represents the probability that category cls appears in the i-th row and j-th column of the labeled radar heatmap.

5. The target detection method based on millimeter-wave radar as described in claim 1, characterized in that, Training the teacher network and student network using the unlabeled radar heatmap data and the labeled radar heatmap data includes: The labelless radar heatmap data is subjected to a first data enhancement to obtain a first data sequence, and the labelless radar heatmap data is subjected to a second data enhancement to obtain a second data sequence. The first data sequence is forward-propagated through the teacher network to generate a first pseudo-label, and the second data sequence is forward-propagated through the student network to generate a first predicted value. Calculate the mean squared error loss between the first predicted value and the first pseudo label, and update the student network parameters through backpropagation of gradients, while updating the teacher network parameters using an exponential moving average method. The labeled radar heatmap data is subjected to a third data enhancement to obtain a third data sequence, and the labeled radar heatmap data is subjected to a fourth data enhancement to obtain a fourth data sequence. The third data sequence is forward-propagated through the teacher network to generate a second pseudo-label, and the fourth data sequence is forward-propagated through the student network to generate a second predicted value. The cross-entropy loss between the second predicted value and the true label and the mean squared error loss between the second predicted value and the second pseudo label are calculated as a weighted sum. The student network parameters are updated through backpropagation of the gradient, while the teacher network parameters are updated using an exponential moving average method.

6. A computer-readable storage medium, characterized in that, It stores a target detection program based on millimeter-wave radar, which, when executed by the processor, implements the target detection method based on millimeter-wave radar as described in any one of claims 1-5.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the target detection method based on millimeter-wave radar as described in any one of claims 1-5.

Images