A micro-doppler radar data enhancement method, system, terminal and storage medium

Abstract

The application discloses a kind of micro-doppler radar data enhancement method, system, terminal and storage medium, the method includes: micro-doppler feature sample is input to the automatic encoder-decoder based on auxiliary classifier, and output synthetic micro-doppler feature sample and category label;Low-dimensional representation and corresponding category label obtained by the automatic encoder-decoder based on auxiliary classifier are input to the generator based on time-frequency causality, and output probability distribution considering time causal relationship;According to the low-dimensional representation of given real sample, according to the category label of real sample as generation condition, new representation is generated from low-dimensional representation, and synthetic sample is generated from new representation.The application can generate new sample using the network trained using real sample, perform data enhancement, and the performance of downstream classification task can be significantly improved by data enhancement.

Description

Technical Field

[0001] This invention relates to the field of data enhancement technology, and in particular to a micro-Doppler radar data enhancement method, system, terminal, and computer-readable storage medium. Background Technology

[0002] Human Activity Recognition (HAR) based on radar micro-Doppler features has attracted significant research interest due to its wide application in surveillance and human-computer interaction. Traditional HAR models are typically based on signal processing techniques, while recent deep learning models have made significant progress in performance. However, acquiring radar data often requires substantial manpower and time, leading to insufficient labeled samples when training deep learning models.

[0003] Some existing generative methods are dedicated to producing natural-looking samples, especially in the field of visual signals. The main purpose of these methods is to provide viewers with a satisfying visual experience.

[0004] Even with class labels provided, the realism of the generated samples is crucial for HAR tasks. For example, a widely adopted solution is to generate synthetic samples based on real labels using adversarial loss or its variants. Simply put, the generator and discriminator are trained simultaneously to make the distribution of synthetic samples approximate the distribution of real samples. However, due to the scarcity of real microDoppler data, the discriminator quickly overfits during simultaneous training, leading to a decrease in sample realism.

[0005] An alternative solution is to construct a latent feature space for real samples using an autoencoder (AE) architecture without employing any discriminator. Assuming the features of the real samples form a manifold in the latent space, synthetic samples can be generated by decoding random features on the manifold. In acquiring these random features, most generation methods ignore the properties of Doppler features, treating them merely as optical images. Unlike optical images, the continuous columns of micro-Doppler features are correlated with each other, essentially constituting a time series. This causality persists after feature encoding. Therefore, when generating micro-Doppler features, temporal causality needs to be considered when acquiring random features.

[0006] Existing methods typically model radar samples following a specific distribution directly at the pixel or time level. However, in micro-Doppler features, many pixels are dominated by noise rather than signal. Therefore, directly modeling micro-Doppler features is unreasonable; instead, it is desirable to model their low-dimensional representation, as low-dimensional representations contain less noise.

[0007] It is worth noting that the positional topological relationships between elements in the low-dimensional representation are essentially the same as those in the corresponding micro-Doppler features. The low-dimensional representation is obtained from the micro-Doppler features using an encoder and a quantizer. The encoder contains only convolution operations and is translation-invariant. The quantizer simply replaces the encoded features at each element position with discrete indices. Therefore, element dependencies should be considered when modeling the low-dimensional representation.

[0008] While some researchers have encoded predefined category labels as conditions when generating samples, most existing methods simply use hard labels in data augmentation, meaning that the generated samples are expected to definitely belong to a certain category. However, in recognition or classification tasks, samples with ambiguous categories have a greater impact on the classification boundary in the feature space than samples that explicitly belong to a certain category.

[0009] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0010] The main objective of this invention is to provide a micro-Doppler radar data enhancement method, system, terminal, and computer-readable storage medium, aiming to solve the problems of poor performance and low naturalness and authenticity of human activity recognition models based on radar micro-Doppler features in the prior art.

[0011] To achieve the above objectives, the present invention provides a micro-Doppler radar data enhancement method, which includes the following steps:

[0012] The micro-Doppler feature samples are input into the automatic encoder-decoder based on the auxiliary classifier, and the synthesized micro-Doppler feature samples and class labels are output to complete the training of the automatic encoder-decoder based on the auxiliary classifier.

[0013] The low-dimensional representation and corresponding class label obtained by the automatic encoder and decoder based on the auxiliary classifier are input into the time-frequency causal generator, and the output is a probability distribution that takes into account the time causal relationship, so as to complete the training of the time-frequency causal generator.

[0014] The trained autoencoder based on the auxiliary classifier extracts a low-dimensional representation from the given real samples. The trained time-frequency causal generator generates a new representation from the low-dimensional representation based on the class label of the real samples as the generation condition. The trained autoencoder based on the auxiliary classifier generates synthetic samples from the new representation.

[0015] Optionally, in the micro-Doppler radar data augmentation method, the automatic encoder-decoder based on the auxiliary classifier includes: an encoder, a quantizer, a decoder, and an auxiliary classifier;

[0016] The process of inputting micro-Doppler feature samples into an automatic encoder-decoder based on an auxiliary classifier and outputting synthesized micro-Doppler feature samples and class labels specifically includes:

[0017] Acquire micro-Doppler feature samples and input the micro-Doppler feature samples into the automatic encoder-decoder based on the auxiliary classifier;

[0018] By working together with the encoder and the quantizer, a low-dimensional representation containing time and category information is extracted.

[0019] The decoder uses the low-dimensional representation to generate a synthetic microDoppler feature sample.

[0020] The auxiliary classifier uses the low-dimensional representation to extract the category information of the micro-Doppler feature samples and generates category labels.

[0021] The output target of the decoder is the input of the encoder; the output target of the auxiliary classifier is the label input to the encoder.

[0022] Optionally, in the micro-Doppler radar data augmentation method, the time-frequency causal generator includes: multiple causal blocks, residual and attention blocks, and sampling blocks;

[0023] The step of inputting the low-dimensional representation and corresponding class label obtained by the automatic encoder-decoder based on the auxiliary classifier into the time-frequency causality-based generator, and outputting a probability distribution considering time causality, specifically includes:

[0024] The low-dimensional representation and corresponding class label output by the automatic encoder-decoder based on the auxiliary classifier are used as input to the time-frequency causal generator;

[0025] The input is transformed into a probability distribution that takes into account temporal causality through multiple causal blocks and residual and attention blocks;

[0026] Random sampling is performed on the probability distribution that considers temporal causality to obtain a new representation. The probability distribution that considers temporal causality is optimized through repeated random sampling processes to make the output representation close to the input representation, with the input category label as the generation condition.

[0027] Optionally, the micro-Doppler radar data augmentation method, wherein the step of using the trained automatic encoder-decoder based on the auxiliary classifier to extract a low-dimensional representation based on a given real sample, using the trained time-frequency causal generator to generate a new representation from the low-dimensional representation based on the class label of the real sample as the generation condition, and using the trained automatic encoder-decoder based on the auxiliary classifier to generate synthetic samples from the new representation, specifically includes:

[0028] Using the encoder and quantizer in the trained autoencoder based on the auxiliary classifier, a low-dimensional representation is extracted from a given real sample;

[0029] Using the trained time-frequency causal generator, combined with the class labels of the given real samples as generation conditions, a new representation is generated from the extracted low-dimensional representation;

[0030] Synthetic samples are generated from the newly generated representation using the decoder in the trained autoencoder based on the auxiliary classifier.

[0031] Optionally, the microDoppler radar data augmentation method, wherein the step of generating synthetic samples from the new representation using the trained automatic encoding and decoding based on the auxiliary classifier, further includes:

[0032] If it is necessary to generate samples with fuzzy categories, low-dimensional representations are extracted from two different real samples. A hybrid method module is designed to obtain hybrid representations and soft labels through the hybrid method module. Synthetic samples are then generated sequentially through the time-frequency causal generator and the decoder of the autoencoder based on the auxiliary classifier, which are trained in turn. The synthetic samples are then associated with the soft labels. Finally, the auxiliary classifier of the autoencoder based on the auxiliary classifier is used to select the synthetic samples that meet the requirements.

[0033] Optionally, the micro-Doppler radar data augmentation method, wherein the step of generating synthetic samples sequentially through the decoder of the time-frequency causal generator and the automatic encoder-decoder based on the auxiliary classifier, further includes:

[0034] If the class information is inconsistent during the sample synthesis process, the synthesized sample is discarded by the auxiliary classifier in the trained automatic encoder-decoder based on the auxiliary classifier.

[0035] Optionally, in the micro-Doppler radar data augmentation method, the conditional distribution of the low-dimensional representation is:

[0036]

[0037] Among them, v mLet v represent the m-th column vector of the low-dimensional representation, c represent the category label associated with the low-dimensional representation, and v 1：m-1 Indicates the transition from v1 to v m-1 A vector sequence, where m represents a column in low dimension, n represents a row in low dimension, N represents the number of rows in low dimension, and d (m) Let z represent the fundamental Doppler frequency of the m-th column in the low-dimensional representation. [m，n] z represents the magnitude of the nth element in the m-th column of the low-dimensional representation. [m，n-1] z represents the magnitude of the (n-1)th element in the m-th column of the low-dimensional representation. [m，n+1] This represents the magnitude of the (n+1)th element in the m-th column of the low-dimensional representation. This represents the d-th column in the m-th column of the low-dimensional representation. (m) The magnitude of each element, p(v) m |v 1：m-1 c) represents the conditional distribution in low-dimensional representation. This represents the amplitude distribution of the fundamental frequency elements in the m-th column, where the amplitude distribution depends on the first m-1 columns. This indicates that the column m contains values ​​greater than d. (m) The amplitude distribution of the fundamental frequency elements, the amplitude distribution depends on d (m) The Doppler frequency elements up to n-1, This indicates that the m-th column contains values ​​less than d. (m) The amplitude distribution of the fundamental frequency elements, the amplitude distribution depends on d (m) The Doppler frequency element up to n+1.

[0038] Furthermore, to achieve the above objectives, the present invention also provides a micro-Doppler radar data enhancement system, wherein the micro-Doppler radar data enhancement system comprises:

[0039] An automatic codec training module is used to input micro-Doppler feature samples into an automatic codec based on an auxiliary classifier, and output synthesized micro-Doppler feature samples and class labels to complete the training of the automatic codec based on the auxiliary classifier.

[0040] The generator training module is used to input the low-dimensional representation and corresponding category label obtained by the automatic encoder and decoder based on the auxiliary classifier into the time-frequency causality-based generator, and output a probability distribution that considers the time causality relationship to complete the training of the time-frequency causality-based generator.

[0041] The sample generation module is used to extract a low-dimensional representation from a given real sample using the trained autoencoder based on the auxiliary classifier, generate a new representation from the low-dimensional representation using the trained time-frequency causal generator based on the class label of the real sample as the generation condition, and generate a synthetic sample from the new representation using the trained autoencoder based on the auxiliary classifier.

[0042] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a micro-Doppler radar data enhancement program stored in the memory and executable on the processor, wherein when the micro-Doppler radar data enhancement program is executed by the processor, it implements the steps of the micro-Doppler radar data enhancement method as described above.

[0043] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a micro-Doppler radar data enhancement program, which, when executed by a processor, implements the steps of the micro-Doppler radar data enhancement method as described above.

[0044] In this invention, micro-Doppler feature samples are input into an autoencoder / decoder based on an auxiliary classifier, which outputs synthesized micro-Doppler feature samples and class labels to complete the training of the autoencoder / decoder based on the auxiliary classifier. The low-dimensional representation and corresponding class labels obtained from the autoencoder / decoder based on the auxiliary classifier are input into a time-frequency causality-based generator, which outputs a probability distribution considering time causality to complete the training of the time-frequency causality-based generator. The trained autoencoder / decoder based on the auxiliary classifier extracts a low-dimensional representation based on given real samples. The trained time-frequency causality-based generator uses the class labels of the real samples as generation conditions to generate a new representation from the low-dimensional representation. The trained autoencoder / decoder based on the auxiliary classifier generates synthetic samples from the new representation. This invention, by training the network using real samples, can generate new samples using the trained network for data augmentation, which can significantly improve the performance of downstream classification tasks. Attached Figure Description

[0045] Figure 1 This is a flowchart of a preferred embodiment of the micro-Doppler radar data enhancement method of the present invention;

[0046] Figure 2 This is a schematic diagram of the overall framework in a preferred embodiment of the micro-Doppler radar data enhancement method of the present invention.

[0047] Figure 3This is a schematic diagram of the training process of an automatic encoder-decoder based on an auxiliary classifier in a preferred embodiment of the micro-Doppler radar data augmentation method of the present invention;

[0048] Figure 4 This is a schematic diagram of the training process of a time-frequency causal generator in a preferred embodiment of the micro-Doppler radar data enhancement method of the present invention.

[0049] Figure 5 This is a schematic diagram of the sample synthesis stage in a preferred embodiment of the micro-Doppler radar data enhancement method of the present invention;

[0050] Figure 6 This is a structural diagram of a time-frequency causal generator in a preferred embodiment of the micro-Doppler radar data enhancement method of the present invention;

[0051] Figure 7 This is a schematic diagram illustrating the effect of the data augmentation sample ratio on the downstream classifier in a preferred embodiment of the micro-Doppler radar data augmentation method of the present invention.

[0052] Figure 8 This is a schematic diagram comparing synthetic samples and real samples in a preferred embodiment of the micro-Doppler radar data enhancement method of the present invention;

[0053] Figure 9 This is a structural diagram of a preferred embodiment of the micro-Doppler radar data enhancement system of the present invention;

[0054] Figure 10 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0056] The micro-Doppler radar data enhancement method described in the preferred embodiment of the present invention, such as... Figure 1 and Figure 2 As shown, the micro-Doppler radar data enhancement method includes the following steps:

[0057] Step S10: Input the micro-Doppler feature samples into the automatic encoder-decoder based on the auxiliary classifier, and output the synthesized micro-Doppler feature samples and class labels to complete the training of the automatic encoder-decoder based on the auxiliary classifier.

[0058] Specifically, such as Figure 2 and Figure 3 As shown, the Automatic Encoder-Decoder (ACAE) based on the Auxiliary Classifier (ACAE) has the following detailed architecture: Figure 2 The diagram (shown in yellow) includes: an encoder (En), a quantizer (Q), a decoder (De), and an auxiliary classifier (AC). The encoder (En) consists of multiple convolutional layers and residual blocks, used to extract features from the input micro-Doppler image. Since the encoder is primarily based on convolutional operations, it can preserve the topological relationships between pixels. Given a low-dimensional representation, the ACAE can reconstruct a sample from the low-dimensional representation using the decoder (De); the decoder's structure is symmetric to the encoder's. Since the decoder does not introduce any randomness, the reconstructed sample is considered to have the same class information as the given low-dimensional representation. To facilitate the acquisition of this class information, this invention uses the auxiliary classifier (AC) to obtain a label vector from the low-dimensional representation, containing the predicted class probabilities associated with the low-dimensional representation and the corresponding micro-Doppler image. Furthermore, the auxiliary classifier (AC) can also optimize the performance of the encoder. The auxiliary classifier (AC) consists of two convolutional layers and two fully connected layers (FCs), each convolutional layer followed by a rectified linear unit (ReLU), and the two fully connected layers followed by a softmax activation function. Through this architecture, ACAE can not only effectively extract and reconstruct low-dimensional representations of samples, but also ensure that the generated samples have accurate category information, thereby improving the quality and diversity of sample generation.

[0059] like Figure 3 As shown, micro-Doppler feature samples are acquired and input into the automatic encoder-decoder based on the auxiliary classifier. Through the combined action of the encoder and the quantizer, a low-dimensional representation containing temporal and category information is extracted. The decoder uses this low-dimensional representation to generate a synthetic micro-Doppler feature sample. The auxiliary classifier uses this low-dimensional representation to extract the category information of the micro-Doppler feature sample and generates category labels. The output target of the decoder is the input of the encoder; the output target of the auxiliary classifier is the label input to the encoder.

[0060] In other words, the purpose of introducing an automatic encoder-decoder (ACAE) based on an auxiliary classifier into the data augmentation model is twofold: first, given a microDoppler feature sample (i.e. Figure 3 Given an input sample X, ACAE can extract its low-dimensional representation using an encoder and quantizer. This extracted low-dimensional representation should contain both temporal and class information of the input sample. Secondly, given a representation containing class information (i.e., ... Figure 3 The ACAE can generate a synthetic micro-Doppler feature sample (i.e., input class label c) through the decoder. Figure 3 Output samples Meanwhile, the class labels generated by the auxiliary classifier can extract class information from a given representation (i.e., Figure 3 Output category labels To achieve the above objectives, this invention trains ACAE using a self-supervised learning paradigm, such as... Figure 3 As shown. In short, during training, the output target of the decoder is the input of the encoder, i.e., the real microDoppler feature samples with class labels; simultaneously, the output target of the auxiliary classifier is the label input to the encoder. This invention considers the training of ACAE as the first stage of training.

[0061] Step S20: Input the low-dimensional representation and corresponding category label obtained by the automatic encoder and decoder based on the auxiliary classifier into the time-frequency causal generator, and output the probability distribution considering the time causal relationship to complete the training of the time-frequency causal generator.

[0062] Specifically, such as Figure 2 and Figure 4 As shown, the Time-Frequency Causality-based Representation Generator (TFCRG) includes multiple causal blocks (CBs), residual and attention blocks (RABs), and sampling blocks (SBs). It requires a low-dimensional feature sample and its corresponding class label as input. The purpose of this part is to generate a new representation with the same distribution as the input representation. The low-dimensional representation and corresponding class label output by the automatic encoder / decoder based on the auxiliary classifier are input to the time-frequency causality-based generator. After passing through multiple causal blocks and the residual and attention blocks, the input is transformed into a probability distribution considering temporal causality. Random sampling is performed on this probability distribution to obtain a new representation, such as... Figure 4 As shown, the training of TFCRG, also known as the second-stage training in this invention, is performed after ACAE training and freezing. Since random sampling in TFCRG is repeated multiple times during training, the optimization of the probability distribution is equivalent to the optimization of the random sampling results. The purpose of the second-stage training is to make the output representation approximate the input representation, using the input class label as the generation condition.

[0063] Existing methods typically model radar samples following a specific distribution directly at the pixel or time level. However, in micro-Doppler features, many pixels are dominated by noise rather than signal. Therefore, directly modeling micro-Doppler features is unreasonable; instead, it is desirable to model their low-dimensional representation, as low-dimensional representations contain less noise.

[0064] It is worth noting that the positional topological relationships between elements in the low-dimensional representation are essentially the same as those in the corresponding micro-Doppler features. Low-dimensional representations are obtained from the micro-Doppler features using an encoder and a quantizer. The encoder contains only convolution operations, which are translation-invariant. Furthermore, the quantizer simply replaces the encoded features at each element position with discrete indices. The horizontal axis of the low-dimensional representation represents the time frame, and the vertical axis represents the Doppler frequency. Therefore, element dependencies should be considered when modeling the low-dimensional representation. Based on the properties of micro-Doppler features, this invention primarily models two types of dependencies: temporal causality and frequency causality.

[0065] Temporal causality refers to the principle that the motion of a given target in the real world generally exhibits temporal continuity and stationarity. This means that the target's velocity at the current time largely depends on its velocity at previous times. Since the Doppler feature map contains target velocity information in columns, it shows the dependency between the current time frame and previous time frames. This characteristic of Doppler signals is well inherited in low-dimensional Doppler signals, and this characteristic is also well inherited in low-dimensional representations. Therefore, the m-th column vector in the low-dimensional representation should depend on the previous m-1 column vectors.

[0066] Frequency causality refers to the dependence of Doppler frequencies in micro-Doppler features and their low-dimensional representations. In the HAR context, Doppler frequencies distinguish the motion velocities of different parts of the human body. Due to the physical constraints and connectivity of body parts, their velocities are difficult to make independent of each other. Taking a walking person as an example, this invention readily shows that the movement of the arm or leg depends on the movement of the torso. Furthermore, the movement of the wrist and fingers is constrained by the corresponding arm. Generally speaking, the motion relationships of body parts can be described as a base component (such as the torso) and hierarchically dependent components (such as the arms and fingers). In micro-Doppler features, the Doppler frequencies with the largest amplitudes are typically contributed by the base component, serving as the fundamental frequencies of the micro-motions of the dependent components. Compared to the base component, the dependent components have a wider range of motion velocities, resulting in Doppler frequencies higher or lower than the fundamental frequency.

[0067] Therefore, when modeling each column of the low-dimensional representation, this invention first aims to estimate its fundamental Doppler frequency from the corresponding feature map. Due to the consistency of motion speed in the real world, the fundamental Doppler frequency is stable across consecutive frames. Therefore, this invention estimates the fundamental Doppler frequency of the m-th column as the frequency with the largest amplitude in the previous column. For the first column, for simplicity, the fundamental Doppler frequency is set to an intermediate frequency. After modeling the fundamental frequency of the m-th column, it is also necessary to model the elements with frequencies higher or lower than the fundamental Doppler frequency that are related to other elements in that column. To this end, the conditional probabilities of these elements are constructed sequentially using the chain rule. Formulaically, this invention can model the conditional distribution of the low-dimensional representation (using the chain rule to obtain the formula, which is then used to design the model framework; specifically, the design of the two masks in TFCRG) as follows:

[0068]

[0069] Among them, v m Let v represent the m-th column vector of the low-dimensional representation, c represent the category label associated with the low-dimensional representation, and v 1：m-1 Indicates the transition from v1 to v m-1 A vector sequence, where m represents a column in low dimension, n represents a row in low dimension, N represents the number of rows in low dimension, and d (m) Let z represent the fundamental Doppler frequency of the m-th column in the low-dimensional representation. [m，n] z represents the magnitude of the nth element in the m-th column of the low-dimensional representation. [m，n-1] z represents the magnitude of the (n-1)th element in the m-th column of the low-dimensional representation. [m，n+1] This represents the magnitude of the (n+1)th element in the m-th column of the low-dimensional representation. This represents the d-th column in the m-th column of the low-dimensional representation. (m) The magnitude of each element, p(v) m |v 1：m-1 c) represents the conditional distribution in low-dimensional representation. This represents the amplitude distribution of the fundamental frequency elements in the m-th column, where the amplitude distribution depends on the first m-1 columns. This indicates that the column m contains values ​​greater than d. (m) The amplitude distribution of the fundamental frequency elements (i.e., for n > d) (m) The distribution of the elements further depends on d (m) The amplitude distribution depends on the Doppler frequency elements up to n-1, and the amplitude distribution depends on the frequency elements up to d. (m) The Doppler frequency elements up to n-1, This indicates that the m-th column contains values ​​less than d. (m) The amplitude distribution of the fundamental frequency elements, the amplitude distribution depends on d (m) Doppler frequency elements up to n+1 (i.e., n < d) (m) Its distribution depends on d (m)(Up to the n+1 Doppler frequency elements). By combining the formulas of a total of M columns in the low-dimensional representation, this invention can describe the dependencies of elements in the low-dimensional representation and take into account time-frequency causality.

[0070] Based on the above modeling, this invention develops TFCRG, a model designed to generate new low-dimensional representations of the same size using a low-dimensional representation of size N×M and its associated class labels. In TFCRG, embedding is performed first, followed by the development of N×M concatenated causal blocks (e.g., 2×3 represents a matrix of shape [2, 3]), followed by three residual and attention blocks (RABs), to generate an amplitude distribution matrix of size N×M×B, where B is the quantization order of the amplitude values ​​in the low-dimensional representation. New representations can then be generated by randomly sampling on the generated distribution.

[0071] Since the chain rule is used to model low-dimensional representations sequentially, this invention must follow the same order when generating new samples based on the representation model. Therefore, this invention designs a causal block and generates each element of the low-dimensional representation sequentially through a looping causal block. To update a low-dimensional representation of shape N×M, this invention requires looping the causal block N×M times. In each loop, the causal block takes the low-dimensional representation and the embedded category as input and outputs the updated low-dimensional representation. As the conditional distribution of the low-dimensional representation is shown in the formula above, the generation order of the low-dimensional representation along the time dimension is column-by-column. When generating the m-th column, the condition depends on the m-1 column vectors representing the previous time frame. Therefore, this invention designs a time frame mask, as shown... Figure 5 As shown. The perceptual elements of the time frame mask only include the elements in the first m-1 columns. The time frame mask is updated once every N new elements (i.e., every time a new column is generated).

[0072] Within the frequency dimension of each column, the order in which the low-dimensional representations are generated is from d... (m) This diffuses to each element. When generating the nth element in the mth column, the condition depends not only on the m-1 column vector representing the previous time frame, but also on the other elements in the already generated mth column. Therefore, this invention designs a frequency element mask, such as... Figure 5 As shown. The perceptual element coverage of the frequency element mask represents the frequency from d. (m) Elements up to n. The frequency element mask is updated once each time a new element is generated.

[0073] Subsequently, since residual convolutional blocks can effectively alleviate the gradient vanishing problem and promote deep network training, while attention blocks can dynamically adjust weights to capture more meaningful representations, this invention uses a cascaded combination of residual convolutional blocks and attention blocks to help the TFCRG structure generate higher quality low-dimensional representations.

[0074] Finally, this invention designs a sampling block for final sampling. The sampling block consists of a convolutional layer, a ReLU activation function layer, and a sampling layer, using different sampling functions during the training and testing phases.

[0075] Step S30: Using the trained automatic encoder-decoder based on the auxiliary classifier, extract a low-dimensional representation based on the given real sample; using the trained time-frequency causal generator, generate a new representation from the low-dimensional representation based on the class label of the real sample as the generation condition; and using the trained automatic encoder-decoder based on the auxiliary classifier to generate a synthetic sample from the new representation.

[0076] Specifically, after training ACAE and TFCRG, the direct method for generating new samples from given real samples and their corresponding class labels includes three steps: First, using the encoder and quantizer in the trained autoencoder / decoder based on the auxiliary classifier, a low-dimensional representation is extracted from the given real samples; second, using the trained time-frequency causal generator, combined with the class label of the given real samples as the generation condition, a new representation is generated from the low-dimensional representation extracted in the first step using TFCRG; third, using the decoder in the trained autoencoder / decoder based on the auxiliary classifier, a synthetic sample is generated from the newly generated representation.

[0077] The generated sample category using the above method is limited to the category label of the given sample. In other words, the generated sample belongs to a specific category. However, generating samples with ambiguous categories is more meaningful in the context of HAR. Therefore, this invention proposes to... Figure 6 The architecture shown generates new samples by first extracting two representations from two different real samples. Based on these two representations and their corresponding class labels, a fusion method module is designed to obtain a hybrid representation and a soft label. Subsequently, decoders for TFCRG and ACAE are sequentially included to generate a synthetic sample, which should be associated with the soft label. If the class information is inconsistent during sample synthesis, the ACAE auxiliary classifier will eventually discard the synthetic sample.

[0078] To evaluate the effectiveness of the designed model and the data augmentation method for data augmentation tasks, this invention applies augmented data to train various downstream classification tasks, including common and existing state-of-the-art classification models: AlexNet, ResNet, VGG, and DCNN. A line graph is plotted based on the median classification accuracy obtained from ten experiments, as shown below. Figure 7As shown, when the augmentation ratio (rate) is 0, i.e., no augmentation method is used, the classification performance of most downstream classification tasks is the worst. As the data augmentation ratio increases, the performance of downstream classification tasks gradually improves. When the size of the augmented dataset reaches six to seven times that of the original dataset, the overall classification effect of the model is most significant. These data demonstrate that the data augmentation model and data augmentation method proposed in this invention can significantly improve the performance of downstream classification tasks.

[0079] In addition, this invention also includes a qualitative analysis. For example... Figure 8 As shown, compared with real data samples, the data augmentation model of the present invention can generate new samples with extremely high similarity.

[0080] This invention aims to generate micro-Doppler feature samples to enhance training data for HAR tasks. It not only solves the problems of naturalness and realism in sample generation in existing technologies, but also achieves significant progress in handling category ambiguity and sample diversity, thereby significantly improving the performance of human activity recognition models based on radar micro-Doppler features.

[0081] This invention proposes a data augmentation network. After training the network with real samples, it can be used to generate new samples for data augmentation. In this network, this invention proposes a time-frequency causal relationship representation generator, which maintains the positional topological relationship between elements and models the time-frequency dependency relationship between elements, and can be used to generate new samples.

[0082] Furthermore, such as Figure 9 As shown, based on the above-described micro-Doppler radar data enhancement method, the present invention also provides a micro-Doppler radar data enhancement system, wherein the micro-Doppler radar data enhancement system includes:

[0083] The automatic codec training module 51 is used to input micro-Doppler feature samples into the automatic codec based on the auxiliary classifier and output synthesized micro-Doppler feature samples and class labels to complete the training of the automatic codec based on the auxiliary classifier.

[0084] The generator training module 52 is used to input the low-dimensional representation and corresponding category label obtained by the automatic encoder and decoder based on the auxiliary classifier into the time-frequency causal generator, and output a probability distribution that considers the time causal relationship, so as to complete the training of the time-frequency causal generator.

[0085] The sample generation module 53 is used to extract a low-dimensional representation from a given real sample using the trained automatic encoder-decoder based on the auxiliary classifier, generate a new representation from the low-dimensional representation using the trained time-frequency causal generator based on the class label of the real sample as the generation condition, and generate a synthetic sample from the new representation using the trained automatic encoder-decoder based on the auxiliary classifier.

[0086] Furthermore, such as Figure 10 As shown, based on the above-mentioned micro-Doppler radar data enhancement method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 10 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0087] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a micro-Doppler radar data enhancement program 40, which can be executed by the processor 10 to implement the micro-Doppler radar data enhancement method of this application.

[0088] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the micro-Doppler radar data enhancement method.

[0089] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0090] In one embodiment, when the processor 10 executes the micro-Doppler radar data enhancement program 40 in the memory 20, it implements the steps of the micro-Doppler radar data enhancement method as described above.

[0091] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a micro-Doppler radar data enhancement program, which, when executed by a processor, implements the steps of the micro-Doppler radar data enhancement method as described above.

[0092] In summary, this invention provides a micro-Doppler radar data augmentation method, system, terminal, and storage medium. The method includes: inputting micro-Doppler feature samples into an automatic encoder-decoder based on an auxiliary classifier, outputting synthesized micro-Doppler feature samples and class labels to complete the training of the automatic encoder-decoder based on the auxiliary classifier; inputting the low-dimensional representation and corresponding class labels obtained by the automatic encoder-decoder based on the auxiliary classifier into a time-frequency causality-based generator, outputting a probability distribution considering time causality to complete the training of the time-frequency causality-based generator; using the trained automatic encoder-decoder based on the auxiliary classifier to extract a low-dimensional representation based on given real samples; using the trained time-frequency causality-based generator to generate a new representation from the low-dimensional representation based on the class labels of the real samples as generation conditions; and using the trained automatic encoder-decoder based on the auxiliary classifier to generate synthetic samples from the new representation. This invention utilizes real samples to train the network, and then uses the trained network to generate new samples for data augmentation. Simultaneously, data augmentation can significantly improve the performance of downstream classification tasks.

[0093] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0094] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0095] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A micro-Doppler radar data enhancement method, characterized in that, The micro-Doppler radar data enhancement method includes: The micro-Doppler feature samples are input into the automatic encoder-decoder based on the auxiliary classifier, and the synthesized micro-Doppler feature samples and class labels are output to complete the training of the automatic encoder-decoder based on the auxiliary classifier. The low-dimensional representation and corresponding class label obtained by the automatic encoder and decoder based on the auxiliary classifier are input into the time-frequency causal generator, and the output is a probability distribution that takes into account the time causal relationship, so as to complete the training of the time-frequency causal generator. The time-frequency causal generator includes: multiple causal blocks, residual and attention blocks, and sampling blocks; The step of inputting the low-dimensional representation and corresponding class label obtained by the automatic encoder-decoder based on the auxiliary classifier into the time-frequency causality-based generator, and outputting a probability distribution considering time causality, specifically includes: The low-dimensional representation and corresponding class label output by the automatic encoder-decoder based on the auxiliary classifier are used as input to the time-frequency causal generator; The input is transformed into a probability distribution that takes into account temporal causality through multiple causal blocks and residual and attention blocks; Random sampling is performed on the probability distribution that considers time causality to obtain a new representation. The probability distribution that considers time causality is optimized through repeated random sampling processes to make the output representation close to the input representation, with the input category label as the generation condition. The trained autoencoder based on the auxiliary classifier extracts a low-dimensional representation from the given real samples. The trained time-frequency causal generator generates a new representation from the low-dimensional representation based on the class label of the real samples as the generation condition. The trained autoencoder based on the auxiliary classifier generates synthetic samples from the new representation.

2. The micro-Doppler radar data enhancement method according to claim 1, characterized in that, The automatic encoder-decoder based on the auxiliary classifier includes: an encoder, a quantizer, a decoder, and an auxiliary classifier; The process of inputting micro-Doppler feature samples into an automatic encoder-decoder based on an auxiliary classifier and outputting synthesized micro-Doppler feature samples and class labels specifically includes: Acquire micro-Doppler feature samples and input the micro-Doppler feature samples into the automatic encoder-decoder based on the auxiliary classifier; By working together with the encoder and the quantizer, a low-dimensional representation containing time and category information is extracted. The decoder uses the low-dimensional representation to generate a synthetic microDoppler feature sample. The auxiliary classifier uses the low-dimensional representation to extract the category information of the micro-Doppler feature samples and generates category labels. The output target of the decoder is the input of the encoder; the output target of the auxiliary classifier is the label input to the encoder.

3. The micro-Doppler radar data enhancement method according to claim 2, characterized in that, The process of using the trained autoencoder based on the auxiliary classifier to extract a low-dimensional representation from a given real sample, using the trained time-frequency causal generator to generate a new representation from the low-dimensional representation based on the class label of the real sample as the generation condition, and using the trained autoencoder based on the auxiliary classifier to generate synthetic samples from the new representation specifically includes: Using the encoder and quantizer in the trained autoencoder based on the auxiliary classifier, a low-dimensional representation is extracted from a given real sample; Using the trained time-frequency causal generator, combined with the class labels of the given real samples as generation conditions, a new representation is generated from the extracted low-dimensional representation; Synthetic samples are generated from the newly generated representation using the decoder in the trained autoencoder based on the auxiliary classifier.

4. The micro-Doppler radar data enhancement method according to claim 2, characterized in that, The process of generating synthetic samples from the new representation using the trained auxiliary classifier-based automatic encoding and decoding further includes: If it is necessary to generate samples with fuzzy categories, low-dimensional representations are extracted from two different real samples. A hybrid method module is designed to obtain hybrid representations and soft labels through the hybrid method module. Synthetic samples are then generated sequentially through the time-frequency causal generator and the decoder of the autoencoder based on the auxiliary classifier, which are trained in turn. The synthetic samples are then associated with the soft labels. Finally, the auxiliary classifier of the autoencoder based on the auxiliary classifier is used to select the synthetic samples that meet the requirements.

5. The micro-Doppler radar data enhancement method according to claim 4, characterized in that, The process of generating synthetic samples sequentially through the trained time-frequency causal generator and the decoder of the automatic encoder-decoder based on the auxiliary classifier, and then further includes: If the class information is inconsistent during the sample synthesis process, the synthesized sample is discarded by the auxiliary classifier in the trained automatic encoder-decoder based on the auxiliary classifier.

6. The micro-Doppler radar data enhancement method according to claim 1, characterized in that, The conditional distribution of the low-dimensional representation is as follows: ； in, The first dimension represents the low-dimensional representation. Column vector, This represents the category labels associated with the low-dimensional representation. express arrive Vector sequence, The lower dimension represents a certain column. To represent a row in lower dimensions, This represents the number of rows in the low-dimensional representation. In low-dimensional representation, the first The basic Doppler frequency of the column, In low-dimensional representation, the first The first in the list The amplitude of each element, In low-dimensional representation, the first The first in the list The amplitude of each element, In low-dimensional representation, the first The first in the list The amplitude of each element, In low-dimensional representation, the first The first in the list The amplitude of each element, The conditional distribution representing the low-dimensional representation. Indicates the first The amplitude distribution of the fundamental frequency elements in the column, the amplitude distribution depends on the preceding List, Indicates the first Columns greater than The amplitude distribution of the fundamental frequency elements, the amplitude distribution depends on the fundamental frequency elements. arrive Doppler frequency elements Indicates the first Less than in the column The amplitude distribution of the fundamental frequency elements, the amplitude distribution depends on arrive Doppler frequency elements.

7. A micro-Doppler radar data enhancement system, characterized in that, The micro-Doppler radar data enhancement system is used to implement the micro-Doppler radar data enhancement method according to any one of claims 1-6, and the micro-Doppler radar data enhancement system includes: An automatic codec training module is used to input micro-Doppler feature samples into an automatic codec based on an auxiliary classifier, and output synthesized micro-Doppler feature samples and class labels to complete the training of the automatic codec based on the auxiliary classifier. The generator training module is used to input the low-dimensional representation and corresponding category label obtained by the automatic encoder and decoder based on the auxiliary classifier into the time-frequency causality-based generator, and output a probability distribution that considers the time causality relationship to complete the training of the time-frequency causality-based generator. The sample generation module is used to extract a low-dimensional representation from a given real sample using the trained autoencoder based on the auxiliary classifier, generate a new representation from the low-dimensional representation using the trained time-frequency causal generator based on the class label of the real sample as the generation condition, and generate a synthetic sample from the new representation using the trained autoencoder based on the auxiliary classifier.

8. A terminal, characterized in that, The terminal includes: a memory, a processor, and a micro-Doppler radar data enhancement program stored in the memory and executable on the processor. When the micro-Doppler radar data enhancement program is executed by the processor, it implements the steps of the micro-Doppler radar data enhancement method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a micro-Doppler radar data enhancement program, which, when executed by a processor, implements the steps of the micro-Doppler radar data enhancement method as described in any one of claims 1-6.

Images