A method for cross-domain recognition of human body action based on a diffusion model

Abstract

The application belongs to the technical field of human action recognition, and particularly relates to a millimeter wave radar human action cross-domain recognition method based on a diffusion model. The method comprises the following steps: constructing a data expansion module, which is used for learning source domain and target domain data distribution and generating expansion samples; constructing a feature alignment module, which receives corresponding confidence scores of image feature outputs, wherein the confidence scores are binary matrices consistent with feature map sizes; constructing a feature extraction module, which is used for extracting features of input image data and transmitting the extracted image features to the feature alignment module, and is also used for taking the confidence scores output by the image alignment module as weights to weight the image features and take the weighted image features as updated image features; constructing an FC layer, which is used for recognizing human actions according to the updated image features; and using a trained neural network to realize human action cross-domain recognition.

Description

Technical Field

[0001] This invention belongs to the field of human motion recognition technology, specifically relating to a millimeter-wave radar cross-domain human motion recognition method based on a diffusion model. Background Technology

[0002] In human motion recognition solutions, millimeter-wave radar possesses unique advantages such as non-contact operation and strong adaptability, and has gradually become one of the mainstream solutions. The core principle of millimeter-wave radar is to accurately measure the target's distance, speed, and angle by emitting millimeter-wave electromagnetic waves and receiving the echo signals reflected from the target. The human motion data collected by millimeter-wave radar is highly correlated with environmental layout, sensor parameters, and individual user differences. Changes in these factors can cause significant variations in the data distribution. Furthermore, radar data acquisition is difficult and highly specialized; existing datasets are often limited in size and lack sufficient sample diversity, making it difficult to collect enough labeled data independently for each user to train a personalized model in practical applications. These problems make it difficult for the model to adapt to the complex needs across users and scenarios in real-world situations, leading to a decrease in recognition accuracy.

[0003] Existing human motion recognition methods based on millimeter-wave radar primarily rely on deep learning (DL) technology. Traditional DL methods employ supervised training on large-scale labeled data to enable the model to learn the complex mapping relationship between radar signals and specific motion categories. However, these traditional deep learning methods heavily depend on the assumption that training and test data follow the same distribution. Once real-world applications involve changes in data distribution across users, environments, or devices, the feature distribution learned by the model in the source domain shifts from that in the target domain, leading to a significant decline in recognition performance and exhibiting problems such as insufficient generalization ability and poor robustness. The source domain refers to the training domain with a large amount of labeled data, while the target domain refers to the deployment domain in practical applications that is unlabeled or has scarce labeled data.

[0004] Domain adaptation is an effective means of addressing the problem of data distribution discrepancies. Its core idea is that when there are differences in data distribution between the source and target domains, knowledge learned in the source domain can be transferred to the target domain by reducing or aligning the distribution differences, allowing the model to maintain good performance in the target domain. However, such schemes still suffer from low accuracy and insufficient generalization ability, and they require a large number of samples, paying little attention to the impact of insufficient samples on the effectiveness of millimeter-wave radar human motion domain adaptation recognition methods. Summary of the Invention

[0005] In view of this, the present invention proposes a cross-domain human motion recognition method based on a diffusion model for millimeter-wave radar. This method can improve the accuracy of cross-domain recognition while reducing the need for labeled data.

[0006] The technical solution for implementing the present invention is as follows:

[0007] In a first aspect, the present invention provides a millimeter-wave radar method for cross-domain human motion recognition based on a diffusion model, comprising the following steps: Sample data augmentation: Construct a data augmentation module to learn the data distribution of the source and target domains and generate augmented samples; Neural Network Construction: A feature alignment module is constructed, which receives the confidence scores corresponding to the image features and outputs them. The confidence scores are binary matrices of the same size as the feature map. A feature extraction module is constructed, which extracts features from the input image data and transmits the extracted image features to the feature alignment module. It also uses the confidence scores output by the image alignment module as weights to weight the image features, resulting in updated image features. A fully connected (FC) layer is constructed to perform human action recognition based on the updated image features. Neural network training involves training the neural network using augmented sample data. Action recognition: The collected millimeter-wave radar images of human actions are input into the neural network to achieve cross-domain recognition of human actions.

[0008] Optionally, the updated image features described in this invention are:

[0009] in, This represents the confidence score output by the image alignment module; This represents the features extracted from the input image data by the feature extraction module.

[0010] Optionally, the adversarial loss function during training of the feature alignment module of the present invention is as follows:

[0011] Wherein, the source domain labels are a matrix of all zeros with the same size as the feature map, and the target domain labels are a matrix of all one values ​​with the same size as the feature map. The representation domain discriminator for source domain image features confidence score, The representation domain discriminator for target image features confidence score, Indicates the distribution of data from the source domain. Medium-sampled source domain samples And according to Calculate the expectation. Indicates the distribution of data from the target domain Mid-sample target domain samples And according to Calculate the expected value.

[0012] Optionally, the loss function for training the feature extraction module of the present invention is designed as follows:

[0013] in, The multi-class cross-entropy loss used in the representation method This indicates the feature extraction module. Indicates a fully connected layer. Represents source domain samples Its corresponding real label The joint probability distribution.

[0014]

[0015] in, Indicates the feature alignment module. Indicates the first Data after step diffusion.

[0016] The feature extraction module is trained by combining the two loss functions mentioned above.

[0017] Optionally, the present invention constructs a feature alignment module based on the Denoising Diffusion Probability Model (DDPM).

[0018] Secondly, the present invention provides a millimeter-wave radar human motion cross-domain recognition device based on a diffusion model, comprising: The feature alignment module receives the confidence scores corresponding to the image features output, wherein the confidence scores are binary matrices of the same size as the feature map; The feature extraction module is used to extract features from the input image data and transmit the extracted image features to the feature alignment module. On the other hand, the confidence score output by the image alignment module is used as a weight to weight the image features and then the weighted image features are used as the updated image features. The FC layer is used for human motion recognition based on updated image features.

[0019] Beneficial effects: First, the method of this invention is applied to cross-domain recognition of human movements using millimeter-wave radar. Firstly, a data augmentation module is constructed to generate a large number of augmented samples in both the source and target domains, ensuring sufficient data for training. Secondly, a feature alignment module is constructed, which trains the domain-adaptive recognition network using adversarial learning loss, enabling the network to extract common features between the source and target domains, thus enhancing the network's generalization ability even when target domain labels are missing. Simultaneously, a feature extraction module is constructed, which weights the network output through feature alignment attention and inputs it into the next iteration, making the model more sensitive to differences in features, accelerating convergence while improving model performance.

[0020] Secondly, this invention can be applied to real-world scenarios with complex cross-user and cross-platform requirements, such as health monitoring and smart homes, which can improve the accuracy of cross-domain recognition while reducing the need for labeled data. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A cross-domain human motion recognition method based on a diffusion model using millimeter-wave radar; Figure 2 A schematic diagram of a sample radar signature dataset from the University of Glasgow. Figure 3 Schematic diagram of TI radar human motion dataset samples; Figure 4 Data augmentation results; Figure 5 Ablation experiment results; Figure 6 Results of method comparison. Detailed Implementation

[0023] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0024] It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0025] It should be noted that the following description covers various aspects of embodiments within the scope of the appended claims. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0026] The design concept of this invention is as follows: First, a data augmentation module is constructed, which gradually converges the initial distribution to the target distribution through information propagation and diffusion during the iteration process, thereby greatly increasing the number of samples; second, adversarial learning forces the network to extract common features between the source domain and the target domain; finally, a feature alignment attention mechanism is added, which makes the model more sensitive to the parts where features differ, thereby accelerating the convergence speed and improving the model performance.

[0027] like Figure 1 As shown in the figure, this embodiment presents a millimeter-wave radar method for cross-domain human motion recognition based on a diffusion model. The specific process is as follows: Sample data augmentation: Construct a data augmentation module to learn the data distribution of the source and target domains and generate augmented samples; Neural Network Construction: A feature alignment module is constructed, which receives the confidence scores corresponding to the image features and outputs them. The confidence scores are binary matrices of the same size as the feature map. A feature extraction module is constructed, which extracts features from the input image data and transmits the extracted image features to the feature alignment module. It also uses the confidence scores output by the image alignment module as weights to weight the image features, resulting in updated image features. A fully connected (FC) layer is constructed to perform human action recognition based on the updated image features. Neural network training involves training the neural network using augmented sample data. Action recognition: The collected millimeter-wave radar images of human actions are input into the neural network to achieve cross-domain recognition of human actions.

[0028] The following section will provide a detailed explanation of the sample data expansion and neural network construction processes described above.

[0029] like Figure 1 As shown, this invention proposes a millimeter-wave radar method for cross-domain human motion recognition based on a diffusion model, the specific steps of which are as follows: Step 1: Based on the denoising diffusion probabilistic model (DDPM) data augmentation module, learn the data distribution of the source domain and the target domain, and generate a large number of augmented samples respectively.

[0030] To alleviate the data sample pressure during network training, a data augmentation module (DDPM) is used to learn the distribution of the original data samples and generate a large number of independent and identically distributed samples. The basic idea is to treat the input as a probability distribution, gradually add noise to the data, and then learn an inverse denoising process to gradually recover new samples conforming to the original data distribution from pure noise. DDPM consists of two processes: a forward process and a backward process. Both the forward and backward processes are parameterized Markov chains: the forward process gradually adds noise to the original data, and the backward process generates data from the noise.

[0031] Forward process: For the original data Each step in the diffusion process is related to the previous step. Add Gaussian noise: (1) in, It is a Gaussian noise variance sequence. This represents the total number of steps in the diffusion process. By continuously adding noise, when... When it is large enough, the final result is It will infinitely tend towards Gaussian random noise. The entire diffusion process is a Markov chain: (2) In actual diffusion processes, raw data can be used directly. To any Step Sampling was performed to obtain .make and By continuously recursively applying this method, we can eventually obtain: (3) Reverse process: the real It is unknown, but it can be obtained by fitting using a neural network. For the original data... Each step in the diffusion process is related to the previous step. Add Gaussian noise: (4) (5) in, , The distribution is a parameterized Gaussian distribution, whose mean and variance are given by the trained neural network. For the parameters of the neural network, , It needs to be obtained from the network output.

[0032] Step 2: Construct a feature alignment module and train the domain adaptation recognition network using adversarial learning loss so that the network can extract common features between the source and target domains.

[0033] In Generative Adversarial Networks (GANs), the generator G is responsible for generating samples to confuse the discriminator, while the discriminator... It attempts to distinguish between real and fake samples, aiming to generate data consistent with the distribution of the training set. The overall optimization function of GAN is shown below: (6) In the formula, represent Based on real data distribution, represent The data is derived from the generated data distribution. The goal of GANs is to optimize the generator to minimize the loss while training the discriminator to maximize the probability of correctly assigning labels, so that the model eventually reaches Nash equilibrium. Inspired by GANs, an adversarial domain adaptation approach is used to guide the feature learning process of the domain adaptation recognition network. The training process of adversarial domain adaptation is a game between the feature extractor and the domain discriminator: the domain discriminator is trained to distinguish between source domain features and target domain features, while the feature extractor learns feature representations with domain invariance to confuse the domain discriminator. After training, the features learned by the neural network are class discriminative and are common features of the source and target domains. Therefore, the following adversarial loss function can be constructed: (7) As can be seen from the loss function, the core of the feature alignment module is a discriminator D. The training objective of this module is to determine whether the input features are extracted from the source domain. Or features extracted from the target domain The source domain labels are represented as an all-zero matrix of the same size as the feature map, and the target domain labels are represented as an all-one matrix of the same size as the feature map. The specific method for aligning the differences between the source and target domains involves subtracting the extracted features from the labels and calculating the loss. Backpropagation then enables the domain discriminator D to better distinguish between source and target domain features, thereby better supervising the feature extraction module's feature selection.

[0034] Step 3: Construct a feature extraction module and weight the network output by using feature alignment attention.

[0035] For target image features Confidence score of the domain discriminator show Is it a local match? The distribution of confidence scores. A lower confidence score indicates that the region is well aligned, making it difficult for the discriminator to distinguish the source of the features; A higher confidence score indicates poor alignment in the region, and the domain discriminator can distinguish the source of the features. It can be observed that the confidence score and the attention function have some similarity. Therefore, feature alignment attention can be added, and... The confidence scores are used to weight the features, encouraging the model to focus on matching features in poorly aligned regions. Furthermore, tanh activation is added as a normalization layer to normalize the confidence scores to the [0,1] interval. Then, a residual attention mechanism is used to compute new features. (8) The feature extraction module has two main tasks: First, it needs to ensure that the extracted features can perform the subsequent recognition task with high quality. Since only the source domain data has labels to support training, the loss function is defined as: (9) in, The multi-class cross-entropy loss used in the representation method This indicates the feature extraction module. This indicates a fully connected layer.

[0036] Secondly, the feature extraction module needs to extract common features between the source and target domains, so that the extracted features can confuse the discriminator trained separately. Therefore, the loss function is defined as: (10) in, This indicates the feature alignment module.

[0037] Based on the loss function, it is clear that to satisfy the two tasks of the feature extraction module, two loss functions need to be combined for training. This is because the domain discriminator... With feature extraction module The two training tasks are opposite, and the ideal network reaches Nash equilibrium after training.

[0038] Step 4: Construct the FC layer for human motion recognition based on the updated image features.

[0039] Thus, the construction of a millimeter-wave radar human motion cross-domain recognition neural network based on a diffusion model has been completed.

[0040] Example: In this embodiment, a field experiment was conducted based on the University of Glasgow radar feature dataset and the TI radar human motion dataset. Both datasets contain four types of actions: standing up, sitting down, drinking water, and falling. Each action has 100 sets of data. The time-frequency analysis results are as follows: Figure 2 and Figure 3As shown, the former is considered as the labeled source domain, and the latter as the unlabeled target domain data. The goal of the experiment is to transfer the labeled knowledge from the data to the unlabeled target domain, and ultimately achieve high-precision recognition in the target domain. Since the radar frequency band, observation angle, experimental scene, and subjects of the source and target domains are all different, the two sets of data have significant distribution differences, which can be used to verify the proposed method.

[0041] First, the data is augmented using step 1. After training, 1000 additional data sets are added for each behavior. The original data and augmented data of the two datasets are as follows: Figure 4 As shown. To verify the effectiveness of the algorithm, we compared the proposed algorithm with baseline methods and conducted ablation experiments to verify the effectiveness of each module. The experimental results are as follows. Figure 5 As shown in the figure, the proposed cross-domain human motion recognition method can effectively improve the target domain recognition accuracy. Furthermore, experiments were conducted on the selected dataset using other methods and compared with the proposed method; the experimental results are shown below. Figure 6 As shown in the figure, the superiority of the proposed method can be seen.

[0042] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A millimeter-wave radar method for cross-domain human motion recognition based on a diffusion model, characterized in that, Includes the following steps: Sample data augmentation: Construct a data augmentation module to learn the data distribution of the source and target domains and generate augmented samples; Neural network construction: Construct a feature alignment module, which receives the confidence scores corresponding to the image features output, and the confidence scores are binary matrices with the same size as the feature map; A feature extraction module is constructed to extract features from the input image data and transmit the extracted image features to the feature alignment module. On the other hand, the confidence score output by the image alignment module is used as a weight to weight the image features and obtain the updated image features. An FC layer is constructed to perform human action recognition based on the updated image features. Neural network training involves training the neural network using augmented sample data. Action recognition: The collected millimeter-wave radar images of human actions are input into the neural network to achieve cross-domain recognition of human actions.

2. The millimeter-wave radar human motion cross-domain recognition method based on a diffusion model according to claim 1, characterized in that, The updated image features are: in, This represents the confidence score output by the image alignment module; This represents the features extracted from the input image data by the feature extraction module.

3. The millimeter-wave radar human motion cross-domain recognition method based on a diffusion model according to claim 2, characterized in that, The adversarial loss function during the training of the feature alignment module is: Wherein, the source domain labels are a matrix of all zeros with the same size as the feature map, and the target domain labels are a matrix of all one values ​​with the same size as the feature map. The representation domain discriminator for source domain image features confidence score, The representation domain discriminator for target image features confidence score, Indicates the distribution of data from the source domain. Medium-sampled source domain samples And according to Calculated expectation, Indicates the distribution of data from the target domain Mid-sample target domain samples And according to The expected value is calculated.

4. The millimeter-wave radar human motion cross-domain recognition method based on a diffusion model according to claim 3, characterized in that, The loss function for training the feature extraction module is designed as follows: in, The multi-class cross-entropy loss used in the representation method This indicates the feature extraction module. Indicates a fully connected layer. Represents source domain samples Its corresponding real label The joint probability distribution of ; in, Indicates the feature alignment module. Indicates the first Data after step diffusion; The feature extraction module is trained by combining the two loss functions mentioned above.

5. The millimeter-wave radar human motion cross-domain recognition method based on a diffusion model according to claim 1, characterized in that, A feature alignment module is constructed based on the Denoising Diffusion Probability Model (DDPM).

6. A millimeter-wave radar human motion cross-domain recognition device based on a diffusion model, characterized in that, include: The feature alignment module receives the confidence scores corresponding to the image features output, wherein the confidence scores are binary matrices of the same size as the feature map; The feature extraction module is used to extract features from the input image data and transmit the extracted image features to the feature alignment module. On the other hand, it uses the confidence score output by the image alignment module as a weight to weight the image features and then uses the weighted image features as the updated image features. The FC layer is used for human motion recognition based on updated image features.

7. The millimeter-wave radar human motion cross-domain recognition device based on a diffusion model according to claim 6, characterized in that, The updated image features are: in, This represents the confidence score output by the image alignment module; This represents the features extracted from the input image data by the feature extraction module.

8. The millimeter-wave radar human motion cross-domain recognition device based on the diffusion model according to claim 7, characterized in that, The adversarial loss function during the training of the feature alignment module is: Wherein, the source domain labels are a matrix of all zeros with the same size as the feature map, and the target domain labels are a matrix of all one values ​​with the same size as the feature map. The representation domain discriminator for source domain image features confidence score, The representation domain discriminator for target image features confidence score, Indicates the distribution of data from the source domain. Medium-sampled source domain samples And according to Calculated expectation, Indicates the distribution of data from the target domain Mid-sample target domain samples And according to The expected value is calculated.

9. The millimeter-wave radar human motion cross-domain recognition device based on the diffusion model according to claim 8, characterized in that, The loss function for training the feature extraction module is designed as follows: in, The multi-class cross-entropy loss used in the representation method This indicates the feature extraction module. Indicates a fully connected layer. Represents source domain samples Its corresponding real label The joint probability distribution of ; in, Indicates the feature alignment module. Indicates the first Data after step diffusion; The feature extraction module is trained by combining the two loss functions mentioned above.

10. The millimeter-wave radar human motion cross-domain recognition device based on a diffusion model according to claim 1, characterized in that, The feature alignment module is a generative adversarial network.

Images