Activity identification method and device based on characterization guide multi-mode diffusion model

Through the multimodal diffusion model based on characterization guidance, multimodal representation and global representation are extracted, the overfitting problem of multimodal data in wearable human activity recognition is solved, the accuracy of activity recognition is improved, and the medical treatment is achieved. Application prospects in the fields of health care and exercise monitoring.

CN119066475AActive Publication Date: 2024-12-03ZHEJIANG UNIV
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411556232.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-04
Publication Date
2024-12-03
Estimated Expiration
2044-11-04

AI Technical Summary

Technical Problem

In the field of wearable human activity recognition, the overfitting of multimodal data is serious, and existing data augmentation methods are difficult to effectively utilize label-free data, resulting in insufficient diversity of generated data and loss of modal details.

Method used

Using a multimodal diffusion model based on characterization guidance, multimodal representation and global representation are extracted through self-supervised model and diffusion model training, and multimodal activity data with modal details and consistency are generated for training activity recognition models. , solve the overfitting problem and improve the recognition accuracy.

Benefits of technology

Effectively utilize labelless data on complex multimodal activity data for data augmentation, generate data with modal details and consistent activity information, improve the accuracy of activity recognition, and have broad application prospects in areas such as health care and sports monitoring .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066475A_ABST
    Figure CN119066475A_ABST
Patent Text Reader

Abstract

The invention discloses an activity identification method and device based on a characterization guide multi-mode diffusion model. Comprising the following steps: acquiring original multi-modal activity data, preprocessing the original multi-modal activity data, and constructing a label-free data set and a label data set; training a multi-modal self-supervised model by adopting the label-free data set so as to extract a multi-modal representation containing a complex mode and a global representation containing structural information from the data; introducing a modal layer guided by multi-modal characterization and a global layer guided by global characterization into the diffusion model, and training the diffusion model by adopting the label-free data set to generate multi-modal activity data; the activity recognition model is trained by using the labeled data set and the multi-modal activity data generated by the post-diffusion model based on the labeled data set, so that the over-fitting problem of the activity recognition model can be solved, the activity recognition accuracy is improved, and the method has wide application prospects in the fields of medical care, motion monitoring and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of wearable human activity recognition, and in particular relates to an activity recognition method and device based on a representation-guided multimodal diffusion model. Background Art

[0002] Wearable Human Activity Recognition (WHAR) uses wearable sensor data to infer human activities and has a wide range of applications in scenarios such as sports monitoring, health support, and work assistance. Today, more and more deep learning-based activity recognition models use multimodal data, such as accelerometer, gyroscope, and magnetometer data, to obtain complementary activity information, thereby achieving higher accuracy than using single-modal data.

[0003] Because sensor data must be labeled when collected, it is costly to obtain large amounts of diverse labeled data. Therefore, human activity recognition models are prone to overfitting problems, which are particularly serious in multimodal situations due to the increase in model parameters. The overfitting problem can usually be solved by increasing labeled data using data enhancement methods. In the field of wearable human activity recognition, existing data enhancement methods can be mainly divided into three types: traditional data enhancement methods, data enhancement methods based on Generative Adversarial Network (GAN), and data enhancement methods based on Diffusion Model.

[0004] Traditional data enhancement methods, such as rotation, permutation, and scaling, rely on human experience and may destroy key information contained in the data, such as temporal context information, sensor posture, signal change intensity, etc.

[0005] The data enhancement method based on GAN solves the above problems, but the task of the discriminator in GAN is to judge the authenticity of the sample, without judging the diversity of the generated data, resulting in the generator not being able to learn the complete data distribution. Therefore, GAN has the problem of insufficient diversity in generated data.

[0006] The data enhancement method based on the diffusion model has the training goal of optimizing the negative log-likelihood loss. It can learn the complete data distribution without the problem of insufficient generated data diversity. In the field of wearable human activity recognition, the data enhancement method based on the diffusion model uses statistical features, such as mean and variance, to guide the diffusion model. It can use a large amount of unlabeled data to enhance a small amount of labeled data. However, statistical features mainly describe the data as a whole and will lose local detail information, causing the enhanced data to lose some complex patterns and structures. In addition, this method is only designed for unimodal data, and directly extending it to multimodal data will lose modal details. Summary of the invention

[0007] In view of the above, an object of the present invention is to provide an activity recognition method and apparatus based on a characterization-guided multimodal diffusion model, so as to solve the problem of how to use the diffusion model to improve the accuracy of activity recognition based on multimodal data by the activity recognition model.

[0008] To achieve the above-mentioned object of the invention, an embodiment provides an activity recognition method based on a representation-guided multimodal diffusion model, comprising the following steps: Data preprocessing stage: Obtain the original multimodal activity data and construct unlabeled and labeled datasets after preprocessing: Self-supervised model training phase: Use unlabeled datasets to train a multimodal self-supervised model to extract multimodal representations containing complex patterns and global representations containing structural information from the data; Diffusion model training phase: introduce a modal layer guided by multimodal representation and a global layer guided by global representation into the diffusion model, and use an unlabeled dataset to train the diffusion model to generate multimodal activity data; Activity recognition model training phase: The activity recognition model is trained using the labeled dataset and the multimodal activity data generated by the trained diffusion model based on the labeled dataset, and the trained activity recognition model is used for activity recognition.

[0009] Preferably, an unlabeled dataset is used to train a multimodal self-supervised model so that it can extract multimodal representations containing complex patterns and global representations containing structural information from the data, including: Build a self-supervised model with multiple independent modality encoders, attention networks, and multiple independent modality decoders; Unlabeled data samples are selected from the unlabeled data set and randomly masked to obtain masked data, and multiple independent modal encoders are used to encode each modality in the masked data to obtain a multimodal representation containing complex patterns; After concatenating the multimodal representations, the attention network is used to obtain a global representation containing structural information, and multiple independent modality decoders are used to decode each modality in the global representation to obtain reconstructed data. A mask reconstruction loss is constructed based on the reconstructed data and unlabeled data samples, and the mask reconstruction loss is used to train the self-supervised model to update the self-supervised model parameters.

[0010] The mask reconstruction loss uses mean square error, expressed as: ; in, represents the mask reconstruction loss, and represent unlabeled data samples and reconstructed data respectively, Represents unlabeled data samples From the distribution of the unlabeled dataset , Express expectations, Represents the square of the L2 norm.

[0011] Preferably, a modal layer guided by multimodal representation and a global layer guided by global representation are introduced into the diffusion model, and an unlabeled dataset is used to train the diffusion model to generate multimodal activity data, including: (1) Multiple modal layers and global layers are introduced into the denoising network of the diffusion model, wherein each modal layer and global layer includes multiple continuous downsampling modules and multiple upsampling modules, and the global layer also includes a fusion module. The number of downsampling modules and upsampling modules in each modal layer is equal, and a jump connection is formed between the downsampling modules and the corresponding upsampling modules; the number of downsampling modules and upsampling modules in the global layer is also equal, the output of the last downsampling module is directly connected to the input of the corresponding upsampling module, and the remaining downsampling modules are connected to the corresponding upsampling modules. Jump connections are formed; (2) Selecting unlabeled data samples from the unlabeled dataset, and using the trained self-supervised model to extract multimodal representations and global representations based on the unlabeled data samples, and averaging the multimodal representations and global representations along a fixed length to obtain the multimodal representations and global representations used as guidance; (3) For each time step, the following process is performed: (3-1) Noising the input data to obtain noisy data, and using two encoding networks to encode the time steps to obtain the first time step encoding and the second time step encoding; (3-2) Using the noisy data as the input of the denoising network, each modal layer processes the noisy data corresponding to one modality, using multiple downsampling modules in each modal layer, and using the first time step encoding and the modal representation corresponding to each modality extracted from the multimodal representation as the guiding condition, extracting the modal features of each modality from the input noisy data; (3-3) Using the fusion module of the global layer to fuse the modal features of all modalities to obtain fusion features, using multiple downsampling modules and upsampling modules in the global layer, and taking the second time step encoding and global representation as guiding conditions, extracting global features from the input fusion features; (3-4) utilizing multiple upsampling modules in each modality layer, and taking the first time step encoding and the modal representation corresponding to each modality extracted from the multimodal representation as guiding conditions, extracting the denoised data of each modality from the input global features, and the denoised data of all modalities constitute the generated multimodal activity data; (4) A reconstruction loss is constructed based on the unlabeled data samples and the generated multimodal activity data, and the reconstruction loss is used to train the diffusion model to update the denoising network parameters in the diffusion model.

[0012] Preferably, the same processing process adopted by each downsampling module is: calculating the first scaling parameter and the first translation parameter based on the input guidance condition, and then scaling and translating the input data or input features of each downsampling module using the first scaling parameter and the first translation parameter, and then obtaining the output features after convolution and activation processing; for each modal layer, the input data of the first downsampling module is the noisy data, the input features of the other downsampling modules are the output features of the previous downsampling modules of each modal layer, and the output of the last downsampling module is the modal feature; for the global layer, the input features of the first downsampling module are the fusion features, and the input features of the other downsampling modules are the output features of the previous downsampling modules in the global layer; The same processing procedure adopted by each upsampling module is: the second scaling parameter and the second translation parameter are calculated based on the input guidance condition, and then the input features of each upsampling module are scaled and translated using the second scaling parameter and the second translation parameter, and then the output features are obtained after deconvolution and activation processing; for the global layer, the input features of the first upsampling module are the output features of the last downsampling module in the global layer, the input features of other upsampling modules are the output features of the previous upsampling module in the global layer and the output features of the downsampling modules in the same layer through jump connections, and the output features of the last upsampling module are the global features; for each modal layer, the input features of the first upsampling module are the global features output by the last upsampling module in the global layer and the output features of the downsampling modules in the same layer through jump connections, the input features of other upsampling modules are the output features of the previous upsampling modules in the modal layer and the output features of the downsampling modules in the same layer through jump connections, and the output of the last upsampling module is the denoised data.

[0013] Preferably, the modal features of all modalities are fused using a fusion module of the global layer to obtain fused features, including: The modal features of all modalities are spliced ​​to obtain the spliced ​​features, and the spliced ​​features are input into the multi-layer convolutional layer to obtain the modal weight matrix. At the same time, the spliced ​​matrix is ​​input into two convolutional layers to obtain the first fusion parameter and the second fusion parameter. The modal weight matrix is ​​processed with the first fusion parameter to obtain the modal weight, and the modal weight is processed with the second fusion parameter to obtain the fusion feature.

[0014] Preferably, the reconstruction loss adopts mean square error, which is expressed as: ; in, represents the reconstruction loss, and represent unlabeled data samples and generated multimodal activity data, respectively. Represents unlabeled data samples From the distribution of the unlabeled dataset , Indicates the noise in the noise adding process comes from a mean of 0 and a variance of The normal distribution of represents the time step, Represents the square of the L2 norm.

[0015] Preferably, training the activity recognition model using the labeled data set and the multimodal activity data generated by the trained diffusion model based on the labeled data set includes: Select labeled data samples from the labeled data set, use the trained self-supervised model to generate multimodal representations and global representations based on the labeled data samples, and take the average along a fixed length to obtain the multimodal representations and global representations used as guidance; The trained diffusion model is used to perform multi-step denoising and denoising on the initial random noise, and multimodal representation and global representation are introduced as guiding conditions in the denoising process to generate multimodal activity data. The labeled data samples selected from the labeled dataset and the generated modal activity data are input into the activity recognition model to obtain the classification results, and the classification loss is constructed based on the classification results and the true labels. The activity recognition model is trained using the classification loss to update the activity recognition model parameters.

[0016] Preferably, the classification loss adopts cross entropy loss, which is expressed as: ; in, represents the classification loss, represents the true label, Indicates the classification results corresponding to the labeled data samples, Represents the classification result corresponding to the generated modal activity data, represents the cross entropy loss, Express expectations, Represents labeled data samples From the distribution of the labeled dataset , The noise of the noise adding process comes from a mean of 0 and a variance of The normal distribution of .

[0017] To achieve the above-mentioned object of the invention, an embodiment of the present invention further provides an activity recognition device based on a representation-guided multimodal diffusion model, comprising: The data preprocessing unit is used to obtain the original multimodal activity data and construct unlabeled and labeled data sets after preprocessing: A self-supervised model training unit, which is used to train a multimodal self-supervised model using an unlabeled data set so that it can extract a multimodal representation containing complex patterns and a global representation containing structural information from the data; A diffusion model training unit, which is used to introduce a modal layer guided by multimodal representation and a global layer guided by global representation into the diffusion model, and train the diffusion model with an unlabeled data set to generate multimodal activity data; The activity recognition model training unit is used to train the activity recognition model using the labeled data set and the multimodal activity data generated by the trained diffusion model based on the labeled data set, and the trained activity recognition model is used for activity recognition.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention introduces the modal layer guided by multimodal representation and the global layer guided by global representation into the diffusion model. It can use a large amount of unlabeled data to perform data enhancement on a small amount of labeled data on complex multimodal activity data, generate complex multimodal activity data with modal details and consistent activity information, solve the overfitting problem of activity recognition models, and improve the accuracy of activity recognition. It has broad application prospects in the fields of medical care and sports monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flow chart of an activity recognition method based on a representation-guided multimodal diffusion model provided by an embodiment; Figure 2 is a flowchart of an activity recognition method based on a characterization-guided multimodal diffusion model provided by an embodiment; Figure 3 is a schematic diagram of the structure of the self-supervision model provided in the embodiment; Figure 4 is a schematic diagram of the structure of a denoising network in a diffusion model provided in an embodiment; Figure 5 is a schematic diagram of the structure of an upsampling module and a downsampling module provided in an embodiment; Figure 6 It is a structural schematic diagram of an activity recognition device based on a characterization-guided multimodal diffusion model provided in an embodiment. DETAILED DESCRIPTION

[0021] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0022] like Figure 1 and Figure 2As shown, an activity recognition method based on a characterization-guided multimodal diffusion model provided by an embodiment includes the following steps: S1, data preprocessing stage: obtain the original multimodal activity data and construct unlabeled datasets and labeled datasets after preprocessing.

[0023] The specific process of the data preprocessing stage in the embodiment includes: S1-1, obtain multimodal time series data of different users performing different activities collected by wearable devices, including Multimodal activity data collected by various sensors.

[0024] S1-2, perform outlier elimination and normalization on the collected multimodal activity data, use a sliding window to divide the processed data to obtain data samples, and divide them into unlabeled data sets and labeled data sets according to whether they contain labels. The data samples of the unlabeled data set are , , Indicates sensor The corresponding single modality activity data, Indicates the number of sensors, represents the length of the sliding window, Indicates sensor The number of channels, Represents the data space, and the data samples of the labeled data set are ,in Indicates that there is a true label The data sample, , Represents a collection of activity category labels.

[0025] Specifically, the preprocessing process is: a) Eliminate outliers in multimodal activity data, replace values ​​outside the normal range with the maximum or minimum value of the normal range, and replace missing values ​​with 0; b) Normalize the processed data, the formula is as follows: ; in is the original value, is the minimum value of the channel where the original value is located, is the maximum value of the channel where the original value is located, is the normalized value; c) Artificially set the time window size based on experience , the overlapping degree of sliding windows is 50%, and the normalized data is divided by sliding windows to obtain data samples ; d) An unlabeled dataset is constructed based on each data sample, and labeled data samples and corresponding labels are used as samples that need data enhancement to form a labeled dataset.

[0026] S1-3, divide the unlabeled dataset and the labeled dataset into batches of fixed size.

[0027] In this step, the total number of unlabeled dataset batches is , the total number of batches of labeled datasets is , the formula is as follows: ; ; in, represents the total number of samples in the unlabeled dataset, Represents the batch size of the unlabeled dataset, which is set manually based on experience. Represents the total number of samples in the labeled dataset, Represents the batch size of the labeled dataset, which is set manually based on experience.

[0028] S2, self-supervised model training phase: an unlabeled dataset is used to train a multimodal self-supervised model to extract multimodal representations containing complex patterns and global representations containing structural information from the data.

[0029] In the embodiment, Figure 3 As shown in the figure, the self-supervised model includes multiple independent modal encoders, an attention network, and multiple independent modal decoders. Based on the self-supervised model, multimodal representations containing complex patterns and global representations containing structural information can be extracted from the input multimodal activity data. The specific training process of the self-supervised model is: S2-1, select a batch of unlabeled samples from the unlabeled dataset, and for each unlabeled data sample in the selected batch , repeat step S2-2.

[0030] S2-2, for unlabeled data samples Perform random masking to obtain masked data Afterwards, the data is reconstructed through self-supervised model calculation; During random mask processing, the unlabeled data samples Each value in has a 20% probability of being replaced by 0, realizing the masking process; use K The mask data is encoded by two independent modal encoders. Each mode in k Encode to obtain multimodal representations containing complex patterns ,in, , representing a single mode k The corresponding modal features are: represents the feature dimension; wherein each modality encoder has the same structure, consisting of two one-dimensional convolutional layers, where the convolution operation is performed along the time dimension; Stitching multimodal representations Then use the attention network to obtain a global representation containing structural information Specifically, the multimodal representation All the individual modal representations in the concatenation are concatenated to obtain the global representation ; use K The independent modality decoders are used to represent the global Decode each mode in the reconstructed data ; Among them, each modality decoder has the same structure, which consists of a fully connected layer.

[0031] S2-3, for all unlabeled data samples in the current batch , calculate the reconstruction data and unlabeled data samples The mask reconstruction loss , and use the mask reconstruction loss Train the self-supervised model to update the self-supervised model parameters, minimizing the mask reconstruction loss during training For the goal, Using mean square error, it is expressed as: ; in, Represents unlabeled data samples From the distribution of the unlabeled dataset , Express expectations, Represents the square of the L2 norm.

[0032] S2-4, when the number of training iterations has not been reached, that is, when the training is not completed, steps S2-1 to S2-3 are repeated until all batches of the unlabeled dataset participate in the training, and the training ends when the specified number of training iterations is reached.

[0033] S3, diffusion model training stage: a modal layer guided by multimodal representation and a global layer guided by global representation are introduced into the diffusion model, and an unlabeled dataset is used to train the diffusion model to generate multimodal activity data.

[0034] In the embodiment, Figure 4As shown, multiple modal layers and global layers are introduced into the denoising network of the diffusion model, wherein each modal layer and global layer includes multiple continuous downsampling modules and multiple upsampling modules, and the global layer also includes a fusion module, wherein the number of downsampling modules and upsampling modules in each modal layer is equal, and a jump connection is formed between the downsampling modules and the corresponding upsampling modules; the number of downsampling modules and upsampling modules in the global layer is also equal, the output of the last downsampling module is directly connected to the input of the corresponding upsampling module, and a jump connection is formed between the remaining downsampling modules and the corresponding upsampling modules. The specific training process of the diffusion model is: S3-1, select a batch of unlabeled data samples from the unlabeled data set, and for each unlabeled data sample in the batch , repeat steps S3-2 ~ S3-3.

[0035] S3-2, fix the self-supervised model, and use the trained self-supervised model to extract multimodal representation and global representation based on unlabeled data samples; specifically, the unlabeled data samples are input into the trained self-supervised model and multimodal representation is extracted through forward reasoning. and global representation , and then the multimodal representation and global representation Along a fixed length (such as the sliding window size l ) is averaged to obtain a multimodal representation used as a guide and global representation ,in, Representing modality k The corresponding modal representation.

[0036] S3-3, multimodal representation and global representation are used as guiding conditions, and the diffusion model is trained on unlabeled data samples. Specifically, the training process of the diffusion model includes: randomly sampling time steps ,in Indicates the maximum number of steps in the diffusion model, for each time step , carry out the following process; (a) Noising the input data to obtain the noisy data, and using two encoding networks to encode the time steps to obtain the first time step encoding and the second time step encoding; Specifically, for the input data Add noise to get the noised data ,in, Representing modality k At time step t The corresponding noise-added data, specific input data and from the normal distribution Random noise collected in According to the time step The noise addition process is achieved by adding in proportion.

[0037] For each time step t , two encoding networks are used to encode the time step t Encode separately to get the first time step encoding and the second time step encoding , where the encoding network can adopt a fully connected layer.

[0038] (b) Noise-added data As the input of the denoising network, each modality layer processes one modality k The corresponding noise-added data , using multiple downsampling modules in each modality layer and encoding at the first time step and from multimodal representations Each modality extracted k Corresponding modal representation As a guide condition, the input noise data Extract each modality k At time step t The modal characteristics of , Indicates the feature length after downsampling and the data after noise addition The modal characteristics of all modes are .

[0039] like Figure 5 As shown, each downsampling module in each modal layer adopts the same processing process, specifically: based on the input guided condition first time step encoding and modal representation Calculate the first scaling parameter and the first translation parameter , for example, using SiLU activation function (SiLU) and fully connected layer (Linear) based on and Calculate the first scaling parameter and the first translation parameter , and then use and After scaling and translating the input data or input features of each downsampling module, the output features are obtained after one-dimensional convolution (Conv1D) and SiLU activation function (SiLU), which is expressed as: ; ; in, Represents the input data or input features for mode k. The input data for the first downsampling module is the noised data. , the input features of other downsampling modules are the output features of the previous downsampling module in each modal layer, and the output of the last downsampling module is the modal k The corresponding modal features , Represents the output features of each downsampling module in each modality layer.

[0040] (c) Using the fusion module of the global layer to fusion the modal features of all modalities Fusion to obtain fusion features , using multiple downsampling modules and upsampling modules in the global layer and encoding at the second time step and global representation As a guiding condition, the fusion features from the input Extracting global features ; Modal features for all modes in the embodiment Fusion to obtain fusion features The specific process is: for all modal features After concatenating the modal features of each mode, the concatenated features are obtained. , and the splicing features Input into the multi-layer convolutional layer to obtain the modal weight matrix , while concatenating the matrices Input into two convolutional layers respectively to get the first fusion parameter and the second fusion parameter , using the first fusion parameter Modal weight matrix Processing to obtain modal weight , using the second fusion parameter Weight of modal Processing to obtain fusion features , the formula is: ; ; like Figure 5 As shown in Figure 2, each downsampling module in the global layer uses the same processing as the downsampling module in the modal layer, specifically: the second time step encoding based on the input guidance condition and global representation Calculate the first scaling parameter and the first translation parameter , for example, using SiLU activation function (SiLU) and fully connected layer (Linear) based on and calculate and , and then use and After scaling and translating the input data or input features of each downsampling module, the output features are obtained after one-dimensional convolution (Conv1D) and SiLU activation function (SiLU), which is expressed as: ; ; in, Represents the input data or input features of each downsampling module in the global layer. The input data of the first downsampling module is the fusion feature , the input features of other downsampling modules are the output features of the previous downsampling module in the global layer, Represents the output features of each downsampling module in the global layer.

[0041] like Figure 5 As shown in Figure 2, the specific implementation process of each upsampling module in the global layer is basically the same as that of the downsampling module, except that the one-dimensional convolution replaces the one-dimensional deconvolution layer. The specific process is: based on the input guided condition, the second time step encoding and global representation Calculate the second scaling parameter and the second translation parameter , for example, using SiLU activation function (SiLU) and fully connected layer (Linear) based on and calculate and , and then use and After scaling and translating the input features of each upsampling module, the output features are obtained after one-dimensional deconvolution (DeConv1D) and SiLU activation function (SiLU), which is expressed as: ; ; in, Represents the input features of each upsampling module in the global layer, where the input features of the first upsampling module are the output features of the last downsampling module in the global layer, the input features of other upsampling modules are the output features of the previous upsampling module in the global layer and the output features of the downsampling modules in the same layer connected by jumps, and the output features of the last upsampling module are the global features , Represents the output features of each upsampling module in the global layer.

[0042] (d) Utilize multiple upsampling modules in each modality layer and encode at the first time step and from multimodal representations Each modality extracted from k Corresponding modal representation As a guiding condition, from the global features of the input Extract each modality k The denoised data , the denoised data of all modes As generated multimodal activity data.

[0043] like Figure 5 As shown, the upsampling module in each modal layer adopts the same processing as the upsampling module in the global layer, specifically: the first time step encoding based on the input guidance condition and modal representation Calculate the second scaling parameter and the second translation parameter , for example, using SiLU activation function (SiLU) and fully connected layer (Linear) based on and calculate and , and then use and After scaling and translating the input features of each upsampling module, the output features are obtained after one-dimensional deconvolution (DeConv1D) and SiLU activation function (SiLU), which is expressed as: ; ; in, Indicates that for the modal k The input features of the first upsampling module are the global features output by the last upsampling module of the global layer. The input features of other upsampling modules are the output features of the previous upsampling module in the modal layer and the output features of the downsampling module in the same layer connected by jumps. The output of the last upsampling module is the modal layer. k The denoised data , Indicates that for the modal k Output features of each upsampling module.

[0044] For all unlabeled data samples in the current batch, based on the unlabeled data samples and the generated multimodal activity data Constructing the reconstruction loss , using the reconstruction loss Train the diffusion model to update the denoising network parameters in the diffusion model, and minimize the reconstruction loss during training As the target, the reconstruction loss Using mean square error, it is expressed as: ; in, Indicates the noise in the noise adding process comes from a mean of 0 and a variance of The normal distribution of .

[0045] S3-4, when the number of training iterations is not reached, that is, when the training is not completed, repeat steps S3-1 to S3-3 until all batches of the unlabeled dataset participate in the training, and the training ends when the specified number of training iterations is reached. S4, activity recognition model training phase: the activity recognition model is trained using the labeled dataset and the multimodal activity data generated by the trained diffusion model based on the labeled dataset, and the trained activity recognition model is used for activity recognition.

[0046] The specific process of training an activity recognition model using a labeled dataset is as follows: S4-1, select a batch of labeled data samples from the labeled data set, and for each labeled data sample in the batch , repeat steps S4-2 to S4-4.

[0047] S4-2, fix the self-supervised model, use the trained self-supervised model to extract multimodal representation and global representation based on labeled data samples; specifically, Input into the trained self-supervised model and extract multimodal representation through forward reasoning and global representation , and averaged along a fixed length to obtain a multimodal representation used as a guide and global representation .

[0048] S4-3, using multimodal representation and global representation as guiding conditions, using diffusion model to generate multimodal activity data; specifically, using the trained diffusion model to perform multi-step denoising and denoising on the initial random noise, and introducing multimodal representation in the denoising process and global representation As a guide to generate multimodal activity data In each time step, the denoising network is used to denoise the data and then denoise it again, which is then used as the input for the next time step. This cycle is repeated over multiple time steps to generate multimodal activity data. .

[0049] S4-4, using real data and generated data to train the activity recognition model, specifically the labeled data samples selected from the labeled dataset and the generated modal activity data Input to the activity recognition model to get the classification result and , for all labeled data samples in the current batch, based on the classification results and With the true label Constructing classification loss , using classification loss Train the activity recognition model to update the activity recognition model parameters, using the classification loss Minimization is the goal, where the classification loss uses the cross entropy loss, expressed as: The classification loss uses cross entropy loss, which is expressed as: ; in, represents the cross entropy loss, Represents labeled data samples From the distribution of the labeled dataset , The noise of the noise adding process comes from a mean of 0 and a variance of The normal distribution of .

[0050] S4-5, when the number of training iterations has not been reached, that is, when the training is not completed, steps S4-1 to S4-4 are repeated until all batches of the unlabeled dataset participate in the training, and the training ends when the specified number of training iterations is reached.

[0051] The activity recognition model after training is used for activity recognition. When recognizing a specific activity, the multimodal activity data to be recognized is input into the activity recognition model, and the classification result of the recognition is obtained through calculation.

[0052] like Figure 6As shown, the embodiment also provides an activity recognition device 60 based on a representation-guided multimodal diffusion model, including: a data preprocessing unit 61, a self-supervised model training unit 62, a diffusion model training unit 63, and an activity recognition model training unit 64, wherein the data preprocessing unit 61 is used to obtain the original multimodal activity data and construct an unlabeled data set and a labeled data set after preprocessing; the self-supervised model training unit 62 is used to train a multimodal self-supervised model with an unlabeled data set so that it can extract multimodal representations containing complex patterns and global representations containing structural information from the data; the diffusion model training unit 63 is used to introduce a modal layer guided by a multimodal representation and a global layer guided by a global representation into the diffusion model, and use the unlabeled data set to train the diffusion model so that it generates multimodal activity data; the activity recognition model training unit 64 is used to train an activity recognition model with a labeled data set and multimodal activity data generated by a trained diffusion model based on the labeled data set, and the trained activity recognition model is used for activity recognition.

[0053] It should be noted that the activity identification device based on the representation-guided multimodal diffusion model provided in the above embodiment should be illustrated by the division of the above functional modules when performing activity identification. The above functional distribution can be completed by different functional units or modules as needed, that is, the internal structure of the terminal or server is divided into different functional units or modules to complete all or part of the functions described above. In addition, the activity identification device based on the representation-guided multimodal diffusion model provided in the above embodiment and the activity identification method embodiment based on the representation-guided multimodal diffusion model belong to the same concept. The specific implementation process is detailed in the activity identification method embodiment based on the representation-guided multimodal diffusion model, which will not be repeated here.

[0054] In the above-mentioned activity recognition method and device, the modal layer guided by multimodal representation and the global layer guided by global representation are introduced into the diffusion model, which can use a large amount of unlabeled data to perform data enhancement on a small amount of labeled data on complex multimodal data, generate complex multimodal data with modal details and consistent activity information, solve the overfitting problem of the activity recognition model, and improve the accuracy of activity recognition. It has broad application prospects in the fields of medical care, sports monitoring, etc.

[0055] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An activity recognition method based on representation-guided multimodal diffusion model, It is characterized in that The following steps are involved: Data preprocessing stage: Obtain the original multimodal activity data and construct unlabeled and labeled datasets after preprocessing: Self-supervised model training phase: Use unlabeled datasets to train a multimodal self-supervised model to extract multimodal representations containing complex patterns and global representations containing structural information from the data; Diffusion model training phase: introduce a modal layer guided by multimodal representation and a global layer guided by global representation into the diffusion model, and use an unlabeled dataset to train the diffusion model to generate multimodal activity data; Activity recognition model training phase: The activity recognition model is trained using the labeled dataset and the multimodal activity data generated by the trained diffusion model based on the labeled dataset, and the trained activity recognition model is used for activity recognition.

2. The activity recognition method based on the representation-guided multimodal diffusion model according to claim 1, It is characterized in that The multimodal self-supervised model is trained using an unlabeled dataset to extract multimodal representations containing complex patterns and global representations containing structural information from the data, including: Build a self-supervised model with multiple independent modality encoders, attention networks, and multiple independent modality decoders; Unlabeled data samples are selected from the unlabeled data set and randomly masked to obtain masked data, and multiple independent modal encoders are used to encode each modality in the masked data to obtain a multimodal representation containing complex patterns; After concatenating the multimodal representations, the attention network is used to obtain a global representation containing structural information, and multiple independent modality decoders are used to decode each modality in the global representation to obtain reconstructed data. A mask reconstruction loss is constructed based on the reconstructed data and unlabeled data samples, and the mask reconstruction loss is used to train the self-supervised model to update the self-supervised model parameters.

3. The activity recognition method based on the representation-guided multimodal diffusion model according to claim 2, It is characterized in that The mask reconstruction loss uses mean square error, expressed as: ; in, represents the mask reconstruction loss, and represent unlabeled data samples and reconstructed data respectively, Represents unlabeled data samples From the distribution of the unlabeled dataset , Express expectations, Represents the square of the L2 norm.

4. The activity recognition method based on the representation-guided multimodal diffusion model according to claim 1, It is characterized in that A modal layer guided by multimodal representation and a global layer guided by global representation are introduced into the diffusion model, and an unlabeled dataset is used to train the diffusion model to generate multimodal activity data, including: (1) Multiple modal layers and global layers are introduced into the denoising network of the diffusion model, wherein each modal layer and global layer includes multiple continuous downsampling modules and multiple upsampling modules, and the global layer also includes a fusion module. The number of downsampling modules and upsampling modules in each modal layer is equal, and a jump connection is formed between the downsampling modules and the corresponding upsampling modules; the number of downsampling modules and upsampling modules in the global layer is also equal, the output of the last downsampling module is directly connected to the input of the corresponding upsampling module, and the remaining downsampling modules are connected to the corresponding upsampling modules. Jump connections are formed; (2) Selecting unlabeled data samples from the unlabeled dataset, and using the trained self-supervised model to extract multimodal representations and global representations based on the unlabeled data samples, and averaging the multimodal representations and global representations along a fixed length to obtain the multimodal representations and global representations used as guidance; (3) For each time step, the following process is performed: (3-1) Noising the input data to obtain noisy data, and using two encoding networks to encode the time steps to obtain the first time step encoding and the second time step encoding; (3-2) Using the noisy data as the input of the denoising network, each modal layer processes the noisy data corresponding to one modality, using multiple downsampling modules in each modal layer, and using the first time step encoding and the modal representation corresponding to each modality extracted from the multimodal representation as the guiding condition, extracting the modal features of each modality from the input noisy data; (3-3) Using the fusion module of the global layer to fuse the modal features of all modalities to obtain fusion features, using multiple downsampling modules and upsampling modules in the global layer, and taking the second time step encoding and global representation as guiding conditions, extracting global features from the input fusion features; (3-4) utilizing multiple upsampling modules in each modality layer, and taking the first time step encoding and the modal representation corresponding to each modality extracted from the multimodal representation as guiding conditions, extracting the denoised data of each modality from the input global features, and the denoised data of all modalities constitute the generated multimodal activity data; (4) A reconstruction loss is constructed based on the unlabeled data samples and the generated multimodal activity data, and the reconstruction loss is used to train the diffusion model to update the denoising network parameters in the diffusion model.

5. The activity recognition method based on the representation-guided multimodal diffusion model according to claim 4, It is characterized in that The same processing process adopted by each downsampling module is: calculating a first scaling parameter and a first translation parameter based on the input guidance condition, and then scaling and translating the input data or input features of each downsampling module using the first scaling parameter and the first translation parameter, and then obtaining the output features after convolution and activation processing; For each modal layer, the input data of the first downsampling module is the noisy data, the input features of other downsampling modules are the output features of the previous downsampling module of each modal layer, and the output of the last downsampling module is the modal feature; for the global layer, the input features of the first downsampling module are the fusion features, and the input features of other downsampling modules are the output features of the previous downsampling module in the global layer; The same processing process adopted by each upsampling module is: calculating the second scaling parameter and the second translation parameter based on the input guidance condition, and then scaling and translating the input features of each upsampling module using the second scaling parameter and the second translation parameter, and then obtaining the output features after deconvolution and activation processing; For the global layer, the input features of the first upsampling module are the output features of the last downsampling module in the global layer, the input features of other upsampling modules are the output features of the previous upsampling module in the global layer and the output features of the downsampling modules in the same layer through jump connections, and the output features of the last upsampling module are the global features; for each modal layer, the input features of the first upsampling module are the global features output by the last upsampling module in the global layer and the output features of the downsampling modules in the same layer through jump connections, the input features of other upsampling modules are the output features of the previous upsampling modules in the modal layer and the output features of the downsampling modules in the same layer through jump connections, and the output of the last upsampling module is the denoised data.

6. The activity recognition method based on the representation-guided multimodal diffusion model according to claim 4, It is characterized in that The fusion module of the global layer is used to fuse the modal features of all modalities to obtain fusion features, including: The modal features of all modalities are spliced ​​to obtain the spliced ​​features, and the spliced ​​features are input into the multi-layer convolutional layer to obtain the modal weight matrix. At the same time, the spliced ​​matrix is ​​input into two convolutional layers to obtain the first fusion parameter and the second fusion parameter. The modal weight matrix is ​​processed with the first fusion parameter to obtain the modal weight, and the modal weight is processed with the second fusion parameter to obtain the fusion feature.

7. The activity recognition method based on the representation-guided multimodal diffusion model according to claim 4, It is characterized in that The reconstruction loss uses mean square error, expressed as: ; in, represents the reconstruction loss, and represent unlabeled data samples and generated multimodal activity data, respectively. Represents unlabeled data samples From the distribution of the unlabeled dataset , Indicates the noise in the noise adding process comes from a mean of 0 and a variance of The normal distribution of represents the time step, Represents the square of the L2 norm.

8. The activity recognition method based on the representation-guided multimodal diffusion model according to claim 1, It is characterized in that The activity recognition model is trained using the labeled dataset and the multimodal activity data generated by the trained diffusion model based on the labeled dataset, including: Selecting labeled data samples from the labeled data set, using the trained self-supervised model to generate multimodal representations and global representations based on the labeled data samples, and taking the average along a fixed length to obtain the multimodal representations and global representations used as guidance; The trained diffusion model is used to perform multi-step denoising and denoising on the initial random noise, and multimodal representation and global representation are introduced as guiding conditions in the denoising process to generate multimodal activity data. The labeled data samples selected from the labeled dataset and the generated modal activity data are input into the activity recognition model to obtain the classification results, and the classification loss is constructed based on the classification results and the true labels. The activity recognition model is trained using the classification loss to update the activity recognition model parameters.

9. The activity recognition method based on the representation-guided multimodal diffusion model according to claim 8, It is characterized in that The classification loss uses cross entropy loss, which is expressed as: ; in, represents the classification loss, represents the true label, Indicates the classification results corresponding to the labeled data samples, Represents the classification result corresponding to the generated modal activity data, represents the cross entropy loss, Express expectations, Represents labeled data samples From the distribution of the labeled dataset , The noise of the noise adding process comes from a mean of 0 and a variance of The normal distribution of .

10. An activity recognition device based on a representation-guided multimodal diffusion model, It is characterized in that include: The data preprocessing unit is used to obtain the original multimodal activity data and construct unlabeled and labeled data sets after preprocessing: A self-supervised model training unit, which is used to train a multimodal self-supervised model using an unlabeled data set so that it can extract a multimodal representation containing complex patterns and a global representation containing structural information from the data; A diffusion model training unit, which is used to introduce a modal layer guided by multimodal representation and a global layer guided by global representation into the diffusion model, and train the diffusion model with an unlabeled data set to generate multimodal activity data; The activity recognition model training unit is used to train the activity recognition model using the labeled data set and the multimodal activity data generated by the trained diffusion model based on the labeled data set, and the trained activity recognition model is used for activity recognition.

Citation Information

Patent Citations

  • Multi-modal human activity recognition method based on generative adversarial network

    CN110309861A

  • SAR (Synthetic Aperture Radar) recognition database sample attitude expansion method based on depth condition diffusion network

    CN116258874A

  • Self-supervised contrast learning method for human body action recognition based on diffusion model

    CN116415152A

  • Multimodal bacterial colony sample classification and identification method and system based on diffusion model

    CN117292195A

  • Semi-supervised medical image segmentation method based on data enhancement strategy

    CN117710681A