Multi-modal continuous behavior recognition method based on time sequence semantic alignment

Abstract

The invention discloses a multi-modal continuous behavior recognition method based on time sequence semantic alignment. The method comprises the steps that a time sequence semantic alignment module is inserted into a pre-trained frozen ViT-B / 16 model; s2, a new task model is obtained after multiple epochs are trained; s3, calculating a prototype and a variance of each class of the new task to form Gaussian distribution of each class of the new task; s4, obtaining a new class feature set; s5, forming a corresponding new class feature set; s6, acquiring a confrontation feature set; s7, sampling features from Gaussian distribution of new and old tasks to form a new and old sampling feature set, sending the new and old sampling feature set and the adversarial feature set into a classifier of a new task model, and calculating classification loss; s8, calculating disturbance loss, and optimizing the adversarial feature extraction module and a classifier of the new task model together with classification loss; and step S9, carrying out multi-mode continuous behavior identification. According to the invention, new and old features can be synthesized to extract adversarial features of new and old classification boundaries, and multi-modal continuous behavior recognition is realized.

Description

Technical Field

[0001] This invention relates to the field of video image processing, and in particular to a multimodal continuous behavior recognition method based on temporal semantic alignment that combines adversarial feature extraction. Background Technology

[0002] With the widespread adoption of wearable devices, multimodal sensor data is growing exponentially, driving the application of multimodal continuous behavior recognition technology in fields such as healthcare, industrial deployment, and virtual reality. Multimodal continuous behavior recognition requires models to continuously learn new knowledge from multimodal data while retaining old knowledge. However, due to legal restrictions and privacy concerns, the fine-tuning phase for new tasks cannot obtain multimodal data for older categories, leading to a catastrophic forgetting phenomenon in the model, i.e., a significant decline in performance in recognizing older categories.

[0003] Existing multimodal continuous behavior recognition methods primarily rely on Gaussian distribution modeling of old category knowledge, using it as a substitute for old category data for model training. However, real-world multimodal data exhibits significant heterogeneity and complexity, with some old category sample features not conforming to the Gaussian distribution assumption. This leads to incorrect decision boundary delineation between old and new categories, limiting model recognition accuracy. Furthermore, existing methods fail to align temporal information during multimodal feature modeling, resulting in insufficient feature discriminative power and parameter redundancy. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal continuous behavior recognition method based on temporal semantic alignment. The method supervises the temporal semantic orthogonality of multimodal behavior through a temporal semantic alignment module and can extract adversarial features of the new and old classification boundaries by integrating new and old features, thereby realizing multimodal continuous behavior recognition.

[0005] The objective of this invention is achieved through the following technical solution: a multimodal continuous behavior recognition method based on temporal semantic alignment, comprising the following steps:

[0006] Step S1: Insert a temporal semantic alignment module into the pre-trained frozen ViT-B / 16 model;

[0007] Step S2: Obtain temporal semantic alignment multimodal features, calculate classification loss, and calculate temporal semantic orthogonal loss based on temporal semantic alignment multimodal features. Optimize the new task model together with the classification loss, and obtain the new task model after training for multiple epochs.

[0008] Step S3: Calculate the prototype and variance of each class of the new task using the new task model, and form a Gaussian distribution for each class of the new task;

[0009] Step S4: Sample for each old task category from the Gaussian distribution of all old tasks to obtain the old class sampled feature set, and input all multimodal data of the new task into the feature extractor of the new task model to obtain the new class feature set;

[0010] Step S5: For each feature in the old class feature set, find the nearest new class feature in the new class feature set. These nearest new class features form the corresponding new class feature set.

[0011] Step S6: Input the old class sampling feature set and the corresponding new class feature set into the adversarial feature extraction module to obtain the adversarial feature set;

[0012] Step S7: Sample features from the Gaussian distributions of the new and old tasks to form a set of new and old sampled features. Feed this set together with the set of adversarial features into the classifier of the new task model and calculate the classification loss.

[0013] Step S8: Calculate the perturbation loss based on the adversarial feature set and the old class sampled feature set, and optimize the classifier of the adversarial feature extraction module and the new task model together with the classification loss;

[0014] Step S9: Input the multimodal data to be identified into the frozen ViT-B / 16 model containing the temporal semantic alignment module to extract features, and then input the features into the optimized classifier to realize multimodal continuous behavior recognition.

[0015] The beneficial effects of this invention are as follows: First, a temporal semantic alignment module is inserted into the frozen ViT of each modality. Then, the features of each modality are concatenated together, and a temporal semantic orthogonal loss is calculated on the concatenated multimodal features to align the temporal semantics of the multimodal features. Finally, the sampled features of the old category are sampled from the Gaussian distribution of the old category and fed together with the features of the new category into the adversarial feature extraction module to obtain adversarial features that will be classified as the old category, thereby improving the classification boundary between the old and new categories. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the principle of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0018] like Figure 1As shown, this invention aims to construct a temporal semantic alignment module under inter-frame orthogonal supervision to align the temporal semantics of multimodal features. An adversarial feature extraction module extracts adversarial features that will be classified as belonging to the old category from the sampled features of the old category and the features of the new category. First, the frozen ViT of each modality is inserted into the temporal semantic alignment module. Then, the features of each modality are concatenated, and a temporal semantic orthogonal loss is calculated on the concatenated multimodal features to align the temporal semantics of the multimodal features. Finally, sampled features of the old category are sampled from the Gaussian distribution of the old category and fed together with the features of the new category into the adversarial feature extraction module to obtain adversarial features that will be classified as belonging to the old category, thereby improving the classification boundary between the old and new categories.

[0019] This invention is implemented on a GPU server experimental platform and mainly includes the following steps: A temporal semantic alignment module is inserted into a pre-trained frozen ViT-B / 16 model to train a new task model; then, an adversarial feature extraction module is used to further train the classifier of the new task model, including the following steps:

[0020] A multimodal continuous behavior recognition method based on temporal semantic alignment includes the following steps:

[0021] Step S1: Insert a temporal semantic alignment module into the pre-trained frozen ViT-B / 16 model;

[0022] In each Transformer block of the pre-trained frozen ViT-B / 16 model, a temporal semantic alignment module is inserted for the video, gyroscope, and accelerometer modalities respectively. In the Temporal semantic alignment module in a Transformer block It can be represented as:

[0023]

[0024] in, For the first Modalities of MLP layers in a Transformer block Input, It is a scale parameter. Represents ReLU and These represent the downsampling layer and the upsampling layer, respectively.

[0025] In the embodiments of this application, a ViT-B / 16 model that has been trained on the ImageNet dataset (i.e., pre-trained) is used. This is a common existing method. During the training process of this invention, the parameters of the pre-trained ViT-B / 16 model will not change (i.e., are frozen).

[0026] Based on this, only the parameters of the temporal semantic alignment module, the classifier, and the adversarial feature extraction module will change. The training process involves first feeding the multimodal data of the new task into a frozen pre-trained ViT-B / 16 model with the temporal semantic alignment module inserted to extract features, and then feeding the features into the classifier for behavior recognition. During this process, the temporal semantic alignment module and the classifier participate in the training. After the training of the new task data is completed, the parameters of the temporal semantic alignment module are also frozen. The adversarial feature extraction module and the classifier are trained using the new task data and the sampled features of both new and old tasks, with a focus on optimizing the classifier.

[0027] Step S2: Obtain temporal semantic alignment multimodal features, calculate classification loss, and calculate temporal semantic orthogonal loss based on temporal semantic alignment multimodal features. Optimize the new task model together with the classification loss, and obtain the new task model after training for multiple epochs.

[0028] A batch of multimodal data was sampled from the training set and fed into a pre-trained frozen ViT-B / 16 model with a temporal semantic alignment module inserted. Video features, accelerometer features, and gyroscope features were extracted and concatenated along the feature dimension to obtain multimodal features. Multimodal features The data is fed into a classifier, and the cross-entropy loss is calculated based on the classification results. ;

[0029] The training set contains multiple samples, each of which is a multimodal dataset, including video frame sequences, accelerometer data, and gyroscope data.

[0030] In the embodiments of this application, the classifier is a linear layer, the input dimension is the feature dimension, and the output dimension is the number of categories; the classifier outputs the labels corresponding to the features, and combined with the true label distribution of the features, the cross-entropy loss is calculated. .

[0031] For multimodal features Calculate temporal semantic orthogonal loss , and cross-entropy loss The loss of the Temporal Semantic Alignment Module (TSA) is composed of these components. :

[0032]

[0033] in, These are the weights of the temporal semantic orthogonal loss. It can be represented as:

[0034]

[0035] in, Multimodal features normalized to L2 , It is the identity matrix. The number of sampling frames, It is the Frobenius norm;

[0036] Repeat the above process to train for multiple epochs, traversing the entire training set in each epoch to obtain the task. Behavior recognition model;

[0037] In the embodiments of this application, the behavior recognition model for each task consists of two parts: a frozen pre-trained ViT-B / 16 model with a temporal semantic alignment module (TSA) inserted and a classifier. Multimodal data is first processed by the frozen pre-trained ViT-B / 16 model with a TSA inserted to extract features, and then the features are fed into the classifier for behavior recognition. This training process is supervised by the loss of the TSA module.

[0038] Step S3: Calculate the prototype and variance of each class of the new task using the new task model, and form a Gaussian distribution for each class of the new task;

[0039] Through the task Action recognition model extracts task The category prototype for each class and variance , forming a task Gaussian distribution for each class, where For the task To the mission Number of categories For the task To the mission The number of categories;

[0040] Step S4: Sample for each old task category from the Gaussian distribution of all old tasks to obtain the old class sampled feature set, and input all multimodal data of the new task into the feature extractor of the new task model to obtain the new class feature set;

[0041] From the task To the mission For each task, a Gaussian distribution is used. To the mission Category sampling Gejiu Class Sampling Feature Composition and all tasks Feature input task The feature extractor of the behavior recognition model yields a new set of features. ;

[0042] Step S5: For each feature in the old class feature set, find the nearest new class feature in the new class feature set. These nearest new class features form the corresponding new class feature set.

[0043] for Each old class feature sampled is used to calculate the L2 distance in the new class feature set. Find the most recent new class feature, corresponding Each new class feature constitutes a corresponding new class feature set. ;

[0044] Step S6: Input the old class sampling feature set and the corresponding new class feature set into the adversarial feature extraction module (this module is used to optimize the classifier) ​​to obtain the adversarial feature set;

[0045] Will and Simultaneously, the data is fed into the AFE (Adversarial Feature Extraction Module) to obtain the adversarial feature set. Its tags and Consistency:

[0046]

[0047] in, Represents multiple points of cross-attention. and These represent the new type of encoder and the old type of encoder, respectively. Represents the decoder;

[0048] Step S7: Sample features from the Gaussian distributions of the new and old tasks to form a set of new and old sampled features. Feed this set together with the set of adversarial features into the classifier of the new task model and calculate the classification loss.

[0049] From the task To the mission In a Gaussian distribution, samples are taken from each class. Features, whose set is , and adversarial feature set Send them together into the mission In the classifier of the behavior recognition model, the labels corresponding to the features output by the classifier are used to calculate the cross-entropy loss against the true label distribution of the features. , as classification loss;

[0050] Step S8: Calculate the perturbation loss based on the adversarial feature set and the old class sampled feature set, and optimize the classifier of the adversarial feature extraction module and the new task model together with the classification loss.

[0051] Calculate disturbance loss , and classification loss The loss of the adversarial feature extraction module (AFE) is composed of these components. Optimize the Adversarial Feature Extraction (AFE) module and the task Classifier for behavior recognition models:

[0052]

[0053] in, It is the weight of the disturbance loss, the disturbance loss. It can be represented as:

[0054]

[0055] in, This is the mean squared error loss;

[0056] Step S9: Input the multimodal data to be identified into the frozen ViT-B / 16 model containing the temporal semantic alignment module to extract features, and then input the features into the optimized classifier to realize multimodal continuous behavior recognition.

[0057] The foregoing description illustrates and describes a preferred embodiment of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A multimodal continuous behavior recognition method based on temporal semantic alignment, characterized in that: Includes the following steps: Step S1: Insert a temporal semantic alignment module into the pre-trained frozen ViT-B / 16 model; Step S2: Obtain temporal semantic alignment multimodal features, calculate classification loss, and calculate temporal semantic orthogonal loss based on temporal semantic alignment multimodal features. Optimize the new task model together with the classification loss, and obtain the new task model after training for multiple epochs. Step S3: Calculate the prototype and variance of each class of the new task using the new task model, and form a Gaussian distribution for each class of the new task; Step S4: Sample for each old task category from the Gaussian distribution of all old tasks to obtain the old class sampled feature set, and input all multimodal data of the new task into the feature extractor of the new task model to obtain the new class feature set; Step S5: For each feature in the old class feature set, find the nearest new class feature in the new class feature set. These nearest new class features form the corresponding new class feature set. Step S6: Input the old class sampling feature set and the corresponding new class feature set into the adversarial feature extraction module to obtain the adversarial feature set; Step S7: Sample features from the Gaussian distributions of the new and old tasks to form a set of new and old sampled features. Feed this set together with the set of adversarial features into the classifier of the new task model and calculate the classification loss. Step S8: Calculate the perturbation loss based on the adversarial feature set and the old class sampled feature set, and optimize the classifier of the adversarial feature extraction module and the new task model together with the classification loss; Step S9: Input the multimodal data to be identified into the frozen ViT-B / 16 model containing the temporal semantic alignment module to extract features, and then input the features into the optimized classifier to realize multimodal continuous behavior recognition.

2. The multimodal continuous behavior recognition method based on temporal semantic alignment according to claim 1, characterized in that: Step S1 includes: In each Transformer block of the pre-trained frozen ViT-B / 16 model, a temporal semantic alignment module is inserted for the video, gyroscope, and accelerometer modalities respectively. In the Temporal semantic alignment module in a Transformer block Represented as: ； in, For the first Modalities of MLP layers in a Transformer block Input, It is a scale parameter. Represents ReLU and These represent the downsampling layer and the upsampling layer, respectively.

3. The multimodal continuous behavior recognition method based on temporal semantic alignment according to claim 1, characterized in that: Step S2 includes: S201. Sample a batch of multimodal data from the training set and feed it into a pre-trained frozen ViT-B / 16 model with a temporal semantic alignment module inserted. Extract video features, accelerometer features, and gyroscope features, and concatenate them along the feature dimension to obtain multimodal features. Multimodal features The data is fed into a classifier, and the cross-entropy loss is calculated based on the classification results. , as classification loss; The training set contains multiple samples, each of which is a multimodal dataset, including video frame sequences, accelerometer data, and gyroscope data. S202. Multimodal features Calculate temporal semantic orthogonal loss , and cross-entropy loss The loss of the Temporal Semantic Alignment Module (TSA) is composed of these components. : ； in, These are the weights of the temporal semantic orthogonal loss. Represented as: ； in, Multimodal features normalized to L2 , It is the identity matrix. The number of sampling frames, It is the Frobenius norm; S203. Repeat steps S201-S202 to train for multiple epochs, traversing the entire training set in each epoch to obtain the task. The behavior recognition model, i.e., the new task model; The behavior recognition model for each task consists of two parts: a frozen pre-trained ViT-B / 16 model with a temporal semantic alignment module (TSA) inserted and a classifier. Multimodal data is first processed by the frozen pre-trained ViT-B / 16 model with a TSA inserted to extract features, and then the features are fed into the classifier for behavior recognition. This training process is supervised by the loss of the TSA module.

4. The multimodal continuous behavior recognition method based on temporal semantic alignment according to claim 1, characterized in that: Step S3 includes: Through the task Action recognition model extracts task The category prototype for each class and variance , forming a task Gaussian distribution for each class, where For the task To the mission Number of categories For the task To the mission The number of categories.

5. The multimodal continuous behavior recognition method based on temporal semantic alignment according to claim 4, characterized in that: Step S4 includes: From the task To the mission For each task, a Gaussian distribution is used. To the mission Category sampling Gejiu Class Sampling Feature Composition and all tasks Feature input task The feature extractor of the behavior recognition model yields a new set of features. .

6. The multimodal continuous behavior recognition method based on temporal semantic alignment according to claim 5, characterized in that: Step S5 includes: for Each old class feature sampled is used to calculate the L2 distance in the new class feature set. Find the most recent new class feature, corresponding Each new class feature constitutes a corresponding new class feature set. .

7. The multimodal continuous behavior recognition method based on temporal semantic alignment according to claim 6, characterized in that: Step S6 includes: Will and Simultaneously, the data is fed into the adversarial feature extraction module to obtain the adversarial feature set. Its tags and Consistency: ； Where, AFE represents the adversarial feature extraction module. Represents multiple points of cross-attention. and These represent the new type of encoder and the old type of encoder, respectively. Represents the decoder.

8. The multimodal continuous behavior recognition method based on temporal semantic alignment according to claim 7, characterized in that: Step S7 includes: from the task To the mission In a Gaussian distribution, samples are taken from each class. Features, whose set is , and adversarial feature set Send them together into the mission In the classifier of the behavior recognition model, the labels corresponding to the features output by the classifier are used to calculate the cross-entropy loss against the true label distribution of the features. As a classification loss.

9. The multimodal continuous behavior recognition method based on temporal semantic alignment according to claim 8, characterized in that: Step S8 includes: Calculate disturbance loss , and classification loss Together they form the loss of the Adversarial Feature Extraction (AFE) module. Optimize the Adversarial Feature Extraction (AFE) module and the task Classifier for behavior recognition models: ； in, It is the weight of the disturbance loss, the disturbance loss. It can be represented as: ； in, This represents the mean squared error loss.

Images