A first-view incremental behavior recognition method based on a multi-modal adapter

Abstract

The application discloses a first-view incremental behavior recognition method based on a multi-modal adapter, and relates to the field of video image processing.The application aims to transfer the time sequence perception ability in the old task recognition model to the current task recognition model through a time sequence distillation loss, and realizes the incremental recognition of the classification layer to multiple tasks by combining a modal balance adapter, feature Gaussian sampling of the old task recognition model and feature Gaussian sampling of the current task recognition model.First, the multi-modal time sequence perception adapter is inserted in the form of a residual before and after the forward network layer of all the Transformer encoders of the Vit-B / 16 model pre-trained on the frozen ImageNet dataset, so as to realize the extraction of time sequence information from the features of the visual modal, acceleration modal and gyroscope modal in the Transformer encoder, and ensure that the time sequence information of the current task recognition model and the old task recognition model are as close as possible through the time sequence distillation loss.Finally, the feature Gaussian distribution of the current task is constructed by calculating the feature mean and variance of the current task recognition model, and the feature Gaussian sampling of multiple tasks is realized in combination with the feature Gaussian distribution of the old task recognition model, the modal balance adapter is inserted in front of the classification layer of the current task, and the classification layer of the current task is trained again by using the sampling features and the modal balance adapter, so as to improve the recognition ability of the classification layer to each task.

Description

Technical Field

[0001] This invention relates to the field of video image processing, and in particular to a first-view incremental action recognition method based on a multi-modal adapter in action recognition tasks. Background Technology

[0002] With the development of wearable devices, more and more researchers are focusing on enabling intelligent systems to understand and perceive user behavior from a first-person perspective using multimodal data provided by wearable devices, thereby achieving human-computer interaction. Due to the complexity and variability of real-world usage scenarios, deep learning models are required to learn the current recognition task while maintaining high accuracy on previous tasks. However, due to user privacy concerns, training data from previous recognition tasks cannot be obtained when training the model for the current task. In such scenarios, deep learning models directly learning from the current task's training data will lead to forgetting knowledge from previous recognition tasks. To improve the accuracy of first-person incremental behavior recognition in the absence of multimodal data from previous recognition tasks, this invention incrementally learns the temporal perception patterns in different scenarios, starting from the technical route of a multimodal adapter to incrementally learn the visual attention mechanisms of multiple task scenarios, compared to existing methods.

[0003] Traditional incremental learning methods based on pre-trained models freeze the pre-trained model during training and learn multi-task knowledge by continuously expanding the classification layer parameters. However, this learning method cannot prevent the network from forgetting old task knowledge. Existing incremental learning methods based on pre-trained models alleviate the forgetting problem by introducing additional parameters, and can be mainly divided into two categories: the first type guides multi-task recognition through task-shared prompts; the second type learns task plasticity through an adapter and then uses the Gaussian distribution of multi-task features to learn task stability. The first type of method does not learn the contextual characteristics of the task; the second type ignores the learning and transfer of temporal-aware commonalities of multimodal data, and the sampled multimodal data suffers from modality imbalance. Summary of the Invention

[0004] To facilitate the transfer of scene knowledge from old recognition tasks, this invention uses temporal distillation loss to constrain the representation of temporal knowledge in the multimodal temporal perception adapters of the old and current tasks to be as close as possible. To achieve modal balance of sampled features, a modal balancing adapter is designed to dynamically adjust the gradients of each modal adapter.

[0005] Existing research indicates that introducing supervision from old task models when learning temporal information of the current task can improve the model's ability to remember old task scenarios, and balanced multimodal data helps the model recognize intermodal relationships. To improve the model's incremental learning of multiple tasks without stored data, this invention proposes a multimodal temporal-aware adapter network and a modal balancing adapter network. Therefore, the technical solution of this invention is: a first-person perspective incremental behavior recognition method based on a multimodal adapter, the method comprising:

[0006] Step 1: Based on the pre-trained Vit-B / 16 model on the ImageNet dataset, the Vit-B / 16 model includes: 1 Patch Embedding layer, 12 Transformer encoders and 1 classification layer, where the Transformer encoder consists of a multi-head self-attention layer MHSA and a feedforward network layer MLP.

[0007] First, freeze the Patch Embedding layer and all Transformer encoders of Vit-B / 16. Then, insert multimodal temporal-aware adapters before and after the feedforward network layers of all Transformer encoders in the frozen pre-trained Vit-B / 16 model with residual connections. Input the old task dataset into the Vit-B / 16 model with the multimodal temporal-aware adapters inserted. The multimodal temporal-aware adapters consist of 3 downsampling modules, 1 multimodal temporal fusion module F(·), and 3 upsampling modules.

[0008] The three downsampling modules are respectively and The three upsampling modules are respectively and For the visual feature v fed into the i-th multimodal temporal sensing adapter i Acceleration characteristics a i and gyroscope characteristics g i The samples are fed into a downsampling module composed of linear layers. and Obtain downsampling features and

[0009] Step 2: Separate the downsampled features of each modality to obtain the CLS token features of each modality. and Patch features The CLS token features are then concatenated and fed into the multimodal temporal fusion module F(·) to obtain the temporal features f. i =F(CLS) i Then, all CLS token features are combined with the temporal feature f.i The features are then concatenated with the Patch features and fed into a ReLU activation layer σ(·) for non-linear mapping; the mapped features are then fed into an upsampling module consisting of linear layers. and Obtain temporally enhanced visual features v i ′ Acceleration characteristics a i ′ and gyroscope characteristics g i ′ ;

[0010] Then, the temporally enhanced visual features v i ′ Acceleration characteristics a i ′ and gyroscope characteristics g i ′ Add it to the output of the forward network layer of the Transformer encoder;

[0011] Step 3: On the old task data, train the Vit-B / 16 model with the multimodal temporal awareness adapter inserted. The trained Vit-B / 16 model with the multimodal temporal awareness adapter inserted is denoted as the old task recognition model S. The old task has c categories. Calculate the feature mean μ of each category in the old task. o ={μ1,…,μ c} and variance ∑ o ={∑1,…,∑ c Construct the Gaussian distribution of features for the old task.

[0012] Step 4: The number of categories in the current task is dc. The old task classification layer, which previously classified c categories, is incrementally expanded to classify d categories in the current task. The current task dataset is fed into the old task recognition model S to obtain the old model temporal features f of the i-th Transformer encoder's multimodal temporal-aware adapter. i o And the current model temporal features f of the multimodal temporal awareness adapter of the i-th Transformer encoder of the current recognition model T. i n Calculate time-series distillation losses

[0013] The total loss of the current recognition model T is:

[0014]

[0015] in, Let T be the cross-entropy classification loss of the current recognition model on the current task dataset;

[0016] Step 5: Calculate the feature mean μ of each category for the current task using the trained current task recognition model T. n ={μ c+1 ,…,μ d} and variance ∑ n ={∑ c+1 ,…,∑ d Construct the Gaussian distribution of features for the current task. From the characteristic Gaussian distribution of the old task and the characteristic Gaussian distribution of the current task The process involves sampling feature samples of various types to form a feature set. Where the generated feature set of class j N is the number of samples;

[0017] Step 6: Insert three modality balancer adapters before the classification layer of the current task: a visual modality balancer adapter, an acceleration modality balancer adapter, and a gyroscope modality balancer adapter. Each modality balancer adapter consists of a downsampling module, a nonlinear mapping layer, and an upsampling module. For the t-th batch B... t Data First, break it down into visual generation features. Acceleration generation features and gyroscope generation features Then, they are sent to the corresponding modal adapters:

[0018]

[0019] Among them, U v U a and U g These represent the upsampling modules for the visual, accelerometer, and gyroscope modes, respectively. σ(·) represents the ReLU layer, a nonlinear mapping layer. v D a and D g These represent downsampling modules for the three modalities of vision, acceleration, and gyroscope, respectively.

[0020] Then, the output of the modality balancing adapter A(·) is added to the generated features of the corresponding modality to obtain the visual features of modality balancing. acceleration characteristics and gyroscope characteristics After being concatenated, the data is fed into the classification layer of the current task to calculate the cross-entropy classification loss and the gradient.

[0021] Step 7: Decompose the current classification layer weights into visual weights. Acceleration weight and gyroscope weight W tg The sum of the predicted outputs of each modality is calculated by combining the bias b of the classification layer as the contribution of each modality;

[0022] Then, based on the contribution of each mode, the gradient modulation coefficient of each mode is calculated.

[0023]

[0024] Where m represents the mode, This indicates the contribution of the corresponding mode;

[0025] In calculating the gradient modulation coefficient Then, the corresponding coefficients are multiplied onto the gradient of the corresponding modality balancer, and the SGD optimizer is used to update the modality balancer and the classification layer parameters of the current task based on the gradient.

[0026] Step 8: After training the modality balance adapter and the classification layer of the current task for 20 epochs using the generated feature set, replace the classification layer of the current recognition model T with the classification layer of the current task that has been retrained using the generated feature set, so as to achieve incremental recognition of the behavior categories of the current task and the old task.

[0027] Furthermore, in step 2, the temporally enhanced visual features v are obtained. i ′ Acceleration characteristics a i ′ and gyroscope characteristics g i ′ The specific formula is expressed as follows:

[0028]

[0029] Concat(·) is a concatenation operation.

[0030] Furthermore, the time-sequential distillation loss in step 4 The calculation method is as follows:

[0031]

[0032] Where, ‖·‖2 represents the Euclidean norm, and the number of Transformer encoders is 12.

[0033] Furthermore, the method for calculating the contribution of each mode in step 7 is as follows:

[0034]

[0035] Where d is the number of categories up to the current task, b represents the bias of the classification layer, and y represents the category label.

[0036] The first-person incremental behavior recognition method proposed in this invention, based on a multimodal adapter, achieves an average incremental recognition accuracy improvement of 3.41 percentage points on the public dataset UESTC-MMEA-CL without storing old task data, and a final recognition accuracy improvement of 4.1 percentage points. Attached Figure Description

[0037] Figure 1 A schematic diagram of a Vit-B / 16 model with a multimodal timing-aware adapter inserted.

[0038] Figure 2 This is a schematic diagram of a first-view incremental behavior recognition method based on a multimodal adapter. Detailed Implementation

[0039] This invention aims to transfer the common temporal knowledge of multimodal data to the current task model under the supervision of the old task model, and to achieve modal balancing of the sampled features. The classification layer for the current task is then retrained using the balanced sampled features. First, a multimodal temporal awareness adapter learns the temporal knowledge of the old task, establishing a Gaussian distribution of features from the old task. Then, when learning the current task, the temporal knowledge of the old task is distilled into the current model, and a modal balancing adapter is used to modally balance the Gaussian sampled features of the old and current tasks, retraining the classification layer for the current task.

[0040] Step 1: Insert a multimodal temporal awareness adapter into the pre-trained Vit-B / 16 model on the frozen ImageNet dataset. Train the multimodal temporal awareness adapter and the classification layer of the old task using the old task dataset to obtain the old task recognition model. Step 2: Calculate the feature mean and variance of each class in the old task dataset using the old task recognition model.

[0041] Step 3: When training on the current task dataset, synchronously input the data into the old task recognition model, calculate the temporal distillation loss between the temporal information of the old task and the current task, and optimize the current task recognition model together with the classification loss.

[0042] Step 4: Calculate the mean and variance of features for each class in the current task dataset using the current task recognition model;

[0043] Step 5: Insert a modality balancing adapter before the classification layer of the current task, input the sampled features of the old task and the current task into the modality balancing adapter, and calculate the classification loss;

[0044] Step 6: Calculate the contribution of each modality based on the classification layer weights, and then calculate the gradient modulation coefficients of each modality. Multiply these coefficients by the gradient of the corresponding modality balancing adapter, and update the parameters of the modality balancing adapter and the classification layer of the current task. The average incremental recognition accuracy is improved by 3.41 percentage points compared to the state-of-the-art method, and the final recognition accuracy is improved by 4.1 percentage points. Table 1 shows the performance comparison between the method of this invention and other methods in the UESTC-MMEA-CL 4-task × 8-class setting per task. The average recognition accuracy of task k represents the recognition accuracy of the recognition model after training on task k for the categories of task k and previous tasks. The final recognition accuracy represents the average recognition accuracy of the last task, and the average incremental recognition accuracy represents the mean of the average recognition accuracy of all tasks.

[0045] Table 1

[0046] method Final recognition accuracy Average incremental recognition accuracy FeTrIL[1] 82.60 89.73 FOSTER[2] 84.45 91.35 SSIAT[3] 89.97 92.57 Method of the present invention 94.07 95.98

[0047] [1] Petit G, Popescu A, Schindler H, et al. Fetril: Feature translation for exemplar-free class-incremental learning [C] / / Proceedings of the IEEE / CVFwinter conference on applications of computer vision. 2023: 3911-3920.

[0048] [2] For Wang FY, Zhou DW, Ye HJ, et al. Foster: Feature boosting and compression for class-incremental learning [C] / / European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 398-414.

[0049] [3] Tan Y, Zhou Q, Xiang X, et al. Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 23252-23262.

Claims

1. A first-view incremental behavior recognition method based on a multimodal adapter, the method comprising: Step 1: Based on the pre-trained Vit-B / 16 model on the ImageNet dataset, the Vit-B / 16 model includes: 1 Patch Embedding layer, 12 Transformer encoders and 1 classification layer, where the Transformer encoder consists of a multi-head self-attention layer MHSA and a feedforward network layer MLP. First, freeze the Patch Embedding layer and all Transformer encoders of Vit-B / 16. Then, insert multimodal temporal-aware adapters before and after the feedforward network layers of all Transformer encoders in the frozen pre-trained Vit-B / 16 model with residual connections. Input the old task dataset into the Vit-B / 16 model with the multimodal temporal-aware adapters inserted. The multimodal temporal-aware adapters consist of 3 downsampling modules, 1 multimodal temporal fusion module F(·), and 3 upsampling modules. The three downsampling modules are respectively and The three upsampling modules are respectively and For the visual feature v fed into the i-th multimodal temporal sensing adapter i Acceleration characteristics a i and gyroscope characteristics g i The samples are fed into a downsampling module composed of linear layers. and Obtain downsampling features and Step 2: Separate the downsampled features of each modality to obtain the CLS token features of each modality. and Patch features The CLS token features are then concatenated and fed into the multimodal temporal fusion module F(·) to obtain the temporal features f. i =F(CLS) i Then, all CLS token features are combined with the temporal feature f. i The features are then concatenated with the Patch features and fed into a ReLU activation layer σ(·) for non-linear mapping; the mapped features are then fed into an upsampling module consisting of linear layers. and Obtain temporally enhanced visual features v i ′ Acceleration characteristics a i ′ and gyroscope characteristics g i ′ ; Then, the temporally enhanced visual features v i ′ Acceleration characteristics a i ′ and gyroscope characteristics g i ′ Add it to the output of the forward network layer of the Transformer encoder; Step 3: On the old task data, train the Vit-B / 16 model with the multimodal temporal awareness adapter inserted. The trained Vit-B / 16 model with the multimodal temporal awareness adapter inserted is denoted as the old task recognition model S. The old task has c categories. Calculate the feature mean μ of each category in the old task. o ={μ1,…,μ c } and variance ∑ o ={∑1,…,∑ c Construct the Gaussian distribution of features for the old task. Step 4: The number of categories in the current task is dc. The old task classification layer, which previously classified c categories, is incrementally expanded to classify d categories in the current task. The current task dataset is fed into the old task recognition model S to obtain the old model temporal features f of the i-th Transformer encoder's multimodal temporal-aware adapter. i o And the current model temporal features f of the multimodal temporal awareness adapter of the i-th Transformer encoder of the current recognition model T. i n Calculate time-series distillation losses The total loss of the current recognition model T is: in, Let T be the cross-entropy classification loss of the current recognition model on the current task dataset; Step 5: Calculate the feature mean μ of each category for the current task using the trained current task recognition model T. n ={μ c+1 ,…,μ d } and variance ∑ n ={∑ c+1 ,…,∑ d Construct the Gaussian distribution of features for the current task. From the characteristic Gaussian distribution of the old task and the characteristic Gaussian distribution of the current task The process involves sampling feature samples of various types to form a feature set. Where the generated feature set of class j N is the number of samples; Step 6: Insert three modality balancer adapters before the classification layer of the current task: a visual modality balancer adapter, an acceleration modality balancer adapter, and a gyroscope modality balancer adapter. Each modality balancer adapter consists of a downsampling module, a nonlinear mapping layer, and an upsampling module. For the t-th batch B... t Data First, break it down into visual generation features. Acceleration generation features and gyroscope generation features Then, they are sent to the corresponding modal adapters: Among them, U v U a and U g These represent the upsampling modules for the visual, accelerometer, and gyroscope modes, respectively. σ(·) represents the ReLU layer, a nonlinear mapping layer. v D a and D g These represent downsampling modules for the three modalities of vision, acceleration, and gyroscope, respectively. Then, the output of the modality balancing adapter A(·) is added to the generated features of the corresponding modality to obtain the visual features of modality balancing. acceleration characteristics and gyroscope characteristics After being concatenated, the data is fed into the classification layer of the current task to calculate the cross-entropy classification loss and the gradient. Step 7: Decompose the current classification layer weights into visual weights. Acceleration weight and gyroscope weight W t g The sum of the predicted outputs of each modality is calculated by combining the bias b of the classification layer as the contribution of each modality; Then, based on the contribution of each mode, the gradient modulation coefficient of each mode is calculated. Where m represents the mode, This indicates the contribution of the corresponding mode; In calculating the gradient modulation coefficient Then, the corresponding coefficients are multiplied onto the gradient of the corresponding modality balancer, and the SGD optimizer is used to update the modality balancer and the classification layer parameters of the current task based on the gradient. Step 8: After training the modality balance adapter and the classification layer of the current task for 20 epochs using the generated feature set, replace the classification layer of the current recognition model T with the classification layer of the current task that has been retrained using the generated feature set, so as to achieve incremental recognition of the behavior categories of the current task and the old task.

2. The first-view incremental behavior recognition method based on a multimodal adapter as described in claim 1, characterized in that, In step 2, the temporally enhanced visual feature v is obtained. i ′ Acceleration characteristics a i ′ and gyroscope characteristics g i ′ The specific formula is expressed as follows: Concat(·) is a concatenation operation.

3. The first-view incremental behavior recognition method based on a multimodal adapter as described in claim 1, characterized in that, Time-sequential distillation loss in step 4 The calculation method is as follows: Where, ‖·‖2 represents the Euclidean norm, and the number of Transformer encoders is 12.

4. The first-view incremental behavior recognition method based on a multimodal adapter as described in claim 1, characterized in that, The method for calculating the contribution of each mode in step 7 is as follows: Where d is the number of categories up to the current task, b represents the bias of the classification layer, and y represents the category label.

Images