First view angle incremental behavior identification method based on multi-mode adapter

Abstract

The invention discloses a first view angle incremental behavior identification method based on a multi-mode adapter, and relates to the field of video image processing. According to the method, the time sequence perception capability in the old task identification model is migrated to the current task identification model through the time sequence distillation loss, and the incremental identification of the classification layer on multiple tasks is realized through the feature Gaussian sampling of the old task identification model and the feature Gaussian sampling of the current task identification model in combination with the modal balance adapter. The method comprises the following steps: firstly, inserting a multi-mode time sequence sensing adapter in the front and back of forward network layers of all Transform encoders of a Vit-B / 16 model pre-trained on a frozen Image Net data set in a residual form, so as to extract time sequence information from the features of a visual mode, an acceleration mode and a gyroscope mode in the Transform encoders; and the time sequence information of the current task identification model and the old task identification model is close to each other as far as possible through the time sequence distillation loss. And finally, constructing the feature Gaussian distribution of the current task by calculating the feature mean value and variance of the current task recognition model, realizing the feature Gaussian sampling of multiple tasks in combination with the feature Gaussian distribution of the old task recognition model, and inserting a modal balance adapter in front of the classification layer of the current task. And training the classification layer of the current task again by using the sampling features and the modal balance adapter so as to improve the recognition capability of the classification layer on each task.

Description

Technical Field

[0001] The present invention relates to the field of video image processing, and in particular to a first-person perspective incremental action recognition method (Incremental Egocentric Action Recognition) based on a multi-modal adapter (Multi-modal Adapter) in an action recognition task. Background Art

[0002] With the development of wearable devices, more and more researchers have begun to focus on allowing intelligent systems to understand and perceive user behavior from a first-person perspective through multimodal data provided by wearable devices, thereby achieving human-computer interaction. Due to the complexity and variety of actual usage scenarios, deep learning models are required to learn current recognition tasks while ensuring high accuracy for old recognition tasks. However, due to user privacy issues, it is impossible to obtain training data for old recognition tasks when training the model for the current task. In this scenario, the deep learning model will forget the knowledge of the old recognition task when directly learning the current task training data. In order to improve the first-person perspective incremental behavior recognition accuracy of the deep learning model in the absence of multimodal data for the old recognition task, and incrementally learn the temporal perception rules in different scenarios, compared with the existing methods, the present invention incrementally learns the visual attention mechanism of multiple task scenarios based on the technical route of the multimodal adapter.

[0003] Traditional incremental learning methods based on pre-trained models freeze the pre-trained model when training data, and learn multi-task knowledge by continuously expanding the parameters of the classification layer. However, this learning method cannot restrain the network from forgetting old task knowledge. Existing incremental learning methods based on pre-trained models alleviate the forgetting problem by introducing additional parameters, and can be mainly divided into two categories: the first type of method guides the recognition of multi-tasks through task-sharing prompts; the second type of method learns the plasticity of tasks through adapters, and then uses the Gaussian distribution of multi-task features to learn the stability of tasks. The first type of method does not learn the scene characteristics of the task; the second type of method ignores the learning and transfer of the temporal perception commonality of multimodal data, and the sampled multimodal data has the problem of modal imbalance. Summary of the invention

[0004] In order to achieve the transfer of scene knowledge of the old recognition task, the present invention uses the temporal distillation loss to constrain the multimodal temporal perception adapter of the old task and the current task to express temporal knowledge as close as possible. In order to achieve the modal balance of the sampling features, a modal balance adapter is designed to dynamically adjust the gradients of each modal adapter.

[0005] Existing studies have shown that introducing supervision from old task models when learning the temporal information of the current task can improve the model's ability to remember old task scenarios, and balanced multimodal data helps the model recognize the relationship between modalities; in order to improve the model's incremental learning of multiple tasks without storing data, the present invention proposes a multimodal temporal perception adapter network and a modal balance adapter network; therefore, the technical solution of the present invention is: a first-person incremental behavior recognition method based on a multimodal adapter, the method comprising:

[0006] Step 1: Based on the Vit-B / 16 model pre-trained on the ImageNet dataset, the Vit-B / 16 model includes: 1 Patch Embedding layer, 12 Transformer encoders and 1 classification layer, where the Transformer encoder consists of a multi-head self-attention layer MHSA and a forward network layer MLP;

[0007] First, freeze the Patch Embedding layer and all Transformer encoders of Vit-B / 16. Then, insert the multimodal time-aware adapter before and after the forward network layer of all Transformer encoders of the frozen pre-trained Vit-B / 16 model in the form of residual connection, and input the old task dataset to the Vit-B / 16 model with the multimodal time-aware adapter inserted. The multimodal time-aware adapter consists of 3 downsampling modules, 1 multimodal time fusion module F(·) and 3 upsampling modules in sequence.

[0008] The three downsampling modules are respectively and The three upsampling modules are: and For the visual feature v fed into the i-th multimodal temporal perception adapter i , acceleration characteristics a i and gyroscope characteristic g i , respectively, are fed into a downsampling module consisting of a linear layer and Get downsampled features and

[0009] Step 2: Separate the downsampled features of each modality to obtain the CLS token features of each modality and Patch Features The CLS token features are concatenated and sent to the multimodal time series fusion module F(·) to obtain the time series feature f i =F(CLS i ); Then, all CLS token features are added with the timing feature fi After that, it is concatenated with the Patch feature and sent to the ReLU activation layer σ(·) for nonlinear mapping; the mapped feature is then sent to the upsampling module composed of linear layers. and Get the temporally enhanced visual feature v i ′ , acceleration characteristics a i ′ and gyroscope characteristic g i ′ ;

[0010] Then, the temporally enhanced visual feature v i ′ , acceleration characteristics a i ′ and gyroscope characteristic g i ′ Added to the forward network layer output of the Transformer encoder;

[0011] Step 3: On the old task data, train the Vit-B / 16 model with the multimodal temporal perception adapter inserted. The trained Vit-B / 16 model with the multimodal temporal perception adapter inserted is recorded as the old task recognition model S; the old task has a total of c categories, and the feature mean μ of each category of the old task is calculated o ={μ1,…,μ c} and variance ∑ o ={∑1,…,∑ c}, construct the characteristic Gaussian distribution of the old task

[0012] Step 4: The number of categories of the current task is dc. The old task classification layer that originally classified c categories is incrementally expanded to the current task classification layer that classifies d categories; the current task dataset is sent to the old task recognition model S to obtain the old model temporal features f of the multimodal temporal perception adapter of the i-th Transformer encoder i o , and the current model temporal features f of the multimodal temporal perception adapter of the i-th Transformer encoder of the current recognition model T i n Calculating the time distillation loss

[0013] The total loss of the current recognition model T is:

[0014]

[0015] in, is the cross entropy classification loss of the current recognition model T for the current task dataset;

[0016] Step 5: Use the trained current task recognition model T to calculate the feature mean μ of each category of the current task n ={μ c+1 ,…,μ d} and variance ∑ n ={∑ c+1 ,…,∑ d}, construct the characteristic Gaussian distribution of the current task From the characteristic Gaussian distribution of the old task And the characteristic Gaussian distribution of the current task In the above example, we sample the feature samples of each category to form a generated feature set. The generated feature set of the jth class is N is the number of samples;

[0017] Step 6: Insert three modal balance adapters before the classification layer of the current task, namely: visual modal balance adapter, acceleration modal balance adapter, and gyroscope modal balance adapter; each modal balance adapter consists of a downsampling module, a nonlinear mapping layer, and an upsampling module; for the tth batch B t Data First split into visual generation features Acceleration Generation Feature and gyroscope generation features Then, they are sent to the corresponding modal adapters:

[0018]

[0019] Among them, U v , U a and U g Represent the upsampling modules of the three modes of vision, acceleration and gyroscope, σ(·) represents the nonlinear mapping ReLU layer, and D v , D a and D g Represent the downsampling modules of the three modes: vision, acceleration, and gyroscope;

[0020] Then, the output of the modal balance adapter A(·) is added to the generated features of the corresponding mode to obtain the visual features of modal balance Acceleration characteristics and gyroscope characteristics After being spliced ​​together, they are sent to the classification layer of the current task to calculate the cross entropy classification loss and the gradient;

[0021] Step 7: Split the current classification layer weights into visual weights Acceleration Weight and the gyroscope weight W tg , and combined with the bias b of the classification layer, the sum of the predicted outputs of each mode is calculated as the contribution of each mode;

[0022] Then, according to the contribution of each mode, the gradient modulation coefficient of each mode is calculated

[0023]

[0024] Where m represents the mode, Indicates the contribution of the corresponding mode;

[0025] In calculating the gradient modulation coefficient After that, the corresponding coefficient is multiplied by the gradient of the corresponding modal balance adapter, and the SGD optimizer is used to update the modal balance adapter and the classification layer parameters of the current task according to the gradient;

[0026] Step 8: After training the modality balance adapter and the classification layer of the current task with the generated feature set for 20 epochs, replace the classification layer of the current recognition model T with the classification layer of the current task trained again with the generated feature set to achieve incremental recognition of the behavior categories of the current task and the old task.

[0027] Furthermore, the temporally enhanced visual feature v obtained in step 2 is i ′ , acceleration characteristics a i ′ and gyroscope characteristic g i ′ The specific formula is expressed as:

[0028]

[0029] Among them, Concat(·) is a concatenation operation.

[0030] Furthermore, the sequential distillation loss in step 4 is The calculation method is:

[0031]

[0032] Among them, ‖·‖2 represents the Euclidean norm, and the number of Transformer encoders is 12.

[0033] Furthermore, the calculation method of the contribution of each mode in step 7 is:

[0034]

[0035] Among them, d is the number of categories up to the current task, b represents the bias of the classification layer, and y represents the category label.

[0036] A first-person perspective incremental behavior recognition method based on a multimodal adapter proposed in the present invention achieves an average incremental recognition accuracy of 3.41 percentage points higher than the most advanced method on the public dataset UESTC-MMEA-CL without storing old task data, and the final recognition accuracy is improved by 4.1 percentage points. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the Vit-B / 16 model with a multimodal temporal perception adapter inserted.

[0038] Figure 2 Schematic diagram of a first-person perspective incremental behavior recognition method based on a multimodal adapter. DETAILED DESCRIPTION

[0039] The present invention aims to transfer the temporal common knowledge of multimodal data to the current task model under the supervision of the old task model, and realize the modal balance of the sampling features, and retrain the classification layer of the current task through the balanced sampling features. First, the temporal knowledge of the old task is learned through the multimodal temporal perception adapter, and the characteristic Gaussian distribution of the old task is established. Then, when learning the current task, the temporal knowledge of the old task is distilled into the current model, and the modal balance adapter is used to perform modal balance on the Gaussian sampling features of the old task and the current task, and the classification layer of the current task is retrained.

[0040] Step 1: Insert the multimodal temporal perception adapter into the Vit-B / 16 model pre-trained on the frozen ImageNet dataset, train the multimodal temporal perception adapter and the classification layer of the old task through the old task dataset, and obtain the old task recognition model; Step 2: Calculate the feature mean and variance of each class of the old task dataset through the old task recognition model;

[0041] Step 3: When training on the current task dataset, synchronously input the data into the old task recognition model, calculate the temporal distillation loss between the temporal information of the old task and the current task, and optimize the current task recognition model together with the classification loss;

[0042] Step 4: Calculate the feature mean and variance of each category of the current task dataset through the current task recognition model;

[0043] Step 5: Insert the modality balance adapter before the classification layer of the current task, input the sampled features of the old task and the current task into the modality balance adapter, and calculate the classification loss;

[0044] Step 6: Calculate the contribution of each mode according to the weight of the classification layer, and then calculate the gradient modulation coefficient of each mode, multiply the coefficient by the gradient of the corresponding modal balance adapter, and update the parameters of the classification layer of the modal balance adapter and the current task. The average incremental recognition accuracy is 3.41 percentage points higher than the most advanced method, and the final recognition accuracy is 4.1 percentage points higher. Table 1 shows the performance comparison of the method of the present invention and other methods under the setting of 4 tasks × each task contains 8 categories in UESTC-MMEA-CL. The average recognition accuracy of task k represents the recognition accuracy of the recognition model after task k training is completed for the categories of task k and previous tasks. The final recognition accuracy represents the average recognition accuracy of the last task, and the average incremental recognition accuracy represents the mean of the average recognition accuracy of all tasks.

[0045] Table 1

[0046] method Final recognition accuracy Average incremental recognition accuracy FeTrIL[1] 82.60 89.73 FOSTER[2] 84.45 91.35 SSIAT[3] 89.97 92.57 Method of the present invention 94.07 95.98

[0047] [1] Petit G, Popescu A, Schindler H, et al. Fetril: Feature translation for exemplar-free class-incremental learning [C] / / Proceedings of the IEEE / CVFwinter conference on applications of computer vision. 2023: 3911-3920.

[0048] [2] For Wang FY, Zhou DW, Ye HJ, et al. Foster: Feature boosting and compression for class-incremental learning [C] / / European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 398-414.

[0049] [3] Tan Y, Zhou Q, Xiang X, et al. Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024:23252-23262.

Claims

1. A first-person perspective incremental behavior recognition method based on a multimodal adapter, the method comprising: Step 1: Based on the Vit-B / 16 model pre-trained on the ImageNet dataset, the Vit-B / 16 model includes: 1 Patch Embedding layer, 12 Transformer encoders and 1 classification layer, where the Transformer encoder consists of a multi-head self-attention layer MHSA and a forward network layer MLP; First, freeze the Patch Embedding layer and all Transformer encoders of Vit-B / 16. Then, insert the multimodal time-aware adapter before and after the forward network layer of all Transformer encoders of the frozen pre-trained Vit-B / 16 model in the form of residual connection, and input the old task dataset to the Vit-B / 16 model with the multimodal time-aware adapter inserted. The multimodal time-aware adapter consists of 3 downsampling modules, 1 multimodal time fusion module F(·) and 3 upsampling modules in sequence. The three downsampling modules are respectively and The three upsampling modules are: and For the visual feature v fed into the i-th multimodal temporal perception adapter i , acceleration characteristics a i and gyroscope characteristic g i , respectively, are fed into a downsampling module consisting of a linear layer and Get downsampled features and Step 2: Separate the downsampled features of each modality to obtain the CLS token features of each modality and Patch Features The CLS token features are concatenated and sent to the multimodal time series fusion module F(·) to obtain the time series feature f i =F(CLS i ); Then, all CLS token features are added with the timing feature f i After that, it is concatenated with the Patch feature and sent to the ReLU activation layer σ(·) for nonlinear mapping; the mapped feature is then sent to the upsampling module composed of linear layers. and Get the temporally enhanced visual feature v i ′ , acceleration characteristics a i ′ and gyroscope characteristic g i ′ ; Then, the temporally enhanced visual feature v i ′ , acceleration characteristics a i ′ and gyroscope characteristic g i ′ Added to the forward network layer output of the Transformer encoder; Step 3: On the old task data, train the Vit-B / 16 model with the multimodal temporal perception adapter inserted. The trained Vit-B / 16 model with the multimodal temporal perception adapter inserted is recorded as the old task recognition model S; the old task has a total of c categories, and the feature mean μ of each category of the old task is calculated o ={μ1,…,μ c } and variance ∑ o ={∑1,…,∑ c }, construct the characteristic Gaussian distribution of the old task Step 4: The number of categories of the current task is dc. The old task classification layer that originally classified c categories is incrementally expanded to the current task classification layer that classifies d categories; the current task dataset is sent to the old task recognition model S to obtain the old model temporal features f of the multimodal temporal perception adapter of the i-th Transformer encoder i o , and the current model temporal features f of the multimodal temporal perception adapter of the i-th Transformer encoder of the current recognition model T i n Calculating the time distillation loss The total loss of the current recognition model T is: in, is the cross entropy classification loss of the current recognition model T for the current task dataset; Step 5: Use the trained current task recognition model T to calculate the feature mean μ of each category of the current task n ={μ c+1 ,…,μ d } and variance ∑ n ={∑ c+1 ,…,∑ d }, construct the characteristic Gaussian distribution of the current task From the characteristic Gaussian distribution of the old task And the characteristic Gaussian distribution of the current task In the above example, we sample the feature samples of each category to form a generated feature set. The generated feature set of the jth class is N is the number of samples; Step 6: Insert three modal balance adapters before the classification layer of the current task, namely: visual modal balance adapter, acceleration modal balance adapter, and gyroscope modal balance adapter; each modal balance adapter consists of a downsampling module, a nonlinear mapping layer, and an upsampling module; for the tth batch B t Data First split into visual generation features Acceleration Generation Feature and gyroscope generation features Then, they are sent to the corresponding modal adapters respectively: Among them, U v , U a and U g Represent the upsampling modules of the three modes of vision, acceleration and gyroscope, σ(·) represents the nonlinear mapping ReLU layer, and D v , D a and D g Represent the downsampling modules of the three modes: vision, acceleration, and gyroscope; Then, the output of the modal balance adapter A(·) is added to the generated features of the corresponding mode to obtain the visual features of modal balance. Acceleration characteristics and gyroscope characteristics After being spliced ​​together, they are sent to the classification layer of the current task to calculate the cross entropy classification loss and the gradient; Step 7: Split the current classification layer weights into visual weights Acceleration Weight and the gyroscope weight W t g , and combined with the bias b of the classification layer, the sum of the predicted outputs of each mode is calculated as the contribution of each mode; Then, according to the contribution of each mode, the gradient modulation coefficient of each mode is calculated Where m represents the mode, Indicates the contribution of the corresponding mode; In calculating the gradient modulation coefficient After that, the corresponding coefficient is multiplied by the gradient of the corresponding modal balance adapter, and the SGD optimizer is used to update the modal balance adapter and the classification layer parameters of the current task according to the gradient; Step 8: After training the modality balance adapter and the classification layer of the current task with the generated feature set for 20 epochs, replace the classification layer of the current recognition model T with the classification layer of the current task trained again with the generated feature set to achieve incremental recognition of the behavior categories of the current task and the old task.

2. A first-person perspective incremental behavior recognition method based on a multimodal adapter as claimed in claim 1, characterized in that: The temporally enhanced visual feature v is obtained in step 2 i ′ , acceleration characteristics a i ′ and gyroscope characteristic g i ′ The specific formula is expressed as: Among them, Concat(·) is a concatenation operation.

3. The first-person perspective incremental behavior recognition method based on a multimodal adapter as claimed in claim 1, characterized in that: The sequential distillation loss in step 4 The calculation method is: Among them, ‖·‖2 represents the Euclidean norm, and the number of Transformer encoders is 12.

4. The first-person perspective incremental behavior recognition method based on a multimodal adapter as claimed in claim 1, characterized in that: The calculation method of the contribution of each mode in step 7 is: Among them, d is the number of categories up to the current task, b represents the bias of the classification layer, and y represents the category label.

Images