Zero-sample multi-mode first-view-angle behavior recognition method introduced based on visual language knowledge

Abstract

The invention discloses a zero-sample multi-modal first-view behavior recognition method introduced based on visual language knowledge, and belongs to the field of multi-modal behavior recognition. The method comprises the following steps: firstly, inputting a visual mode into a pre-trained CLIP visual encoder to extract visual features, extracting corresponding features from an accelerometer mode spectrogram and a gyroscope mode spectrogram which are subjected to STFT transformation, and extracting text features from a text through a pre-trained CLIP text encoder; and then the visual features are transmitted to an adapter module, and zero sample knowledge and new adaptive feature knowledge are dynamically combined to obtain final visual features. And the accelerometer mode and the gyroscope mode obtain final inertial sensor characteristics through the inertial sensor fusion module. And finally, inputting the text, vision and inertial sensor features into a multi-modal fusion module together, fully considering the problem of alignment among different modals, and effectively improving the modal fusion effect. The method is satisfactory in a zero-sample multi-modal first-view behavior recognition task.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal behavior recognition, and in particular to a zero-sample multimodal first-person perspective behavior recognition method based on the introduction of visual language knowledge. The method can realize behavior recognition for a multimodal first-person perspective data set under a zero-sample setting. Background Art

[0002] In recent years, the task of action recognition in first-person videos has made great progress due to widely annotated datasets and significantly improved deep learning architectures. Egocentric action recognition (EAR) focuses on recognizing human actions in first-person videos, enriching more realistic human-environment interactions, and has attracted increasing attention due to its wide range of applications in human-computer interaction, motion analysis, and video summarization. However, it is impractical to collect and annotate video samples for every possible interaction between objects, so recognizing previously unseen actions remains a challenging task.

[0003] Recently, visual language models pre-trained on large-scale datasets have shown excellent zero-shot generalization capabilities in various visual recognition tasks. The strategy of transferring visual language knowledge from powerful visual language models and extending it to zero-shot egocentric action recognition is becoming an effective and promising approach. Existing methods include LAVILA, GPT4Ego, etc., which perform global video-text matching by directly calculating the cosine similarity of class name and class tag features, achieving excellent performance. However, the above methods cannot adapt well to different downstream datasets and need further fine-tuning. In addition, current research focuses more on zero-shot first-person action recognition of visual modality, and multimodal action recognition tasks remain largely unexplored. Due to the introduction of multimodal information, the alignment of different modalities is not ideal. How to integrate and align multimodal information is a problem that needs to be considered. Therefore, in order to solve the above problems, we propose a zero-shot multimodal first-person action recognition method based on the introduction of visual language knowledge, which better integrates and aligns multimodal information, and performs satisfactorily on the zero-shot multimodal first-person action recognition task. Summary of the invention

[0004] Existing zero-shot action recognition methods based on visual language models cannot adapt well to downstream data sets. If all parameters of the model are optimized, due to over-parameterization and lack of sufficient training samples, it will lead to overfitting to specific data sets and loss of its original generalization ability. Existing zero-shot action recognition methods based on visual language models have not been explored in multimodal action recognition tasks. After the introduction of multimodal data, different modalities contain more complementary information, but it is difficult to establish a corresponding relationship between these information.

[0005] Since the pre-trained visual language model CLIP has powerful zero-sample image recognition capabilities, the present invention uses the visual and text encoders of the CLIP model as feature extractors. A lightweight adapter module is proposed in the visual branch, and only the parameters in the adapter module are fine-tuned to learn new knowledge to adapt to the pre-trained visual language model. The generalization ability is improved by dynamically combining zero-sample knowledge with new adaptive feature knowledge. A multimodal fusion module is proposed, which uses the global information supplemented by the inertial sensor to weight the output of the extracted frame of the visual branch, thereby realizing the fusion and alignment of the visual modality and the inertial sensor modality. The present invention conducts a preliminary exploration of the zero-sample multimodal first-person behavior recognition method based on the visual language model, fully considers the problem of alignment between different modalities, effectively improves the effect of modal fusion, and achieves the most advanced behavior recognition performance. Therefore, the technical solution of the present invention is: a zero-sample multimodal first-person behavior recognition method based on the introduction of visual language knowledge, the method comprising:

[0006] Step 1: Use wearable smart glasses to collect data. Collect synchronized visual data and sensor data through the first-person camera and sensor on the glasses respectively, and construct a multimodal first-person behavior dataset.

[0007] Step 2: For visual data, capture a video and input it into the frozen CLIP visual encoder to extract the visual features U v ; For the text branch, the text is the category name of the behavior in the video, and the names of k categories are put into C I In the I Input to the frozen CLIP text encoder to get the text feature F t ;

[0008] Step 3: For inertial sensor data, which consists of a three-dimensional time series, each dimension of data is converted into a two-dimensional spectrum graph through fast Fourier transform, and then the two-dimensional spectrum graph is extracted into inertial sensor features including accelerometer features U through visual Transformer a , gyroscope characteristic U g ;

[0009] Step 4: Transform the visual features U v Input to the adapter module, the output of the adapter module is consistent with the visual feature U v After splicing, the final visual feature F is obtained after adaptive combination of new and old knowledge v ;

[0010] Step 5: Set the Accelerometer Feature U a , gyroscope characteristic U g Input into the inertial sensor fusion module SA(·) to obtain the inertial sensor feature FIS ;

[0011] Step 6: Transform the text feature F t , visual feature F v , inertial sensor characteristic F i They are input together into the multimodal fusion module to obtain the output recognition result.

[0012] Furthermore, the adapter module A(·) in step 4 is composed of two layers of linear transformations, and its specific formula is as follows:

[0013] A(U v )=ReLU(U v T W1)W2

[0014] In the formula, ReLU is the activation function, W1 and W2 are learnable parameters;

[0015] The output of the adapter module is compared with the visual feature U v The splicing method is:

[0016] F v =αA(U v ) T +βU v

[0017] In the formula, α and β are residual coefficients, A(U v ) is the new knowledge obtained after fine-tuning the adapter.

[0018] Furthermore, the specific method of step 5 is:

[0019] Z=[U a ||U g ]

[0020] [Q, K, V] = [W q Z, W k Z, W v Z]

[0021]

[0022] In the formula, || represents the splicing operation, Z is the input of the inertial sensor, and the input Z is passed through three different projection matrices W q , W k , W v Correspondingly, we get Q, K, V, d is the feature dimension, h is the number of attention heads, and finally we get the inertial sensor feature F i .

[0023] Furthermore, the specific method of the multimodal fusion module is:

[0024] Step 6.1: The feature F of the inertial sensor with global information i With visual features F v Do the inner product and get the importance score S;

[0025] Step 6.2: Calculate visual features F v With the text feature F t The similarity of the matching is obtained to obtain the similarity matrix M ij ;

[0026] Step 6.3: Combine the importance score S and the similarity matrix M ij Do the inner product to get the similarity of each frame in the video, then take the weighted sum of the similarities of all frames and predict the behavior of the current video based on the summation result.

[0027] Furthermore, the specific method of step 6.3 is:

[0028] The importance score S and the similarity matrix M ij Do the inner product and update the similarity of different frame outputs to get F′ v F t , the formula is as follows:

[0029]

[0030]

[0031] F IS Represents the inertial sensor features of the entire video, represents the visual features of the i-th frame, represents the text feature of the jth frame, and the final weighted similarity score, that is, the prediction probability, is as follows:

[0032]

[0033] Among them, τ is the temperature of Softmax, k is the number of categories, P j Represents the predicted probability of category j; the one with the highest score is selected as the prediction result.

[0034] The present invention first inputs the visual modality into the pre-trained CLIP visual encoder to extract visual features, extracts corresponding features from the accelerometer modality and gyroscope modality spectrogram after STFT transformation, and extracts text features from the text through the pre-trained CLIP text encoder. The visual features are then passed into the adapter module, and the zero-sample knowledge is dynamically combined with the new adaptive feature knowledge to obtain the final visual features. The accelerometer modality and gyroscope modality are passed through the inertial sensor fusion module to obtain the final inertial sensor features. Finally, the text, vision, and inertial sensor features are input into the multimodal fusion module together, fully considering the alignment problem between different modalities, and effectively improving the effect of modal fusion. This method performs satisfactorily on the zero-sample multimodal first-person perspective behavior recognition task. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the process of the present invention.

[0036] Figure 2 This is a schematic diagram of the structure of the multimodal fusion module of the present invention. DETAILED DESCRIPTION

[0037] We propose a zero-shot multimodal first-person action recognition method based on visual language knowledge. Existing zero-shot action recognition methods based on visual language models cannot adapt well to downstream datasets, and the multimodal action recognition task remains largely unexplored. This paper designs a zero-shot first-person action recognition method based on visual language knowledge for multimodal scenarios. The overall architecture of this method is as follows: Figure 1 As shown, in order of steps Figure 1 Provide detailed explanation; Figure 1 Middle,U v represents the visual features extracted by the visual encoder; F t Represents the text features extracted by the text encoder; U a , U g They represent the accelerometer features and gyroscope features extracted by the visual Transformer model respectively; STFT is short-time Fourier transform; the adapter module consists of two layers of linear transformation, FC represents the fully connected layer, and ReLU represents the ReLU activation function operation; the multimodal fusion module is described in detail in step 5; W in the inertial sensor fusion module q ,W k ,W v are three different projection matrices in the self-attention mechanism; F v represents the visual features after the adaptive combination of new and old knowledge after the adapter module; F IS is the inertial sensor characteristic.

[0038] Step 1: For visual data, we uniformly sample 8 frames from a video and then input them into the frozen CLIP visual encoder to extract visual features U v For the text branch, the names of k categories C i ={C1,…,C k}Insert the fixed template "a photo of a C i ”, input it into the frozen CLIP text encoder to obtain the text feature F t .

[0039] Step 2: For inertial sensor data, they are composed of three-dimensional time series. Each dimension of data is converted into a two-dimensional spectrum through fast Fourier transform (STFT), and then extracted into inertial sensor features including accelerometer features U through visual transformer. a , gyroscope characteristic U g .

[0040] Step 3: Transform the visual features U v The input is sent to the adapter module, and only a small number of parameters in the adapter module are used to learn new knowledge to adapt to the pre-trained visual language model. Specifically, the learnable adapter module A(·) is composed of two layers of linear transformations, and its specific formula is as follows:

[0041] A(U v )=ReLU(U v T W1)W2#(1.1)

[0042] In the formula, ReLU is the activation function, and W1 and W2 are learnable parameters. v For visual features.

[0043] Since the original pre-trained encoder of CLIP already has powerful representation capabilities, it only needs to be lightly adapted in the form of residuals, which can retain the original generalization ability and dynamically combine zero-shot knowledge with new adaptive feature knowledge. Specifically, the new knowledge obtained by fine-tuning is added to the original visual feature U through residual connections. v , the formula is as follows:

[0044] F v =αA(U v ) T +βU v #(1.2)

[0045] In the formula, α and β are residual coefficients, indicating the degree of retention of the original knowledge. v ) is the new knowledge obtained through fine-tuning of the adapter, U v is the original visual feature, Fv It is the visual feature after the final adaptive combination of new and old knowledge.

[0046] Step 4: Set the Accelerometer Feature U a , gyroscope characteristic U g Input into the inertial sensor fusion module SA(·), the specific formula is as follows:

[0047] Z=[U a ||U g ]#(1.3)

[0048] [Q,K,V]=[W q Z, W k Z, W v Z]#(1.4)

[0049]

[0050] In the formula, || represents the splicing operation, Z is the input of the inertial sensor, and the input Z is passed through three different projection matrices W q , W k , W v Get Q, K, V. d is the feature dimension, h is the number of attention heads. Finally, the inertial sensor feature F is obtained. IS =SA(Z).

[0051] Step 5: Since CLIP is designed for the visual modality, it is not possible to fuse the inertial sensor modality with the visual modality at the feature level. Therefore, we propose a multimodal fusion module. Finally, the text feature F t , visual feature F v , inertial sensor characteristic F is The multimodal fusion module is input together to obtain the output result. Further, the multimodal fusion module is described in detail, such as Figure 2 shown. Figure 2 middle, represents the visual features of the extracted frame, represents the text features of k categories, F IS represents the inertial sensor feature, S = {S1, ...S8} represents the importance scores of the 8 frames respectively, represents the similarity score between the visual feature of the i-th frame and the text feature of the j-th category, P j = {P1, ...P k} represents the final weighted similarity score.

[0052] Specifically, after obtaining the visual feature F v and the inertial sensor modal characteristics F isFinally, the features of the inertial sensor with global information are inner-producted with the visual features of the 8 frames to obtain the importance scores S of the 8 frames. The features of the inertial sensor with global information supplement the visual features of the extracted frames. The inertial sensor modality and the visual modality are aligned by the inner product method without changing the visual features and thus affecting the subsequent output results. The formula is as follows:

[0053]

[0054] Then calculate the similarity between the 8 frames of visual features and text features, and get the similarity matrix M ij , the formula is as follows:

[0055]

[0056] The importance score S is then weighted on the similarity between the visual and textual matches by subtracting the importance score S from the similarity matrix M ij Do the inner product and update the similarity of different frame outputs to get F′ v F t , the formula is as follows:

[0057]

[0058] make

[0059]

[0060] The final weighted similarity score, i.e. the predicted probability, is as follows:

[0061]

[0062] Among them, τ is the temperature of Softmax, k is the number of categories, P j represents the prediction probability of category j. Finally, the one with the highest score is selected as the prediction result, thus realizing the fusion of visual modality and inertial sensor modality.

[0063] In the present invention, we innovatively propose a zero-shot multimodal first-person action recognition method based on the introduction of visual language knowledge. First, the visual modality is input into the pre-trained CLIP visual encoder to extract visual features, and the corresponding features are extracted from the accelerometer modality and gyroscope modal spectrum after STFT transformation through the visual Transformer model, and the text is extracted from the text features through the pre-trained CLIP text encoder. Then the visual features are passed into the adapter module, and the zero-shot knowledge is dynamically combined with the new adaptive feature knowledge to obtain the final visual features. The accelerometer modality and gyroscope modality are passed through the inertial sensor fusion module to obtain the final inertial sensor features. Finally, the text, vision, and inertial sensor features are input into the multimodal fusion module together, fully considering the alignment problem between different modalities, and effectively improving the effect of modal fusion. Our method performs satisfactorily on the zero-shot multimodal first-person action recognition task.

[0064] In the present invention, for the visual and text branches, we selected the CLIP model of VIT-B-32 scale open sourced by OpenAI, and used the pre-trained weights of the visual encoder and text encoder therein as the initialization weights of the visual encoder and text encoder in the present invention. For the inertial sensor branch, we used ViT-B / 16 as the initialization weight of the visual Transformer model. In the present invention, the parameters of the visual Transformer model, the adapter module, and the inertial sensor fusion module are all trainable.

Claims

1. A zero-sample multimodal first-person perspective behavior recognition method based on visual language knowledge introduction, the method comprising: Step 1: Use wearable smart glasses to collect data. Collect synchronized visual data and sensor data through the first-person camera and sensor on the glasses respectively, and construct a multimodal first-person behavior dataset. Step 2: For visual data, capture a video and input it into the frozen CLIP visual encoder to extract the visual features U v ; For the text branch, the text is the category name of the behavior in the video, and the names of k categories are put into C i In the i Input to the frozen CLIP text encoder to get the text feature F t ; Step 3: For inertial sensor data, which consists of a three-dimensional time series, each dimension of data is converted into a two-dimensional spectrum graph through fast Fourier transform, and then the two-dimensional spectrum graph is extracted into inertial sensor features including accelerometer features U through visual Transformer a , gyroscope characteristic U g ; Step 4: Transform the visual features U v Input to the adapter module, the output of the adapter module is consistent with the visual feature U v After splicing, the final visual feature F is obtained after adaptive combination of new and old knowledge v ; Step 5: Set the Accelerometer Feature U a , gyroscope characteristic U g Input into the inertial sensor fusion module SA(·) to obtain the inertial sensor feature F IS ; Step 6: Transform the text feature F t , visual feature F v , inertial sensor characteristic F i They are input together into the multimodal fusion module to obtain the output recognition result.

2. The zero-sample multimodal first-person perspective behavior recognition method based on visual language knowledge introduction as claimed in claim 1, characterized in that: The adapter module A(·) in step 4 is composed of two layers of linear transformations, and its specific formula is as follows: TO THE v )=ReLU(U v T W1)W2 In the formula, ReLU is the activation function, W1 and W2 are learnable parameters; The output of the adapter module is compared with the visual feature U v The splicing method is: F v =αA(U v ) T +βU v In the formula, α and β are residual coefficients, A(U v ) is the new knowledge obtained after fine-tuning the adapter.

3. The zero-sample multimodal first-person perspective behavior recognition method based on visual language knowledge introduction as claimed in claim 1, characterized in that: The specific method of step 5 is: Z=[U a ||U g ] [Q,K,V]=[W q Z,W k Z,W v Z] In the formula, || represents the splicing operation, Z is the input of the inertial sensor, and the input Z is passed through three different projection matrices W q ,W k ,W v Correspondingly, we get Q, K, V, where d is the feature dimension and h is the number of attention heads. Finally, we get the inertial sensor feature F i .

4. The zero-sample multimodal first-person perspective behavior recognition method based on visual language knowledge introduction as claimed in claim 1, characterized in that: The specific method of the multimodal fusion module is: Step 6.1: The feature F of the inertial sensor with global information i With visual features F v Do the inner product and get the importance score S; Step 6.2: Calculate visual features F v With the text feature F t The similarity of the matching is obtained to obtain the similarity matrix M ij ; Step 6.3: Combine the importance score S and the similarity matrix M ij Do the inner product to get the similarity of each frame in the video, then take the weighted sum of the similarities of all frames and predict the behavior of the current video based on the summation result.

5. The zero-sample multimodal first-person perspective behavior recognition method based on visual language knowledge introduction as claimed in claim 1, characterized in that: The specific method of step 6.3 is: The importance score S and the similarity matrix M ij Do the inner product and update the similarity of different frame outputs to get F v ′ F t , the formula is as follows: F IS Represents the inertial sensor features of the entire video, represents the visual features of the i-th frame, represents the text feature of the jth frame, and the final weighted similarity score, that is, the prediction probability, is as follows: Among them, τ is the temperature of Softmax, k is the number of categories, P j Represents the predicted probability of category j; the one with the highest score is selected as the prediction result.

Images