A bidirectional domain closure behavior recognition method

Abstract

The application relates to a bidirectional domain gap bridging behavior recognition method, which comprises the following steps: acquiring an RGB image frame sequence to be recognized and a category text thereof; inputting the RGB image frame sequence to be recognized into a text prompt generation module to obtain a generated text; inputting the RGB image frame sequence to be recognized, the category text thereof and the generated text into a behavior recognition model to obtain the action category of the video to be recognized; the behavior recognition model comprises a first text encoder module, an image encoder module, a time sequence encoder module, a second text encoder module, an interaction module, a fusion module and a similarity calculation module. The application can effectively reduce the domain gap between a pre-training model and a behavior recognition task, thereby significantly improving the applicability and performance of a CLIP model in the field of behavior recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a bidirectional domain healing behavior recognition method. BACKGROUND

[0002] Behavior recognition is one of the most basic but also most challenging tasks in video understanding. It plays a crucial role in many visual applications, such as micro-video recommendation, sports video analysis, automatic driving, etc. With the rapid increase of video quantity on the Internet, large-scale action recognition has become a key problem that needs to be solved urgently. A powerful action recognition method should be able to understand the semantic information in the video, and even be able to automatically describe the content, such as human action and complex events, and achieve the accuracy comparable to human beings.

[0003] In the past few years, behavior recognition based on convolutional neural networks and Transformers has made significant progress. However, most of the existing works follow the closed-set learning setting, where all the classes are predefined. This approach is impractical for many real-world applications, such as automatic tagging of web videos, where there is no information about new video classes during the training process. Therefore, it is very challenging for the known class method to train a classifier to recognize unseen or unfamiliar classes.

[0004] With the development of pre-trained large models, recent research in image classification shows that guiding large-scale contrastive language-image pre-training (CLIP) models to solve classification tasks can significantly enhance the generalization ability of existing models. The CLIP model is a kind of vision-language (VL) model, which learns joint representations from pairs of text and images of network scale, and then aligns the representations to a shared embedding space through simple noise contrastive learning. However, since the CLIP model is pre-trained on large-scale image-text data, there is a significant domain gap between its pre-training task and the downstream video behavior recognition task, and direct application of the CLIP model will result in poor performance. Therefore, in order to fully exploit the potential of pre-trained models in the field of behavior recognition, it is necessary to adapt the CLIP model from both visual and textual aspects to bridge the domain gap.

[0005] To solve the domain adaptation problem from images to videos, the common methods currently include directly fine-tuning end-to-end, and extracting temporal cues through simple post-fusion, and the post-fusion method usually generates video representation by using inter-frame attention mechanism or frame-level embedding fusion. In addition, some methods try to introduce carefully designed learnable components (such as self-attention layers, visual decoders or visual prompt modules) in the encoder of CLIP. However, when dealing with interfering pictures in videos, the attention of the model is prone to deviate to the background and pay insufficient attention to the moving subject, thereby restricting the overall performance.

[0006] In addition, there are also some works trying to solve the domain gap problem between the narrative text adopted by CLIP pre-training and the concise behavior labels in downstream tasks, for example, generating more rich semantic details from behavior labels to make up for the lack of semantic information of behavior labels. However, the inventors of the present application have noticed that the way of generating prompts from class names has limitations: the generated class prompts may not be completely applicable to each video sample, and it is also difficult to cover all rich details related to actions in the video in detail. At the same time, such methods often only focus on bridging the semantic domain from labels to descriptions, ignoring the visual domain adaptation, which leads to insufficient temporal expression ability of the model and makes it difficult to achieve robust performance. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a bidirectional domain bridging behavior recognition method, which can effectively reduce the domain gap between the pre-trained model and the behavior recognition task, thereby significantly improving the applicability and performance of the CLIP model in the field of behavior recognition.

[0008] The technical solution adopted by the present application to solve the technical problem is: providing a bidirectional domain bridging behavior recognition method, comprising the following steps:

[0009] obtaining a sequence of RGB image frames to be recognized and its category text;

[0010] inputting the sequence of RGB image frames to be recognized into a text prompt generation module to obtain generated text;

[0011] inputting the sequence of RGB image frames to be recognized, its category text and the generated text into a behavior recognition model to obtain the action category of the video to be recognized, the behavior recognition model comprising:

[0012] a first text encoder module for encoding the generated text to obtain generated text features;

[0013] an image encoder module for generating first image features according to the sequence of RGB image frames to be recognized;

[0014] a timing encoder module integrated with the motion prompt generation module, configured to generate a second image feature according to the to-be-identified RGB image frame sequence under the guidance of the motion prompt generation module;

[0015] a second text encoder module configured to encode the category text to obtain a category text feature;

[0016] an interaction module configured to encode the generative text feature and the first image feature to obtain a more discriminative image feature with text modality information introduced;

[0017] a fusion module configured to fuse the more discriminative image feature and the second image feature to obtain a fusion feature;

[0018] a similarity calculation module configured to calculate a similarity between the category text feature and the fusion feature.

[0019] The inputting the to-be-identified RGB image frame sequence into the text prompt generation module to obtain the generative text specifically comprises:

[0020] generating a text label according to the to-be-identified RGB image frame sequence using a video description generation model;

[0021] generating an action keyword according to all possible labels given and the provided generative text label, and providing the action keyword as part of an instruction to the text prompt generation module;

[0022] providing a number of examples and giving a specification instruction of a system prompt, requiring the text prompt generation module to generate a text without noise according to the provided text label and action keyword to obtain the generative text.

[0023] The first text encoder module comprises:

[0024] a first tokenizer configured to encode the generative text to obtain a word vector of the generative text;

[0025] a generative text encoder adopting a text encoder structure of a CLIP model, configured to encode the word vector of the generative text to obtain a generative text feature;

[0026] The generative text encoder participates in fine-tuning of a first period during training.

[0027] The image encoder module comprises:

[0028] an image conversion unit configured to convert the to-be-identified RGB image frame sequence into an image embedding sequence;

[0029] An image encoder, a visual encoder adopting a CLIP model, is configured to encode the image embedding sequence to obtain first image features;

[0030] The image encoder keeps all parameters frozen during training.

[0031] The temporal encoder module comprises:

[0032] An image conversion unit is configured to convert the sequence of RGB image frames to be recognized into an image embedding sequence.

[0033] A temporal encoder, a pre-trained visual encoder adopting a CLIP model, is configured to encode the image embedding sequence under the guidance of a motion cue generation module to obtain second image features.

[0034] The motion cue generation module inputs image embedding features of each layer of the temporal encoder, splices image embedding features of adjacent frames to obtain paired frame features, uses a convolution module to extract the similarity of the paired frame features to obtain similarity features, reduces the dimension of the similarity features through weight extraction to obtain the weight of the similarity features, and uses the weight of the similarity features to weight the image embedding features to obtain motion subject summaries of each RGB image frame. The motion subject summaries of each frame are interacted through a one-dimensional convolution module to obtain motion subject cues of the RGB image frames.

[0035] The fusion module is composed of a standard multi-head self-attention mechanism, a feedforward neural network, and an average pooling layer.

[0036] The second text encoder module comprises:

[0037] A second tokenizer is configured to encode the category text to obtain a word vector of the category text.

[0038] A category text encoder, a text encoder structure adopting a CLIP model, is configured to encode the word vector of the category text to obtain a category text feature.

[0039] The category text encoder keeps all parameters frozen during training.

[0040] The interaction module performs cross-attention calculation on the patch level using the generated text feature as a query, the first image feature as a key and a value, extracts local visual region features strongly related to the text cue, and obtains more distinctive image features introducing text modal information.

[0041] The behavior recognition model adopts a two-stage training method, wherein the first stage fine-tunes the first text encoder module, the interaction module, and the fusion module, and the second stage fine-tunes the temporal encoder module and the fusion module.

[0042] The total loss of the behavior recognition model in the first stage training includes a cross-entropy loss and a cosine loss of the timing encoder module, and the total loss in the second stage training is a cross-entropy loss, wherein the cross-entropy loss is: wherein L entro is a cross-entropy loss, sim() is a similarity calculation, v i is a feature representation of the i-th sample, c i is a feature representation of the correct class of the i-th sample, τ is a temperature hyperparameter for scaling the similarity, c j is a feature representation of the j-th class.

[0043] Advantages

[0044] Compared with the prior art, the application has the following advantages and positive effects: the application makes the model focus more on the movement itself based on the visual domain motion subject prompt, and the timing clues obtained are clearer and more robust; the generated text prompt is used as an intermediate bridge in the text domain to guide the formation of a visual-text prompt-label bidirectional information transmission pipeline, which is easy to match. The application realizes the bidirectional healing of the CLIP model in the visual-text domain in the behavior recognition task, and fully utilizes the potential of the VL model in the video behavior recognition task. In particular, the method provided by the application not only achieves the best performance in the full-supervised task, but also performs well in the zero-shot and few-shot tasks. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flowchart of the behavior recognition method of the bidirectional domain healing of the embodiment of the application;

[0046] Figure 2 is a generation flowchart of the text prompt generation module of the embodiment of the application;

[0047] Figure 3 is a structural schematic diagram of the behavior recognition model in the embodiment of the application;

[0048] Figure 4 is a structural schematic diagram of the motion prompt generation module in the embodiment of the application;

[0049] Figure 5 is a structural schematic diagram of the image encoder and the text encoder in the embodiment of the application. DETAILED DESCRIPTION

[0050] The application will be further described below with reference to specific embodiments. It should be understood that these embodiments are only used to explain the application and not used to limit the scope of the application. Furthermore, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content of the application, and these equivalent forms also fall within the scope of the appended claims of the application.

[0051] Embodiments of the application relate to a bidirectional domain-bridging behavior recognition method, as shown in the following steps: Figure 1

[0052] Step 1, obtaining a sequence of RGB image frames to be recognized and its category text.

[0053] Step 2, inputting the sequence of RGB image frames to be recognized into a text prompt generation module to obtain a generated text.

[0054] In order to make up for the domain gap in text when the CLIP model is migrated to the downstream behavior recognition task, and solve the problem that the existing text prompt learning method does not match the video content accurately, the embodiment obtains a descriptive text strongly related to the sequence of RGB image frames to be recognized as a prompt to guide the text domain bridging through the text prompt generation module.

[0055] Considering that obtaining manual video annotations is a costly and labor-intensive task, the embodiment uses a zero-shot video description generation model to automatically generate annotations. The video description generation model used can be a video extension version of the image-level ZeroCap model based on GPT-2, which can directly obtain descriptive annotations (caption) from downstream unseen videos without additional training. The embodiment first generates 20 captions from the sequence of RGB image frames to be recognized at one time, uses the CLIP score to sort the obtained captions from high to low, and selects the highest scoring caption as the text prompt input to the text prompt generation module.

[0056] However, the way of directly obtaining content prompts from the video description generation model may bring noisy and ethically problematic texts due to the hallucination tendency of the generative LLM. In order to generate high-quality text prompts, the embodiment further adopts the form of instruction-following data to fully utilize the prior semantic knowledge of the LLM to filter the captions generated by the video description generation model.

[0057] The specific process is shown in Figure 2 The text prompt generation module of the application is based on GPT-3.5, and for an image X v , the generated caption X c , and its associated action keywords X k ​, create a set of canonical instructions X i that instruct the assistant to perform data filtering, returning generated text C k after filtering out content unrelated to action keywords X p for guiding cross-modal interaction. Specifically, the filtering step of data is as follows:

[0058] generate corresponding action keywords X k by GPT-3.5 from the hard prompt "a video described as [X c ] may be related to the label [X k ] in the label set [K]". Subsequently, the action keywords X k are provided to GPT-3.5 as part of the instructions to guide it to limit the filtered video description within the corresponding category range, reducing action-unrelated noise;

[0059] provide several examples X e to facilitate GPT-3.5 understanding and generating similar high-quality text;

[0060] give a canonical instruction X i of a system prompt, requiring GPT-3.5 to generate generated text C c without noise according to the provided caption X k and action keywords X p ;

[0061] input the above content to GPT-3.5, and GPT-3.5 returns generated text C p .

[0062] The generated text prompt obtained by the text prompt generation module in this embodiment is closely related to the action label and has rich video-related descriptive semantic details, which serves as a bridge to bridge the domain gap between descriptive text and concise labels.

[0063] Step 3, input the to-be-recognized RGB image frame sequence, its category text and the generated text into the behavior recognition model to obtain the action category of the to-be-recognized video.

[0064] As shown in Figure 3 , the behavior recognition model in this embodiment includes:

[0065] a first text encoder module for encoding the generated text to obtain generated text features;

[0066] an image encoder module for generating first image features according to the to-be-recognized RGB image frame sequence;

[0067] a time sequence encoder module integrated with a motion prompt generation module, configured to generate a second image feature according to the sequence of RGB image frames to be recognized under the guidance of the motion prompt generation module;

[0068] a second text encoder module configured to encode the category text to obtain a category text feature;

[0069] an interaction module configured to encode the generative text feature and the first image feature to obtain more distinctive image information with the introduction of text modal information;

[0070] a fusion module configured to fuse the visual-text prompt-label information and the second image feature to obtain a fusion feature;

[0071] a similarity calculation module configured to calculate the similarity between the category text feature and the fusion feature.

[0072] The first text encoder module comprises: a first tokenizer configured to encode the generative text to obtain a word vector of the generative text; and a generative text encoder adopting a text encoder structure of a CLIP model and configured to encode the word vector of the generative text to obtain a generative text feature. The second text encoder module comprises: a second tokenizer configured to encode the category text to obtain a word vector of the category text; and a category text encoder adopting a text encoder structure of a CLIP model and configured to encode the word vector of the category text to obtain a category text feature.

[0073] The generative text encoder and the category text encoder both adopt the text encoder structure of the original CLIP model, which comprises L c Transformer encoder layers. The overall structure of the Transformer encoder layer is shown in Figure 5 The tokenizer encodes the corresponding input text to obtain a word vector. The parameters of the category text encoder are kept frozen throughout, and the generative text encoder participates in the first-stage fine-tuning.

[0074] After the last layer of the Transformer encoder of the generative text encoder, the generative text encoder directly outputs the generative text feature e p = [CLS, E p1 , E p2 , …, E pN ] and fuses it with the first image feature generated by the image encoder module. The category text encoder directly extracts the class token in the category text feature as a text representation for calculating the similarity between the fusion feature output by the fusion module.

[0075] The image encoder module in the embodiment includes: an image conversion (i.e. image Tokenization) unit for converting the to-be-identified RGB image frame sequence into an image embedding sequence; an image encoder adopting a visual encoder of a CLIP model for encoding the image embedding sequence to obtain a first image feature; and the image encoder keeps the parameters frozen during training.

[0076] The image conversion unit follows the provisions of ViT, and for a video input of V∈R T×H×W×3 (H and W herein represent the height and width resolution of the picture respectively, and T represents the number of video frames), after passing through the image conversion unit, the image is converted into an image embedding sequence, and the t-th frame embedding sequence is composed of N non-overlapping patches and a [CLS] token, and is represented as: where d is the feature dimension. Finally, in order to maintain the relative spatial position relationship of each node in the image feature sequence, a learnable position encoding is added to the frame embedding sequence.

[0077] The image encoder mainly functions to overcome the generalization problem caused by the text gap in combination with the interaction module. The image encoder adopts a visual encoder of a CLIP model, freezes the visual encoder of the CLIP model, provides fine-grained semantic alignment of the visual knowledge with the generated text prompts, causes the visual and the text to be associated in a relatively rich semantic space, and thus more easily anchors the visual content from the text.

[0078] The input of the image encoder is an image embedding sequence v *,0 ∈R T× (N+1) ×d After L-layer Transformer encoding, the output is v *,L ∈R T× (N+1)×d, and v *,L is sent to the interaction module together with the output of the generated text encoder to obtain a final frame representation. As shown in Figure 5 , the specific structure of the image encoder is the same as that of the generated text encoder.

[0079] The interaction module in the embodiment forms a visual-text prompt-label information flow by taking the generated text produced by the first text encoder module as an intermediate bridge, and realizes the bridging of the text domain.

[0080] In the visual-text prompt direction, the embodiment extracts necessary visual details related to the behavior label by using a text prompt containing rich semantics. The interaction module encodes the video frame and the generated text prompt through the frozen CLIP image encoder and the fine-tuned generated text encoder respectively, takes the generated text feature as a query, and takes the first image feature At the patch level, cross-attention calculations are performed using the key and value to extract local visual region features strongly correlated with the text prompt, represented as: Finally, the token representing the frame is taken from the embedded features of the frame after the interaction.

[0081] In the text prompt-label direction, this implementation is Add on the basis Multi-frame fusion is performed to extract video representations, which are then used to calculate similarity with the category features encoded by the text encoder. Textual semantic information related to the tags is progressively introduced into the video representation, bridging the gap between video and tags. Through these strategies, this implementation uses text prompts as an intermediary bridge, constructing a bidirectional information flow channel between visual, textual prompts, and tags through an interactive process guided by generative text prompts, thus making the video-tag pair more closely connected in the latent space.

[0082] The temporal encoder module in this embodiment includes: an image conversion unit, used to convert the RGB image frame sequence to be identified into an image embedding sequence; and a temporal encoder, a pre-trained visual encoder using the CLIP model, used to encode the image embedding sequence under the guidance of the motion cue generation module to obtain a second image feature.

[0083] The image conversion unit is the same as the image conversion unit in the image encoding module, and will not be described again here.

[0084] In this embodiment, the motion cues generation module is integrated into the timing encoder. The specific generation process is as follows:

[0085] In a temporal encoder, the embedding features of the l-th layer are represented as: For each layer of input image embedding features, the image embedding features of adjacent frames are concatenated to obtain paired frame features. The similarity of paired frame features is extracted using a convolutional module to obtain similarity features.

[0086]

[0087] Similarity features Dimensionality reduction is performed using a weight extraction module to obtain the weights W of the similarity features. m,l The image embedding features are weighted using the weights of similarity features to obtain the moving subject cue for each RGB image frame. Subsequently, a cosine loss constraint is imposed on the motion subject summary of consecutive frames, guiding the weight extraction module to optimize the generated weight mask, so that the weight is large in the similar area of adjacent frames and small in the different area. Because the similar area between frames is more likely to correspond to the motion subject in the video, the attention ability of the model to the motion subject is enhanced. Among them, the weight of the similarity feature and the motion subject prompt can be represented as:

[0088]

[0089]

[0090] For each frame motion subject summary, the embodiment further adopts an adapter structure, which first reduces the dimension and then increases the dimension, uses a 1D convolution module for inter-frame subject motion information transmission, and obtains the motion subject prompt corresponding to each frame. In general, the structure of the motion subject prompt generation module is as shown in Figure 4

[0091] The temporal encoder adopts the pre-trained visual encoder of the CLIP model, which contains L layers of Transformer encoding layers, which are mainly composed of self-attention mechanism and feedforward neural network, and is used to extract image feature information. In particular, in order to extract temporal information robust to background interference, the motion subject prompt generation module is inserted at the front end of each layer of the Transformer encoding layer, which relies on the embedding features of the previous layer to generate the motion subject prompt, which is used to guide the attention distribution of the current layer.

[0092] Specifically, the calculation of the lth layer of the temporal encoder can be represented as:

[0093]

[0094] Among them, is the output of the motion subject prompt generation module, and the output of the motion subject prompt generation module is added to the end of the frame embedding sequence of each frame, and after completing the self-attention guidance of the lth layer of the temporal encoder, the motion subject prompt is discarded to obtain the frame embedding sequence output of the lth layer.Finally, the category token of the frame embedding sequence of the Lth layer is used as the output of the temporal encoder.

[0095] The temporal encoder module in the embodiment guides the global and local temporal coding process using the motion subject prompt, avoiding the interference caused by the non-motion subject of the picture subject, so that more robust temporal clues can be obtained.

[0096] The fusion module in the embodiment aims to efficiently fuse the T-frame embedding sequence output by the interaction module and the temporal encoder module, integrate the feature representations of each frame, and generate a video representation that is more suitable for the requirements of the identification task. The fusion module is composed of a standard multi-head self-attention mechanism, a feedforward neural network, and an average pooling layer to fully capture the global correlation between frames and improve the expression ability of the video representation.

[0097] The similarity module in the embodiment can adopt a cosine similarity module, and the calculation of the similarity score s V can be represented by the following formula:

[0098] s V = sim(v, c)

[0099] where sim() represents the cosine similarity calculation.

[0100] During the training process, the symmetric cross-entropy loss is used to maximize the similarity between the matched video-label pairs and minimize the similarity of the non-matched sample pairs. The cross-entropy loss is represented as:

[0101]

[0102] where L entro is the cross-entropy loss, sim() is the similarity calculation, τ is the temperature hyperparameter for scaling the similarity, i represents the sample index in the training data, v i is the feature representation of the i-th sample. c i is the feature representation of the correct class of the i-th sample, j represents the index of the class, and all classes are traversed, c j is the feature representation of class j.

[0103] To avoid excessive dependence on the input text prompt, the embodiment adopts a two-stage training method. The first stage fine-tunes the first text encoder and the interaction module, and the second stage fine-tunes the temporal encoder module. During the entire training process, the fusion module participates in fine-tuning. The total loss in the first stage is divided into two parts, the first part is the cross-entropy loss described above, and the second part is the cosine loss in the temporal encoder module. The total loss is expressed as follows:

[0104] L = L entro + L cos′

[0105] The cross-entropy loss L entro in the second stage only calculates the cross-entropy loss between the second image feature output by the temporal encoder module and the category text feature, focusing on the fine-tuning of the temporal encoder. In the inference process, the similarity scores between the fusion features and the text of each category are calculated, and the category with the highest similarity score is taken as the final classification result.

[0106] It can be found that the method of the embodiment makes the model focus more on the movement itself based on the visual domain of the movement subject prompt, the obtained time sequence clue is clearer and more robust, and the generated text prompt is used as an intermediate bridge in the text domain, which guides the formation of a bidirectional information transmission pipeline of visual-text prompt-label, which is easy to match. The embodiment realizes the bidirectional bridging of the visual-text domain of the CLIP model in the behavior recognition task, and fully plays the potential of the VL model in the video behavior recognition task. In particular, the method provided by the embodiment not only realizes the most advanced performance in the full supervision task, but also performs well in the zero-shot and few-shot tasks.

Claims

1. A behavior recognition method for bidirectional domain closure, characterized in that, The method comprises the following steps: obtaining a sequence of RGB image frames to be recognized and a category text thereof; inputting the sequence of RGB image frames to be recognized into a text prompt generation module to obtain a generated text; inputting the sequence of RGB image frames to be recognized, the category text thereof and the generated text into a behavior recognition model to obtain an action category of the sequence of RGB image frames to be recognized, the behavior recognition model comprising: a first text encoder module configured to encode the generated text to obtain a generated text feature; an image encoder module configured to generate a first image feature according to the sequence of RGB image frames to be recognized; a time sequence encoder module integrated with a motion prompt generation module, configured to generate a second image feature according to the sequence of RGB image frames to be recognized under the guidance of the motion prompt generation module; wherein the motion prompt generation module, for each layer of image embedding features of the time sequence encoder, splices image embedding features of adjacent frames to obtain paired frame features, uses a convolution module to extract the similarity of the paired frame features to obtain a similarity feature, reduces the dimension of the similarity feature through weight extraction to obtain a weight of the similarity feature, and uses the weight of the similarity feature to weight the image embedding features to obtain a motion subject summary of each RGB image frame, and uses a one-dimensional convolution module to interact the motion subject summaries of each frame to obtain a motion subject prompt of the RGB image frames; a second text encoder module configured to encode the category text to obtain a category text feature; an interaction module configured to encode the generated text feature and the first image feature to obtain a more distinctive image feature with text modal information introduced; a fusion module configured to fuse the more distinctive image feature with the second image feature to obtain a fusion feature; a similarity calculation module configured to calculate the similarity between the category text feature and the fusion feature.

2. The method of claim 1, wherein the behavior recognition method is bidirectional domain closure. The inputting of the sequence of RGB image frames to be recognized into the text prompt generation module to obtain the generated text specifically comprises: generating a text label according to the sequence of RGB image frames to be recognized using a video description generation model; generating an action keyword according to all possible labels given and the generated text label provided, and providing the action keyword as part of an instruction to the text prompt generation module; providing several examples and giving a specification instruction of a system prompt, and requiring the text prompt generation module to generate a text without noise according to the provided text label and action keyword to obtain the generated text.

3. The method of claim 1, wherein the behavior recognition method is bidirectional domain agnostic. The first text encoder module comprises: a first tokenizer configured to encode the generated text to obtain a word vector of the generated text; a generated text encoder adopting a text encoder structure of a CLIP model, configured to encode the word vector of the generated text to obtain the generated text feature; The generated text encoder participates in fine-tuning of a first period during training.

4. The method of claim 1, wherein the behavior recognition method is bidirectional domain closure. The image encoder module comprises: an image conversion unit configured to convert the sequence of RGB image frames to be recognized into an image embedding sequence; An image encoder, a visual encoder adopting a CLIP model, is configured to encode the image embedding sequence to obtain first image features; The image encoder keeps all parameters frozen during training.

5. The bidomain reconciled behavior identification method of claim 1, wherein, The temporal encoder module comprises: An image conversion unit is configured to convert the sequence of RGB image frames to be recognized into an image embedding sequence; A temporal encoder, a pre-trained visual encoder adopting a CLIP model, is configured to encode the image embedding sequence under the guidance of the motion prompt generation module to obtain second image features.

6. The bidomain reconciled behavior identification method of claim 1, wherein, The fusion module is composed of a standard multi-head self-attention mechanism, a feedforward neural network, and an average pooling layer.

7. The bidomain reconciled behavior identification method of claim 1, wherein, The second text encoder module comprises: A second word segmenter is configured to encode the category text to obtain word vectors of the category text; A category text encoder, a text encoder structure adopting a CLIP model, is configured to encode the word vectors of the category text to obtain category text features; The category text encoder keeps all parameters frozen during training.

8. The bidomain reconciled behavior identification method of claim 1, wherein, The interaction module performs cross-attention calculation on the patch level by taking the generated text features as a query and the first image features as keys and values, extracts local visual region features strongly related to the text prompt, and obtains more distinctive image features with the introduction of text modal information.

9. The bidomain reconciled behavior identification method of claim 1, wherein, The behavior recognition model adopts a two-stage training method, wherein the first stage fine-tunes the first text encoder module, the interaction module, and the fusion module, and the second stage fine-tunes the temporal encoder module and the fusion module.

10. The bidomain healing behavioral identification method of claim 9, wherein, The total loss of the behavior recognition model in the first stage training includes a cross-entropy loss and a cosine loss of the time series encoder module, and the total loss in the second stage training is a cross-entropy loss, wherein the cross-entropy loss is: where L entro is the cross-entropy loss, sim() is the similarity calculation, v i is the feature representation of the i-th sample, c i is the feature representation of the correct class of the i-th sample, τ is a temperature hyperparameter used to scale the similarity, and c j is the feature representation of the class j.

Images