Bidirectional domain bridging behavior recognition method

Abstract

The invention relates to a behavior recognition method for bidirectional domain bridging. The method comprises the following steps: acquiring a to-be-recognized RGB image frame sequence and a category text of the to-be-recognized RGB image frame sequence; inputting the to-be-identified RGB image frame sequence into a text prompt generation module to obtain a generative text; and inputting the RGB image frame sequence to be identified, the category text of the RGB image frame sequence and the generative text into a behavior identification model to obtain an action category of the video to be identified, the behavior recognition model comprises a first text encoder module, an image encoder module, a time sequence encoder module, a second text encoder module, an interaction module, a fusion module and a similarity calculation module. According to the method, the domain gap between the pre-training model and the behavior recognition task can be effectively reduced, so that the applicability and the performance of the CLIP model in the behavior recognition field are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a behavior recognition method with two-way domain bridging. Background Art

[0002] Action recognition is one of the most basic but also the most challenging tasks in video understanding. It plays a vital role in many visual applications, such as micro-video recommendation, sports video analysis, autonomous driving, etc. With the rapid increase in the number of videos on the Internet, large-scale action recognition has become a critical problem that needs to be solved urgently. A powerful action recognition method should be able to understand the semantic information in the video and even automatically describe the content, such as human actions and complex events, and achieve accuracy comparable to that of humans.

[0003] In the past few years, significant progress has been made in action recognition based on convolutional neural networks and Transformers. However, most existing works follow the setting of closed-set learning, where all categories are pre-defined. This approach is impractical for many real-world applications, such as automatic labeling of online videos, where there is no information about the categories of new videos during training. Therefore, it is very challenging for methods with known categories to train classifiers for recognizing unseen or unfamiliar categories.

[0004] With the development of large pre-trained models, recent studies in image classification have shown that bootstrapping large-scale contrastive language-image pretraining (CLIP) models to solve classification tasks can significantly enhance the generalization ability of existing models. The CLIP model is a type of vision language (VL) model that learns joint representations from web-scale text and image pairs, and then aligns the representations to a shared embedding space via simple noise contrastive learning. However, since the CLIP model is pre-trained on large-scale image-text data, there is a significant domain gap between its pre-training task and the downstream video action recognition task, and directly applying the CLIP model will lead to poor performance. Therefore, in order to fully exploit the potential of pre-trained models in the field of action recognition, it is necessary to domain adapt the CLIP model from both visual and text aspects to bridge the domain gap.

[0005] In order to solve the problem of domain adaptation from images to videos, common methods currently include direct end-to-end fine-tuning and extracting temporal cues through simple post-fusion. Post-fusion methods usually use inter-frame attention mechanisms or frame-level embedding fusion to generate video representations. In addition, some methods try to introduce carefully designed learnable components (such as self-attention layers, visual decoders, or visual cue modules) into the CLIP encoder. However, when these methods process disruptive images in videos, the model's attention tends to be biased towards the background and not enough attention is paid to the moving subject, which restricts the overall performance.

[0006] In addition, there are some works that attempt to address the domain gap problem between the network-scale narrative text used in CLIP pre-training and the concise behavior labels in downstream tasks, such as generating richer semantic details from behavior labels to make up for the lack of semantic information in behavior labels. However, the inventors of the present invention have noticed that there are limitations to the method of generating prompts only from category names: the generated category prompts may not be fully applicable to each video sample, and it is difficult to cover all the rich details related to the action in the video in detail. At the same time, such methods often only focus on bridging the semantic domain from labels to descriptions, ignoring visual domain adaptation, which leads to insufficient temporal expression capabilities of the model and difficulty in achieving robust performance. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a bidirectional domain bridging behavior recognition method, which can effectively narrow the domain gap between the pre-trained model and the behavior recognition task, thereby significantly improving the applicability and performance of the CLIP model in the field of behavior recognition.

[0008] The technical solution adopted by the present invention to solve the technical problem is: to provide a behavior recognition method for bidirectional domain bridging, comprising the following steps:

[0009] Get the RGB image frame sequence to be identified and its category text;

[0010] Inputting the to-be-recognized RGB image frame sequence into a text prompt generation module to obtain a generated text;

[0011] The RGB image frame sequence to be identified and its category text and the generated text are input into a behavior recognition model to obtain the action category of the video to be identified. The behavior recognition model includes:

[0012] A first text encoder module, used for encoding the generated text to obtain generated text features;

[0013] An image encoder module, configured to generate a first image feature according to the RGB image frame sequence to be identified;

[0014] A temporal encoder module, integrated with a motion cue generation module, configured to generate a second image feature according to the RGB image frame sequence to be identified under the guidance of the motion cue generation module;

[0015] A second text encoder module is used to encode the category text to obtain category text features;

[0016] An interaction module, configured to encode the generative text feature and the first image feature to obtain a more distinguishing image feature that introduces text modal information;

[0017] A fusion module, used for fusing the more distinguishing image feature with the second image feature to obtain a fused feature;

[0018] The similarity calculation module is used to calculate the similarity between the category text feature and the fusion feature.

[0019] The step of inputting the to-be-recognized RGB image frame sequence into a text prompt generation module to obtain a generated text specifically includes:

[0020] Generate text annotations based on the RGB image frame sequence to be identified using a video description generation model;

[0021] Generate an action keyword based on all possible tags given and the generated text annotations provided, and provide the action keyword as part of the instruction to the text prompt generation module;

[0022] Several examples are provided, and a standard instruction of a system prompt is given, requiring the text prompt generation module to generate noise-free text according to the provided text annotations and action keywords to obtain a generated text.

[0023] The first text encoder module comprises:

[0024] A first word segmenter, used to encode the generated text to obtain a word vector of the generated text;

[0025] A generative text encoder, which uses the text encoder structure of the CLIP model, is used to encode the word vectors of the generative text to obtain generative text features;

[0026] The generative text encoder participates in fine-tuning in a first phase during training.

[0027] The image encoder module comprises:

[0028] An image conversion unit, used for converting the to-be-recognized RGB image frame sequence into an image embedding sequence;

[0029] An image encoder, using a visual encoder of a CLIP model, for encoding the image embedding sequence to obtain a first image feature;

[0030] The parameters of the image encoder remain frozen throughout the training process.

[0031] The timing encoder module comprises:

[0032] An image conversion unit, used for converting the to-be-recognized RGB image frame sequence into an image embedding sequence;

[0033] A temporal encoder, using a pre-trained visual encoder of the CLIP model, for encoding the image embedding sequence under the guidance of the motion cue generation module to obtain a second image feature;

[0034] The motion prompt generation module inputs image embedding features of each layer of the temporal encoder, splices the image embedding features of adjacent frames to obtain paired frame features, uses a convolution module to extract the similarity of paired frame features to obtain similarity features, reduces the dimension of the similarity features through weight extraction to obtain the weight of the similarity features, uses the weight of the similarity features to weight the image embedding features to obtain a summary of the motion subject of each RGB image frame, interacts the motion subject summary of each frame through a one-dimensional convolution module, and obtains a motion subject prompt for the RGB image frame.

[0035] The fusion module consists of a standard multi-head self-attention mechanism, a feedforward neural network and an average pooling layer.

[0036] The second text encoder module comprises:

[0037] A second word segmenter is used to encode the category text to obtain a word vector of the category text;

[0038] A category text encoder adopts the text encoder structure of the CLIP model to encode the word vector of the category text to obtain the category text feature;

[0039] The parameters of the categorical text encoder remain frozen throughout the training process.

[0040] The interactive module uses the generative text features as queries and the first image features as keys and values ​​to perform cross-attention calculations at the patch level, extracts local visual area features that are strongly correlated with text prompts, and obtains more discriminative image features that introduce text modal information.

[0041] The behavior recognition model adopts a two-stage training method, wherein the first stage fine-tunes the first text encoder module, the interaction module and the fusion module, and the second stage fine-tunes the timing encoder module and the fusion module.

[0042] The total loss of the behavior recognition model during the first stage training includes the cross entropy loss and the cosine loss of the temporal encoder module, and the total loss during the second stage training is the cross entropy loss, and the cross entropy loss is: Among them, L entro is the cross entropy loss, sim() is the similarity calculation, v i is the feature representation of the i-th sample, c i is the feature representation of the correct category of the i-th sample, τ is the temperature hyperparameter used to scale the similarity, and c j is the feature representation of category j.

[0043] Beneficial Effects

[0044] Due to the adoption of the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art: the present invention enables the model to focus more on the movement itself based on the motion subject prompt in the visual domain, and the obtained timing clues are clearer and more robust. In the text domain, the generative text prompt is used as an intermediate bridge to guide the formation of a two-way information transmission pipeline of visual-text prompt-label, which is easy to match. The present invention realizes the two-way bridging of the visual-text domain of the CLIP model in the behavior recognition task, and gives full play to the potential of the VL model in the video behavior recognition task. In particular, the method provided by the present invention not only achieves the most advanced performance in the fully supervised task, but also performs well in zero-shot and few-shot tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a behavior recognition method for bidirectional domain bridging according to an embodiment of the present invention;

[0046] Figure 2 is a generation flow chart of a text prompt generation module in an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of the structure of a behavior recognition model in an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of the structure of a motion prompt generating module in an embodiment of the present invention;

[0049] Figure 5 It is a schematic diagram of the structure of an image encoder and a text encoder in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0051] The embodiments of the present invention relate to a behavior recognition method for bidirectional domain bridging, such as Figure 1 As shown, the following steps are included:

[0052] Step 1: Obtain the RGB image frame sequence to be identified and its category text.

[0053] Step 2: input the RGB image frame sequence to be identified into a text prompt generation module to obtain generated text.

[0054] In order to make up for the domain gap in text when the CLIP model is migrated to the downstream behavior recognition task, and to solve the problem of inaccurate matching between prompts and video content in existing text prompt learning methods, this embodiment obtains descriptive text that is strongly related to the RGB image frame sequence to be recognized as a prompt to guide the bridging of the text domain through a text prompt generation module.

[0055] Considering that obtaining manual video annotations is a costly and labor-intensive task, this embodiment uses a zero-shot video description generation model to automatically generate annotations. The video description generation model used can be a video extension version of the image-level ZeroCap model based on GPT-2, which can directly obtain descriptive annotations (captions) from unseen downstream videos without additional training. This embodiment first generates 20 captions at one time based on the RGB image frame sequence to be identified, sorts the obtained captions from high to low using the CLIP score, and selects the caption with the highest score as the text prompt input into the text prompt generation module.

[0056] However, the method of obtaining content prompts directly from the video description generation model will produce noisy and ethically problematic texts due to the hallucination tendency of the generative LLM. In order to generate high-quality text prompts, this embodiment further adopts the form of instruction following data and makes full use of the prior semantic knowledge of LLM to filter the captions generated by the video description generation model.

[0057] The specific process is as follows Figure 2 The text prompt generation module of the present invention is based on GPT-3.5. v Generated caption X c and its associated action keyword X k, create a set of canonical instructions X i , instructing the assistant to filter data and filter out data related to the action keyword X k Return the generated text C after irrelevant content p Used to guide cross-modal interaction. Specifically, the data filtering steps are as follows:

[0058] Generate corresponding action keyword X k , through GPT-3.5 by "a description as [X c ] may have the same label [X] as the one in the label set [K] k ] related to”. Then, the action keyword X k Provided as part of the instructions to GPT-3.5 to guide it to limit the filtered video description to the corresponding category range and reduce noise unrelated to the action;

[0059] Provide a few examples X e , which makes it easier for GPT-3.5 to understand and generate similar high-quality text;

[0060] Give a system prompt standard instruction X i , requiring GPT-3.5 to provide caption X c and action keyword X k Generate noise-free generative text C p ;

[0061] Input the above content to GPT-3.5, and GPT-3.5 returns the generated text C p .

[0062] In this embodiment, the generative text prompts obtained by the text prompt generation module are closely centered around the behavior labels and have rich video-related descriptive semantic details, serving as an intermediate bridge to bridge the domain gap between descriptive text and concise labels.

[0063] Step 3: input the RGB image frame sequence to be identified, its category text and the generated text into a behavior recognition model to obtain the action category of the video to be identified.

[0064] like Figure 3 As shown, the behavior recognition model in this embodiment includes:

[0065] A first text encoder module, used for encoding the generated text to obtain generated text features;

[0066] An image encoder module, configured to generate a first image feature according to the RGB image frame sequence to be identified;

[0067] A temporal encoder module, integrated with a motion cue generation module, configured to generate a second image feature according to the RGB image frame sequence to be identified under the guidance of the motion cue generation module;

[0068] A second text encoder module is used to encode the category text to obtain category text features;

[0069] An interaction module, configured to encode the generative text feature and the first image feature to obtain more distinguishing image information that introduces text modal information;

[0070] A fusion module, used for fusing the visual-textual prompt-label information with the second image feature to obtain a fused feature;

[0071] The similarity calculation module is used to calculate the similarity between the category text feature and the fusion feature.

[0072] The first text encoder module includes: a first word segmenter, which is used to encode the generative text to obtain the word vector of the generative text; a generative text encoder, which adopts the text encoder structure of the CLIP model, is used to encode the word vector of the generative text to obtain the generative text feature. The second text encoder module includes: a second word segmenter, which is used to encode the category text to obtain the word vector of the category text; a category text encoder, which adopts the text encoder structure of the CLIP model, is used to encode the word vector of the category text to obtain the category text feature.

[0073] Among them, both the generative text encoder and the category text encoder adopt the text encoder structure of the original CLIP model, which includes L c The overall structure of the Transformer encoder layer is as follows: Figure 5 As shown in Figure 2. The word segmenter encodes the corresponding input text to obtain the word vector. The parameters of the categorical text encoder remain frozen throughout the process, and the generative text encoder participates in the fine-tuning of the first stage.

[0074] After the last layer of Transformer encoder of the generative text encoder, the generative text encoder directly outputs the generated text feature e p =[CLS,E p1 ,E p2 ,…,E pN ] and fused with the first image feature generated by the image encoder module. The category text encoder directly extracts the class token in the category text feature as the text representation, which is used to calculate the similarity between the fusion feature output by the fusion module.

[0075] The image encoder module in this embodiment includes: an image conversion (i.e., image tokenization) unit, which is used to convert the RGB image frame sequence to be identified into an image embedding sequence; an image encoder, which uses a visual encoder of the CLIP model, which is used to encode the image embedding sequence to obtain a first image feature; the parameters of the image encoder remain frozen throughout the training process.

[0076] The image conversion unit follows the rules of ViT. For a V∈R T×H×W×3 The video input (here H and W represent the height and width resolution of the picture, respectively, and T represents the number of video frames) is converted into an image embedding sequence after passing through the image conversion unit. The t-th frame embedding sequence consists of N non-overlapping patches and [CLS] tokens, expressed as: Among them, d is the feature dimension. Finally, in order to maintain the relative spatial position relationship of each node in the image feature sequence, a learnable position encoding is introduced and added to the frame embedding sequence.

[0077] The main function of the image encoder is to combine with the interaction module to overcome the generalization problem caused by the text gap. It adopts the visual encoder of the CLIP model. By freezing the visual encoder of the CLIP model, it provides visual knowledge that is fine-grained and semantically aligned with the generative text prompts, so that vision and text are associated in a richer semantic space, making it easier to anchor visual content from text.

[0078] The input of the image encoder is the image embedding sequence v *,0 ∈R T× (N+1) ×d , after L layers of Transformer encoding, the output is v *,L ∈R T× (N+1)×d, v *,L Together with the output of the generative text encoder, it is fed into the interactive module to obtain the final frame representation. Figure 5 As shown, the specific structure of the image encoder is the same as that of the generative text encoder.

[0079] The interactive module in this embodiment uses the generative text generated by the first text encoder module as an intermediate bridge to form a visual-text prompt-label information flow, thereby achieving text domain bridging.

[0080] In the visual-textual prompt direction, this embodiment uses semantically rich textual prompts to extract necessary visual details related to the behavior label. The interactive module encodes the video frame and the generated textual prompt respectively through the frozen CLIP image encoder and the fine-tuned generative text encoder, and converts the generated text features into As the query query, the first image feature As the key and value, cross attention calculation is performed at the patch level to extract the local visual area features that are strongly related to the text prompt, expressed as: Finally, the class token of the frame embedding feature after interaction is taken as the frame representation

[0081] In the text prompt-label direction, this embodiment Add on the basis of Multi-frame fusion is performed to extract video representation, which is used to calculate similarity with the category features encoded by the text encoder. Text semantic information related to the label is gradually introduced into the video representation, shortening the distance between the video and the label. Through the above strategy, this implementation method uses text prompts as an intermediate bridge, and through the interactive process guided by generative text prompts, a two-way information flow pipeline of vision-text prompts-labels is constructed, making the video-label pair more closely connected in the latent space.

[0082] The temporal encoder module in this embodiment includes: an image conversion unit, which is used to convert the RGB image frame sequence to be identified into an image embedding sequence; a temporal encoder, which uses a pre-trained visual encoder of the CLIP model, and is used to encode the image embedding sequence under the guidance of the motion cue generation module to obtain a second image feature.

[0083] Among them, the image conversion unit is the same as the image conversion unit in the image encoding module, and will not be repeated here.

[0084] The motion hint generation module in this embodiment is integrated into the timing encoder, and the specific generation process is as follows:

[0085] In the temporal encoder, the embedding feature of the lth layer is represented as: For each layer of input image embedding features, the image embedding features of adjacent frames are concatenated to obtain paired frame features Use the convolution module to extract the similarity of paired frame features and obtain similarity features

[0086]

[0087] Similarity Features The weight extraction module is used to reduce the dimension and obtain the weight W of the similarity feature. m,l , weight the image embedding features using the weight of the similarity feature to obtain the motion subject hint of each RGB image frame Then, a cosine loss constraint is imposed on the summary of the moving subject in consecutive frames, guiding the weight extraction module to optimize the generated weight mask so that the weights in the similar regions of adjacent frames are large and the weights in the different regions are small. Because similar regions between frames are more likely to correspond to the moving subject in the video, the model's ability to focus on the moving subject is enhanced. Among them, the weights of the similarity features and the moving subject hints can be expressed as:

[0088]

[0089]

[0090] For the summary of the motion subject in each frame, this embodiment further adopts an adapter structure, first reducing the dimension and then increasing the dimension, using a 1D convolution module to transfer the subject motion information between frames, and obtain the motion subject prompt corresponding to each frame. In general, the structure of the motion subject prompt generation module is as follows Figure 4 shown.

[0091] The temporal encoder uses a pre-trained visual encoder of the CLIP model, which includes L layers of Transformer encoding layers, which are mainly composed of a self-attention mechanism and a feedforward neural network for extracting image feature information. In particular, in order to extract temporal information that is robust to background interference, this embodiment inserts a motion subject prompt generation module at the front end of each layer of Transformer encoding layer, relying on the embedded features of the previous layer to generate motion subject prompts for guiding the attention distribution of the current layer.

[0092] Specifically, the calculation of the lth layer of the temporal encoder can be expressed as:

[0093]

[0094] in, The output of the motion subject prompt generation module is Frame embedding sequence added to each frame At the end, after completing the self-attention guidance of the lth layer of the temporal encoder, the motion subject prompt is discarded to obtain the frame embedding sequence output of the lth layer Finally, the category token of the frame embedding sequence of the Lth layer is used as the output of the temporal encoder.

[0095] The temporal encoder module in this embodiment uses motion subject cues to dominate the global and local temporal encoding process, avoiding interference caused by non-moving subjects in the picture, so that more robust temporal cues can be obtained.

[0096] The fusion module in this implementation is designed to efficiently fuse the T-frame embedding sequence output by the interaction module and the temporal encoder module, and to generate a video representation that is more suitable for the recognition task requirements by integrating the feature representations of each frame. The fusion module is composed of a standard multi-head self-attention mechanism, a feedforward neural network, and an average pooling layer to fully capture the global correlation between frames, thereby improving the expressiveness of the video representation.

[0097] The similarity module in this embodiment can adopt a cosine similarity module, and the similarity score s V The calculation can be expressed by the following formula:

[0098] s V =sim(v,c)

[0099] Among them, sim() represents cosine similarity calculation.

[0100] During the training process, this implementation uses a symmetric cross entropy loss to maximize the similarity between matching video-label pairs and minimize the similarity between non-matching sample pairs. The cross entropy loss is expressed as:

[0101]

[0102] Among them, L entro is the cross entropy loss, sim() is the similarity calculation, τ is the temperature hyperparameter used to scale the similarity, i represents the sample index in the training data, and v i is the feature representation of the i-th sample. i is the feature representation of the correct category of the i-th sample, j represents the index of the category, traverses all categories, c j is the feature representation of category j.

[0103] In order to avoid over-reliance on input text prompts, this implementation adopts a two-stage training method. The first stage fine-tunes the first text encoder and the interaction module, and the second stage fine-tunes the temporal encoder module. The fusion module participates in the fine-tuning during the entire training process. The total loss in the first stage is divided into two parts. The first part is the cross entropy loss mentioned above, and the second part is the cosine loss in the temporal encoder module. The total loss is expressed as follows:

[0104] L=L entro +L cos′

[0105] The second stage cross entropy loss L entro Only the cross entropy loss between the second image feature and the category text feature output by the temporal encoder module is calculated, focusing on the fine-tuning of the temporal encoder. During the inference process, the similarity score between the fused feature and the text of each category is calculated, and the category with the highest similarity score is taken as the final classification result.

[0106] It is not difficult to find that the method of this embodiment makes the model more focused on the movement itself based on the motion subject prompt in the visual domain, and the obtained timing clues are clearer and more robust. In the text domain, the generative text prompt is used as an intermediate bridge to guide the formation of a two-way information transmission pipeline of visual-text prompt-label, which is easy to match. This embodiment realizes the two-way bridging of the visual-text domain of the CLIP model in the behavior recognition task, and gives full play to the potential of the VL model in the video behavior recognition task. In particular, the method provided by the present invention not only achieves the most advanced performance in the fully supervised task, but also performs well in zero-shot and few-shot tasks.

Claims

1. A bidirectional domain bridging behavior recognition method, characterized in that: The following steps are involved: Get the RGB image frame sequence to be identified and its category text; Inputting the to-be-recognized RGB image frame sequence into a text prompt generation module to obtain a generated text; The RGB image frame sequence to be identified and its category text and the generated text are input into a behavior recognition model to obtain the action category of the video to be identified. The behavior recognition model includes: A first text encoder module, used for encoding the generated text to obtain generated text features; An image encoder module, configured to generate a first image feature according to the RGB image frame sequence to be identified; A temporal encoder module, integrated with a motion cue generation module, configured to generate a second image feature according to the RGB image frame sequence to be identified under the guidance of the motion cue generation module; A second text encoder module is used to encode the category text to obtain category text features; An interaction module, configured to encode the generative text feature and the first image feature to obtain a more distinguishing image feature that introduces text modal information; A fusion module, used for fusing the more distinguishing image feature with the second image feature to obtain a fused feature; The similarity calculation module is used to calculate the similarity between the category text feature and the fusion feature.

2. The behavior recognition method for bidirectional domain bridging according to claim 1 is characterized in that: The step of inputting the to-be-recognized RGB image frame sequence into a text prompt generation module to obtain a generated text specifically includes: Generate text annotations based on the RGB image frame sequence to be identified using a video description generation model; Generate an action keyword based on all possible tags given and the generated text annotations provided, and provide the action keyword as part of the instruction to the text prompt generation module; Several examples are provided, and a standard instruction of a system prompt is given, requiring the text prompt generation module to generate noise-free text according to the provided text annotations and action keywords to obtain a generated text.

3. The behavior recognition method for bidirectional domain bridging according to claim 1 is characterized in that: The first text encoder module comprises: A first word segmenter, used to encode the generated text to obtain a word vector of the generated text; A generative text encoder, which uses the text encoder structure of the CLIP model, is used to encode the word vectors of the generative text to obtain generative text features; The generative text encoder participates in fine-tuning in a first phase during training.

4. The behavior recognition method for bidirectional domain bridging according to claim 1 is characterized in that: The image encoder module comprises: An image conversion unit, used for converting the to-be-recognized RGB image frame sequence into an image embedding sequence; An image encoder, using a visual encoder of a CLIP model, for encoding the image embedding sequence to obtain a first image feature; The parameters of the image encoder remain frozen throughout the training process.

5. The behavior recognition method of two-way domain bridging according to claim 1, characterized in that: The timing encoder module comprises: An image conversion unit, used for converting the to-be-recognized RGB image frame sequence into an image embedding sequence; A temporal encoder, using a pre-trained visual encoder of the CLIP model, for encoding the image embedding sequence under the guidance of the motion cue generation module to obtain a second image feature; The motion prompt generation module inputs image embedding features of each layer of the temporal encoder, splices the image embedding features of adjacent frames to obtain paired frame features, uses a convolution module to extract the similarity of paired frame features to obtain similarity features, reduces the dimension of the similarity features through weight extraction to obtain the weight of the similarity features, uses the weight of the similarity features to weight the image embedding features to obtain a summary of the motion subject of each RGB image frame, interacts the motion subject summary of each frame through a one-dimensional convolution module, and obtains a motion subject prompt for the RGB image frame.

6. The behavior recognition method for bidirectional domain bridging according to claim 1, characterized in that: The fusion module consists of a standard multi-head self-attention mechanism, a feedforward neural network and an average pooling layer.

7. The behavior recognition method for bidirectional domain bridging according to claim 1, characterized in that: The second text encoder module comprises: A second word segmenter is used to encode the category text to obtain a word vector of the category text; A category text encoder adopts the text encoder structure of the CLIP model to encode the word vector of the category text to obtain the category text feature; The parameters of the categorical text encoder remain frozen throughout the training process.

8. The behavior recognition method for bidirectional domain bridging according to claim 1, characterized in that: The interactive module uses the generative text features as queries and the first image features as keys and values ​​to perform cross-attention calculations at the patch level, extracts local visual area features that are strongly correlated with text prompts, and obtains more discriminative image features that introduce text modal information.

9. The behavior recognition method of two-way domain bridging according to claim 1, characterized in that: The behavior recognition model adopts a two-stage training method, wherein the first stage fine-tunes the first text encoder module, the interaction module and the fusion module, and the second stage fine-tunes the timing encoder module and the fusion module.

10. The behavior recognition method of bidirectional domain bridging according to claim 9, characterized in that: The total loss of the behavior recognition model during the first stage training includes the cross entropy loss and the cosine loss of the temporal encoder module, and the total loss during the second stage training is the cross entropy loss, and the cross entropy loss is: Among them, L entro is the cross entropy loss, sim() is the similarity calculation, v i is the feature representation of the i-th sample, c i is the feature representation of the correct category of the i-th sample, τ is the temperature hyperparameter used to scale the similarity, and c j is the feature representation of category j.

Images