A zero-shot action recognition method based on multimodal cue learning

Abstract

The present invention discloses a zero-shot behavior recognition method based on multimodal cue learning, which expands CLIP into a fine-grained multimodal spatiotemporal understanding model, and simultaneously improves the specificity and universality of multimodal spatiotemporal understanding through plug-and-play frame-level multi-scale temporal cue (MTP) and spatiotemporal semantic enhancement. The proposed multimodal dynamic cue framework is initialized using the pre-trained weights of the CLIP model and freezes the weights of the text encoder. For the visual encoder, refined spatiotemporal visual features are obtained by inserting additional learnable multi-scale temporal cue modules into different layers of the visual encoder. For semantic text, the understanding of different behavior categories is improved by expanding and refining the semantic knowledge represented by the category. The method of the present invention shows good performance compared with mainstream methods.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to a zero-sample behavior recognition method based on multimodal cue learning. Background Art

[0002] Zero-shot behavior recognition aims to identify new behaviors not seen during model training. By understanding video content and establishing semantic relationships between behavior categories, the model can identify categories not seen during training. Zero-shot behavior recognition can alleviate the problems of scarce labeled behavior data or severely imbalanced data distribution, providing important support for the development of intelligent video understanding and artificial intelligence technology.

[0003] In related technologies, the training paradigm for zero-shot action recognition models primarily consists of three parts: visual feature extraction, semantic feature extraction, and feature mapping. The general framework followed by zero-shot action recognition is to project the visual embedding of video samples and the semantic embedding of corresponding category labels into the same embedding space for association learning. Through auxiliary semantic information and the association between corresponding categories, the model is able to transfer knowledge from seen to unseen classes in the embedding space. Traditional methods typically require the design of some hand-crafted features or the use of readily available pre-trained feature extractors, followed by knowledge representation or transfer learning methods, focusing primarily on modeling visual and semantic associations. Many recent works leverage the efficient visual-language alignment capabilities of the Multimodal Foundational Image Processing (CLIP) model to achieve universal action recognition through additional temporal modeling. Furthermore, some work focuses on using large language models to expand CLIP's expertise to promote zero-shot generalization capabilities. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a zero-shot behavior recognition method based on multimodal cue learning, which expands CLIP into a fine-grained multimodal spatiotemporal understanding model, and simultaneously improves the specificity and universality of multimodal spatiotemporal understanding through plug-and-play frame-level multi-scale temporal cues (MTP) and spatiotemporal semantic enhancement. The proposed multimodal dynamic cue framework is initialized using the pre-trained weights of the CLIP model and freezes the weights of the text encoder. For the visual encoder, refined spatiotemporal visual features are obtained by inserting additional learnable multi-scale temporal cue modules into different layers of the visual encoder. For semantic text, the understanding of different behavior categories is improved by expanding and refining the semantic knowledge represented by the categories. The method of the present invention shows good performance compared with mainstream methods.

[0005] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0006] Step 1: For a given T-frame video clip Where C, H, and W represent the number of channels, height, and width of the video frame, respectively; each video frame is divided into N non-overlapping patches through the patch embedding layer. Where N = HW / P 2 , the spatial size of each patch is P×P;

[0007] This process also includes the operation of splicing additional category embeddings to obtain the input of the first layer of the encoder corresponding to the tth frame D represents the embedding dimension; let Represents the embedded features input to the lth layer visual encoder, and obtains the input of the l+1th layer

[0008]

[0009] Among them, each visual encoding layer first passes through a layer normalization LN and a multi-head self-attention mechanism MHSA in a skip connection manner, and then passes through layer normalization and a feedforward network MLP;

[0010] Step 2: The multi-scale temporal hint module MTP first uses the window shift mask WSM and multi-scale temporal cross attention MS-TCA to respectively remove the category representation of the query obtained by multi-head self-attention. key Sum The embedding of is calculated:

[0011]

[0012] in, is the original mapping weight from the l-th layer visual encoder, Represents the time scale δ i The corresponding intermediate features, N′ and S, respectively represent the number of visible representations after the window shift mask WSM and the number of time scales of the multi-scale temporal attention MS-TCA;

[0013] Step 3: The global information of visible patches is modeled through layer normalization and feed-forward networks to enhance the interaction of different spatial locations in each frame:

[0014]

[0015] Step 4: Use the weight matrix W δ ∈R (S×N′)×P Will The mapping obtains a frame-specific timing prompt of length P, which represents the spatiotemporal information contained in the video:

[0016]

[0017] The timing prompt and Concatenate as input to the next layer Multi-head self-attention is respectively For query representation, For the representation of keys and values;

[0018] Step 5: For semantic text, use spatiotemporal text enhancement to obtain refined text hints C st ={C s ,C t}, after passing through the text encoder, we get the text embedding c st ={c s ,c t};

[0019] Step 6: According to and c st Calculate the fine-grained alignment scores of all visible categories, and optimize the learnable parameters of the model in multi-scale temporal cues by maximizing the score of the correct category while minimizing the scores of other categories; in the inference stage, the recognition results of the model on unknown behavior categories are compared with the original CLIP model according to and c s The obtained recognition results are aggregated for decision making to obtain the final recognition result.

[0020] Preferably, the multi-scale temporal prompt MTP is as follows:

[0021] The proposed multi-scale temporal cueing consists of two processing modules: (1) the window shift mask (WSM) module aims to mask the patches in each processed frame and discard patches at a certain ratio to reduce the computational cost of the subsequent cross-attention mechanism; (2) the multi-scale temporal cross attention (MS-TCA) aims to process the remaining patches after masking, and enables the self-attention mechanism to jointly focus on spatial and temporal positions by performing channel mixing at multiple time scales;

[0022] (1) Window shift mask WSM:

[0023] The window shift mask sequentially discards patches at different spatial positions by shifting the shape of the window mask of 2*2 spatial size in the time dimension; the masking process is expressed as:

[0024] M t,i =φ(M t-1,i |M 1:t-2,i ),i∈{1,…,R}

[0025] Among them, M t,irepresents the mask map of the t-th frame in the i-th multi-scale temporal prompt module, φ(·|·) is a periodic function that generates a mask map based on the mask frames from 1 to t-1; then, the mask map M t,i Applied to the query, key, and value representations to produce the corresponding representations of the window shift:

[0026]

[0027] in, Represent N′ spatial queries, and feed the key and value representations of the frame into the l-th layer visual encoder respectively;

[0028] (2) Multi-scale Temporal Cross Attention MS-TCA:

[0029] The key after window shift mask is represented as k′ t First, the time series information is fused by indexing the channel information before and after the moment; Indicated by k′ t Index from d s to d e The mixing process corresponding to the time scale δ is expressed as:

[0030]

[0031] Among them, d δ =γ·D is a hyperparameter for mixing index channels;

[0032] Introducing multiple time scales Let k′ t All bonds with a range of δ represent a mixture, i.e. from arrive

[0033] When δ = 2, the multi-scale mixed bond representation is obtained

[0034]

[0035] Without introducing any additional parameters, the calculation of multi-scale temporal cross attention is expressed as:

[0036]

[0037] Preferably, the spatiotemporal text enhancement is as follows:

[0038] The spatiotemporal text enhancement includes a two-stage prompting process, namely the construction of behavioral semantic knowledge graph and semantic spatiotemporal decoupling;

[0039] (1) Behavioral semantic knowledge graph (ASKG);

[0040] ASKG decomposes categories from spatial representations and temporal dynamics into semantic concepts and their relationships. ASKG abstracts different behavior categories into a graph structure by representing behaviors, related objects, and sub-behaviors as nodes in a knowledge graph and their relationships as edges. Finally, by aggregating the outputs of different behavior categories, ASKG brings together structured representations of behavioral semantics and relationships at different granularities.

[0041] (2) semantic spatiotemporal decoupling (SSTD);

[0042] Semantic spatiotemporal decoupling obtains extended text prompts by decomposing text prompts into spatial and temporal levels; then the hard prompt template is spliced ​​with the output to obtain the final spatiotemporal extended text prompt; using SSTD, different objects and behavior entities are extracted respectively, and text prompts are generated according to the structured semantic information of ASKG; spatial text prompt C s Description of static appearance obtained by prompting object relation triples, temporal text prompts C t Capturing the temporal dynamics through the hint-action relation triples.

[0043] Preferably, the fine-grained alignment is specifically as follows:

[0044] The final visual representation for the i-th frame Right now Model based on N st Spatiotemporal text hint embedding Get its alignment score corresponding to the k-th class to achieve fine-grained alignment of frame to cue and cue to frame:

[0045]

[0046] Then, the overall alignment score is obtained by averaging the two symmetrical spatiotemporal alignment scores above. And use cross entropy loss to optimize the model.

[0047] Preferably, the decision aggregation is specifically:

[0048] During inference, the model's score is compared with the original CLIP score according to and The decision scores of the kth category are aggregated; the category with the highest aggregate score is selected as the final predicted category of the video. The final aggregate score of the kth category is calculated as follows:

[0049]

[0050] The beneficial effects of the present invention are as follows:

[0051] The proposed method is compared with other methods on three common benchmarks: UCF101, HMDB51 and Kinectis600. Experimental results show that the proposed method shows the best performance when using the ViT-L / 14 encoder, achieving 87.9% and 56.5% accuracy on UCF and HMDB, respectively. Compared with other CLIP-based methods, the performance of this method on UCF and HMDB is improved by 7.3% and 6.6% respectively over the second-best method BIKE. By incorporating the proposed SSTD module into Text4Vis and directly replacing its category prompt without any retraining, Text4Vis(+SSTD) achieves significant improvements, achieving +4.6% and +3.1% improvements on the two datasets, respectively, indicating that the proposed semantic enhancement method has strong adaptability. It is worth noting that our model (ViT-B / 16) even achieves better performance than BIKE based on ViT-L / 14 backbone on UCF (82.7% vs 80.6%) and HMDB (52.8% vs 49.8%). BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the method of the present invention.

[0053] Figure 2 It is the overall framework of the multimodal dynamic prompt MDP of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings and examples.

[0055] Existing traditional methods are often limited by the expressive power of features and the coverage of knowledge, and are difficult to handle novel and complex behaviors. Methods based on multimodal pre-training models are limited by the small amount of information in category-level text prompts, lack the exploration of the rich semantic knowledge behind human behavior, and ignore the huge difference between the descriptive narratives involved in pre-training and such short action category names, which reduces the distinguishability of the semantic space. Methods that use large language models to expand CLIP's expertise often focus too much on semantic expansion and lack the spatiotemporal context understanding of the visual space. The above methods are unable to collaboratively possess multimodal spatiotemporal context understanding, resulting in the inability to distinguish fine-grained spatiotemporal differences in similar behaviors, affecting the performance of zero-sample behavior recognition. In order to overcome the shortcomings of the prior art, the present invention provides a zero-sample behavior recognition method based on multimodal prompt learning.

[0056] like Figure 1 and Figure 2 , Figure 2This is the overall framework of the multimodal dynamic prompt MDP of the present invention, where part (a) represents the multi-scale temporal prompt module MTP that automatically generates temporal prompts for specific frames through window shift mask WSM and multi-scale temporal cross attention MS-TCA, part (b) represents the spatiotemporal text enhancement to obtain fine spatial and temporal text prompts, and part (c) represents the calculation of the alignment score in a fine-grained frame-prompt alignment method to improve the performance of the model in zero-sample behavior recognition.

[0057] Overall framework:

[0058] The present invention provides a multimodal dynamic prompt (MDP) framework to achieve zero-shot action recognition. The framework extends CLIP into a fine-grained multimodal spatiotemporal understanding model, and improves the specificity and universality of multimodal spatiotemporal understanding through plug-and-play frame-level multi-scale temporal prompts (MTP) and spatiotemporal semantic enhancement. Overall, the proposed multimodal dynamic prompt framework is initialized using the pre-trained weights of the CLIP model and freezes the weights of the text encoder. For the visual encoder, refined spatiotemporal visual features are obtained by inserting additional learnable multi-scale temporal prompt modules into different layers of the visual encoder.

[0059] Specifically, for a given T-frame video clip Where C, H, and W represent the number of channels, height, and width of the video frame, respectively. Each frame is divided into N non-overlapping patches through the patch embedding layer. Where N = HW / P 2 , the spatial size of each patch is P×P. This process also includes concatenating the category representation to obtain the input of the first layer of the encoder corresponding to the tth frame make Represents the embedded features input to the lth layer visual encoder. Then, the input of the l+1th layer is obtained by the following operation

[0060]

[0061] Among them, each visual encoding layer first passes through a layer normalization (LN) and a multi-head self-attention mechanism (MHSA) in a skip connection manner, and then passes through layer normalization and a feedforward network (MLP).

[0062] At the same time, the Multi-scale Temporal Prompt Module (MTP) first uses Window Shift Mask (WSM) and Multi-scale Temporal Cross Attention (MS-TCA) to respectively remove the category representation of the query obtained by multi-head self-attention. key Sum The embedding of is calculated:

[0063]

[0064] in, is the original mapping weight from the l-th layer visual encoder, in addition, Represents the time scale δ i The corresponding intermediate features, N′ and S represent the number of visible representations after the window shift mask WSM and the number of temporal scales of the multi-scale temporal attention MS-TCA.

[0065] Then, The global information of visible patches is modeled through layer normalization and feed-forward networks to enhance the interaction of different spatial locations in each frame:

[0066]

[0067] Next, use the weight matrix W δ ∈R (S×N′)×P Will The mapping results in a frame-specific timing cue of length P, which concisely represents the spatiotemporal information contained in the video:

[0068]

[0069] The timing prompt and Concatenate as input to the next layer Multi-head self-attention is respectively For query representation, A representation of keys and values.

[0070] For the text branch, unlike the general method that uses standard hard prompts to obtain category representation, this method uses spatiotemporal text enhancement to obtain refined text prompts C st ={C s ,C t}, after passing through the text encoder, we get the text embedding c st ={c s ,c t}.

[0071] Finally, according to and c st Compute the fine-grained alignment scores of all visible categories and optimize the learnable parameters of the model in multi-scale temporal cues by maximizing the score of the correct category while minimizing the scores of other categories. In the inference phase, the recognition results of the model on unknown behavior categories are compared with the original CLIP model according to and c sThe obtained recognition results are aggregated at the decision level to obtain the final recognition result.

[0072] Multi-scale Temporal Prompt (MTP):

[0073] Multi-scale temporal cues mainly include two key processing modules: (1) the window shift mask (WSM) module aims to mask the patches in each processed frame and discard patches at a certain ratio to reduce the computational cost of the subsequent cross-attention mechanism; (2) the multi-scale temporal cross attention (MS-TCA) aims to process the remaining patches after masking, and by performing channel mixing at multiple time scales, the self-attention mechanism can jointly focus on spatial and temporal positions.

[0074] (1) Window Shift Mask (WSM):

[0075] Considering that there is a large amount of temporal redundant information in the video, the window shift mask sequentially discards patches at different spatial positions by shifting the shape of the 2*2 spatial window mask in the temporal dimension. The masking process can be expressed as:

[0076] M t,i =φ(M t-1,i |M 1:t-2,i ),i∈{1,…,R}

[0077] Among them, M t,i represents the mask map of the t-th frame in the i-th multi-scale temporal prompt module, and φ(·|·) is a periodic function that generates a mask map based on the mask frames from 1 to t-1. Then, the mask map M t,i Applied to the query, key, and value representations to produce the corresponding representations of the window shift:

[0078]

[0079] in, Representing N′ spatial queries, the key and value representations of the frame are fed into the l-th layer visual encoder respectively. The window shift mask preserves the essential spatiotemporal correlations in the video and enables multi-scale temporal cross attention to effectively mix channel information across windows and time scales.

[0080] (2) Multi-Scale Temporal Cross Attention (MS-TCA):

[0081] After the window shift mask operation, a lightweight multi-scale temporal cross attention module is introduced to enhance the basic dynamic perception between frames by capturing the temporal correlation information between multiple groups of frame tuples corresponding to different time intervals.

[0082] Specifically, in order to expand the dynamic perception of temporal cues, the key representation k′ after window shift masking is t First, the time series information is fused by indexing the channel information before and after the moment. Indicated by k′ t Index from d s to d e The mixing process corresponding to the time scale δ is expressed as:

[0083]

[0084] Among them, d δ =γ·D is a hyperparameter for the mixed index channel. In order to expand richer dynamic perception and enhance spatiotemporal information fusion, multiple time scales are introduced in a similar way. Let k′ t All bonds with a range of δ represent a mixture, i.e. from arrive Take δ = 2 as an example, and use the same method to obtain the key representation after multi-scale mixing

[0085]

[0086] Then, without introducing any additional parameters, the computation of multi-scale temporal cross attention can be expressed as:

[0087]

[0088] Spatial-temporal text enhancements:

[0089] In addition to enhancing visual representations through temporal cues, this paper also leverages GPT-3.5 to address the low semantic differentiation of action categories by expanding textual cues to incorporate refined behavioral semantic knowledge. Spatiotemporal text enhancement primarily involves a two-stage cuing process: constructing a behavioral semantic knowledge graph and decoupling semantic spacetime.

[0090] (1) Behavioral Semantic Knowledge Graph (ASKG)

[0091] ASKG decomposes categories from spatial representations and temporal dynamics into refined semantic concepts and their relationships. Specifically, ASKG abstracts different behavior categories into a graph structure by representing behaviors, related objects, and sub-behaviors as nodes in a knowledge graph and their relationships as edges. Taking the behavior "abseiling" as an example, the text prompt "Return the object entity list containing TopK(5≤K≤10)most relevant objects / sub-actions involved in action:{abseiling}," is used to obtain the corresponding semantic concepts, and "Find the proper predicate names that concisely describe the relationship between each object / sub-action pair chosen from the entity list" is used to obtain the semantic relationships between the corresponding concepts. Finally, by aggregating the outputs obtained from different behavior categories, ASKG systematically brings together structured representations of behavioral semantics and associations at different granularities.

[0092] (2) Semantic Spatiotemporal Decoupling (SSTD)

[0093] In order to introduce the semantic concepts in ASKG into the model, semantic spatiotemporal decoupling obtains extended text prompts by decomposing text prompts into spatial and temporal levels. Similar to the ASKG construction process, SSTD is implemented using the following text prompt: "Try to complete the whole sentence according to each relation triples: This is an example of {abseiling},…". The hard prompt template is then concatenated with the output to obtain the final spatiotemporal extended text prompt. Using SSTD, rich text prompts are generated based on the structured semantic information of ASKG by extracting different object and behavior entities respectively. Spatial text prompt C s The description is obtained by prompting object relation triples to obtain static appearance description, while the temporal text prompt C t Capturing the temporal dynamic description by prompting action relationship triplets can provide additional rich behavior semantic knowledge and enhance the performance of zero-shot behavior recognition.

[0094] Fine-grained alignment:

[0095] Specifically, for the final visual representation of frame i Right now Model based on N st Spatiotemporal text hint embedding Get its alignment score corresponding to the kth class to achieve fine-grained alignment of frame to prompt and prompt to frame:

[0096]

[0097] Then, the overall alignment score is obtained by averaging the two symmetric spatiotemporal alignment scores mentioned above. And use cross entropy loss to optimize the model.

[0098] and decision aggregation during inference, the model's score is compared with the original CLIP according to and The decision scores of the kth class are aggregated. The class with the highest aggregate score is selected as the final predicted class of the video. The final aggregate score of the kth class is calculated as follows:

[0099]

[0100] Example:

[0101] The model is pre-trained on Kinectis400 using two official CLIP backbones: ViT-B / 16 and ViT-L / 14. The initial learning rate is 5×10 -4 , and then use the AdamW optimizer after the cosine decay scheduler with a decay of 5×10 -6 Only the weights of the multi-scale temporal cue modules were optimized for 15 epochs. 6 (for ViT-B / 16) and 12 (for ViT-L / 14) multi-scale temporal cue modules were plugged on top of CLIP, with a cue length of 8 and a 224 spatial input size. By default, each input clip consists of 8 frames, using 2 temporal views and 3 spatial views of each video.

[0102] For a given test video, we sample T frames to form a video clip V. The video clip is passed through the visual encoder of the multimodal dynamic prompt model to obtain a frame-level dynamic visual representation. In addition, we use the pre-trained model of the original CLIP to obtain a frame-level static visual representation. At the same time, according to the predefined unseen categories, we obtain spatiotemporal text prompts through spatiotemporal text enhancement. The final visual representation of the i-th frame is Right now Model based on N st Spatiotemporal text hint embedding Get its alignment score corresponding to the k-th class to achieve fine-grained alignment of frame to cue and cue to frame:

[0103]

[0104] Then, the overall alignment score is obtained by averaging the two symmetric spatiotemporal alignment scores mentioned above. The scores of the multimodal dynamic cue model were compared with the original CLIP model according to and The decision scores of the kth class are aggregated. The class with the highest aggregate score is selected as the final predicted class of the video. The final aggregate score of the kth class is calculated as follows:

[0105]

Claims

1. A zero-shot behavior recognition method based on multimodal cue learning, characterized by: The steps include: Step 1: For a given T-frame video clip Where C, H, and W represent the number of channels, height, and width of the video frame, respectively; each video frame is divided into N non-overlapping patches through the patch embedding layer. Where N = HW / P 2 , the spatial size of each patch is P×P; This process also includes the operation of splicing additional category embeddings to obtain the input of the first layer of the encoder corresponding to the tth frame D represents the embedding dimension; let Represents the embedded features input to the lth layer visual encoder, and obtains the input of the l+1th layer Among them, each visual encoding layer first passes through a layer normalization LN and a multi-head self-attention mechanism MHSA in a skip connection manner, and then passes through layer normalization and a feedforward network MLP; Step 2: The multi-scale temporal hint module MTP first uses the window shift mask WSM and multi-scale temporal cross attention MS-TCA to respectively remove the category representation of the query obtained by multi-head self-attention. key Sum The embedding of is calculated: in, is the original mapping weight from the l-th layer visual encoder, Represents the time scale δ i The corresponding intermediate features, N′ and S, respectively represent the number of visible representations after the window shift mask WSM and the number of time scales of the multi-scale temporal attention MS-TCA; Step 3: The global information of visible patches is modeled through layer normalization and feed-forward networks to enhance the interaction of different spatial locations in each frame: Step 4: Use the weight matrix W δ ∈R (S×N′)×P Will The mapping obtains a frame-specific timing prompt of length P, which represents the spatiotemporal information contained in the video: The timing prompt and Concatenate as input to the next layer Multi-head self-attention is respectively For query representation, For the representation of keys and values; Step 5: For semantic text, use spatiotemporal text enhancement to obtain refined text hints C st ={C s ,C t }, after passing through the text encoder, we get the text embedding c st ={c s ,c t }; Step 6: According to and c st Calculate the fine-grained alignment scores of all visible categories, and optimize the learnable parameters of the model in multi-scale temporal cues by maximizing the score of the correct category while minimizing the scores of other categories; in the inference stage, the recognition results of the model on unknown behavior categories are compared with the original CLIP model according to and c s The obtained recognition results are aggregated for decision making to obtain the final recognition result.

2. The zero-shot behavior recognition method based on multimodal cue learning according to claim 1, characterized in that: The multi-scale timing prompt MTP is as follows: The proposed multi-scale temporal cueing consists of two processing modules: (1) the window shift mask (WSM) module aims to mask the patches in each processed frame and discard patches at a certain ratio to reduce the computational cost of the subsequent cross-attention mechanism; (2) the multi-scale temporal cross attention (MS-TCA) aims to process the remaining patches after masking, and enables the self-attention mechanism to jointly focus on spatial and temporal positions by performing channel mixing at multiple time scales; (1) Window shift mask WSM: The window shift mask sequentially discards patches at different spatial positions by shifting the shape of the window mask of 2*2 spatial size in the time dimension; the masking process is expressed as: M t,i =φ(M t-1,i |M 1:t-2,i ),o∈{1,…,R} Among them, M t,i represents the mask map of the t-th frame in the i-th multi-scale temporal prompt module, φ(·|·) is a periodic function that generates a mask map based on the mask frames from 1 to t-1; then, the mask map M t,i Applied to the query, key, and value representations to produce the corresponding representations of the window shift: in, Represent N′ spatial queries, and feed the key and value representations of the frame into the l-th layer visual encoder respectively; (2) Multi-scale Temporal Cross Attention MS-TCA: The key after window shift mask is represented as k′ t First, the time series information is fused by indexing the channel information before and after the moment; Indicated by k′ t Index from d s to d e The mixing process corresponding to the time scale δ is expressed as: Among them, d δ =γ·D is a hyperparameter for mixing index channels; Introducing multiple time scales Let k′ t All bonds with a range of δ represent a mixture, i.e. from arrive When δ = 2, the multi-scale mixed bond representation is obtained Without introducing any additional parameters, the calculation of multi-scale temporal cross attention is expressed as:

3. The zero-sample behavior recognition method based on multimodal cue learning according to claim 2 is characterized in that: The spatiotemporal text enhancement is specifically as follows: The spatiotemporal text enhancement includes a two-stage prompting process, namely the construction of behavioral semantic knowledge graph and semantic spatiotemporal decoupling; (1) Behavioral semantic knowledge graph (ASKG); ASKG decomposes categories from spatial representations and temporal dynamics into semantic concepts and their relations; ASKG abstracts different behavior categories into a graph structure by representing behaviors, related objects, and sub-behaviors as nodes in a knowledge graph and their relationships as edges. Finally, by aggregating the outputs of different behavior categories, ASKG brings together structured representations of behavior semantics and associations at different granularities. (2) semantic spatiotemporal decoupling (SSTD); Semantic spatiotemporal decoupling obtains extended text prompts by decomposing text prompts into spatial and temporal levels; then the hard prompt template is spliced ​​with the output to obtain the final spatiotemporal extended text prompt; using SSTD, different objects and behavior entities are extracted respectively, and text prompts are generated according to the structured semantic information of ASKG; spatial text prompt C s Description of static appearance obtained by prompting object relation triples, temporal text prompts C t Capturing the temporal dynamics through the hint-action relation triples.

4. The zero-sample behavior recognition method based on multimodal cue learning according to claim 3 is characterized in that: The fine-grained alignment is as follows: The final visual representation for frame i Right now Model based on N st Spatiotemporal text hint embedding Get its alignment score corresponding to the k-th class to achieve fine-grained alignment of frame to cue and cue to frame: Then, the overall alignment score is obtained by averaging the two symmetrical spatiotemporal alignment scores above. And use cross entropy loss to optimize the model.

5. The zero-sample behavior recognition method based on multimodal cue learning according to claim 4 is characterized in that: The decision aggregation is specifically: During inference, the model's score is compared with the original CLIP score according to and The decision scores of the kth category are aggregated; the category with the highest aggregate score is selected as the final predicted category of the video. The final aggregate score of the kth category is calculated as follows:

Images