Spatio-temporal saliency cueing method and apparatus for open vocabulary video action recognition

Abstract

This invention provides a spatiotemporal saliency cues method and apparatus for open-vocabulary video action recognition, relating to the field of computer vision technology. It includes: inputting an input video to a video-level spatiotemporal saliency cues module; obtaining a coarse-grained saliency map through cross-modal aligned attention branches; obtaining a fine-grained saliency map through instance-level self-attention branches; fusing the two to obtain a video-level spatiotemporal saliency cues map; performing element-wise multiplication with the input video and introducing residual connections to obtain enhanced discriminative action features; injecting the fine-grained saliency map into the input video to generate spatiotemporal action features; and inputting these features into a token-level spatiotemporal saliency cues module to obtain the open-vocabulary video action recognition result. This invention proposes a novel spatiotemporal saliency cues method that can dynamically focus on instance-level discriminative action regions and generate multiple levels of action saliency cues that complement each other, thereby significantly enhancing the model's generalization ability.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a spatiotemporal saliency cueing method and apparatus for open-vocabulary video action recognition. Background Technology

[0002] Action recognition is a core task in video understanding, aiming to accurately identify human actions within input video clips—specifically, determining the type of action performed by a subject over a given period. Compared to static visual tasks like image classification, action recognition requires simultaneously modeling dynamic changes over time and semantic representations in space, placing higher demands on models in areas such as temporal modeling, fine-grained difference discrimination, and multi-scale feature integration. However, when applied to the real world, a significant problem arises: real-world action categories are open-ended and constantly changing, making it impossible to exhaustively define and label all possible action categories beforehand. Traditional methods, trained and tested on closed category sets, often lack generalization ability when faced with unfamiliar new action categories, making them ill-suited for the complex action recognition needs of open worlds.

[0003] To address this issue, OVAR (Open-vocabulary Action Recognition) has been proposed as an emerging task setting. Unlike traditional closed settings, OVAR requires the model to be trained under supervision only on "base classes" and tested on "new classes" of actions. Its core challenge lies in how to enable the model to learn transferable and interpretable action semantic features from limited labeled data, and to guide the model to reason and match new categories of actions in a unified semantic space through natural language descriptions. This task not only emphasizes cross-modal semantic alignment (the correspondence between video and language), but also requires the model to possess zero-shot recognition or few-shot learning capabilities to achieve accurate recognition of unseen classes.

[0004] In recent years, thanks to the development of large-scale visual language pre-trained models (such as CLIP (Contrastive Language-Image Pre-Training), ALIGN (A Large-scale Image and Noisy-text Embedding), VideoCLIP, etc.), researchers can construct a unified semantic embedding space, mapping video content and language descriptions to the same space, and completing action recognition by calculating the similarity between the two. This framework eliminates the dependence on fixed category labels, providing a scalable foundation for open-world action recognition. As an important component of open-world understanding, open-lexical action recognition is increasingly demonstrating its research value and practical application prospects. In intelligent security scenarios, the model can be used to identify sudden abnormal behaviors or rare actions (such as falls); in autonomous driving and assisted driving systems, it can be used to identify complex interactive actions of pedestrians, vehicles, or cyclists; in intelligent sports analysis, OVAR can identify various body postures, gestures, or tactical actions to achieve refined training evaluation; and in fields such as human-computer interaction, content retrieval, and remote rehabilitation, this task can also be flexibly invoked through natural language, greatly expanding the generalization ability and practicality of video intelligence systems.

[0005] Currently, open-vocabulary video action classification algorithms utilize Vision-Language Models (VLMs), such as CLIP. These models are pre-trained on large-scale image-text alignment data, enabling zero-shot or few-shot generalization to unseen actions. While CLIP has demonstrated remarkable zero-shot performance on diverse downstream image tasks through large-scale pre-training, it cannot naturally capture temporal dynamics in videos. To address this, a series of works have proposed compensating for CLIP's shortcomings by introducing temporal modeling modules or lightweight adapters. For example, methods such as ActionCLIP, X-CLIP, ST-Adapter, Open-VCLIP, AIM, ViFi-CLIP, and TC-CLIP all incorporate inter-frame interaction modeling or additional temporal adaptation structures on top of CLIP, allowing visual language models to better adapt to video understanding tasks. These methods generally prioritize lightweight and scalability in their design to enhance temporal awareness without compromising CLIP's original semantic representation capabilities. Besides temporal modeling, other research has shifted its focus to optimizing cross-modal cues. For example, methods such as Vita-CLIP, VPT, XOV-Action, and AP-CLIP refine the semantic alignment process between video and language by learning more expressive multimodal cues. They shorten the modality gap through dynamic, task-relevant cue design, enabling the model to distinguish action categories at a fine-grained level, thus achieving higher recognition accuracy in open-vocabulary scenarios. Recent research has also seen new approaches to further enhance the adaptability of CLIP in video action understanding. FROSTER (FROZEN CLIP IS A STRONG TEACHER FOR OPEN-VOCABULARY ACTION RECOGNITION) treats frozen CLIP as a teacher model, guiding the student model to learn video-specific representations through residual feature distillation, thereby maintaining generalization ability while avoiding overfitting. This distillation framework effectively solves the core contradiction of "how to enhance temporal features without destroying pre-trained semantics."Building on this, Open-MeDe (Learning to Generalize without Bias for Open-Vocabulary Action Recognition) proposes a framework that combines meta-optimization and static debiasing. It mitigates the inherent static bias of CLIP through a cross-batch meta-learning strategy, enabling the model to better balance the relationship between static visual semantics and dynamic temporal modeling, thereby further improving its robustness and generalization performance on open-vocabulary action recognition tasks.

[0006] Furthermore, prompt learning has become a key technique for efficiently adapting visual language models to downstream tasks, such as CLIP. It maintains the model's zero-shot generalization ability with only a very small number of parameter updates. Early prompting methods learned prompts through context optimization, significantly improving performance in few-shot scenarios and reducing the need for retraining the entire model. Subsequently, research proposed conditional prompts to enhance cross-domain generalization capabilities.

[0007] In this direction, various improvement methods have emerged. For example, multimodal cue learning improves zero-shot transfer performance by combining cues from visual and linguistic branches; other methods constrain the cue learning process to prevent the model from forgetting general knowledge acquired during VLM pre-training. Other works insert adjustable visual tokens into the visual encoder for task adaptation. Recently, methods have leveraged rich text representations in pre-trained models to jointly improve model performance during distillation and inference stages; other methods introduce meta-visual cues to refine channel-level attention distribution, enabling the model to capture more valuable details with less attention resources; simultaneously, research decomposes visual representations into interpretable text elements and aligns them with attention heads and spatial locations, thereby enhancing the model's interpretability and cross-modal understanding.

[0008] Video action recognition is a challenging video understanding task. Current methods suffer from insufficient open-ended capabilities, relying almost entirely on pre-defined action categories for training. While existing methods have achieved good performance on closed categories through model architecture optimization and improved training device capabilities, most struggle with recognizing unfamiliar new categories, rendering them unsuitable for real-world applications.

[0009] 1) Lack of Effective Spatiotemporal Modeling: Existing open-vocabulary action recognition methods generally lack effective spatiotemporal modeling capabilities. Most visual language models (such as CLIP) are primarily pre-trained based on image-text alignment, thus possessing a strong advantage in capturing static spatial semantic features. However, the core of video action recognition lies in modeling the dynamic evolution process in the temporal dimension, which is precisely the inherent weakness of image-level pre-trained models. In practical tasks, many actions are difficult to distinguish based on a single frame image alone, requiring reliance on cross-frame temporal relationships and the continuity of action stages. For example, "waving" and "pushing a door" may present extremely similar hand gestures in a single frame, but they differ significantly in their temporal evolution trajectories. Without spatiotemporal dependency modeling, the model often confuses these actions, leading to classification errors. Furthermore, the background in videos often coexists with the action subject and occupies a large spatial area. Without an effective spatiotemporal attention mechanism, the model is easily interfered with by redundant backgrounds, thereby weakening its focus on the discriminative region of the action. Therefore, how to further improve the spatiotemporal modeling capability while maintaining the strong semantic alignment capability of visual language models is the primary challenge in open-vocabulary action recognition.

[0010] 2) Limited Knowledge of New Categories: Existing open-vocabulary action recognition methods face significant challenges in recognizing new categories. Since the model primarily relies on base class samples during training, these categories possess ample labeled data and clear semantic representations, making it easier for the model to learn stable discrimination patterns. However, when applied to unseen new categories, the model often exhibits insufficient generalization ability and a significant drop in accuracy. This "training-test category bias" problem is particularly pronounced in open-vocabulary settings. First, the imbalanced category distribution causes the model to develop a biased learning strategy towards base classes during training, making it difficult to actively transfer semantics to new categories in the linguistic space. Second, although visual language models possess cross-modal alignment capabilities, their semantic representation capabilities cannot be fully utilized without spatiotemporal modeling and instance-aware mechanisms for new categories, thus limiting the transfer value of linguistic knowledge in new category recognition. Furthermore, under limited or zero-sample supervision, the model is prone to catastrophic forgetting, i.e., losing memory of the original base classes during adaptation to new categories, further leading to overall performance instability. Therefore, how to effectively activate the prior language knowledge in the visual language model while preserving the base class discrimination ability, and achieve cross-modal semantic transfer and spatiotemporal modeling of new categories, is a key problem that urgently needs to be solved in open-vocabulary action recognition. Summary of the Invention

[0011] To address the issue that most existing spatiotemporal action detection algorithms focus on optimizing the identification and localization of closed action categories (i.e., predefined, limited number of behavior types), these methods can provide relatively accurate results in specific scenarios, such as behavior monitoring common in sports event analysis or surveillance environments. However, their design and training methods limit their application scope, failing to fully consider the complexity and diversity of the real world and thus unable to handle the technical challenges of dealing with the complex and diverse behavior categories in the real world. This invention provides a spatiotemporal saliency cueing method and apparatus for open-vocabulary video action recognition. The technical solution is as follows:

[0012] On the one hand, a spatiotemporal saliency cueing method for open-vocabulary video action recognition is provided. This method is implemented by a spatiotemporal saliency cueing device for open-vocabulary video action recognition, and includes:

[0013] S1. Obtain the input video to be recognized.

[0014] S2. Input the input video to the video-level spatiotemporal saliency cues module. Obtain a coarse-grained saliency map through the cross-modal aligned attention branch of the video-level spatiotemporal saliency cues module. Obtain a fine-grained saliency map through the instance-level self-attention branch of the video-level spatiotemporal saliency cues module. Fuse the coarse-grained saliency map and the fine-grained saliency map to obtain the video-level spatiotemporal saliency cues map.

[0015] S3. Multiply the video-level spatiotemporal saliency cue map element-wise with the input video and introduce residual connections to obtain the enhanced discriminative action features.

[0016] S4. Inject fine-grained saliency maps into the input video to generate instance-level spatiotemporal action features.

[0017] S5. Input the input video, the enhanced discriminative action features, and the instance-level spatiotemporal action features into the Token-level spatiotemporal saliency cue module to obtain the open-vocabulary video action recognition results.

[0018] Optionally, S2 includes:

[0019] S21. Input the input video into the cross-modal aligned attention branch of the video-level spatiotemporal saliency cue module to generate a category-independent coarse-grained saliency map.

[0020] S22. Input the input video to the instance-level self-attention branch of the video-level spatiotemporal saliency cues module and extract a fine-grained saliency map that reflects local dynamic changes.

[0021] S23. The coarse-grained saliency map and the fine-grained saliency map are concatenated along the feature dimension and input into the fusion network to obtain the fused saliency map.

[0022] S24. The fused saliency map is used to obtain a video-level spatiotemporal saliency cue map through the Sigmoid activation function.

[0023] Optionally, the cross-modal alignment attention branch leverages the text-image alignment capability of the CLIP model and generates a category-independent coarse-grained saliency map through guiding text.

[0024] Optionally, the instance-level self-attention branch includes a lightweight 3D convolutional network module; wherein the lightweight 3D convolutional network module consists of multiple 3D convolutional layers, BatchNorm layers, and ReLU layers.

[0025] Optionally, the fusion network consists of two 3D convolutional layers, a BatchNorm layer, and a ReLU layer.

[0026] Optionally, S5 includes:

[0027] S51. Input instance-level spatiotemporal action features into a lightweight 3D convolutional network module and an MLP module to extract instance adaptive cue tokens.

[0028] S52. Introduce a teacher model learnable token and a student model learnable token. Add the teacher model learnable token and the student model learnable token to the instance adaptive prompt token to form the teacher model context prompt information and the student model context prompt information, respectively.

[0029] S53. Input the contextual prompts and input video of the teacher model into the teacher model to obtain the video embedding representation and text embedding representation of the teacher model; input the contextual prompts and input video of the student model into the student model to obtain the video embedding representation and text embedding representation of the student model.

[0030] S54. Input the enhanced discriminative action features into the student model to obtain the extracted video embedding representation and text embedding representation.

[0031] S55. Match the student model video embedding representation and the enhanced discriminative video embedding representation with the student model text representation, and obtain the prediction results of the two action categories by calculating the dot product.

[0032] S56. The prediction results of the two action categories are weighted and fused, and the open vocabulary video action recognition results are output through the softmax function.

[0033] S57. Based on the video and text embedding representations of the teacher model, the video and text embedding representations of the student model, and the enhanced discriminative video embedding representation, construct classification loss and feature distillation loss. Jointly optimize the classification loss and feature distillation loss to obtain the fine-tuned student model. Optionally, the classification loss is calculated using cross-entropy loss based on the student model video and text embedding representations and the extracted video and text embedding representations.

[0034] Feature distillation loss includes visual distillation loss and text distillation loss; visual distillation loss is calculated by comparing the feature differences between the teacher model video embedding representation, the student model video embedding representation, and the enhanced discriminative video embedding representation; text distillation loss is calculated by comparing the feature differences between the teacher model text embedding representation and the student model text embedding representation.

[0035] On the other hand, a spatiotemporal saliency cueing device for open-vocabulary video action recognition is provided. This device is applied to a spatiotemporal saliency cueing method for open-vocabulary video action recognition, and includes:

[0036] The data acquisition module is used to acquire the input video to be recognized.

[0037] The video-level spatiotemporal saliency cues module is used to input the video into the video-level spatiotemporal saliency cues module. A coarse-grained saliency map is obtained through the cross-modal aligned attention branch of the video-level spatiotemporal saliency cues module; a fine-grained saliency map is obtained through the instance-level self-attention branch of the video-level spatiotemporal saliency cues module; and the video-level spatiotemporal saliency cues map is obtained by fusing the coarse-grained saliency map and the fine-grained saliency map.

[0038] The Token-level spatiotemporal saliency cue module is used to perform element-wise multiplication of the video-level spatiotemporal saliency cue map with the input video and introduce residual connections to obtain enhanced discriminative action features; a fine-grained saliency map is injected into the input video to generate instance-level spatiotemporal action features; the input video, enhanced discriminative action features, and instance-level spatiotemporal action features are input into the Token-level spatiotemporal saliency cue module to obtain the open-vocabulary video action recognition result.

[0039] Optionally, the video-level spatiotemporal saliency cue module is further used for:

[0040] S21. Input the input video into the cross-modal aligned attention branch of the video-level spatiotemporal saliency cue module to generate a category-independent coarse-grained saliency map.

[0041] S22. Input the input video to the instance-level self-attention branch of the video-level spatiotemporal saliency cues module and extract a fine-grained saliency map that reflects local dynamic changes.

[0042] S23. The coarse-grained saliency map and the fine-grained saliency map are concatenated along the feature dimension and input into the fusion network to obtain the fused saliency map.

[0043] S24. The fused saliency map is used to obtain a video-level spatiotemporal saliency cue map through the Sigmoid activation function.

[0044] Optionally, the cross-modal alignment attention branch leverages the text-image alignment capability of the CLIP model and generates a category-independent coarse-grained saliency map through guiding text.

[0045] Optionally, the instance-level self-attention branch includes a lightweight 3D convolutional network module; wherein the lightweight 3D convolutional network module consists of multiple 3D convolutional layers, BatchNorm layers, and ReLU layers.

[0046] Optionally, the fusion network consists of two 3D convolutional layers, a BatchNorm layer, and a ReLU layer.

[0047] Optionally, the token-level spatiotemporal saliency hint module is further used for:

[0048] S51. Input instance-level spatiotemporal action features into a lightweight 3D convolutional network module and an MLP module to extract instance adaptive cue tokens.

[0049] S52. Introduce a teacher model learnable token and a student model learnable token. Add the teacher model learnable token and the student model learnable token to the instance adaptive prompt token to form the teacher model context prompt information and the student model context prompt information, respectively.

[0050] S53. Input the contextual prompts and input video of the teacher model into the teacher model to obtain the video embedding representation and text embedding representation of the teacher model; input the contextual prompts and input video of the student model into the student model to obtain the video embedding representation and text embedding representation of the student model.

[0051] S54. Input the enhanced discriminative action features into the student model to obtain the enhanced discriminative video embedding representation.

[0052] S55. Match the student model video embedding representation and the enhanced discriminative video embedding representation with the student model text representation, and obtain the prediction results of the two action categories by calculating the dot product.

[0053] S56. The prediction results of the two action categories are weighted and fused, and the open vocabulary video action recognition results are output through the softmax function.

[0054] S57. Based on the video and text embedding representations of the teacher model, the video and text embedding representations of the student model, and the enhanced discriminative video embedding representation, construct classification loss and feature distillation loss. Jointly optimize the classification loss and feature distillation loss to obtain the fine-tuned student model. Optionally, the classification loss is calculated using cross-entropy loss based on the student model video and text embedding representations and the extracted video and text embedding representations.

[0055] Feature distillation loss includes visual distillation loss and text distillation loss; visual distillation loss is calculated by comparing the feature differences between the teacher model video embedding representation, the student model video embedding representation, and the enhanced discriminative video embedding representation; text distillation loss is calculated by comparing the feature differences between the teacher model text embedding representation and the student model text embedding representation.

[0056] On the other hand, a spatiotemporal saliency prompting device for open-vocabulary video action recognition is provided, wherein the spatiotemporal saliency prompting device for open-vocabulary video action recognition...

[0057] The prompting device includes: a processor; and a memory storing computer-readable instructions that, when executed by the processor, implement any of the methods described above for spatiotemporal saliency prompting for open-vocabulary video action recognition.

[0058] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described spatiotemporal saliency cues for open-vocabulary video action recognition methods.

[0059] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0060] This invention proposes a novel spatiotemporal saliency cueing model for open-vocabulary action recognition. This model can capture instance-level discriminative action regions, significantly enhancing the ability of visual language models to recognize base and new class actions.

[0061] A video-level spatiotemporal saliency cue module was designed. By fusing cross-modal aligned attention and instance-level self-attention, it effectively extracts discriminative action features, highlights key action regions, and suppresses irrelevant background information.

[0062] A token-level spatiotemporal saliency cue module is introduced, and instance adaptive context is integrated into the visual language model based on a teacher-student architecture, thereby improving the model's generalization ability to new action categories.

[0063] Extensive experiments conducted on multiple open-source action recognition benchmark datasets demonstrate that the SSP model proposed in this invention achieves state-of-the-art performance in this task. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart of a spatiotemporal saliency cues method for open-vocabulary video action recognition provided by an embodiment of the present invention;

[0066] Figure 2 This is the overall model architecture diagram provided in the embodiments of the present invention;

[0067] Figure 3 This is a visualization comparison of the attention of the method of the present invention with FROSTER and TextSpan on the K400 dataset provided in this embodiment of the invention;

[0068] Figure 4 This is a block diagram of a spatiotemporal saliency prompting device for open vocabulary video action recognition provided in an embodiment of the present invention;

[0069] Figure 5 This is a schematic diagram of the structure of a spatiotemporal saliency prompting device for open vocabulary video action recognition provided in an embodiment of the present invention. Detailed Implementation

[0070] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0071] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0072] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0073] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0074] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0075] This invention provides a spatiotemporal saliency cueing method for open-vocabulary video action recognition. This method can be implemented by a spatiotemporal saliency cueing device for open-vocabulary video action recognition, which can be a terminal or a server. Figure 1 , Figure 2 The flowchart shown is for a spatiotemporal saliency cueing method for open-vocabulary video action recognition. The processing flow of this method may include the following steps:

[0076] S1. Obtain the input video to be recognized.

[0077] In one feasible implementation, the research objective of this invention is to establish a simple and effective open-vocabulary video action recognition technology for the real world. The aim is to construct an open conceptual space by leveraging conceptual knowledge learned from massive amounts of image-text pairs using a visual language pre-trained model. This knowledge can then be fine-tuned and transferred to downstream perceptual tasks using a small amount of training data from downstream tasks. The model will be able to identify relevant entities of different granularities in a video based on arbitrary descriptions.

[0078] S2. Input the input video to the video-level spatiotemporal saliency cues module. Obtain a coarse-grained saliency map through the cross-modal aligned attention branch of the video-level spatiotemporal saliency cues module. Obtain a fine-grained saliency map through the instance-level self-attention branch of the video-level spatiotemporal saliency cues module. Fuse the coarse-grained saliency map and the fine-grained saliency map to obtain the video-level spatiotemporal saliency cues map.

[0079] Optionally, step S2 above may include the following steps S21-S24:

[0080] S21. Input the input video into the cross-modal aligned attention branch of the video-level spatiotemporal saliency cue module to generate a category-independent coarse-grained saliency map.

[0081] Among them, the cross-modal alignment attention branch utilizes the image-text alignment capability of the CLIP model and generates a category-independent coarse-grained saliency map through guiding text.

[0082] S22. Input the input video to the instance-level self-attention branch of the video-level spatiotemporal saliency cues module and extract a fine-grained saliency map that reflects local dynamic changes.

[0083] The instance-level self-attention branch includes a lightweight 3D convolutional network module; the lightweight 3D convolutional network module consists of multiple 3D convolutional layers, BatchNorm layers, and ReLU layers.

[0084] S23. The coarse-grained saliency map and the fine-grained saliency map are concatenated along the feature dimension and input into the fusion network to obtain the fused saliency map.

[0085] The fusion network consists of two 3D convolutional layers, a BatchNorm layer, and a ReLU layer.

[0086] S24. The fused saliency map is used to obtain a video-level spatiotemporal saliency cue map through the Sigmoid activation function.

[0087] In one feasible implementation, the video-level spatiotemporal saliency cueing module of the present invention extracts discriminative action features from the input video by fusing cross-modal aligned attention and instance-level self-attention, thereby improving the recognition capability of the pre-trained visual language model.

[0088] Specifically, the module comprises two branches: a cross-modal aligned attention branch and an instance-level self-attention branch, used to mine coarse-grained and fine-grained motion saliency regions from the input video, respectively. The cross-modal aligned attention branch leverages CLIP's image-text alignment capabilities to generate category-independent coarse-grained saliency maps using guiding text (such as "a video of a person performing an action"). This process encodes the video frame-by-frame and integrates the output information from all multi-head attention within the Transformer, resulting in a full-time saliency response map. The instance-level self-attention branch models the original video using a lightweight 3D convolutional network, automatically extracting fine-grained saliency maps reflecting local dynamic changes. This network consists of multiple layers of 3D convolutions, BatchNorm, and ReLU, with the final layer using upsampling to preserve spatial resolution.

[0089] Furthermore, the saliency maps generated by the two branches are semantically complementary: the former provides semantically guided global action awareness, while the latter emphasizes fine-grained local action regions at the instance level. This invention concatenates the two along the feature dimension and inputs them into a fusion network. This fusion network consists of two layers of 3D convolutions, BatchNorm, and ReLU, and finally outputs the final video-level spatiotemporal saliency cue map through a sigmoid activation function.

[0090] S3. Multiply the video-level spatiotemporal saliency cue map element-wise with the input video and introduce residual connections to obtain the enhanced discriminative action features.

[0091] In one feasible implementation, the saliency cue map is multiplied element-wise with the original video, and residual connections are introduced to obtain enhanced discriminative action features. This enhancement mechanism helps to highlight key action regions while retaining effective information from the original video, thereby improving the overall feature representation capability.

[0092] S4. Inject fine-grained saliency maps into the input video to generate instance-level spatiotemporal action features.

[0093] In one feasible implementation, after obtaining a fine-grained instance-level action saliency map in the video-level spatiotemporal saliency cue module, this invention injects it into the original video to generate instance-level spatiotemporal action features. This saliency enhancement feature not only preserves the temporal structure of the original video but also highlights action-related regions, helping to improve the model's generalization ability to new categories. Subsequently, this invention inputs this feature into the teacher model and the student model as the basis for further learning.

[0094] S5. Input the input video, the enhanced discriminative action features, and the instance-level spatiotemporal action features into the Token-level spatiotemporal saliency cue module to obtain the open-vocabulary video action recognition results.

[0095] Optionally, step S5 above may include the following steps S51-S58:

[0096] S51. Input instance-level spatiotemporal action features into a lightweight 3D convolutional network module and an MLP module to extract instance adaptive cue tokens.

[0097] S52. Introduce a teacher model learnable token and a student model learnable token. Add the teacher model learnable token and the student model learnable token to the instance adaptive prompt token to form the teacher model context prompt information and the student model context prompt information, respectively.

[0098] S53. Input the contextual prompts and input video of the teacher model into the teacher model to obtain the video embedding representation and text embedding representation of the teacher model; input the contextual prompts and input video of the student model into the student model to obtain the video embedding representation and text embedding representation of the student model.

[0099] S54. Input the enhanced discriminative action features into the student model to obtain the enhanced discriminative video embedding representation.

[0100] S55. Match the student model video embedding representation and the enhanced discriminative video embedding representation with the student model text representation, and obtain the prediction results of the two action categories by calculating the dot product.

[0101] S56. The prediction results of the two action categories are weighted and fused, and the open vocabulary video action recognition results are output through the softmax function.

[0102] S57. Based on the video embedding representation and text embedding representation of the teacher model, the video embedding representation and text embedding representation of the student model, and the enhanced discriminative video embedding representation, construct the classification loss and feature distillation loss, and perform joint optimization based on the classification loss and feature distillation loss to obtain the fine-tuned student model.

[0103] In one feasible implementation, the token-level spatiotemporal saliency cue module of the present invention, after obtaining a fine-grained instance-level action saliency map from the video-level spatiotemporal saliency cue module, injects it into the original video to generate instance-level spatiotemporal action features. This saliency enhancement feature not only preserves the temporal structure of the original video but also highlights the action-related regions, helping to improve the model's generalization ability to new categories. Subsequently, the present invention inputs this feature into the teacher model and the student model as the basis for further learning.

[0104] Specifically, this invention first designs a lightweight 3D convolutional network and an MLP (Multilayer Perceptron) module to extract instance-adaptive cue tokens from enhanced video features. These tokens can be viewed as compressed representations of the current video action patterns, possessing a certain temporal modeling capability. Next, this invention introduces a set of learnable tokens for both the teacher and student models, and adds them to the aforementioned instance-adaptive tokens to form the final contextual cue information, thereby creating a spatiotemporal context adapted to the current video instance.

[0105] During inference, the teacher model uses frozen CLIP weights, while the student model undergoes fine-tuning. Both models combine their respective instance-adaptive cue tokens to obtain corresponding video and text embeddings. Simultaneously, the student model also incorporates discriminative action features generated by the video-level spatiotemporal saliency cue module to extract another set of embeddings. This dual-path design allows the teacher model to maintain strong generalization capabilities while enabling the student model to better adapt to feature learning for new action categories. Ultimately, these visual and text embeddings are used for classification prediction and feature distillation, improving the overall model's performance in open-vocabulary action recognition tasks.

[0106] Optionally, the classification loss is calculated using cross-entropy loss based on the student model's video embedding representation and text embedding representation, as well as the extracted video embedding representation and text embedding representation.

[0107] Feature distillation loss includes visual distillation loss and text distillation loss; visual distillation loss is calculated by comparing the feature differences between the teacher model video embedding representation and the student model video embedding representation, and an auxiliary branch is introduced to strengthen the constraints; text distillation loss is calculated by comparing the feature differences between the teacher model text embedding representation and the student model text embedding representation.

[0108] In one feasible implementation, model training specifically includes: for the input video, joint optimization is performed by minimizing the sum of classification loss and feature distillation loss. The final objective function consists of a weighted classification loss and a feature distillation loss. Specifically, the classification loss is calculated using cross-entropy loss based on two visual embeddings extracted by the student model: the student model's video embedding representation and the enhanced discriminative video embedding representation. This loss considers not only the prediction results obtained from the student model's encoded video but also the prediction results obtained from the enhanced discriminative video embedding generated by the video-level spatiotemporal saliency cue module; both jointly optimize classification performance. Regarding feature distillation, this invention simultaneously constrains both the visual and textual modalities. The visual distillation loss is calculated by comparing the feature differences between the teacher model and the student model under two different visual representations; while the text distillation loss calculates the differences in text feature representations between the teacher and student models. To enhance representational capability, the student model output is first passed through a two-layer MLP projection head with residual connections before being aligned with the teacher output. By jointly optimizing the above two loss terms, the model of this invention can effectively improve the generalization ability to new categories while retaining the base class recognition capability.

[0109] Furthermore, the model inference process specifically includes: based on the input video, the video-level spatiotemporal saliency cues module first extracts discriminative action features to highlight key spatiotemporal regions. Subsequently, the fine-tuned CLIP takes the original video and the saliency-enhanced video as input to generate two sets of visual embedding features. These two sets of visual features are then matched with their corresponding text features, and the prediction results for the two action categories are obtained through dot product calculation. Finally, the two prediction results are weighted and fused, and the final classification result is output through softmax.

[0110] Tables 1 and 2 present the experimental results of the proposed SSP model and existing mainstream methods on four open lexical action recognition datasets: K400, HMDB-51, UCF-101, and SSv2. Taking the K400 dataset as an example, as shown in Table 1, the proposed SSP model was compared with other methods in a base-to-novel setting. The results show that SSP achieved the best performance on all evaluation metrics. Specifically, SSP achieved accuracies of 78.2%, 65.6%, and 71.4% in the Base, Novel, and HM partitions, respectively. Compared with the strong baseline method FROSTER, which uses a CLIP model with freezing and fine-tuning, SSP improved performance by 0.4%, 1.3%, and 1.0% in the three settings, respectively. Furthermore, compared with the Open-MeDe method, the proposed SSP model improved performance by 1.0%, 1.8%, and 1.5%, respectively. These performance improvements demonstrate that, thanks to the carefully designed video-level and token-level spatiotemporal saliency cueing mechanism of this invention, SSP not only maintains strong discriminative ability in base class action recognition, but also exhibits excellent generalization performance in new class action recognition.

[0111] Similar conclusions were reached on other datasets (such as HMDB-51, UCF-101, and SSv2). For example, on the HMDB-51 dataset, SSP improved performance by 3.1% compared to FROSTER in the Novel setting and by 3.2% compared to Open-MeDe in the HM setting. On the UCF-101 dataset, SSP achieved accuracies of 96.1%, 81.2%, and 88.0% in the Base, Novel, and HM settings, respectively, representing improvements of 0.8%, 1.2%, and 1.0% over FROSTER, and improvements of 1.2%, 2.7%, and 2.1% over Open-MeDe. On the more challenging SSv2 dataset, SSP consistently outperformed FROSTER across all evaluation metrics, achieving a 1.5% improvement in the Novel setting; and a further 1.7% improvement over Open-MeDe in the HM setting.

[0112] Table 1: Results of the model on the K400 and HMDB-51 datasets

[0113]

[0114] Table 2: Results of the model on the UCF101-24 and SSv2 datasets

[0115]

[0116] Furthermore, to verify the effectiveness of the suggestion mechanism of this invention, this invention demonstrates multiple instances from the K400 dataset, including input videos and attention maps generated by TextSpan, FROSTER, and the SSP proposed in this invention, such as... Figure 3 As shown, the SSP of this invention can accurately highlight key action areas in multiple action categories, such as "golf putting," "spraying paint," and "dancing Macarena." In contrast, the partial attention maps generated by TextSpan and FROSTER appear more scattered and even lack key areas, especially in the first and second columns of "spraying paint," and the second and third columns of "dancing Macarena."

[0117] These visualization results demonstrate that the method of the present invention, benefiting from video-level and token-level spatiotemporal saliency cues, is able to more accurately capture spatiotemporal regions related to actions, thereby improving the understanding of action categories.

[0118] This invention proposes a novel SSP (Spatial-Temporal Saliency Prompting) model for open-lexical action recognition. SSP comprises two core modules: video-level spatiotemporal saliency cues and token-level spatiotemporal saliency cues (T-SSP). These two modules provide saliency cues at different levels and complement each other, thereby enhancing the model's ability to recognize actions. This model can capture instance-level discriminative action regions, significantly improving the ability of visual language models to recognize both base and novel action classes.

[0119] Furthermore, a video-level spatiotemporal saliency cue module was designed to guide pre-trained visual language models (such as CLIP) to focus on important regions related to action execution. Its core is the generation of a video-level spatiotemporal saliency map, which is composed of two mechanisms: cross-modal aligned attention and instance-level self-attention. Cross-modal aligned attention utilizes CLIP's image-text alignment capability, combined with general textual descriptions (such as "a video of a person performing an action"), to generate a coarse-grained, category-agnostic action saliency map, providing initial semantic guidance. Simultaneously, instance-level self-attention extracts a fine-grained instance-level action saliency map from the original video through a lightweight 3D convolutional network, automatically capturing key dynamic regions in the video. These two mechanisms complement each other; by fusing cross-modal aligned attention and instance-level self-attention, key action regions are highlighted while redundant background interference is suppressed, ultimately generating a video-level spatiotemporal saliency cue, which is then injected into the video features to extract discriminative action features.

[0120] Furthermore, this invention proposes a token-level spatiotemporal saliency cue module, introducing instance-adaptive contextual information to enhance the generalization ability of the visual language model for new action categories. Specifically, this invention constructs a model with a teacher-student structure, integrating instance-adaptive contextual information into the visual language model based on this structure. The teacher model uses a frozen CLIP to maintain its original strong generalization ability, while the student model performs controllable fine-tuning of the CLIP. Through knowledge distillation, the teacher model improves the generalization performance of the student model. This includes two key components: learnable cue tokens and instance-level spatiotemporal saliency information. The former models the common semantics across all action categories, while the latter extracts discriminative personalized features from specific instances. The combination of these two significantly improves the model's adaptability to different action instances, enabling the student model to achieve performance improvements on new category recognition tasks while maintaining the generalization advantages of the teacher model.

[0121] In this embodiment of the invention, a novel spatiotemporal saliency cue model is proposed for open-vocabulary action recognition. This model can capture instance-level discriminative action regions, significantly enhancing the ability of visual language models to recognize base and new class actions.

[0122] A video-level spatiotemporal saliency cue module was designed. By fusing cross-modal aligned attention and instance-level self-attention, it effectively extracts discriminative action features, highlights key action regions, and suppresses irrelevant background information.

[0123] A token-level spatiotemporal saliency cue module is introduced, and instance adaptive context is integrated into the visual language model based on a teacher-student architecture, thereby improving the model's generalization ability to new action categories.

[0124] Extensive experiments conducted on multiple open-source action recognition benchmark datasets demonstrate that the SSP model proposed in this invention achieves state-of-the-art performance in this task.

[0125] Figure 4 This is a block diagram illustrating a spatiotemporal saliency cuing device for open-vocabulary video action recognition according to an exemplary embodiment. The device is used in a spatiotemporal saliency cuing method for open-vocabulary video action recognition. (Refer to...) Figure 4 The device includes a data acquisition module 310, a video-level spatiotemporal saliency prompting module 320, and a token-level spatiotemporal saliency prompting module 330. Among them:

[0126] The data acquisition module 310 is used to acquire the input video to be recognized.

[0127] The video-level spatiotemporal saliency cues module 320 is used to input the input video into the video-level spatiotemporal saliency cues module, obtain a coarse-grained saliency map through the cross-modal aligned attention branch of the video-level spatiotemporal saliency cues module, obtain a fine-grained saliency map through the instance-level self-attention branch of the video-level spatiotemporal saliency cues module, and fuse the coarse-grained saliency map and the fine-grained saliency map to obtain the video-level spatiotemporal saliency cues map.

[0128] The Token-level spatiotemporal saliency cue module 330 is used to perform element-wise multiplication of the video-level spatiotemporal saliency cue map with the input video and introduce residual connections to obtain enhanced discriminative action features; inject fine-grained saliency maps into the input video to generate instance-level spatiotemporal action features; and input the input video, enhanced discriminative action features, and instance-level spatiotemporal action features into the Token-level spatiotemporal saliency cue module to obtain open-vocabulary video action recognition results.

[0129] In this embodiment of the invention, a novel spatiotemporal saliency cue model is proposed for open-vocabulary action recognition. This model can capture instance-level discriminative action regions, significantly enhancing the ability of visual language models to recognize base and new class actions.

[0130] A video-level spatiotemporal saliency cue module was designed. By fusing cross-modal aligned attention and instance-level self-attention, it effectively extracts discriminative action features, highlights key action regions, and suppresses irrelevant background information.

[0131] A token-level spatiotemporal saliency cue module is introduced, and instance adaptive context is integrated into the visual language model based on a teacher-student architecture, thereby improving the model's generalization ability to new action categories.

[0132] Extensive experiments conducted on multiple open-source action recognition benchmark datasets demonstrate that the SSP model proposed in this invention achieves state-of-the-art performance in this task.

[0133] Figure 5 This is a schematic diagram of the structure of a spatiotemporal saliency cueing device for open vocabulary video action recognition provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the spatiotemporal saliency cueing device for open-vocabulary video action recognition may include the above-mentioned... Figure 4 The spatiotemporal saliency cueing device for open-vocabulary video action recognition is shown. Optionally, the spatiotemporal saliency cueing device 410 for open-vocabulary video action recognition may include a first processor 2001.

[0134] Optionally, the spatiotemporal saliency cueing device 410 for open vocabulary video action recognition may also include a memory 2002 and a transceiver 2003.

[0135] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0136] The following is combined with Figure 5 The various components of the spatiotemporal saliency cueing device 410 for open-vocabulary video action recognition are described in detail below:

[0137] The first processor 2001 is the control center of the spatiotemporal saliency cues device 410 for open-vocabulary video action recognition. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0138] Optionally, the first processor 2001 can perform various functions of the spatiotemporal saliency cueing device 410 for open vocabulary video action recognition by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0139] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 5 CPU0 and CPU1 are shown in the diagram.

[0140] In a specific implementation, as one example, the spatiotemporal saliency cueing device 410 for open-vocabulary video action recognition may also include multiple processors, for example... Figure 5 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0141] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0142] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the spatiotemporal saliency cueing device 410 for open-vocabulary video action recognition. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0143] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0144] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 5 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0145] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the spatiotemporal saliency cueing device 410 for open-vocabulary video action recognition. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0146] It should be noted that, Figure 5 The structure of the spatiotemporal saliency cueing device 410 for open vocabulary video action recognition shown in the figure does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0147] Furthermore, the technical effects of the spatiotemporal saliency cueing device 410 for open vocabulary video action recognition can be referred to the technical effects of the spatiotemporal saliency cueing method for open vocabulary video action recognition described in the above method embodiments, and will not be repeated here.

[0148] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0149] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0150] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0151] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0152] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0153] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0154] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0156] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0159] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A spatiotemporal saliency cues method for open-vocabulary video action recognition, characterized in that, The method includes: S1. Obtain the input video to be recognized; S2. Input the input video to the video-level spatiotemporal saliency cue module. Obtain a coarse-grained saliency map through the cross-modal aligned attention branch of the video-level spatiotemporal saliency cue module; obtain a fine-grained saliency map through the instance-level self-attention branch of the video-level spatiotemporal saliency cue module; fuse the coarse-grained saliency map and the fine-grained saliency map to obtain the video-level spatiotemporal saliency cue map. S3. Multiply the video-level spatiotemporal saliency cue map element-wise with the input video and introduce residual connections to obtain the enhanced discriminative action features; S4. Inject fine-grained saliency maps into the input video to generate instance-level spatiotemporal action features; S5. Input the input video, the enhanced discriminative action features, and the instance-level spatiotemporal action features into the Token-level spatiotemporal saliency cue module to obtain the open-vocabulary video action recognition results.

2. The spatiotemporal saliency cues for open-vocabulary video action recognition according to claim 1, characterized in that, The S2 includes: S21. Input the input video into the cross-modal aligned attention branch of the video-level spatiotemporal saliency cue module to generate a category-independent coarse-grained saliency map; S22. Input the input video into the instance-level self-attention branch of the video-level spatiotemporal saliency cues module to extract a fine-grained saliency map that reflects local dynamic changes; S23. The coarse-grained saliency map and the fine-grained saliency map are concatenated along the feature dimension and input into the fusion network to obtain the fused saliency map. S24. The fused saliency map is used to obtain a video-level spatiotemporal saliency cue map through the Sigmoid activation function.

3. The spatiotemporal saliency cues for open-vocabulary video action recognition according to claim 2, characterized in that, The cross-modal alignment attention branch utilizes the text-image alignment capability of the CLIP model and generates a category-independent coarse-grained saliency map through guiding text.

4. The spatiotemporal saliency cues for open-vocabulary video action recognition according to claim 2, characterized in that, The instance-level self-attention branch includes a lightweight 3D convolutional network module; wherein, the lightweight 3D convolutional network module consists of multiple 3D convolutional layers, BatchNorm layers, and ReLU layers.

5. The spatiotemporal saliency cues for open-vocabulary video action recognition according to claim 2, characterized in that, The fusion network consists of two 3D convolutional layers, a BatchNorm layer, and a ReLU layer.

6. The spatiotemporal saliency cues for open-vocabulary video action recognition according to claim 1, characterized in that, The S5 includes: S51. Input instance-level spatiotemporal action features into a lightweight 3D convolutional network module and an MLP module to extract instance adaptive cue tokens; S52. Introduce a teacher model learnable token and a student model learnable token. Add the teacher model learnable token and the student model learnable token to the instance adaptive prompt token respectively to form the teacher model context prompt information and the student model context prompt information. S53. Input the contextual prompts and input video of the teacher model into the teacher model to obtain the video embedding representation and text embedding representation of the teacher model; input the contextual prompts and input video of the student model into the student model to obtain the video embedding representation and text embedding representation of the student model. S54. Input the enhanced discriminative action features into the student model to obtain the enhanced discriminative video embedding representation; S55. Match the student model video embedding representation and the enhanced discriminative video embedding representation with the student model text representation, and obtain the prediction results of the two action categories by calculating the dot product. S56. The prediction results of the two action categories are weighted and fused, and the open vocabulary video action recognition results are output through the softmax function. S57. Based on the video embedding representation and text embedding representation of the teacher model, the video embedding representation and text embedding representation of the student model, and the enhanced discriminative video embedding representation, construct the classification loss and feature distillation loss, and perform joint optimization based on the classification loss and feature distillation loss to obtain the fine-tuned student model.

7. The spatiotemporal saliency cues method for open-vocabulary video action recognition according to claim 6, characterized in that, The classification loss is calculated using cross-entropy loss based on the student model's video embedding representation and text embedding representation, as well as the extracted video embedding representation and text embedding representation. Feature distillation loss includes visual distillation loss and text distillation loss; visual distillation loss is calculated by comparing the feature differences between the teacher model video embedding representation, the student model video embedding representation, and the enhanced discriminative video embedding representation; text distillation loss is calculated by comparing the feature differences between the teacher model text embedding representation and the student model text embedding representation.

8. A spatiotemporal saliency prompting device for open-vocabulary video action recognition, wherein the spatiotemporal saliency prompting device for open-vocabulary video action recognition is used to implement the spatiotemporal saliency prompting method for open-vocabulary video action recognition as described in any one of claims 1-7, characterized in that, The device includes: The data acquisition module is used to acquire the input video to be recognized; The video-level spatiotemporal saliency cueing module is used to input the video into the video-level spatiotemporal saliency cueing module. A coarse-grained saliency map is obtained through the cross-modal aligned attention branch of the video-level spatiotemporal saliency cueing module; a fine-grained saliency map is obtained through the instance-level self-attention branch of the video-level spatiotemporal saliency cueing module; and the coarse-grained saliency map and the fine-grained saliency map are fused to obtain the video-level spatiotemporal saliency cueing map. The Token-level spatiotemporal saliency cue module is used to perform element-wise multiplication of the video-level spatiotemporal saliency cue map with the input video and introduce residual connections to obtain enhanced discriminative action features; a fine-grained saliency map is injected into the input video to generate instance-level spatiotemporal action features; the input video, enhanced discriminative action features, and instance-level spatiotemporal action features are input into the Token-level spatiotemporal saliency cue module to obtain the open-vocabulary video action recognition result.

9. A spatiotemporal saliency cueing device for open-vocabulary video action recognition, characterized in that, The spatiotemporal saliency cueing device for open-vocabulary video action recognition includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images