Motion recognition method based on time sequence decoupling of CLIP

Abstract

The invention discloses an action recognition method based on time sequence decoupling of CLIP. The method comprises the following steps: collecting a video image of a target area; and inputting the video image into the trained action recognition model to obtain an action recognition result. Wherein the action recognition model comprises a text decoding branch and an image decoding branch, the image decoding branch is provided with an instance perception time sequence decoupling module used for decomposing frame-level features into content-based features and style residual features, and the instance perception time sequence decoupling module comprises an instance normalization module and a double-time-sequence global-local attention module. The embodiment normalization module achieves style normalization operation of input video frames, the double-time-sequence global-local attention module conducts feature decoupling on style residual features, cross-frame long-range dependence is modeled through global time sequence attention, and fine-grained correlation of adjacent frames is captured through local time sequence attention. According to the method, effective balance between supervision and generalization is realized, and the accuracy and robustness of video action recognition are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence application technology, and more specifically, to a CLIP-based action recognition method for temporal decoupling. Background Technology

[0002] Visual language models consist of an image encoder and a text encoder. They learn multimodal representations through large-scale image-text pre-training, thereby learning a common feature space between image and text labels. For example, CLIP (Contrastive Language-Image Pre-training, a text-image pre-training model using contrastive learning) and ALIGN models have zero-shot learning performance and generalization ability. Research based on models such as CLIP can improve the performance of downstream tasks by leveraging pre-trained models. For instance, CoOp, CLIP-Adapter, and TipAdapter use pre-trained CLIP models to improve transfer capabilities for few-shot tasks, PointCLIP transfers CLIP to point cloud understanding tasks, and DenseCLIP transfers knowledge to dense prediction tasks.

[0003] Currently, there are several CLIP-based video recognition methods, which can generally be divided into full fine-tuning and partial fine-tuning approaches. In full fine-tuning, Action-CLIP applies CLIP to the video action recognition field, adding extra temporal layers to learn temporal information in addition to fine-tuning the CLIP model. STAN uses an auxiliary network for temporal feature extraction. ViFi-CLIP and Text4Vis only fine-tune CLIP without adding extra components to learn temporal information. Partial fine-tuning primarily freezes the parameters of the pre-trained CLIP model by using additional parameters for training. Most existing CLIP-based video action recognition models are implemented using partial fine-tuning, aiming to achieve a balance between temporal accuracy and zero-shot performance. For example, AdaptFormer proposes a lightweight module, AdaptFormer, for efficiently adapting pre-trained visual Transformers to various image and video recognition tasks, significantly improving model performance while maintaining low parameter tuning. ST-Adapter, through a lightweight adapter, freezes the main parameters of the pre-trained model while introducing a small number of trainable parameters to adapt to video tasks, balancing model performance and computational efficiency. X-CLIP uses a cross-frame communication Transformer and a multi-frame ensemble Transformer to handle inter-frame information exchange and video-level feature aggregation, respectively. AIM proposes a method to efficiently adapt pre-trained image models to video action recognition tasks using a lightweight adapter, achieving excellent performance while maintaining low parameter tuning. M2-CLIP freezes the CLIP backbone and adds an adapter after the transformer, improving the temporal representation capability of the video encoder and enhancing semantic label information learning.

[0004] A key factor in video action recognition models is how to utilize temporal information. Early deep learning methods used 2D convolution plus optical flow to learn temporal information through changes in optical flow. 3D convolution-based video action recognition can directly extract spatiotemporal features to capture inter-frame temporal dependencies. However, due to the limited receptive field of convolutional networks (CNNs), they cannot handle long-term temporal information. Subsequent research has gradually replaced convolution with transformer-based methods. For example, TimeTransformer reduces computation by decomposing spatiotemporal attention through distributed attention computation. The ViViT model employs various strategies to process the temporal and spatial information of videos. The Video Swin Transformer model extends the design of SwinTransformer by introducing a hierarchical Transformer architecture to process videos, further reducing computation. The transformer-based semi-supervised video action recognition model SVFormer reduces reliance on dataset annotations. Recent research based on Mamba, such as Video Mamba, addresses local redundancy and global dependencies in video understanding and overcomes the limitations of transformers in modeling long videos.

[0005] To bridge the gap between images and videos, researchers have proposed many novel methods for CLIP. Some schemes employ cue learning, proposing cue-based learning schemes on either the visual or text side and adding additional temporal layers to learn temporal information, such as using cross-frame communication and multi-frame ensembles to handle temporal modeling relationships. Other schemes focus on end-to-end CLIP fine-tuning. However, these schemes either fail to learn effectively on both the text and visual sides, or suffer from degraded zero-shot generalization ability of CLIP due to fine-tuning the visual decoder or backbone network. Although recent works have achieved a good balance between zero-shot and fully supervised learning through full CLIP fine-tuning, the lack of temporal components results in poor performance for some fine-grained action recognition tasks. Furthermore, there are efficient fine-tuning methods that freeze the CLIP backbone, adding an adapter after the CLIP visual-text transformer layer to learn temporal and textual information with a lower parameter count, and finally using a multi-task decoder to improve the overall learning ability of the framework. However, these methods typically fail to achieve an effective balance between supervision and generalization. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a CLIP-based action recognition method for temporal decoupling. This method includes:

[0007] Acquire video images of the target area;

[0008] The video image is input into a trained action recognition model to obtain action recognition results;

[0009] The action recognition model is a CLIP-based dual-temporal decoupling network, comprising a text decoding branch and an image decoding branch. The image decoding branch includes an instance-aware temporal decoupling module for decomposing frame-level features into content-based features and style residual features. This instance-aware temporal decoupling module includes an instance normalization module and a dual-temporal global-local attention module. The instance normalization module performs style normalization on the input video frames, and the dual-temporal global-local attention module decouples the style residual features. It models long-range dependencies across frames through global temporal attention and captures fine-grained relationships between adjacent frames through local temporal attention.

[0010] Compared with existing technologies, the advantages of this invention are that, addressing the problems of insufficient utilization of supervision signals and limited cross-domain generalization ability of CLIP models in video action recognition tasks, this invention proposes a CLIP-based dual-temporal decoupled network for video action recognition. This optimizes the supervised learning process and significantly improves the model's generalization ability to unknown video scenes and action categories. This invention achieves an effective balance between supervision and generalization, and improves the accuracy and robustness of video action recognition.

[0011] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0013] Figure 1 This is a flowchart of a CLIP-based action recognition method for time-series decoupling according to an embodiment of the present invention;

[0014] Figure 2 This is a schematic diagram of a CLIP-based action recognition method for time-series decoupling according to an embodiment of the present invention.

[0015] Figure 3 This is a comparison chart of experimental results of different video action recognition models according to an embodiment of the present invention on the HMDB51 dataset;

[0016] In the attached diagram, MS represents maximum similarity; BN represents batch normalization; FC represents fully connected components; IN represents instance normalization; BDR represents boundary-aware double recovery loss; and Pool represents temporal pooling. Detailed Implementation

[0017] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0018] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0019] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0020] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0021] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0022] This invention constructs a CLIP-based DualTemporal Decoupling Network for Video Action Recognition (TDN-CLIP) as an action recognition model, which maintains supervision and improves generalization. The Instance-aware Temporal Decoupling Module (ITDM) decomposes the frame-level features output by CLIP into content-based features and style residual features to eliminate style differences in video frames (e.g., effectively eliminating lighting, background, and other styles). The style residual features are then sent to a Dual Temporal Global-Local Attention (DTA) module for feature decoupling. Global Temporal Attention (GTA) models long-range dependencies across frames, while Local Temporal Attention (LTA) captures fine-grained correlations between adjacent frames. Finally, global-local temporal collaborative attention recalibrates the residual features, decoupling them into features useful and useless to the action features. By fusing decoupled useful and useless features with content-based features, a highly discriminative video representation is formed, preserving its cross-domain invariant action semantics and improving its generalization across different datasets. Furthermore, during the action recognition model training process, Boundary-Aware Dual-Recovery Contrastive Loss (BAD-RCL) is used to enhance its judgment between irrelevant and relevant action features and its reconstruction of inter-frame features. Hard-margin constraints force the entropy value of useful features to be significantly lower than that of baseline features, ensuring that key regions are not ignored, while soft-margin constraints handle blurred action boundaries, avoiding excessive suppression of useful features.

[0023] Specifically, in combination Figure 1 and Figure 2 As shown, the provided CLIP-based temporal decoupling action recognition method includes the following steps:

[0024] Step S110: Construct a CLIP-based dual temporal decoupling network as an action recognition model. The model includes a text decoding branch and an image decoding branch. The image decoding branch is equipped with an instance-aware temporal decoupling module, which is used to perform style normalization on the input video frames and decouple the style residual features using global temporal attention and local temporal attention.

[0025] Figure 2This is a structural diagram of the action recognition model, which generally includes a text encoder, an image encoder, and an instance-aware temporal decoupling module (ITDM). The instance-aware temporal decoupling module contains an instance normalization module (IN) and uses a dual temporal global-local attention module (DTA) for feature decoupling. The dual temporal global-local attention module includes global temporal attention (GTA) for modeling long-range dependencies across frames and local temporal attention (LTA) for capturing fine-grained correlations between adjacent frames.

[0026] Instance normalization (IN) performs a style normalization operation to reduce differences between samples, which helps enhance the network's generalization ability. In this embodiment of the invention, instance normalization is used to reduce instance style differences between input video frames, expressed as:

[0027]

[0028] in, This represents the video frame input at time t. This represents the normalized video frame. i, j, and t represent the i-th sample, the j-th channel, and the t-th frame, respectively. and It calculates the mean and variance of each video frame independently, γ. j β and β are parameters learned from the data. Considering that simply using instance normalization might filter out some important information, thus weakening discriminative ability, we extract and recover task-specific discriminative features from the information removed by instance normalization. Then, we decompose these features into task-related and task-independent features using Boundary-Aware Dual-Recovery Contrastive Loss (BAD-RCL).

[0029] Specifically, while instance normalization removes style differences, it may also remove some important feature information. To overcome this shortcoming, residual features are used. Extract category-related features and restore them to the network. Represented as:

[0030]

[0031] Then, Feature decoupling is achieved through a learnable channel-based temporal attention mechanism. The channel-based temporal attention mechanism is... (D is the dimension of each feature, b1 represents the temporal attention of the first channel), using b to... Decoupled into two parts, namely features related to action characteristics. (B represents the batch size, and T represents the number of video frames) and features unrelated to action features. They are represented as follows:

[0032]

[0033] in, The j-th channel represents the feature. Features related to the action are... With normalized features Adding them together yields the action-related feature Z for output enhancement. + , is represented as:

[0034]

[0035] Features unrelated to action category With normalization characteristics The summation yields the suppressed category-independent action feature Z. - , is represented as:

[0036]

[0037] Z + Perform global average pooling along the time dimension to obtain features. Represented as:

[0038] v = AvgPool(Z) +) (7)

[0039] The video-level feature v and the text feature are compared using cosine similarity calculation. For example, the loss function is set as follows:

[0040]

[0041] in, Let represent the embedding vector of the i-th video feature. Let represent the text embedding vector corresponding to the correct category of the i-th video. Let represent the text embedding vector of the j-th category, sim represent the cosine similarity, and τ represent the temperature coefficient. Normalize the features... And the suppressed irrelevant feature Z - It also performs average pooling along the time dimension, and then passes it through a fully connected layer to transform the dimension into the number of action categories, represented as:

[0042]

[0043] v vuseful =softmax(liner(AvgPool(Z)) + ))) (10)

[0044] v useless =softmax(liner(AvgPool(Z)) - (11)

[0045] Among them, v base V represents the features after normalization. useful Represents enhanced category-related features, v useless This represents suppressing category-independent features, and liner represents a fully connected layer.

[0046] Step S120: Train the action recognition model using the set boundary-aware dual-recovery contrastive loss function.

[0047] In one embodiment, a boundary-aware dual-recovery contrastive loss (BAD-RCL) is used to train the action recognition model. Specifically, to facilitate the distinction between action-related and action-independent features, a boundary-aware dual-recovery contrastive loss is designed by comparing the discriminative power of features before and after recovery. This loss method is used to distinguish between identity-related features. Restore to normalized features The ability to distinguish actions after a symptom will be enhanced, and features irrelevant to the action will be recognized. Restore to normalized features The discriminative power of post-features will weaken. The main idea is to bring positive samples closer and push negative samples further away. This loss mainly consists of two parts: hard margin constraint and soft margin constraint. The hard margin constraint forces the entropy value of useful features to be significantly lower than that of the baseline features, ensuring that the network model focuses on key action segments. For example, the formula for the hard margin constraint is as follows:

[0048]

[0049] Where margin is a pre-defined positive threshold number. Soft-margin constraints handle fuzzy action boundaries, avoiding excessive suppression of potentially useful features. The formula is as follows:

[0050]

[0051] in, L represents the soft-split constraint loss. hard L represents the hard-interval constraint loss. soft1 L represents the soft constraint for enhancing features. soft2 The soft constraint representing the inhibition feature, v useful，j V represents the enhancement features of the j-th frame. base,j V represents the normalized feature of the j-th frame. useless,j This represents the suppression feature of the j-th frame.

[0052] The BAD-RCL loss is:

[0053]

[0054] Here, α is a hyperparameter that can be determined according to actual needs or simulation. For example, α can be set to 0.1 or other values.

[0055] Global attention enhances motion-related channels and suppresses background and noise channels, adaptively extracting motion-related features for recovery. Local attention primarily aims to capture short-term motion models, enhancing climactic frames and mitigating the influence of irrelevant frames. Fusing global and local attention enables interaction between the two; global attention provides global feature selection, while local attention supplements local motion enhancement, thereby achieving enhanced response at keyframes and suppressing redundant spatiotemporal regions. For example, the global attention formula is expressed as:

[0056] b global =σbig(W c2 ·δ(W c1 GAP T (S))) (17)

[0057] Among them, b global Indicates global attention. Global average pooling along the time dimension. 1x1 convolutions for dimensionality reduction and expansion, where r represents the scaling factor, δ represents the GELU activation function, and σ represents the Sigmoid activation function. The local attention formula is expressed as:

[0058] b local =σ(W t GAP T big(δ(BN(DWConv(S))))) (18)

[0059] Among them, b local Local attention is represented by DWConv, one-dimensional convolution by DWConv, batch normalization by BN, and GAP by GAP. T Average pooling over time, W t This represents the learnable weight matrix. Finally, the local and global temporal attentions are summed, with the following formula:

[0060]

[0061] Where λ∈[0,1]. b is the feature obtained by multiplying each video frame.

[0062] Step S130: For the actual acquired video images, obtain the action recognition results using the trained action recognition model.

[0063] After the action recognition model is trained, optimized model parameters, such as weights and biases, can be obtained. During model application, the collected video images of the target area are input into the action recognition model for instance normalization and dual temporal feature decoupling to obtain video representations. Based on these video representations, action recognition results are then obtained. The action recognition model of this invention can be widely applied in fields such as security monitoring, traffic management, smart homes, entertainment recommendations, virtual reality, augmented reality, and robotics.

[0064] To further verify the effectiveness of the present invention, zero-shot, base-to-novel, and few-shot experiments were conducted on four public datasets: UCF-101, HMDB-51, Kinetics-400 (300,000 videos), and Something-Something V2 (220,000 videos). Figure 3 This is a comparison of zero-shot experimental results on the HMDB51 dataset. The experimental results show that the generalization and supervised performance of this invention are significantly improved, and it achieves state-of-the-art (SOTA) performance on multiple tasks.

[0065] In summary, this invention is the first to apply the concept of decoupling to video action recognition and CLIP. By designing an ITDM module to enhance video frames, it decomposes the frame-level features output by CLIP into content-based features and style residual features to eliminate style differences within video frames. Furthermore, it utilizes DTA to learn inter-frame temporal information and decouple video frame features. Global temporal attention models long-range dependencies across frames, while local temporal attention captures fine-grained correlations between adjacent frames for temporal information learning and feature enhancement. In addition, during model training, BDR-Loss enhancement is designed to distinguish and recover features between irrelevant and relevant action features. Compared to existing technologies, this invention uses a visual language model as the backbone network, not only transferring CLIP to the video domain but also enhancing its ability to discriminate fine-grained actions, improving temporal information learning, and increasing generalization. This invention not only fully utilizes the good generalization and robustness of the visual language model but also improves the accuracy of action recognition and reduces errors in various application scenarios.

[0066] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0067] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0068] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0069] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0070] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0071] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0072] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0073] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0074] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A CLIP-based action recognition method with temporal decoupling, comprising the following steps: Acquire video images of the target area; The video image is input into a trained action recognition model to obtain action recognition results; The action recognition model is a CLIP-based dual-temporal decoupling network, comprising a text decoding branch and an image decoding branch. The image decoding branch includes an instance-aware temporal decoupling module for decomposing frame-level features into content-based features and style residual features. This instance-aware temporal decoupling module includes an instance normalization module and a dual-temporal global-local attention module. The instance normalization module performs style normalization on the input video frames, and the dual-temporal global-local attention module decouples the style residual features. It models long-range dependencies across frames through global temporal attention and captures fine-grained relationships between adjacent frames through local temporal attention.

2. The method according to claim 1, characterized in that, The instance normalization module performs style normalization according to the following formula: in, This represents the video frame input at time t. This represents the normalized video frame, where i, j, and t represent the i-th sample, the j-th channel, and the t-th frame, respectively. and These are the mean and variance of the video frames, respectively, and γ. j β and β are the learned parameters, and IN represents the instance normalization operation.

3. The method according to claim 2, characterized in that, The action recognition model is trained based on the following loss function: in: in, It is a boundary-aware dual-recovery contrast loss. L represents the soft-split constraint loss. hard L represents the hard-interval constraint loss. soft 1 L represents the soft constraint for enhancing features. soft 2 The soft constraint representing the inhibition feature, v useful,j U represents the enhancement feature of the j-th frame. base,j V represents the normalized feature of the j-th frame. useless,j This represents the suppression feature of the j-th frame, where margin is a pre-set positive threshold, α is a hyperparameter, and M represents the number of frames.

4. The method according to claim 3, characterized in that, The enhancement feature and the suppression feature are uniformly represented according to the following formula: v useful =softmar(liner(AvgPool(Z + ))) v useless =softmax(liner(AvgPool(Z - ))) in: Among them, Z + It is an enhanced action-related feature, Z - It is a category-independent action feature, v useful S represents the enhanced category-related features. + It is a feature related to action characteristics, S - It is a feature unrelated to action characteristics, v useless This represents suppressing category-independent features, liner represents a fully connected layer, and AvgPool represents average pooling.

5. The method according to claim 4, characterized in that, The global temporal attention and the local temporal attention are calculated according to the following formulas: b global =σbig(W c2 ·δ(W c1 ·GAP T (S))) b lobal =σ(W t ·GAP T big(δ(BN(DWConv(S)))) Among them, b global Indicates global attention, b local GAP represents local attention. T W represents global average pooling along the time dimension. c1 It is a 1x1 convolution with reduced dimensionality, W c2 It is a 1x1 convolution with increased dimensionality, where r represents the scaling factor, δ represents the GELU activation function, σ represents the Sigmoid activation function, SWConv represents one-dimensional convolution, BN represents batch normalization, and GAP represents... T Average pooling over time, W t This represents the learnable weight matrix. It is a residual characteristic.

6. The method according to claim 5, characterized in that, The global attention and the local attention are fused according to the following formula: Where λ∈[0,1] is a set parameter.

7. The method according to claim 5, characterized in that, The residual characteristic S is calculated according to the following formula: in, This represents the video frame input at time t. This represents the video frame after instance normalization.

8. The method according to claim 5, characterized in that, The residual feature S is decoupled through learnable channel temporal attention to obtain... and 9. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images