An action recognition method and system based on global-local adaptive guidance and video guided motion aggregation

Abstract

This invention discloses an action recognition method and system based on global-local adaptive guidance and video-guided motion aggregation. The method constructs an action recognition model to perform action recognition on the test video. In the action recognition model, the test video is processed by a video backbone encoder and a visual guidance encoder, respectively. A global-local adaptive guidance module performs multi-scale enhancement and cross-granularity attention aggregation on the global and local block labels output by the visual guidance branch to obtain frame-level guidance features. A video-guided motion aggregation module performs fine-grained and coarse-grained explicit motion mining on the video sequence labels output by the video backbone encoder, and uses the frame-level guidance features to perform gating enhancement on the motion information. Then, spatial pooling and temporal pooling are used to obtain frame-level action representations and video-level semantic representations. The output of the video-guided motion aggregation module is further processed and classified to obtain the action recognition result.

Description

Technical Field

[0001] This invention belongs to the field of video understanding and deep learning technology, specifically relating to an action recognition method and system based on global-local adaptive guidance and video-guided motion aggregation. Background Technology

[0002] Action recognition is one of the core tasks in video understanding, and it is widely used in scenarios such as intelligent security, sports analysis, behavior retrieval, human-computer interaction, and intelligent monitoring. Existing methods typically adopt three technical approaches: one is based on two-dimensional convolution and temporal aggregation, another is based on three-dimensional convolution, and the third is based on video Transformer.

[0003] While existing video Transformer-based methods possess strong global modeling capabilities, they still suffer from the following problems in practical applications:

[0004] 1. When the backbone network encodes block tags uniformly, it is easy to ignore local detail areas that are closely related to action discrimination, resulting in insufficient differentiation between key action areas and background areas.

[0005] 2. Existing guidance branches often only use frame-level global semantics and fail to make full use of the local block-level structural information within each frame, making it difficult to effectively couple global semantics and local details.

[0006] 3. Many video classification methods use simple average pooling or coarse aggregation to compress the video backbone output, which weakens cross-frame motion relationships and local displacement patterns, making it difficult to recognize fine-grained actions and complex interactive actions.

[0007] 4. Some motion enhancement solutions rely on additional optical flow, pose key points, or external supervision information, which increases training and deployment costs.

[0008] Therefore, a new action recognition method is needed that, without relying on additional motion supervision, enhances the ability of global semantics to guide local regions and the ability to discover explicit motion patterns, so as to improve the accuracy and robustness of action recognition models in complex scenarios. Summary of the Invention

[0009] To address the limitations of existing video Transformer-based action recognition methods, such as the backbone network's tendency to overlook closely related local details when uniformly encoding block labels, the guide branch's reliance on frame-level global semantics without fully utilizing local block-level structural information, and the weakening of cross-frame motion relationships and local displacement patterns due to simple average pooling, this invention proposes an action recognition method and system based on global-local adaptive guidance and video-guided motion aggregation. This method effectively improves action recognition accuracy and robustness while eliminating the need for additional optical flow or keypoint supervision, thus reducing training and deployment overhead.

[0010] In a first aspect, the present invention provides an action recognition method based on global-local adaptive guidance and video-guided motion aggregation, the method comprising:

[0011] Acquire the video to be tested; preprocess the video to obtain the video frame sequence;

[0012] The video frame sequence is input into the constructed action recognition model for action recognition; the action recognition model includes a video backbone encoder, a visual guidance encoder, a global-local adaptive guidance module, a video guidance motion aggregation module, a temporal position encoder, a category text embedding module, a Transformer decoder, and a classification head;

[0013] The video backbone encoder is used to extract video sequence markers from the video frame sequence;

[0014] The visual guidance encoder is used to extract global and local block markers from the video frame sequence; the global-local adaptive guidance module processes the global and local block markers through global and local processing branches respectively, and introduces an attention mechanism in the fusion branch to aggregate the processing results of the global and local processing branches to obtain frame-level guidance features.

[0015] The video-guided motion aggregation module is used to fuse video sequence markers and frame-level guided features to obtain the final frame-level action representation and the final video-level semantic representation.

[0016] The temporal position encoder is used to perform temporal position encoding on the final frame-level action representation; the category text embedding module is used to embed the action category into the final video-level semantic representation to obtain the category query vector.

[0017] The Transformer decoding module uses the final frame-level action representation after temporal position encoding as the source sequence, and the category query vector as the target sequence for interactive decoding. The decoding result is input to the classification head, and the predicted score corresponding to different action categories is output. After threshold determination, the final multi-label action recognition result is obtained, realizing action recognition of the tested video.

[0018] Preferably, the video-guided motion aggregation module includes fine-grained motion branches, coarse-grained motion branches, a multi-scale fusion unit, a guidance gating unit, a spatial pooling unit, and a temporal pooling unit. In the video-guided motion aggregation module, video sequence markers are split into classification markers and block markers, and the block markers are converted into grid markers. The grid markers are processed through fine-grained and coarse-grained motion branches respectively to obtain appearance enhancement features and explicit motion features corresponding to the fine-grained and coarse-grained motion branches. The multi-scale fusion unit fuses the outputs of the fine-grained and coarse-grained motion branches to obtain comprehensive appearance enhancement features and comprehensive explicit motion features. The guidance gating unit fuses frame-level guidance features and comprehensive explicit motion features to obtain explicit motion guidance features. The spatial pooling unit fuses explicit motion guidance features, block markers, and comprehensive appearance enhancement features, and then further processes them to obtain the final frame-level action representation. The temporal pooling unit fuses explicit motion guidance features and frame-level guidance features, and then further processes them to obtain the final video-level semantic representation.

[0019] Preferably, both the fine-grained motion branch and the coarse-grained motion branch introduce local motion modeling units to process the mesh markers. In the local motion modeling unit, after layer normalization of the mesh markers, the mesh markers of the current frame are linearly mapped to obtain query features, and the mesh markers of the next frame are linearly mapped to obtain key features and value features respectively. Dimensional permutation and reshaping operations are performed on the key features and value features respectively, transforming them into a two-dimensional image format. Local windows are extracted centered on each spatial location through an unfolding operation, and then reshaped into a multi-head format to form local window key features and local window value features. Relevance scores are calculated based on query features and local window key features. On the one hand, the relevance scores are normalized and weighted summed with the local window value features, and then reshaped and linearly mapped to obtain appearance enhancement features. On the other hand, the relevance scores are flattened and then subjected to layer normalization and multi-layer mapping to obtain explicit motion features.

[0020] Preferably, in the spatial pooling unit, block labels, comprehensive appearance enhancement features, and explicit motion guidance features are fused to obtain fused block labels; after concatenating the fused block labels with the explicit motion guidance features, layer normalization and multi-layer linear mapping are performed to obtain weight scores; the weight scores are normalized to obtain spatial weights; the fused block labels are weighted and summed using the spatial weights to obtain frame-level action representations; after injecting classification labels into the frame-level action representations, layer normalization is performed to obtain the final frame-level action representation.

[0021] As a preferred approach, in the temporal pooling unit, the explicit motion guidance features are averaged in the spatial dimension, and the result is concatenated with the frame-level guidance features and then processed by layer normalization and linear mapping to obtain a temporal weight score. Softmax normalization is performed on the temporal weight score to obtain the temporal weight. The frame-level guidance features are weighted and summed using the temporal weight to obtain the video-level semantic representation. The video-level semantic representation is then processed by layer normalization to obtain the final video-level semantic representation.

[0022] Preferably, in the guidance gating unit, the frame-level guidance features are processed sequentially through layer normalization, linear mapping, and activation functions to generate channel gating weights; the channel gating weights are then multiplied element-wise with the comprehensive explicit motion features to obtain explicit motion guidance features.

[0023] Preferably, the global processing branch sequentially performs linear mapping, layer normalization, nonlinear activation, and reshaping operations on the global markers to obtain query features; the local processing branch processes local block markers through a multi-scale local enhancement submodule to obtain local enhancement markers; on one hand, the local enhancement markers are bottleneck mapped and then multiplied element-wise with the local enhancement markers to obtain key features; on the other hand, the local enhancement markers are linearly mapped to obtain value features; the fusion branch performs additive attention calculation on the query features and key features, and performs normalization along the spatial block dimension to obtain attention weights; the attention weights are applied to the value features and weighted summation is performed along the spatial block dimension to obtain frame-level context features; the frame-level context features are residually concatenated with the global markers and then layer normalization is performed to obtain frame-level guiding features.

[0024] Preferably, in the multi-scale local enhancement submodule, layer normalization, reshaping, and convolution operations are performed sequentially on the local block labels to obtain a compressed feature map; multiple parallel depthwise convolutions are used to process the compressed feature map, and the processing results are fused and then sequentially convolution and reshaping are performed to obtain enhanced features; the enhanced features are injected into the local block labels in the form of residuals to obtain local enhanced labels.

[0025] Preferably, the video backbone encoder uses a pre-trained TimeSformer network; the visual guidance encoder uses a pre-trained CLIP visual coding network; the temporal position encoder uses sine-cosine position coding to inject temporal position information into the frame-level action representation; the category text embedding module includes a concatenated text encoder and a category semantic query construction module; the text encoder uses a pre-trained CLIP text coding network; the text encoder is used to generate category text embeddings based on predefined action categories; the category semantic query construction module is used to fuse the category text embeddings with the final video-level semantic representation to obtain a category query vector.

[0026] Secondly, the present invention provides an action recognition system based on global-local adaptive guidance and video-guided motion aggregation, which is used to execute the above-mentioned action recognition method; the action recognition system includes a data acquisition unit, an image preprocessing unit, and an action recognition unit; the data acquisition unit is used to acquire the video under test; the image preprocessing unit is used to preprocess the video under test; the action recognition unit is used to store the action recognition model and use the action recognition model to perform action recognition on the input video frame sequence.

[0027] Thirdly, the present invention provides a computer device; the device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described action recognition method.

[0028] Fourthly, the present invention provides a computer-readable storage medium; the medium stores a computer program, which, when executed by a processor, implements the above-described action recognition method.

[0029] Fifthly, the present invention provides a computer program product; the product includes a computer program / instruction that, when executed by a processor, implements the above-described action recognition method.

[0030] The beneficial effects of this invention are:

[0031] 1. In the global-local adaptive guidance module, the present invention constructs queries with global tags and constructs keys and values ​​with enhanced local block tags. It achieves cross-granularity information convergence through additive attention and element-wise aggregation, enabling the guidance features of each frame to simultaneously perceive global semantics and local action details, realizing deep coupling between global semantics and local details. It is especially suitable for action recognition scenarios where there are significant local changes in human body parts and complex backgrounds.

[0032] 2. This invention employs a multi-scale local enhancement submodule in the global-local adaptive guidance module. This module uses 12 parallel 3D deep convolutional branches to cover a combination of 3 spatial dilation rates and 4 temporal dilation rates. It adaptively adjusts the importance of local motion modes at different scales through learnable fusion weights and controls the injection intensity of enhancement features through learnable gating coefficients. This elevates the guidance information from "coarse semantic cues" to "local key region expression under global semantic constraints," enhancing the important information of moving parts such as the face and hands.

[0033] 3. In the video-guided motion aggregation module, this invention employs a dual-branch structure of fine-grained and coarse-grained motion branches to simultaneously perform local cross-frame correlation modeling at different spatial resolutions on a block-level spatial grid. This approach balances local detailed motion with larger-scale displacement patterns, improving adaptability to complex actions, occluded scenes, and multi-subject interaction scenarios. It enables multi-scale motion mining, accurately models the spatiotemporal evolution characteristics of actions while reducing computational overhead, and provides an efficient solution for end-to-end action recognition.

[0034] 4. This invention utilizes the frame-level guidance features output by the global-local adaptive guidance module to perform channel-gated modulation on explicit motion features, so that the enhancement of motion information is guided by global semantics and local key regions, effectively suppressing background interference and noise motion.

[0035] 5. In the video-guided motion aggregation module, this invention achieves hierarchical filtering from frame-level key block selection to video-level key frame selection through motion-aware spatial pooling and temporal pooling, effectively suppressing background interference and redundant frame interference, and avoiding information loss caused by traditional simple average pooling. Attached Figure Description

[0036] Figure 1 This is a structural diagram of the action recognition model in this invention.

[0037] Figure 2 This is a structural diagram of the global-local adaptive guidance module in this invention.

[0038] Figure 3 This is a structural diagram of the multi-scale local enhancement submodule in this invention.

[0039] Figure 4 This is a structural diagram of the video-guided motion aggregation module in this invention.

[0040] Figure 5 This is a structural diagram of the local motion modeling unit in this invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings.

[0042] Example 1

[0043] An action recognition method based on global-local adaptive guidance and video-guided motion aggregation is proposed. The action recognition system includes a data acquisition unit, an image preprocessing unit, a training set construction unit, a training unit, and an action recognition unit. The data acquisition unit acquires video frame sequences. The image preprocessing unit preprocesses the images in each video frame sequence to obtain preprocessed image sequences. The training set construction unit constructs a training set based on the preprocessed image sequences. The action recognition unit stores the action recognition model and uses it to perform action recognition on the input video frame sequences, thereby outputting one or more corresponding action categories. The training unit trains the action recognition model using the training set.

[0044] The action recognition method includes the following steps:

[0045] Step 1: Building the dataset

[0046] This embodiment uses the Charades multi-label action recognition dataset. Charades is a large-scale indoor daily behavior video dataset, crowdsourced from various home scenarios. It has been widely used in multi-label action recognition tasks to validate the effectiveness of new methods, demonstrating strong representativeness and influence. This dataset contains 157 action categories, covering various interaction patterns of indoor daily behavior, characterized by multiple actions coexisting, large variations in action duration, and strong background interference. The training and test sets contain 7811 and 1814 video samples, respectively. The video content is in RGB format, with a frame rate of 24fps and an average video length of approximately 30 seconds.

[0047] Step 2: Data Preprocessing

[0048] Each video in the dataset is decoded into a continuous RGB image sequence, then evenly divided into T segments along the time dimension, and one frame is randomly sampled from each segment as the model input. When the number of video frames is less than T frames, it is padded using resampling or linear interpolation. In this embodiment, T is 10 frames.

[0049] Images in the training set were randomly cropped at multiple scales (1, 0.875, 0.75, and 0.66), resulting in images of size 224×224. Then, random horizontal flipping was performed, along with random color perturbations (brightness 0.4, contrast 0.4, saturation 0.2, hue 0.1) with a probability of 0.8 and random grayscale conversion with a probability of 0.2. Finally, the images were tensorized and normalized according to the mean [0.48145466, 0.4578275, 0.40821073] and standard deviation [0.26862954, 0.26130258, 0.27577711]. These data augmentation operations—multi-scale random cropping, color perturbation, and random grayscale conversion—effectively broaden the diversity of the dataset, enhance the robustness of the model, mitigate the risk of overfitting during training, and optimize the model's performance on the dataset.

[0050] The images in the test set are first scaled to 256, then cropped to a center to obtain a 224×224 input image, and the same tensor quantization and normalization operations are performed to ensure the consistency and stability of the input.

[0051] Step 3: Construct an action recognition model

[0052] like Figure 1 As shown, the action recognition model includes a video backbone encoder, a visually guided encoder (CLIP Image Encoder), a global-local adaptive guidance module (Global-Local Attention), a video guided motion aggregation module (Video Guided Motion Aggregation), a temporal position encoder, a category text embedding module (text encoder + category semantic query construction module), a Transformer decoder (Transformer Encoder), and a classification head.

[0053] In the action recognition model, video sequence markers are extracted from the video frame sequence using a video backbone encoder, and frame-level guidance features are extracted from the video frame sequence using a cascaded visual guidance encoder and a global-local adaptive guidance module. The video sequence markers and frame-level guidance features are then fused and processed by a video-guided motion aggregation module to obtain the final frame-level action representation and the final video-level semantic representation. The final frame-level action representation and the final video-level semantic representation are then processed by a temporal position encoder and a category-based text embedding module, respectively, and the processing results are input into a Transformer decoder to obtain category-related features. A classification head is then used to process the category-related features to obtain a predicted score vector, which serves as the output of the action recognition model.

[0054] The video backbone encoder uses a pre-trained TimeSformer network. The input is a video frame sequence of T frames, which is 224×224. The block size is 16×16. Therefore, each frame corresponds to 14×14=196 spatial block labels. With one classification label, the video backbone encoder outputs a video sequence label with the shape B×(1+T×196)×768. Here, B is the batch size and 768 is the feature dimension.

[0055] The visual-guided encoder uses a pre-trained CLIP visual coding network (ViT-B / 16) to encode each frame in the input video frame sequence separately and extract the global markers corresponding to each frame. (i.e., CLS markers) and 196 local block markers The global marker shape is B×T×768, and the local block marker shape is B×T×196×768. The parameters of the visual guided encoder are frozen during training and do not participate in gradient updates. When processing video frame sequences, the visual guided encoder rearranges the input video frame sequence from shape B×T×C×H×W to (B×T)×C×H×W and then extracts visual features frame by frame. Subsequently, the extracted global and local block markers are rearranged according to the batch and time dimensions to obtain a global marker of shape B×T×768 and a local block marker of shape B×T×196×768, respectively, and then input into the global-local adaptive guided module.

[0056] like Figure 2 As shown, the global-local adaptive guidance module includes a global processing branch, a local processing branch, and a fusion branch. In the global processing branch, global tags are... Linear mapping, layer normalization, and GELU nonlinear activation are applied to form query features of shape B×T×768. This is further expanded to B×T×196×768 to align with the local block label dimensions. In the local processing branch, a multi-scale local enhancement submodule is used to enhance the local block labeling. The process yields locally enhanced markers; these markers are then subjected to bottleneck mapping (768→96→768) and multiplied element-wise with the locally enhanced markers to form a bond feature of shape B×T×196×768. A linear mapping is performed on the local enhancement markers to form a value feature of shape B×T×196×768. In the fusion branch, query features will be... Key features Additive attention computation is performed, and softmax normalization is applied along the spatial block dimension to obtain attention weights; these attention weights are then applied to the value features. The frame-level context features are then weighted and summed along the spatial block dimension to obtain a frame-level context feature of shape B×T×768. After Dropout, the frame-level context features are then compared with the global label. By performing residual connections and layer normalization, the frame-level guidance feature with a shape of B×T×768 is obtained from the output of the global-local adaptive guidance module.

[0057] The global-local adaptive guidance module enables the guidance features of each frame to simultaneously perceive global semantics and local action details, making it particularly suitable for action recognition scenarios where there are significant local changes in human body parts and complex backgrounds. Unlike using only global markers as guidance, this module elevates the process from "coarse semantic cues" to "local key region representation under global semantic constraints".

[0058] like Figure 3 As shown, in the multi-scale local enhancement submodule, the local block with shape B×T×196×768 is first marked. Layer normalization is performed to stabilize subsequent convolution calculations. Then, the normalized local block labels are rearranged into a 3D spatiotemporal feature map of shape B×768×T×14×14, and channel compression is performed using 1×1×1 3D convolutions with a compression ratio r=16, resulting in a compressed feature map of shape B×48×T×14×14. This compressed feature map is then input into multiple parallel 3D depthwise convolution branches, each using a 3×3×3 convolution kernel with different temporal and spatial dilation rates. In this embodiment, the temporal dilation rate is set to 1 to 4, and the spatial dilation rate to 1 to 3, forming 12 multi-scale convolution branches. Each branch's convolution operation uses grouped convolution (the number of groups equals the number of compressed channels, 48). The output shape of each branch is B×48×T×14×14. The outputs of the 12 branches are weighted and fused using learnable fusion weights σ. The fusion weights are initialized to a uniform distribution (i.e., each weight is 1 / 12), and the importance of different scale spatiotemporal neighborhoods is adaptively adjusted through backpropagation of gradients. The fused feature map is restored to its original 768 channels using a 1×1×1 3D convolution, resulting in an enhanced feature map of shape B×768×T×14×14, which is then rearranged back to B×T×196×768. Finally, the enhanced features are injected as residuals into the local block labels using a learnable gating coefficient α (initialized to 0). The local enhancement labels output by the multi-scale local enhancement submodule are obtained. ,Right now Initializing α=0 ensures that the original features are preserved during the initial training phase.

[0059] like Figure 4As shown, the video-guided motion aggregation module includes fine-grained motion branching, coarse-grained motion branching, multi-scale fusion units, guided gating units, spatial pooling units, and temporal pooling units. Within the video-guided motion aggregation module, video sequence labels are split into classification labels. (Shape is B×1×768) and block markers (Shape B×(T×196)×768) and reshapes the block markers into two-dimensional mesh markers of shape B×T×14×14×768, rearranging the flattened one-dimensional patch token sequence back into a two-dimensional spatial mesh. The fine-grained motion branch performs local motion modeling on the mesh markers, obtaining the appearance enhancement features and explicit motion features corresponding to the fine-grained motion branch. The coarse-grained motion branch first performs average pooling downsampling on the mesh markers with a stride of 2, obtaining a low-resolution feature map of shape B×T×7×7×768, then performs local motion modeling, and upsamples the result back to the original 14×14 resolution using bilinear interpolation, obtaining the appearance enhancement features and explicit motion features corresponding to the coarse-grained motion branch. A multi-scale fusion unit is used to fuse the outputs of the fine-grained and coarse-grained motion branches, obtaining the comprehensive appearance enhancement features and comprehensive explicit motion features. A guidance gating unit is used to fuse the frame-level guidance features and the comprehensive explicit motion features, obtaining the explicit motion guidance features. Spatial pooling units are used to process explicit motion guidance features, block markers, and comprehensive appearance enhancement features to obtain the final frame-level action representation. Temporal pooling units are used to process explicit motion guidance features and frame-level guidance features to obtain the final video-level semantic representation.

[0060] like Figure 5As shown, in the local motion modeling unit, the input mesh markers (shape B×T×H×W×768) are first subjected to layer normalization. After layer normalization, the data is divided into the current frame path and the next frame path: in the current frame path, the normalized mesh markers are reshaped into current frame markers with shape (B×T)×(H×W)×768, projected onto the query space through linear mapping, and then reshaped into a multi-head format (8 attention heads, each head with a dimension of 768 / 8=96), forming the query feature Q. In the next frame path, a next frame marker is constructed (for the last frame, it is copied as the next frame). The next frame marker is then processed through two independent linear mappings to obtain key features and value features. Permute and reshape operations are performed on the key and value features respectively, transforming them into a two-dimensional image format with a shape of (B×T)×768×H×W. Then, an unfold operation is performed to extract local windows centered at each spatial location (the local window size for fine-grained motion branches is 7×7, with a window area of ​​49; the local window size for coarse-grained motion branches is 5×5, with a window area of ​​25). These are then reshaped into a multi-head format, forming local window key features K and local window value features V. The element-wise dot product between the current frame query feature Q and the next frame local window key feature K is calculated. After scaling by a scaling factor of 1 / √d_k (d_k=96), a learnable relative positional bias (shape 8×window area) is introduced to obtain the relevance score. The relevance score is fed into both the appearance and motion branches: In the appearance branch, the relevance score is normalized using Softmax, then weighted and summed with the local window value feature V of the next frame. This sum is then reshaped and linearly mapped to obtain the appearance enhancement feature. In the motion branch, the reshaped relevance score is flattened into a vector of shape (B×T)×(H×W)×(window area × 8). After layer normalization, it is mapped through two layers of MLP (the first linear mapping is followed by a GELU activation function, then a second linear mapping), and finally, a single linear mapping layer is applied to obtain the explicit motion feature. Both the appearance enhancement feature and the explicit motion feature have a shape of B×T×H×W×768.

[0061] In the multi-scale fusion unit, the appearance enhancement features and explicit motion features of the coarse-grained motion branch are fused with the corresponding outputs of the fine-grained motion branch through a learnable coarse-grained scaling factor to obtain the comprehensive appearance enhancement features and comprehensive explicit motion features.

[0062] In the guidance gating unit, the frame-level guidance features are sequentially processed through layer normalization, linear mapping (768→768), and the Sigmoid activation function to generate channel gating weights (with a shape of B×T×1×1×768). The channel gating weights are then multiplied element-wise with the comprehensive explicit motion features to obtain the explicit motion guidance features.

[0063] In the space pooling unit, the block is marked The integrated appearance enhancement features and explicit motion guidance features are fused using learnable scaling factors (both appearance scaling factor and motion scaling factor are initialized to 0) to obtain a fused block label, i.e., fused block label = block label + appearance scaling factor × integrated appearance enhancement features + motion scaling factor × explicit motion guidance features. Initializing the scaling factor to 0 ensures that the model behavior is consistent with that without modules in the early stages of training, and the optimal fusion ratio is gradually learned through gradient optimization. The fused block label and explicit motion guidance features are concatenated along the channel dimension (the dimension becomes 768 × 2 = 1536), and after layer normalization and two layers of linear mapping (1536 → 768 → 1), the weight score corresponding to each spatial block is output. Softmax normalization is performed on the weight scores of all 196 spatial blocks in each frame to obtain spatial weights. Then, the fused block label is weighted and summed using the spatial weights to obtain a frame-level action representation of shape B × T × 768. The classification label is injected into the frame-level action representation according to the learnable scaling factor (initialized to 0), and the final frame-level action representation is obtained through layer normalization.

[0064] In the temporal pooling unit, the frame-level guidance features and the frame-level summary of the explicit motion guidance features (the mean of the explicit motion guidance features in the spatial dimension is taken, with a shape of B×T×768) are concatenated in the channel dimension (the dimension becomes 768+768=1536). After layer normalization and two layers of linear mapping (1536→768→1), the temporal weight score corresponding to each frame is output. Softmax normalization is performed on the temporal weight scores of all T frames to obtain the temporal weights. Then, the frame-level guidance features are weighted and summed using the temporal weights to obtain a video-level semantic representation with a shape of B×1×768. After layer normalization, the final video-level semantic representation is obtained.

[0065] The temporal position encoder uses sine-cosine position coding to inject temporal position information into the frame-level motion representation. The frame-level motion representation is linearly mapped (768→512) and then injected with position coding, which serves as the source sequence (B×T×512) for the Transformer decoder.

[0066] The text encoder uses a pre-trained CLIP text encoding network to construct a text sequence based on 157 predefined action category names, and obtains a category text embedding with a shape of 157×512 through the text encoding network.

[0067] The category semantic query construction module receives the category text embedding and the final video-level semantic representation. It expands the category text embedding to a shape of B×157×512 in the batch dimension and copies the final video-level semantic representation to a shape of B×157×768 in the category dimension. Then, it concatenates the category text embedding and the final video-level semantic representation in the channel dimension to obtain a fused semantic representation with a shape of B×157×1280. It then projects this fused semantic representation onto a 512-dimensional feature space through a linear mapping to obtain a category query vector with a shape of B×157×512.

[0068] The Transformer decoder in this invention adopts the standard Transformer structure, uses frame-level action representation as the source sequence and class query vector as the target sequence, and enables interactive decoding of the class query vector with key action frames in the video to output class-related features with shape B×157×512.

[0069] The classification head used in this invention adopts a GroupWiseLinear structure with independent parameter modeling by category. That is, each action category is assigned an independent weight vector (1×157×512) and bias (1×157). The action scores of each category are output by weighted summation of elements, and the final output is a prediction score vector of shape B×157.

[0070] Step 4: Train the action recognition model using the dataset, and then use the trained model to identify actions in the test video, outputting predicted scores for 157 action categories. After thresholding, the final multi-label action recognition result is obtained. During training, the binary cross-entropy loss function (BCEWithLogitsLoss) is used, which is suitable for multi-label action recognition tasks. Simultaneously, the Adam optimizer is used for parameter updates, with an initial learning rate of 1×10⁻⁻⁶. 5 The CosineAnnealingWarmRestarts learning rate scheduling strategy (T0=10) was adopted. Furthermore, the gradient norm was pruned during training with a pruning threshold of 1.0. The CLIP image encoding network parameters in the visual guided encoder were kept frozen, and the video backbone encoder, global-local adaptive guided module, video guided motion aggregation module, Transformer decoder, and classification head were jointly optimized.

[0071] Step 5: Method Evaluation

[0072] The action recognition model was tested and evaluation metrics were output. The main evaluation metric used was micro mAP (micro-mean accuracy). The performance of the action recognition model in this invention on the Charades dataset in ablation experiments is shown in Table 1.

[0073] Table 1 shows the ablation experiments of this invention on the Charades dataset.

[0074]

[0075] As shown in Table 1, using only the global-local adaptive guidance module improves accuracy, increasing the micro mAP on the test set from 47.57% to 47.95%. Further adding the video-guided motion aggregation module further improves the micro mAP to 49.81%, while using only the video-guided motion aggregation module also yields a greater accuracy improvement, increasing the micro mAP on the test set from 47.57% to 49.43%. This indicates that the two modules proposed in this invention are highly complementary. The global-local adaptive guidance module enhances the representation ability of key regions, while the video-guided motion aggregation module further enhances the ability to model cross-frame temporal relationships. Their collaborative work significantly improves action recognition performance.

[0076] The performance of the action recognition model in this invention is compared with that of MsqNet on the Charades dataset, and the results are shown in Table 2.

[0077] Table 2 shows the performance comparison of the present invention with other methods on the Charades dataset.

[0078]

[0079] Example 2

[0080] This invention proposes a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of an action recognition method based on global-local adaptive guidance and video-guided motion aggregation.

[0081] Example 3

[0082] This invention proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of an action recognition method based on global-local adaptive guidance and video-guided motion aggregation.

[0083] Example 4

[0084] This invention proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of an action recognition method based on global-local adaptive guidance and video-guided motion aggregation.

[0085] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. An action recognition method based on global-local adaptive guidance and video-guided motion aggregation, characterized in that: The method includes: Acquire the video to be tested; preprocess the video to obtain the video frame sequence; The video frame sequence is input into the constructed action recognition model for action recognition; the action recognition model includes a video backbone encoder, a visual guidance encoder, a global-local adaptive guidance module, a video guidance motion aggregation module, a temporal position encoder, a category text embedding module, a Transformer decoder, and a classification head; The video backbone encoder is used to extract video sequence markers from the video frame sequence; The visual guidance encoder is used to extract global and local block markers from the video frame sequence; the global-local adaptive guidance module processes the global and local block markers through global and local processing branches respectively, and introduces an attention mechanism in the fusion branch to aggregate the processing results of the global and local processing branches to obtain frame-level guidance features. The video-guided motion aggregation module is used to fuse video sequence markers and frame-level guided features to obtain the final frame-level action representation and the final video-level semantic representation. The temporal position encoder is used to perform temporal position encoding on the final frame-level motion representation; the category semantic query construction module is used to embed the motion category into the final video-level semantic representation to obtain the category query vector. The Transformer decoding module uses the final frame-level action representation after temporal position encoding as the source sequence, and the category query vector as the target sequence for interactive decoding. The decoding result is input to the classification head, and the predicted score corresponding to different action categories is output. After threshold determination, the final multi-label action recognition result is obtained, realizing action recognition of the tested video. 2.The motion recognition method based on global-local adaptive guidance and video guided motion aggregation according to claim 1, characterized in that: The video-guided motion aggregation module includes fine-grained motion branches, coarse-grained motion branches, a multi-scale fusion unit, a guidance gating unit, a spatial pooling unit, and a temporal pooling unit. In this module, video sequence markers are split into classification markers and block markers, and the block markers are converted into grid markers. The grid markers are processed through the fine-grained and coarse-grained motion branches respectively to obtain the appearance enhancement features and explicit motion features corresponding to the fine-grained and coarse-grained motion branches. The multi-scale fusion unit fuses the outputs of the fine-grained and coarse-grained motion branches to obtain the comprehensive appearance enhancement features and comprehensive explicit motion features. The explicit motion guidance feature is obtained by fusing frame-level guidance features and comprehensive explicit motion features using a guidance gating unit; the explicit motion guidance feature, block markers and comprehensive appearance enhancement features are further fused using a spatial pooling unit and then processed to obtain the final frame-level action representation; the final video-level semantic representation is obtained by fusing explicit motion guidance features and frame-level guidance features using a temporal pooling unit and then processed to obtain the final video-level semantic representation. 3.The method of claim 2, wherein: Both the fine-grained motion branch and the coarse-grained motion branch introduce local motion modeling units to process the mesh markers. In the local motion modeling unit, after layer normalization of the mesh markers, the mesh markers of the current frame are obtained through linear mapping to obtain query features, and the mesh markers of the next frame are obtained through linear mapping to obtain key features and value features respectively. Dimensional displacement and reshaping operations are performed on the key features and value features respectively, and after transforming them into a two-dimensional image format, local windows are extracted with each spatial location as the center through an expansion operation, and then reshaped into a multi-head format to form local window key features and local window value features. The relevance score is calculated based on query features and local window key features. On the one hand, the relevance score is normalized and then weighted and summed with the local window value features. After reshaping and linear mapping, the appearance enhancement features are obtained. On the other hand, after flattening the correlation scores, explicit motion features are obtained through layer normalization and multi-layer mapping. 4.The method of claim 2, wherein: In the spatial pooling unit, block labels, comprehensive appearance enhancement features, and explicit motion guidance features are fused to obtain fused block labels; after concatenating the fused block labels with the explicit motion guidance features, layer normalization and multi-layer linear mapping are performed to obtain weight scores; Normalize the weighted scores to obtain spatial weights; The frame-level action representation is obtained by weighted summation of the fused block markers using spatial weights. After injecting the classification tags into the frame-level action representation, the final frame-level action representation is obtained through layer normalization.

5. The method of claim 2, wherein the method is based on global-local adaptive guidance and video guided motion aggregation. In the temporal pooling unit, the explicit motion guidance features are averaged in the spatial dimension, and the result is concatenated with the frame-level guidance features. After layer normalization and linear mapping, a temporal weight score is obtained. Softmax normalization is performed on the temporal weight score to obtain the temporal weight. The frame-level guidance features are weighted and summed using the temporal weight to obtain the video-level semantic representation. The video-level semantic representation is then processed by layer normalization to obtain the final video-level semantic representation.

6. The method of claim 2, wherein the method is based on global-local adaptive guidance and video guided motion aggregation. In the guidance gating unit, the frame-level guidance features are processed sequentially through layer normalization, linear mapping, and activation functions to generate channel gating weights; the channel gating weights are then multiplied element-wise with the comprehensive explicit motion features to obtain explicit motion guidance features.

7. The action recognition method based on global-local adaptive guidance and video-guided motion aggregation according to claim 1, characterized in that: The global processing branch sequentially performs linear mapping, layer normalization, nonlinear activation, and reshaping operations on the global labels to obtain query features. The local processing branch processes local block labels through a multi-scale local enhancement submodule to obtain local enhanced labels. On one hand, the local enhanced labels are bottleneck mapped and multiplied element-wise with the local enhanced labels to obtain key features. On the other hand, the local enhanced labels are linearly mapped to obtain value features. The fusion branch performs additive attention calculation on the query features and key features and performs normalization along the spatial block dimension to obtain attention weights. Attention weights are applied to the value features and weighted summation is performed along the spatial block dimension to obtain frame-level context features. The frame-level context features are then residually concatenated with the global labels, and layer normalization is performed to obtain frame-level guiding features.

8. The method of claim 7, wherein the method is based on global-local adaptive guidance and video guided motion aggregation. In the multi-scale local enhancement submodule, layer normalization, reshaping, and convolution operations are performed sequentially on the local block labels to obtain compressed feature maps; multiple parallel depthwise convolutions are used to process the compressed feature maps, and the processing results are fused and then convolution and reshaping are performed sequentially to obtain enhanced features; The enhanced features are injected into the local block labels in the form of residuals to obtain local enhanced labels.

9. The method of claim 1, wherein: The video backbone encoder uses a pre-trained TimeSformer network; the visual guidance encoder uses a pre-trained CLIP visual coding network; the temporal position encoder uses sine-cosine position coding to inject temporal position information into frame-level action representation; the category text embedding module includes a cascaded text encoder and a category semantic query construction module; the text encoder uses a pre-trained CLIP text coding network.

10. An action recognition system based on global-local adaptive guidance and video guided motion aggregation, characterized in that: This system is used to perform an action recognition method based on global-local adaptive guidance and video-guided motion aggregation as described in claim 1. The action recognition system includes a data acquisition unit, an image preprocessing unit, and an action recognition unit. The data acquisition unit is used to acquire the video under test. The image preprocessing unit is used to preprocess the video under test. The action recognition unit is used to store the action recognition model and use the action recognition model to perform action recognition on the input video frame sequence.

Images