Method for constructing video behavior recognition model, video behavior recognition method, model, system and storage medium

Abstract

The application relates to the technical field of deep learning, in particular to a video behavior recognition model construction method, a video behavior recognition method, a model, a system and a storage medium. On the one hand, the feature complementary mapping module is used to enhance the two-dimensional space features in the video data set to avoid the loss of details of the space features; on the other hand, the adaptive temperature regulation dynamic attention feature fusion module TDAF is used to deeply fuse the enhanced two-dimensional space features and three-dimensional time sequence features, a variable temperature parameter is introduced as a non-fixed learning strategy, the action amplitude in the video is scaled, and then corresponding adaptive weights are generated to adjust the recognition power consumption of the model, so that accurate behavior recognition in a complex dynamic scene is realized. The application aims to solve the problem of how to realize accurate behavior recognition in a complex dynamic scene through a non-fixed learning strategy mode.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a method for constructing a video behavior recognition model, a video behavior recognition method, a model, a system, and a storage medium. Background Technology

[0002] Video behavior recognition, a popular research area in computer vision, is widely used in scenarios such as intelligent surveillance and human-computer interaction. Currently, some research has applied deep learning technology to video behavior recognition, leveraging the powerful learning capabilities of deep learning models for rapid behavior identification.

[0003] However, traditional deep learning models are still insufficient in recognizing behavior in complex dynamic scenes: In complex dynamic scenes, the distribution of targets in video images is relatively dense, and the behavior patterns of targets vary greatly. The fixed learning strategy adopted by traditional deep learning models is difficult to simultaneously recognize large-scale foreground and small-scale background targets, as well as the increased computational overhead caused by different behavior patterns.

[0004] In view of this, this application proposes a method for constructing a video behavior recognition model, which aims to achieve accurate behavior recognition in complex dynamic scenes through a non-fixed learning strategy. Summary of the Invention

[0005] The main purpose of this application is to provide a method for constructing a video behavior recognition model, which aims to solve the problem of how to achieve accurate behavior recognition in complex dynamic scenes through a non-fixed learning strategy.

[0006] To achieve the above objectives, this application provides a method for constructing a video behavior recognition model, the method comprising:

[0007] (1) Construct a feature complementary mapping module, so that when the feature complementary mapping module performs two-dimensional spatial feature semantic feature enhancement, it includes the following steps:

[0008] The two-dimensional spatial features in the video dataset are divided into shallow features and deep features, and the spatial resolution of the shallow features and the deep features are aligned.

[0009] The aligned shallow features are concatenated with the deep features, and the resulting concatenated features are introduced into a spatial channel attention mechanism to enhance key channels, thus obtaining enhanced features.

[0010] The deep features are added to the enhanced features by residual connection to obtain enhanced two-dimensional spatial features;

[0011] (2) Construct an adaptive temperature-controlled dynamic attention feature fusion module, so that when performing feature fusion enhancement, the adaptive temperature-controlled dynamic attention feature fusion module includes the following steps:

[0012] The enhanced two-dimensional spatial features and three-dimensional temporal features in the video dataset are obtained, and the enhanced two-dimensional spatial features and three-dimensional temporal features of the same scale are concatenated in the channel dimension to obtain the concatenated fused feature map.

[0013] Extract the fusion semantic vector from the spliced ​​fusion feature map, generate fusion weights through a weight learning network, and introduce a variable temperature parameter for scaling to obtain adaptive weights. The magnitude of the variable temperature parameter is positively correlated with the behavioral amplitude in the video dataset.

[0014] The adaptive weights, along with the channel attention weights and spatial attention weights obtained after cross-attention processing of the video dataset, are fused and weighted to obtain fused features.

[0015] Optionally, the steps of extracting the fused semantic vector from the concatenated fused feature map, generating fusion weights for the two-dimensional and three-dimensional features through a weight learning network, and introducing a variable temperature parameter for softmax temperature scaling to obtain adaptive weights include:

[0016] Two-dimensional spatial features With three-dimensional temporal features Global average pooling is performed after concatenation:

[0017]

[0018] In the formula, Indicates global average pooling;

[0019] Weight vectors are generated using a fully connected MLP layer, and a temperature parameter is introduced. Softmax scaling is applied to obtain adaptive weights;

[0020] .

[0021] Optionally, the step of fusing and weighting the input features with the adaptive weights, the channel attention weights obtained after channel attention processing of the input features, and the spatial attention weights to obtain the fused features includes:

[0022] Obtain Channel Attention Weights :

[0023]

[0024] In the formula, For global average pooling, For global max pooling, Here, F is the Sigmoid activation function, and F is the input feature.

[0025] And obtain spatial attention weights :

[0026]

[0027] In the formula, Conv7*7 represents a convolution of size 7*7, AvgPool represents average pooling, and MaxPool represents max pooling.

[0028] The input features are fused and weighted based on adaptive weights, channel attention weights, and spatial attention weights:

[0029]

[0030] In the formula, F TDAM As a feature of fusion, Indicates original features; Indicates enhanced features, This indicates element-wise multiplication;

[0031] Optionally, the expression for the enhanced two-dimensional spatial features is:

[0032]

[0033] in:

[0034]

[0035]

[0036]

[0037]

[0038]

[0039] In the formula, To enhance the features of two-dimensional space; This represents the deep features after applying spatial attention weights. This represents the shallow features after applying channel attention weights; Spatial attention weights, Channel attention weights; This represents shallow features that have been upsampled to the same resolution as the deep features. This represents the deep features after performing a 1x1 convolution operation; Indicates deep features; Indicates shallow features; For global average pooling; This is the Sigmoid activation function.

[0040] Furthermore, to achieve the above objectives, this application also provides a video behavior recognition method, which uses the video behavior recognition model construction method described in any of the preceding claims to obtain a video behavior recognition model, wherein the video behavior recognition model includes a feature complementary mapping module and an adaptive temperature-controlled dynamic attention feature fusion module, and the video behavior recognition method includes the following steps:

[0041] The input video sequence is subjected to frame-by-frame feature extraction to obtain input features, which include two-dimensional spatial features and three-dimensional temporal features.

[0042] The feature semantics of the two-dimensional spatial features are enhanced by the feature complement mapping module to obtain enhanced two-dimensional spatial features;

[0043] The enhanced two-dimensional spatial features and the three-dimensional temporal features are deeply fused by the adaptive temperature-controlled dynamic attention feature fusion module to obtain fused features.

[0044] The fused features are input into the classification head, and the behavior category prediction results of the video sequence are output.

[0045] Optionally, the behavior category prediction result includes regression prediction result and classification prediction result. The step of inputting the fused features into the classification head and outputting the behavior category prediction result of the video sequence includes:

[0046] In decoupled mode, independent fusion modules and dynamic weight learning networks are built for the bounding box regression branch and classification branch respectively. Adaptive weights are used to output regression prediction results and classification prediction results respectively, so as to reduce feature conflicts between multiple tasks:

[0047]

[0048] In the formula, These are all adaptive weights for the bounding box regression branch. All are adaptive weights for the classification branches; This represents the upsampled three-dimensional temporal features. It represents the characteristics of two-dimensional space.

[0049] Furthermore, to achieve the above objectives, this application also provides a video behavior recognition model obtained using the video behavior recognition model construction method described above, wherein the video behavior recognition model includes:

[0050] The input module is used to receive and preprocess video frame sequences;

[0051] A spatial backbone module is constructed by building a feature complementary mapping module at layer P5 of the feature pyramid of the spatial backbone module to obtain enhanced two-dimensional spatial features;

[0052] The temporal backbone module is used to extract three-dimensional temporal features;

[0053] An adaptive temperature-controlled dynamic attention feature fusion module is used to deeply fuse enhanced two-dimensional spatial features and three-dimensional temporal features;

[0054] The output module is used to generate and output video behavior recognition results.

[0055] In addition, to achieve the above objectives, this application also provides a computer system, the computer system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the method for constructing a video behavior recognition model as described in any of the preceding claims or the steps of the video behavior recognition method as described in any of the preceding claims.

[0056] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for constructing a video behavior recognition model as described in any of the preceding claims or the steps of the video behavior recognition method as described in any of the preceding claims.

[0057] This application has at least the following beneficial effects:

[0058] 1. By leveraging the synergistic effect of the FCM feature complementarity mapping module and the TDAF fusion module, the problem of insufficient feature extraction in traditional methods is effectively solved;

[0059] 2. The TDAF module introduces an adaptive temperature control strategy and dynamic weight learning to achieve intelligent deep integration of 2D spatial features and 3D temporal features.

[0060] 3. In response to the characteristic of drastic changes in target scale in videos, the multi-scale attention mechanism can dynamically balance feature representation at different resolutions.

[0061] 4. An improved Task Alignment Learning (TAL) loss function was set up, which effectively alleviates the class imbalance problem through triple constraints of classification, regression and distribution.

[0062] 5. By combining spatial-temporal dual data augmentation strategies and temporal consistency constraints, the model exhibits good generalization performance under different lighting conditions, shooting angles, and background environments, making it suitable for various practical video analysis scenarios. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating the method for constructing a video behavior recognition model according to an embodiment of this application;

[0064] Figure 2 This is a schematic diagram of the architecture of the feature complementary mapping module involved in an embodiment of this application;

[0065] Figure 3 This is an architecture diagram of the adaptive temperature-controlled dynamic attention fusion module involved in an embodiment of this application;

[0066] Figure 4 This is a schematic diagram of the structure of the multi-scale attention mechanism involved in the embodiments of this application;

[0067] Figure 5 This is a schematic diagram illustrating the training process and loss function of the video behavior recognition model involved in the embodiments of this application;

[0068] Figure 6 Point activation map of the conventional baseline model involved in the embodiments of this application.

[0069] Figure 7 This is a region activation map of the method proposed in this application, which is involved in the embodiments of this application;

[0070] Figure 8 This is a graph showing the changes in fusion weights in a continuous motion scenario as described in the embodiments of this application.

[0071] Figure 9 This is a graph showing the changes in fusion weights in a burst motion scenario as described in the embodiments of this application.

[0072] Figure 10 This is the video behavior recognition model involved in the embodiments of this application;

[0073] Figure 11 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0074] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0075] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0076] First Embodiment

[0077] Reference Figure 1 This embodiment provides a method for constructing a video behavior recognition model. Compared with traditional video behavior recognition models, the model proposed in this embodiment creatively proposes two modules: a feature complementary mapping module FCM and an adaptive temperature-controlled dynamic attention feature fusion module TDAF.

[0078] The complementary feature mapping module is used to solve the problem of detail loss in two-dimensional spatial features in deep networks. In this embodiment, a complementary feature mapping module is constructed so that when performing two-dimensional spatial feature semantic feature enhancement, the complementary feature mapping module includes the following steps:

[0079] S10, divide the two-dimensional spatial features in the video dataset into shallow features and deep features, and align the spatial resolution of the shallow features and the deep features;

[0080] In some optional implementations, the video dataset is preprocessed: Spatially, operations such as random cropping, horizontal flipping, ±15-degree random rotation, and color jitter of intensity 0.2 are performed to enhance the model's robustness to target position shifts, viewpoint changes, and lighting differences, while also expanding the diversity of training samples. Temporally, a random temporal cropping strategy is used to dynamically extract fixed-length segments from the original video as input, combined with a variable time step sampling mechanism to enhance the model's ability to model different action rhythms; a 10% probability is used to randomly discard some video frames to simulate frame loss or occlusion that may occur in real-world scenes, further improving the model's robustness to temporal incompleteness.

[0081] In some optional implementations, the channel dimension is divided at a preset ratio of 3:1 to obtain branches for shallow features and branches for deep features; bilinear interpolation upsampling is performed on the shallow feature map to adjust its spatial resolution to align with that of the deep features. Figure 1 To, received:

[0082]

[0083] S20, the aligned shallow features are spliced ​​with the deep features, and the resulting spliced ​​features are introduced into the spatial channel attention mechanism to enhance key channels, thereby obtaining enhanced features;

[0084] In some alternative implementations, the stitching method can be cascaded convolutions to extract fine-grained spatial features:

[0085]

[0086] During the concatenation process, a 1×1 convolution operation is performed on the deep features to adjust their channel dimensions while preserving semantic information.

[0087]

[0088] In some alternative implementations, a Squeeze-and-Excitation (SE) attention mechanism is introduced to enhance feature representation. This SE attention mechanism includes parallel spatial attention computation and channel attention computation.

[0089] Spatial attention weights are extracted using convolutional layers;

[0090]

[0091] Channel attention weights are extracted using global average pooling and fully connected layers;

[0092]

[0093] S30, the deep features and the enhanced features are added together through residual connection to obtain enhanced two-dimensional spatial features;

[0094] In some alternative implementations, a cross-attention mechanism is used to achieve complementary fusion of shallow and deep features. Spatial attention weights are applied to deep features, and channel attention weights are applied to shallow features. Feature recombination is then performed to obtain the final output enhanced two-dimensional spatial features. :

[0095]

[0096] in,

[0097]

[0098] In the above formula, To enhance the features of two-dimensional space; This represents the deep features after applying spatial attention weights. This represents the shallow features after applying channel attention weights; Spatial attention weights, Channel attention weights; This represents shallow features that have been upsampled to the same resolution as the deep features. This represents the deep features after performing a 1x1 convolution operation; Indicates deep features; Indicates shallow features; For global average pooling; This is the Sigmoid activation function.

[0099] Furthermore, an adaptive temperature-controlled dynamic attention feature fusion module (TDAF) is constructed. This module achieves deep fusion of 2D spatial features and 3D temporal features through multiple attention mechanisms and temperature control strategies. Therefore, when performing feature fusion enhancement, the adaptive temperature-controlled dynamic attention feature fusion module includes the following steps:

[0100] S40, Obtain the enhanced two-dimensional spatial features and three-dimensional temporal features from the video dataset, and concatenate the enhanced two-dimensional spatial features and three-dimensional temporal features of the same scale in the channel dimension to obtain the concatenated fused feature map;

[0101] In some alternative implementations, the enhanced 2D spatial features are input in list form by scale, while the 3D features are input as single tensors (single frames); and the number of channels and spatial resolution of each feature are obtained for subsequent alignment and fusion.

[0102]

[0103]

[0104] S50, extract the fusion semantic vector from the spliced ​​fusion feature map, generate fusion weights through a weight learning network, introduce a variable temperature parameter for scaling, and obtain adaptive weights, wherein the magnitude of the variable temperature parameter is positively correlated with the behavior amplitude in the video dataset;

[0105] In some alternative implementations, a weight vector is generated using a fully connected layer, and a temperature parameter is introduced. Perform Softmax scaling;

[0106]

[0107] in, This is a temperature parameter used to control the sharpness of the weight distribution;

[0108] It should be noted that in video action recognition tasks, when the input video contains subtle movements such as "raising a hand" or "nodding," the model will automatically learn smaller actions. The value makes the weight distribution steeper, thus forcing the model to focus on the most discriminative spatiotemporal feature channels; while when the input is a large-amplitude action such as "walking" or "jumping", the model will learn a larger value. The value is adjusted to smooth the weight distribution, thereby utilizing more global contextual information. This mechanism endows the TDAF module with adaptive perception capabilities for different behavioral patterns, which is a key feature that distinguishes it from traditional static attention mechanisms.

[0109] S60 combines the adaptive weights with the channel attention weights and spatial attention weights obtained after cross-attention processing of the video dataset to fuse and weight the input features, thereby obtaining fused features.

[0110] In some alternative implementations, global average pooling and global max pooling are performed in parallel to extract average features and maximum features, respectively. These features are then processed by a shared multilayer perceptron (MLP), summed, and activated by a sigmoid function to generate channel attention weights.

[0111]

[0112] In the formula, For global average pooling, For global max pooling, Here, F is the Sigmoid activation function, and F is the input feature.

[0113] Calculate the importance weights of spatial locations to generate spatial attention weights:

[0114]

[0115] In the formula, Conv7*7 represents a convolution of size 7*7, AvgPool represents average pooling, and MaxPool represents max pooling.

[0116] The input features are fused and weighted based on adaptive weights, channel attention weights, and spatial attention weights:

[0117]

[0118] In the formula, F TDAM As a feature of fusion, Indicates original features; Indicates enhanced features, This indicates element-wise multiplication.

[0119] In this embodiment, after the completion of the Complementary Feature Mapping (TCM) module and the Adaptive Temperature-Controlled Dynamic Attention Feature Fusion (TDAF) module, the remaining necessary architecture of the video behavior recognition model is constructed in accordance with the traditional model building method, and will not be described in detail in this embodiment.

[0120] In the technical solution provided in this embodiment, on the one hand, the feature complementary mapping module enhances the two-dimensional spatial features in the video dataset to avoid the loss of spatial feature details; on the other hand, the adaptive temperature-controlled dynamic attention feature fusion module TDAF deeply fuses the enhanced two-dimensional spatial features and three-dimensional temporal features, and introduces a variable temperature parameter as a non-fixed learning strategy to scale the magnitude of the action in the video, thereby generating corresponding adaptive weights to adjust the recognition computing power overhead of the model, thus achieving accurate behavior recognition in complex dynamic scenes.

[0121] Second Embodiment

[0122] Based on the first embodiment, this embodiment provides a method for improving a video behavior recognition model or constructing its infrastructure:

[0123] (1) Construction of dual-stream network architecture

[0124] A dual-stream network framework is constructed, comprising a 2D spatial feature extraction branch and a 3D spatiotemporal feature modeling branch. The 2D spatial feature extraction branch is used to extract spatial details and semantic features of video frames, while the 3D spatiotemporal feature modeling branch is used to capture long-term temporal action logic of video sequences. A Feature Complementary Mapping (FCM) module is integrated into the dual-stream network architecture to enhance the feature representation capability of the 2D backbone network; an Adaptive Temperature-Regulated Dynamic Attention Fusion (TDAF) module is integrated to achieve deep cross-fusion of 2D spatial features and 3D temporal features. The detection head module adopts a decoupled design, mapping the fused features to two independent output spaces, outputting the target's bounding box regression result and action category prediction result, respectively.

[0125] (2) Construction of 2D backbone network

[0126] In this embodiment, the 2D feature extraction backbone is initialized based on the YOLOv8n network structure, and a feature extraction network with CSP-Darknet53 as the core structure is constructed to balance the lightweight nature of the model with the feature representation capability.

[0127] The basic convolutional module, as the fundamental building block of the entire network, integrates convolutional layers (Conv), batch normalization, and the SiLU activation function in sequence.

[0128] A C2f residual connection module is constructed. This module uses 1x1 convolution for dimensionality reduction, 3x3 convolution for feature extraction, and residual connection to achieve bottleneck design, which increases feature gradient capability while reducing computational complexity.

[0129] In this embodiment, a Feature Complementary Mapping (FCM) module is integrated at the key nodes of the feature pyramid, and the asymmetric channel segmentation characteristics of the FCM module are used to process multi-scale feature information.

[0130] (3) Constructing a multi-scale feature pyramid

[0131] Leveraging the inherent multi-level structure of YOLOv8, multi-scale spatial features are extracted simultaneously, constructing a feature pyramid comprising P3 (28×28, 64 channels), P4 (14×14, 128 channels), and P5 (7×7, 256 channels). The specific implementation steps are as follows:

[0132] S100, based on the CSPDarknet53 architecture, extracts features at different depth levels.

[0133] S200 constructs a Feature Pyramid Network (FPN) to establish a top-down feature fusion path, enabling the transfer and fusion of deep semantic features to shallow layers.

[0134] S300 normalizes feature maps at different scales to ensure the stability of numerical distribution.

[0135] S400 deploys the FCM module simultaneously at the P3, P4, and P5 levels of the feature pyramid to achieve cross-scale feature enhancement.

[0136] S500 formats the processed multi-scale features into a standard feature pyramid output.

[0137] S600, constructing the DarkNet_FCM backbone network:

[0138] Layer P1 performs initial feature extraction, using a large 6x6 convolutional kernel to process the 640x640 input image, transforming the 3-channel RGB image into a 1D 320x320 feature map. Layer P2 processes shallow features, performing a 2x downsampling through 3x3 convolutions to transform the 1D 320x320 feature map into a 2D 160x160 feature map. Layers P3 and P4 expand the feature map channel width dimensions to 4D and 8D, respectively. Layer P5 handles low-resolution features with high semantic information, achieving a 16D channel width dimension. Applying FCM's asymmetric channel segmentation and cross-attention mechanism in layer P5 maximizes feature representation while avoiding unnecessary computational overhead on shallow features.

[0139] S700, construct the DarkFPN_FCM feature pyramid network:

[0140] The feature fusion path achieves information transfer between feature maps of different resolutions through three key convolutional layers. Layer H6 handles the feature transfer from P5 to P4, using a 1×1 convolution to map the deepest semantic features from a 16-dimensional channel width to an 8-dimensional channel width, providing rich semantic guidance for mid-layer features. Layer H4 maps the fused features from layer P4 from an 8-dimensional channel width to a 4-dimensional channel width, achieving feature transfer to layer P3. Layer H2, as the final output layer, performs final processing and optimization on the features from layer P3, maintaining an output dimension of 2-dimensional channel width. An FCM module is inserted into all layers to achieve complementary multi-scale feature mappings, enhancing feature representation capabilities.

[0141] (4) Constructing a 3D backbone network

[0142] During the 3D backbone network construction phase, the system uses shufflnetv2 2.0x as the temporal modeling backbone and loads RGB weight files pre-trained based on ImageNet to initialize the network, thereby accelerating convergence and improving generalization.

[0143] A complete temporal feature extraction pathway is constructed by stacking 3D convolutional layers. In input processing, the model receives 16 consecutive video frames, performs end-to-end 3D convolution operations via the ShufflNetv2 network, and finally outputs multi-level spatiotemporal feature maps, uniformly represented in T×H×W×C format (time×height×width×channel), providing a high-dimensional and structured temporal representation foundation for subsequent cross-modal fusion with 2D spatial features.

[0144] (5) Construct a multi-scale attention feature fusion module

[0145] The scale factor is configured as [1, 2, 4], corresponding to the original resolution, 2x downsampling, and 4x downsampling feature maps, respectively. Self-attention maps are computed independently for each scale, generating attention outputs at scale 1 (fine local details), scale 2 (mid-range actions), and scale 4 (long-range overall pose) to capture behavioral semantics at different spatial scales.

[0146] Third Embodiment

[0147] As an alternative embodiment, based on any embodiment, refer to Figure 2The schematic diagram of the Feature Complementary Mapping (FCM) module is shown. In this embodiment, the module is deployed at the P5 level of the feature pyramid. The input is a feature map of size C×H×W, and the output is a calibration feature with consistent dimensions. The FCM module consists of a channel attention branch, a spatial attention branch, a feature enhancement branch, and a feature fusion unit. The input feature X is fed into each branch in parallel: the channel attention branch performs global average pooling on X, generates channel weights through two fully connected layers and a sigmoid function; the spatial attention branch performs GAP and GMP on X and concatenates them, generates spatial weights through a 7×7 convolution and a sigmoid function; the feature enhancement branch uses 3×3 convolution, BatchNorm, ReLU, and 1×1 convolution to obtain enhanced features. The feature fusion unit calculates the final output of the module in an element-wise manner. This module is used at the H2 / H4 / H6 scale to improve the classification, localization, and temporal alignment features of downstream detection heads.

[0148] Fourth embodiment

[0149] As an alternative embodiment, based on any embodiment, refer to Figure 3 The diagram shows the architecture of the adaptive temperature-controlled dynamic attention fusion module. The module inputs two-dimensional multi-scale features F_2D (B×C×H×W) and three-dimensional temporal features F_3D (B×C×D×H×W), which are upsampled and aligned to obtain an aligned temporal feature tensor. These two features are fed in parallel into a dynamic weight learning submodule (global pooling, MLP, temperature controller) to generate normalized fusion weights. The fusion strategy unit selects and executes attention, addition, multiplication, or concatenation based on the fusion weights to obtain the fused feature input. The attention module consists of a main branch and an enhanced channel attention branch (CAM). The main branch sequentially performs 1×1 convolution, 3×3 convolution, batch normalization (BN), and SiLU activation on the input fusion features for feature reshaping and noise suppression. The CAM branch simultaneously models channel and spatial attention on the same input, generating weights that interact with the features. The outputs of the two branches are summed through residuals to form the fusion result, establishing a stable gradient propagation path and improving the effectiveness of the attention representation. The outputs of the two branches are added together to form a fused feature Y (B×C×H×W), which is used by the detection head for classification, localization and temporal alignment; the module supports coupled mode (i.e. shared fusion) and decoupled mode (i.e. classification / regression are fused separately).

[0150] Fifth embodiment

[0151] As an optional embodiment, based on the first embodiment, referring to Figure 4The diagram illustrates the structure of the multi-scale attention mechanism. The system starts with an input feature tensor (size H×W×C) and sets up an original resolution self-attention branch, a two-resolution branch, a four-resolution branch, and a global attention branch. The original branch performs a linear mapping on the input features to obtain "q, k, v" and calculates the attention response; the two-resolution and four-resolution branches perform 2x and 4x average pooling respectively, generating attention and then aligning it to a uniform spatial size via bilinear upsampling; the global branch performs global average pooling on the input features, obtains channel weights through a fully connected layer, and broadcasts them to the spatial dimension. The multi-scale weight learning unit concatenates the attention from each branch with the channel weights, inputs them into a multilayer perceptron, and obtains the fusion weights (weights sum to 1) through a normalization function. The weighted fusion unit applies each attention and input feature element-wise according to the fusion weights, outputting a fusion feature tensor with the size maintained at H×W×C. Through joint modeling dependent on "local-cross-scale-global" and sample adaptive weight selection, it achieves collaborative calibration of details and semantics, integrating across the feature pyramid network at various scales for subsequent detection, localization, and temporal alignment tasks.

[0152] Fifth embodiment

[0153] As an alternative embodiment, based on any embodiment, refer to Figure 5 The diagram illustrates the training process and loss function of the video action recognition model. In this embodiment, a Task Aligned Learning (TAL) loss function is used, comprehensively considering classification accuracy and bounding box regression accuracy. The classification loss employs Binary Cross-Entropy (BCE) loss to optimize action category prediction. The regression loss combines CIoU loss and Distribution Focal Loss (DFL) to optimize bounding box localization accuracy, especially for small targets with severe occlusion. Dynamic sample allocation is based on SimOTA or a dynamic Top-k strategy, adaptively selecting high-quality positive sample anchors to address the imbalance between positive and negative samples.

[0154] Furthermore, the AdamW optimizer is used for backpropagation updates, and an exponential moving average (EMA) mechanism is introduced to maintain a shadow model whose parameters are a weighted average of historical parameters. This is used to smooth out oscillations during training and improve the generalization ability of the final model.

[0155] The video stream to be tested is input into the trained model, which is then processed by dual-stream backbone extraction, TDAF fusion, and decoupled detection head, and directly outputs spatiotemporal bounding boxes with action category labels and confidence scores.

[0156] Verification of Examples

[0157] This embodiment verifies the recognition effect of the video behavior recognition model obtained by the construction method of the video behavior recognition model involved in the foregoing embodiments.

[0158] The model was evaluated on the UCF101-24 dataset, using Frame-mAP (@IoU=0.5) as the core accuracy metric. Furthermore, we measured the model's actual inference speed (FPS) on an NVIDIA RTX 3090 GPU to rigorously assess its edge deployment potential.

[0159] On the one hand, this embodiment compares our proposed method with existing state-of-the-art (SOTA) methods, and the results are shown in Table 1:

[0160] Table 1. Performance Comparison Table

[0161]

[0162] As can be seen, the method proposed in this application has a significant advantage in the lightweight track. Compared to YOWOv3 (Zhu et al.)'s 81.44% mAP, Ours achieves 83.85% mAP on this configuration, an improvement of +2.41%. More importantly, the actual inference speed of this configuration reaches 43.37 FPS, far exceeding the real-time standard (30 FPS), proving that this method can significantly improve accuracy without sacrificing speed.

[0163] On the other hand, we also conducted ablation experiments and analyses, the results of which are shown in Table 2:

[0164] Table 2. Comparison of Ablation Experiment Results

[0165]

[0166] As can be seen, on UCF101-24, the original YOWOv3 (Nano without FCM), which serves as the baseline model, only achieved 88.00% mAP. After introducing FCM gain, the mAP improved to 88.62%. Furthermore, replacing the simple attention CFAM with the TDAF fusion strategy proposed in this application further improved the mAP to 89.20%. The TDAF proposed in this application achieved the best performance, demonstrating the superiority of the method proposed in this application.

[0167] In addition, to visually demonstrate the recognition effect of the proposed method, a visualization analysis was performed on the feature heatmap (Grad-CAM) and dynamic weight changes, referring to... Figure 6 and Figure 7 The point activation maps of the conventional baseline model and the region activation maps of the method proposed in this application are shown respectively.

[0168] from Figure 6 The heatmap shows that the baseline model only exhibits weak and isolated bright spots in the center or localized areas of the player's (plateholder, batter) body, predominantly in cool tones (cyan / yellow), with very few high-confidence red areas. This indicates that even with a larger YOLOv8-Medium backbone, the model still falls short in recognizing small targets. Although the model "sees" local textures, the feature responses are extremely weak and sparse. It fails to form a complete semantic understanding of the "human body" target, resembling more of a simple local texture matching, making it difficult to accurately locate the subject of the action.

[0169] and Figure 7 In the heatmap, strong and continuous red highlighted areas are observed. These highlighted areas completely cover the torsos and limbs of the pitcher, batter, and outfielder, even accurately capturing distal limbs with smaller movements. This visually demonstrates that the proposed method successfully enhances the expressive power of features. The model not only knows "there's a point here," but also clearly recognizes "this is a complete person." The features exhibit extremely high confidence and semantic completeness, proving that explicit spatial detail injection effectively retrieves geometric structures forgotten by deep networks, thus achieving a perceptual leap from "point" to "domain."

[0170] In addition, this embodiment also plotted the curve of the fusion weight changing over time and analyzed it in conjunction with pixel-level motion intensity. The result is as follows: Figure 8 and Figure 9 The diagrams show the changes in fusion weights for continuous motion scenarios and explosive motion scenarios, respectively.

[0171] from Figure 8 As we can see, fencing is a continuous game of strategy. We can see that the red curve (3D temporal weights) remains very stable at a high level of around 0.65. This indicates that during continuous motion, the model "intelligently" determines that motion information is more important than the static background, thus continuously weighting the 3D branches, demonstrating a stable dependence on motion cues.

[0172] from Figure 9 As can be easily seen, in the segment between Frames 30 and 40, which is the instant the player jumps to dunk, the movement is extremely fast and accompanied by dramatic changes. The model keenly captured this explosive point, and the red curve (3D weight) instantly soared to over 0.70.

[0173] In summary, the method proposed in this application does not rigidly fix weights, but rather adaptively adjusts attention according to the intensity of the action, much like the human visual system. This explains why our dynamic fusion strategy significantly outperforms static stitching schemes in high-performance scenarios—it utilizes spatial texture in static frames and temporal cues in dynamic frames, achieving optimal complementarity of spatiotemporal information.

[0174] As one implementation, this application embodiment also relates to a video behavior recognition method, which uses a video behavior recognition model obtained by the construction method of the video behavior recognition model as described in any of the preceding claims. The video behavior recognition model includes a feature complementary mapping module and an adaptive temperature-controlled dynamic attention feature fusion module. The video behavior recognition method includes the following steps:

[0175] S1000, perform frame-by-frame feature extraction on the input video sequence to obtain input features, which include two-dimensional spatial features and three-dimensional temporal features;

[0176] S2000, the feature semantics of the two-dimensional spatial features are enhanced by the feature complementary mapping module to obtain enhanced two-dimensional spatial features;

[0177] S3000, through the adaptive temperature-controlled dynamic attention feature fusion module, deeply fuses the enhanced two-dimensional spatial features and the three-dimensional temporal features to obtain fused features;

[0178] S4000, input the fused features into the classification head, and output the behavior category prediction result of the video sequence.

[0179] Further and optionally, the behavior category prediction result includes regression prediction result and classification prediction result. The step of inputting the fused features into the classification head and outputting the behavior category prediction result of the video sequence includes:

[0180] In decoupled mode, independent fusion modules and dynamic weight learning networks are built for the bounding box regression branch and classification branch respectively. Adaptive weights are used to output regression prediction results and classification prediction results respectively, so as to reduce feature conflicts between multiple tasks:

[0181]

[0182] In the formula, These are all adaptive weights for the bounding box regression branch. All are adaptive weights for the classification branches; This represents the upsampled three-dimensional temporal features. It represents the characteristics of two-dimensional space.

[0183] As one implementation scheme, refer to Figure 10The illustrated diagram shows a video behavior recognition model architecture. This application also relates to a video behavior recognition model obtained using the construction method described above, wherein the video behavior recognition model includes:

[0184] The input module is used to receive and preprocess video frame sequences;

[0185] A spatial backbone module is constructed by building a feature complementary mapping module at layer P5 of the feature pyramid of the spatial backbone module to obtain enhanced two-dimensional spatial features;

[0186] The temporal backbone module is used to extract three-dimensional temporal features;

[0187] An adaptive temperature-controlled dynamic attention feature fusion module is used to deeply fuse enhanced two-dimensional spatial features and three-dimensional temporal features;

[0188] The output module is used to generate and output video behavior recognition results.

[0189] In this embodiment, the video behavior recognition model sequentially includes an input module, a spatial backbone module, a temporal backbone module, an adaptive temperature-controlled dynamic attention feature fusion module, a detection head module, and an output module. The input module receives and preprocesses video frame sequences in T×H×W×C format. The spatial backbone is based on YOLOv8, introducing FCM at the P5 layer of the feature pyramid to calibrate and enhance multi-scale features; the temporal backbone uses the ShuffleNetv2 model to extract cross-frame motion and temporal relationships. The feature fusion module uses the TDAF fusion module to weight and fuse features in the channel, spatial, and temporal dimensions and implement dynamic weight learning. The detection head has a multi-scale structure, responsible for classification, localization, and temporal alignment; training can incorporate TAL loss to balance the various subtasks. The output module generates action categories and confidence scores. The modules are connected according to the path "input → (spatial / temporal) feature extraction → fusion → detection → output," with data flowing unidirectionally to complete video behavior detection.

[0190] As one implementation scheme, Figure 11 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0191] like Figure 11As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0192] Those skilled in the art will understand that Figure 11 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0193] like Figure 11 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.

[0194] exist Figure 11 In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.

[0195] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:

[0196] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0197] (1) Construct a feature complementary mapping module, so that when the feature complementary mapping module performs two-dimensional spatial feature semantic feature enhancement, it includes the following steps:

[0198] The two-dimensional spatial features in the video dataset are divided into shallow features and deep features, and the spatial resolution of the shallow features and the deep features are aligned.

[0199] The aligned shallow features are concatenated with the deep features, and the resulting concatenated features are introduced into a spatial channel attention mechanism to enhance key channels, thus obtaining enhanced features.

[0200] The deep features are added to the enhanced features by residual connection to obtain enhanced two-dimensional spatial features;

[0201] (2) Construct an adaptive temperature-controlled dynamic attention feature fusion module, so that when the adaptive temperature-controlled dynamic attention feature fusion module performs feature fusion enhancement, it includes the following steps:

[0202] The enhanced two-dimensional spatial features and three-dimensional temporal features in the video dataset are obtained, and the enhanced two-dimensional spatial features and three-dimensional temporal features of the same scale are concatenated in the channel dimension to obtain the concatenated fused feature map.

[0203] Extract the fusion semantic vector from the spliced ​​fusion feature map, generate fusion weights through a weight learning network, and introduce a variable temperature parameter for scaling to obtain adaptive weights. The magnitude of the variable temperature parameter is positively correlated with the behavioral amplitude in the video dataset.

[0204] The adaptive weights, along with the channel attention weights and spatial attention weights obtained after cross-attention processing of the video dataset, are fused and weighted to obtain fused features.

[0205] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0206] The input video sequence is subjected to frame-by-frame feature extraction to obtain input features, which include two-dimensional spatial features and three-dimensional temporal features.

[0207] The feature semantics of the two-dimensional spatial features are enhanced by the feature complement mapping module to obtain enhanced two-dimensional spatial features;

[0208] The enhanced two-dimensional spatial features and the three-dimensional temporal features are deeply fused by the adaptive temperature-controlled dynamic attention feature fusion module to obtain fused features.

[0209] The fused features are input into the classification head, and the behavior category prediction results of the video sequence are output.

[0210] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.

[0211] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the video behavior recognition model construction method described in the above embodiments.

[0212] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0213] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0214] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0215] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0216] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0217] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0218] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0219] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0220] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for constructing a video behavior recognition model, characterized in that, The method for constructing the video behavior recognition model includes the following steps: (1) Construct a feature complementary mapping module, so that when the feature complementary mapping module performs two-dimensional spatial feature semantic feature enhancement, it includes the following steps: The two-dimensional spatial features in the video dataset are divided into shallow features and deep features, and the spatial resolution of the shallow features and the deep features are aligned. The aligned shallow features are concatenated with the deep features, and the resulting concatenated features are introduced into a spatial channel attention mechanism to enhance key channels, thus obtaining enhanced features. The deep features are added to the enhanced features by residual connection to obtain enhanced two-dimensional spatial features; (2) Construct an adaptive temperature-controlled dynamic attention feature fusion module, so that when the adaptive temperature-controlled dynamic attention feature fusion module performs feature fusion enhancement, it includes the following steps: The enhanced two-dimensional spatial features and three-dimensional temporal features in the video dataset are obtained, and the enhanced two-dimensional spatial features and three-dimensional temporal features of the same scale are concatenated in the channel dimension to obtain the concatenated fused feature map. Extract the fusion semantic vector from the spliced ​​fusion feature map, generate fusion weights through a weight learning network, and introduce a variable temperature parameter for scaling to obtain adaptive weights. The magnitude of the variable temperature parameter is positively correlated with the behavioral amplitude in the video dataset. The adaptive weights, along with the channel attention weights and spatial attention weights obtained after cross-attention processing of the video dataset, are fused and weighted to obtain fused features.

2. The method for constructing a video behavior recognition model as described in claim 1, characterized in that, The steps of extracting the fusion semantic vector from the concatenated fusion feature map, generating fusion weights for two-dimensional and three-dimensional features through a weight learning network, and introducing a variable temperature parameter for softmax temperature scaling to obtain adaptive weights include: Two-dimensional spatial features With three-dimensional temporal features Global average pooling is performed after concatenation: ； In the formula, Indicates global average pooling; Weight vectors are generated using a fully connected MLP layer, and a temperature parameter is introduced. Softmax scaling is applied to obtain adaptive weights: 。 3. The method for constructing a video behavior recognition model as described in claim 2, characterized in that, The step of fusing and weighting the input features with the adaptive weights, the channel attention weights obtained after channel attention processing of the input features, and the spatial attention weights to obtain the fused features includes: Obtain Channel Attention Weights : ； In the formula, For global average pooling, For global max pooling, Here, F is the Sigmoid activation function, and F is the input feature. And obtain spatial attention weights : ； In the formula, Conv7*7 represents a convolution of size 7*7, AvgPool represents average pooling, and MaxPool represents max pooling. The input features are fused and weighted based on adaptive weights, channel attention weights, and spatial attention weights: ； In the formula, F TDAM As a feature of fusion, Indicates original features; Indicates enhanced features, This indicates element-wise multiplication.

4. The method for constructing a video behavior recognition model as described in claim 1, characterized in that, The expression for the enhanced two-dimensional spatial features is: ； in: ； ； ； ； ； In the formula, To enhance the features of two-dimensional space; This represents the deep features after applying spatial attention weights. This represents the shallow features after applying channel attention weights; Spatial attention weights, Channel attention weights; This represents shallow features that have been upsampled to the same resolution as the deep features. This represents the deep features after performing a 1x1 convolution operation; Indicates deep features; Indicates shallow features; For global average pooling; This is the Sigmoid activation function.

5. A video behavior recognition method, characterized in that, A video behavior recognition model obtained by constructing a video behavior recognition model according to any one of claims 1 to 4, the video behavior recognition model comprising a feature complementary mapping module and an adaptive temperature-controlled dynamic attention feature fusion module, the video behavior recognition method comprising the following steps: The input video sequence is subjected to frame-by-frame feature extraction to obtain input features, which include two-dimensional spatial features and three-dimensional temporal features. The feature semantics of the two-dimensional spatial features are enhanced by the feature complement mapping module to obtain enhanced two-dimensional spatial features; The enhanced two-dimensional spatial features and the three-dimensional temporal features are deeply fused by the adaptive temperature-controlled dynamic attention feature fusion module to obtain fused features. The fused features are input into the classification head, and the behavior category prediction results of the video sequence are output.

6. The video behavior recognition method as described in claim 5, wherein the behavior category prediction result includes regression prediction result and classification prediction result, and the step of inputting the fused features into a classification head and outputting the behavior category prediction result of the video sequence includes: In decoupled mode, independent fusion modules and dynamic weight learning networks are built for the bounding box regression branch and classification branch respectively. Adaptive weights are used to output regression prediction results and classification prediction results respectively, so as to reduce feature conflicts between multiple tasks: ； In the formula, These are all adaptive weights for the bounding box regression branch. All are adaptive weights for the classification branches; This represents the upsampled three-dimensional temporal features. It represents the characteristics of two-dimensional space.

7. A video behavior recognition model obtained by constructing a video behavior recognition model according to any one of claims 1 to 4, characterized in that, The video behavior recognition model includes: The input module is used to receive and preprocess video frame sequences; A spatial backbone module is constructed by building a feature complementary mapping module at layer P5 of the feature pyramid of the spatial backbone module to obtain enhanced two-dimensional spatial features. The temporal backbone module is used to extract three-dimensional temporal features; An adaptive temperature-controlled dynamic attention feature fusion module is used to deeply fuse enhanced two-dimensional spatial features and three-dimensional temporal features; The output module is used to generate and output video behavior recognition results.

8. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the method for constructing a video behavior recognition model as described in any one of claims 1 to 4 or the video behavior recognition method as described in any one of claims 5 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for constructing a video behavior recognition model as described in any one of claims 1 to 4 or the video behavior recognition method as described in any one of claims 5 to 6.

Images