A video behavior recognition method based on a frequency domain attention mechanism

Abstract

The application discloses a video behavior recognition method based on a frequency domain attention mechanism and belongs to the technical field of computer vision. The method comprises the following steps: obtaining a video to be recognized and dividing the video into continuous video segments; inputting the video segments into a pre-trained behavior recognition model, wherein the model comprises a two-dimensional convolutional neural network branch for extracting spatial appearance features, a three-dimensional convolutional neural network branch for extracting space-time dynamic features, and a fusion detection head for feature fusion and recognition output; wherein the three-dimensional convolutional neural network branch comprises a multi-head local frequency spectrum attention module, the module realizes frequency domain feature enhancement by performing frequency domain transformation, single-head internal frequency domain interaction, adaptive frequency spectrum modulation and complex activation function processing on a space-time feature map. The application effectively separates low-frequency behavior signals and high-frequency environmental noise through frequency domain feature enhancement, and significantly improves the accuracy and robustness of behavior recognition on self-built data sets and public data sets.

Description

Technical Field

[0001] This invention relates to the field of computer vision and video analysis technology, and more specifically, to a video behavior recognition method based on a frequency domain attention mechanism, which is particularly suitable for refined behavior monitoring of captive animals in complex environments. Background Technology

[0002] In recent years, with the increasing efforts to protect captive animals, the importance of monitoring animal behavior in captive environments has become increasingly prominent. Behavioral research is crucial for improving husbandry management, enhancing animal welfare, and promoting reproductive success. Currently, the collection of captive animal behavioral data mainly relies on manual observation, which is not only inefficient and subjective but also difficult to achieve continuous, 24 / 7 behavioral monitoring. Although video surveillance systems have been widely used in captive environments, generating a large amount of behavioral video data, the lack of effective automated analysis tools results in low data resource utilization and delays in identifying key behavioral events, potentially missing important breeding signals or health warnings.

[0003] With the development of deep learning technology, behavior recognition methods based on convolutional neural networks have made significant progress. In early studies, Matkowski et al. used a small-scale facial image dataset to achieve individual panda recognition. Subsequent studies have further improved recognition accuracy through head segmentation, multi-feature fusion, and other methods. However, these methods are mainly based on static images and are difficult to capture dynamic changes in behavior. Duan et al. pointed out that behavior recognition requires temporal sequence information and constructed a video dataset for panda behavior, which has promoted the development of video behavior analysis.

[0004] In the field of general action recognition, existing mainstream methods include 3D convolutional neural network-based methods, two-stream networks, SlowFast networks, and the YOWO series models. Among them, the YOWO series models, by combining 2D and 3D convolutional neural networks, achieve the fusion of spatial and spatiotemporal features, demonstrating excellent performance in real-time detection. However, when these general action recognition models are directly applied to the recognition of captive animal behavior, the following technical problems are encountered:

[0005] First, captive animals have unique fur colors that often blend closely with the environmental background, making feature extraction difficult. At the same time, their behavior is mostly characterized by low-frequency, slow, and continuous movement patterns, while environmental disturbances, such as swaying vegetation, changes in light and shadow, and reflections, are mainly distributed in the high-frequency range. The frequency domain features of the two overlap, making it difficult for existing models to effectively separate behavioral signals from environmental noise.

[0006] Second, existing attention mechanisms mainly focus on channel domain or spatial domain features. For example, the compression-excitation network (SE) only models the relationship between channels, and although the convolutional block attention module (CBAM) combines channel and spatial information, neither of them considers frequency domain features. In recent years, frequency domain attention methods have mostly been combined with the visual transformer (ViT) architecture. Although they can model global temporal dependencies, they are difficult to effectively capture local frequency domain features and have high computational complexity.

[0007] Third, in captive environments, factors such as fence obstruction and changes in lighting lead to information loss, further increasing the difficulty of behavior recognition; existing models are not robust enough when facing complex scenarios such as partial occlusion and motion blur.

[0008] In summary, the existing technology lacks a video behavior recognition method that can effectively utilize frequency domain information and improve the accuracy and robustness of behavior recognition in complex environments. Summary of the Invention

[0009] The present invention aims to solve the above-mentioned problems in the prior art and provides a video behavior recognition method based on frequency domain attention mechanism. By introducing frequency domain feature enhancement, the method improves the ability to extract low-frequency behavior signals and suppresses high-frequency environmental noise, thereby improving the accuracy and robustness of behavior recognition.

[0010] To achieve the above objectives, this invention provides a video behavior recognition method based on a frequency domain attention mechanism, comprising the following steps:

[0011] Step S1: Obtain the video to be identified and divide the video to be identified into video segments containing multiple consecutive frames of images;

[0012] Step S2: Input the video clip into a pre-trained behavior recognition model, the behavior recognition model comprising:

[0013] A two-dimensional convolutional neural network branch is used to extract the spatial appearance features of the video segment;

[0014] A three-dimensional convolutional neural network branch is used to extract the spatiotemporal dynamic features of the video segment;

[0015] And a fusion detection head, used to fuse the spatial appearance features and spatiotemporal dynamic features, and output behavior recognition results.

[0016] The two-dimensional convolutional neural network branch includes a C3k2 structure, a C2PSA module, a spatial pyramid pooling module, and a dual-coordinate attention feature enhancement module. The C3k2 structure is used for parameter-adaptive multi-scale feature extraction. The C2PSA module combines multi-head attention and a feedforward network to enhance feature representation. The dual-coordinate attention feature enhancement module is used to enhance spatial relationship modeling.

[0017] The three-dimensional convolutional neural network branch includes at least one multi-head local spectral attention module, which is used to perform frequency domain transformation and feature enhancement on the input spatiotemporal feature map.

[0018] Furthermore, the multi-head local spectral attention module processes the input spatiotemporal feature map, specifically including the following sub-steps:

[0019] Step S21: Flatten the input feature map into a one-dimensional sequence in the spatial dimension and perform layer normalization;

[0020] Step S22: Divide the normalized feature map into multiple attention heads along the channel dimension;

[0021] Step S23: Perform a real-number fast Fourier transform on the sequence within each attention head to obtain the complex feature tensor in the frequency domain;

[0022] Step S24: Decompose the complex feature tensor into real and imaginary parts, and perform dimension permutation in the frequency domain dimension;

[0023] Step S25: Perform grouped convolution and standard convolution operations on the real and imaginary parts after permutation respectively to achieve frequency domain interaction within a single head;

[0024] Step S26: Reassemble the processed real and imaginary parts into a complex number and perform spectrum modulation through an adaptive modulation module. The adaptive modulation module consists of a global average pooling layer and a multilayer perceptron, and is used to generate spectrum modulation coefficients.

[0025] Step S27: The amplitude of the modulated complex feature is adjusted using a complex activation function while keeping the phase unchanged;

[0026] Step S28: Perform an inverse fast Fourier transform on the processed complex features to restore them to the time domain, and perform a residual connection with the original input to output the enhanced feature map.

[0027] Furthermore, the fusion detection head includes a spatial-channel collaborative attention mechanism to achieve collaborative feature modeling of spatial and channel dimensions.

[0028] Furthermore, the method also includes the following steps: constructing a behavior recognition dataset, the dataset containing multiple behavior categories, each behavior category corresponding to a preset number of video samples, the video samples being from captive environment monitoring videos, and divided into training set and test set according to a preset ratio.

[0029] Furthermore, the training process of the behavior recognition model includes: iteratively training the training set using a preset initial learning rate, optimizer, and weight decay parameters until the loss function converges.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] This invention combines frequency domain analysis with a multi-head attention mechanism by designing a multi-head local spectral attention module, fully utilizing the energy distribution characteristics of target behavior in the frequency domain. Specifically, the behavior of captive animals is characterized by low-frequency, slow movements, with their energy concentrated in the low-frequency band of the spectrum, while environmental disturbances, such as vegetation swaying and reflections, are mainly distributed in the high-frequency band. This invention effectively separates behavioral signals from environmental noise through frequency domain transformation and single-head internal frequency domain interaction, achieving targeted feature enhancement. Experimental results show that on a self-built captive animal behavior dataset, the average frame-level accuracy of this invention reaches 73.94%, which is 6.81 percentage points higher than the baseline model YOWOv2-Tiny.

[0032] The technical solution of this invention not only performs excellently in animal behavior recognition tasks, but also demonstrates outstanding generalization ability on publicly available human action recognition datasets. On the UCF101-24 dataset, the average frame-level precision of this invention reaches 81.74%, and the average video-level precision reaches 50.70%. On the JHMDB21 dataset, the average frame-level precision reaches 60.98%, and the average video-level precision reaches 69.81%. These results indicate that the frequency domain feature enhancement method proposed in this invention is universal and can be widely applied to behavior recognition tasks in different scenarios.

[0033] This invention improves recognition accuracy while maintaining low computational complexity. Specifically, the computational load of this invention is 3.35G floating-point operations and the number of parameters is 14.9M, achieving a good balance between accuracy and efficiency, and is suitable for practical deployment needs. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the captive environment in an embodiment of the present invention;

[0035] Figure 2 This is an overall architecture diagram of the improved YOWOv2 behavior recognition model in this embodiment of the invention;

[0036] Figure 3 This is a schematic diagram of the structure of a two-dimensional convolutional neural network branch in an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of the structure of a three-dimensional convolutional neural network branch in an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of the fusion detection head in an embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram illustrating the working principle of the multi-head local spectrum attention module in an embodiment of the present invention;

[0040] Figure 7 This is a schematic diagram of the behavior recognition results in an embodiment of the present invention;

[0041] Figure 8 This is a schematic diagram of the behavior recognition results in an embodiment of the present invention;

[0042] Figure 9 This is a schematic diagram of the behavior recognition results in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in further detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the following embodiments are only for illustrating the invention and are not intended to limit the invention.

[0044] This embodiment provides a method for constructing a dataset for behavioral recognition of captive animals, where the captive environment is as follows: Figure 1 As shown; the dataset is named the PBA dataset; the data sources are mainly publicly available online video footage and self-shot footage; the video footage covers the behavior of captive animals under different times, lighting conditions, and shooting angles;

[0045] During the data collection process, a total of 2,369 video clips were selected, each with a duration of 30 seconds. Each video clip was labeled at a sampling rate of 1 frame per second, resulting in 137,314 behavioral tags. The labeled behavioral categories were divided into three main categories: action, activity, and pregnancy-related, covering a total of 24 specific behaviors. Among them, action included behaviors such as continuous lying down, continuous standing, continuous sitting, continuous prone, continuous standing, running, walking, climbing, rubbing, and rolling; activity included behaviors such as drinking, eating, moving, resting, exploring, playing, marking, excretion, and grooming; and pregnancy-related behaviors included behaviors such as grasping, licking the chest and abdomen, licking the vulva, curling up, and rubbing the vulva.

[0046] The dataset is divided into a training set and a test set in a 9:1 ratio, with the training set containing 123,104 annotations and the test set containing 14,210 annotations.

[0047] Those skilled in the art will understand that the above-described dataset construction method is merely an exemplary implementation of the present invention. In practical applications, parameters such as video duration, sampling frequency, and behavior category classification can be adjusted according to specific target objects and scenarios.

[0048] The overall architecture of the behavior recognition model Figure 2 As shown, this embodiment provides an improved behavior recognition model, which is optimized based on the YOWOv2-Tiny architecture. The behavior recognition model adopts a dual-stream architecture, including a two-dimensional convolutional neural network branch, a three-dimensional convolutional neural network branch, and a fusion detection head.

[0049] The two-dimensional convolutional neural network branch is used to extract the spatial appearance features of the input video clip; the input video clip consists of 16 consecutive frames, each with a size of 224×224 pixels; the two-dimensional convolutional neural network branch adopts an improved feature extraction network, the specific structure of which is as follows: Figure 3 As shown;

[0050] A 3D convolutional neural network branch is used to extract spatiotemporal dynamic features of video clips. This branch uses ShuffleNetV2 as its backbone network and embeds the multi-head local spectral attention module proposed in this invention to achieve frequency domain feature enhancement. The specific structure of the 3D convolutional neural network branch is as follows: Figure 4 As shown;

[0051] The fusion detection head is used to fuse features extracted from 2D and 3D branches, and outputs behavior classification and behavior localization results. The fusion detection head adopts a decoupled structure and introduces a spatial-channel collaborative attention mechanism to replace the channel attention in the original model, achieving collaborative feature modeling of spatial and channel dimensions. The specific structure of the fusion detection head is as follows: Figure 5 As shown.

[0052] Two-dimensional convolutional neural network branches, such as Figure 3 As shown in the figure, this embodiment provides a detailed description of the two-dimensional convolutional neural network branch; the two-dimensional convolutional neural network branch includes an input layer, a feature extraction backbone network, and an output layer;

[0053] The feature extraction backbone network adopts the C3k2 structure for parameter adaptive multi-scale feature extraction. The C3k2 structure enables the network to dynamically adjust the receptive field according to different input features by adaptively selecting the convolution kernel size, thereby more effectively capturing spatial features at different scales.

[0054] Following the C3k2 structure, the network sequentially includes a C2PSA module and a spatial pyramid pooling module. The C2PSA module combines a multi-head attention mechanism with a feedforward network to enhance feature representation capabilities. The spatial pyramid pooling module is used for multi-scale feature encoding, capturing multi-scale contextual information through different pooling sizes.

[0055] In addition, the two-dimensional convolutional neural network branch also includes a dual-coordinate attention feature enhancement module; this module is used to enhance spatial relationship modeling. By considering spatial attention in both the horizontal and vertical directions at the same time, it effectively captures the pose and orientation features of the target object, which is particularly important for recognizing animal behaviors with complex pose changes.

[0056] The spatial appearance feature map output by the two-dimensional convolutional neural network branch is input into the fusion detection head and fused with the spatiotemporal dynamic features extracted by the three-dimensional branch.

[0057] Multi-head local spectral attention modules, such as Figure 6 As shown, this embodiment provides a detailed description of the multi-head local spectrum attention module proposed in this invention;

[0058] The design of the multi-head local spectrum attention module is based on the following technical insights: In the task of recognizing behavior in captive animals, the target behavior is usually manifested as a low-frequency, slow, and continuous movement pattern, and its energy is concentrated in the low-frequency band in the frequency domain; while the interference in the complex environmental background, such as vegetation swaying, reflection, and changes in lighting, is mainly distributed in the high-frequency band; by targeting the enhancement of frequency domain features, the discriminativeness of behavioral features can be effectively improved.

[0059] Let the input feature map be ,in For the number of channels, These represent the height and width of the feature map, respectively. Representing the real number field; the processing flow of the multi-head local spectral attention module for the input feature map is as follows:

[0060] The first step is shape transformation and normalization; this involves transforming the input feature map. Flattening it in the spatial dimension and merging the height and width dimensions, we get... , The flattened feature map is then represented; subsequently, layer normalization is performed on this tensor to stabilize the training process.

[0061] The second step is multi-head segmentation; the normalized feature map is segmented along the channel dimension. There are 1 attention head, and the channel dimension of each head is 1. After division, we obtain , This represents the feature map after multi-head segmentation;

[0062] The third step is frequency domain transformation; for each attention head, the sequence dimension ( Perform a real-number fast Fourier transform to obtain the complex characteristic tensor in the frequency domain. ,in For the frequency domain dimension, The number of channels for each head, Represents the field of complex numbers;

[0063] The fourth step involves decomposing and permuting the real and imaginary parts of the complex tensor. Decompose into real part and the virtual part ,Right now Subsequently, the real and imaginary parts are dimensionally replaced to change the frequency domain dimension. With feature dimension The fifth step involves swapping positions and performing frequency domain interaction within each attention head. This involves processing the real and imaginary tensors after the permutation in the frequency domain. Specifically, first, grouped convolutions are performed, where the number of groups equals the number of attention heads. The number of input channels is The number of output channels is , The expansion coefficients are then used; a non-linear transformation is performed through an activation function, followed by standard convolution to restore the number of channels. The above convolution operations are all performed in the frequency domain, enabling information exchange between different frequency components within a single head.

[0064] Step 6: Adaptive Spectrum Modulation. The real and imaginary parts after frequency domain interaction are replaced back to their original dimensions and recombined into complex form. Subsequently, the spectrum is modulated through an adaptive modulation module. This module first performs global average pooling on the normalized feature sequence to obtain context statistics. Then, the statistics are input into a multilayer perceptron to generate spectrum modulation parameters, including amplitude adjustment coefficients and phase offset. The modulation process includes multiplicative modulation (amplitude scaling) and additive modulation (phase offset).

[0065] Step 7, Complex Activation Function; To maintain the integrity of phase information, this embodiment uses an improved complex activation function: ;in, For complex modulus, This indicates that the GELU activation function is used. To prevent division by zero for small constants, This represents complex number multiplication; the activation function only adjusts the magnitude, keeping the phase unchanged.

[0066] Step 8: Inverse transform and residual concatenation; perform inverse fast Fourier transform on the modulated complex features to restore them to the time domain; finally, perform residual concatenation with the original input feature map to output the enhanced feature map.

[0067] In a preferred embodiment, the number of attention heads in the multi-head local spectral attention module Set to 2, expansion factor Set it to 2; by adjusting the above parameters, the computational complexity and feature representation ability of the model can be balanced.

[0068] Model training: This embodiment describes the training process of the behavior recognition model.

[0069] The training environment is configured as follows: an Intel Core i3-12100F CPU and an NVIDIA RTX 2080 Super GPU (8GB VRAM) are used, the operating system is Windows 10, and the model is developed and trained based on the CUDA 11.8 and PyTorch deep learning framework.

[0070] The training hyperparameters were set as follows: the initial learning rate was 1e-4, the optimizer was AdamW, the weight decay parameter was 5e-4, a linear warm-up strategy was adopted, and the number of warm-up iterations was 500; the input video segments were uniformly sampled as 16 frames, the image size was adjusted to 224×224 pixels, and the batch size was 8; to ensure the reproducibility of the experiment, a fixed random seed of 88 was used throughout.

[0071] The number of training epochs is adjusted according to different datasets: For the UCF101-24 dataset, 7 epochs are trained, and the learning rate is halved after epochs 1, 2, 3, and 4; For the JHMDB21 dataset, 10 epochs are trained, and the learning rate is halved after epochs 3, 4, 5, and 6; For the self-built dataset, 30 epochs are trained, and the learning rate decay strategy is the same as that for JHMDB21.

[0072] Experimental Results and Verification: To verify the effectiveness of the technical solution of this invention, multiple sets of comparative experiments and ablation experiments were conducted in this embodiment.

[0073] 1. Experimental setup

[0074] The experiment uses frame-level average precision as the main evaluation index, which measures the detection accuracy of behavior localization boxes in each frame of image under the condition that the cross-union ratio threshold is 0.5. For the public dataset, video-level average precision is also used as a supplementary evaluation index, which measures the localization accuracy of behavior channels in three-dimensional space.

[0075] 2. Comparative experiment with mainstream methods

[0076] The method of this invention was compared with existing mainstream behavior recognition methods on a self-built dataset. The comparison methods included C3D, I3D, SlowFast, YOWO, YOWOv2 series and YOWOv3 series models. The experimental results are shown in Table 1.

[0077] Table 1. Comparison of experimental results with mainstream methods

[0078]

[0079] Experimental results show that the method of this invention achieves an average frame-level accuracy of 73.94%, which is superior to all comparative methods. Specifically, the method of this invention improves accuracy by 6.81 percentage points compared to the baseline model YOWOv2-Tiny (67.13%). It is worth noting that although the YOWOv2-Large model achieves an accuracy of 72.68%, its computational cost is 53.6 G floating-point operations, which is much higher than the 3.35 G floating-point operations of the method of this invention. This indicates that the method of this invention maintains high computational efficiency while improving accuracy, achieving a good balance between accuracy and efficiency.

[0080] Comparative experiments on public datasets further validated the generalization ability of the present invention;

[0081] Table 2 Comparative Experiment Results on Public Datasets

[0082]

[0083] Table 2 shows that on the UCF101-24 dataset, the average frame-level accuracy of the method of this invention is 81.74%, and the average video-level accuracy is 50.70%, which are 1.24 and 0.79 percentage points higher than YOWOv2-Tiny, respectively. On the JHMDB21 dataset, the average frame-level accuracy of the method of this invention is 60.98%, and the average video-level accuracy is 69.81%, which is also better than the comparison method.

[0084] 3. Comparative Experiment on Attention Mechanisms

[0085] To verify the effectiveness of the multi-head local spectral attention module proposed in this invention, it was compared with existing mainstream attention mechanisms. The comparison methods included compressed-excitation networks, efficient channel attention, coordinate attention, convolutional block attention modules, spectral transformers, and spectral gated attention. The experimental results are shown in Table 3.

[0086] Table 3. Results of comparative experiments on attention mechanisms

[0087]

[0088] Experimental results show that the multi-head local spectral attention module proposed in this invention achieves a frame-level average accuracy of 73.94% on a self-built dataset, which is significantly better than all the comparison methods. Among them, the best performing comparison method, the convolutional block attention module, achieves 70.84%. This invention improves upon the convolutional block attention module by 3.10 percentage points, demonstrating the important value of frequency domain feature modeling for animal behavior recognition.

[0089] 4. Ablation test

[0090] To verify the effectiveness of each component of the present invention, ablation experiments were conducted. Using YOWOv2-Tiny as the baseline model, C3k2 structure, dual-coordinate attention, spatial-channel collaborative attention and multi-head local spectral attention modules were added step by step. The experimental results are shown in Table 4.

[0091] Table 4 Ablation Experiment Results

[0092]

[0093] Experimental results show that adding each module individually can improve performance; among them, the spatial-channel collaborative attention module, when used alone, improves the average frame-level accuracy to 69.38%; the multi-head local spectrum attention module, when used alone, improves it to 69.15%; when all four modules are used together, the performance reaches the optimal level, with an average frame-level accuracy of 73.94%, which verifies the synergistic effect between the modules.

[0094] Further ablation of sub-modules of the multi-head local spectrum attention module;

[0095] Table 5 Ablation Experiment Results of MLSA Submodule

[0096]

[0097] As shown in Table 5, the average frame-level accuracy is 70.73% when only the adaptive modulation module is used; the accuracy is 69.92% when only the single-head internal frequency domain interaction module is used; and the accuracy reaches 73.94% when both are used in combination. This proves that the functions of the two sub-modules are complementary and together constitute a complete frequency domain feature enhancement mechanism.

[0098] 5. Hyperparameter Analysis

[0099] The key hyperparameters of the multi-head local spectral attention module were analyzed;

[0100] Table 6. Influence of segmentation head number and expansion coefficient parameters on the results of the MLSA attention mechanism.

[0101]

[0102] The results in Table 6 show that the model performs best when the number of attention heads is 2 and the expansion coefficient is 2, with an average frame-level precision of 73.94%. Performance decreases to varying degrees when deviating from these settings, indicating that appropriate attention head numbers and expansion coefficients help achieve effective frequency domain feature interaction.

[0103] Table 7. Influence of Dropout Rate Parameter on MLSA Attention Mechanism

[0104]

[0105] Table 7 shows that the model performs best when the dropout rate is 0.2. Too high or too low a dropout rate will lead to a decrease in performance, indicating that appropriate regularization helps to prevent overfitting.

[0106] 6. Visual Analysis

[0107] Figures 7 to 9 The results of behavior recognition using the method of the present invention in real-world scenarios are demonstrated. Under complex backgrounds and partial occlusion conditions, the model can reliably detect "continuous sitting" and "eating" behaviors with confidence levels of 0.94 and 0.95, respectively. Under conditions of small targets and image blur, the model successfully identifies "continuous prone" and "resting" behaviors with confidence levels of 0.8 and 0.91, respectively. Under conditions of overexposure and motion blur, the model can still accurately locate "climbing" and "moving" behaviors. The above visualization results verify the robustness of the method of the present invention in complex scenarios.

[0108] The video behavior recognition method based on frequency domain attention mechanism provided by this invention can be widely used in fields such as captive animal behavior monitoring, intelligent zoo management, and animal behavior research. In particular, this invention can effectively cope with challenging scenarios such as complex environments, occlusion, and changes in lighting, and achieve high-precision and robust behavior recognition, which has good prospects for industrial application.

[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, several improvements and modifications can be made without departing from the spirit and principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A video behavior recognition method based on a frequency domain attention mechanism, characterized in that, Includes the following steps: Step S1: Obtain the video to be identified and divide the video to be identified into video segments containing multiple consecutive frames of images; Step S2: Input the video clip into a pre-trained behavior recognition model, the behavior recognition model comprising: A two-dimensional convolutional neural network branch is used to extract the spatial appearance features of the video segment; A three-dimensional convolutional neural network branch is used to extract the spatiotemporal dynamic features of the video segment; And a fusion detection head, used to fuse the spatial appearance features and spatiotemporal dynamic features, and output behavior recognition results.

2. The video behavior recognition method based on frequency domain attention mechanism according to claim 1, characterized in that, The two-dimensional convolutional neural network branch includes a C3k2 structure, a C2PSA module, a spatial pyramid pooling module, and a dual-coordinate attention feature enhancement module; The C3k2 structure is used for parameter-adaptive multi-scale feature extraction; the C2PSA module combines multi-head attention and feedforward networks to enhance feature representation; and the dual-coordinate attention feature enhancement module is used to enhance spatial relationship modeling.

3. The video behavior recognition method based on frequency domain attention mechanism according to claim 1, characterized in that, The three-dimensional convolutional neural network branch includes at least one multi-head local spectral attention module, which is used to perform frequency domain transformation and feature enhancement on the input spatiotemporal feature map; The multi-head local spectral attention module processes the input spatiotemporal feature map, specifically including the following sub-steps: Step S21: Flatten the input feature map into a one-dimensional sequence in the spatial dimension and perform layer normalization; Step S22: Divide the normalized feature map into multiple attention heads along the channel dimension; Step S23: Perform a real-number fast Fourier transform on the sequence within each attention head to obtain the complex feature tensor in the frequency domain; Step S24: Decompose the complex feature tensor into real and imaginary parts, and perform dimension permutation in the frequency domain dimension; Step S25: Perform grouped convolution and standard convolution operations on the real and imaginary parts after permutation respectively to achieve frequency domain interaction within a single head; Step S26: Reassemble the processed real and imaginary parts into a complex number and perform spectrum modulation through an adaptive modulation module. The adaptive modulation module consists of a global average pooling layer and a multilayer perceptron, and is used to generate spectrum modulation coefficients. Step S27: The amplitude of the modulated complex feature is adjusted using a complex activation function while keeping the phase unchanged; Step S28: Perform an inverse fast Fourier transform on the processed complex features to restore them to the time domain, and perform a residual connection with the original input to output the enhanced feature map.

4. The video behavior recognition method based on frequency domain attention mechanism according to claim 1, characterized in that, The fusion detection head includes a spatial-channel collaborative attention mechanism, which is used to achieve collaborative feature modeling of spatial and channel dimensions.

5. The video behavior recognition method based on frequency domain attention mechanism according to claim 1, characterized in that, The method further includes the following steps: constructing a behavior recognition dataset, which contains multiple behavior categories, each behavior category corresponding to a preset number of video samples, the video samples being from captive environment monitoring videos, and divided into training set and test set according to a preset ratio.

6. The video behavior recognition method based on frequency domain attention mechanism according to claim 1, characterized in that, The training process of the behavior recognition model includes: using a preset initial learning rate, optimizer, and weight decay parameters, iteratively training the training set until the loss function converges.

Images