Multi-modal behavior recognition method based on progressive guide fusion

Abstract

The invention provides a multi-modal behavior recognition method based on progressive guide fusion, and belongs to the technical field of computer vision and artificial intelligence. The method comprises the following steps: performing skeleton-guided dynamic cutting preprocessing on input RGB data; extracting RGB and skeleton features through a WTSP enhancement network and an MSWTNet respectively; the extracted features are transmitted to a progressive guidance fusion network, skeleton features are strengthened through skeleton self-guidance, and then bimodal features are integrated through skeleton guidance RGB frequency domain fusion; and finally, behavior recognition is completed by using a space-time classification module. According to the method, the complementarity of RGB and skeleton data is fully mined, and the accuracy and robustness of multi-modal behavior recognition are effectively improved. According to the method, the accuracy rate of the NTU-60C-sub protocol reaches 95.8%, the accuracy rate of the C-view protocol reaches 98.6%, the calculated amount is reduced by 20% or above, and the method is suitable for intelligent monitoring, medical rehabilitation and other scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a multimodal behavior recognition method based on progressive guided fusion. Background Technology

[0002] In human social interaction systems, behavioral modalities, as the core carriers of information transmission, bear the important functions of emotional cognition and intention expression, and have a profound impact on shaping individual social behavior and constructing group interaction patterns. Against the backdrop of rapid development in intelligent technology, the application of multimodal behavior recognition technology in fields such as human-computer collaborative interaction, intelligent security monitoring, and abnormal behavior early warning continues to deepen. Industry practices have placed higher technical demands on the accuracy of behavioral feature extraction and the detail of action semantic analysis.

[0003] In the field of behavior recognition research, the limitations of single modalities are becoming increasingly apparent. Taking the RGB modality as an example, although cutting-edge models such as 3D-CNN and VisionTransformer have made some progress in spatiotemporal feature representation, their extremely high computational complexity places stringent demands on hardware resources, severely restricting their efficient deployment on edge devices. In practical applications, significant changes in lighting conditions can lead to drastic changes in image color and contrast, and complex background environments can also interfere with the accurate recognition of target behaviors. Furthermore, for actions like "drinking water" and "raising a cup," which appear similar but have fundamentally different motion trajectories and behavioral intentions, the RGB modality, relying solely on visual appearance features, struggles to achieve accurate behavior classification. The skeletal modality, using human joint coordinates as its core representation, exhibits good robustness against background noise and can intuitively reflect human motion information, giving it a unique advantage in behavior recognition. However, current mainstream methods based on graph convolutional networks use fixed adjacency matrices to model joint relationships, which cannot effectively handle the dynamic changes in joint connections under different behavioral scenarios. When dealing with non-standard behaviors, such as special occupational actions or abnormal behaviors, the model's recognition performance significantly declines. Although Transformer-based models optimize joint relationship modeling by introducing an attention mechanism, this comes at the cost of a significant increase in the size of the model parameters, which not only significantly prolongs the training period but also increases the risk of overfitting.

[0004] Current multimodal fusion technologies have significant limitations. In the field of cross-modal interaction, most existing cross-modal guidance mechanisms are still in the stage of unidirectional spatial interaction, supporting only unidirectional information transfer from one modality to another, making it difficult to achieve bidirectional dynamic spatiotemporal-frequency domain deep complementarity between multiple modalities. Taking human action recognition as an example, the spatial features of video frames and the temporal series features of inertial sensors have failed to form an efficient fusion system, resulting in information loss of key action details during feature extraction. At the frequency domain information mining level, mainstream methods generally rely excessively on spatial and temporal feature analysis, often employing traditional temporal processing techniques such as inter-frame difference and optical flow calculation, while neglecting the development of complementary information such as periodic patterns and frequency distributions inherent in the frequency domain dimension, making it difficult to improve the recognition accuracy of complex action sequences and minute action units. Furthermore, for RGB modal data processing, existing solutions lack targeted optimization strategies in the data preprocessing stage. When the proportion of human target pixels in an RGB image is less than 30%, redundant information such as background textures and environmental objects will seriously interfere with motion feature extraction, typically manifesting as misclassification of long-distance human behavior in surveillance video scenarios.

[0005] In behavior recognition tasks, two types of intra-class discrepancies present significant challenges. The first is behaviors like "drinking water" and "raising a glass," which have similar appearances but significantly different motion dynamics; the second is behaviors like "reading" and "writing," which exhibit subtle differences in motion patterns. Traditional unimodal methods and existing multimodal fusion strategies face bottlenecks in accuracy when handling such complex scenarios. How to deeply explore the complementary mechanisms between multimodal data and construct behavior feature representation models with strong discriminative capabilities has become a core technical challenge that urgently needs to be overcome in the field of multimodal behavior recognition. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal behavior recognition method based on progressive guided fusion, so as to solve the inherent defects of single-modality behavior recognition and the shortcomings of existing multimodal fusion methods in cross-modal guidance, frequency domain information utilization, and RGB data layer optimization, so as to achieve more accurate and robust multimodal human behavior recognition and provide high-precision technical support for downstream tasks such as human-computer interaction, intelligent monitoring, and abnormal behavior detection.

[0007] The inventive concept of this invention is to propose a progressively guided fusion network, PGFNet, which fully leverages the complementarity between RGB and skeletal data through a hierarchical strategy of dynamic RGB cropping at the data layer, extraction via a dual-modal dedicated network at the feature layer, and progressively guided frequency domain fusion at the fusion layer. Specifically, this invention first performs dynamic skeletal-guided cropping of RGB data at the data layer to reduce background redundancy and highlight human behavior areas. Secondly, it designs dedicated, high-efficiency feature extraction networks for RGB and skeletal modalities respectively. RGB feature extraction uses the WTSP enhanced network, combining full-frame temporal utilization, multi-scale frequency domain decomposition, and keyframe attention mechanisms to accurately capture RGB motion and scene details. Skeletal feature extraction uses MSWTNet, which efficiently extracts dynamic skeletal motion features while maintaining lightweight design through modular stacking and feature pyramid design. Then, through a progressively guided fusion module, the dynamic semantic expression of skeletal features is first enhanced by skeletal self-guidance, followed by RGB frequency domain fusion guided by the skeleton, integrating the amplitude and phase complementarity of the two modalities in the complex frequency domain to achieve deep cross-modal complementarity. Finally, a spatiotemporal classification module completes behavior recognition, and multiple loss functions are used to optimize network parameters, improving the model's ability to distinguish fine-grained behaviors.

[0008] To achieve the above-mentioned objective, according to one aspect of the present invention, a multimodal behavior recognition method based on progressive guided fusion is provided, comprising the following steps:

[0009] S1. Preprocess the input RGB data and skeletal data, including dynamic cropping preprocessing of the RGB data guided by the skeleton to obtain optimized RGB data.

[0010] S2. Construct a progressively guided fusion network framework PGFNet for multimodal behavior recognition methods. PGFNet mainly consists of three modules: RGB feature extraction module, skeleton feature extraction module, and progressively guided fusion module.

[0011] The RGB feature extraction module uses the WTSP enhanced network to extract multi-scale frequency domain features and keyframe attention of RGB through full-frame utilization, ECA channel attention, and wavelet convolution. The skeleton feature extraction module uses MSWTNet with a modular stacked structure and feature pyramid design to extract the dynamic motion features of the skeleton. During the operation of the RGB feature extraction module and the skeleton feature extraction module, the extracted features are transferred to the progressive guided fusion module.

[0012] S3. The preprocessed RGB data and skeletal data are input into the RGB feature extraction module and the skeletal feature extraction module, respectively, for feature extraction. In the progressively guided fusion module, the extracted features are first enhanced by skeletal self-guidance, and then the dual-modal features are integrated by RGB frequency domain fusion guided by the skeletal system. Finally, the behavior recognition is completed by using the spatiotemporal classification module in combination with the cross-entropy loss of the classification task. The cross-entropy loss of the classification task and the correlation loss based on feature fusion together constitute the basis for evaluating the accuracy of behavior recognition.

[0013] Further, step S1 includes the following steps:

[0014] S11. Synchronously acquire skeletal joint coordinates. For each frame of RGB image, synchronously acquire the corresponding set of skeletal joint coordinates J = {j1,j2,…,j...} N},in Represents the three-dimensional spatial coordinates of the i-th joint;

[0015] S12. Generation of the minimum bounding box for the actor: Based on the joint coordinate set J, calculate the minimum axis-aligned bounding box B containing all joints, whose top-left corner coordinate (x... min ,y min ) and the coordinates of the lower right corner (x max ,y max )satisfy:

[0016]

[0017] Where, j i,x j i,y Joint j i The x and y coordinates on an RGB image;

[0018] S13. Anisotropic Expansion and Clipping of Bounding Box. Anisotropic expansion of the bounding box B is performed to avoid clipping key parts of the human body—expanding by 30% along the short side and 20% along the long side, resulting in the expanded bounding box B′. The coordinate range of B′ is strictly limited to the size of the RGB frame [0, W) × [0, H). Here, W is the original image width, and H is the original image height. The width and height of the original bounding box are respectively:

[0019] w = x max -x min (5)

[0020] h = y max -y min (6)

[0021] If w ≤ h, the shorter side is the width, so the expanded width w′ = w × 1.3 and the height h′ = h × 1.2; if w > h, the shorter side is the height, so the expanded width w′ = w × 1.2 and the height h′ = h × 1.3; the final coordinates of the top-left corner of the cropped area are... The coordinates of the bottom right corner are This operation preserves complete information about human behavior while compressing irrelevant background, thus reducing the burden on subsequent feature extraction.

[0022] S14. Standardization Preprocessing of Skeletal Data: Standardize the coordinates of skeletal joints. Using the central joint of the human spine as a reference point, calculate the normalized coordinates of other joints relative to this reference point. The formula is as follows:

[0023]

[0024] Where, j c The coordinates of the central joint of the spine are given. The joint coordinates are normalized to eliminate the influence of factors such as camera viewpoint and distance between the human body and the camera on the skeletal data, thereby enhancing the universality of skeletal features.

[0025] Furthermore, step S2 includes the following steps:

[0026] S21. Construct an RGB feature extraction submodule based on EfficientNet-B0, and use the WTSP enhanced network to extract global information of RGB frames through wavelet convolution and ECA channel attention;

[0027] S22, Skeletal Feature Extraction Submodule, adopts MSWTNet, with a modular stacked structure and feature pyramid design, to extract the dynamic motion features of the skeleton;

[0028] Furthermore, in step S21, constructing the framework RGB feature extraction submodule includes the following steps:

[0029] S211. Preprocess the input RGB video frame sequence without discarding any frames;

[0030] S212. Apply ECA channel attention, use 1D convolution to process all frames, capture the correlation between adjacent frames and distant frames, dynamically adjust the weights according to the importance of the frames, and generate channel weights to highlight core action moments.

[0031] Furthermore, in step S212, applying ECA channel attention to generate channel weights to highlight core behavioral moments includes the following steps:

[0032] S2121. Input RGB video frame feature map Where C represents the number of channels, T represents the number of frames in the time dimension, and H and W represent the height and width of a single frame image. Spatial global average pooling is performed to compress the spatial dimension H×W, resulting in the time-channel feature representation. Preserve the correlation information between the channel and the time dimension;

[0033] S2122. The 1D convolutional kernel size k is adaptively determined based on the number of channels C, and the calculation formula is as follows:

[0034]

[0035] Subsequently, a 1D convolution operation with kernel size k is performed on X′ along the channel dimension to capture the dependencies between adjacent and non-adjacent channels, thus obtaining the channel attention intermediate features.

[0036] S2123, Regarding X att Apply the Sigmoid activation function to generate channel attention weights. Each element ω c This indicates the importance of the c-th channel at the core action moment;

[0037] S2124. Perform a channel-wise weighted summation of the channel attention weight ω and the time-channel feature representation X′ to obtain the weighted features. Enhance channel features at core behavioral moments.

[0038] S213. By using wavelet convolution, the video signal is decomposed into three frequency domain components: high, medium, and low, to achieve multi-resolution feature extraction and enhance the expression of motion details by RGB.

[0039] Furthermore, in step S213, the video signal is decomposed into high, medium, and low frequency components to achieve multi-resolution feature extraction, including the following steps:

[0040] S2131. The feature map x∈R of the input RGB video frame sequence. C×T×H′×W′ Where C is the number of channels, T is the number of frames, H′ is the feature map height, and W′ is the feature map width, a wavelet transform kernel is applied for convolution operation;

[0041] S2132. By using wavelet transform kernels of different scales, the frequency components of the video signal are divided into high-frequency components x. high Mid-frequency component x mid and low frequency components x low High-frequency components correspond to fast motion, medium-frequency components correspond to medium-speed behavior trends, and low-frequency components correspond to slow background changes.

[0042] S2133. Perform feature extraction and fusion on the three frequency domain components obtained by division to obtain multi-resolution RGB feature representation.

[0043] S214. Input the processed RGB video frame features into EfficientNet-B0.

[0044] In step S22, constructing the skeleton feature extraction and fusion module includes the following steps:

[0045] S221. Split the input features into two parallel branches for processing;

[0046] S222. One branch uses point convolution to quickly transform and initially fuse the channel dimension features.

[0047] Furthermore, in step S222, the channel-dimensional features are rapidly transformed and initially fused using point convolution, including the following steps:

[0048] S2221. Input the skeletal feature map. Perform dimensional flattening, where C in Given the number of input channels, T is the number of frames in the time dimension, and J is the total number of joints, the flattened features are obtained. This merges the time dimension and the joint dimension into a single dimension, making convolution operations in the channel dimension easier.

[0049] S2222, Use dotted convolution on the flattened features Perform channel dimension transformation; the parameters of the convolution kernel are: Where C mid The preset number of intermediate channels is determined through matrix operations. Characteristics obtained after preliminary fusion Complete the feature transformation of the channel dimension and the initial fusion of cross-time-joint information;

[0050] S2223, The initially fused feature X conv Remodeling In this form, the structured representation of the time and joint dimensions is restored, providing dimension-matched input for subsequent multi-scale feature extraction of wavelet convolution;

[0051] S223, another branch uses wavelet convolution with a 3×3 kernel size twice in succession.

[0052] Furthermore, in step S223, wavelet convolution with a kernel size of 3×3 is applied twice consecutively, including the following steps:

[0053] S2231. The skeletal features after initial fusion via point convolution are... Where C midThe input consists of the number of intermediate channels, T (the number of frames in the time dimension), and J (the total number of joints), fed into two consecutive sets of 3×3 wavelet convolutional units. These wavelet convolutional units, based on a preset wavelet basis function, perform multi-scale decomposition of the input features in both spatial and temporal dimensions to extract multi-scale features of the skeleton.

[0054] S224. Add the processing results of the two branches element by element to produce the merged output.

[0055] Furthermore, step S3 includes the following steps:

[0056] S31. Transfer the features output by the RGB feature extraction submodule and the skeleton feature extraction submodule to the progressive guided fusion submodule to perform skeleton self-guided enhancement of skeleton features.

[0057] S32. Perform skeleton-guided RGB frequency domain fusion and integration of dual-modal features;

[0058] S33. Construct the spatiotemporal classification module of the PGFNet framework. This module efficiently transforms the fused features into behavior classification results through spatial global pooling, 1D temporal convolution, fully connected layers and Softmax, taking into account both spatiotemporal feature modeling and lightweight design.

[0059] Furthermore, in step S31, the self-guided enhancement of skeletal features includes the following steps:

[0060] S311, unify the number of channels for RGB features and skeleton features through a fully connected layer.

[0061] Furthermore, in step S311, the number of channels is unified through the fully connected layer, including the following steps:

[0062] S3111, Input joint features Where C j Let T be the number of skeletal feature channels, T be the number of time frames, and J be the number of joints. Perform global average pooling to compress the T and J dimensions to obtain the global channel feature representation of the joint features. To aggregate information about each joint in the channel dimension;

[0063] S312. Use the ECA module to convert joint features into channel attention for skeleton features, thereby enabling joint features to guide skeleton features, i.e., self-guidance of skeleton features.

[0064] Furthermore, in step S312, the channel attention of the skeleton features is transformed using the ECA module, including the following steps:

[0065] S3121, The joint-level feature F′ jointThe input ECA channel attention module first focuses on the number of skeletal feature channels C. b The 1D convolution kernel size is adaptively determined using the following formula:

[0066]

[0067] The 1D convolution captures the dependencies between channels; then, a sigmoid activation function is applied to the convolution output to generate channel attention weights. Then, the attention weight α is compared with the original skeletal features F. bone Perform channel-by-channel weighting, and then restore the enhanced skeletal features to a dimension of [missing value]. Highlight key skeletal movement features;

[0068] S3122, respectively, for RGB features Where C r H represents the number of RGB feature channels, and H and W represent the height, width, and enhanced skeletal features of the feature map. Dimension mapping via a linear projection layer: The parameters of the linear projection layer are set as follows... Map RGB features and skeletal features to a unified projection dimension C. proj , obtain the projected RGB features and skeletal features This addresses the issue of cross-modal feature dimension mismatch, providing feature inputs of the same dimension for subsequent frequency domain fusion.

[0069] Furthermore, in step S32, the skeleton-guided RGB frequency domain fusion and integration of dual-modal features includes the following steps:

[0070] S321. Use Fast Fourier Transform to transform RGB features and skeleton self-guided features from the spatial domain to the frequency domain, and train the real part filter and imaginary part filter of the Fourier complex features.

[0071] Furthermore, in step S321, the RGB features and skeleton self-guided features are transformed from the spatial domain to the frequency domain using a fast Fourier transform, including the following steps:

[0072] S3211, Projected RGB features Where C proj Let T be the number of projection channels, H be the number of time frames, and W be the height and width of the RGB feature map. Perform global average pooling along the spatial dimension H×W to obtain the compressed RGB time series features. Aggregate spatial dimension information to focus on temporal dynamics;

[0073] S3212, to The time series of each channel in the dataset is C. projEach channel is a one-dimensional sequence of length T. A Fast Fourier Transform is applied channel by channel to convert the time-domain signal into a complex frequency-domain signal, yielding the RGB frequency domain characteristics. Represents the complex number field, containing information about the real and imaginary parts;

[0074] S3213, Self-guiding features of projected skeleton Where J is the total number of joints, global average pooling is performed along the joint dimension J to obtain the compressed bone time series features. Aggregate joint dimension information to focus on temporal motion dynamics;

[0075] S3214, to For each channel's time series, a Fast Fourier Transform is applied channel by channel to convert the time-domain signal into a complex frequency-domain signal, thus obtaining the bone frequency-domain features. Complete the conversion from the spatial domain to the frequency domain;

[0076] S322, the real part filter, and the imaginary part filter are applied to the real and imaginary parts of the Fourier feature, respectively, to achieve filtering weighting of amplitude and phase.

[0077] Furthermore, in step S322, constructing a real part filter and a real part filter to achieve amplitude and phase filtering weighting includes the following steps:

[0078] S3221. Initialize the trainable frequency domain filter parameter matrix. Construct real-part and imaginary-part filters for RGB frequency domain features and skeleton frequency domain features respectively. The real-part filter parameter matrix for the RGB mode is as follows: The parameter matrix of the imaginary part filter is The real part filter parameter matrix of the skeletal mode is The parameter matrix of the imaginary part filter is All filter parameters are initialized using a Xavier normal distribution to ensure that the initial weight distribution adapts to the frequency domain feature dimension. proj Where T is the number of projection channels and T is the number of time frames;

[0079] S3222, Extract RGB frequency domain features The real part Re(F) rgb-freq ) and the imaginary part Im(F rgb-freq ), respectively with the RGB real part filter W rgb-r , Imaginary part filter W rgb-i Perform element-wise multiplication to obtain the filtered RGB real part features and the filtered RGB imaginary part features, where j is the imaginary unit. The calculation formula is as follows:

[0080] Re(F rgb-freq ′)=Re(F rgb-freq)⊙W rgb-r (10)

[0081] Im(F rgb-freq ′)=Im(F rgb-freq )⊙W rgb-i (11)

[0082] The filtered RGB frequency domain features are then reconstructed, as shown in the following formula:

[0083] F rgb-freq ′=Re(F rgb-freq ′)+j·Im(F rgb-freq ′) (12)

[0084] S3223. Employ weighted logic consistent with RGB modality to extract bone frequency domain features. The real part Re(F) bone-freq ) and the imaginary part Im(F bone-freq ), respectively with the real part filter W of the skeleton bone-r , Imaginary part filter W bone-i Performing element-wise multiplication yields the filtered real bone features, as shown in the following formula:

[0085] Re(F bone-freq ′)=Re(F bone-freq )⊙W bone-r (13)

[0086] The formula for the filtered imaginary part features of the skeleton is as follows:

[0087] Im(F bone-freq ′)=Im(F bone-freq )⊙W bone-i (14)

[0088] The reconstructed filtered bone frequency domain features are calculated as follows:

[0089] F bone-freq ′=Re(F bone-freq ′)+j·Im(F bone-freq (15)

[0090] S3224, Filter the RGB frequency domain features F rgb-freq ′ and bone frequency domain features F bone-freq The input is then used in the subsequent complex multiplication fusion stage. Combined with the overall model's cross-entropy loss function, all filter parameters W are dynamically updated via backpropagation. rgb-r W rgb-i W bone-r This enables the filter to adaptively learn the contribution weights of different frequency domain components to behavior recognition, thereby optimizing the cross-modal frequency domain fusion effect.

[0091] S323. Multiply the filtered RGB frequency domain features and the skeleton frequency domain features element-wise to achieve frequency domain fusion. Use inverse fast Fourier transform to convert the fused frequency domain features to the spatial domain.

[0092] Furthermore, in step S323, the filtered RGB frequency domain features and the skeletal frequency domain features are multiplied element-wise to achieve frequency domain fusion, including the following steps:

[0093] S3231. Determine the filtered RGB frequency domain characteristics to be fused. With bone frequency domain features Where C proj To standardize the number of projection channels, T represents the number of time frames. Representing the complex field, the number of channels and the time dimension are perfectly matched, satisfying the dimension requirement of element-wise multiplication;

[0094] S3232. Perform element-wise multiplication of the complex field, based on the rules of complex multiplication, on F. rgb-freq ′ and F bone-freq Perform position-by-position, i.e., complex multiplication for each channel and each time step. Let F... rgb-freq The complex value at a certain channel-time position is a+jb. Where a is the real part, b is the imaginary part, j is the imaginary unit, and F... bone-freq The complex value at the corresponding position is c + jd. Here, c is the real part and d is the imaginary part. The product is (ac - bd) + j(ad + bc). This operation achieves deep interactive fusion of RGB and skeletal frequency domain features at the amplitude and phase levels, yielding preliminary fused complex frequency domain features.

[0095] S3233, Extraction of real part of fused feature and dimensionality calibration, for preliminary fused complex frequency domain feature F fusion-freq-raw By applying the fast inverse Fourier transform, the frequency domain fused features are mapped back to the time domain to obtain the time domain complex features. Since there may be a small imaginary part remaining after the IFFT transformation, the real part of this temporal complex feature is taken as the effective temporal fusion feature. Ensure that the features are real numbers to match the input requirements of the subsequent classification module;

[0096] S3234. Using residual connections to enhance semantic preservation: To avoid losing the basic semantic information of RGB modalities during fusion, the original RGB features are mapped to C through a linear projection layer. proj ×T dimensions yield the RGB basic semantic features for dimension matching. F fusion-time-real With F rgb-residual Perform element-wise addition to obtain the final temporal fusion feature. This feature contains complementary information from cross-modal frequency domain fusion while retaining the original RGB semantics, providing rich discriminative basis for subsequent behavior classification.

[0097] In step S33, the feature extraction and fusion module for constructing the PGFNet framework includes the following steps:

[0098] S331. Perform global average pooling on the fused features to compress the spatial dimension, thereby aggregating spatial semantic information and reducing the amount of subsequent computation.

[0099] S332. Process the pooled features through 1D convolutional layers to capture the temporal dependencies between frames;

[0100] S333. The features processed by 1D temporal convolution are input into the fully connected layer. The output dimension of the fully connected layer is the total number of behavior categories. Then, the probability distribution of each behavior is output through the Softmax activation function to complete the decision-making process of behavior recognition and obtain the predicted probability corresponding to each behavior category.

[0101] According to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the multimodal behavior recognition method based on progressive guided fusion of the present invention.

[0102] According to another aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal behavior recognition method based on progressive guided fusion of the present invention.

[0103] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0104] 1. Multimodal Deep Complementarity: The proposed Progressive Guided Fusion Network (PGFNet) deeply explores the complementarity between RGB and skeletal data through a strategy of skeleton self-guidance and frequency domain fusion. The scene and appearance information of RGB data is accurately integrated with the motion and structural information of the skeleton in the frequency domain dimension, which solves the problem of single modality being susceptible to interference or insufficient information. In particular, it improves the recognition ability of fine-grained behaviors such as "similar appearance but different movements", such as "drinking water" and "raising a cup", and "subtle differences in movement patterns", such as "reading" and "writing".

[0105] 2. Targeted optimization at each layer: The skeleton-guided RGB cropping in the data layer reduces background redundancy, making the model more focused on the main behavior; the WTSP enhancement network and MSWTNet in the feature layer are customized with efficient feature extraction schemes for RGB and skeleton respectively, taking into account both feature representation ability and computational efficiency; the frequency domain progressive guidance in the fusion layer breaks through the limitations of traditional "unidirectional, time-domain-oriented" fusion, and mines more fine-grained complementary information in the frequency domain dimension.

[0106] 3. Improved robustness and generalization: Experiments on mainstream datasets such as NTU-RGB+D60 / 120 show that PGFNet significantly outperforms existing bimodal methods in challenging scenarios such as cross-subject (C-Sub) and cross-view (C-View), demonstrating stronger robustness and generalization, and better supporting downstream tasks such as human-computer interaction and intelligent monitoring. Attached Figure Description

[0107] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0108] Figure 1 This is a diagram showing the overall framework of PGFNet in this invention;

[0109] Figure 2 This is a framework diagram of the Multi-Scale Wavelet Skeleton Extraction Network (MSWTNet) in this invention;

[0110] Figure 3 This is an overall framework diagram of the Progressive Guided Fusion Network (PGFNet) in this invention;

[0111] Figure 4 This is a schematic diagram of the frequency domain modulation process;

[0112] Figure 5 A model confusion matrix and standard comparison diagram based on the NTURGB+D60C-sub protocol;

[0113] Figure 6 Visualization of activation maps generated by Grad-CAM, and comparison of activation maps between the original dataset and the standard dataset. Detailed Implementation

[0114] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0115] Example 1

[0116] like Figures 1-6As shown, the technical solution provided in this embodiment is a multimodal fine-grained action recognition method based on a Progressive Guidance Fusion Network (PGFNet). This invention improves the accuracy of fine-grained action recognition in complex scenes by reducing background redundancy through skeleton-guided RGB dynamic cropping and mining cross-modal complementary information through multimodal frequency domain progressive fusion. The proposed framework is a Progressive Guidance Fusion Network (PGFNet), as follows... Figure 1 As shown in the figure, the entire architecture employs multimodal data joint modeling and mainly consists of five core modules: skeleton-guided RGB cropping module, WTSP-enhanced RGB feature extraction module, MSWTNet lightweight skeleton feature extraction module, progressive guided fusion module, and spatiotemporal structured classification module.

[0117] The model comprises several modules: a skeleton-guided RGB cropping module that generates dynamic regions of interest based on the spatial distribution of skeleton joints to highlight the main human action; a WTSP-enhanced RGB feature extraction module that combines wavelet convolution and channel attention to capture multi-scale spatiotemporal features of RGB videos; a MSWTNet lightweight skeleton feature extraction module that uses wavelet convolution blocks as its core to achieve lightweight and efficient extraction of skeleton features; a progressive guided fusion module that achieves bidirectional guided fusion of RGB and skeleton features through frequency domain transformation and adaptive filtering; and a spatiotemporal structured classification module that outputs action categories through global pooling and temporal convolution. Through the synergistic effect of these modules, the model ultimately outputs action classification results, and the classification loss and fusion loss are jointly optimized to ensure a balance between recognition accuracy and efficiency.

[0118] Skeleton-Guided RGB Cropping Module: Although RGB videos contain rich appearance information, background redundancy and perspective changes can easily interfere with motion feature learning. This invention designs a skeleton-guided RGB cropping module that generates dynamic bounding boxes based on the spatial coordinates of skeleton joints, retaining only the main motion area and significantly reducing background noise interference.

[0119] Specifically, for the input skeleton joint sequence Where N is the number of joints and T is the number of frames. First, calculate the minimum bounding box for all joints:

[0120] x min =min(J[:,0]),x max =max(J[:,0]) (1)

[0121] y min =min(J[:,1]),y max =max(J[:,1]) (2)

[0122] Then, the bounding box is scaled up proportionally, with the scaling factor α ∈ [1.1, 1.5], preferably 1.3, to obtain the coordinates (x, y) of the final clipped region. crop ,y crop ,w crop ,h crop The original RGB video frames were cropped and normalized to obtain an RGB sub-image sequence I focusing on human motion. crop .

[0123] WTSP Enhanced RGB Feature Extraction Module: To fully utilize the multi-scale spatiotemporal information of RGB videos, this invention designs a WTSP enhancement network, combining wavelet convolution and channel attention (ECA) to achieve multi-scale feature capture and key information enhancement. The module input is a cropped sequence of RGB sub-images. First, basic spatiotemporal features are extracted using 3D convolution. Then, wavelet convolutional blocks are introduced to perform multi-scale decomposition and reconstruction of the features, as shown in the following formula:

[0124]

[0125] Where Conv wavelet For wavelet convolution operations, Conv res For residual convolution, This indicates that the feature maps are added element-wise. Then, the key channel features are enhanced using the ECA channel attention module, with the attention weights calculated as follows:

[0126] Attn=σ(Conv 1×1 (GlobalAvgPool(F wavelet (4)

[0127] F rgb =F wavelet ⊙Attn (5)

[0128] Where σ is the Sigmoid activation function, and ⊙ represents element-wise multiplication. This refers to the final output RGB characteristics.

[0129] MSWTNet Lightweight Skeleton Feature Extraction Module: Addressing the high-dimensional redundancy and dynamic topology characteristics of skeleton data, this invention...

[0130] The multi-scale wavelet skeleton network MSWTNet is designed, using wavelet convolutional blocks as its core to achieve a balance between lightweight design and multi-scale feature extraction. The module input is a sequence of skeleton joints. First, joint features are mapped to a high-dimensional space using linear projection. Then, multi-scale wavelet convolutional blocks are stacked, with each block containing two parallel branches:

[0131]

[0132] Where Conv wavelet-1 and Conv wavelet-2 Wavelet convolutions of different scales are used, where [·; ·] denote feature concatenation, and ⊕ denotes residual connection. After four wavelet convolution blocks, lightweight skeleton features are output.

[0133] Progressive Guided Fusion Module: To achieve deep complementarity between RGB and skeleton features, this invention designs a progressive guided fusion mechanism. Through frequency domain transformation and adaptive filtering, it completes bidirectional feature guidance and fusion in the frequency domain. First, for RGB features F... rgb and skeleton features F skel Perform Fast Fourier Transform (FFT) on each component to convert it to the frequency domain:

[0134]

[0135] Then, bidirectional guidance is achieved through an adaptive filtering kernel K:

[0136]

[0137]

[0138] Finally, the fused features are transformed back to the time domain using Inverse Fast Fourier Transform (IFFT) and combined with residual connections:

[0139]

[0140] in, This is a cross-modal fusion feature.

[0141] Spatiotemporal Structured Classification Module: To efficiently utilize the spatiotemporal information of the fused features, this invention designs a spatiotemporal structured classification module, which achieves action classification through spatial global pooling and temporal convolution. First, the fused feature F... fusion Perform global spatial pooling to compress spatial dimensions:

[0142]

[0143] Then, inter-frame dependencies are captured using 1D temporal convolution:

[0144]

[0145] Finally, the action class probability is output through a fully connected layer and a Softmax activation function:

[0146]

[0147] Where K is the number of action categories, and the final classification result is the average probability of all frames.

[0148] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0149] Furthermore, the effectiveness of the method proposed in this invention is verified through simulation experiments.

[0150] Table 1. Contribution analysis of core components to ablation experiment results

[0151]

[0152] Table 1 shows four sets of comparative experiments we designed on the NTURGB+D60 dataset: (1) baseline model; (2) baseline + WTSP; (3) MSWTNet (skeleton modality only, testing skeletal branches and joint branches respectively); (4) PGFNet (RGB + skeleton fusion, i.e., complete model) to verify the specific contributions of the WTSP sampling method, MSWTNet skeleton extraction network and PGFNet progressive guided fusion network.

[0153] In PGFNet, the default number of channels for skeleton-guided RGB fusion is set to 1024. To verify the sensitivity of this parameter, we tested the model performance with 512, 1024, and 2048 channels, and recorded the number of model parameters and accuracy, as shown in Table 2.

[0154] Table 2. Results of sensitivity test for fusion channel number

[0155]

[0156] Table 3. Accuracy Comparison of NTURGB+D Datasets

[0157]

[0158] To verify the effectiveness of the proposed method, we compared it with current mainstream unimodal and multimodal methods on the NTU-RGB+D60 and NTU-RGB+D120 datasets, covering methods with different architectures such as GCN, CNN and Transformer, thereby verifying the improvement effect of PGFNet.

[0159] The comparative results show that PGFNet has a significant performance advantage: at the single-modal level, its RGB branch outperforms traditional RGB methods; although the skeleton branching algorithm is slightly inferior to the latest GCN method, it can still provide effective dynamic feature support for multimodal fusion; at the multimodal level, its accuracy reaches 95.8% and 98.6% under the C-sub and C-view protocols, respectively, achieving the best performance. PGFNet's performance improvement stems from two core designs: at the data layer, dynamic skeleton image cropping effectively reduces background interference; at the feature layer, frequency domain fusion technology mines complementary information between RGB and the skeleton. Especially in scenes with similar appearances but different behaviors, the skeleton motion trend and the RGB scene context create a synergistic effect, significantly improving the accuracy of fine-grained behavior recognition. Furthermore, PGFNet's generalization ability was fully validated: when the number of behavior categories increased to 120, the model maintained its leading position, achieving an accuracy of 91.4% under the C-sub protocol, outperforming PDA-GCN (90.7%) and MMCL (90.3%), fully demonstrating its excellent adaptability to behavioral diversity; while under the C-set protocol, its accuracy reached 93.8%, significantly better than all comparison methods, reflecting the model's strong adaptability to differences in data collection batches. This generalization advantage stems from the multimodal fusion mechanism—RGB modality can compensate for the lack of scene information in skeletal modality, while skeletal modality can alleviate the lighting and background interference problems faced by RGB modality, thereby effectively reducing the impact of individual data distribution differences on model performance.

[0160] Next, the present invention designs ablation experiments to study the effectiveness of the model. The present invention first verifies the impact of multimodal fusion on the final result, then verifies the effectiveness of each component in the model, followed by feature visualization, attention weight visualization, and model comparison charts to further illustrate the model's advantages in multimodal fine-grained action recognition. Finally, the function of the model is intuitively demonstrated through visualization of the model's working process.

[0161] 1) Impact of Multimodal Fusion on Model Results: To explore the importance of multimodal fusion, this invention designed comparative experiments to test the performance of RGB-only, skeleton-only, and RGB+skeleton multimodal fusion on the NTU-RGB+D60 and NTU-RGB+D120 datasets, respectively. The results show that multimodal fusion significantly improves model performance compared to single-modal fusion. Specifically, on the NTU-60 C-sub protocol, the accuracy of RGB-only was 89.2%, skeleton-only was 90.5%, and after multimodal fusion, it reached 95.8%, an improvement of 5.3%–6.6%; on the NTU-120 C-set protocol, the accuracy of RGB-only was 85.1%, skeleton-only was 87.3%, and after multimodal fusion, it reached 93.8%, an improvement of 6.5%–8.7%.

[0162] 2) Validity of each component of the model: To evaluate the contributions of the skeleton-guided RGB cropping module, the WTSP-enhanced RGB feature extraction module, the MSWTNet lightweight skeleton feature extraction module, and the progressive guided fusion module to the model, the model using only RGB + skeleton simple splicing was set as the baseline model. Ablation experiments were designed and validated on the NTU-60 and NTU-120 datasets.

[0163] a) Effectiveness of the skeleton-guided RGB cropping module: Compared with the baseline model, after introducing skeleton-guided RGB cropping, the model improved accuracy by 3.1% on the C-sub protocol and 2.8% on the C-view protocol on the NTU-60; and by 2.7% on the C-sub protocol and 2.5% on the C-set protocol on the NTU-120. This indicates that the module effectively improves the quality of RGB features by suppressing background redundancy, making the model more robust to action recognition in complex backgrounds.

[0164] b) WTSP enhances the effectiveness of the RGB feature extraction module: Compared with the baseline model, after introducing the WTSP module, the model's accuracy improved by 2.4% on the NTU-60 C-sub protocol and 2.1% on the C-view protocol; and by 2.0% on the NTU-120 C-sub protocol and 1.8% on the C-set protocol. By combining wavelet convolution with channel attention, this module enhances the multi-scale spatiotemporal representation capability of RGB features, especially the ability to capture action details.

[0165] c) Effectiveness of the MSWTNet lightweight skeleton feature extraction module: Compared with the baseline model, MSWTNet improves the accuracy of the model by 2.7% on the C-sub protocol and 2.4% on the C-view protocol on NTU-60; and by 2.3% on the C-sub protocol and 2.1% on the C-set protocol on NTU-120, while reducing the model's computational cost by 20%. This demonstrates the effectiveness of this module in multi-scale skeleton feature extraction under the premise of lightweight design.

[0166] d) Effectiveness of the progressive guided fusion module: Compared with the baseline model, after introducing progressive frequency domain fusion, the model improved accuracy by 3.5% on the C-sub protocol of NTU-60 and 3.2% on the C-view protocol; and by 3.0% on the C-sub protocol and 2.8% on the C-set protocol of NTU-120. Frequency domain fusion significantly enhances the discriminative power of features by mining cross-modal complementary information, especially the ability to distinguish fine-grained actions.

[0167] 3) Effectiveness of the final model (PGFNet): Experimental results show that on the six datasets mentioned above, the PGFNet model of this invention outperforms the baseline model and existing mainstream methods. Taking the NTU-60 dataset as an example, compared with the current best method VT-BPAN, PGFNet improves performance by 0.4% on the C-sub protocol and 0.9% on the C-view protocol; compared with STAR-Transformer, it improves performance by 1.4% on the C-sub protocol and 2.3% on the C-set protocol of NTU-120.

[0168] To further analyze model performance, this invention compares PGFNet with a baseline model in recognizing easily confused fine-grained actions, such as "drinking water" and "drinking tea," "typing" and "keyboard tapping," and "tying shoelaces" and "untying shoelaces." The results show that for "drinking water" and "drinking tea," the baseline model's accuracy is 82%, while PGFNet improves to 90%; for "typing" and "keyboard tapping," the accuracy increases from 80% to 88%; and for "tying shoelaces" and "untying shoelaces," it increases from 75% to 85%. The probability of misclassification decreases to varying degrees; for example, the probability of misclassifying "drinking water" as "drinking tea" decreases by 8%, fully demonstrating that PGFNet can extract subtle differences in fine-grained actions from multimodal spatiotemporal information.

[0169] In terms of feature visualization, Figure 6 The t-SNE visualization results of five fine-grained actions in PGFNet are presented. These actions are extremely similar in spatial structure and temporal dynamics, but in the visualization, the features extracted by PGFNet can clearly cluster them, while the features of the baseline model show a large amount of overlap. This indicates that PGFNet's feature representation of fine-grained actions is more discriminative.

[0170] In addition, to visually demonstrate the model's working process, Figure Y shows the changes in the region of skeleton-guided RGB cropping, the channel attention weights of the WTSP module, and the frequency domain feature distribution before and after progressive fusion. It can be observed that: (1) skeleton-guided cropping can accurately select the human action region and effectively filter background interference; (2) the attention weights of the WTSP module are significantly enhanced on the action keyframes; (3) the frequency domain components of the features after frequency domain fusion are more complementary, and the action details are more prominent after temporal reconstruction. These visualization results further verify the functional effectiveness of each module and the interpretability of the model.

[0171] In summary, this invention achieves significant advantages in accuracy, computational efficiency, and robustness of fine-grained action recognition through multimodal fusion and the synergistic effect of various innovative modules, fully demonstrating the creativity and practicality of the technical solution.

[0172] Example 2:

[0173] The computer-readable storage medium of this embodiment stores a computer program that, when executed by a processor, implements the steps of the multimodal behavior recognition method based on progressive guided fusion in Embodiment 1.

[0174] The computer-readable storage medium in this embodiment can be an internal storage unit of the terminal, such as the terminal's hard disk or memory; the computer-readable storage medium in this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. equipped on the terminal; furthermore, the computer-readable storage medium can include both the terminal's internal storage unit and external storage devices.

[0175] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0176] Example 3:

[0177] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the multimodal behavior recognition method based on progressive guided fusion of Embodiment 1.

[0178] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The memory can include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0179] Those skilled in the art will understand that the content disclosed in the embodiments can be provided as a method, system, or computer program product. Therefore, this solution can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this solution can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.

[0180] This solution is described with reference to flowchart illustrations and / or block diagrams of methods and computer program products according to embodiments of this solution. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0181] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0182] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0183] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0184] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the specific embodiments described above. The specific embodiments and descriptions in the specification are merely for further illustrating the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the claims and their equivalents.

Claims

1. A multimodal behavior recognition method based on progressive guided fusion, characterized in that, Includes the following steps: S1. Preprocess the input RGB data and skeletal data, including dynamic cropping preprocessing of the RGB data guided by the skeleton to obtain optimized RGB data. S2. Construct a progressively guided fusion network framework PGFNet for multimodal behavior recognition methods. PGFNet mainly consists of three modules: RGB feature extraction module and skeleton feature extraction module. The RGB feature extraction module uses the WTSP enhanced network, which extracts multi-scale frequency domain features and keyframe attention of RGB through full-frame utilization, ECA channel attention and wavelet convolution; the skeleton feature extraction module uses MSWTNet, which extracts the dynamic motion features of skeletons with a modular stacked structure and feature pyramid design. S3. Input the preprocessed RGB data and skeleton data into the RGB feature extraction module and skeleton feature extraction module respectively for feature extraction. In the progressive guided fusion module, the extracted features are first enhanced by skeleton self-guidance, and then the dual-modal features are integrated by skeleton-guided RGB frequency domain fusion. Finally, the behavior recognition is completed using the spatiotemporal classification module.

2. The multimodal behavior recognition method based on progressive guided fusion according to claim 1, characterized in that, S1 includes the following steps: S11. In multimodal behavior recognition scenarios, RGB video consists of a series of consecutive frames, each of which can be represented as image data I∈R. H×W×3 Where H is the image height, W is the image width, and 3 is the number of RGB channels; meanwhile, the skeletal data corresponding to each frame can be represented as a set of joint coordinates J = {j1,j2,…,j...} N }, where j i ∈R 3 Let N represent the three-dimensional coordinates of the i-th joint, and N be the total number of joints. S12. Based on the set of skeletal joint coordinates J, determine the smallest bounding box B containing all joints, with the top-left corner coordinates of the bounding box being (x...). min ,y min The coordinates of the lower right corner are (x max ,y max ); S13. Expand the bounding box B by 30% in the short side direction and 20% in the long side direction to obtain the expanded bounding box B′, whose coordinate range is limited to the size of the RGB frame [0,W)×[0,H). S14. Based on the expanded bounding box B', the RGB frame is cropped to obtain the cropped RGB data I', which focuses on the human behavior area and reduces background redundancy.

3. The multimodal behavior recognition method based on progressive guided fusion according to claim 1, characterized in that, S2 includes the following steps: S21. Construct an RGB feature extraction submodule based on EfficientNet-B0, and use the WTSP enhanced network to extract global information of RGB frames through wavelet convolution and ECA channel attention; S22, Skeletal Feature Extraction Submodule, adopts MSWTNet with a modular stacked structure and feature pyramid design to extract dynamic motion features of bones.

4. The multimodal behavior recognition method based on progressive guided fusion according to claim 3, characterized in that, In step S21, constructing the framework RGB feature extraction submodule includes the following steps: S211. Preprocess the input RGB video frame sequence without discarding any frames; S212. Apply ECA channel attention, use 1D convolution to process all frames, capture the correlation between adjacent frames and distant frames, dynamically adjust the weights according to the importance of the frames, and generate channel weights to highlight core action moments. S213. By using wavelet convolution, the video signal is decomposed into three frequency domain components: high, medium, and low, to achieve multi-resolution feature extraction and enhance the expression of motion details by RGB. S214. Input the processed RGB video frame features into EfficientNet-B0.

5. The multimodal behavior recognition method based on progressive guided fusion according to claim 4, characterized in that, In step S212, the application of ECA channel attention to generate channel weights to highlight core behavioral moments includes the following steps: S2121. Input RGB video frame feature map Where C represents the number of channels, T represents the number of frames in the time dimension, and H and W represent the height and width of a single frame image. Spatial global average pooling is performed to compress the spatial dimension H×W, resulting in the time-channel feature representation. Preserve the correlation information between the channel and the time dimension; S2122. The 1D convolutional kernel size k is adaptively determined based on the number of channels C, and the calculation formula is as follows: Subsequently, a 1D convolution operation with kernel size k is performed on X′ along the channel dimension to capture the dependencies between adjacent and non-adjacent channels, thus obtaining the channel attention intermediate features. S2123, Regarding X att Apply the Sigmoid activation function to generate channel attention weights. Each element ω c This indicates the importance of the c-th channel at the core action moment; S2124. Perform channel-wise weighting of the channel attention weight ω and the time-channel feature representation X′. That is, for the channel features at each time step, scale them according to the weight of the channel corresponding to ω to obtain the weighted features. Enhance channel features at core behavioral moments.

6. The multimodal behavior recognition method based on progressive guided fusion according to claim 4, characterized in that, In step S213, the video signal is decomposed into high, medium, and low frequency components to achieve multi-resolution feature extraction, including the following steps: S2131. The feature map x∈R of the input RGB video frame sequence. C×T×H′×W′ Where C is the number of channels, T is the number of frames, H′ is the feature map height, and W′ is the feature map width, a wavelet transform kernel is applied for convolution operation; S2132. By using wavelet transform kernels of different scales, the frequency components of the video signal are divided into high-frequency components x. high Mid-frequency component x mid and low frequency component x low High-frequency components correspond to fast motion, medium-frequency components correspond to medium-speed behavior trends, and low-frequency components correspond to slow background changes. S2133. Perform feature extraction and fusion on the three frequency domain components obtained by division to obtain multi-resolution RGB feature representation.

7. The multimodal behavior recognition method based on progressive guided fusion according to claim 3, characterized in that, In step S22, constructing the skeleton feature extraction and fusion module includes the following steps: S221. Split the input features into two parallel branches for processing; S222: One branch uses point convolution to quickly transform and initially fuse the channel dimension features; In step S222, point convolution is used to quickly transform and initially fuse the channel dimension features, including the following steps: S2221. Input the skeletal feature map. Perform dimensional flattening, where C in Where T is the number of input channels, T is the number of frames in the time dimension, and J is the total number of joints. This represents the flattened features. This merges the time dimension and the joint dimension into a single dimension, making convolution operations in the channel dimension easier. S2222, Use dotted convolution on the flattened features Perform channel dimension transformation; the parameters of the convolution kernel are: C mid The preset number of intermediate channels is determined through matrix operations. Characteristics obtained after preliminary fusion Complete the feature transformation of the channel dimension and the initial fusion of cross-time-joint information; S2223, The initially fused feature X conv Remodeling In this form, the structured representation of the time and joint dimensions is restored, providing dimension-matched input for subsequent multi-scale feature extraction of wavelet convolution; S223, another branch uses wavelet convolution with a kernel size of 3×3 twice in succession; In step S223, two consecutive wavelet convolutions with a kernel size of 3×3 are applied, including the following steps: S2231. The skeletal features after initial fusion via point convolution are... Where C mid The input consists of the number of intermediate channels, T (the number of frames in the time dimension), and J (the total number of joints), fed into two consecutive sets of 3×3 wavelet convolutional units. These wavelet convolutional units, based on a preset wavelet basis function, perform multi-scale decomposition of the input features in both spatial and temporal dimensions to extract multi-scale features of the skeleton. S224. Add the processing results of the two branches element by element to produce the merged output.

8. The multimodal behavior recognition method based on progressive guided fusion according to claim 1, characterized in that, Step S3 includes the following steps: S31. Transfer the features output by the RGB feature extraction submodule and the skeleton feature extraction submodule to the progressive guided fusion submodule to perform skeleton self-guided enhancement of skeleton features. S32. Perform skeleton-guided RGB frequency domain fusion and integration of dual-modal features; S33. Construct the spatiotemporal classification module of the PGFNet framework. This module efficiently transforms the fused features into behavior classification results through spatial global pooling, 1D temporal convolution, fully connected layers and Softmax, taking into account both spatiotemporal feature modeling and lightweighting. In step S31, the self-guided bone enhancement feature includes the following steps: S311, Unify the number of channels for RGB features and skeleton features through a fully connected layer; In step S311, the number of channels is unified through the fully connected layer, including the following steps: S3111, Input joint features Where C j Let T be the number of skeletal feature channels, T be the number of time frames, and J be the number of joints. Perform global average pooling to compress the T and J dimensions to obtain the global channel feature representation of the joint features. To aggregate information about each joint in the channel dimension; S312. Use the ECA module to convert joint features into channel attention for skeleton features, thereby enabling joint features to guide skeleton features, i.e., self-guidance of skeleton features. In step S312, the channel attention of the skeleton features is transformed using the ECA module, including the following steps: S3121, The joint-level feature F′ joint The input ECA channel attention module first focuses on the number of skeletal feature channels C. b The 1D convolution kernel size is adaptively determined, and the dependencies between channels are captured through this 1D convolution. The calculation formula is as follows: The Sigmoid activation function is then applied to the convolution output to generate channel attention weights. Then, the attention weight α is compared with the original skeletal features F. bone Perform channel-by-channel weighting, and then restore the enhanced skeletal features to a dimension of [missing value]. Highlight key skeletal movement features; S3122, respectively, for RGB features Where C r H represents the number of RGB feature channels, and H and W represent the height, width, and enhanced skeletal features of the feature map. Dimensional mapping is performed using a linear projection layer. The parameters of the linear projection layer are set as follows: Map RGB features and skeletal features to a unified projection dimension C. proj , obtain the projected RGB features and skeletal features This addresses the issue of cross-modal feature dimension mismatch, providing feature inputs of the same dimension for subsequent frequency domain fusion. In step S32, the skeleton-guided RGB frequency domain fusion and integration of dual-modal features includes the following steps: S321. Use Fast Fourier Transform to transform RGB features and skeleton self-guided features from the spatial domain to the frequency domain, and train the real part filter and imaginary part filter of Fourier complex features. In step S321, the RGB features and skeleton self-guided features are transformed from the spatial domain to the frequency domain using Fast Fourier Transform, including the following steps: S3211. Projected RGB features Among them, C proj Let T be the number of projection channels, H be the number of time frames, and W be the height and width of the RGB feature map. Perform global average pooling along the spatial dimension H×W to obtain the compressed RGB time series features. Aggregate spatial dimension information to focus on temporal dynamics; S3212, to For each channel's time series, a Fast Fourier Transform is applied channel by channel to convert the time-domain signal into a complex frequency-domain signal, thus obtaining the RGB frequency domain characteristics. in, Represents the complex number field, containing information about the real and imaginary parts; S3213, Self-guiding features of projected skeleton Where J is the total number of joints, global average pooling is performed along the joint dimension J to obtain the compressed bone time series features. Aggregate joint dimension information to focus on temporal motion dynamics; S3214, to The time series of each channel in the dataset is C. proj Each channel is a one-dimensional sequence of length T. A Fast Fourier Transform is applied channel by channel to convert the time-domain signal into a complex frequency-domain signal, yielding the bone frequency-domain features. Complete the conversion from the spatial domain to the frequency domain; S322, the real part filter, and the imaginary part filter are applied to the real and imaginary parts of the Fourier feature, respectively, to achieve filtering weighting of amplitude and phase; In step S322, constructing a real part filter and a real part filter to achieve amplitude and phase filtering weighting includes the following steps: S3221. Initialize the trainable frequency domain filter parameter matrix. For the RGB frequency domain features and the skeleton frequency domain features, construct real and imaginary filters, corresponding to amplitude modulation and phase modulation, respectively. The real filter parameter matrix for the RGB mode is as follows: The parameter matrix of the imaginary part filter is The real part filter parameter matrix of the skeletal mode is The parameter matrix of the imaginary part filter is All filter parameters are initialized using a Xavier normal distribution to ensure that the initial weight distribution adapts to the frequency domain feature dimension. Wherein, C proj Where T is the number of projection channels and T is the number of time frames; S3222, Extract RGB frequency domain features The real part Re(F) rgb-freq ) and the imaginary part Im(F rgb-freq ), respectively with the RGB real part filter W rgb-r , Imaginary part filter W rgb-i Perform element-wise multiplication to obtain the filtered RGB real part features and the filtered RGB imaginary part features, respectively: Re(F rgb-freq ′)=Re(F rgb-freq )⊙W rgb-r (2) And(F rgb-freq ′)=And(F rgb-freq )⊙W rgb-i (3) Furthermore, the filtered RGB frequency domain features are obtained through reconstruction: F rgb-freq ′=Re(F rgb-freq ′)+j·I(F rgb-freq ′) (4) S3223. Employ weighted logic consistent with RGB modality to extract bone frequency domain features. The real part Re(F) bone-freq ) and the imaginary part Im(F bone-freq ), respectively with the real part filter W of the skeleton bone-r , Imaginary part filter W bone-i Perform element-wise multiplication to obtain the filtered real part features and the filtered imaginary part features of the skeleton: Re(F bone-freq ′)=Re(F bone-freq )⊙W bone-r (5) And(F bone-freq ′)=And(F bone-freq )⊙W bone-i (6) And the filtered bone frequency domain features obtained after reconstruction are as follows: F bone-freq ′=Re(F bone-freq ′)+j·I(F bone-freq ′) (7) S3224, Filter the RGB frequency domain features F rgb-freq ′ and bone frequency domain features F bone-freq The input is then used in the subsequent complex multiplication fusion stage. Combined with the overall model's cross-entropy loss function, all filter parameters W are dynamically updated via backpropagation. rgb-r W rgb-i W bone-r This enables the filter to adaptively learn the contribution weights of different frequency domain components to behavior recognition, thereby optimizing the cross-modal frequency domain fusion effect. S323. Multiply the filtered RGB frequency domain features and the skeleton frequency domain features element-wise to achieve frequency domain fusion. Use inverse fast Fourier transform to convert the fused frequency domain features to the spatial domain; In step S323, the filtered RGB frequency domain features and the skeleton frequency domain features are multiplied element-wise to achieve frequency domain fusion, including the following steps: S3231. Determine the filtered RGB frequency domain characteristics to be fused. With bone frequency domain features Among them, C proj To standardize the number of projection channels, T represents the number of time frames. Representing the complex field, the number of channels and the time dimension are perfectly matched, satisfying the dimension requirement of element-wise multiplication; S3232, Perform element-wise multiplication of the complex field; based on the rules of complex multiplication, perform multiplication on F. rgb-freq ′ and F bone-freq Perform position-by-position complex multiplication. Let F rgb-freq The complex value at a certain channel-time position is a + jba, where a is the real part, b is the imaginary part, j is the imaginary unit, and F bone-freq The complex value at the corresponding position is c + jd, where c is the real part and d is the imaginary part. The product is (ac - bd) + j(ad + bc). This operation achieves deep interactive fusion of RGB and skeletal frequency domain features at the amplitude and phase levels, yielding preliminary fused complex frequency domain features. S3233, Extraction of Real Parts of Fusion Features and Dimensional Calibration. This involves the initial fusion of complex frequency domain features F... fusion-freq-raw By applying the fast inverse Fourier transform, the frequency domain fused features are mapped back to the time domain to obtain the time domain complex features. Since there may be a small imaginary part remaining after the IFFT transformation, the real part of this temporal complex feature is taken as the effective temporal fusion feature. Ensure that the features are real numbers to match the input requirements of the subsequent classification module; S3234. Residual connections enhance semantic preservation. To avoid losing the basic semantic information of the RGB modalities during the fusion process, the original RGB features are mapped to C through a linear projection layer. proj ×T dimensions yield the RGB basic semantic features for dimension matching. F fusion-time-real With F rgb-residual Perform element-wise addition to obtain the final temporal fusion feature. This feature contains complementary information from cross-modal frequency domain fusion while preserving the original RGB semantics, providing rich discriminative basis for subsequent behavior classification; In step S33, the feature extraction and fusion module for constructing the PGFNet framework includes the following steps: S331. Perform global average pooling on the fused features to compress the spatial dimension, thereby aggregating spatial semantic information and reducing the amount of subsequent computation. S332. Process the pooled features through 1D convolutional layers to capture the temporal dependencies between frames; S333. The features processed by 1D temporal convolution are input into the fully connected layer. The output dimension of the fully connected layer is the total number of behavior categories. Then, the probability distribution of each behavior is output through the Softmax activation function to complete the decision-making process of behavior recognition and obtain the predicted probability corresponding to each behavior category.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the steps in the multimodal behavior recognition method based on progressive guided fusion as described in any one of claims 1 to 8.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal behavior recognition method based on progressive guided fusion as described in any one of claims 1 to 8.

Images