Human action recognition method and system based on energy driving and hypergraph guided frequency decoupling

Abstract

The present application relates to the field of behavior recognition, and provides a human action recognition method and system based on energy driving and supergraph guided frequency decoupling. The method comprises: extracting skeleton sequence feature representation from acquired video data, inputting an HFM-Net network, and the HFM-Net network comprises: (1) a supergraph guided frequency decoupling module extracts static structure features and dynamic motion features in the spatiotemporal features, respectively constructs a structure-oriented supergraph and a motion-oriented supergraph, uses high-order topology guidance induced by the supergraph to guide the frequency decoupling of the spatial feature aggregation of the corresponding branch, and obtains the decoupled spatial enhanced features; (2) an energy driving sparse time attention module acquires a frame-level motion energy signal, adaptively allocates different attention neighborhood ranges to different time frames according to the frame-level motion energy signal, performs attention calculation only in the sparse neighborhood corresponding to each frame, obtains time enhanced features, and outputs an action recognition result through a classifier.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition technology, and in particular to a human action recognition method and system based on energy-driven and hypergraph-guided frequency decoupling. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Human behavior recognition is an important research direction in computer vision, aiming to automatically understand the semantics of human actions in video sequences. It has wide application value in scenarios such as intelligent monitoring, human-computer interaction, and medical rehabilitation. Early methods used RGB video as input, which were easily affected by factors such as changes in lighting, differences in viewing angle, and cluttered backgrounds. With the rapid development and widespread application of depth sensors, behavior recognition methods based on human skeletons have attracted widespread attention due to their insensitivity to changes in visual appearance. These methods use human joint coordinate sequences as input, treating joints as nodes and skeletal connections as edges, constructing a spatiotemporal graph structure that can capture the kinematic features of the human body with relatively low computational overhead.

[0004] Currently, skeleton-based methods mainly rely on graph convolutional networks (GCNNs) and Transformers. GCNNs extract spatial features using skeletal graph topology, while Transformers model dynamic connections between joints through self-attention. However, existing methods still have the following shortcomings: First, standard graph convolutions rely only on paired node connections, lacking the ability to capture high-order collaborative relationships between multiple functionally related joints at long distances, thus limiting the model's representation of complex limb coordination movements. Second, during spatial feature aggregation, stable static structural patterns and motion-sensitive dynamic change patterns are coupled and entangled, making it difficult for the model to focus on truly discriminative motion features, reducing the recognition performance of fine-grained movements. Third, existing temporal modeling often employs a global dense attention mechanism, introducing a large amount of redundant interaction in weak motion or static periods and easily amplifying noise interference; at the same time, spatial motion modeling and temporal context modeling are independent of each other, lacking unified motion intensity signal coordination, resulting in insufficient adaptability of the model to changes in movement speed and differences in execution style. Summary of the Invention

[0005] To address the technical problems mentioned above, this invention provides a human motion recognition method and system based on energy-driven and hypergraph-guided frequency decoupling. In spatial modeling, a hypergraph-guided frequency decoupling module is designed to separate static structure from dynamic motion. By constructing structure-guided and motion-guided hypergraphs, high-order dependencies between joints are captured, achieving frequency-decoupled spatial feature aggregation. In temporal modeling, an energy-driven sparse temporal attention module is designed. A unified frame-level motion energy signal is used to adaptively optimize the temporal attention range, achieving efficient temporal context modeling. In feature collaboration, the learning of the spatial motion branch and the temporal attention module is coordinated through shared motion energy signals, enhancing the model's adaptability to changes in motion speed.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling.

[0007] A human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling includes: Extract the human skeleton structure from the acquired video data and convert it into a human skeleton graph structure representation, then extract the skeleton sequence feature representation; Based on the skeleton sequence feature representation, a Hypergraph-guided Frequency-decoupling Network with Motion-energymodulation (HFM-Net) is used to obtain action recognition results. The HFM-Net network includes a preprocessing layer, ten sequentially stacked spatiotemporal blocks, and a classifier. Each of the first to seventh spatiotemporal blocks contains a graph convolution module and a temporal convolution module. The eighth and ninth spatiotemporal blocks are both hypergraph-guided frequency-decoupling modules. The tenth spatiotemporal block is a joint refinement block, containing both a hypergraph-guided frequency-decoupling module and an energy-driven sparse temporal attention module. The preprocessing layer is used to perform channel expansion and positional encoding injection on the skeleton sequence feature representation to generate initial spatiotemporal features. Based on the initial spatiotemporal features, local spatiotemporal features are extracted from the first to seventh spatiotemporal blocks. For local spatiotemporal features, the static structural features and dynamic motion features are extracted from the spatiotemporal features by using the hypergraph-guided frequency decoupling module of the eighth, ninth, and tenth spatiotemporal blocks. Structure-guided hypergraphs and motion-guided hypergraphs are constructed respectively. Spatial features are aggregated through frequency decoupling using these hypergraphs to obtain decoupled spatial enhancement features. Based on these spatial enhancement features, the energy-driven sparse temporal attention module of the tenth spatiotemporal block is used. The frame-level motion energy signal output by the hypergraph-guided frequency decoupling module adaptively guides sparse temporal interactions to obtain temporal enhancement features. Based on these temporal enhancement features, a classifier is used to obtain the action recognition results.

[0008] Furthermore, the method involves extracting static structural features and dynamic motion features from the spatiotemporal features to construct a structure-guided hypergraph and a motion-guided hypergraph, respectively; the method includes: Based on local spatiotemporal features, the normalized frame difference between adjacent frames is extracted to obtain the motion change tensor. The static structural features are obtained by averaging the local spatiotemporal features along the time dimension; The dynamic motion characteristics are obtained by averaging the motion change tensor along the time dimension. The joint activation is obtained by averaging the motion change tensor along the channel and time dimensions and then normalizing it joint by joint. Based on static structural features and joint activation, a structure-guided hypergraph correlation matrix is ​​constructed. Based on the splicing of static structural features and dynamic motion features and joint activation, a motion-guided hypergraph correlation matrix is ​​constructed.

[0009] Furthermore, the spatial feature aggregation through frequency decoupling using structure-guided hypergraphs and motion-guided hypergraphs yields decoupled spatial enhancement features; the method includes: By multiplying the transposes of the structure-guided hypergraph incidence matrix and the motion-guided hypergraph incidence matrix and row normalization respectively, the structure-guided higher-order topological matrix and the motion-guided higher-order topological matrix are induced. The motion change tensor is averaged along the channel and joint dimensions and then normalized frame by frame to obtain the frame-level motion energy signal; Using frame-level motion energy signals, motion branch modulation gating is calculated to modulate the motion change tensor with energy, thereby obtaining motion energy characteristics; The structure-space attention matrix is ​​learned from local spatiotemporal features using a structural branch, and combined with a structure-guided higher-order topological matrix to output the structure-space attention features. These features are then aggregated with the local spatiotemporal features to obtain the structure aggregation matrix. The motion-space attention matrix is ​​learned from motion energy features using a motion branch, and combined with a motion-guided higher-order topological matrix to output the motion-space attention features. These features are then aggregated with the motion energy features to obtain the motion aggregation matrix. The spatial augmentation features are obtained by weighted fusion of the structural aggregation matrix, the motion aggregation matrix, and the residual branch output features based on the physical skeleton connection graph.

[0010] Furthermore, the structure-guided hypergraph association matrix is ​​expressed by the following formula:

[0011] The motion-guided hypergraph correlation matrix is ​​expressed by the following formula:

[0012] in, , The number of superedges. The number of joints is represented by a row, where each row indicates the membership degree of a joint on each hyperedge. and For learnable linear projection, It is a static structural feature. It is a dynamic motion characteristic. For joint activation, is the numerical stability constant.

[0013] Furthermore, a consistency loss is applied to the deep network layer of the hypergraph-guided frequency decoupling module, and the structural space attention features and motion space attention features are propagated along the corresponding higher-order topology, respectively, and the propagated features are constrained to maintain consistency with the static structural features and dynamic motion features, respectively.

[0014] Furthermore, the method based on spatial enhancement features employs an energy-driven sparse temporal attention module of the tenth spatiotemporal block, adaptively guiding sparse temporal interaction through the frame-level motion energy signal output by the hypergraph-guided frequency decoupling module to obtain temporal enhancement features; the method includes: Spatial enhancement features are applied to temporal convolution to obtain local temporal features. Local temporal features are aggregated along the joint dimension to obtain temporal tokens. Conditional positional encoding is applied, and adjacent frame context is injected into each temporal token through depthwise separable temporal convolution and layer normalization. Finally, the basic temporal attention score matrix is ​​calculated. The frame-level motion energy signal output by the supergraph guiding frequency decoupling module is zero-padded and aligned in the first frame to obtain the frame-level motion energy alignment signal, and then the frame-level motion energy alignment signal is converted into time motion gating; The temporal motion gating is used as a key-side saliency bias and superimposed on the basic temporal attention score matrix to obtain the modulation score matrix; Based on the frame-level motion energy alignment signal, linear interpolation is performed between the preset minimum attention budget and the maximum attention budget to obtain the attention neighborhood size corresponding to each query frame; For each query frame, in the modulation score matrix In the row, select the one with the highest score other than itself. Each non-self frame, together with itself, constitutes the sparse temporal neighborhood of the query frame; Softmax normalization is performed only within the sparse temporal neighborhood to obtain the sparse temporal attention matrix, and the attention weights of unselected frames are reset to zero; Aggregate temporal context based on sparse temporal attention matrix and adjacent frame context; The aggregated temporal context is extended back to the joint dimension via broadcasting and then residually fused with the local temporal features to obtain temporally enhanced features.

[0015] Furthermore, the loss of the HFM-Net network is:

[0016] in, Cross-entropy classification loss, The weighting coefficient for consistency loss. This is due to deep consistency loss. The symbol represents the network spatiotemporal block index, and the summation symbol indicates that a consistency loss is applied to all spatiotemporal blocks from the eighth to the tenth.

[0017] A second aspect of the present invention provides a human motion recognition system based on energy-driven and hypergraph-guided frequency decoupling.

[0018] A human motion recognition system based on energy-driven and hypergraph-guided frequency decoupling, comprising: The feature extraction module is configured to: extract the human skeleton structure from the acquired video data, convert it into a human skeleton graph structure representation, and extract the skeleton sequence feature representation; The action recognition module is configured to: obtain action recognition results based on the skeleton sequence feature representation using an HFM-Net network; the HFM-Net network includes: a preprocessing layer, ten sequentially stacked spatiotemporal blocks, and a classifier; wherein, each of the first to seventh spatiotemporal blocks contains a graph convolution module and a temporal convolution module; the eighth and ninth spatiotemporal blocks are both hypergraph-guided frequency decoupling modules; the tenth spatiotemporal block is a joint refinement block, containing a hypergraph-guided frequency decoupling module and an energy-driven sparse temporal attention module; the preprocessing layer is used to perform channel expansion and positional encoding injection on the skeleton sequence feature representation to generate initial spatiotemporal features; based on the initial spatiotemporal features, through the first to seventh spatiotemporal blocks, each spatiotemporal block contains a graph convolution module and a temporal convolution module; the eighth and ninth spatiotemporal blocks are both hypergraph-guided frequency decoupling modules; the tenth spatiotemporal block is a joint refinement block, containing a hypergraph-guided frequency decoupling module and an energy-driven sparse temporal attention module; the preprocessing layer is used to perform channel expansion and positional encoding injection on the skeleton sequence feature representation to generate initial spatiotemporal features; based on the initial spatiotemporal features, the first to ten spatiotemporal blocks are processed by the HFM-Net network. The seventh spatiotemporal block extracts local spatiotemporal features. Based on these features, static structural and dynamic motion features are extracted from the spatiotemporal features through the hypergraph-guided frequency decoupling modules of the eighth, ninth, and tenth spatiotemporal blocks. Structure-guided and motion-guided hypergraphs are constructed respectively, and spatial features are aggregated through frequency decoupling using these hypergraphs to obtain decoupled spatial enhancement features. Based on these spatial enhancement features, the energy-driven sparse temporal attention module of the tenth spatiotemporal block is used to adaptively guide sparse temporal interactions using the frame-level motion energy signal output by the hypergraph-guided frequency decoupling module to obtain temporal enhancement features. Based on these temporal enhancement features, a classifier is used to obtain action recognition results.

[0019] A third aspect of the present invention provides a computer device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling as described in the first aspect above.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute steps in the human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling as described in the first aspect above.

[0021] The fifth aspect of the present invention provides a computer program product or computer program.

[0022] This invention provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling as described in the first aspect above.

[0023] Compared with the prior art, the beneficial effects of the present invention are: In this invention, the hypergraph-guided frequency decoupling module constructs two types of hypergraphs, structure-guided and motion-guided, to extend paired node relationships into multi-node higher-order relationships with hyperedge connections, effectively capturing the collaborative patterns between multiple long-distance functionally related joints. At the same time, it separates static structural features and dynamic motion features into different branches for processing, avoiding mutual coupling and interference between the two, and improving the model's ability to recognize complex limb coordination movements and fine-grained movements.

[0024] In this invention, the energy-driven sparse temporal attention module utilizes a unified frame-level motion energy signal to adaptively allocate different attention budgets to frames with different motion intensities. During periods of weak motion or static conditions, it employs a compact attention neighborhood to reduce redundant interactions and noise interference, while maintaining a wide temporal receptive field during periods of strong motion to capture the complete action context. This reduces computational overhead while improving the accuracy of temporal modeling.

[0025] In this invention, frame-level motion energy signals are used simultaneously to regulate the feature enhancement intensity of spatial motion branches and the neighborhood allocation of sparse temporal attention, realizing collaborative learning of spatial motion modeling and temporal context modeling. This enables the model to adaptively handle rhythm and speed changes of different actions, enhancing the model's robustness to differences in execution style. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] Figure 1 This is a flowchart illustrating a human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling, as shown in an embodiment of the present invention. Figure 2 This is a diagram illustrating the overall algorithm framework of the human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling, as shown in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of the HFM-Net network shown in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the supergraph guiding frequency decoupling module shown in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an energy-driven sparse temporal attention module as shown in an embodiment of the present invention; Figure 6 This is a structural diagram of a human motion recognition system based on energy-driven and hypergraph-guided frequency decoupling, as shown in an embodiment of the present invention. Figure 7 This is a structural diagram of a computer device shown in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] Figure 1 This is a flowchart illustrating a human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling, as shown in an embodiment of the present invention; see reference. Figure 1 The method includes: Extract the human skeleton structure from the acquired video data and convert it into a human skeleton graph structure representation, then extract the skeleton sequence feature representation; Based on the skeleton sequence feature representation, an HFM-Net network is used to obtain action recognition results. The HFM-Net network includes a preprocessing layer, ten sequentially stacked spatiotemporal blocks, and a classifier. Each of the first to seventh spatiotemporal blocks contains a graph convolution module and a temporal convolution module. The eighth and ninth spatiotemporal blocks are both hypergraph-guided frequency decoupling modules. The tenth spatiotemporal block is a joint refinement block, containing a hypergraph-guided frequency decoupling module and an energy-driven sparse temporal attention module. The preprocessing layer is used to perform channel expansion and positional encoding injection on the skeleton sequence feature representation to generate initial spatiotemporal features. Based on the initial spatiotemporal features, the first to seventh spatiotemporal blocks are used to extract... Local spatiotemporal features are extracted. Based on these features, static structural features and dynamic motion features are extracted from the spatiotemporal features through the hypergraph-guided frequency decoupling module of the eighth, ninth, and tenth spatiotemporal blocks. Structure-guided hypergraphs and motion-guided hypergraphs are constructed respectively. Spatial features are aggregated through frequency decoupling using the structure-guided and motion-guided hypergraphs to obtain decoupled spatial enhancement features. Based on these spatial enhancement features, an energy-driven sparse temporal attention module of the tenth spatiotemporal block is used. The frame-level motion energy signal output by the hypergraph-guided frequency decoupling module is adaptively guided to guide sparse temporal interaction to obtain temporal enhancement features. Based on these temporal enhancement features, a classifier is used to obtain the action recognition results.

[0032] This invention captures high-order dependencies between joints through hypergraph modeling, separates static structure and dynamic motion through frequency decoupling, and coordinates spatial motion modeling and temporal context modeling through shared motion energy signals. It effectively solves the problems of insufficient modeling of high-order joint dependencies, mutual coupling between static structure and dynamic motion, and temporal interaction redundancy in existing methods, thereby improving the accuracy of behavior recognition while reducing computational overhead.

[0033] Figure 2 This is a diagram illustrating the overall algorithm framework of a human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling, as shown in an embodiment of the present invention; see reference. Figure 2 The human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling includes the following steps: Step 1: Obtain the human skeleton sequence dataset for model training and convert it into a graph representation.

[0034] To obtain the human skeleton sequence and convert it into a spatiotemporal feature representation, this embodiment uses three publicly available datasets: NTU RGB+D 60, NTU RGB+D 120, or NW-UCLA, to extract the human skeleton structure from each frame of the image. Specifically, human joints are treated as nodes, and skeletal connections are treated as edges, constructing a graph structure representation, denoted as […]. ,in For a set of joint nodes, This represents the total number of joint nodes in each frame. Represents the edge set. The feature representation of the skeleton sequence is as follows: Where B is the batch size, C is the number of channels, T is the number of time frames, and V is the number of joint nodes contained in each frame.

[0035] For different skeleton datasets, in this embodiment, the number of joints per frame in the NTU RGB+D 60 dataset is... (25 joints per person in a two-person exercise), the joint dimensions of multiple people are combined into one. Each node, with skeleton sequences uniformly sampled to 64 frames, and input channel dimension C set to 96. NW-UCLA dataset single-frame joint count. After spatial embedding, the input channel dimension is mapped to... .

[0036] Step 2: Establish an HFM-Net network based on hypergraph-guided frequency decoupling and energy-driven sparse temporal attention to learn the skeleton data and obtain classification results.

[0037] refer to Figure 3The HFM-Net model consists of a preprocessing layer, ten stacked spatiotemporal blocks, and a classifier. The preprocessing layer performs channel expansion and positional encoding injection on the input skeleton sequence to generate initial spatiotemporal features.

[0038] In the structural design of the ten spatiotemporal blocks, the first to seventh spatiotemporal blocks are conventional spatiotemporal construction modules, each containing a graph convolution module and a temporal convolution module for extracting shallow local spatiotemporal features. The eighth and ninth spatiotemporal blocks are hypergraph-guided frequency decoupling modules, while the tenth spatiotemporal block is a joint refinement block, containing both a hypergraph-guided frequency decoupling module and an energy-driven sparse temporal attention module. Specifically, the hypergraph-guided frequency decoupling modules in the eighth, ninth, and tenth spatiotemporal blocks separate the static structure from the dynamic motion in the deep features, aggregating spatial features with frequency decoupling through structure-guided and motion-guided hypergraphs. The energy-driven sparse temporal attention module in the tenth spatiotemporal block adaptively guides sparse temporal interactions using frame-level motion energy signals. The enhanced features processed by the ten spatiotemporal blocks are then input into a classifier to complete behavior recognition.

[0039] The Hypergraph-guided frequency decoupling module is activated in the deep layers of the network, which allows the shallow stage to focus on the extraction of spatiotemporal features of local details, while the deep stage performs high-order spatial modeling of frequency decoupling based on the obtained local features. This avoids the introduction of high-order topology too early in the shallow stage due to insufficient feature extraction, which would lead to feature mixing.

[0040] Step 3: Supergraph guides the frequency decoupling module.

[0041] refer to Figure 4 The specific implementation of the hypergraph-guided frequency decoupling module is as follows: For input features First, extract the normalized frame difference between adjacent frames. All frame differences Stacking along the time dimension yields the motion change tensor. .

[0042] right Static structural features are obtained by averaging along the time dimension. ; right The dynamic motion characteristics are obtained by averaging along the time dimension. ; right After averaging along the channel and time dimensions, joint activation is normalized to obtain the joint activation. Characterizes the degree of motion activity of each joint: ; right After averaging along the channel and joint dimensions, the signal is normalized frame by frame to obtain the frame-level motion energy signal. : .

[0043] in, , , is the numerical stability constant. The joint activation degree is obtained by averaging the motion changes along the channel and time dimensions and then normalizing it joint by joint. ; The frame-level motion energy is obtained by averaging the motion changes along the channel and joint dimensions and then normalizing it frame by frame. .

[0044] Based on static structural features and joint activation Construct the structure-guided hypergraph association matrix:

[0045] splicing based on static structural features and dynamic motion features and joint activation Construct the motion-guided hypergraph correlation matrix:

[0046] in, , The number of superedges. The number of joints is represented by a row, where each row indicates the membership degree of a joint on each hyperedge. and For learnable linear projection, This is a numerical stability constant used to prevent overflow in denominators or logarithmic operations. In this embodiment, it is set to a value of [value missing]. Softmax is performed along the joint dimension. In the above design, the logarithmic term converts joint activation into prior biases, such that stable joints are preferentially assigned to the structure-guided hypergraph, and active joints are preferentially assigned to the motion-guided hypergraph.

[0047] By multiplying the transpose of the incidence matrix and performing row normalization, structure-oriented and motion-oriented higher-order topological matrices are induced, respectively: ,

[0048] in, These are structure-oriented higher-order topological matrices and motion-oriented higher-order topological matrices, respectively, with the structure-oriented higher-order topological matrix... For example, The Each element represents a joint. and The higher-order correlation strength between nodes connected by a hypergraph is a value that is relatively large if two nodes frequently appear on the same hyperedge, even if they are not physically connected.

[0049] Using frame-level motion energy signals Computational motion branch modulation gating:

[0050] in, For the Sigmoid function, temperature parameter Used to control the steepness of the mapping from energy to the gating value. The larger the value, the smoother the gating change. The smaller the value, the more sensitive the gating is to energy differences. In this embodiment... The value is 1.0. Because... The factor of 2 makes the gain 1 in weak motion and gradually enhances the response in high motion frames.

[0051] Energy modulation of the motion change tensor:

[0052] To enhance motion response in high-motion frames and preserve change information in low-motion frames, This indicates element-wise multiplication.

[0053] The structural and motion branches each combine their respective learnable spatial attention with the corresponding hypergraph topology: ,

[0054] in, and The spatial attention matrix is ​​learned from the input features. and It is a learnable scalar used to control the guidance strength of the hypergraph topology. and Initializing it to 0 allows the model to primarily rely on learnable spatial attention during the initial training phase, while gradually introducing hypergraph prior knowledge as training progresses.

[0055] Based on the fused spatial attention features and Perform feature aggregation, and the structure branch outputs a structure aggregation matrix:

[0056] Motion branch outputs motion aggregation matrix:

[0057] in, This represents a feature aggregation operation weighted by the attention matrix. Represents the tensor due to motion changes The time dimension is After spatial aggregation, zeros need to be padded at the beginning of the time dimension to restore the original frame number. Zeros are padded at the beginning of the time dimension to restore the frame rate. Meanwhile, the physical skeleton diagram provides residual paths. The three branches are mediated by learnable scalars. , Perform weighted fusion:

[0058] in, , The residual branch output is based on the physical skeleton connection graph and is used to preserve the original skeleton topology information to prevent the hypergraph topology learning from deviating from the human physical structure.

[0059] Apply a consistency loss at a deeper network layer to propagate static structural features and motion features along their corresponding higher-order topologies:

[0060] The features after constraint propagation must remain consistent with the original features:

[0061] in, The Frobenius norm, or the square root of the sum of the squares of all elements in a matrix, is used to measure the degree of difference between two feature matrices. This consistency loss forces the features propagated from the hypergraph to remain semantically consistent with the original structural and motion features, thus ensuring that each hypergraph branch focuses on its corresponding feature type.

[0062] Step 4: Energy-driven sparse temporal attention module.

[0063] refer to Figure 5 The specific implementation of the energy-driven sparse temporal attention module is as follows: Spatial features after fusion Applying temporal convolution yields local temporal features. Frame-level motion energy Zero-padding alignment in the first frame And convert it into time motion gating:

[0064] in, It is a time-gated motion signal, where each element takes a value between (0,1) and represents the motion intensity of the corresponding time frame. For the Sigmoid function, The temperature parameter controls the steepness of the mapping from energy to the gating value. Therefore, each frame is assigned a differentiable motion intensity metric. ( (For frame index).

[0065] Time tokens are obtained by aggregating local temporal features along the joint dimension. Conditional positional encoding is applied, injecting adjacent frame context into each token through depthwise separable temporal convolution and layer normalization:

[0066] in, It is a depthwise separable temporal convolution with a kernel size of 3, and LN is the layer normalization. Unlike fixed-position coding, conditional position coding adaptively extracts local positional information from adjacent frames, preserving short-term temporal continuity.

[0067] Time token encoded from conditional position Calculate the base time attention score matrix:

[0068] in, This is a function that generates inter-frame pairwise fundamental fractional matrices.

[0069] Subsequently, time motion gating was introduced. As keyframe The significance bias modulates the fundamental fraction matrix: the temporal motion gating value of each frame is used. This is superimposed on the base attention score, giving frames with high motion intensity a greater weight in the attention calculation, thus yielding the modulation score matrix. , its first Line 1 The column elements are:

[0070] in, Index the query frame and the key frame separately. These are learnable modulation coefficients. This operation allows temporal attention to adaptively adjust its importance based on the motion intensity of the keyframes.

[0071] Each query frame is adaptively calculated based on frame-level motion energy. Time attention budget:

[0072] in, , , and The preset minimum and maximum neighborhood ratios, It is a rounding function. This is the rounding function. High-energy query frames receive a larger attention budget to preserve broader temporal interactions, while low-energy query frames use a more compact neighborhood.

[0073] For each query frame In the modulation fractional matrix In the row, select the one with the highest score other than itself. Each non-self frame, together with itself, constitutes the sparse temporal neighborhood of the query frame. The sparse neighborhood set of all query frames is denoted as . Among them, the first sparse neighborhood of each query frame Defined as:

[0074] in, , Return the highest score One index, This indicates the number of additional keyframes selected besides the query frame itself, which, together with the query frame, constitute a size of [size missing]. The sparse temporal neighborhood. Select the highest-scoring frame from non-self frames. Each frame. Representation matrix No. All columns remaining after removing the column containing the current column from the current row.

[0075] Only Softmax normalization is performed internally to obtain the sparse temporal attention matrix. , its first The row is calculated as follows: ,

[0076] in, Represents the modulation fractional matrix. The row belongs to the neighborhood elements, This indicates a location that does not belong to the neighborhood. The attention weights for unselected frames are reset to zero. This is based on the sparse attention matrix. Aggregate time context :

[0077] Or equivalent:

[0078] in, The time token feature after conditional location encoding. This represents an attention-weighted aggregation operation along the time dimension. The aggregated context. Then, it is expanded back to the joint dimension via broadcasting, and... Perform residual fusion.

[0079]

[0080] in, These are learnable residual coefficients that control the contribution of sparse temporal context.

[0081] Step 5: Extract and fuse spatiotemporal features through the network, and use the Softmax classifier to predict the behavior category.

[0082] Will Global average pooling is performed along the spatial dimension to aggregate discriminative features from each joint and frame, resulting in a sample-level feature vector. This feature vector is then input into a fully connected layer, normalized using the Softmax function, and outputs the predicted probability of the action category. The total model loss is a weighted sum of the cross-entropy classification loss and the deep consistency loss.

[0083] in, Cross-entropy classification loss, The weighting coefficients for consistency loss are used to balance the importance of classification tasks and hypergraph semantic preservation tasks. The summation symbol indicates that a consistency loss is applied to all spatiotemporal blocks from the 8th to the 10th spatiotemporal blocks.

[0084] The above combination Figure 1 The human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling provided by the embodiments of the present invention has been described in detail. Next, the human motion recognition system based on energy-driven and hypergraph-guided frequency decoupling provided by the embodiments of the present invention will be described in conjunction with the accompanying drawings.

[0085] Figure 6 This is a schematic diagram of the structure of a human motion recognition system based on energy-driven and hypergraph-guided frequency decoupling, as shown in an embodiment of the present invention. Figure 6 The system described in this invention includes: The feature extraction module is configured to: extract the human skeleton structure from the acquired video data, convert it into a human skeleton graph structure representation, and extract the skeleton sequence feature representation; The action recognition module is configured to: obtain action recognition results based on the skeleton sequence feature representation using an HFM-Net network; the HFM-Net network includes: a preprocessing layer, ten sequentially stacked spatiotemporal blocks, and a classifier; wherein, each of the first to seventh spatiotemporal blocks contains a graph convolution module and a temporal convolution module; the eighth and ninth spatiotemporal blocks are both hypergraph-guided frequency decoupling modules; the tenth spatiotemporal block is a joint refinement block, containing a hypergraph-guided frequency decoupling module and an energy-driven sparse temporal attention module; the preprocessing layer is used to perform channel expansion and positional encoding injection on the skeleton sequence feature representation to generate initial spatiotemporal features; based on the initial spatiotemporal features, through the first to seventh spatiotemporal blocks, each spatiotemporal block contains a graph convolution module and a temporal convolution module; the eighth and ninth spatiotemporal blocks are both hypergraph-guided frequency decoupling modules; the tenth spatiotemporal block is a joint refinement block, containing a hypergraph-guided frequency decoupling module and an energy-driven sparse temporal attention module; the preprocessing layer is used to perform channel expansion and positional encoding injection on the skeleton sequence feature representation to generate initial spatiotemporal features; based on the initial spatiotemporal features, the first to ten spatiotemporal blocks are processed by the HFM-Net network. The seventh spatiotemporal block extracts local spatiotemporal features. Based on these features, static structural and dynamic motion features are extracted from the spatiotemporal features through the hypergraph-guided frequency decoupling modules of the eighth, ninth, and tenth spatiotemporal blocks. Structure-guided and motion-guided hypergraphs are constructed respectively, and spatial features are aggregated through frequency decoupling using these hypergraphs to obtain decoupled spatial enhancement features. Based on these spatial enhancement features, the energy-driven sparse temporal attention module of the tenth spatiotemporal block is used to adaptively guide sparse temporal interactions using the frame-level motion energy signal output by the hypergraph-guided frequency decoupling module to obtain temporal enhancement features. Based on these temporal enhancement features, a classifier is used to obtain action recognition results.

[0086] According to embodiments of the present invention, the human motion recognition system based on energy-driven and hypergraph-guided frequency decoupling can correspond to the execution of the method described in the embodiments of the present invention, and the above and other operations and / or functions of each module of the human motion recognition system based on energy-driven and hypergraph-guided frequency decoupling are respectively for implementing Figure 1 , Figure 2 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0087] See Figure 7 The diagram shows the structure of a computer device, which includes a processor, a communication interface, and a computer-readable storage medium. The processor, communication interface, and computer-readable storage medium are connected via a bus or other means. The communication interface is used to receive and send data. The computer-readable storage medium can be stored in the computer device's memory. The computer-readable storage medium stores computer programs, including program instructions, and the processor executes the program instructions stored in the computer-readable storage medium. The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device, adapted to implement one or more instructions, specifically adapted to load and execute one or more instructions to implement the corresponding steps in the embodiment of the human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling.

[0088] This embodiment provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device.

[0089] Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0090] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-described embodiment of the human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling.

[0091] This embodiment provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding steps in the above-described embodiment of the human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling.

[0092] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0093] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0097] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling, characterized in that, include: Extract the human skeleton structure from the acquired video data and convert it into a human skeleton graph structure representation, then extract the skeleton sequence feature representation; Based on the skeleton sequence feature representation, an HFM-Net network is used to obtain action recognition results. The HFM-Net network includes a preprocessing layer, ten sequentially stacked spatiotemporal blocks, and a classifier. Each of the first to seventh spatiotemporal blocks contains a graph convolution module and a temporal convolution module. The eighth and ninth spatiotemporal blocks are both hypergraph-guided frequency decoupling modules. The tenth spatiotemporal block is a joint refinement block, containing a hypergraph-guided frequency decoupling module and an energy-driven sparse temporal attention module. The preprocessing layer is used to perform channel expansion and positional encoding injection on the skeleton sequence feature representation to generate initial spatiotemporal features. Based on the initial spatiotemporal features, the first to seventh spatiotemporal blocks are used to extract... Local spatiotemporal features are extracted. Based on these features, static structural features and dynamic motion features are extracted from the spatiotemporal features through the hypergraph-guided frequency decoupling module of the eighth, ninth, and tenth spatiotemporal blocks. Structure-guided hypergraphs and motion-guided hypergraphs are constructed respectively. Spatial features are aggregated through frequency decoupling using the structure-guided and motion-guided hypergraphs to obtain decoupled spatial enhancement features. Based on these spatial enhancement features, an energy-driven sparse temporal attention module of the tenth spatiotemporal block is used. The frame-level motion energy signal output by the hypergraph-guided frequency decoupling module is adaptively guided to guide sparse temporal interaction to obtain temporal enhancement features. Based on these temporal enhancement features, a classifier is used to obtain the action recognition results.

2. The human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling according to claim 1, characterized in that, The method involves extracting static structural features and dynamic motion features from spatiotemporal features, and constructing structure-guided hypergraphs and motion-guided hypergraphs respectively; the method includes: Based on local spatiotemporal features, the normalized frame difference between adjacent frames is extracted to obtain the motion change tensor. The static structural features are obtained by averaging the local spatiotemporal features along the time dimension; The dynamic motion characteristics are obtained by averaging the motion change tensor along the time dimension. The joint activation is obtained by averaging the motion change tensor along the channel and time dimensions and then normalizing it joint by joint. Based on static structural features and joint activation, a structure-guided hypergraph correlation matrix is ​​constructed. Based on the splicing of static structural features and dynamic motion features and joint activation, a motion-guided hypergraph correlation matrix is ​​constructed.

3. The human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling according to claim 2, characterized in that, The spatial feature aggregation through frequency decoupling using structure-guided hypergraphs and motion-guided hypergraphs yields decoupled spatial enhanced features; the method includes: By multiplying the transposes of the structure-guided hypergraph incidence matrix and the motion-guided hypergraph incidence matrix and row normalization respectively, the structure-guided higher-order topological matrix and the motion-guided higher-order topological matrix are induced. The motion change tensor is averaged along the channel and joint dimensions and then normalized frame by frame to obtain the frame-level motion energy signal; Using frame-level motion energy signals, motion branch modulation gating is calculated to modulate the motion change tensor with energy, thereby obtaining motion energy characteristics; The structure-space attention matrix is ​​learned from local spatiotemporal features using a structural branch, and combined with a structure-guided higher-order topological matrix to output the structure-space attention features. These features are then aggregated with the local spatiotemporal features to obtain the structure aggregation matrix. The motion-space attention matrix is ​​learned from motion energy features using a motion branch, and combined with a motion-guided higher-order topological matrix to output the motion-space attention features. These features are then aggregated with the motion energy features to obtain the motion aggregation matrix. The spatial augmentation features are obtained by weighted fusion of the structural aggregation matrix, the motion aggregation matrix, and the residual branch output features based on the physical skeleton connection graph.

4. The human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling according to claim 1, characterized in that, The structure-guided hypergraph association matrix is ​​expressed by the following formula: The motion-guided hypergraph correlation matrix is ​​expressed by the following formula: in, , The number of superedges. The number of joints is represented by a row, where each row indicates the membership degree of a joint on each hyperedge. and For learnable linear projection, It is a static structural feature. It is a dynamic motion characteristic. For joint activation, is the numerical stability constant.

5. The human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling according to claim 1, characterized in that, The method, based on spatial enhancement features, employs an energy-driven sparse temporal attention module of the tenth spatiotemporal block. It adaptively guides sparse temporal interaction through the frame-level motion energy signal output by the hypergraph-guided frequency decoupling module to obtain temporal enhancement features. The method includes: Spatial enhancement features are applied to temporal convolution to obtain local temporal features. Local temporal features are aggregated along the joint dimension to obtain temporal tokens. Conditional positional encoding is applied, and adjacent frame context is injected into each temporal token through depthwise separable temporal convolution and layer normalization. Finally, the basic temporal attention score matrix is ​​calculated. The frame-level motion energy signal output by the supergraph guiding frequency decoupling module is zero-padded and aligned in the first frame to obtain the frame-level motion energy alignment signal, and then the frame-level motion energy alignment signal is converted into time motion gating; The temporal motion gating is used as a key-side saliency bias and superimposed on the basic temporal attention score matrix to obtain the modulation score matrix; Based on the frame-level motion energy alignment signal, linear interpolation is performed between the preset minimum attention budget and the maximum attention budget to obtain the attention neighborhood size corresponding to each query frame; For each query frame, in the modulation score matrix In the row, select the one with the highest score other than itself. Each non-self frame, together with itself, constitutes the sparse temporal neighborhood of the query frame; Softmax normalization is performed only within the sparse temporal neighborhood to obtain the sparse temporal attention matrix, and the attention weights of unselected frames are reset to zero; Aggregate temporal context based on sparse temporal attention matrix and adjacent frame context; The aggregated temporal context is extended back to the joint dimension via broadcasting and then residually fused with the local temporal features to obtain temporally enhanced features.

6. The human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling according to claim 1, characterized in that, The loss of the HFM-Net network is: in, Cross-entropy classification loss, The weighting coefficient for consistency loss, This is due to deep consistency loss. The symbol represents the network spatiotemporal block index, and the summation symbol indicates that a consistency loss is applied to all spatiotemporal blocks from the eighth to the tenth.

7. A human motion recognition system based on energy-driven and hypergraph-guided frequency decoupling, characterized in that, include: The feature extraction module is configured to: extract the human skeleton structure from the acquired video data, convert it into a human skeleton graph structure representation, and extract the skeleton sequence feature representation; The action recognition module is configured to: obtain action recognition results based on the skeleton sequence feature representation using an HFM-Net network; the HFM-Net network includes: a preprocessing layer, ten sequentially stacked spatiotemporal blocks, and a classifier; wherein, each of the first to seventh spatiotemporal blocks contains a graph convolution module and a temporal convolution module; the eighth and ninth spatiotemporal blocks are both hypergraph-guided frequency decoupling modules; the tenth spatiotemporal block is a joint refinement block, containing a hypergraph-guided frequency decoupling module and an energy-driven sparse temporal attention module; the preprocessing layer is used to perform channel expansion and positional encoding injection on the skeleton sequence feature representation to generate initial spatiotemporal features; based on the initial spatiotemporal features, through the first to seventh spatiotemporal blocks, each spatiotemporal block contains a graph convolution module and a temporal convolution module; the eighth and ninth spatiotemporal blocks are both hypergraph-guided frequency decoupling modules; the tenth spatiotemporal block is a joint refinement block, containing a hypergraph-guided frequency decoupling module and an energy-driven sparse temporal attention module; the preprocessing layer is used to perform channel expansion and positional encoding injection on the skeleton sequence feature representation to generate initial spatiotemporal features; based on the initial spatiotemporal features, the first to ten spatiotemporal blocks are processed by the HFM-Net network. The seventh spatiotemporal block extracts local spatiotemporal features. Based on these features, static structural and dynamic motion features are extracted from the spatiotemporal features through the hypergraph-guided frequency decoupling modules of the eighth, ninth, and tenth spatiotemporal blocks. Structure-guided and motion-guided hypergraphs are constructed respectively, and spatial features are aggregated through frequency decoupling using these hypergraphs to obtain decoupled spatial enhancement features. Based on these spatial enhancement features, the energy-driven sparse temporal attention module of the tenth spatiotemporal block is used to adaptively guide sparse temporal interactions using the frame-level motion energy signal output by the hypergraph-guided frequency decoupling module to obtain temporal enhancement features. Based on these temporal enhancement features, a classifier is used to obtain action recognition results.

8. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and execute the steps of the human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps in the human motion recognition method based on energy-driven and hypergraph-guided frequency decoupling as described in any one of claims 1-6.

Images