A spatio-temporal decoupled human behavior recognition method, device and equipment based on dynamic semantic guidance mask

Abstract

The application discloses a kind of spatio-temporal decoupling human behavior recognition method, device and equipment based on dynamic semantic guide mask, comprising: extracting human skeleton sequence data from original video and carrying out data enhancement;Based on dynamic semantic guide mask mechanism, the skeleton sequence after data enhancement is masked in operation in two dimensions of space and time;Skeleton sequence data after masking operation is sent into query encoder and momentum encoder respectively, spatial representation and time representation are obtained respectively, cross-domain contrast loss is constructed, contrast learning training is carried out to obtain human behavior recognition model;The video to be identified is input into human behavior recognition model, and the prediction result of human behavior in the video to be identified is obtained.The spatio-temporal decoupling human behavior recognition method, device and equipment based on dynamic semantic guide mask of the application significantly improve human behavior recognition precision and environmental adaptability without manual label.

Description

Technical Field

[0001] This invention relates to the field of computer vision and neuromorphic computing, and in particular to a spatiotemporal decoupled human behavior recognition method, apparatus and device based on dynamic semantic guidance mask. Background Technology

[0002] Human behavior recognition, as a key technology for artificial intelligence systems to understand the physical world, is gradually expanding from controlled experimental environments to the open real world, and is being widely used in fields such as motion interaction, security monitoring, and rehabilitation assistance.

[0003] In recent years, the rise of the human-machine collaborative computing paradigm has driven the development of "embodied intelligence." This paradigm simulates human perception and decision-making mechanisms through brain-like neuromorphic computing, providing a new framework for interpretable behavior analysis that incorporates spatiotemporal dynamic characteristics. Simultaneously, the introduction of cognitive ergonomics has promoted semantic understanding and safety assessment of human movement, enabling action recognition to no longer rely solely on low-level motion features but to integrate high-level semantic cues and dynamic behavioral patterns.

[0004] In terms of data modalities, current research is dedicated to overcoming the limitations of traditional visual methods. With the widespread adoption of depth sensors, skeleton data, with its advantages of lightweight design, strong privacy protection, and precise joint localization, has become a mainstream data source for advanced applications such as action recognition, skill transfer, and motion prediction. In understanding human behavior and modeling robot action cognition, the biomimetic properties of skeletons effectively combine ergonomic posture parameters with motion dynamics characteristics. Notably, breakthroughs in self-supervised learning have greatly advanced research on behavior recognition in poorly labeled scenarios, which is the direct motivation for this research to focus on self-supervised learning methods. By pre-setting tasks, the model can autonomously learn meaningful feature representations from unlabeled data.

[0005] AimCLR is a self-supervised action recognition method based on contrastive learning, which improves model robustness through extreme data augmentation and feature dropout modules. AimCLR performs skeletal self-supervision through random view transformations and global contrastive learning, but its masking and cropping are decoupled from semantics, making it prone to missing key information when encountering joint occlusion or noise insertion, resulting in insufficient robustness. MaskCLR is an attention-guided contrastive learning framework that uses a probabilistic masking strategy to occlude key limb parts, forcing the model to learn to rely on other motion features for action classification, thereby enhancing its adaptability to pose estimation errors and subtle motion changes. MaskCLR uses attention-guided probabilistic masks and multi-level contrastive learning to improve noise resistance, but it only focuses on spatially significant joints, lacks temporal dynamic adaptation, and the mask strength needs to be manually set, failing to accommodate different action and noise scenarios.

[0006] However, real-world environments are rife with interference factors, such as camera occlusion, sensor noise, and frame drops, leading to insufficient robustness and significant performance fluctuations in existing behavior recognition methods. Therefore, a behavior recognition method with stronger robustness in complex dynamic scenes is needed. This invention constructs a dynamic spatiotemporal mask using "joint degree centrality + inter-frame motion difference," and combines this with space-time decoupling coding and cross-domain contrastive learning to cover key joints and key frames while adapting to noise, significantly improving the robustness of behavior recognition in complex dynamic scenes. Summary of the Invention

[0007] This invention provides a spatiotemporally decoupled human behavior recognition method based on dynamic semantic guidance mask, which significantly improves the accuracy and environmental adaptability of human behavior recognition without the need for manual labeling.

[0008] The technical solution of the present invention is as follows:

[0009] A spatiotemporal decoupling human behavior recognition method based on dynamic semantic guidance masks includes the following steps:

[0010] (1) Extract human skeleton sequence data from the original video and perform data augmentation on the skeleton sequence data;

[0011] (2) Based on the dynamic semantic guidance masking mechanism, the data-enhanced skeleton sequence is masked in both spatial and temporal dimensions. In the spatial dimension, the masking degree centrality of the joints is higher than the threshold, and in the temporal dimension, the frames with significant motion changes are masked.

[0012] (3) The skeleton sequence data after the masking operation is sent to the query encoder and the momentum encoder respectively. Both the query encoder and the momentum encoder include a feature extraction unit and a spatiotemporal parallel decoupling unit to obtain spatial representation and temporal representation respectively. Cross-domain contrastive loss is constructed and contrastive learning training is performed.

[0013] A human behavior recognition model is obtained after training;

[0014] (4) Input the video to be identified into the human behavior recognition model to obtain the prediction results of human behavior in the video to be identified.

[0015] To address the shortcomings of existing skeletal motion recognition methods, such as "random mask information leakage, spatial-temporal feature fragmentation, and insufficient robustness," this invention proposes a spatiotemporally decoupled human behavior recognition method based on dynamic semantic-guided masks. This method adaptively constructs a mask sampling pool based on the degree centrality of joints in the spatial dimension and on the motion difference between frames in the temporal dimension. Combined with a parallel decoupled encoder and cross-domain contrastive learning loss, it significantly improves recognition accuracy and environmental adaptability without the need for manual labeling. This method can be deployed in scenarios such as fall detection in elderly care and embodied monitoring robots.

[0016] In step (1), data augmentation of the skeletal sequence data includes: performing spatial rotation, mirror flipping, or shearing on the skeletal sequence.

[0017] In step (2), the masking operation on the data-enhanced skeletal sequence in the spatial dimension based on the dynamic semantic guidance masking mechanism includes:

[0018] (2-1i) Calculate the degree centrality of each joint in each frame of the action sequence. Defined as: ,in Represents a node The number of edges connected;

[0019] (2-1ii) Add each node i to the set according to its degree centrality: N represents the total number of key points in a single frame;

[0020] (2-1iii) From A fixed number of joints are sampled and masked.

[0021] In step (2-1iii), the number of sampling masked joints is set manually as needed. Joints with higher degree centrality are repeatedly included in the sampling pool more often, thereby ensuring that the masking operation prioritizes covering key structural regions in the skeletal diagram (such as core joints like the shoulder, elbow, and hip). By masking these high-centrality nodes, the model is guided to focus on the semantic features of low-interest regions, thereby enhancing spatial generalization ability.

[0022] In step (2), the masking operation on the data-enhanced skeletal sequence in the time dimension based on the dynamic semantic guidance masking mechanism includes:

[0023] (2-2i) Calculate the attention value at each time step in the skeletal sequence. :

[0024] ;

[0025] in, Number of channels; This refers to the number of joints. For the number of people; For the t-th frame, the p-th character, and the... Joints, First Motion attention values ​​for each coordinate channel; Motion attention values Given a skeletal sequence , Indicates the first t The complete skeleton of frame t, the inter-frame motion difference of frame t. Defined as: ;

[0026] (2-1ii) Screening from the skeletal sequence It is masked by a minimum fixed number of frames.

[0027] In step (2-1ii), the frames in the skeleton sequence are arranged according to... Sort the frames from smallest to largest, and select the first F frames for masking. The number of frames F to be masked can be set according to specific needs.

[0028] Based on the above calculations, frames with significant motion changes correspond to... Lower values ​​mean these frames are considered less important and are removed during masking; while the most stable frames with the least change correspond to... A higher value indicates a frame that is considered important.

[0029] By masking the skeleton sequence with a dynamic semantic-guided masking mechanism, content that is highly related to action semantics is hidden, guiding the model to actively explore and learn other potential features that are not hidden, thereby obtaining richer and more interference-resistant action expressions.

[0030] Step (3) includes:

[0031] (3-1) Extract the skeleton sequence after masking operation through the feature extraction unit. intermediate spacetime representation ;

[0032] (3-2) Representation of intermediate spacetime Perform dimensional transformation to obtain preliminary spatial features. and preliminary time series characteristics Preliminary spatial features Projection converted into spatial embedding Preliminary time series characteristics Projection to Temporal Embedding ;

[0033] (3-3) Further refinement of the decoupled spatial embedding based on the self-attention mechanism module and the feedforward module and time embedding The spatial representations are obtained respectively. and time representation ;

[0034] (3-4) The query encoder uses projection to decouple the spatial representation. and time representation Transform them into the final spatial representation respectively and final time representation ,Will and Integration introduces a global perspective representation ;

[0035] The momentum encoder represents the decoupled spatial representation through projection. and time representation Transform them into the final spatial representation respectively and final time representation ,Will and Integration introduces a global perspective representation ;

[0036] (3-5) Construct a quadruple-level contrastive loss function structure and perform contrastive learning training:

[0037] ;

[0038] , , and These are the weighting coefficients for the four-way losses; Indicates comparative loss;

[0039] During training, the parameters of the query encoder are updated through backpropagation, and the parameters of the momentum encoder are updated through the momentum strategy.

[0040] The feature extraction units for the query encoder and momentum encoder are graph convolutional neural networks (GCN).

[0041] In step (3-1), for each joint point Its output characteristics for:

[0042] ;

[0043] Indicates the normalization factor; Key points Neighboring joints (referring to joint points) The set of directly connected joints, i.e., the skeleton, is modeled as a joint-connection graph, where each joint is a node and the physiological connections between joints are edges. Key points The feature vectors of the neighboring nodes; This is a weight matrix that changes over time; It is a nonlinear transformation matrix; It is the ReLU activation function; This is the weighting function.

[0044] In step (3-2), the preliminary spatial features are... Projection converted into spatial embedding The formula is:

[0045] ;

[0046] in, , It is a trainable weight matrix. , It is a partial vector. Indicates the activation function;

[0047] Preliminary time series characteristics Projection to Temporal Embedding The formula is:

[0048] ;

[0049] in, , It is a trainable weight matrix. , It is a partial vector. This represents the activation function.

[0050] Step (3-3) includes:

[0051] (3-3i) Embedding space Projected onto the query, key, and value spaces, the attention score between joints is calculated by scaling the dot product, and then normalized using Softmax:

[0052] ;

[0053] This represents the output feature matrix of the i-th attention head in the spatial dimension; , and These are the query, key, and value matrices obtained through linear transformation. It is the scaling factor;

[0054] Compare the spatial attention results with... By combining residual connections and layer normalization, feature stability is enhanced:

[0055] ;

[0056] This is the refined spatial feature matrix; This is the concatenated attention matrix; This is processed as a fully connected layer; It is a layer normalization function; This indicates the number of attention heads in the spatial dimension;

[0057] ;

[0058] For layer normalization function; It is a feedforward network. This is a max pooling operation;

[0059] (3-3ii) Embedding time Projected onto the query, key, and value spaces, the attention score between joints is calculated by scaling the dot product, and then normalized using Softmax:

[0060] ;

[0061] This represents the output feature matrix of the i-th attention head in the time dimension; , and These are the query, key, and value matrices obtained through linear transformation. It is the scaling factor;

[0062] Compare the spatial attention results with... By combining residual connections and layer normalization, feature stability is enhanced:

[0063] ;

[0064] This is the refined spatial feature matrix; This is the concatenated attention matrix; This is processed as a fully connected layer; It is a layer normalization function; This indicates the number of attention heads over time.

[0065] ;

[0066] For layer normalization function; It is a feedforward network. This is a max pooling operation.

[0067] In steps (3-5), the InfoNCE framework is used to calculate the sample contrast loss. For any contrast pair... u and v The loss is defined as:

[0068] ;

[0069] in, for u and v Similarity score, , For temperature parameters, For two feature vectors u andv The inner product; Indicates the negative sample set In the middle, calculation u With all negative samples The similarity scores between them are then summed exponentially.

[0070] The present invention also provides a spatiotemporally decoupled human behavior recognition device based on dynamic semantic guidance mask, comprising:

[0071] The data processing module extracts human skeletal sequence data from the original video and performs data augmentation on the skeletal sequence data;

[0072] The dynamic semantic guidance masking module performs masking operations on the data-enhanced skeleton sequence in both spatial and temporal dimensions based on the dynamic semantic guidance masking mechanism. In the spatial dimension, it masks joints with a degree centrality higher than the threshold, and in the temporal dimension, it masks frames with significant motion changes.

[0073] The contrastive learning training module feeds the masked skeletal sequence data into the query encoder and momentum encoder respectively. Both the query encoder and momentum encoder include a feature extraction unit and a spatiotemporal parallel decoupling unit to obtain spatial and temporal representations respectively. A cross-domain contrastive loss is constructed and contrastive learning training is performed. After training, a human behavior recognition model is obtained.

[0074] The human behavior recognition module inputs the video to be recognized into the human behavior recognition model to obtain the prediction results of human behavior in the video.

[0075] The present invention also provides a spatiotemporally decoupled human behavior recognition device based on dynamic semantic guidance mask, including a memory, a processor, and a computer program stored in the memory and executable on the processor;

[0076] The memory stores a human behavior recognition model trained by the above method; the video to be recognized is input into the human behavior recognition model, and the processor executes the computer program to predict the human behavior in the video to be recognized.

[0077] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0078] (1) Significantly enhanced robustness: It maintains high accuracy under extreme perturbations such as coordinate jitter, frame insertion / deletion, and occlusion, and there is no crash phenomenon in the case of complex perturbation.

[0079] (2) High information utilization: Based on dynamic semantic masking, the model is effectively guided to discover semantic clues in low-interest regions and stable frames, avoiding random masking from destroying structural information.

[0080] (3) Parallel decoupled architecture is more discriminative: Spatial-temporal domain learning avoids the fuzzy processing of topological and temporal dependencies in traditional coupled models, and further enhances feature expression by combining cross-layer fusion, resulting in higher recognition accuracy.

[0081] (4) No manual annotation required, cross-platform friendly: The entire chain of self-learning from data augmentation to model optimization reduces the dependence on manual annotation, lowers data costs, and improves adaptability to new or less annotated scenarios. Attached Figure Description

[0082] Figure 1 This is a flowchart of a spatiotemporal decoupling human behavior recognition method based on dynamic semantic guidance masks.

[0083] Figure 2 This is a schematic diagram of a spatiotemporal decoupling human behavior recognition device based on dynamic semantic guidance masks. Detailed Implementation

[0084] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0085] The technical solution provided by this invention is a method and system for robust human behavior recognition, which combines a dynamic semantic-guided masking strategy, self-supervised contrastive learning, and a spatiotemporal decoupling network for human behavior recognition.

[0086] The specific process of the spatiotemporal decoupling human behavior recognition method based on dynamic semantic guidance mask of this invention is as follows: Figure 1 As shown, it includes the following steps:

[0087] S101 raw video frame input and bone sequence extraction

[0088] Extracting bone sequences from raw video frames using a pose estimator S .

[0089] S102 Data Augmentation Generates Dual Views

[0090] For skeletal sequences S Data augmentation by performing spatial rotation, mirror flipping, and shearing yields two positive samples.

[0091] Data augmentation is used to augment the skeletal sequence. S Randomly transformed into different enhanced versions, these are considered positive sample pairs. For skeletal data, this invention employs spatial rotation, mirror flipping, and shearing affine enhancement strategies.

[0092] (1) Spatial rotation based on skeletal sequence S The TTaking a frame as an example, its joint set can be represented as , V T For skeletal sequence S The T The complete set of joints in a frame, which contains the three-dimensional coordinates of all joints in that frame; V T，j For the first T The first frame j The specific coordinates of each joint; J This represents the total number of joints contained in each bone in a single frame.

[0093] By using Euler quaternions, rotate counterclockwise around the x, y, and z axes to an angle. α T , β T and γ T Generate new coordinates.

[0094] (2) Mirror flip enhancement is achieved by reversing the skeletal sequence. S The left and right structures are used to introduce mirror symmetry. For skeletal sequences S Each frame in T Joint assembly The reflection is processed along the sagittal plane. During this process, the coordinates of the left and right joints are interchanged, while the joints located at the midline remain in their original positions.

[0095] (3) Shear enhancement is a linear transformation applied to the three-dimensional coordinate space of a body joint, which causes the joint structure to tilt at random angles. Shear transformation matrix ,in The remaining positions are randomly set shearing factors.

[0096] S103 Dynamic Semantic Guidance Mask

[0097] (1) Calculation of spatial mask weights

[0098] Spatial mask weights calculate the degree centrality of each joint, outputting a probability distribution of joint importance. In the spatial dimension, the degree centrality metric measures the importance of joints, based on node connectivity. The definition is as follows: ,in Represents a node The number of edges connected.

[0099] To construct a dynamic spatiotemporal masking sampling pool, this invention uses joint degree centrality as a spatial importance index to perform importance-based weighted sampling of nodes in the skeletal graph structure. Specifically, firstly, the degree centrality of each joint in the entire action sequence (i.e., the connection strength between the node and other nodes) is calculated, reflecting its topological importance in the spatial dimension; then, the sampling probability of each node is determined based on the degree centrality value. Joints with higher degree centrality are repeatedly included in the sampling pool more often, thereby ensuring that the masking operation preferentially covers key structural regions in the skeletal graph (such as core joints like the shoulder, elbow, and hip). Each node i is added to the set according to its centrality value: A joint The centrality indicates a multiset, meaning that the higher the centrality, the greater the probability of sampling. Centrality is converted into sampling probability, eliminating the need for explicit normalization; N represents the total number of joints in a single frame.

[0100] (2) Application of spatial mask

[0101] final By sampling a fixed number of key points k, a mask set is constructed: The mask needs to be k nodes (nodes are placed at infinity), J k This represents the final selected subset of k joint indices; j represents J. k Local index variables. By masking these highly central nodes, the model is guided to focus on the semantic features of low-interest regions, thereby enhancing spatial generalization ability.

[0102] (3) Time mask calculation

[0103] The temporal mask weights are used to calculate the motion difference between adjacent frames and output the frame mask probability distribution.

[0104] In the temporal dimension, frames with intense motion (such as leg swings during running or rapid arm movements during waving) typically contain a large amount of transient information, but are also susceptible to noise and sensor jitter. By preferentially masking these frames, the model is forced to rely on more consistent motion patterns in stable frames (such as torso posture during running or shoulder position during waving) to learn action semantics, thereby enhancing robustness to temporal disturbances.

[0105] To extract temporal motion information from sequence data, dynamic weights are calculated based on inter-frame motion differences, and the motion attention distribution is obtained by calculating the square of the absolute value of the inter-frame differences. Given a skeletal sequence... , Indicates the first t Frame 1 complete skeleton; 1st t The inter-frame motion difference is defined as The distribution of motor attention is defined as follows: By averaging across all channels, joints, and individual human bodies, the first...t Overall motion attention value of the frame (Frame-level dynamic motion attention):

[0106] ;

[0107] in, For the number of channels, For the number of joints, For the number of people, For the first t Frame, First p The first person, the Joints, First Local motion attention values ​​for each coordinate channel. , , These represent the coordinate channel index, joint index, and character index, respectively.

[0108] The specific calculation method is as follows: first, calculate all channels (totaling 100,000 channels). ), all joints (total number of ) and all personnel (total number of The corresponding local motion attention value The summation is then performed, and the result is divided by the product of the number of channels, the number of joints, and the number of personnel to obtain the attention value at each time step. This calculation method allows frames with significant motion changes to correspond to... Lower values ​​mean these frames are considered less important; while the most stable frames with the least change correspond to... A higher value indicates a frame that is considered important.

[0109] (4) Apply time mask

[0110] Filter out The smallest F-frame (i.e., the high-motion frame, where F is a manually defined value) is masked (deleted). This mechanism guides the model to strengthen the stable temporal features in static frames by masking highly dynamic frames.

[0111] Therefore, a robust feature representation method is adopted. This method introduces a dynamic semantic-guided masking strategy during the training phase to mask the content in the skeletal action sequence that is highly related to the semantics of the action. This guides the model to actively explore and learn other potential features that are not masked, thereby obtaining a richer and more interference-resistant action expression.

[0112] S104 trains the recognition model based on contrastive learning.

[0113] (1) The present invention employs parallel spatiotemporal decoupled feature extraction, wherein the graph convolutional network (GCN) is used to capture spatial structure information and temporal dynamic features, while the self-attention mechanism focuses on temporal dynamic features.

[0114] Network for enhanced input skeleton (C×T×V) is processed to generate an intermediate spacetime representation. (C1×T×V). For each target node Its output characteristics By aggregating its domain joints (Refers to the target node) The set of directly connected nodes, i.e., the skeleton, is modeled as a joint-connection graph, where each joint is a node, and the physiological connections between joints are adjacent nodes within an edge. The features were calculated.

[0115] ;

[0116] Specifically, it means the target node. At any moment Output characteristics This is equivalent to its domain All adjacent nodes The features are used to perform a weighted summation. Among them, These are the feature vectors of adjacent nodes; It is a nonlinear transformation performed on the features of adjacent nodes; It is a weight matrix that changes over time and is used to capture dynamic information over time; It is the ReLU activation function; It is a weighting function used to evaluate the importance of the transformed features; Used to normalize the contribution of the aggregation.

[0117] In each training step, the input data is split into two branches, namely two augmentation sequences: query data and input data. and key data These two sequences are considered positive samples and mapped to the embedding space using a mapping function. The embedding of the query data is represented as follows: The embedded representation of key data is as follows .

[0118] Supporting efficient contrastive learning (bringing positive samples closer and pushing negative samples further away), the model introduces a dynamic queue to store negative sample features and uses a momentum update strategy to stabilize the key encoder parameters. The queue stores historical key features k as negative samples (samples that do not belong to the same action as the current query feature). To avoid training instability caused by overly frequent key encoder parameter updates, a momentum strategy is used to update its parameters. After each training round, new key features are added to the queue, while the oldest features are removed, keeping the queue size fixed. During optimization, the query encoder is updated only through backpropagation, while the key encoder is updated using a momentum-based strategy.

[0119] ;

[0120] in, and These represent the parameters for the key encoder and query encoder, respectively. Key encoder parameters Update to original parameters Multiply by momentum coefficient and Multiply by the encoder parameters The sum. Updated only through backpropagation. The parameters, this momentum update method makes The evolution ratio Smoother.

[0121] Intermediate spatiotemporal representation extracted in the preliminary stage A dimensional transformation is performed on (C1×T×V) to obtain preliminary spatial features. and preliminary time series characteristics Preliminary spatial features Projection is transformed into spatial embedding at a higher semantic level, as shown in the formula:

[0122] ;

[0123] in, Indicates spatial embedding; The initial spatial features obtained after aggregation by a graph convolutional network (GCN) include the topological relationships between joints and local spatial dependencies. , It is a trainable weight matrix. , It is a partial vector. This represents the activation function.

[0124] The core of temporal feature processing, its design logic and spatial embedding This creates a symmetrical correspondence, while also making targeted optimizations for the dynamic characteristics of the time dimension:

[0125] ;

[0126] in, Indicates timing embedding, It is a preliminary temporal feature obtained through preliminary feature extraction in the time dimension, which includes basic temporal information such as the inter-frame motion amplitude, speed change, and periodic pattern of the action. , It is a trainable weight matrix. , It is a partial vector. This represents the activation function.

[0127] During feature extraction, spatiotemporal interactions are intertwined: Focus on the spatial topology (static structure) of joints. Focusing on the temporal dynamics (dynamic changes) of actions, the model forms a complex information structure through cross-domain comparative learning, jointly supporting the complete expression of action semantics. To further refine the decoupled spatial and temporal features, the model introduces a self-attention mechanism and multi-layer processing (layer normalization, feedforward network) to improve the discriminativeness of features.

[0128] Spatial self-attention mechanism:

[0129] Spatial self-attention is used to uncover nonlocal dependencies between human joints. Spatial embedding extracted from GCN is then applied. (V represents the total number of joints, C represents the total number of channels) Projected onto the query ( ),key( ),value( Three spaces.

[0130] Attention scores between joints are calculated by scaling the dot product, and then normalized using Softmax.

[0131] ;

[0132] in, This represents the output feature matrix of the i-th attention head; , and It is a query, key, and value matrix obtained through linear transformation. middle Indicates transpose. It is the scaling factor.

[0133] Compare the spatial attention results with... By combining residual connections and layer normalization, feature stability is enhanced:

[0134] ;

[0135] Among them, the refined spatial feature matrix By analyzing the spliced ​​attention matrix After the fully connected layer After processing, with spatial embedding The result is obtained by adding the ingredients together and then performing layer normalization. Indicates the number of heads of attention;

[0136] ;

[0137] in, It is a layer normalization function. It is a feedforward network, and the max pooling operation further enhances the spatial representation. .

[0138] Temporal self-attention mechanism:

[0139] Temporal self-attention is used to capture long-range dependencies of actions over time, resolving temporal patterns of actions. Its design is symmetrical to spatial self-attention but emphasizes temporal dynamics.

[0140] Temporal features extracted from GCN (T is the total number of frames) Projected to the query ( ),key( ),value( Three spaces.

[0141] Attention scores between joints are calculated by scaling the dot product, and then normalized using Softmax.

[0142] ;

[0143] in, This represents the output feature matrix of the i-th attention head; , and It is a query, key, and value matrix obtained through linear transformation. middle Indicates transpose. It is the scaling factor.

[0144] After spatiotemporal feature aggregation using GCN, the resulting temporal embedding along the time dimension is denoted as... First, the attention matrix in the time dimension is processed by concatenating the multi-head attention matrices, and then passed through a fully connected layer. Perform feature transformation, and then combine with temporal embedding. Perform residual connections and layer normalization. The operation yielded preliminary refined time characteristics. :

[0145] ;

[0146] To further improve the discriminative power of time features, the initially refined time features will be... The input is fed into the feedforward network in the time dimension. The processing is performed within this feedforward network, which consists of a multilayer perceptron and is capable of performing more complex nonlinear transformations on temporal features. Residual connections are performed, layer normalization is applied, and finally, global average pooling in the time dimension is used to compress the time dimension, resulting in the final refined and enhanced time features. :

[0147] .

[0148] (2) Cross-domain contrast loss function

[0149] To utilize the spatial features after decoupling and time characteristics The query encoder obtains the final representation through projection. and .Will and Fusion to introduce a global perspective representation Similarly, the key encoder obtains the final representation of spatial and temporal features through projection. and .Will and Integration, .

[0150] To quantify the differences between these representations, a four-tuple-level contrastive loss function structure is formed, combining momentum encoding with a dynamic queue of negative samples:

[0151] ;

[0152] , , and These are the weighting coefficients for the four loss terms. These coefficients are set to balance the contribution of each loss term. This indicates a comparative loss.

[0153] For skeletons extracted from different videos, negative sample pairs are formed due to their different content, and their similarity in the latent space should be minimized. To this end, we use the InfoNCE framework to calculate the sample contrast loss. For any contrast pair... u and v The loss is defined as:

[0154] ;

[0155] Where H represents the set of negative samples; , It's a temperature parameter. It calculates the inner product of two eigenvectors, which measures the similarity between the two vectors in a direction. Indicates the negative sample set In the middle, calculation u With all negative samples The similarity scores between them are calculated and summed exponentially.

[0156] S105 Model Application

[0157] The video to be identified is input into the trained recognition model to obtain the prediction results of human behavior in the video.

[0158] like Figure 2 As shown, the present invention also provides a spatiotemporally decoupled human behavior recognition device based on dynamic semantic guidance mask, comprising:

[0159] The data processing module extracts human skeletal sequence data from the original video and performs data augmentation on the skeletal sequence data;

[0160] The dynamic semantic guidance masking module performs masking operations on the data-enhanced skeletal sequence in both spatial and temporal dimensions based on the dynamic semantic guidance masking mechanism. It masks joints with a degree centrality higher than the threshold in the spatial dimension and frames with significant motion changes in the temporal dimension.

[0161] The contrastive learning training module feeds the masked skeletal sequence data into the query encoder and momentum encoder respectively. Both the query encoder and momentum encoder include a feature extraction unit and a spatiotemporal parallel decoupling unit to obtain spatial and temporal representations respectively. A cross-domain contrastive loss is constructed and contrastive learning training is performed. After training, a human behavior recognition model is obtained.

[0162] The human behavior recognition module inputs the video to be recognized into the human behavior recognition model to obtain the prediction results of human behavior in the video.

[0163] The present invention also provides a spatiotemporally decoupled human behavior recognition device based on dynamic semantic guidance mask, including a memory, a processor, and a computer program stored in the memory and executable on the processor;

[0164] The memory stores the human behavior recognition model obtained by training using the above method; the video to be recognized is input into the human behavior recognition model, and the processor executes a computer program to predict the human behavior in the video to be recognized.

[0165] To verify the stability and anti-interference capability of this invention in a real home-based elderly care environment, experiments were conducted using the NTU RGB + D60 dataset with cross-subject partitioning, resulting in a total of 25 joints. The skeletal behavior recognition model was trained at the highest perturbation intensity (0.3), and then evaluated at five perturbation intensities (0, 0.05, 0.1, 0.2, and 0.3) using Top-1 accuracy as the metric. Three main perturbation scenarios were designed: temporal perturbation, spatial perturbation, and spatiotemporal coupling perturbation. The experimental settings and results are as follows:

[0166] (1) Time perturbation

[0167] ① Dynamic coordinate jitter

[0168] Within a randomly selected continuous time window, a sinusoidal perturbation is applied to the joint point coordinates to simulate periodic jitter caused by an accidental collision between a wheelchair or nursing trolley and the camera bracket. When the perturbation intensity reaches 30% of the full frame length, the Top-1 accuracy of the method of this invention can be maintained at 62.8%. The coordinate jitter is shown in Table 1.

[0169] ② Frame reordering perturbation

[0170] The frame sequence was randomly shuffled at proportions of 5%, 10%, 20%, and 30% to evaluate the model's robustness to changes in motion order, simulating frame order misalignment caused by transmission delay. Experiments showed that the Top-1 accuracy fluctuated by no more than ±2% under each perturbation level. Frame order rearrangements are shown in Table 1.

[0171] ③ Temporal manifold distortion

[0172] By injecting or deleting 5%, 10%, 20%, and 30% of frames, the model simulated an elderly person walking slowly while holding onto a wall or suddenly falling rapidly. As shown in Table 1, frame injection simulates an elderly person walking slowly while holding onto a wall, while frame deletion simulates a sudden fall and rapid descent. Under all intensities, the model maintained a relatively stable Top-1 accuracy (fluctuation range less than 3%).

[0173] The results of the time perturbation experiment are shown in Table 1. Five intensity levels were set: 0 (no perturbation), 0.05, 0.1, 0.2, and 0.3, which are represented by numbers 1-5.

[0174] Table 1

[0175]

[0176] (2) Spatial disturbance

[0177] ① Motion blur

[0178] Gaussian noise (mean 0, standard deviation 0.1) was added to the joint coordinates at percentages of 5%, 10%, 20%, and 30% to simulate insufficient lighting at night and slight camera shake from an inexpensive camera. Even at maximum noise intensity, the Top-1 accuracy remained above 67%. Motion blur is shown in Table 2.

[0179] ② Covering up key areas

[0180] Key kinematic regions—left arm, right arm, head, torso, and legs—were partially occluded using simulated furniture, crutches, or blankets. The impact of missing key regions on recognition was assessed. Top-1 accuracy remained above 60% under all occlusion conditions, as shown in Table 2.

[0181] ③ Joint point missing perturbation

[0182] Randomly discarding 5%, 10%, 20%, and 30% of key points simulates the occlusion / reflection of the camera by the metal frame when an elderly person uses a walking aid, resulting in key points that cannot be recovered. Even with 30% key point loss, the Top-1 accuracy still reaches 62.2%. Key point loss is shown in Table 2.

[0183] (3) Spatiotemporal coupling disturbance

[0184] First, under the maximum noise intensity of motion blur, frames with a percentage of 5%, 10%, 20%, and 30% are randomly deleted to consider the superposition effect of multiple interferences and simulate the actual "shaking-lag" concurrent situation (such as network packet loss + camera shake). Under the strongest superposition perturbation, the Top-1 accuracy still remains above 60%. Spatiotemporal coupling is shown in Table 2.

[0185] The experimental results of temporal perturbation and spatiotemporal coupling perturbation are shown in Table 2. Five intensity levels were set: 0 (no perturbation applied), 0.05, 0.1, 0.2, and 0.3, represented by numbers 1-5. For occlusion of critical parts, instead of being classified according to perturbation intensity, the left arm, right arm, head, torso, and legs were identified using numbers 1-5.

[0186] Table 2

[0187]

[0188] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A spatio-temporal decoupled human behavior recognition method based on dynamic semantic guidance mask, characterized in that, The method comprises the following steps: (1) extracting human skeleton sequence data from the original video, and performing data enhancement on the skeleton sequence data; (2) performing mask operation on the data-enhanced skeleton sequence in the spatial and temporal dimensions based on a dynamic semantic guided mask mechanism, and masking the joint nodes with a high centrality higher than a threshold in the spatial dimension and the frames with significant motion changes in the temporal dimension; (3) inputting the skeleton sequence data after the mask operation into a query encoder and a momentum encoder, respectively, the query encoder and the momentum encoder both comprising a feature extraction unit and a space-time parallel decoupling unit, obtaining spatial representation and temporal representation, respectively, constructing cross-domain contrast loss, and performing contrast learning training, comprising: (3-1) extracting the masked bone sequence by a feature extraction unit intermediate spatio-temporal representation ; (3-2) Intermediary spatio-temporal representation Perform dimension transformation to obtain preliminary spatial features and preliminary temporal features ; Project the preliminary spatial features into spatial embeddings , and project the preliminary temporal features into temporal embeddings ; (3-3) Further refining the decoupled spatial embedding based on a self-attention mechanism module and a feed-forward module and temporal embedding , respectively, to obtain spatial representation and temporal representation ; (3-4) The query encoder transforms the decoupled spatial representation and temporal representation into a final spatial representation and a final temporal representation respectively, fusing and into a global view representation ; Momentum encoders transform decoupled spatial representations and temporal representations into final spatial representations and final temporal representations respectively and fused into a global view representation ; (3-5) constructing a four-tuple level contrast loss function structure and performing contrast learning training: ； , , and are weight coefficients of four-way loss, respectively; denotes the contrastive loss; In the training process, the query encoder parameters are updated through back propagation, and the momentum encoder parameters are updated through a momentum strategy; After the training is completed, a human behavior recognition model is obtained; (4) inputting a to-be-recognized video into the human behavior recognition model to obtain a prediction result of human behavior in the to-be-recognized video.

2. The spatio-temporal decoupled human behavior recognition method based on dynamic semantic guidance mask according to claim 1, characterized in that, In step (1), the data enhancement on the skeleton sequence data comprises: performing spatial rotation, mirror flipping or shearing on the skeleton sequence.

3. The spatio-temporal decoupled human behavior recognition method based on dynamic semantic guidance mask according to claim 1, characterized in that, In step (2), the mask operation on the data-enhanced skeleton sequence in the spatial dimension based on the dynamic semantic guided mask mechanism comprises: (2-1i) calculating a degree centrality of each of the junction nodes in each frame in the sequence of actions, the degree centrality is defined as: wherein denotes the number of edges connected to the node (2-1 ii) Add each node i to the set according to its degree centrality: ; N represents the total number of single-frame key nodes; (2-1 iii) masking from sampling a fixed number of joints.

4. The spatio-temporal decoupled human action recognition method based on dynamic semantic guidance mask of claim 1, wherein, In step (2), the mask operation on the data-enhanced skeleton sequence in the temporal dimension based on the dynamic semantic guided mask mechanism comprises: (2-2i) Computing attention values for each time step in the skeletal sequence : ； wherein, is the number of channels; is the number of joints; is the number of persons; is the motion attention value of the t-th frame, the p-th person, the joint, the coordinate channel; the motion attention value , given the skeleton sequence , represents the complete skeleton of the t-th frame, the inter-frame motion difference of the t-th frame t is defined as: ;​ (2-1 ii) screening out from the skeletal sequence The smallest fixed number of frames masks it.

5. The spatio-temporal decoupled human action recognition method based on dynamic semantic guidance mask according to claim 1, characterized in that, In step (3-2), the preliminary spatial features Projection transformation to spatial embedding , which is ； wherein, , is a trainable weight matrix, , is a bias vector, denotes an activation function; The preliminary timing features are combined into a single feature vector The projection is converted into a time embedding with the formula: ； wherein, , is a trainable weight matrix, , is a bias vector, denotes an activation function.

6. The spatio-temporal decoupled human action recognition method based on dynamic semantic guidance mask according to claim 1, characterized in that, Step (3-3) comprises: (3-3i) Spatial Embedding Project to the three spaces of query, key, value, compute attention scores between joints by scaled dot-product, then normalize by Softmax: ； denotes the i-th spatial dimension attention head output feature matrix; , and are query, key and value matrices obtained by linear transformation, is a scaling factor; The spatial attention results are combined, respectively, with In combination, the feature stability is enhanced by residual connections and layer normalization: ； is a refined spatial feature matrix; is a concatenated attention matrix; is a fully connected layer processing; is a layer normalization function; denotes the number of spatial dimension attention heads; ； is a layer normalization function; is a feed forward network, is a max pooling operation; (3-3ii) Time Embedding Project to query, key, value three spaces, calculate the attention score between joints through scaled dot product, and then normalized by Softmax: ； denotes the i-th attention head output feature matrix in the time dimension; 、 and are query, key and value matrices obtained by linear transformation, is a scaling factor; The spatial attention results are combined, respectively, with In combination, the feature stability is enhanced by residual connections and layer normalization: ； is a refined spatial feature matrix; is a concatenated attention matrix; is a fully connected layer processing; is a layer normalization function; denotes the number of heads in the temporal dimension. ； is a layer normalization function; is a feed-forward network, is a max-pooling operation.

7. The spatio-temporal decoupled human action recognition method based on dynamic semantic guidance mask according to claim 1, characterized in that, In step (3-5), the InfoNCE framework is used to calculate the sample contrastive loss, and for any contrast pair u and v , the loss is defined as: ； wherein, is u and v a similarity score of , is a temperature parameter, is an inner product of two feature vectors u and v ; denotes the set of negative samples , the similarity scores between u and all negative samples are computed and exponentially summed.

8. A device for spatio-temporal decoupled human behavior recognition based on dynamic semantic guidance masks, characterized in that comprises: The data processing module extracts human skeleton sequence data from the original video, and performs data enhancement on the skeleton sequence data; The dynamic semantic guided mask module performs mask operation on the data-enhanced skeleton sequence in the spatial and temporal dimensions based on a dynamic semantic guided mask mechanism, and masks the joint nodes with a high centrality higher than a threshold in the spatial dimension and the frames with significant motion changes in the temporal dimension; The contrast learning training module inputs the skeleton sequence data after the mask operation into a query encoder and a momentum encoder, respectively, the query encoder and the momentum encoder both comprising a feature extraction unit and a space-time parallel decoupling unit, obtains spatial representation and temporal representation, respectively, constructs cross-domain contrast loss, and performs contrast learning training, comprising: extracting the masked bone sequence by a feature extraction unit intermediate spatio-temporal representation ; intermediate spatio-temporal representation performing dimension transformation to obtain preliminary spatial features and preliminary temporal features ; projecting the preliminary spatial features into a spatial embedding , and projecting the preliminary temporal features into a temporal embedding ; further refining the decoupled spatial embedding based on a self-attention mechanism module and a feed-forward module and a temporal embedding , respectively, resulting in a spatial representation and a temporal representation ; The query encoder transforms the decoupled spatial representation and temporal representation into a final spatial representation and a final temporal representation respectively, fusing and into a global view representation ; Momentum encoders transform decoupled spatial representations and temporal representations into final spatial representations and final temporal representations respectively and fuse into a global view representation ; constructing a four-tuple level contrast loss function structure and performing contrast learning training: ； , , and are weight coefficients of four-way loss, respectively; denotes the contrastive loss; In the training process, the query encoder parameters are updated through back propagation, and the momentum encoder parameters are updated through a momentum strategy; The human behavior recognition module inputs a to-be-recognized video into the human behavior recognition model to obtain a prediction result of human behavior in the to-be-recognized video.

9. A spatio-temporal decoupled human behavior recognition device based on dynamic semantic guided mask, characterized in that, comprise a memory, a processor, and a computer program stored in the memory and executable on the processor; The memory stores a human behavior recognition model; a to-be-recognized video is input into the human behavior recognition model, and the processor executes the computer program to realize prediction of human behavior in the to-be-recognized video; The human behavior recognition model is trained by the steps in the method of any one of claims 1-7.

Images