Space-time decoupling human body behavior recognition method, device and equipment based on dynamic semantic guide mask

Abstract

The invention discloses a space-time decoupling human behavior recognition method, device and equipment based on a dynamic semantic guide mask. The method comprises the following steps: extracting human skeleton sequence data from an original video and performing data enhancement; based on a dynamic semantic guide mask mechanism, performing mask operation on the skeleton sequence after data enhancement in two dimensions of space and time; respectively sending the skeleton sequence data after mask operation into a query encoder and a momentum encoder, respectively obtaining spatial representation and time representation, constructing cross-domain contrast loss, and carrying out contrast learning training to obtain a human behavior recognition model; and inputting the to-be-recognized video into the human body behavior recognition model to obtain a prediction result of the human body behavior in the to-be-recognized video. According to the space-time decoupling human body behavior recognition method, device and equipment based on the dynamic semantic guide mask, the human body behavior recognition precision and the environmental adaptability are remarkably improved under the condition that manual labeling is not needed.

Description

Technical Field

[0001] The present invention relates to the technical field of intersection of computer vision and neuromorphic computing, and in particular to a method, device and equipment for spatiotemporal decoupling of human behavior recognition based on dynamic semantically guided masks. Background Art

[0002] Human behavior recognition, as a key technology for artificial intelligence systems to understand the physical world, is gradually expanding from controlled experimental environments to the open real world. It is widely used in areas such as somatosensory interaction, security monitoring, and rehabilitation assistance.

[0003] In recent years, the rise of the human-computer collaborative computing paradigm has driven the development of "embodied intelligence." This paradigm, which simulates human perception and decision-making mechanisms through brain-inspired neuromorphic computing, provides a new framework for interpretable behavioral analysis that incorporates spatiotemporal dynamics. Simultaneously, the introduction of cognitive ergonomics has advanced the semantic understanding and safety assessment of human motion, enabling action recognition to move beyond relying solely on low-level motion features and instead integrate high-level semantic cues with dynamic behavioral patterns.

[0004] In terms of data modality, current research is working to overcome the limitations of traditional visual methods. With the prevalence of depth sensors, skeleton data has become a mainstream data source for advanced applications such as action recognition, skill transfer, and motion prediction, thanks to its lightweight, privacy-preserving, and precise joint positioning. In understanding human behavior and modeling robot motion cognition, the biomimetic properties of skeletons effectively combine ergonomic posture parameters with motion dynamics. Notably, breakthroughs in self-supervised learning have greatly promoted the study of action recognition in sparsely labeled scenarios, which is also the direct motivation for this study to focus on self-supervised learning methods. Pre-set tasks enable the model to autonomously learn meaningful feature representations from unlabeled data.

[0005] AimCLR is a self-supervised action recognition method based on contrastive learning, which improves model robustness through extreme data augmentation and feature dropout modules. AimCLR uses random view transformations and global contrastive learning for skeletal self-supervision, but its masking and cropping are decoupled from semantics, making it prone to missing key information when encountering joint occlusion or noise insertion, resulting in insufficient robustness. MaskCLR is an attention-guided contrastive learning framework. This method uses a probabilistic masking strategy to occlude key body parts, forcing the model to learn to rely on other motion features for action classification, thereby enhancing its adaptability to pose estimation errors and subtle motion changes. MaskCLR uses attention-guided probabilistic masking and multi-level contrastive learning to improve noise resistance, but it only focuses on spatially salient joints, lacks temporal dynamic adaptation, and requires manual setting of mask strength, making it unable to account for different motion and noise scenarios.

[0006] However, there are a large number of interference factors in the real environment, such as camera occlusion, sensor noise, frame loss, etc., which lead to the lack of robustness of existing behavior recognition methods in practical applications and significant performance fluctuations. Therefore, there is a need for a behavior recognition method with stronger robustness in complex dynamic scenes. The present invention constructs a dynamic spatiotemporal mask through "joint degree centrality + inter-frame motion difference", and combines space-time decoupling coding with cross-domain comparative learning to cover key joints and key frames and adaptive noise, significantly improving the robustness of behavior recognition in complex dynamic scenes. Summary of the Invention

[0007] The present invention provides a spatiotemporal decoupled human behavior recognition method based on dynamic semantic guided masking, which significantly improves the accuracy of human behavior recognition and environmental adaptability without the need for manual labeling.

[0008] The technical solutions of the present invention are as follows: A spatiotemporal decoupled human behavior recognition method based on dynamic semantic guided masking includes the following steps: (1) Extract human skeleton sequence data from the original video and perform data enhancement on the skeleton sequence data; (2) Based on the dynamic semantics-guided masking mechanism, the skeleton sequence after data augmentation is masked in both spatial and temporal dimensions. In the spatial dimension, joints with degree centrality higher than the threshold are masked, and in the temporal dimension, frames with significant motion changes are masked. (3) The skeleton sequence data after masking operation is fed into the query encoder and momentum encoder respectively. Both the query encoder and momentum encoder include a feature extraction unit and a spatiotemporal parallel decoupling unit to obtain spatial representation and temporal representation respectively, construct cross-domain contrast loss, and perform contrastive learning training; After the training, a human behavior recognition model is obtained; (4) Input the video to be identified into the human behavior recognition model to obtain the prediction results of the human behavior in the video to be identified.

[0009] In response to the defects of existing skeletal action recognition such as "random mask information leakage, spatial-temporal feature separation, and insufficient robustness", the spatiotemporal decoupled human action recognition method based on dynamic semantic-guided mask in the present invention adaptively constructs a mask sampling pool in the spatial dimension based on the degree centrality of joint points and in the temporal dimension based on the inter-frame motion difference, and cooperates with a parallel decoupling encoder and cross-domain contrastive learning loss to significantly improve recognition accuracy and environmental adaptability without the need for manual labeling. It can be deployed in elderly care fall detection and embodied monitoring robot scenarios.

[0010] In step (1), data enhancement of the skeleton sequence data includes: performing spatial rotation, mirror flipping or shearing on the skeleton sequence.

[0011] In step (2), the masking operation on the data-enhanced skeleton sequence in the spatial dimension based on the dynamic semantic guided masking mechanism includes: (2-1i) Calculate the degree centrality of each joint point in each frame in the action sequence, degree centrality Defined as: ,in Representation node The number of connected edges; (2-1ii) Add each node i to the set according to its degree centrality: ; N represents the total number of joint points in a single frame; (2-1iii) From A fixed number of joint points are sampled for masking.

[0012] In step (2-1iii), the number of joint points sampled and masked is manually set as needed. Joint points with higher degree centrality are repeatedly included in the sampling pool more often, ensuring that the masking operation prioritizes coverage of key structural areas in the skeleton graph (such as core joints such as the shoulder, elbow, and hip). By masking these high-centrality nodes, the model is guided to focus on the semantic features of low-attention areas, thereby enhancing spatial generalization capabilities.

[0013] In step (2), the masking operation on the skeleton sequence after data enhancement in the time dimension based on the dynamic semantic guided masking mechanism includes: (2-2i) Calculate the attention value of each time step in the skeleton sequence : ; in, is the number of channels; is the number of joints; is the number of human bodies; For the tth frame, the pth person, Joints, The motion attention value of the coordinate channel; motion attention value , given a skeleton sequence , Indicates the t Frame complete skeleton, inter-frame motion difference of frame t Defined as: ; (2-1ii) Screening from bone sequences A minimum fixed number of frames will mask it.

[0014] In step (2-1ii), the frames in the skeleton sequence are arranged according to Sort by size, and select the first F frames for masking. The number of frames F to be masked can be set based on specific needs.

[0015] Through the above calculation, the frames with significant motion changes correspond to A lower value means that these frames are considered less important and are deleted during masking; stable frames with the smallest changes correspond to Higher values ​​are considered important frames.

[0016] By masking based on a dynamic semantics-guided masking mechanism, the content in the skeleton sequence that is highly relevant to the action semantics is masked, guiding the model to actively explore and learn other potential features that are not masked, thereby obtaining richer and more interference-resistant action expressions.

[0017] Step (3) includes: (3-1) Extract the skeleton sequence after mask operation through the feature extraction unit Intermediate space-time representation of ; (3-2) Representation of intermediate spacetime Perform dimension transformation to obtain preliminary spatial features and preliminary time series characteristics ; The preliminary spatial features Projection into spatial embedding , the preliminary time series features Projection into temporal embedding ; (3-3) Further refine the decoupled spatial embedding based on the self-attention mechanism module and the feedforward module and time embedding , and obtain the spatial representation and time representation ; (3-4) The query encoder decouples the spatial representation by projection and time representation Converted into final spatial representation and the final time representation ,Will and Fusion introduces global perspective representation ; The momentum encoder decouples the spatial representation by projecting and time representation Converted into final spatial representation and the final time representation ,Will and Fusion introduces global perspective representation ; (3-5) Construct a quadruple-level contrast loss function structure for contrastive learning training: ; 、 、 and are the weight coefficients of the four-way loss respectively; represents contrast loss; During training, the query encoder parameters are updated via back-propagation, and the momentum encoder parameters are updated via the momentum strategy.

[0018] The feature extraction unit of the query encoder and momentum encoder is a graph convolutional neural network (GCN).

[0019] In step (3-1), for each joint point , its output features for: ; represents the normalization factor; For joint points Neighborhood joints (referring to the joint points The set of directly connected joints, i.e. the skeleton is modeled as a joint-connection graph, where each joint is a node and the physiological connections between joints are edges); For joint points The feature vectors of the adjacent nodes; is the weight matrix that changes with time; is the nonlinear transformation matrix; is the ReLU activation function; is the weight function.

[0020] In step (3-2), the preliminary spatial features Projection into spatial embedding , the formula is: ; in, 、 is a trainable weight matrix, 、 is the bias vector, represents the activation function; The preliminary time series characteristics Projection into temporal embedding , the formula is: ; in, 、 is a trainable weight matrix, 、 is the bias vector, Represents the activation function.

[0021] Step (3-3) includes: (3-3i) Embedding space Projected into the query, key, and value spaces, the attention scores between joints are calculated by scaling the dot product and then normalized by Softmax: ; Represents the output feature matrix of the i-th attention head in the spatial dimension; 、 and are respectively the query, key and value matrices obtained by linear transformation, is the scaling factor; The spatial attention results are respectively compared with Combined, feature stability is enhanced through residual connection and layer normalization: ; is the refined spatial feature matrix; is the concatenated attention matrix; Processed by the fully connected layer; is the layer normalization function; Indicates the number of attention heads in the spatial dimension; ; is the layer normalization function; is a feedforward network, It is the maximum pooling operation; (3-3ii) Embedding time Projected into the query, key, and value spaces, the attention scores between joints are calculated by scaling the dot product and then normalized by Softmax: ; Represents the output feature matrix of the i-th attention head in the time dimension; 、 and are respectively the query, key and value matrices obtained by linear transformation, is the scaling factor; The spatial attention results are respectively compared with Combined, feature stability is enhanced through residual connection and layer normalization: ; is the refined spatial feature matrix; is the concatenated attention matrix; Processed by the fully connected layer; is the layer normalization function; Indicates the number of attention heads in the time dimension; ; is the layer normalization function; is a feedforward network, It is the maximum pooling operation.

[0022] In steps (3-5), the InfoNCE framework is used to calculate the sample contrast loss. u and v , the loss is defined as: ; in, for u and v The similarity score of , is the temperature parameter, are two eigenvectors u and v The inner product of Indicates that in the negative sample set In the calculation u With all negative samples The similarity scores between them are summed exponentially.

[0023] The present invention also provides a temporal-spatial decoupling human behavior recognition device based on dynamic semantic guided mask, comprising: The data processing module extracts human skeleton sequence data from the original video and performs data enhancement on the skeleton sequence data; The dynamic semantics-guided masking module performs masking operations on the data-augmented skeleton sequence in both spatial and temporal dimensions based on the dynamic semantics-guided masking mechanism. In the spatial dimension, it masks joints whose centrality is higher than the threshold, and in the temporal dimension, it masks frames with significant motion changes. The contrastive learning training module feeds the masked skeleton sequence data into the query encoder and momentum encoder, respectively. Both the query encoder and momentum encoder include a feature extraction unit and a spatiotemporal parallel decoupling unit to obtain spatial and temporal representations, respectively. Cross-domain contrastive loss is then constructed to perform contrastive learning training. After training, a human action recognition model is obtained. The human behavior recognition module inputs the video to be recognized into the human behavior recognition model to obtain the prediction results of the human behavior in the video to be recognized.

[0024] The present invention also provides a spatiotemporal decoupled human behavior recognition device based on dynamic semantic guided mask, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; The memory stores a human behavior recognition model obtained by training using the above method; the video to be recognized is input into the human behavior recognition model, and the processor executes the computer program to predict the human behavior in the video to be recognized.

[0025] Compared with the prior art, the present invention has the following beneficial effects: (1) Robustness is greatly enhanced: it maintains high accuracy under extreme disturbances such as coordinate jitter, frame insertion and deletion, and occlusion, and there is no crash in complex disturbance scenarios.

[0026] (2) High information utilization: Based on dynamic semantic masks, the model is effectively guided to discover semantic clues in low-attention areas and stable frames, avoiding random masks from destroying structural information.

[0027] (3) Parallel decoupling architecture is more discriminative: spatial-temporal domain learning avoids the fuzzy treatment of topological and temporal dependencies in traditional coupling models, and combines cross-layer fusion to further enhance feature expression, resulting in higher recognition accuracy.

[0028] (4) No need for manual labeling, cross-platform friendly: Full-link self-learning from data enhancement to model optimization reduces dependence on manual labeling, reduces data costs, and improves adaptability in new scenarios or scenarios with few labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flowchart of the spatiotemporal decoupled human action recognition method based on dynamic semantic guided mask.

[0030] Figure 2 Schematic diagram of the structure of the spatiotemporal decoupled human behavior recognition device based on dynamic semantic guided mask. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0032] The technical solution provided by the present invention is a method and system for robust human behavior recognition, which combines a dynamic semantics-guided masking strategy, self-supervised contrastive learning, and a spatiotemporal decoupling network to form a human behavior recognition system and method.

[0033] The specific process of the spatiotemporal decoupling human behavior recognition method based on dynamic semantic guided mask is as follows: Figure 1 As shown, the following steps are included: S101 inputs the original video frame and extracts the skeleton sequence Extract skeleton sequences from raw video frames through pose estimator S .

[0034] S102 data enhancement to generate dual views Bone Sequence S Data enhancement by spatial rotation, mirror flipping and shearing is performed to obtain two positive samples.

[0035] Using data enhancement operations, give the skeleton sequence S Randomly transform them into different enhanced versions and regard them as positive sample pairs. For skeletal data, the present invention uses spatial rotation, mirror flipping and shear affine enhancement strategies.

[0036] (1) Spatial rotation with bone sequence S No. T Frame as an example, its joint set can be expressed as , V T For bone sequence S No. T The complete set of joints for a frame, which contains the 3D coordinates of all joints in that frame; V T，j For the T Frame No. j The specific coordinates of each joint; J Indicates the total number of joints each skeleton contains in a single frame.

[0037] By using Euler quaternion, rotate counterclockwise around the x, y, and z axes to the angle α T 、 β T and γ T , generate new coordinates.

[0038] (2) Mirror flip enhancement is achieved by reversing the bone sequence S The left and right structures are used to introduce mirror symmetry. S Each frame in T , joint set Reflection is performed along the sagittal plane. During this process, the coordinates of the left and right joints are swapped, while the midline joints remain in their original positions.

[0039] (3) Shear enhancement is a linear transformation of the three-dimensional coordinate space of the body joints, which causes the joint structure to tilt at random angles. Shear transformation matrix ,in , and the remaining positions are randomly set shear factors.

[0040] S103 Dynamic Semantic Guided Mask (1) Spatial mask weight calculation The spatial mask weights calculate the degree centrality of each joint and output the probability distribution of joint importance. In the spatial dimension, the importance of joints is measured based on the degree centrality index of the joints, which is measured based on the connectivity of the nodes. The definition is as follows: ,in Representation node The number of connected edges.

[0041] To construct a dynamic spatiotemporal mask sampling pool, this paper uses joint degree centrality as a spatial importance indicator and performs importance-based weighted sampling on nodes in the skeletal graph structure. Specifically, the degree centrality of each joint in the entire action sequence (i.e., the strength of the connection between the node and other nodes) is first calculated to reflect its topological importance in the spatial dimension. The sampling probability of each node is then determined based on the degree centrality value. Joints with higher degree centrality are repeatedly included in the sampling pool more often, thereby ensuring that the masking operation preferentially covers key structural areas in the skeletal graph (such as core joints such as shoulders, elbows, and hips). Each node i is added to the set according to its centrality value: ; A joint Represents a multiple set, indicating that the higher the centrality, the greater the probability of being drawn. The centrality is converted into sampling probability without explicit normalization; N represents the total number of joints in a single frame.

[0042] (2) Applying spatial mask final A fixed number of joint points k are sampled in , and a mask set is constructed: The number of nodes to be masked is k (the nodes are placed at infinity), J k represents the final selected k joint index subsets; j represents J k By masking these high-centrality nodes, the model is guided to focus on the semantic features of low-attention areas, thereby enhancing spatial generalization ability.

[0043] (3) Time mask calculation The temporal mask weights calculate the motion difference between adjacent frames and output the frame mask probability distribution.

[0044] In the temporal dimension, frames with intense motion (such as leg swings during running or rapid arm movements during waving) typically contain a wealth of transient information but are also susceptible to noise and sensor jitter. By prioritizing these frames, the model is forced to rely on more consistent motion patterns in stable frames (such as torso posture during running or shoulder position during waving) to learn action semantics, thereby enhancing robustness to temporal disturbances.

[0045] In order to extract the motion information of the time dimension from the sequence data, the dynamic weight is calculated based on the inter-frame motion difference, and the motion attention distribution is calculated by the square of the absolute value of the inter-frame difference. , Indicates the t Frame complete skeleton; t The inter-frame motion difference is defined as The motion attention distribution is defined as: By averaging all channels, joints and human individuals, we get t The comprehensive motion attention value of the frame (Frame-level dynamic motion attention): ; in, is the number of channels, is the number of joints, is the body number, For the t Frame, p Individual characters, Joints, The local motion attention value of the coordinate channel. 、 、 Represents coordinate channel index, joint index, and character index respectively.

[0046] The specific calculation method is to first calculate all channels (the total number is ), all joints (total ) and all personnel (total ) corresponds to the local motion attention value Sum the result and then divide it by the product of the number of channels, the number of joints and the number of people to get the attention value for each time step This calculation method makes the frames with significant motion changes correspond to Lower values ​​mean that these frames are considered less important frames; stable frames with minimal changes correspond to Higher values ​​are considered important frames.

[0047] (4) Applying time mask Filter out The smallest F frames (i.e., high-motion frames, where F is artificially defined) are masked (deleted). This mechanism masks frames with high dynamic changes and guides the model to enhance stable temporal features in static frames.

[0048] Therefore, a robust feature representation method is adopted. This method introduces a dynamic semantics-guided masking strategy in the training stage to mask the content in the skeletal action sequence that is highly relevant to the action semantics, and guides the model to actively explore and learn other potential features that are not masked, so as to obtain richer and more interference-resistant action expression.

[0049] S104 trains the recognition model based on contrastive learning (1) The present invention adopts parallel spatiotemporal decoupled feature extraction, in which the graph convolutional network (GCN) is used to capture spatial structure information and temporal dynamic features, while the self-attention mechanism focuses on the temporal dynamic features.

[0050] The network is trained on the enhanced input skeleton (C×T×V) is processed to generate an intermediate spatiotemporal representation (C1×T×V). For each target node , its output features is achieved by aggregating its domain joints (referring to the target node A set of directly connected nodes, i.e. the skeleton is modeled as a joint-connection graph, where each joint is a node and the physiological connections between joints are edges) The features are calculated.

[0051] ; The specific meaning is that the target node At the moment Output features , which is equivalent to its domain All adjacent nodes The features of are weighted summed. is the feature vector of the adjacent nodes; It is a nonlinear transformation of the features of adjacent nodes; It is a time-varying weight matrix used to capture temporal dynamic information; is the ReLU activation function; Is a weight function used to evaluate the importance of the transformed features; Used to normalize the contributions of the aggregation.

[0052] In each training step, the input data is divided into two branches, i.e., two augmentation sequences: query data and key data These two sequences are regarded as positive samples and mapped to the embedding space through the mapping function. The embedding representation of the query data is , the embedding of key data is expressed as .

[0053] To support efficient contrastive learning (pulling positive samples closer and pushing negative samples further away), the model introduces a dynamic queue to store negative sample features and uses a momentum update strategy to stabilize the key encoder parameters. The queue is used to store historical key features k, which serve as negative samples (samples that do not belong to the same action as the current query feature). To prevent training instability caused by overly frequent key encoder parameter updates, a momentum strategy is used to update its parameters. After each training round, new key features are added to the queue, and the oldest features are removed to maintain a fixed queue size. During optimization, only the query encoder is updated via backpropagation, while the key encoder is updated using a momentum-based strategy.

[0054] ; in, and Represents the parameters of the key encoder and query encoder respectively. Parameters of the key encoder Update to original parameters Multiply by the momentum coefficient and Multiply by the query encoder parameter The sum of . Updated only by backpropagation Parameters, this momentum update method makes The evolution ratio Smoother.

[0055] Initially extracted intermediate spatiotemporal representation (C1×T×V) performs dimension transformation to obtain preliminary spatial features and preliminary time series characteristics . The preliminary spatial features The projection is transformed into a spatial embedding at a higher semantic level, and the formula is: ; in, Represents spatial embedding; It is the preliminary spatial feature obtained after aggregation by the graph convolutional network (GCN), including the topological relationship between joints and local spatial dependencies; 、 is a trainable weight matrix, 、 is the bias vector, Represents the activation function.

[0056] The core link of time series feature processing, its design logic and spatial embedding A symmetrical response is formed, and targeted optimization is carried out based on the dynamic characteristics of the time dimension: ; in, represents temporal embedding, It is the preliminary timing feature obtained through preliminary feature extraction in the time dimension, which contains basic timing information such as the inter-frame motion amplitude, speed change, and periodic pattern of the action. 、 is a trainable weight matrix, 、 is the bias vector, Represents the activation function.

[0057] During feature extraction, spatiotemporal interactions are intertwined: Focus on the spatial topology of the joint (static structure), Focusing on the temporal dynamics (dynamic changes) of actions, the two form a complex information structure through cross-domain comparative learning, jointly supporting the complete expression of action semantics. To further refine the decoupled spatial and temporal features, the model introduces a self-attention mechanism and multi-layer processing (layer normalization and feedforward networks) to improve feature discriminability.

[0058] Spatial Self-Attention Mechanism: Spatial self-attention is used to mine the non-local dependencies between human joints. The spatial embedding extracted by GCN (V is the total number of joints, C is the total number of channels) projected onto the query ( ),key( ),value( )Three spaces.

[0059] The attention score between joints is calculated by scaling the dot product and then normalized by Softmax: ; in, Represents the output feature matrix of the i-th attention head; 、 and are the query, key, and value matrices obtained through linear transformation, middle represents transpose, is the scaling factor.

[0060] The spatial attention results are respectively compared with Combined, feature stability is enhanced through residual connection and layer normalization: ; Among them, the refined spatial feature matrix Is the concatenated attention matrix After the fully connected layer After processing, and spatial embedding Add and then perform layer normalization. Indicates the number of attention heads; ; in, is the layer normalization function, It is a feedforward network, and the maximum pooling operation further enhances the spatial representation .

[0061] Temporal Self-Attention Mechanism: Temporal self-attention is used to capture the long-range dependencies of actions in the temporal dimension and analyze the temporal patterns of actions. Its design is symmetrical to spatial self-attention but focuses on temporal dynamics: The temporal features extracted by GCN (T is the total number of frames) projected onto the query ( ),key( ),value( )Three spaces.

[0062] The attention score between joints is calculated by scaling the dot product and then normalized by Softmax: ; in, Represents the output feature matrix of the i-th attention head; 、 and are the query, key, and value matrices obtained through linear transformation, middle represents transpose, is the scaling factor.

[0063] After aggregating spatiotemporal features through GCN, the obtained temporal embedding of the time dimension is recorded as First, the attention matrix of the time dimension is processed, the multi-head attention matrix is ​​spliced, and then the full connection layer is used to Perform feature conversion and then embed it with time series Perform residual connection and layer normalization Operation to obtain preliminary refined time characteristics : ; In order to further improve the discriminability of time features, the initially refined time features , a feed-forward network input to the time dimension The feedforward network is composed of multi-layer perceptrons and can perform more complex nonlinear transformations on temporal features. Perform residual connection and normalize through layers, and finally use global average pooling of time dimension to compress time dimension and obtain the final refined and enhanced time features. : .

[0064] (2) Cross-domain contrast loss function To utilize the decoupled spatial features and time characteristics , the query encoder obtains the final representation by projection and .Will and Fusion, to introduce global perspective representation Similarly, the key encoder obtains the final representation of spatial and temporal features through projection and .Will and Fusion, get .

[0065] To quantify the differences between these representations, a quadruple-level contrastive loss function structure is formed, combining momentum encoding with a dynamic queue of negative samples: ; 、 、 and are the weight coefficients of the four-way loss respectively. These coefficients are set to balance the contribution of each loss term. represents contrast loss.

[0066] For skeletons extracted from different videos, since their contents are different, they form negative sample pairs, and their similarity in the latent space should be minimized. To this end, we use the InfoNCE framework to calculate the sample contrast loss. For any contrast pair u and v , the loss is defined as: ; Among them, H represents the negative sample set; , is the temperature parameter, What is calculated is the inner product of two eigenvectors, which can be used to measure the similarity of the two vectors in direction; Indicates that in the negative sample set In the calculation u With all negative samples The similarity scores between them are summed exponentially.

[0067] S105 Model Application The video to be identified is input into the trained recognition model to obtain the prediction results of the human behavior in the video to be identified.

[0068] like Figure 2 As shown, the present invention also provides a temporal and spatial decoupling human behavior recognition device based on dynamic semantic guided mask, comprising: The data processing module extracts human skeleton sequence data from the original video and performs data enhancement on the skeleton sequence data; The dynamic semantics-guided masking module performs masking operations on the data-augmented skeleton sequence in both spatial and temporal dimensions based on the dynamic semantics-guided masking mechanism. The spatial dimension masking is used to mask joints with degree centrality above the threshold, and the temporal dimension masking is used to mask frames with significant motion changes. The contrastive learning training module feeds the masked skeleton sequence data into the query encoder and momentum encoder, respectively. Both the query encoder and momentum encoder include a feature extraction unit and a spatiotemporal parallel decoupling unit to obtain spatial and temporal representations, respectively. Cross-domain contrastive loss is then constructed to perform contrastive learning training. After training, a human action recognition model is obtained. The human behavior recognition module inputs the video to be recognized into the human behavior recognition model to obtain the prediction results of the human behavior in the video to be recognized.

[0069] The present invention also provides a spatiotemporal decoupled human behavior recognition device based on dynamic semantic guided masks, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; The memory stores a human behavior recognition model obtained by training the above method; the video to be recognized is input into the human behavior recognition model, and the processor executes a computer program to predict the human behavior in the video to be recognized.

[0070] To verify the stability and anti-interference capabilities of this invention in a real-life home-based elderly care environment, experiments were conducted using the cross-subject partition of the NTU RGB + D60 dataset, with a total of 25 joints. The skeletal behavior recognition model was trained at the highest perturbation intensity (0.3) and then evaluated at five perturbation intensities (0, 0.05, 0.1, 0.2, and 0.3) using Top-1 accuracy as the metric. Three categories of perturbation scenarios were designed: temporal perturbation, spatial perturbation, and spatiotemporal coupled perturbation. The experimental setup and results are as follows: (1) Time disturbance ① Dynamic coordinate jitter Sine wave perturbations were applied to the joint coordinates within randomly selected continuous time windows to simulate the periodic jitter caused by accidental collisions between a wheelchair or nursing cart and the camera mount. When the perturbation intensity reached 30% of the full frame length, the proposed method maintained a top-1 accuracy of 62.8%. Table 1 shows the coordinate jitter.

[0071] ②Frame reordering disturbance The model's robustness to changes in motion order was evaluated by randomly shuffling the frame sequence at ratios of 5%, 10%, 20%, and 30%, simulating frame misalignment caused by transmission delays. Experiments showed that the Top-1 accuracy fluctuated within ±2% at all perturbation intensities. The frame reordering is shown in Table 1.

[0072] ③ Temporal manifold distortion By injecting or deleting 5%, 10%, 20%, and 30% of the frames, we simulated elderly people walking slowly with the help of a wall or suddenly falling and falling. As shown in Table 1, frame injection simulated elderly people walking slowly with the help of a wall, while frame deletion simulated sudden falls and falling. Under all intensities, the model maintained a relatively stable Top-1 accuracy (fluctuation within 3%).

[0073] The results of the time perturbation experiment are shown in Table 1. Five intensity levels are set: 0 (no perturbation), 0.05, 0.1, 0.2, and 0.3, which are represented by numbers 1-5.

[0074] Table 1

[0075] (2) Spatial disturbance ① Motion blur Gaussian noise (mean 0, standard deviation 0.1) was added to the joint coordinates at ratios of 5%, 10%, 20%, and 30% to simulate low-light conditions at night and slight camera jitter. Even at the maximum noise intensity, the top-1 accuracy remained above 67%. Table 2 shows motion blur.

[0076] ②Key parts blocking We occluded the left and right arms, head, torso, and legs, simulating partial occlusion by furniture, a cane, or a blanket, to evaluate the impact of missing key parts on recognition. Top-1 accuracy remained above 60% under all occlusion conditions. Table 2 shows the occlusions.

[0077] ③ Joint point missing disturbance We randomly discarded 5%, 10%, 20%, and 30% of the joints to simulate the occlusion / reflection of the metal frame of an elderly person using a walker, resulting in the inability to recover the joints. Even with 30% of the joints missing, the top-1 accuracy still reached 62.2%. Table 2 shows the missing joints.

[0078] (3) Space-time coupling disturbance First, we applied maximum motion blur noise intensity, then randomly deleted 5%, 10%, 20%, and 30% of the frames. This allowed for the combined effects of multiple interferences and simulated real-world jitter and lag scenarios (e.g., network packet loss and camera shake). Even with the strongest combined perturbation, the top-1 accuracy remained above 60%. Table 2 shows the spatiotemporal coupling.

[0079] The experimental results for temporal perturbations and spatiotemporal coupling perturbations are shown in Table 2. Five perturbation intensity levels were set: 0 (no perturbation), 0.05, 0.1, 0.2, and 0.3, represented by numbers 1-5. For key area occlusion, instead of perturbation intensity, the left arm, right arm, head, torso, and legs were categorized using the same numbers 1-5.

[0080] Table 2

[0081] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A spatiotemporal decoupling human behavior recognition method based on dynamic semantic guided mask, characterized by: The following steps are involved: (1) Extract human skeleton sequence data from the original video and perform data enhancement on the skeleton sequence data; (2) Based on the dynamic semantics-guided masking mechanism, the skeleton sequence after data augmentation is masked in both spatial and temporal dimensions. In the spatial dimension, joints with degree centrality higher than the threshold are masked, and in the temporal dimension, frames with significant motion changes are masked. (3) The skeleton sequence data after masking operation is fed into the query encoder and momentum encoder respectively. Both the query encoder and momentum encoder include a feature extraction unit and a spatiotemporal parallel decoupling unit to obtain spatial representation and temporal representation respectively, construct cross-domain contrast loss, and perform contrastive learning training; After the training, a human behavior recognition model is obtained; (4) Input the video to be identified into the human behavior recognition model to obtain the prediction results of the human behavior in the video to be identified.

2. The method for human behavior recognition based on spatiotemporal decoupling of dynamic semantic guided masks according to claim 1 is characterized in that: In step (1), data enhancement of the skeleton sequence data includes: performing spatial rotation, mirror flipping or shearing on the skeleton sequence.

3. The method for human behavior recognition based on spatiotemporal decoupling of dynamic semantic guided mask according to claim 1, characterized in that: In step (2), the masking operation on the data-enhanced skeleton sequence in the spatial dimension based on the dynamic semantic guided masking mechanism includes: (2-1i) Calculate the degree centrality of each joint point in each frame in the action sequence, degree centrality Defined as: ,in Representation node The number of connected edges; (2-1ii) Add each node i to the set according to its degree centrality: ; N represents the total number of joint points in a single frame; (2-1iii) From A fixed number of joint points are sampled for masking.

4. The method for human behavior recognition based on spatiotemporal decoupling of dynamic semantic guided masks according to claim 1, characterized in that: In step (2), the masking operation on the skeleton sequence after data enhancement in the time dimension based on the dynamic semantic guided masking mechanism includes: (2-2i) Calculate the attention value of each time step in the skeleton sequence : ； in, is the number of channels; is the number of joints; is the number of human bodies; For the tth frame, the pth person, the Joints, The motion attention value of the coordinate channel; motion attention value , given a skeleton sequence , Indicates the t Frame complete skeleton, inter-frame motion difference of frame t Defined as: ; (2-1ii) Screening from bone sequences A minimum fixed number of frames will mask it.

5. The method for human behavior recognition based on spatiotemporal decoupling of dynamic semantic guided mask according to claim 1, characterized in that: Step (3) includes: (3-1) Extract the skeleton sequence after mask operation through the feature extraction unit Intermediate space-time representation of ; (3-2) Representation of intermediate spacetime Perform dimension transformation to obtain preliminary spatial features and preliminary time series characteristics ; The preliminary spatial features Projection into spatial embedding , the preliminary time series features Projection into temporal embedding ; (3-3) Further refine the decoupled spatial embedding based on the self-attention mechanism module and the feedforward module and time embedding , and obtain the spatial representation and time representation ; (3-4) The query encoder decouples the spatial representation by projection and time representation Converted into final spatial representation and the final time representation ,Will and Fusion introduces global perspective representation ; The momentum encoder decouples the spatial representation by projecting and time representation Converted into final spatial representation and the final time representation ,Will and Fusion introduces global perspective representation ; (3-5) Construct a quadruple-level contrast loss function structure for contrastive learning training: ； 、 、 and are the weight coefficients of the four-way loss respectively; represents contrast loss; During training, the query encoder parameters are updated via back-propagation, and the momentum encoder parameters are updated via the momentum strategy.

6. The method for human behavior recognition based on spatiotemporal decoupling of dynamic semantic guided mask according to claim 5, characterized in that: In step (3-2), the preliminary spatial features Projection into spatial embedding , the formula is: ； in, 、 is a trainable weight matrix, 、 is the bias vector, represents the activation function; The preliminary time series characteristics Projection into temporal embedding , the formula is: ； in, 、 is a trainable weight matrix, 、 is the bias vector, Represents the activation function.

7. The method for human behavior recognition based on spatiotemporal decoupling of dynamic semantic guided mask according to claim 5, characterized in that: Step (3-3) includes: (3-3i) Embedding space Projected into the query, key, and value spaces, the attention scores between joints are calculated by scaling the dot product and then normalized by Softmax: ； Represents the output feature matrix of the i-th attention head in the spatial dimension; 、 and are respectively the query, key and value matrices obtained by linear transformation, is the scaling factor; The spatial attention results are respectively compared with Combined, feature stability is enhanced through residual connection and layer normalization: ； is the refined spatial feature matrix; is the concatenated attention matrix; Processed by the fully connected layer; is the layer normalization function; Indicates the number of attention heads in the spatial dimension; ； is the layer normalization function; is a feedforward network, It is the maximum pooling operation; (3-3ii) Embedding time Projected into the query, key, and value spaces, the attention scores between joints are calculated by scaling the dot product and then normalized by Softmax: ； Represents the output feature matrix of the i-th attention head in the time dimension; 、 and are the query, key, and value matrices obtained through linear transformation, is the scaling factor; The spatial attention results are respectively compared with Combined, feature stability is enhanced through residual connection and layer normalization: ； is the refined spatial feature matrix; is the concatenated attention matrix; Processed by the fully connected layer; is the layer normalization function; Indicates the number of attention heads in the time dimension; ； is the layer normalization function; is a feedforward network, It is the maximum pooling operation.

8. The method for human behavior recognition based on spatiotemporal decoupling of dynamic semantic guided mask according to claim 5, characterized in that: In steps (3-5), the InfoNCE framework is used to calculate the sample contrast loss. u and v , the loss is defined as: ； in, for u and v The similarity score of , is the temperature parameter, are two eigenvectors u and v The inner product of Indicates that in the negative sample set In the calculation u With all negative samples The similarity scores between them are summed exponentially.

9. A temporal-spatial decoupling human behavior recognition device based on dynamic semantic guided mask, characterized in that: include: The data processing module extracts human skeleton sequence data from the original video and performs data enhancement on the skeleton sequence data; The dynamic semantics-guided masking module performs masking operations on the data-augmented skeleton sequence in both spatial and temporal dimensions based on the dynamic semantics-guided masking mechanism. In the spatial dimension, it masks joints whose centrality is higher than the threshold, and in the temporal dimension, it masks frames with significant motion changes. The contrastive learning training module feeds the masked skeleton sequence data into the query encoder and momentum encoder, respectively. Both the query encoder and momentum encoder include a feature extraction unit and a spatiotemporal parallel decoupling unit to obtain spatial and temporal representations, respectively, construct a cross-domain contrastive loss, and perform contrastive learning training. After the training, a human behavior recognition model is obtained; The human behavior recognition module inputs the video to be recognized into the human behavior recognition model to obtain the prediction results of the human behavior in the video to be recognized.

10. A spatiotemporal decoupling human behavior recognition device based on dynamic semantic guided mask, characterized in that: comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; The memory stores a human behavior recognition model; the video to be recognized is input into the human behavior recognition model, and the processor executes the computer program to predict the human behavior in the video to be recognized; The human behavior recognition model is obtained by training through the steps in the method according to any one of claims 1 to 8.

Images