Human behavior recognition method and system based on dynamic spatio-temporal modeling and semantic quantization
By combining global dynamic joint weighting and spatial graph attention branches with semantic quantization, the problems of insufficient joint importance allocation and codebook collapse in skeleton behavior recognition are solved, achieving adaptive focusing on discriminative motion patterns and stable discrete semantic understanding.
Patent Information
- Application Number
- CN202610921879.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-25
- Publication Date
- 2026-08-25
AI Technical Summary
Existing skeleton behavior recognition methods lack dynamic importance calibration of joints in the spatiotemporal dimension, making it difficult to adaptively allocate differentiated weights. They also lack discrete semantic understanding, resulting in insufficient ability to focus on discriminative motion patterns and the existence of codebook collapse problems.
A global dynamic joint weighting module and a spatial graph attention branch are used in parallel and collaboratively. The importance of joints is captured by time-adaptive weighting and spatial graph attention. The semantic quantization module discretizes the joint features into semantic indexes, constructs a semantic graph and fuses it with the physical skeleton graph. A dual-path allocation strategy is used for differentiable feature reconstruction and loss function optimization.
It achieves adaptive focusing on discriminative motion patterns, improves the interpretability and generalization ability of the model, alleviates the codebook collapse problem, and maintains the stability of end-to-end training.
Smart Images
Figure CN122637482A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of behavior recognition technology, and in particular relates to a method and system for human behavior recognition based on dynamic spatiotemporal modeling and semantic quantization. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Human behavior recognition has been widely applied in scenarios such as intelligent monitoring and medical rehabilitation. Early methods based on RGB video were easily affected by factors such as lighting and background, and had high computational costs. Methods based on skeleton data represent key points as a three-dimensional coordinate sequence, which has the advantages of small data volume and robustness to environmental changes, making it suitable for graph convolutional network modeling.
[0004] Existing skeleton behavior recognition methods mainly improve in two aspects: first, by adaptively adjusting the graph topology through learnable adjacency matrices or attention mechanisms to capture action dependencies beyond physical connections; and second, by enhancing temporal dynamic perception through multi-scale temporal convolution or self-attention. However, existing skeleton behavior recognition methods lack calibration of the dynamic importance of joints in the spatiotemporal dimensions, making it difficult to adaptively assign differentiated weights based on action samples, resulting in insufficient ability to focus on discriminative motion patterns.
[0005] Secondly, existing adaptive graph learning outputs continuous association weights, which cannot be mapped to the semantic question of "which action primitive to execute," lacking discretized semantic understanding and thus limiting interpretability and generalization ability. Furthermore, the introduction of discrete representation learning suffers from codebook collapse, and existing methods do not utilize discrete semantic indexes to guide graph structure construction, failing to achieve deep coupling between semantics and graph reasoning. Summary of the Invention
[0006] To overcome the shortcomings of the prior art, the present invention provides a method and system for human behavior recognition based on dynamic spatiotemporal modeling and semantic quantization.
[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: Firstly, a method for human behavior recognition based on dynamic spatiotemporal modeling and semantic quantization is disclosed, including: Obtain the human skeleton sequence and convert it into a spatiotemporal feature representation; Construct a Transformer network based on dynamic spatiotemporal modeling and semantic quantization, including a preprocessing layer, a progressive feature extraction layer, and a classifier; The progressive feature extraction layer consists of a shallow layer and a deep layer: the shallow layer adaptively allocates the spatiotemporal weights of joints through a global dynamic joint weighting module and outputs dynamically enhanced features; the deep layer uses a global dynamic joint weighting and semantic quantization module to discretize features into semantic indices using a learnable codebook, constructs a semantic graph based on index consistency through hard allocation and merges it with the physical skeleton graph, and performs differentiable feature reconstruction through soft allocation to output semantically enhanced features; the semantically enhanced features are then input into a classifier to complete recognition.
[0008] Furthermore, the global dynamic joint weighting operation is composed of a time-adaptive weighting branch and a spatial graph attention branch in parallel.
[0009] Furthermore, the time-adaptive weighted branch includes: performing global average pooling along the joint dimension to aggregate spatial information, then generating a channel-level temporal attention map through two layers of one-dimensional convolution and ReLU activation function and Sigmoid activation function, and multiplying it element-wise with the original features to achieve temporal calibration; performing average pooling and max pooling on the temporal calibration features along the time dimension respectively, and concatenating the two pooling results along the channel dimension to obtain the aggregated features.
[0010] Furthermore, the dynamic scoring network in the spatial graph attention branch consists of three 1×1 convolutions with progressively compressed channels, and outputs joint importance weights after passing through the tanh activation function.
[0011] Furthermore, the semantic quantization operation adopts a dual-path allocation strategy: the hard allocation path assigns each joint to the semantic prototype with the highest similarity, constructs a semantic map based on the prototype consistency of joints within the same frame, and the semantic map and the physical skeleton map are added together by learnable weights to form a fusion map; the soft allocation path generates a prototype probability distribution through temperature-scaled Softmax, obtains reconstructed features by weighted summation of codebook vectors, and uses a pass-through estimator to ensure gradient propagation.
[0012] Furthermore, the progressive feature extraction layer consists of four progressive stages. The first and second stages are shallow stages, which only perform global dynamic joint weighting operations. The third and fourth stages are deep stages, which perform both global dynamic joint weighting operations and semantic quantization operations.
[0013] Furthermore, it also includes updating the codebook using commitment loss and codebook loss, where commitment loss constrains encoder features to move closer to the codebook, and codebook loss constrains the codebook to move closer to the encoder feature distribution, and avoids conflict between the two by stopping gradient operator.
[0014] Secondly, a human behavior recognition system based on dynamic spatiotemporal modeling and semantic quantization is disclosed, including: Data acquisition module: Acquires human skeleton sequences and converts them into spatiotemporal feature representations; Model building module: Constructs a Transformer network based on dynamic spatiotemporal modeling and semantic quantization, including a preprocessing layer, a progressive feature extraction layer, and a classifier; The progressive feature extraction layer consists of a shallow layer and a deep layer: the shallow layer adaptively allocates the spatiotemporal weights of joints through the global dynamic joint weighting module and outputs dynamically enhanced features; the deep layer uses the global dynamic joint weighting and semantic quantization module to discretize the features into semantic indices using a learnable codebook, constructs a semantic graph based on index consistency through hard allocation and merges it with the physical skeleton graph, and performs differentiable feature reconstruction through soft allocation to output semantically enhanced features. Behavior recognition module: Input semantically enhanced features into the classifier to complete the recognition.
[0015] Thirdly, a computer device is disclosed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the steps of the method described above.
[0016] Fourthly, a computer-readable storage medium is disclosed having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.
[0017] The above one or more technical solutions have the following beneficial effects: In this invention, the global dynamic joint weighting module integrates a temporally adaptive weighting branch and a spatial graph attention branch in parallel. The temporally adaptive weighting branch highlights keyframes through channel-level temporal attention, and then captures both continuous motion trends and instantaneous peak events through complementary concatenation of average pooling and max pooling along the time dimension. The spatial graph attention branch captures global joint dependencies through self-attention, dynamically infers the saliency weight of each joint through a bottleneck scoring network, and enhances it using residual excitation. The combination of these two approaches enables the model to adaptively focus on discriminative spatiotemporal locations based on specific action samples, solving the problem of existing methods lacking sample adaptability in the allocation of spatiotemporal joint importance.
[0018] In this invention, the semantic quantization module quantizes continuous features of the skeleton into discrete action primitives, and constructs a semantic graph based on the prototype consistency of joints through hard path assignment. This semantic graph explicitly connects physically unconnected joints that participate in the same action primitive, capturing high-order joint collaboration relationships that traditional physical graphs cannot express, and overcoming the limitation of existing graph models that can only express pairwise joint relationships.
[0019] In this invention, a dual-path allocation strategy decouples discrete semantic graph construction from differentiable feature reconstruction. The hard-allocation path generates semantic indexes for graph construction, while the soft-allocation path generates probability distributions for feature reconstruction using temperature-scaled Softmax and employs a pass-through estimator to ensure gradient propagation. Simultaneously, the codebook is updated bidirectionally through commitment loss and codebook loss. This design maintains end-to-end training capability while introducing a discrete semantic structure, mitigating the codebook collapse problem common in existing vector quantization methods, and making discrete semantic learning more stable and efficient.
[0020] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0022] Figure 1 This is a diagram illustrating the algorithm framework of the human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization according to Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the global dynamic joint weighting mechanism of the human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization according to Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of the semantic quantization module of the human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization according to Embodiment 1 of the present invention. Detailed Implementation
[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0024] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.
[0025] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0026] Example 1 The inventors, in their research on skeleton behavior recognition, discovered that existing technologies typically improve in two directions. First, in topology learning, learnable adjacency matrices or attention mechanisms enable graph topology to adaptively adjust based on input samples, capturing action-specific dependencies beyond physical connections. Second, in temporal modeling, multi-scale temporal convolutions or self-attention mechanisms enhance the ability to perceive the temporal dynamics of actions. However, existing methods still suffer from the following three core shortcomings: First, there is a lack of dynamic calibration capabilities for the spatiotemporal importance of joints. In actual human movements, the importance of different joints varies significantly at different times. Existing methods typically treat all time frames and all joints equally, or simply aggregate features using a fixed physical graph or global learning graph. They fail to adaptively assign differentiated weights to different joints at different times based on specific action samples, resulting in insufficient ability to focus on discriminative motion patterns.
[0027] Second, while existing adaptive graph learning methods can output continuous association weights between joints, these continuous representations are inherently difficult to map to human-understandable action semantics. Continuous weights can indicate the strength of the association between two joints, but they cannot answer the semantic question of "what kind of action primitive is this joint performing." Human movement is essentially a combination of reusable action primitives. An ideal model should not only detect statistical correlations but also map each joint feature to a discrete action primitive category, thereby possessing stronger interpretability and generalization ability.
[0028] Third, existing methods face two key technical challenges when introducing discrete representation learning. The first is the codebook collapse problem, where only a small number of code vectors are effectively used during training, while the majority remain idle due to insufficient gradient updates, resulting in a significant waste of codebook capacity. The second is that existing methods treat quantization as an independent feature compression step, failing to utilize the semantic indexes generated by discretization to guide the construction of inter-joint graph structures, thus missing the opportunity to deeply couple action semantics with graph reasoning.
[0029] Based on this, such as Figure 1 As shown, this embodiment discloses a human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization, including: Step 1: Obtain the human skeleton motion recognition dataset and convert it into a spatiotemporal feature representation. This embodiment uses three publicly available datasets: NTU RGB+D 60, NTU RGB+D 120, or NW-UCLA, to extract the human skeleton structure from each frame of the image. Specifically, human joints are treated as nodes, and skeletal connections are treated as edges, constructing a graph structure representation, denoted as . ,in For a set of joint nodes, This represents the total number of joint nodes in each frame. Represents the edge set. The feature representation of the skeleton sequence is as follows: Where B is the batch size, C is the number of channels, T is the number of time frames, and V is the number of joint nodes contained in each frame.
[0030] For different skeleton datasets, in this embodiment, the number of joints per frame in the NTU RGB+D 60 dataset is... (24 joints per person in a two-person exercise), the joint dimensions of multiple people are combined into one. 1 node The number of joints per frame. For the number of people, the skeleton sequence was uniformly sampled to 64 frames, and the input channel dimension C was set to 96. NW-UCLA dataset single frame joint count. Input channel dimension .
[0031] Step 2: Construct a Transformer network based on dynamic spatiotemporal modeling and semantic quantization. The Transformer network includes a preprocessing layer, a progressive feature extraction layer, and a classifier.
[0032] The preprocessing layer consists of an initial mapping layer and a learnable position encoding. The initial mapping layer maps the original joint coordinate features to a high-dimensional feature space and injects position index information into each spatiotemporal position to generate initial spatiotemporal features.
[0033] The progressive feature extraction layer consists of multiple processing stages from shallow to deep. The shallow stage only performs global dynamic joint weighting operation, while the deep stage performs both global dynamic joint weighting operation and semantic quantization operation. Specifically, the progressive feature extraction layer includes four progressive stages: the first stage, the second stage, the third stage, and the fourth stage. The first and second stages focus on learning basic skeleton features and include a global dynamic joint weighting module, which is used to perform global dynamic joint weighting operations. It highlights keyframes through a time-adaptive weighting branch and infers joint importance through a spatial graph attention branch, thereby achieving the initial processing from basic spatiotemporal features to dynamic saliency focus.
[0034] In this embodiment, as Figure 2 As shown, the global dynamic joint weighting module is composed of two parts: a time-adaptive weighting branch and a spatial graph attention branch, which work together to achieve dynamic spatiotemporal enhancement of skeleton features.
[0035] For input features ,in For batch size, For the number of channels, For time frames, The number of joints per frame. To determine the number of people, the joint dimensions of multiple individuals are first merged and reshaped into... To ensure uniform processing of multiple skeletons.
[0036] The time-adaptive weighted branch highlights keyframes through channel-level temporal attention and captures continuous motion trends and instantaneous peak events through average pooling and max pooling along the time dimension. Specific implementations include: First, global average pooling is performed along the joint dimension to aggregate the spatial information of all joints while preserving the temporal structure:
[0037] Subsequently, adaptive average pooling is performed along the time dimension on the pooling result to compress the time information and obtain a global time feature description. The data is then fed into the skeleton keyframe compression excitation module, which consists of two layers of one-dimensional convolutions and an intermediate ReLU activation function. Finally, a channel-level temporal attention map is generated using the Sigmoid activation function.
[0038] in It is the Sigmoid activation function. Use ReLU activation function. Apply channel-level temporal attention plots. Broadcast the data and multiply it element-wise with the original features to obtain the time-calibrated features:
[0039] in This represents element-wise multiplication. This operation amplifies feature channels that are strongly correlated with the current action and suppresses irrelevant channels based on the characteristics of the action, thereby achieving adaptive enhancement of keyframes.
[0040] Based on this, average pooling and max pooling are performed on the temporal calibration features along the time dimension to capture the continuous motion trend. With instantaneous peak motion : ,
[0041] The two pooling results are concatenated along the channel dimension to obtain an aggregated feature that takes into account both stationary and abrupt motion information. :
[0042] Reshape the spliced features into This serves as the input for the subsequent spatial graph attention branch.
[0043] The spatial graph attention branch captures global joint dependencies through self-attention, then dynamically infers the importance weights of each joint via a bottleneck scoring network, and enhances the original features using residual excitation. Specific implementations include: Aggregation features First, the similarity between all joint pairs is calculated using scaled dot product self-attention to generate a globally perceptive joint representation. The self-attention calculation is as follows:
[0044]
[0045] scaling factor To prevent gradient saturation, This is a globally perceptible feature. The updated feature of each joint is a weighted sum of the original features of all joints, enabling spatially distant but functionally related joints to interact directly.
[0046] Global awareness features Reshaping The data is then fed into a dynamic scoring network. This network consists of three 1×1 convolutional layers with progressively reduced dimensionality, followed by batch normalization and a ReLU activation function after each layer.
[0047] The first convolutional layer converts the channels from Compress to The second layer is further compressed to The third layer projects onto a single channel, outputting the original importance score for each joint. Finally, the weights are constrained to a value between 1 and 2 using the tanh activation function. Within the range of 1,1):
[0048] The specific implementation of residual stimulus enhancement includes: The joint importance weights are applied to the original temporal calibration features using residual excitation, resulting in dynamically enhanced features with adaptive spatiotemporal joint weighting.
[0049] in This is element-wise multiplication. When When the corresponding joint features are magnified, it indicates that the joint is of high importance to the current action determination; when When this occurs, the characteristics of the corresponding joint are suppressed, indicating that the joint contributes little to the current action or is irrelevant noise. Residual connection This ensures that the original feature information is preserved even when the weights are zero, and also facilitates gradient flow during backpropagation.
[0050] The third and fourth stages include a global dynamic joint weighting module and a semantic quantization module. Based on dynamic spatiotemporal modeling, discrete semantic structuring is further introduced. After constructing a semantic graph through hard allocation and fusing it with the physical graph, graph convolutional inference is performed to achieve deep feature refinement from dynamic focusing to semantic understanding.
[0051] like Figure 3 As shown, the semantic quantization module uses a dual-path allocation strategy to map continuous features to a learnable motion prototype codebook, thereby achieving discrete semantic modeling of skeleton features.
[0052] Augmentation features for input First, reshape it into a two-dimensional matrix. ,in The total number of joint features across all spatiotemporal locations. Construct a learnable codebook. ,storage prototype vectors Each prototype This represents a motion primitive. For each joint feature vector... With each prototype Calculate the cosine similarity:
[0053] Among them, the similarity matrix The semantic affinity between each joint feature and each action primitive prototype was captured, laying the foundation for subsequent dual-path assignment.
[0054] In the semantic quantization module, hard path assignment assigns each joint to the semantic prototype with the highest similarity. A semantic map is constructed based on the prototype consistency of joints within the same frame. The semantic map and the physical skeleton map are added together using learnable weights to form a fusion map. The specific implementation is as follows: For each joint feature The prototype index with the highest similarity is selected as its discrete semantic label:
[0055] When two joints in the same frame and When assigned to the same prototype, they are determined to participate in the same action primitive, and a connection is established in the semantic graph. This leads to the construction of a semantic adjacency matrix:
[0056] in Iverson (in parentheses) and Each represents a batch The Middle Frame number The joint and the first Each joint is assigned a discrete prototype index; when two indices are the same, meaning the two joints are assigned to the same action primitive prototype, the value of the parentheses is 1, establishing a connection in the semantic graph; otherwise, the value is 0, indicating that the two joints are not semantically related. The semantic adjacency matrix constructed in this way only has edges at joint pairs that share the same prototype index, thus explicitly connecting physically unconnected joints that participate in the same action primitive.
[0057] Semantic graph With physical skeleton diagram By fusing learnable weights, a fused adjacency matrix is obtained for graph convolution:
[0058] The physical skeleton diagram mentioned above Based on the predefined natural connections of the human skeleton, its construction method is as follows: For each frame... For each joint node, define an adjacency matrix. If the joint With joints If there is a skeletal connection between them, meaning they are directly connected by bones in their physiological structure, then the corresponding matrix elements... If there is no direct skeletal connection, then At the same time, considering the bidirectional nature of human movement, let It is a symmetric matrix, that is This matrix remains fixed during training and does not participate in parameter updates.
[0059] The learnable fusion weights are dynamically predicted based on input features using a lightweight multilayer perceptron, enabling the model to adaptively balance the contributions of physical and semantic connections according to the characteristics of each sample. A stage-specific scaling factor is used, with a smaller value in shallow stages to maintain dependence on physical connections, and a gradually increasing value in deeper stages to enhance the semantic graph's ability to model higher-order joint relationships. Together, these factors allow the fused graph to retain the inherent skeletal connection priors of the human body while flexibly introducing action semantic associations that transcend physical connections.
[0060] In the semantic quantization module, the soft-assigned path generates a prototype probability distribution using temperature-scaled Softmax, reconstructs the features by weighted summation of the codebook vectors, and employs a pass-through estimator to ensure gradient propagation. The specific implementation is as follows: Since the argmax operation in the hard-assigned path is non-differentiable, hindering gradient propagation, a soft-assignment strategy is employed in parallel to provide a differentiable quantization approximation. This is achieved through temperature parameters. Scaled softmax transforms cosine similarity into a probability distribution for each joint across all prototypes:
[0061] in Indicating joint features Assigned to prototype The probability satisfies Temperature parameters Control the sharpness of the distribution. The larger the value, the more concentrated the probability distribution is on the prototype with the highest similarity; The smaller the value, the smoother the distribution and the more balanced the weights of each prototype.
[0062] A dynamic adjustment strategy is used during training. By gradually decaying the initial value, the model explores the codebook structure with a smoother distribution in the early stages of training and approximates the hard allocation result with a sharper distribution in the later stages of training, thus ensuring the stability of codebook learning.
[0063] The reconstructed features are obtained by weighted summation of the codebook vectors based on the probability distribution:
[0064] To achieve end-to-end training while preserving the discrete semantic index, a pass-through estimator is used to combine the reconstructed features with the original features:
[0065] in This is the stopping gradient operator. Its function is to calculate the features at their original values during the forward propagation, i.e., the output... It itself; during the backpropagation process, the part in parentheses Setting the gradient to zero allows the gradient to be directly passed to the gradient. .
[0066] Therefore, forward computation uses quantized reconstructed features to introduce discrete semantics, while backward computation transmits gradients losslessly to the encoder, ensuring end-to-end training of the model. Thus, the hard-assignment path is responsible for generating discrete semantic indexes for graph construction, while the soft-assignment path is responsible for providing smooth gradient signals for end-to-end training. The two paths each perform their respective functions, jointly ensuring the functional integrity of semantic quantization and training stability.
[0067] Step 3: Input the semantically enhanced features into the classifier to complete behavior recognition.
[0068] The semantically enhanced features, after being processed in four stages, are fed into the classifier. The prediction probability of the action category is generated through a fully connected layer. During the model training process, a joint loss function is used to optimize the parameters. This joint loss function consists of classification cross-entropy loss and quantization loss.
[0069] The specific implementation of the classification cross-entropy loss is as follows: given the final classification prediction value output by the classification head. and the action realism label The classification losses are as follows:
[0070] Quantization loss comprises commitment loss and codebook loss, balancing classification task accuracy with discrete semantic representation learning. The specific implementation of quantization loss includes updating the codebook during training using both commitment loss and codebook loss, with both employing mean absolute error loss.
[0071] The two complement each other: commitment loss Encoder features Pull to codebook This causes the encoder to "commit" to a discrete prototype; codebook loss codebook Pull to encoder features This aligns the codebook and encoder distributions. The stopping gradient operator prevents the two objectives from interfering with each other, achieving stable collaborative learning. The total quantized loss is:
[0072] in Balance the contributions from both sides.
[0073] To monitor the health of the codebook, calculate the perplexity of soft allocation:
[0074] in Average usage for each prototype. The closer the confusion level is... The more uniform the codebook utilization and the lower the perplexity, the greater the risk of codebook collapse.
[0075] In summary, the total loss function for model training is defined as the weighted sum of the classification loss and the quantization loss:
[0076] in, For classification cross-entropy loss, To quantify the loss, The hyperparameter is used to balance the two types of losses and adjust the contribution of quantization loss to the total loss.
[0077] As an optional implementation, in order to make full use of the various motion representations of skeleton data, a multimodal fusion strategy is adopted. Joint coordinates, skeleton vectors, joint motion differences and skeleton motion differences are used as four input modalities. Each modality is trained independently as a model branch. The Softmax prediction scores of the four branches are added together to obtain the final fusion score, and the behavior category prediction result is output.
[0078] Example 2 Based on the method described in Implementation 1, the purpose of this embodiment is to provide human behavior recognition based on dynamic spatiotemporal modeling and semantic quantization, including: Data acquisition module: Acquires human skeleton sequences and converts them into spatiotemporal feature representations; Model building module: Constructs a Transformer network based on dynamic spatiotemporal modeling and semantic quantization, including a preprocessing layer, a progressive feature extraction layer, and a classifier; The progressive feature extraction layer consists of a shallow layer and a deep layer: the shallow layer adaptively allocates the spatiotemporal weights of joints through the global dynamic joint weighting module and outputs dynamically enhanced features; the deep layer uses the global dynamic joint weighting and semantic quantization module to discretize the features into semantic indices using a learnable codebook, constructs a semantic graph based on index consistency through hard allocation and merges it with the physical skeleton graph, and performs differentiable feature reconstruction through soft allocation to output semantically enhanced features. Behavior recognition module: Input semantically enhanced features into the classifier to complete the recognition.
[0079] Example 3 The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.
[0080] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium.
[0081] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above-described method.
[0082] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.
[0083] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.
[0084] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization, characterized in that, include: Obtain the human skeleton sequence and convert it into a spatiotemporal feature representation; A Transformer network based on dynamic spatiotemporal modeling and semantic quantization is constructed, including a preprocessing layer, a progressive feature extraction layer, and a classifier; The progressive feature extraction layer consists of a shallow layer and a deep layer: the shallow layer adaptively allocates the spatiotemporal weights of joints through the global dynamic joint weighting module and outputs dynamically enhanced features; the deep layer uses the global dynamic joint weighting and semantic quantization module to discretize the features into semantic indices using a learnable codebook, constructs a semantic graph based on index consistency through hard allocation and merges it with the physical skeleton graph, and performs differentiable feature reconstruction through soft allocation to output semantically enhanced features. The semantically enhanced features are input into the classifier to complete the recognition.
2. The human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization as described in claim 1, characterized in that, The global dynamic joint weighting operation consists of a time-adaptive weighting branch and a spatial graph attention branch operating in parallel.
3. The human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization as described in claim 2, characterized in that, The time-adaptive weighted branch includes: performing global average pooling along the joint dimension to aggregate spatial information, then generating a channel-level temporal attention map through two layers of one-dimensional convolution and ReLU activation function and Sigmoid activation function, and multiplying it element-wise with the original features to achieve temporal calibration; performing average pooling and max pooling on the temporal calibration features along the time dimension respectively, and concatenating the two pooling results along the channel dimension to obtain the aggregated features.
4. The human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization as described in claim 2, characterized in that, The dynamic scoring network in the spatial graph attention branch consists of three 1×1 convolutions with progressively compressed channels, and outputs joint importance weights after passing through the tanh activation function.
5. The human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization as described in claim 1, characterized in that, The semantic quantization operation adopts a dual-path allocation strategy: the hard allocation path assigns each joint to the semantic prototype with the highest similarity, and a semantic map is constructed based on the prototype consistency of joints within the same frame. The semantic map and the physical skeleton map are added together by learnable weights to form a fusion map. The soft-assignment path generates a prototype probability distribution through temperature-scaled Softmax, obtains reconstructed features by weighted summation of codebook vectors, and uses a pass-through estimator to ensure gradient propagation.
6. The human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization as described in claim 1, characterized in that, The progressive feature extraction layer consists of four progressive stages. The first and second stages are shallow stages, which only perform global dynamic joint weighting operations. The third and fourth stages are deep stages, which perform both global dynamic joint weighting operations and semantic quantization operations.
7. The human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization as described in claim 1, characterized in that, It also includes updating the codebook using commitment loss and codebook loss, where commitment loss constrains encoder features to move closer to the codebook, and codebook loss constrains the codebook to move closer to the encoder feature distribution, and avoids the conflict between the two by stopping gradient operator.
8. A human behavior recognition system based on dynamic spatiotemporal modeling and semantic quantization, characterized in that, include: Data acquisition module: Acquires human skeleton sequences and converts them into spatiotemporal feature representations; Model building module: Constructs a Transformer network based on dynamic spatiotemporal modeling and semantic quantization, including a preprocessing layer, a progressive feature extraction layer, and a classifier; The progressive feature extraction layer consists of a shallow layer and a deep layer: the shallow layer adaptively allocates the spatiotemporal weights of joints through the global dynamic joint weighting module and outputs dynamically enhanced features; the deep layer uses the global dynamic joint weighting and semantic quantization module to discretize the features into semantic indices using a learnable codebook, constructs a semantic graph based on index consistency through hard allocation and merges it with the physical skeleton graph, and performs differentiable feature reconstruction through soft allocation to output semantically enhanced features. Behavior recognition module: Input semantically enhanced features into the classifier to complete the recognition.
9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the human behavior recognition method based on dynamic spatiotemporal modeling and semantic quantization as described in any one of claims 1-7.