A Timing-Smoothed Human Motion Capture Method and System Guided by a Calorific Value Map

Abstract

This application belongs to the fields of computer vision and human motion capture technology, and discloses a temporal smoothing human motion capture method and system guided by heat maps, aiming to solve the deficiencies of existing methods in temporal consistency and structural stability. This method receives a sequence containing RGB video frames, extracts 2D poses and generates heat maps; fuses the temporal features of RGB and heat maps through a dual-branch temporal attention block (DBTA) to capture temporal action cues; proposes an intra-frame pose reasoning (IPR) module, which introduces a graph attention mechanism to enhance the modeling of the structural relationship between joint points within a single frame to ensure structurally coherent pose prediction. This method significantly improves the accuracy and temporal smoothness of 3D human pose estimation.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and human motion capture, and particularly relates to a method and system for temporal smoothing of human motion capture guided by heat maps. Background Art

[0002] Three-dimensional human mesh recovery is one of the core technologies in the fields of computer vision, human-computer interaction, and intelligent monitoring. Accurate and stable three-dimensional human mesh recovery is of great significance for improving the naturalness of interactive systems, enhancing virtual reality experiences, and promoting medical rehabilitation and sports analysis.

[0003] With the development of deep learning, three-dimensional human mesh recovery methods based on single-frame images have shown good performance. However, these methods mainly rely on single-frame features for prediction and fail to effectively utilize the temporal information of human motions, resulting in problems such as jitter and instability in the estimation results. In contrast, three-dimensional human mesh recovery methods based on video streams improve the prediction stability to a certain extent by modeling temporal relationships. However, existing methods still have deficiencies in capturing long-term and short-term temporal dependencies and have poor adaptability to complex motions and perspective changes. For example, some researchers use a pre-trained CNN network to extract static features for each frame and then train a temporal network to predict SMPL parameters. TCMR uses Transformer to extract the temporal information of static features; MPS-Net proposes motion persistent attention to extract the temporal information of static features. However, these methods rely on static features extracted by CNN and do not fully utilize the dynamic features of human motions, thus having certain limitations in capturing dynamic changes and details.

[0004] In addition, some researchers have made innovative attempts in SMPL parameter regression. For example, MAED pays attention to temporal information through a self-attention mechanism and proposes a KTD structure. This method regresses the pose information of child nodes from parent nodes step by step based on the kinematic tree. However, this method does not fully consider the dynamic dependencies between joint points inside the kinematic structure, and all child nodes default to having the same weight coefficient as the parent node, which limits the model's ability to capture complex human postures.

[0005] In three-dimensional pose estimation based on video streams, significant achievements have been made in learning 2D pose information. However, how to use 2D pose information as a prior condition for the task of temporal three-dimensional human mesh recovery and further improve the accuracy and temporal smoothness of pose estimation remains a challenge.

[0006] Therefore, how to improve the temporal smoothness and enhance the structural consistency while improving the accuracy of human pose estimation remains a key challenge in current research. Summary of the Invention

[0007] The present invention aims to solve the problems of inconsistent timing, unstable structure, and failure to fully utilize dynamic features and structural dependencies in existing human motion capture methods based on video streams, and provides a timing-smoothing human motion capture method and system guided by heat maps.

[0008] In a first aspect, the present invention provides a timing-smoothing human motion capture method guided by heat maps, including the following steps:

[0009] S1: Obtain an RGB video sequence containing multiple frames of images;

[0010] S2: Extract 2D human joint position information from each frame of the RGB video sequence, and generate a corresponding heat map sequence based on the 2D human joint position information;

[0011] S3: Respectively perform feature extraction on the RGB video sequence and the heat map sequence to obtain the timing features of the RGB video sequence and the timing features of the heat map sequence;

[0012] S4: Use a dual-branch timing attention block to fuse the timing features of the RGB video sequence and the timing features of the heat map sequence to obtain fused timing features, where the dual-branch timing attention block includes a multi-head self-attention mechanism and a multi-head cross-attention mechanism;

[0013] S5: Regress SMPL model parameters according to the fused timing features, where the SMPL model parameters include pose parameters, body shape parameters, and camera parameters. Among them, the step of regressing the pose parameters includes using an intra-frame pose inference module to model the structural dependencies between the three-dimensional human joints corresponding to the pose parameters through a graph attention mechanism, and optimizing the pose parameters based on the structural dependencies;

[0014] S6: Output three-dimensional human motion capture data according to the regressed SMPL model parameters.

[0015] As an optional implementation manner of the first aspect of the present application, the step S2 specifically includes: obtaining 2D human joint position information in the RGB video sequence through a 2D pose extractor ; based on the 2D human joint position information, use Gaussian distribution spots to generate a heat map sequence, and the value of the heat map sequence at a position is calculated according to the following formula: , where represents the value of the heat map sequence at the position , and represent the positions of each pixel on the grid, is the standard deviation that controls the diffusion range of the Gaussian distribution.

[0016] As an alternative implementation of the first aspect of the present application, in the step S4, obtaining the fused temporal feature using the dual-branch temporal attention block specifically includes: linearly projecting the temporal features of the RGB video sequence and the temporal features of the heat map sequence into the same low-dimensional feature space respectively; adding learnable temporal position encodings to the temporal features of the mapped RGB video sequence and the temporal features of the heat map sequence respectively; calculating the temporal attention matrix of the temporal features of the mapped RGB video sequence and the temporal attention matrix of the temporal features of the mapped heat map sequence using the multi-head self-attention mechanism to capture the temporal information within each modality; establishing a potential association between the temporal features of the mapped RGB video sequence and the temporal features of the mapped heat map sequence using the multi-head cross-attention mechanism to enhance cross-modal information interaction; performing weighted fusion on the temporal features of the fused RGB video sequence and the heat map sequence using a linear layer and an adaptive weight factor to obtain the fused temporal feature; wherein, the adaptive weight factor is calculated according to the following formula: , where the adaptive weight factor , represents the length of the time series, represents a learnable linear transformation matrix, represents the temporal feature of the RGB video sequence and the temporal feature of the heat map sequence the concatenation operation between modalities; the fused temporal feature is calculated according to the following formula: .

[0017] As an alternative implementation of the first aspect of the present application, in the step S5, using the graph attention mechanism of the intra-frame pose inference module to model the structural dependence between the three-dimensional human joints corresponding to the pose parameters specifically includes: regarding the three-dimensional human joints corresponding to the pose parameters as nodes in the graph; transforming the features of the nodes by learning a trainable weight matrix through the graph attention mechanism; calculating the association strength between the joint nodes through the graph attention mechanism to obtain a graph attention matrix, and the association strength between node and node is calculated according to the following formula: , where, represents the transpose of the shared attention vector, represents the pose node i and k the concatenation operation; pruning the unconnected nodes according to the graph attention matrix.

[0018] As an alternative implementation of the first aspect of the present application, in step S5, when using the graph attention mechanism of the intra-frame pose inference module to model the structural dependence between the three-dimensional human joints corresponding to the pose parameters, it further includes: for each joint node , performing softmax function normalization on the set of corresponding parent nodes in the preset kinematic tree structure. The sum of the attention weights of the set of parent nodes is 1, and the normalized attention weights are calculated according to the following formula: ; based on the topological relationship of the kinematic tree structure and the normalized attention weights, calculating the local pose parameters of the joint node . The local pose parameters are derived by calculating the weight distribution coefficient of the joint node relative to its set of parent nodes, and performing a linear combination of the global pose parameters of the parent nodes and the corresponding edge weights . The calculation method is as shown in the following formula: .

[0019] As an alternative implementation of the first aspect of the present application, in step S5, optimizing the pose parameters based on the structural dependence specifically includes: using a multi-layer perceptron to model the pose parameters of the final SMPL model in combination with the fused temporal features and the calculated local pose parameters.

[0020] As an alternative implementation of the first aspect of the present application, the loss function adopted during the training process of the method includes: a 2D joint position loss calculated based on the difference between the predicted 2D joint positions and the true 2D joint positions; a 3D joint position loss calculated based on the difference between the predicted 3D joint positions and the true 3D joint positions; an SMPL parameter loss calculated based on the difference between the predicted SMPL pose parameters and body shape parameters and the true SMPL parameters; a temporal smoothness loss imposed on the global fused features output by the double-branch temporal attention block to penalize the drastic changes in features between adjacent frames.

[0021] In a second aspect, an embodiment of the present application provides a calorific value map-guided temporal smooth human motion capture system, including:

[0022] An input module configured to obtain an RGB video sequence including multiple frames of images;

[0023] A 2D pose extraction and calorific value map generation module configured to extract 2D human joint position information from each frame of the RGB video sequence and generate a corresponding calorific value map sequence based on the 2D human joint position information;

[0024] A feature extraction module, configured to extract features from the RGB video sequence and the heat value map sequence respectively, to obtain the temporal features of the RGB video sequence and the temporal features of the heat value map sequence;

[0025] A dual-branch temporal attention module, configured to use dual-branch temporal attention blocks to fuse the temporal features of the RGB video sequence and the temporal features of the heat value map sequence to obtain fused temporal features, where the dual-branch temporal attention blocks include a multi-head self-attention mechanism and a multi-head cross-attention mechanism;

[0026] An SMPL regression module, configured to regress SMPL model parameters according to the fused temporal features, where the SMPL model parameters include pose parameters, body shape parameters, and camera parameters. Among them, the step of regressing the pose parameters includes using an intra-frame pose inference module to model the structural dependencies between the three-dimensional human joints corresponding to the pose parameters through a graph attention mechanism, and optimizing the pose parameters based on the structural dependencies;

[0027] An output module, configured to output three-dimensional human motion capture data according to the regressed SMPL model parameters.

[0028] In a third aspect, an embodiment of the present application provides an electronic device, where the electronic device includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0029] In a fourth aspect, an embodiment of the present application provides a readable storage medium, where a program or instruction is stored on the readable storage medium. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0030] Compared with the prior art, the beneficial effects of the present invention include but are not limited to:

[0031] 1. By introducing 2D pose information and converting it into a heat value map as prior knowledge, the accuracy of joint position estimation is enhanced, providing a reliable basis for subsequent three-dimensional pose estimation.

[0032] 2. The proposed dual-branch temporal attention blocks can effectively fuse RGB features and heat value map features, capture the temporal information of both modalities at the same time, and improve temporal consistency and feature expression ability through adaptive fusion.

[0033] 3. The proposed intra-frame pose inference module uses a graph attention mechanism to dynamically model the structural relationships of joint points within a single frame, overcoming the limitations of traditional methods that rely on a fixed kinematic tree structure, and enhancing the structural coherence and accuracy of pose prediction.

[0034] 4. Through the above technical combination, the accuracy and temporal smoothness of 3D human pose estimation are significantly improved, enabling better handling of complex movements and perspective changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 FIG. is a flowchart of a proposed method for temporally smooth human action capture guided by heat value maps;

[0036] Figure 2 FIG. is a schematic diagram of the overall technical framework of the proposed method for temporally smooth human action capture guided by heat value maps;

[0037] Figure 3 FIG. is a structural diagram of the proposed dual-branch temporal attention block (DBTA);

[0038] Figure 4 FIG. is a schematic diagram of iterative SMPL model regression based on the motion tree structure;

[0039] Figure 5 FIG. is a detailed diagram of the proposed in-frame pose reasoning (IPR) module;

[0040] Figure 6 FIG. is a structural diagram of a system for temporally smooth human action capture guided by heat value maps provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0042] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the specification and claims means at least one of the connected objects. The character " / " generally represents an "or" relationship between the associated objects before and after. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined.

[0043] Embodiment 1

[0044] Please refer to Figure 1, which is the implementation flowchart of a time-series smoothing human motion capture method guided by a calorific value map provided by an embodiment of the present invention. Figure 2 It is a schematic diagram of the overall technical framework of the proposed time-series smoothing human motion capture method guided by a calorific value map. The method includes the following steps:

[0045] S1: Obtain an RGB video sequence containing multiple frames of images.

[0046] S2: Extract the 2D human joint position information for each frame of the RGB video sequence, and generate a corresponding calorific value map sequence based on the 2D human joint position information.

[0047] First, an RGB video containing multiple frames is given as input. For each frame in the video sequence, the method extracts its 2D human joint position information. Specifically, the method adopts an existing top-down method to obtain the 2D human joint position information. For example, a pre-trained 2D pose extractor is used to obtain the position information of human joints, and this information is represented as , where represents the number of human joints, depending on the selected 2D pose extractor.

[0048] Next, Gaussian distribution spots are constructed based on the extracted 2D human joint position information and propagated throughout the image through spatial diffusion to generate the corresponding calorific value map . For the given 2D pose joint position , the generation process of the calorific value map can be represented by formula (1):

[0049]

[0050] where represents the value of the calorific value map sequence at position , and represent the positions of each pixel on the grid, is the standard deviation that controls the diffusion range of the Gaussian distribution.

[0051] In the above way, the method converts the key feature points extracted from the 2D pose information into calorific value image information, which is used as prior knowledge for subsequent modal feature fusion.

[0052] S3: Extract features from the RGB video sequence and the calorific value map sequence respectively to obtain the time-series features of the RGB video sequence and the time-series features of the calorific value map sequence.

[0053] In this embodiment, the time-series features of the RGB feature map and the calorific value map feature map are extracted through a convolutional neural network (CNN) and a pose network (Pose Net) respectively, which are represented as: , where , represents the number of channels of the feature map, and

[0054] Specifically, the generated heat map is then input into the Pose Net, which is a lightweight feature extraction network composed of convolutional layers and pooling layers, aiming to learn the key pose information in the heat map, so as to obtain the temporal features of the heat map sequence, denoted as . At the same time, the RGB video sequence extracts its temporal features through a convolutional neural network (CNN), denoted as .

[0055] S4: Use the dual-branch temporal attention block (DBTA) to fuse the temporal features of the RGB video sequence and the temporal features of the heat map sequence to obtain the fused temporal features. The DBTA includes a multi-head self-attention (MHSA) mechanism and a multi-head cross-attention (MCA) mechanism.

[0056] In this embodiment, the dual-branch temporal attention block (DBTA) is used to obtain the fused temporal features, specifically including: linearly projecting the temporal features of the RGB video sequence and the temporal features of the heat map sequence into the same low-dimensional feature space respectively; adding learnable time position encodings to the temporal features of the projected RGB video sequence and the temporal features of the heat map sequence respectively; using the multi-head self-attention (MHSA) mechanism to calculate the temporal attention matrices of the temporal features of the projected RGB video sequence and the temporal features of the projected heat map sequence to capture the temporal information within each modality; using the multi-head cross-attention (MCA) mechanism to establish a potential association between the temporal features of the projected RGB video sequence and the temporal features of the projected heat map sequence to enhance cross-modal information interaction; using a linear layer and an adaptive weight factor to perform weighted fusion on the temporal features of the fused RGB video sequence and the temporal features of the heat map sequence to obtain the fused temporal features.

[0057] Specifically, Figure 3 shows the model architecture of the proposed dual-branch temporal attention block (DBTA), aiming to learn the temporal consistency between the RGB image and the heat map. First, a linear projection method is used to project the input temporal features of the RGB video sequence and the temporal features of the heat map sequence and into the same low-dimensional feature space, with the aim of ensuring the dimensional alignment of the two features. Subsequently, the learnable time position encoding represents the length of the time series, is the dimension of the features. The purpose of this step is to introduce temporal position information so that the model can capture the dynamic features that change over time. The core of the dual-branch temporal attention block is to establish an image relationship model across time frames. DBTA adopts the multi-head self-attention (MHSA) mechanism to calculate the temporal attention matrix of, which is used to capture the temporal information between different time frames within the same batch.

[0058] Although and can capture certain temporal information within their respective feature spaces, they still lack direct cross-modal interaction capabilities and cannot fully integrate information from different modalities. Therefore, to further enhance the information interaction between different modal features, this method introduces the cross-attention mechanism (Multi-head Cross Attention, MCA) to establish and the potential association between the two modalities. Its mechanism is similar to MHSA. By using the matrix to calculate the vectors of their respective modalities, an association is established between different modal features, thereby effectively enhancing the global feature representation ability. As shown in Equation (2):

[0059]

[0060] where , L represents the depth of the DBTA block. Since and belong to different feature spaces respectively, direct fusion may lead to information loss. Therefore, this method uses a linear layer to fuse and through the adaptive weight factor to control the contribution of different features in the final representation. The calculation method of the weight is shown in Equation (3):

[0061]

[0062] where the adaptive weight factor , represents the length of the time series, represents the learnable linear transformation matrix, represents the temporal features of the RGB video sequence and the temporal features of the heat map sequence the concatenation operation between modalities. Finally, this method uses the calculated weights and Perform weighted summation of the two features to obtain a global feature representation with better temporal consistency and modal fusion capabilities. , which is calculated as shown in formula (4):

[0063]

[0064] S5: Regress the SMPL model parameters based on the fused temporal features. The SMPL model parameters include pose parameters, body parameters, and camera parameters. The step of regressing the pose parameters includes using the intra-frame pose reasoning (IPR) module to model the structural dependencies between the 3D human joints corresponding to the pose parameters through the graph attention mechanism, and optimize the pose parameters based on the structural dependencies.

[0065] S6: Output 3D human motion capture data based on the regressed SMPL model parameters.

[0066] In this embodiment, the intra-frame pose reasoning (IPR) module graph attention mechanism is used to model the structural dependency between the three-dimensional human joints corresponding to the pose parameters, specifically including: treating the three-dimensional human joints corresponding to the pose parameters as nodes in the graph; transforming the features of the nodes through learning a trainable weight matrix through the graph attention mechanism; and calculating the association strength between the joint nodes through the graph attention mechanism to obtain the graph attention matrix.

[0067] Furthermore, the intra-frame pose reasoning (IPR) module graph attention mechanism is used to model the structural dependencies between the 3D human joints corresponding to the pose parameters, including: for each joint node , its corresponding parent node set in the preset kinematic tree structure The softmax function is normalized on the parent node, and the sum of the attention weights of the parent node set is 1; based on the topological relationship of the kinematic tree structure and the normalized attention weight, the joint nodes are calculated. Local posture parameters, local posture parameters By calculating the joint nodes The weight distribution coefficient of the parent node set relative to the global posture parameter of the parent node and the corresponding edge weight It is derived by linear combination.

[0068] In this embodiment, the posture parameters are optimized based on the structural dependency, specifically including: using a multi-layer perceptron to combine the fused temporal features and the calculated local posture parameters to model the posture parameters of the final SMPL model.

[0069] In 3D human motion capture, using a regressor to obtain SMPL parameters For recovering the pose and shape information of a human body from image data, where represents the pose parameters of the SMPL joints, represents the shape parameters, represents the camera parameters for projecting 3D coordinates into the 2D space. Among them, the pose parameters directly determine the rotation relationship of the skeleton joints. Therefore, the accuracy of its recovery affects the overall accuracy of 3D human mesh recovery to a great extent. Existing methods, such as HMR, usually adopt the image features extracted by a deep neural network to directly regress the relative rotation parameters of the joint points to predict the SMPL pose parameters of the human body, as shown in formula (5):

[0070]

[0071] where represents the regression function, is represented by the 6D rotation of 24 joint points (including 23 joint points and a root node). However, although this end-to-end deep learning-based regression method can achieve the estimation of 3D human information, it often ignores the mutual correlation between joint points when modeling the human body structure, especially the hierarchical dependence relationship of human motion. To address this problem, He et al. proposed a step-by-step regression method based on a kinematic tree. This method hierarchically models the joint nodes according to the hierarchical structure of human motion to enhance the structural rationality of pose prediction. As Figure 4 shown, this method performs step-by-step regression with a specific parent node according to the human kinematic structure. For example, for the given joint point 7, regression is performed with joint 0 as its parent node, and the set of its parent nodes is defined as L = {0, 1, 4}. Therefore, for the regression of a current node, the specific regression method is as shown in formula (6):

[0072]

[0073] where represents the regression function of the pose node, F represents the image features, represents the k th node's pose parameters.

[0074] Although the above-mentioned step-by-step regression method based on the kinematic tree can utilize the hierarchical dependence of human motion to a certain extent, thereby enhancing the structural rationality of pose prediction, this method still has certain limitations. Specifically, this method only relies on the fixed parent-child node relationship for regression and fails to fully consider the dynamic correlation between different joint points within the kinematic structure. In addition, during the regression process, all child nodes are default to have the same weight coefficient as the parent node, ignoring the importance differences of different joints in different motion states, which may limit the model's ability to capture complex human poses.

[0075] To overcome the above deficiencies and enhance the structural relationship between 24 joint points within a single frame, this method uses a graph attention mechanism (GAT) to design weights for node connections in order to capture the dynamic dependence between joint points. This method introduces the graph attention mechanism (GAT), allowing the model to dynamically adjust the association strength between joints, thereby enhancing the ability to model local pose relationships. Specifically, this paper proposes an in-frame pose reasoning (IPR) method that uses GAT to model the joint poses of each frame and dynamically learn the weight distribution between joint points to more accurately capture the human pose structure, such as Figure 5 shown, this method treats each joint point as a node in the graph and transforms the features of each node by learning a trainable weight matrix to obtain its latest representation. On this basis, the graph attention matrix is responsible for measuring the association strength between each joint point, and its calculation method is shown in formula (7):

[0076]

[0077] where, represents the linear activation function, represents the shared attention vector of the nodes in the graph, represents the pose node i and k 's concatenation operation. Through this calculation method, the model is not only restricted by the topological structure of the kinematic tree but can also flexibly capture the dynamic constraint relationships in different motion states. Further, this paper filters the pose nodes and prunes the unconnected nodes. Subsequently, for each node i performs softmax function normalization on its parent node set , and the sum of the attention weights of the parent node set is 1. The specific calculation method is shown in formula (8):

[0078]

[0079] Subsequently, based on the topological relationship of the edge connection weight matrix, this method calculates the weight distribution coefficient of a node relative to its set of parent nodes, and linearly combines the global pose parameters of the parent node with the corresponding edge weights to derive the local pose parameters of the current node , thus realizing hierarchical human pose modeling. This process can be formally expressed as shown in Equation (9): , realizing hierarchical human pose modeling. This process can be formally expressed as shown in Equation (9):

[0080]

[0081] where, represents the set of parent nodes of node i , and represents the 6D rotation parameters of the parent node.

[0082] Finally, the pose regression parameter is modeled through a multi-layer perceptron (MLP) in combination with the local pose parameter and the feature representation , and its formal expression is shown in Equation (10):

[0083]

[0084] In this embodiment, this method further includes a model training process. The loss functions adopted in the model training process include: a 2D joint position loss calculated based on the difference between the predicted 2D joint positions and the true 2D joint positions; a 3D joint position loss calculated based on the difference between the predicted 3D joint positions and the true 3D joint positions; an SMPL parameter loss calculated based on the difference between the predicted SMPL pose parameters and body shape parameters and the true SMPL parameters; a temporal smoothness loss imposed on the global fusion features output by the dual-branch temporal attention block (DBTA) to penalize the drastic changes in features between adjacent frames.

[0085] Specifically, the overall training process of the HG-HMR proposed in this paper is shown in Algorithm 1. In this method, following the method of previous studies, for the position losses of 2D and 3D joints, an L2 loss function is used for calculation. Specifically, the position loss of 2D joints and the position loss of 3D joints are as shown in Equation (11):

[0086]

[0087] where, and respectively represent the predicted 2D and 3D joint positions, Represents the corresponding true joint position. For the SMPL regression parameters, this method uses the SMPL loss function , which comprehensively considers the differences in body shape and pose parameters, as shown in Equation (12):

[0088]

[0089] where represents the predicted joint parameters, represents the predicted body shape parameters, represents the true joint parameters, represents the true body shape parameters.

[0090] To further improve the stability of the model, prevent jitter in the prediction results, and ensure good coherence of the generated motion sequence, this method introduces a smoothness loss on the temporal feature map output by the model at the end . This loss function is shown in Equation (13):

[0091]

[0092] where represents the temporal feature map of the i th frame.

[0093] The training process of the method proposed in this study is as follows:

[0094] Algorithm 1 Overall training process of the proposed method (HG-HMR)

[0095] Input: Training set , 2D pose detector , CNN network , Pose-Net network , DBTA network block , SMPL regressor , GAT network .

[0096] 1. For each training sample (RGB video sequence) in the training dataset do;

[0097] 2. For each frame in the RGB video sequence, extract its 2D joints through Equation (1) and generate the corresponding heat map sequence: ;

[0098] 3. Perform feature encoding: ;

[0099] 4. As shown in formulas (3) and (4), obtain the temporal global features (fused temporal features) of the image through DBTA: ;

[0100] 5. Initially regress the global SMPL parameters (including pose parameters, body shape parameters, and camera parameters) from the fused temporal features through formula (5): ;

[0101] 6. As shown in formula (7), perform in-frame pose reasoning (IPR) on the pose parameters to model the dependency relationships between joint points: ;

[0102] 7. Calculate the normalized parent node attention weights based on the kinematic tree structure and the modeled dependency relationships, as shown in formula (8);

[0103] 8. Parse the local pose parameters according to the pose parameters and attention weights of the parent node, as shown in formula (9): ;

[0104] 9. Combine the fused temporal features and the local pose parameters, regress the final SMPL pose parameters through MLP, and calculate the total loss (including 2D and 3D joint point losses, SMPL parameter losses, and smoothness losses) by combining the body shape parameters and camera parameters, as shown in formulas (11), (12), and (13), and update the model parameters;

[0105] 10. End.

[0106] Experimental results show that the proposed method has achieved excellent performance on multiple datasets, significantly improving the accuracy and temporal smoothness of 3D human pose estimation.

[0107] In summary, in this embodiment, the method and system for temporally smooth human motion capture based on heat map guidance provided by the present invention effectively solve the deficiencies of existing methods by organically combining 2D pose priors, cross-modal temporal feature fusion, and in-frame pose reasoning based on graph attention, providing an effective solution for high-precision and temporally stable human motion capture in video streams.

[0108] Embodiment 2

[0109] Please refer to Figure 6 , which shows the structural schematic diagram of a system for temporally smooth human motion capture based on heat map guidance proposed in the second embodiment of this application. The system includes the following key modules:

[0110] Input module 100, configured to obtain an RGB video sequence containing multiple frames of images;

[0111] The 2D pose extraction and heat map generation module 200 is configured to extract 2D human joint position information from each frame image in the RGB video sequence, and generate a corresponding heat map sequence based on the 2D human joint position information;

[0112] The feature extraction module 300 is configured to perform feature extraction on the RGB video sequence and the heat map sequence respectively, to obtain the temporal features of the RGB video sequence and the temporal features of the heat map sequence;

[0113] The dual-branch temporal attention module 400 is configured to use a dual-branch temporal attention block (DBTA) to fuse the temporal features of the RGB video sequence and the temporal features of the heat map sequence, to obtain fused temporal features, and the DBTA includes a multi-head self-attention (MHSA) mechanism and a multi-head cross-attention (MCA) mechanism;

[0114] The SMPL regression module 500 is configured to regress SMPL model parameters according to the fused temporal features, and the SMPL model parameters include pose parameters, shape parameters, and camera parameters. Among them, the step of regressing the pose parameters includes using an in-frame pose reasoning (IPR) module to model the structural dependencies between the three-dimensional human joints corresponding to the pose parameters through a graph attention mechanism, and optimizing the pose parameters based on the structural dependencies;

[0115] The output module 600 is configured to output three-dimensional human motion capture data according to the regressed SMPL model parameters.

[0116] A temporal smoothing human motion capture system based on heat map guidance in an embodiment of the present application may be a device, or a component, an integrated circuit, or a chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a Network Attached Storage (NAS), a personal computer (PC), etc. The embodiments of the present application do not make specific limitations.

[0117] A timing-smoothing human motion capture system guided by a calorific value map in an embodiment of the present application can be a device with an operating system. The operating system can be the Android operating system, the iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0118] A timing-smoothing human motion capture system guided by a calorific value map provided in an embodiment of the present application can implement Figure 1 each process implemented by a timing-smoothing human motion capture method in a method embodiment guided by a calorific value map. To avoid repetition, it will not be elaborated here.

[0119] Optionally, an embodiment of the present application further provides an electronic device, including a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, it implements each process of the above-mentioned embodiment of the timing-smoothing human motion capture method guided by a calorific value map and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0120] An embodiment of the present application further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above-mentioned embodiment of the timing-smoothing human motion capture method guided by a calorific value map and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0121] Among them, the processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0122] It should be noted that in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0123] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described example methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present application.

[0124] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. A timing-smoothing human motion capture method guided by a calorific value map, characterized in that Including the following steps: S1: Obtain an RGB video sequence containing multiple frames of images; S2: Extract 2D human joint point position information for each frame of image in the RGB video sequence, and generate a corresponding heat map sequence based on the 2D human joint point position information; S3: Respectively perform feature extraction on the RGB video sequence and the heat map sequence to obtain the temporal features of the RGB video sequence and the temporal features of the heat map sequence; S4: Use a dual-branch temporal attention block to fuse the temporal features of the RGB video sequence and the temporal features of the heat map sequence to obtain fused temporal features, where the dual-branch temporal attention block includes a multi-head self-attention mechanism and a multi-head cross-attention mechanism; S5: Regress SMPL model parameters according to the fused temporal features, where the SMPL model parameters include pose parameters, body shape parameters, and camera parameters. Among them, the step of regressing the pose parameters includes using an intra-frame pose inference module to model the structural dependencies between the three-dimensional human joints corresponding to the pose parameters through a graph attention mechanism, and optimizing the pose parameters based on the structural dependencies; S6: Output three-dimensional human motion capture data according to the regressed SMPL model parameters.

2. The method according to claim 1, wherein The step S2 specifically includes: Obtain the 2D human joint position information in the RGB video sequence through a 2D pose extractor ; Based on the 2D human joint point position information, use Gaussian distribution spots to generate a heat map sequence, and the value of the heat map sequence at the position is calculated according to the following formula: ， wherein represents the value of the calorific value map sequence at the position , and represent the positions of each pixel on the grid is the standard deviation that controls the diffusion range of the Gaussian distribution.

3. The method according to claim 1, wherein In the step S4, using the dual-branch temporal attention block to obtain the fused temporal features specifically includes: Linearly project the temporal features of the RGB video sequence and the temporal features of the heat map sequence into the same low-dimensional feature space respectively; Add learnable temporal position encodings to the temporal features of the mapped RGB video sequence and the temporal features of the heat map sequence respectively; Use the multi-head self-attention mechanism to calculate the temporal attention matrix of the temporal features of the mapped RGB video sequence and the temporal attention matrix of the temporal features of the mapped heat map sequence to capture the temporal information within each modality; Use the multi-head cross-attention mechanism to establish a potential association between the temporal features of the mapped RGB video sequence and the temporal features of the mapped heat map sequence to enhance cross-modal information interaction; Use a linear layer and an adaptive weight factor to perform weighted fusion on the temporal features of the fused RGB video sequence and the temporal features of the heat map sequence to obtain the fused temporal features; Among them, the adaptive weight factor is calculated according to the following formula: ， Among them, the adaptive weight factor , represents the length of the time series, represents a learnable linear transformation matrix, represents the temporal features of the RGB video sequence and the temporal features of the heat map sequence the concatenation operation between modalities; The fused temporal features are calculated according to the following formula: 。 4. The method according to claim 1, wherein In the step S5, using the graph attention mechanism of the intra-frame pose inference module to model the structural dependencies between the three-dimensional human joints corresponding to the pose parameters specifically includes: Regard the three-dimensional human joints corresponding to the pose parameters as nodes in the graph; Use the graph attention mechanism to learn a trainable weight matrix to transform the features of the nodes; Calculate the association strength between joint nodes through the graph attention mechanism to obtain a graph attention matrix. The association strength between node and node is calculated according to the following formula: ， Among them, represents the transpose of the shared attention vector, represents the pose node i and k is the splicing operation; Prune the unconnected nodes according to the graph attention matrix.

5. The method according to claim 4, wherein In step S5, using the graph attention mechanism of the intra-frame pose inference module to model the structural dependencies between the three-dimensional human joints corresponding to the pose parameters further includes: For each joint node , perform softmax function normalization on the corresponding set of parent nodes in the preset kinematic tree structure. The sum of the attention weights of the set of parent nodes is 1, and the normalized attention weights are calculated according to the following formula: ； Calculate the local pose parameters of the joint node based on the topological relationship of the kinematic tree structure and the normalized attention weights wherein the local pose parameters are obtained by calculating the weight distribution coefficients of the joint node relative to its set of parent nodes, and linearly combining the global pose parameters of the parent nodes with the corresponding edge weights . The calculation method is shown in the following formula: ​ 。 6. The method according to claim 5, characterized in that, In step S5, optimizing the pose parameters based on the structural dependencies specifically includes: Using a multi-layer perceptron to combine the fused temporal features and the calculated local pose parameters to model the pose parameters of the final SMPL model.

7. The method according to claim 1, characterized in that, The loss function adopted during the training process of the method includes: A 2D joint position loss calculated based on the difference between the predicted 2D joint positions and the true 2D joint positions; A 3D joint position loss calculated based on the difference between the predicted 3D joint positions and the true 3D joint positions; An SMPL parameter loss calculated based on the difference between the predicted SMPL pose parameters and body shape parameters and the true SMPL parameters; A temporal smoothness loss imposed on the globally fused features output by the dual-branch temporal attention block, which is used to penalize the drastic changes in features between adjacent frames.

8. A time-series smoothed human motion capture system guided by a calorific value map, characterized in that, It includes: An input module configured to obtain an RGB video sequence containing multiple frames of images; A 2D pose extraction and heatmap generation module configured to extract 2D human joint position information from each frame image in the RGB video sequence and generate a corresponding heatmap sequence based on the 2D human joint position information; A feature extraction module configured to respectively extract features from the RGB video sequence and the heatmap sequence to obtain the temporal features of the RGB video sequence and the temporal features of the heatmap sequence; A dual-branch temporal attention module configured to use a dual-branch temporal attention block to fuse the temporal features of the RGB video sequence and the temporal features of the heatmap sequence to obtain fused temporal features, and the dual-branch temporal attention block includes a multi-head self-attention mechanism and a multi-head cross-attention mechanism; An SMPL regression module configured to regress SMPL model parameters according to the fused temporal features, and the SMPL model parameters include pose parameters, body shape parameters, and camera parameters. Among them, the step of regressing the pose parameters includes using an intra-frame pose inference module to model the structural dependencies between the three-dimensional human joints corresponding to the pose parameters through a graph attention mechanism and optimizing the pose parameters based on the structural dependencies; An output module configured to output three-dimensional human motion capture data according to the regressed SMPL model parameters.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of a method for temporally smooth human motion capture guided by a heatmap as described in any one of claims 1-7 are implemented.

10. A readable storage medium, characterized in that, A program or instruction is stored on the readable storage medium. When the program or instruction is executed by a processor, the steps of a method for temporally smooth human motion capture guided by a heatmap as described in any one of claims 1-7 are implemented.

Images