Three-dimensional human pose estimation method based on dynamic modeling and bone topology consistency

Abstract

The application discloses a three-dimensional human body posture estimation method based on dynamic modeling and bone topological consistency, multi-modal input construction and feature extraction, feature extraction is respectively performed on input RGB images, two-dimensional joint coordinates and skeleton structure diagrams, and each mode feature after extraction is uniformly mapped to the same dimension, thereby providing a basis for subsequent multi-modal fusion; by sequentially executing five core processes of multi-modal input construction and feature extraction, dynamic credibility modeling and adaptive fusion, bone topological guided repair, multi-scale detail enhancement and three-dimensional regression, and loss function joint optimization, the problem that a static strategy is adopted for multi-modal fusion of an existing three-dimensional human body posture estimation method and the mode contribution proportion cannot be dynamically adjusted according to a scene is effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, and in particular relates to a three-dimensional human pose estimation method based on dynamic modeling and skeletal topological consistency. Background Technology

[0002] Three-dimensional human pose estimation is a core research topic in the field of computer vision, and it has broad application value and prospects in many fields such as human-computer collaboration, security monitoring, medical rehabilitation assessment, and virtual reality interaction.

[0003] Existing methods can be mainly divided into three categories: methods that directly regress the coordinates of three-dimensional joints based on monocular RGB images, methods that improve the accuracy of three-dimensional regression by using two-dimensional pose detection results, and methods that use graph convolutional networks to model the topology of the human skeleton.

[0004] However, existing methods still have the following limitations: multimodal fusion usually relies on static weighting or simple feature splicing strategies, making it difficult to dynamically adjust the weights based on the signal-to-noise ratio of each modality. In complex scenarios such as occlusion, strong lighting, and motion blur, when a certain modality experiences quality degradation, it is easy to interfere with the fusion results.

[0005] Although graph convolution-based methods can propagate information about the human skeletal structure, they fail to be deeply integrated with modal quality assessment mechanisms. Furthermore, due to the lack of explicit kinematic consistency constraints, predicted poses often exhibit structural inconsistencies or jitter.

[0006] To address the aforementioned issues and significantly improve the robustness, accuracy, and stability of 3D human pose estimation in complex environments, this paper proposes a 3D human pose estimation method based on dynamic credibility modeling and skeletal topological consistency constraints. Summary of the Invention

[0007] The purpose of this invention is to provide a three-dimensional human pose estimation method based on dynamic modeling and skeletal topology consistency, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a three-dimensional human pose estimation method based on dynamic modeling and skeletal topology consistency, comprising the following steps: S1. Multimodal input construction and feature extraction: Features are extracted from the input RGB image, two-dimensional joint coordinates and skeleton structure diagram respectively, and the extracted modal features are uniformly mapped to the same dimension to provide a foundation for subsequent multimodal fusion. S2. Dynamic credibility modeling and adaptive fusion: Based on the dimensional unified features of step S1, the credibility of each modality is evaluated in real time and the fusion weight is adaptively adjusted. The features of each modality are weighted and fused through the weights. Then, a self-attention mechanism is introduced to clean up the fused features and suppress modal conflict noise. S3, Skeletal Topology-Guided Repair: Using the purified features from S2 as initial input, a human skeleton map is constructed. The feature representation is enhanced by aggregating joint neighborhood information through a graph convolutional network. At the same time, a neighborhood-weighted repair mechanism is used to reconstruct features for occluded joint nodes with low confidence, thereby achieving accurate repair of occluded joints. S4, Multi-scale detail enhancement and 3D regression: The structural features output by the graph convolutional network after S3 repair are used as the original features. The feature pyramid network is used for multi-scale processing. After fusing the multi-scale features to obtain the enhanced features, they are input into the regression network to output the 3D joint coordinates. S5. Joint optimization of loss function: Based on the 3D joint coordinates output in step S4, the entire network is optimized and trained using the designed joint loss function. At the same time, the accuracy of joint positioning and the kinematic rationality of predicted posture are constrained to complete the 3D human posture estimation in complex scenes.

[0009] Preferably, the RGN image in step S1 Feature extraction via convolutional neural network ,and ; The two-dimensional joint coordinates Obtained through embedding mapping And N=17, which is the fixed number of joint nodes in the human body; The skeleton structure diagram Obtained through convolution mapping ; in, For the real number field, , These represent the height and width of the image and feature map, respectively. The height and width of the feature map after feature extraction. The number of channels in the feature map. , These are the feature extraction functions corresponding to convolutional neural networks, embedding maps, and convolutional maps, respectively.

[0010] Preferably, the specific process of dynamic credibility modeling in step S2 is as follows: for each mode ,in, Modal identifiers used to distinguish RGB images, 2D joint coordinates, and skeleton structure diagrams. The modal features obtained in step S1 are used to obtain global features through a global average pooling operation. , For the first The global features obtained after global average pooling of various modes will be Input the data into the scoring network to obtain a credibility score. ; in, In order to target the A multilayer perceptron scoring network with modal design. For the first The credibility score of each modality; Using the normalized exponential function Convert the confidence scores of each modality into weighting coefficients; The conversion formula is: And satisfy ; in, For the first Modal data Normalized weight coefficients It is a natural exponential function. .

[0011] Preferably, the formula for weighted fusion of multimodal features in step S2 is: ,in, The fused features are the weighted fusion results of multimodal features. , , These are the fusion weight coefficients for the three modalities: RGB, 2D, and sk, respectively. When the self-attention mechanism purifies the fusion features, it will The attention output is calculated by mapping the query matrix Q, key matrix K, and value matrix V respectively. ; The characteristics of purification fusion ; in, For feature dimension, To perform matrix multiplication on the transpose of the query matrix and the key matrix, () is the normalized exponential function. () is the self-attention calculation function. These are characteristics after purification.

[0012] Preferably, in step S3, the constructed human skeleton graph is G=(V,E), where V is the set of nodes consisting of 17 human joint nodes, and E is the set of edges consisting of bone connection edges. A self-connection matrix is ​​obtained based on the human skeleton graph. And construct the degree matrix ; in, This is the initial adjacency matrix of the human skeleton diagram. It is the identity matrix. It is a diagonal degree matrix. Adjacency matrix The Middle Line number Column elements.

[0013] Preferably, in step S3, the feature propagation formula for the graph convolutional network is: ; in, For graph convolutional networks Layer node characteristics, To the purified characteristics of step S2 The initial node features obtained from the index. For graph convolutional networks Layer node characteristics, Let L be the learnable weight matrix of the l-th layer of the graph convolutional network. This is the RelU activation function.

[0014] Preferably, the specific process of the neighborhood weighted repair mechanism in step S3 is as follows: calculate the occluded nodes. With neighboring nodes Feature similarity; The calculation formula is as follows: ; Repair weights are calculated based on similarity. The repair weight formula is: ; The Represents a node The set of neighboring nodes, and the final repaired node characteristics are:

[0015] in, , , Don't be the first in the skeletal diagram , , Features of each joint node For the first Transpose of the features of each node The notation for L2 norm calculation. () is the cosine similarity calculation function. For the first The neighboring nodes of the nth pair Repair weights for each occluded node. Features of the repaired occluded nodes.

[0016] Preferably, in step S4, the specific formula for increasing multi-scale details is as follows: ; ; in, The skeletal topology features are output by the graph convolutional network in step S3. () represents the multi-scale feature processing function of the feature pyramid network. For multi-scale feature sets These are feature maps at different scales. () represents the feature splicing function along the channel dimension. () is the convolution operation function. The final enhanced features are those obtained after multi-scale fusion and dimensionality reduction. The three-dimensional joint coordinates output by the regression network are ,in, This is the result of three-dimensional joint coordinates.

[0017] Preferably, in step S5, the joint regression loss uses an L2 distance constraint, and the calculation formula is as follows: ; The formula for calculating the bone length consistency loss is as follows: ; The formula for the total loss function is: ; in, Let the joint regression loss function be... For the first Predicted 3D coordinates of joint nodes. No. Predicted 3D coordinates of joint nodes. The notation for L2 norm calculation. Let be the bone length consistency loss function. Set of edges connecting bones The number of elements, Indicates the first , Each node represents the two endpoints of the bone connection edge. The symbol for absolute value calculation. For the total loss function, To balance the weighting of the two losses.

[0018] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention effectively solves the problem that existing 3D human pose estimation methods use static strategies for multimodal fusion and cannot dynamically adjust the modal contribution ratio according to the scene by sequentially executing five core processes: multimodal input construction and feature extraction, dynamic credibility modeling and adaptive fusion, skeleton topology-guided repair, multi-scale detail enhancement and 3D regression, and joint optimization of loss function. At the same time, it addresses the shortcomings of modal quality degradation in complex scenes such as occlusion, strong lighting, and motion blur, which interfere with the fusion results. It achieves adaptive weighted fusion and noise reduction of multimodal features. Combined with skeleton topology constraints and joint loss optimization, it significantly improves the robustness and accuracy of 3D human pose estimation in complex scenes, making the pose estimation results more in line with the needs of practical application scenarios.

[0019] 2. This invention addresses the problem that existing graph convolution methods are not deeply coupled with modal quality assessment mechanisms. It uses the cleaned and fused features obtained from dynamic credibility modeling as the initial input for skeletal topology-guided repair. By constructing a human skeletal graph and combining it with a graph convolutional network to aggregate joint neighborhood information, a neighborhood-weighted repair mechanism is designed to accurately reconstruct the features of occluded joint nodes. This achieves a deep integration of modal quality assessment and skeletal topology modeling, effectively making up for the shortcomings of traditional methods in terms of insufficient occlusion recovery capabilities. This makes the joint feature representation in occluded scenarios more accurate and improves the structural integrity of pose estimation.

[0020] 3. This invention addresses the problems of existing methods lacking clear kinematic consistency constraints and unstable predicted posture structures. It improves the positioning accuracy of small-scale joints at the human end of the body through multi-scale detail enhancement. It also designs a joint loss function that includes joint regression loss and bone length consistency loss. While ensuring the accuracy of joint coordinate regression, it constrains the reasonableness of the ratio between predicted bone length and actual length, so that the predicted three-dimensional human posture strictly conforms to the laws of human kinematics. This solves the problems of unreasonable posture structure and jitter, and significantly improves the stability and reasonableness of posture estimation results. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the present invention; Figure 2 This is a framework diagram of the present invention; Figure 3 This is a qualitative result diagram of the three-dimensional attitude estimation of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Please see Figures 1 to 3 The present invention provides a technical solution: Example 1: A 3D human pose estimation method based on dynamic modeling and skeletal topology consistency, comprising the following steps: S1. Multimodal input construction and feature extraction: Features are extracted from the input RGB image, two-dimensional joint coordinates and skeleton structure diagram respectively, and the extracted modal features are uniformly mapped to the same dimension to provide a foundation for subsequent multimodal fusion. S2. Dynamic credibility modeling and adaptive fusion: Based on the dimensional unified features of step S1, the credibility of each modality is evaluated in real time and the fusion weight is adaptively adjusted. The features of each modality are weighted and fused through the weights. Then, a self-attention mechanism is introduced to clean up the fused features and suppress modal conflict noise. S3, Skeletal Topology-Guided Repair: Using the purified features from S2 as initial input, a human skeleton map is constructed. The feature representation is enhanced by aggregating joint neighborhood information through a graph convolutional network. At the same time, a neighborhood-weighted repair mechanism is used to reconstruct features for occluded joint nodes with low confidence, thereby achieving accurate repair of occluded joints. S4, Multi-scale detail enhancement and 3D regression: The structural features output by the graph convolutional network after S3 repair are used as the original features. The feature pyramid network is used for multi-scale processing. After fusing the multi-scale features to obtain the enhanced features, they are input into the regression network to output the 3D joint coordinates. S5. Joint optimization of loss function: Based on the 3D joint coordinates output in step S4, the entire network is optimized and trained using the designed joint loss function. At the same time, the accuracy of joint positioning and the kinematic rationality of predicted posture are constrained to complete the 3D human posture estimation in complex scenes.

[0025] Example 2: A 3D human pose estimation method based on dynamic modeling and skeletal topology consistency, in step S1, the RGN image... Feature extraction via convolutional neural network ,and ; Two-dimensional joint coordinates Obtained through embedding mapping And N=17, which is the fixed number of joint nodes in the human body; Skeletal structure diagram Obtained through convolution mapping ; in, For the real number field, These represent the height and width of the image and feature map, respectively. The height and width of the feature map after feature extraction. The number of channels in the feature map. , These are the feature extraction functions corresponding to convolutional neural networks, embedding maps, and convolutional maps, respectively.

[0026] The specific process of dynamic credibility modeling in step S2 is to model each mode... ,in, Modal identifiers used to distinguish RGB images, 2D joint coordinates, and skeleton structure diagrams. The modal features obtained in step S1 are used to obtain global features through a global average pooling operation. , For the first The global features obtained after global average pooling of various modes will be Input the data into the scoring network to obtain a credibility score. ; in, In order to target the A multilayer perceptron scoring network with modal design. For the first The credibility score of each modality; Using the normalized exponential function Convert the confidence scores of each modality into weighting coefficients; The conversion formula is: And satisfy ; in, For the first Modal data Normalized weight coefficients It is a natural exponential function. .

[0027] The formula for weighted fusion of multimodal features in step S2 is: ,in, The fused features are the weighted fusion results of multimodal features. , These are the fusion weight coefficients for the three modalities: RGB, 2D, and sk, respectively. When purifying fusion features using self-attention mechanisms, Mapped to query matrix Q, key matrix K, and value matrix V respectively, attention output calculation formula ; Features of purification and fusion ; in, For feature dimension, To perform matrix multiplication on the transpose of the query matrix and the key matrix, () is the normalized exponential function. () is the self-attention calculation function. These are characteristics after purification.

[0028] In step S3, the constructed human skeleton graph is G=(V,E), where V is the set of nodes consisting of 17 human joint nodes, and E is the set of edges consisting of skeleton connecting edges. Based on the human skeleton graph, self-connection is introduced to obtain the adjacency matrix. And construct the degree matrix ; in, This is the initial adjacency matrix of the human skeleton diagram. It is the identity matrix. It is a diagonal degree matrix. Adjacency matrix The Middle Line 1 Column elements.

[0029] Example 3: A 3D human pose estimation method based on dynamic modeling and skeletal topology consistency. In step S3, the feature propagation formula for the graph convolutional network is: ; in, For graph convolutional networks Layer node characteristics, To the purified characteristics of step S2 The initial node features obtained from the index. For graph convolutional networks Layer node characteristics, Let L be the learnable weight matrix of the l-th layer of the graph convolutional network. This is the RelU activation function.

[0030] The specific process of the neighborhood weighted repair mechanism in step S3 is as follows: calculate the occluded nodes. With neighboring nodes Feature similarity; The calculation formula is: ; Repair weights are calculated based on similarity. The formula for repair weight is ; Represents a node The set of neighboring nodes, and the final repaired node characteristics are:

[0031] in, , Don't be the first in the skeletal diagram , Features of each joint node For the first Transpose of the features of each node The notation for L2 norm calculation. () is the cosine similarity calculation function. For the first The neighboring nodes of the nth pair Repair weights for each occluded node. Features of the repaired occluded nodes.

[0032] In step S4, the specific formula for adding multi-scale details is as follows: ; ; in, The skeletal topology features are output by the graph convolutional network in step S3. () represents the multi-scale feature processing function of the feature pyramid network. For multi-scale feature sets These are feature maps at different scales. () represents the feature splicing function along the channel dimension. () is the convolution operation function. The final enhanced features are those obtained after multi-scale fusion and dimensionality reduction. The 3D joint coordinates output by the regression network are ,in, This is the result of three-dimensional joint coordinates.

[0033] In step S5, the joint regression loss is constrained by L2 distance, and the calculation formula is as follows: ; The formula for calculating bone length consistency loss is: ; The formula for the total loss function is: ; in, Let the joint regression loss function be... For the first Predicted 3D coordinates of joint nodes. No. Predicted 3D coordinates of joint nodes. The notation for L2 norm calculation. Let be the bone length consistency loss function. Set of edges connecting bones The number of elements, Indicates the first , Each node represents the two endpoints of the bone connection edge. The symbol for absolute value calculation. For the total loss function, To balance the weighting of the two losses.

[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D human pose estimation method based on dynamic modeling and skeletal topology consistency, characterized by: Includes the following steps: S1. Multimodal input construction and feature extraction: Features are extracted from the input RGB image, two-dimensional joint coordinates and skeleton structure diagram respectively, and the extracted modal features are uniformly mapped to the same dimension to provide a foundation for subsequent multimodal fusion. S2. Dynamic credibility modeling and adaptive fusion: Based on the dimensional unified features of step S1, the credibility of each modality is evaluated in real time and the fusion weight is adaptively adjusted. The features of each modality are weighted and fused through the weights. Then, a self-attention mechanism is introduced to clean up the fused features and suppress modal conflict noise. S3, Skeletal Topology-Guided Repair: Using the purified features from S2 as initial input, a human skeleton map is constructed. The feature representation is enhanced by aggregating joint neighborhood information through a graph convolutional network. At the same time, a neighborhood-weighted repair mechanism is used to reconstruct features for occluded joint nodes with low confidence, thereby achieving accurate repair of occluded joints. S4, Multi-scale detail enhancement and 3D regression: The structural features output by the graph convolutional network after S3 repair are used as the original features. The feature pyramid network is used for multi-scale processing. After fusing the multi-scale features to obtain the enhanced features, they are input into the regression network to output the 3D joint coordinates. S5. Joint optimization of loss function: Based on the 3D joint coordinates output in step S4, the entire network is optimized and trained using the designed joint loss function. At the same time, the accuracy of joint positioning and the kinematic rationality of predicted posture are constrained to complete the 3D human posture estimation in complex scenes.

2. The three-dimensional human pose estimation method based on dynamic modeling and skeletal topology consistency according to claim 1, characterized in that: RGN image in step S1 Feature extraction via convolutional neural network ,and ; The two-dimensional joint coordinates Obtained through embedding mapping And N=17, which is the fixed number of joint nodes in the human body; The skeleton structure diagram Obtained through convolution mapping ; in, For the real number field, These represent the height and width of the image and feature map, respectively. The height and width of the feature map after feature extraction. The number of channels in the feature map. , These are the feature extraction functions corresponding to convolutional neural networks, embedding maps, and convolutional maps, respectively.

3. The three-dimensional human pose estimation method based on dynamic modeling and skeletal topology consistency according to claim 1, characterized in that: The specific process of dynamic credibility modeling in step S2 is as follows: for each mode ,in, Modal identifiers used to distinguish RGB images, 2D joint coordinates, and skeleton structure diagrams. The modal features obtained in step S1 are used to obtain global features through a global average pooling operation. , For the first The global features obtained after global average pooling of various modes will be Input the data into the scoring network to obtain a credibility score. ; in, In order to target the A multilayer perceptron scoring network with modal design. For the first The credibility score of each modality; Using the normalized exponential function Convert the confidence scores of each modality into weighting coefficients; The conversion formula is: And satisfy ; in, For the first Modal data Normalized weight coefficients It is a natural exponential function. .

4. The three-dimensional human pose estimation method based on dynamic modeling and skeletal topology consistency according to claim 3, characterized in that: The formula for weighted fusion of multimodal features in step S2 is as follows: ,in, The fused features are the weighted fusion results of multimodal features. , They are RGB, 2D and The fusion weight coefficients corresponding to the three modes; When the self-attention mechanism purifies the fusion features, it will Mapped to query matrix Q and key matrix respectively Sum matrix The attention output calculation formula ; The characteristics of purification fusion ; in, For feature dimension, To perform matrix multiplication on the transpose of the query matrix and the key matrix, () is the normalized exponential function. () is the self-attention calculation function, and is the purified feature.

5. The three-dimensional human pose estimation method based on dynamic modeling and skeletal topology consistency according to claim 1, characterized in that: In step S3, the constructed human skeleton graph is G=(V,E), where V is the set of nodes consisting of 17 human joint nodes, and E is the set of edges consisting of bone connection edges. A self-connection matrix is ​​obtained based on the human skeleton graph. And construct the degree matrix ; in, This is the initial adjacency matrix of the human skeleton diagram. It is the identity matrix. It is a diagonal degree matrix. Adjacency matrix The Middle Line number Column elements.

6. The three-dimensional human pose estimation method based on dynamic modeling and skeletal topology consistency according to claim 5, characterized in that: In step S3, the feature propagation formula for the graph convolutional network is: ; in, For graph convolutional networks Layer node characteristics, To the purified characteristics of step S2 The initial node features obtained from the index. For graph convolutional networks Layer node characteristics, Let L be the learnable weight matrix of the l-th layer of the graph convolutional network. This is the RelU activation function.

7. The three-dimensional human pose estimation method based on dynamic modeling and skeletal topology consistency according to claim 5, characterized in that: The specific process of the neighborhood weighted repair mechanism in step S3 is as follows: calculate the occluded nodes. With neighboring nodes Feature similarity; The calculation formula is as follows: ; Repair weights are calculated based on similarity. The repair weight formula is: ; The Represents a node The set of neighboring nodes, and the final repaired node characteristics are: in, , These are the first in the skeletal diagram. , Features of each joint node For the first Transpose of the features of each node The notation for L2 norm calculation. () is the cosine similarity calculation function, which is the first function to perform the cosine similarity calculation. The neighboring nodes of the nth pair Repair weights for each occluded node. Features of the repaired occluded nodes.

8. The three-dimensional human pose estimation method based on dynamic modeling and skeletal topology consistency according to claim 1, characterized in that: In step S4, the specific formula for increasing multi-scale details is as follows: ； ; in, The skeletal topology features are output by the graph convolutional network in step S3. () represents the multi-scale feature processing function of the feature pyramid network, and () represents the multi-scale feature set. These are feature maps at different scales. () represents the feature concatenation function along the channel dimension. () is the convolution operation function. The final enhanced features are those obtained after multi-scale fusion and dimensionality reduction. The three-dimensional joint coordinates output by the regression network are: ,in, This is the result of three-dimensional joint coordinates.

9. The three-dimensional human pose estimation method based on dynamic modeling and skeletal topology consistency according to claim 1, characterized in that: In step S5, the joint regression loss uses L2 distance constraint, and the calculation formula is as follows: ; The formula for calculating the bone length consistency loss is as follows: ； The formula for the total loss function is: ； in, Let the joint regression loss function be... For the first Predicted 3D coordinates of joint nodes. No. Predicted 3D coordinates of joint nodes. The notation for L2 norm calculation. Let be the bone length consistency loss function. Set of edges connecting bones The number of elements, Indicates the first , Each node represents the two endpoints of the bone connection edge. The symbol for absolute value calculation. For the total loss function, The weighting coefficients are used to balance the two losses.

Images