Fine human body posture estimation method and device, medium and product

Abstract

The invention discloses a refined human body posture estimation method and device, a medium and a product, and relates to the field of artificial intelligence and the technical field of computer vision, and the method comprises the steps: building and training a refined human body posture estimation framework; the refined human body posture estimation framework comprises a joint-limb embedding module, a structure-semantic feature fusion module, a hierarchical joint-limb fusion encoder, a joint-limb decoder and a post-reasoning module which are connected in sequence; on the basis of the to-be-detected image, the refined human body posture estimation is completed through the trained refined human body posture estimation framework, and the precision of human body posture estimation in the crowded scene can be improved by building and training the refined human body posture estimation framework.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and computer vision technology, and in particular to a method, device, medium and product for refined human posture estimation. Background Art

[0002] Multi-person Pose Estimation (MPPE) is a fundamental task in computer vision, which estimates the human pose structure by locating specific predefined human joints. With the advent of deep learning techniques, multi-person pose estimation has made significant progress, which can achieve more accurate and robust estimation. Correctly representing the structure of the human body in crowded scenes remains particularly challenging due to frequent occlusions, different scales, and the complexity of interactions between individuals. The main difficulties in pose estimation in crowded scenes stem from two main problems. First, content interweaving, the same type of joints from different individuals can be close to each other, causing mutual interference. Second, frequent occlusions hinder part-based pose estimation methods that rely on semantic features around specific joints.

[0003] Human pose estimation methods can be mainly divided into two categories: bottom-up and top-down. For bottom-up methods, dealing with occlusion is a difficult point, and the invisible joints caused by occlusion significantly reduce the performance. Faced with these problems, predecessors have continuously innovated bottom-up methods, but the accuracy of bottom-up methods is still challenged when dealing with complex and crowded scenes. For top-down methods, top-down methods first detect all people in the scene and then estimate the pose of each person. This method benefits from the success of object detection technology and can achieve more accurate pose estimation within the detected human bounding box. However, these methods still face the difficulties of content interweaving or occlusion, where one bounding box may capture the joints of multiple people or fail to extract the correct joint-related semantic features. Recent research on human pose estimation has paid more and more attention to pose estimation in crowded scenes. However, when faced with inaccurate image information, it is difficult to achieve accurate key point positioning, which has led to the development of pose refinement technology. In general, human pose estimation technology has made some progress, but accuracy is still problematic in some aspects, especially in crowded and complex scenes, where occlusion and interaction between people can significantly reduce performance. Existing methods, whether bottom-up or top-down, have difficulty estimating pose accurately when joints are occluded or visual information is unclear. Summary of the invention

[0004] The purpose of this application is to provide a refined human posture estimation method, device, medium and product, which can improve the accuracy of human posture estimation in crowded scenes.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a refined human body posture estimation method, comprising:

[0007] Building and training a refined human posture estimation framework; the refined human posture estimation framework includes: a joint-limb embedding module, a structure-semantic feature fusion module, a hierarchical joint-limb fusion encoder, a joint-limb decoder and a post-reasoning module connected in sequence;

[0008] Inputting the image to be tested into a posture estimator to obtain semantic features and initial posture estimation results; the initial posture estimation results include joint data and limb data;

[0009] Determining joint structure features and limb structure features using the joint-limb embedding module according to the initial posture estimation result;

[0010] Determine joint fusion features and limb fusion features using the structure-semantic feature fusion module according to the semantic features, the joint structure features, the limb structure features and a plurality of joint heat maps;

[0011] Determine the joint-limb fusion feature using the hierarchical joint-limb fusion encoder according to the joint fusion feature and the limb fusion feature;

[0012] Determine joint prediction position data and limb prediction data using a joint-limb decoder according to the joint structure features, the limb structure features and the joint-limb fusion features;

[0013] According to the joint prediction position data and the limb prediction data, a post-reasoning module is used to determine the corrected human body posture based on a post-reasoning strategy.

[0014] Optionally, the joint structure is characterized by:

[0015] F J =Pos j +(E J (J init ));

[0016] In the formula, F J Indicates the structural characteristics of the joint, Pos j represents the joint position encoding, E J Represents joint point data, J init Represents joint data;

[0017] The limb structure features are:

[0018] F L =Pos l+(E L (L init );

[0019] In the formula, F L Indicates the structural characteristics of the limbs, Pos l Indicates the limb position code, E L Represents limb data, L init Represents limb data.

[0020] Optionally, the joint fusion is characterized by:

[0021] F fuse_joint =Φ(Concat(F sem ⊙H J ,F struct_joint );

[0022] In the formula, F fuse_joint represents joint fusion features, Φ(*) represents multi-layer perceptron operation, Concat(*) represents concatenation operation, F sem represents the semantic features output by the pose estimator, ⊙ represents the dot product, H J Represents the heat map of all joints, F struct_joint ) represents the structural features of joint points;

[0023] The limb fusion features are:

[0024] F fuse_limb =Φ(Concat(Ω(M j2l ·(F sem ⊙H J )),F struc_limb ));

[0025] In the formula, F fuse_limb represents the limb fusion feature, Ω(*) represents the depth-wise separable convolution with kernel and stride 2, M j2l represents the transformation matrix mapping the connected joint j to the limb l, F struc_limb Represents the structural characteristics of the limbs.

[0026] Optionally, the hierarchical joint-limb fusion encoder includes a limb-joint fusion branch and a joint-limb fusion branch.

[0027] Optionally, the limb-joint fusion branch is:

[0028] F fused_joint =Φ(M adjustj Stack(F root ,F branch ,F leaf )+F limb2joint );

[0029] Among them, F root =Ω root (M root ·F limb );

[0030] F branch =GA(Ω branch (M branch ·F limb )+G root );

[0031] F leaf =GA(Ω leaf (M leaf ·F limb )+G branch );

[0032] In the formula, F fused_joint represents the joint-limb fusion feature based on the joint, (M adjustj represents the joint index matrix, Stack(*) represents the stacking operation, F root Represents the mapping pooling feature from joint to root node, F bracnch Represents the mapping pooling feature from joint to branch node, F leaf ) represents the joint leaf node feature, F limb2joint ) represents the mapping pooling feature from limb to leaf node, Ω root (*) indicates the root node pooling operation, M root Represents the root node to limb mapping transformation matrix, F limb represents limb features, GA(*) represents global attention calculation, Ω branch (*) indicates the branch node pooling operation, M branch represents the branch node to limb mapping transformation matrix, G root represents the global feature of the root node, Ω leaf (*) indicates the leaf node pooling operation, M leaf represents the leaf node to limb mapping transformation matrix, G branch ) represents the fusion of branch nodes;

[0033] The joint-limb fusion branches are:

[0034] F fusd_limb =Φ(M adjustl Stack(F rootl ,F branchl ,F leafl )+F joint2limb );

[0035] F fused_limb represents the limb-based joint-limb fusion feature, M adjustl represents the limb index matrix, Frootl Represents the mapping pooling feature from limb to root node, F branchl Represents the mapping pooling feature from limb to branch node, F leafl Represents node features.

[0036] Optionally, the joint predicted position data is:

[0037]

[0038] In the formula, Represents joint predicted position data, TransformerDecoder(*) represents joint-limb decoder;

[0039] The limb prediction data is:

[0040]

[0041] In the formula, Represents limb prediction data.

[0042] Optionally, the post-reasoning strategy is:

[0043]

[0044] in, represents the joint coordinates of the corrected human posture, represents the coordinates of the first joint in the corrected human posture, represents the coordinates of the second joint in the corrected human posture, represents the coordinates of the Nth joint in the corrected human posture, S j represents the confidence of the jth joint, S k Represents the confidence of the k-th joint, where the k-th joint is the other end point of the limb with the j-th joint as one end point.

[0045] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned refined human body posture estimation method.

[0046] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned refined human body posture estimation method.

[0047] In a fourth aspect, the present application provides a computer program product, including a computer program, which implements the above-mentioned refined human body posture estimation method when executed by a processor.

[0048] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0049] The present application provides a refined human posture estimation method, device, medium and product, and sets a refined human posture estimation framework, including: a joint-limb embedding module, a structure-semantic feature fusion module, a hierarchical joint-limb fusion encoder, a joint-limb decoder and a post-inference module connected in sequence, which fuses semantic features into structural information, fuses joints and limbs into a tree structure, explicitly models limb patterns separately from joint patterns, and treats limbs as vectors with fixed lengths and joints. Significant progress has been made in the accuracy of posture estimation, and the average precision (AP) under various posture estimators and conditions has been significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 This is a flowchart of a refined human posture estimation method in one embodiment of the present application.

[0052] Figure 2 This is a framework diagram for refining human posture estimation in one embodiment of the present application.

[0053] Figure 3 It is a structural schematic diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0056] Despite advances in pose refinement techniques, current methods often fail to effectively address two key issues. First, global structural information needs to be incorporated while retaining accurate local context, a balance that current methods often fail to achieve. Second, different joints have unique biomechanical effects and are affected by various types of noise introduced by the environment, a factor that is often overlooked by existing methods. In an exemplary embodiment, considering the above two factors, a refined human pose estimation method is provided that has high human pose estimation accuracy in crowded scenes.

[0057] like Figure 1 and Figure 2 As shown, a refined human posture estimation method is provided, including:

[0058] Step 101: Build and train a refined human pose estimation framework. The refined human pose estimation framework includes: a joint-limb embedding module, a structure-semantic feature fusion module, a hierarchical joint-limb fusion encoder, a joint-limb decoder, and a post-reasoning module connected in sequence.

[0059] Step 102: Input the image to be tested into the posture estimator to obtain semantic features and initial posture estimation results. The initial posture estimation results include joint data and limb data.

[0060] Step 103: Based on the initial posture estimation result, the joint structure features and limb structure features are determined using the joint-limb embedding module. The joint structure features are:

[0061] F J =Pos j +(E J (J init )).

[0062] In the formula, F J Indicates the structural characteristics of the joint, Pos j represents the joint position encoding, E J Represents joint point data, J init Represents joint data.

[0063] The structural characteristics of the limbs are:

[0064] F L =Pos l +(F L (L init ).

[0065] In the formula, F L Indicates the structural characteristics of the limbs, Pos l Indicates the limb position code, E L Represents limb data, L initRepresents limb data.

[0066] Step 104: Based on the semantic features, joint structure features, limb structure features and multiple joint heat maps, a structure-semantic feature fusion module is used to determine joint fusion features and limb fusion features. The joint fusion features are:

[0067] F fuse_joint =Φ(Concat(F sem ⊙H J ,F struct_joint ).

[0068] In the formula, F fuse_joint represents joint fusion features, Φ(*) represents multi-layer perceptron operation, Concat(*) represents concatenation operation, F sem represents the semantic features output by the pose estimator, ⊙ represents the dot product, H J Represents the heat map of all joints, F struct_joint ) represents the structural features of joint points.

[0069] The characteristics of limb fusion are:

[0070] F fuse_limb =Φ(Concat(Ω(M j2l ·(F sem ⊙H J )),F struc_limb )).

[0071] In the formula, F fuse_limb represents the limb fusion feature, Ω(*) represents the depth-wise separable convolution with kernel and stride 2, M j2l represents the transformation matrix mapping the connected joint j to the limb l, F struc_limb Represents the structural characteristics of the limbs.

[0072] Step 105: Determine the joint-limb fusion feature by using a hierarchical joint-limb fusion encoder according to the joint fusion feature and the limb fusion feature. The hierarchical joint-limb fusion encoder includes a limb-joint fusion branch and a joint-limb fusion branch.

[0073] Limb-joint fusion branches are:

[0074] F fused_joint =Φ(M adjustj Stack(F root ,F branch ,F leaf )+F limb2joint ).

[0075] Among them, F root =Ω root (M root ·Flimb ).

[0076] F branch =GA(Ω branch (M branch ·F limb )+G root ).

[0077] F leaf =GA(Ω leaf (M leaf ·F limb )+G branch ).

[0078] In the formula, F fused_joint represents the joint-limb fusion feature based on the joint, (M adjustj represents the joint index matrix, Stack(*) represents the stacking operation, F root Represents the mapping pooling feature from joint to root node, F bracnch Represents the mapping pooling feature from joint to branch node, F leaf ) represents the joint leaf node feature, F limb2joint ) represents the mapping pooling feature from limb to leaf node, Ω root (*) indicates the root node pooling operation, M root represents the root node to limb mapping transformation matrix, F limb represents limb features, GA(*) represents global attention calculation, Ω branch (*) indicates the branch node pooling operation, M branch represents the branch node to limb mapping transformation matrix, G root represents the global feature of the root node, Ω leaf (*) indicates the leaf node pooling operation, M leaf represents the leaf node to limb mapping transformation matrix, G branch ) indicates the fusion of branch nodes.

[0079] The joint-limb fusion branches are:

[0080] F fusd_lomb =Φ(M adjustl Stack(F rootl ,F branchl ,F leafl )+F joint2limb );

[0081] F fused_limb represents the limb-based joint-limb fusion feature, M adjustl represents the limb index matrix, F rootl Represents the mapping pooling feature from limb to root node, F branchlRepresents the mapping pooling feature from limb to branch node, F leafl Represents node features.

[0082] Step 106: Based on the joint structure features, limb structure features and joint-limb fusion features, a joint-limb decoder is used to determine joint prediction position data and limb prediction data. The joint prediction position data is:

[0083]

[0084] In the formula, Represents the joint prediction position data, and TransformerDecoder(*) represents the joint-limb decoder.

[0085] The limb prediction data is:

[0086]

[0087] In the formula, Represents limb prediction data.

[0088] Step 107: According to the joint prediction position data and the limb prediction data, the post-reasoning module is used to determine the corrected human body posture based on the post-reasoning strategy. The post-reasoning strategy is:

[0089]

[0090] in, represents the joint coordinates of the corrected human posture, represents the coordinates of the first joint in the corrected human posture, represents the coordinates of the second joint in the corrected human posture, represents the coordinates of the Nth joint in the corrected human posture, S j represents the confidence of the jth joint, S k Represents the confidence of the k-th joint, where the k-th joint is the other end point of the limb with the j-th joint as one end point.

[0091] This application provides a refined human posture estimation method that can achieve higher-precision posture estimation on a terminal device and implement related applications based on posture estimation on the terminal device. The core of this processing method is that the joints and limbs are reorganized according to the root-branch-leaf pattern in design and classified according to the mechanical pattern of human body movements. The overall framework diagram is as follows Figure 1 The framework consists of a joint-limb embedding module, a structure-semantic feature fusion module, a hierarchical joint-limb fusion encoder, a joint-limb decoder, and a post-reasoning module, which are connected sequentially.

[0092] Among them, the principle of the joint-limb embedding module is as follows:

[0093] The joint-limb embedding module is responsible for converting the initial pose estimation results into rich feature representations. Given the initial pose coordinates, the confidence score (x,y) represents the coordinates, and s represents the confidence level. This module maps joint coordinates to limb vectors using a predefined configuration. The embedding procedure first initializes the joint and limb embedding layers Emb respectively. j and Emb l , and the corresponding position code Pos j and Pos l , represents the first real number field, N j Represents the number of joint points in the real number domain.

[0094] In order to construct the limb data L init , this application defines the mapping matrix M according to the predefined limb index l , as shown below:

[0095] L init =M l ·J init .

[0096] Among them, M l is the mapping matrix from joint to limb, J init Represents joint data (confidence scores). Calculate the span x of each limb in the x and y directions span and span , and determine the direction θ using:

[0097] θ = arctan(y span ,x span ).

[0098] This direction θ is attached to the limb data.

[0099] Then, the joint data and the calculated limb data are respectively transmitted through embedding and position encoding to obtain the joint structure feature F J and limb structure characteristicsF L :

[0100] F J =Pos j +(E J (J init )).

[0101] F L =Pos l +(F L (L init ).

[0102] The position encoding part uses the sinusoidal position encoding method, which is defined as:

[0103]

[0104] Where t is temperature, c is index, and C is dimension. x i represents any unknown number. The output of this module includes joint F J and limbs F L The rich structural features are used as the input of the subsequent stage InferTrans (i.e., the method described in this embodiment). After the embedding process, the limb data is separated into key point positions as individual body structure data representing different human anatomical structures, and the length and direction of each body part are encoded at the key point position.

[0105] Among them, the principle of the structure-semantic feature fusion module is as follows:

[0106] Since joint and limb data are encoded as high-dimensional structural features, coordinate-based models lack local contextual information and visual clues. Therefore, the present application first follows the design in DGN to extract joint-related semantic features from the preset posture estimator. Specifically, the present application uses a normalized joint heat map as a high-pass filter, and samples semantic features from the last layer before the posture estimator regression head according to the confidence weights encoded in the joint heat map. Based on the confidence weight, a mixed information of all joint features that are encountering target interaction or occlusion is generated, and the normalized heat map weights under the feature sampling window are combined as the visual clue information of the model. Then, the present application uses the transformation matrix of the corresponding joint to construct the limb features, and uses a deep separable convolution to dynamically select the joint feature contribution of each limb. The joint limb semantic feature sampling can be expressed as:

[0107]

[0108] in, is the semantic feature of the pose estimator, is the heat map of joint j. j2l is the transformation matrix that maps the connected joint to a limb, and Ω is a depth-wise separable convolution with a kernel and stride of 2, which dynamically selects the feature with the greatest contribution from two connected joints. and are semantic features of joints and limbs, and All represent real number domain; avgpooling represents average pooling; F vis_j and f j Both represent the semantic features of joint points.

[0109] To further combine the semantic features with the structural features, multi-layer perceptrons (MLPs) are used to densely fuse them after the concatenation operation. The overall semantic and structural fusion module can be described as follows:

[0110]

[0111] Among them, Φ represents the MLPs used for feature fusion, F sem The semantic features F that refer to the pose estimator semantic The process of joint and limb embedding ensures the fusion of local environment and visual cues, introducing and dynamically selecting useful visual cues for joints and limbs, thereby enhancing the representation for subsequent pose refinement.

[0112] The principle of the hierarchical joint-limb fusion encoder is as follows.

[0113] Joint grouping in combination with a hierarchical structure is crucial to capture complex dependencies and different meanings of different body parts. The proposed joint-limb fusion encoder divides joints into three levels: root, branch, and leaf, which facilitates the structured representation of human pose. This hierarchical grouping is inspired by the natural biomechanics of human behavior and effectively handles the differences between joint categories, which is crucial for accurate pose estimation.

[0114] Root joints, such as shoulders and hips, are usually stable, with minimal noise and movement once human motion is determined. These joints usually remain fixed and represent the position of the individual. Branch joints are the conduits for arm and leg motion, transmitting the body's motion to operating joints such as the wrist and ankle. Leaf joints, as the executors of human motion, often have articulation and intertwining of content due to the malleability of the limbs. This clear distinction allows for more precise treatment of the different behaviors and characteristics of joints.

[0115] The fusion phase is divided into two branches: limb-to-joint fusion and joint-to-limb fusion.

[0116] Limb-to-joint fusion. The limb-to-joint fusion branch first defines the transformation matrices for root-to-limb, branch-to-limb, and leaf-to-limb mappings, denoted as M root 、M branch and M leaf These matrices map joint features from corresponding limb features. The root node is aggregated by three branches, using a custom pooling layer Ω with depthwise separable convolution. root Pooling is performed to ensure that the high-dimensional structure effectively condenses natural features while retaining basic information. The limb-to-joint pooling operation is expressed as:

[0117] F root =Ω root (M root ·Flimb ).

[0118] Where F limb is the limb feature. The global root feature aggregates information from higher levels and affects downstream features. branch The branch nodes are pooled and their features are combined with the global root features and residual connections are performed as follows:

[0119] F branch =GA(Ω branch (M branch ·F limb )+G root ).

[0120] For leaf nodes, a similar pooling operation is used, and its features are affected by the global branch features:

[0121] F leaf =GA(Ω leaf (M leaf ·F limb )+G branch ).

[0122] Using the predefined index matrix M adjustj The fused limb-to-joint features are concatenated and adjusted to ensure the correct order. These features are then fused with the residual joint features using MLPs to form a comprehensive feature representation for further processing. The entire limb-joint fusion branch can be formulated as:

[0123] F fused_joint =Φ(M adjustj Stack(F root ,F branch ,F leaf )+F limb2joint ).

[0124] Joint to limb fusion. The entire joint-limb fusion branch can be described as:

[0125] F fused_limb =Φ(M adjust1 Stack(F root1 ,F branch1 ,F leaf1 )+F joint2limb ).

[0126] This hierarchical joint-to-limb fusion encoder effectively captures the complex interdependencies between joints and limbs, enabling robust and accurate pose estimation even in challenging scenarios. By leveraging the stability of root joints, the motion transfer role of branch joints, and the high mobility of leaf joints, the model can handle a variety of dynamic human poses more accurately. The use of depthwise separable convolutions ensures the preservation of essential features while reducing computational complexity. This structured approach enhances the model's ability to handle occlusions and intertwined content, resulting in more reliable and accurate pose estimates.

[0127] The joint-limb decoder works as follows.

[0128] In order to improve the initial pose estimation results and reduce noise, this application adds a joint-limb decoder module after the joint-limb fusion encoder. The fused semantic structure joint-limb feature information is generalized back to the input joint and limb features, thereby improving the accuracy and robustness of the final pose estimation. The decoder module also includes two branches, joints and limbs, with a standard transformer decoder initialized with embedded joint and limb data. The decoder improves the joints and limbs by utilizing the attention mechanism to obtain dependency and context information from the memory features. This process can be expressed as:

[0129]

[0130] The inclusion of the joint-limb decoder serves multiple purposes. First, it allows the model to improve the pose estimation results by incorporating information from fused features, which contain rich semantic and structural context. Second, the attention mechanism inside the transformer decoder helps focus on the most relevant parts of the input features, thereby reducing the impact of noise and improving the accuracy of pose estimation. The model benefits from a refined and denoised representation of the pose data. This can improve performance, especially in challenging scenes where the initial pose estimate may be affected by occlusions or content interleaving.

[0131] The principle of the post-reasoning module is as follows:

[0132] In order to achieve accurate supervision and improvement of pose estimation, the model of this application adopts different loss functions for joint and limb branches. For the joint branch, this application uses the coordinate classification head under the SimCC framework and adopts the following KL divergence loss to supervise the final joint coordinates to ensure accurate joint positioning.

[0133]

[0134] Among them J (x) , J (y) , G (x) , G (y)are the predicted SimCC vector and true SimCC vector of the x-axis and y-axis, σ is the softmax function, N and M are the number of samples and the number of joints. On the other hand, the limb branch uses a simple L1 loss to measure the distance between the improved predicted and true limb lengths. In addition, this application designs a separate head to regress the direction of each limb, which is supervised by a smooth L1 loss combined with the sine of the predicted and true directions. The loss functions of these limb branches are expressed as:

[0135]

[0136] Combining all these losses, the final loss function of the model in this application is:

[0137]

[0138] where α, β, and γ are weighting factors that balance the contribution of each individual loss component, Represents the KL divergence loss of the joint point.

[0139] To maximize the benefits of limb branching, we propose a post-inference strategy to improve the performance of the final result. Limb branching not only improves the learning ability of the model, but also improves the inference accuracy. Using the predicted limb lengths and orientations, we can calculate the joint coordinates based on the most confident root joint. The confidence scores are generated from the SimCC head. We then fuse the joint localizations inferred by the predicted limbs according to the corresponding joint scores. The steps after inference are summarized as follows: Given the joint coordinates, Its confidence score For each joint j in predictedjoints:

[0140]

[0141] Among them, L j represents the predicted position of the limb at joint j, and k represents the other end of the limb connected to joint j. This post-reasoning strategy effectively uses the predicted limb lengths and orientations to refine the joint coordinates, thereby improving the overall accuracy of pose estimation.

[0142] InferTrans is evaluated on three widely used crowd human pose estimation datasets: CrowdPose dataset, OCHuman dataset dataset, and OCPose dataset. In addition, this application also tests the final pose estimation performance of the method on the popular COCO keypoint dataset.

[0143] Results on the CrowdPose dataset. On the crowded pose estimation dataset CrowdPose, InferTrans is evaluated against the state-of-the-art methods, focusing on the average precision (AP) metric at different crowd levels (APeasy, APmedium, and APhard). An overall comparison with the SOTA methods is shown in Table 1. In Table 1, a quantitative comparison with the state-of-the-art methods on the CrowdPose test set is shown. The default input resolution of the experiment is 256×192. The best result is shown in bold and the second best is underlined. The pose optimization results are shown in Table 2, where the default input resolution of the experiment is 256×192, and InferTrans is implemented as a post-refinement model for multiple pose estimators such as HRnet, SimpleBaseline, TransPose, HRFormer, I2R-Net, and butd. As shown in Table 1, the InferTrans proposed in this application converts I 2 The performance of SOTA pose estimators such as R-Net and BUCTD is improved by 0.6 and 0.5 mean average precision respectively. It is worth noting that in Table 2, for previous baselines such as HRnet, SimpleBaseline, TransPose and HRFormer, InferTrans achieves even greater improvements (HRnet4.2 mean average precision, SimpleBaseline5.8 mean average precision, TransPose3.5 mean average precision, HRFormer3.3 mean average precision).

[0144] Table 1 Results on the CrowdPose dataset

[0145]

[0146] Table 2 Quantitative comparison of pose refinement of several pose estimators on the CrowdPose dataset

[0147]

[0148] These results show that InferTrans effectively denoises pose results, exploits semantic features, and discovers structural relationships between joints and prostheses. The greater improvement of TransPose over HRFormer may stem from the fact that our model is built on a transformer-based architecture that works better with the semantic features extracted by TransPose, which is also a transformer-based approach. Overall, the results on the CrowdPose dataset show that InferTrans is an effective post-optimization procedure for conventional pose estimators, validating the combination of structural and semantic information to optimize pose results and mitigate noise introduced by inaccurate visual cues.

[0149] Results on the OCHuman dataset. Table 3 is the comparison results on the OCHuman test set. The model is trained on the OCHuman validation set. As shown in Table 3, this application observes similar phenomena to the results on the CrowdPose dataset. The proposed InferTrans is able to enhance the performance of any off-the-shelf pose estimator, showing significant average precision improvements for each pose estimator (e.g., 8.5 average precision for SimpleBaseline, 3.6 average precision for HRFormer, and 3.8 average precision for transse). It is worth noting that compared with CrowdPose, on OCHuman, InferTrans shows a greater performance improvement, highlighting its effectiveness in optimizing poses in the presence of occlusion.

[0150] Table 3 Comparison of InferTrans trained on the OCHuman validation set with the most advanced methods on the OCHuman test set

[0151]

[0152] Results on the OCPose dataset. We trained InferTrans on the OCPose dataset and compared the results with various methods. OCPose focuses on scenes involving a set of actions, evaluating the pose estimation capabilities under intertwined and interfering individual influences. As shown in Table 4, InferTrans improves the pose estimation results similar to its performance on the CrowdPose and OCHuman datasets. However, the improvements are not significant, with a gain of 5.0 mean average precision for HRnet, 2.8 mean average precision for OPEC-Net, and 2.3 mean average precision for DGN. It is worth noting that both OPEC-Net and DGN belong to the same pose optimization category as InferTrans, but the results still show a significant degree of improvement.

[0153] Table 4 Comparison with the most advanced methods on the OCPose dataset

[0154]

[0155] This emphasizes the benefit of fusing joint and limb representations in improving pose estimation accuracy.

[0156] Results on COCO keypoint dataset. We evaluate the generalization ability of InferTrans on the COCO keypoint dataset. It is worth noting that COCO contains about 48% single-person images, which makes the proposed InferTrans designed for crowded scenes less suitable. As shown in Table 5, InferTrans slightly improves the performance of HRNet, TransPose, and HRFormer by 0.5 mean average precision, 0.4 mean average precision, and 0.2 mean average precision, respectively. This improvement may come from the refinement ability of our model despite minimal mutual interference and occlusion. The results show that even in the absence of content and occlusion interleaving, the rich structural information in InferTrans can still improve some inaccuracies in joint localization.

[0157] Table 5 COCO pose estimation validation set results

[0158]

[0159] A comprehensive analysis of the proposed InferTrans on four benchmark datasets (CrowdPose, OCHuman, OCPose, and COCO Keypoint) demonstrates its strong performance and generalization in various situations.

[0160] Pose estimation accuracy, especially in crowded and heavily occluded settings, as demonstrated by significant AP improvements on the CrowdPose and OCHuman datasets. This enhancement highlights the model's ability to denoise pose results and effectively exploit structural relationships between joints and limbs, supported by semantic features. Although the improvements on the OCPose dataset are slightly less pronounced, InferTrans still shows substantial progress, confirming its usefulness in complex human interactions and mutual interference settings. On the COCO keypoint dataset, where images are mostly characterized by single-person scenes with minimal occlusion, the marginal performance gains suggest the model's inherent refinement capabilities, even if its main advantage is less applicable. Overall, InferTrans performs well in dealing with crowded and occluded environments, demonstrating its versatility and effectiveness as a post-improvement tool to improve the accuracy of state-of-the-art pose estimators.

[0161] Joints and limbs are reorganized in a root-branch-leaf setting and classified according to the biomechanics of human behavior. Root joints and branches are taken as examples. Visualization results between InferTrans and HRNet on the OCHuman dataset are shown. The method can effectively handle dense scenes and accurately locate missing or misestimated joints. Specifically, the proposed InferTrans is able to distinguish misplaced joints from mutual interference (as shown in all samples) and infer occluded joints from a body using predicted joint locations and limb attributes (as shown in the third sample). Even in the case of occlusion, InferTrans can infer reasonable poses, demonstrating its robustness and accuracy.

[0162] InferTrans was implemented using PyTorch on an Nvidia RTX 4090 GPU with 24GB of memory. The training process used the Adam optimizer and the cosine annealing learning rate scheduler. Model training included an epoch for learning rate warmup and spanned 160 stages on three datasets: CrowdPose, OCPose, and COCO keypoint datasets. For the final improved pose estimation results, the SimCC coordinate representation method with a split ratio of 2.0 was adopted. The training batch size was set to 16 for input crops with a resolution of 256×192 and 8 for input crops with a resolution of 384×288. The hidden dimension of the structural and semantic features of each joint or limb was set to 256. The joint-limb fusion encoder consists of 4 layers, and the first layer does not contain joint semantic features. This application compares the evaluation of this application with the state-of-the-art methods on the above datasets, and uses the CrowdPose dataset for all ablation analyses.

[0163] By using a transformer-based architecture to encode global structural information while preserving local context, a hierarchical joint-limb-semantic fusion module is introduced, and limb patterns are explicitly modeled separately, seeking to significantly improve the accuracy and robustness of pose estimation, ultimately improving the performance of traditional methods in crowded scenes and complex situations.

[0164] This application introduces a new pose optimization model that aims to improve the accuracy of human pose estimation, especially in challenging crowded or occluded environments. InferTrans adopts a transformer-based architecture to effectively encode global structural information while maintaining local patterns of joints and limbs. In addition, an innovation of hierarchical fusion module is proposed to integrate semantic features into joint and limb models through a tree structure. This design enables effective information exchange between joints and limbs at different structural levels, improving the model's ability to understand complex poses. In addition, modeling limb structural patterns separately as vectors defining length and direction enhances the ability to infer human pose from limb information alone, significantly improving pose estimation and overall performance. Extensive evaluations on four different datasets (CrowdPose, OCHuman, OCPose, and COCO keypoint datasets) demonstrate the effectiveness of InferTrans. The model achieves significant improvements in pose estimation accuracy, with significant improvements in average precision (AP) across a variety of pose estimators and conditions. These results highlight the potential of InferTrans as a powerful plug-in post-processing tool to improve pose estimation in crowded and complex scenes.

[0165] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a refined human posture estimation method is implemented.

[0166] Those skilled in the art will understand that Figure 3The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0167] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0168] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0170] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0171] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0172] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The present application uses specific examples to illustrate the principles and implementation methods of the present application. The above examples are only used to help understand the method and core ideas of the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A refined human body posture estimation method, characterized in that: include: Build and train a refined human pose estimation framework; The refined human posture estimation framework includes: a joint-limb embedding module, a structure-semantic feature fusion module, a hierarchical joint-limb fusion encoder, a joint-limb decoder and a post-reasoning module connected in sequence; Inputting the image to be tested into a posture estimator to obtain semantic features and initial posture estimation results; the initial posture estimation results include joint data and limb data; Determining joint structure features and limb structure features using the joint-limb embedding module according to the initial posture estimation result; Determine joint fusion features and limb fusion features using the structure-semantic feature fusion module according to the semantic features, the joint structure features, the limb structure features and a plurality of joint heat maps; Determine the joint-limb fusion feature using the hierarchical joint-limb fusion encoder according to the joint fusion feature and the limb fusion feature; Determine joint prediction position data and limb prediction data using a joint-limb decoder according to the joint structure features, the limb structure features and the joint-limb fusion features; According to the joint prediction position data and the limb prediction data, a post-reasoning module is used to determine the corrected human body posture based on a post-reasoning strategy.

2. The refined human posture estimation method according to claim 1, characterized in that: The joint structure features are: F J =Pos j +(E J (J init )); In the formula, F J Indicates the structural characteristics of the joint, Pos j represents the joint position encoding, E J Represents joint point data, J init Represents joint data; The limb structure features are: F L =Pos l +(E L (L init ); In the formula, F L Indicates the structural characteristics of the limbs, Pos l Indicates the limb position code, E L Represents limb data, L init Represents limb data.

3. The refined human posture estimation method according to claim 2, characterized in that: The joint fusion features are: F fuse_joint =Φ(Concat(F sem ⊙H J ,F struct_joint ); In the formula, F fuse_joint represents joint fusion features, Φ(*) represents multi-layer perceptron operation, Concat(*) represents concatenation operation, F sem represents the semantic features output by the pose estimator, ⊙ represents the dot product, H J Represents the heat map of all joints, F struct_joint ) represents the structural features of joint points; The limb fusion features are: F fuse_limb =Φ(Concat(Ω(M j2l ·(F sem ⊙H J )),F struc_limb )); In the formula, F fuse_limb represents the limb fusion feature, Ω(*) represents the depth-wise separable convolution with kernel and stride 2, M j2l represents the transformation matrix mapping the connected joint j to the limb l, F struc_limb Represents the structural characteristics of the limbs.

4. The refined human posture estimation method according to claim 3, characterized in that: The hierarchical joint-limb fusion encoder includes a limb-joint fusion branch and a joint-limb fusion branch.

5. The refined human posture estimation method according to claim 4, characterized in that: The limb-joint fusion branches are: F fused_joint =Φ(M adjustj ·Stack(F root ,F branch ,F leaf )+F limb2joint ); Among them, F root =Ω root (M root ·F limb ); F branch =GA(Ω branch (M branch ·F limb )+G root ); F leaf =GA(Ω leaf (M leaf ·F limb )+G branch ); In the formula, F fused_joint represents the joint-limb fusion feature based on the joint, (M adjustj represents the joint index matrix, Stack(*) represents the stacking operation, F root Represents the mapping pooling feature from joint to root node, F bracnch Represents the mapping pooling feature from joint to branch node, F leaf ) represents the joint leaf node feature, F limb2joint ) represents the mapping pooling feature from limb to leaf node, Ω root (*) indicates the root node pooling operation, M root Represents the root node to limb mapping transformation matrix, F limb represents limb features, GA(*) represents global attention calculation, Ω branch (*) indicates the branch node pooling operation, M branch represents the branch node to limb mapping transformation matrix, G root represents the global feature of the root node, Ω leaf (*) indicates the leaf node pooling operation, M leaf represents the leaf node to limb mapping transformation matrix, G branch ) represents the fusion of branch nodes; The joint-limb fusion branches are: F fusd_limb =Φ(M adjustl ·Stack(F rootl ,F branchl ,F leafl )+F joint2limb ); F fused_limb represents the limb-based joint-limb fusion feature, M adjustl represents the limb index matrix, F rootl Represents the mapping pooling feature from limb to root node, F branchl Represents the mapping pooling feature from limb to branch node, F leafl Represents node features.

6. The refined human posture estimation method according to claim 5, characterized in that: The joint predicted position data is: In the formula, Represents joint predicted position data, TransformerDecoder(*) represents joint-limb decoder; The limb prediction data is: In the formula, Represents limb prediction data.

7. The refined human posture estimation method according to claim 6, characterized in that: The post-reasoning strategy is: in, represents the joint coordinates of the corrected human posture, represents the coordinates of the first joint in the corrected human posture, Represents the coordinates of the second joint in the corrected human posture, represents the coordinates of the Nth joint in the corrected human posture, S j represents the confidence of the jth joint, S k Represents the confidence of the k-th joint, where the k-th joint is the other end point of the limb with the j-th joint as one end point.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the refined human body posture estimation method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the refined human posture estimation method described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the refined human posture estimation method described in any one of claims 1 to 7 is implemented.

Images