Human body posture estimation and tracking system and human body posture estimation and tracking method

Abstract

The present invention provides a human posture estimation and tracking system, comprising: a posture estimation unit, configured to estimate the human posture in the current frame image and output an estimation result; a posture prediction unit, configured to predict the human posture in the current frame based on the dynamic changes of the human posture in multiple consecutive frames of images before the current frame, and output a prediction result; and a fusion unit, configured to match and fuse the estimation result of the posture estimation unit with the prediction result of the posture prediction unit. The fusion method provided by the present invention can simultaneously use the dynamic information from the human posture and the visual information from the image itself. Thanks to the posture prediction of the graph neural network, the human posture estimation and tracking method based on human dynamics modeling provided by the present invention can produce reliable human posture estimation and tracking results even in the presence of motion blur and occlusion.

Description

Technical Field

[0001] This invention relates generally to the field of image recognition technology, and more particularly to a human pose estimation and tracking system, and a method for human pose estimation and tracking of video images. Background Technology

[0002] Existing human pose estimation and tracking systems can be broadly categorized into two types: top-down and bottom-up approaches. Bottom-up methods first generate candidate human joints for all human bodies in an image, then group these candidate joints into groups representing individual human bodies. These grouped joints are then correlated across frames to generate the final human pose estimation and tracking result. Top-down methods first perform human detection, generating bounding boxes for each individual. Then, joint detection is performed within each bounding box to generate the human pose for each individual. Correlating these generated human poses across frames yields the human pose estimation and tracking result.

[0003] While both approaches have achieved promising results, previous methods for human pose estimation and tracking remain prone to missed detections, especially in complex backgrounds and fast-moving scenes. A key reason for this is their heavy reliance on a detection-then-association process: first, keypoints or human figures are estimated from image information; then, these detections are correlated intra-frame or cross-frame to obtain the tracking result. In challenging scenarios (e.g., crowded or blurry scenes), the lack of sufficient image information inevitably leads to missed detections of human keypoints or human detectors, making it impossible to recover these missed keypoints in subsequent processes. Although some previous works have proposed methods to correct and improve this deficiency, they still largely depend on purely visual information and therefore cannot fundamentally solve these missed detection problems.

[0004] The content in the background section is merely technology known to the public and does not necessarily represent existing technology in this field. Summary of the Invention

[0005] In view of at least one deficiency of the prior art, the present invention provides a human pose estimation and tracking system, comprising:

[0006] The pose estimation unit is configured to estimate the human pose in the current frame image and output the estimation result;

[0007] The pose prediction unit is configured to predict the human pose in the current frame based on the dynamic changes of human pose in the previous multiple consecutive frames and output the prediction result.

[0008] The fusion unit is configured to match and fuse the estimation results of the attitude estimation unit and the prediction results of the attitude prediction unit.

[0009] According to one aspect of the invention, the attitude estimation unit is further configured to:

[0010] Perform human detection in the current frame image and generate at least one human bounding box;

[0011] Each human body bounding box is cropped separately, and the cropped image is scaled to a preset size;

[0012] Output feature maps and heat maps of human joints based on the scaled image, and obtain the estimation results based on the heat maps.

[0013] According to one aspect of the invention, the attitude estimation unit is further configured to:

[0014] The scaled image is input into the backbone network, which then outputs feature maps and heat maps of the human joints.

[0015] According to one aspect of the invention, the number of heat maps of the human joints is the number of human joints.

[0016] According to one aspect of the invention, the feature map of the human joint includes: visual features, positional features, and type features of each human joint.

[0017] According to one aspect of the invention, the visual features of the human joints are obtained through the backbone network.

[0018] According to one aspect of the present invention, the positional features of the human body joints are determined by the position of the maximum value in the heatmap corresponding to each joint, the distance from the center of the human body posture to the corresponding human body bounding box, and the image scaling ratio.

[0019] According to one aspect of the invention, the type feature is obtained through a learnable embedding feature matrix.

[0020] According to one aspect of the invention, the attitude prediction unit is further configured to:

[0021] The graph neural network takes the human pose tracking sequence of multiple consecutive frames preceding the current frame as input and outputs the result of predicting the human pose of the current frame.

[0022] According to one aspect of the present invention, the human pose tracking sequence includes a human pose tracking sequence graph, wherein the nodes in the human pose tracking sequence graph are human joints in multiple consecutive frames preceding the current frame, and the lines in the human pose tracking sequence graph are connections between human joints in each frame and connections between corresponding human joints in adjacent frames.

[0023] According to one aspect of the invention, the human posture tracking sequence graph includes: visual features, positional features, and type features of each human joint.

[0024] According to one aspect of the invention, the visual features and type features of the human joints are obtained by the pose estimation unit during the estimation of multiple consecutive frames preceding the current frame.

[0025] According to one aspect of the invention, the positional features of the human joints are obtained by normalizing the center of the human pose in the last frame of the human pose tracking sequence.

[0026] According to one aspect of the invention, the fusion unit is further configured to:

[0027] The estimation results of the posture estimation unit and the prediction results of the posture prediction unit are mapped one-to-one using the Hungarian algorithm, where the similarity is calculated based on the positions of all human joints.

[0028] According to one aspect of the invention, the fusion unit is further configured to:

[0029] Map the first human joint point in the prediction result to the estimation result;

[0030] When there is no corresponding human joint in the estimation result, the first human joint is mapped to the estimation result as the second human joint.

[0031] According to one aspect of the invention, the fusion unit is further configured to:

[0032] Align the center of the heatmap of the estimated result with the center of the heatmap of the predicted result, and then merge them by averaging the heatmaps.

[0033] The present invention also provides a method for human pose estimation and tracking in video images, which is accomplished by the human pose estimation and tracking system described above.

[0034] This invention provides a human pose estimation and tracking system. It inputs a sequence of human pose tracking data from multiple consecutive frames preceding the current frame into a graph neural network, outputting a predicted human pose for the current frame. The predicted human pose is then matched and fused with the human pose detection result of the current frame to obtain the final human pose estimation and tracking result for the current frame. The fusion method provided by this invention can simultaneously utilize dynamic information from the human pose and visual information from the image itself. Thanks to the pose prediction of the graph neural network, the human pose estimation and tracking method based on human dynamics modeling provided by this invention can produce reliable human pose estimation and tracking results even in the presence of motion blur and occlusion. This invention does not rely on the detection of human joints or human bounding boxes, thus solving the problem of missed detections from a more fundamental perspective. Attached Figure Description

[0035] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0036] Figure 1 This invention illustrates a human pose estimation and tracking system according to one embodiment of the present invention;

[0037] Figure 2 The working process of a human posture estimation and tracking system according to one embodiment of the present invention is shown;

[0038] Figure 3 The working process of an attitude prediction unit according to one embodiment of the present invention is shown;

[0039] Figure 4 This illustrates an embodiment of the present invention, which describes the process of aggregating information on human keypoint features in a historical tracking sequence.

[0040] Figure 5 The human posture detection results of one embodiment of the present invention are shown;

[0041] Figure 6 This invention illustrates a method for human pose estimation and tracking in video images, according to one embodiment of the present invention. Detailed Implementation

[0042] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0043] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0044] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection, an electrical connection, or a connection that allows for communication; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0045] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0046] The following disclosure provides many different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the invention. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. In addition, examples of various specific processes and materials are provided in this invention, but those skilled in the art will recognize the application of other processes and / or the use of other materials.

[0047] The embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0048] This invention performs dynamic modeling of human posture using image sequences, and proposes a system for human posture estimation and tracking based on this modeling.

[0049] According to a preferred embodiment of the present invention, such as Figure 1 As shown, the human posture estimation and tracking system 100 provided by the present invention includes a posture estimation unit 110, a posture prediction unit 120, and a fusion unit 130. Wherein:

[0050] The pose estimation unit 110 is configured to estimate the human pose in the current frame image and output the estimation result.

[0051] The pose prediction unit 120 is configured to predict the human pose in the current frame based on the dynamic changes of human pose in multiple consecutive frames of images preceding the current frame, and output the prediction result.

[0052] The fusion unit 130 is configured to match and fuse the estimation results of the attitude estimation unit 110 and the prediction results of the attitude prediction unit 120.

[0053] The following reference Figure 2The working process of the human pose estimation and tracking system 100 is described below. The human pose estimation and tracking system 100 provided by this invention consists of two paths: one for single-frame pose estimation and the other for pose prediction. For the input current frame (t), the pose estimation unit 110 first estimates the human pose of the current frame (t), obtaining a detection result based on pure visual information. Simultaneously, the pose prediction unit 120 makes a pose prediction for the current frame. The pose prediction unit 120 predicts the pose of the current frame (t) based on the human pose tracking results in historical frames, such as (t-1), (t-2), and (t-3) as shown in the figure. The fusion unit 130 matches these two sets of results: the single-frame pose estimation result and the pose prediction result, and fuses the matched poses to obtain the fused human pose. Figure 2 As shown, the motion postures of skaters 1, 3, and 4 were all detected by the posture estimation unit 110. However, skater 2 was not detected by the posture estimation unit 110 because she was occluded by skater 3 in the current frame. Therefore, the motion posture of skater 2 was missed in the posture estimation result. The posture prediction unit 120, based on the human posture tracking results in historical frames (t-1), (t-2), and (t-3), can predict the motion posture of skater 2 in the current frame. Therefore, the motion posture of skater 2 is present in the posture prediction result. Through fusion by the fusion unit 130, the missed motion posture of skater 2 was recovered in the final detection result.

[0054] Existing top-down and bottom-up human pose detection methods rely on the detection of human joints or bounding boxes, making them prone to missed detections when these detection methods fail. For example, in the above embodiment, since skater number 2 is occluded by skater number 3 in the current frame, no bounding box is generated, and therefore, no joint detection or pose estimation is performed for skater number 2. The human pose estimation and tracking system 100 provided in the preferred embodiment of the present invention makes human pose prediction in the current frame based on human pose tracking results in historical frames. This prediction does not rely on the detection of human joints or bounding boxes, thus solving the problem of missed detections from a more fundamental perspective.

[0055] According to a preferred embodiment of the present invention, in the human pose estimation and tracking system 100, the pose estimation unit 110 is further configured to:

[0056] Perform human detection in the current frame image and generate at least one human bounding box;

[0057] Each human body bounding box is cropped separately, and the cropped image is scaled to a preset size;

[0058] Output feature maps and heat maps of human joints based on the scaled image, and generate the estimation results based on the heat maps.

[0059] Continue to refer Figure 2 For single-frame human pose estimation, this invention follows a top-down human pose detection method. First, the human pose estimation unit 110 performs human detection in the current frame, generating at least one human bounding box (as shown in the figure, the bounding box of the detected figure skater is displayed). Then, each human body in the current frame is cropped according to the human bounding box, and the cropped image is rescaled to a fixed size. The scaled image is used as input, and feature maps and heat maps of human joints are output based on the scaled fixed-size image. The human pose estimation result is generated based on the heat map of human joints, and the feature maps of human joints are stored for subsequent human pose prediction.

[0060] According to a preferred embodiment of the present invention, in the human pose estimation and tracking system 100, the pose estimation unit 110 is further configured to:

[0061] The scaled image is input into the backbone network, which then outputs feature maps and heat maps of the human joints.

[0062] For example, if HRNet is used as the backbone network, for each human detection result, the human bounding box is cropped and resized to a fixed size of 384 pixels × 288 pixels. The image within the 384 pixel × 288 pixel human bounding box is input into the HRNet backbone network. During the intermediate process, feature maps of human joints are output and stored. Finally, heatmaps of human joints are output, and human pose estimation results are generated based on the heatmaps of each human joint.

[0063] According to a preferred embodiment of the present invention, in the human posture estimation and tracking system 100, the number of heat maps of human joints is the number of human joints.

[0064] The generated heatmap is typically smaller than the input image; for example, when using HRNet as the backbone network, the heatmap size is 96 pixels × 72 pixels. The number of heatmaps is set to the number of human joints; for example, the common setting for the number of human joints is 15 or 17. This invention uses 15 human joints as an example for illustration. Let H... ijk Let the value at position (i, j) in the k-th heatmap be the value. Then the position of the k-th joint is:

[0065]

[0066] in, It is the position of the k-th joint in the heatmap. The position feature of the k-th joint can be transformed by calculating the position of the maximum value in the k-th heatmap and then based on the distance from the center of the human pose to the human bounding box and the image scaling information.

[0067] The training loss of the single-frame pose estimation model is calculated based on the heatmap. The cropped human sample is first scaled to a fixed size, and the corresponding true joint positions are converted into coordinates in the heatmap. Let l k This represents the actual location of the k-th joint in the heatmap. The heatmap of the actual joints is generated according to a two-dimensional Gaussian distribution.

[0068]

[0069] Where σ 2 Set to 3. Train the human pose estimation model by minimizing the loss:

[0070]

[0071] Where H and W represent the height and width of the heatmap, and K is the number of joints.

[0072] According to a preferred embodiment of the present invention, in the human posture estimation and tracking system 100, the feature map of the human joints includes: visual features, positional features and type features of each human joint.

[0073] According to a preferred embodiment of the present invention, the visual features of each human body joint are obtained directly through a backbone network; the positional features of each human body joint are obtained by calculating the maximum position in the corresponding heatmap, the distance from the human body posture center to the human body bounding box, and the image scaling ratio, and the category features of each human body joint are obtained through an embedded learnable matrix.

[0074] According to a preferred embodiment of the present invention, in the human pose estimation and tracking system 100, the pose prediction unit 130 is further configured to:

[0075] The graph neural network takes the human pose tracking sequence of multiple consecutive frames preceding the current frame as input and outputs the result of predicting the human pose of the current frame.

[0076] Figure 3A flowchart illustrating the human posture dynamics modeling and prediction method of this invention is shown. Given human posture tracking results in historical frames, the graph neural network explicitly captures the dynamic changes in the spatiotemporal human posture and makes a posture prediction in the current frame. In the input graph of the graph neural network, nodes are human joints in all historical frame tracking results, and edges are divided into connections between joints within a frame and connections between all relevant nodes in adjacent frames. This construction of the input graph helps the graph neural network capture the relative positional constraints between joints and model human motion.

[0077] According to a preferred embodiment of the present invention, in the human pose estimation and tracking system 100, the human pose tracking sequence includes a human pose tracking sequence graph, wherein the nodes in the human pose tracking sequence graph are human joints in multiple consecutive frames before the current frame, and the lines in the human pose tracking sequence graph are connections between human joints in each frame and connections between corresponding human joints in adjacent frames.

[0078] For each node in the input graph, this invention uses three features for description: visual features, positional features, and type features. Visual features are derived from the backbone network, while type features are obtained through a learnable embedding feature matrix. Both visual and type features can be output and stored during the pose estimation unit's estimation of poses across multiple consecutive frames prior to the current frame. Positional features are obtained by normalizing the center of the last human pose in the historical frame tracking sequence. The graph neural network updates the features of each node through intertemporal and interspatial message passing and outputs a heatmap prediction of the tracking sequence in the current frame. This will be explained in detail below.

[0079] The key points in the historical trajectory and the potential key points of the human pose in the current frame are used as nodes in the graph neural network (GNN) model.

[0080] For each frame, an input map is constructed at each joint by combining three features. The visual features from the backbone network of the pose estimation unit are v. k The joint type encoding is obtained from a learnable lookup table, and is c. k Its 2D position and confidence level from the attitude estimation unit are p k For potential joints in the current frame, their confidence level is set to 1. The 2D positions of all joints are determined based on the human pose P from the last frame of the tracking sequence. t-1 The center is normalized. Standardizing the joint positions relative to the same center helps capture whole-body motion.

[0081] Where k∈1,2,...K, represents the k-th joint type of a given human posture.

[0082] like Figure 2 The MLP shown is a multilayer perceptron. Using a multilayer perceptron (MLP), all joint features are transformed to have the same dimension and then averaged and pooled together. The final feature of the k-th joint is calculated as follows:

[0083] J k =Pooling(MLP) 视觉 (v k MLP 位置 (p k MLP 类型 (c k ))

[0084] The above describes three MLP encoders for different keypoint features: Visual Feature Encoder (MLP) 视觉 Location Feature Encoder (MLP) 位置 Type Feature Encoder (MLP) 类型 They do not share parameters. When constructing J for potential keypoints in the current frame... k At that time, c k It will be partially ignored.

[0085] The input graph of the Graph Neural Network (GNN) model includes two different types of edges: connections between joints within the same frame and connections across consecutive frames. Intra-frame edges enable the GNN model to capture the relative motion and spatial structure of human joints, while cross-frame edges model the dynamics of human pose over time. Two sets of GNN parameters are used when aggregating features from these two types of edges.

[0086] like Figure 4 As shown, in each layer of the Graph Neural Network (GNN) model, the features of the key points are updated through message passing, i.e.

[0087]

[0088] in This represents the feature of the k-th joint in the l-th layer. Let M represent the neighbor set of the k-th node, M represent the message aggregation function that takes all neighbor sets as input and calculates aggregate features, and [∣∣] represent the concatenation of vectors.

[0089] The Graph Neural Network (GNN) model dynamically updates the features at each node based on what it has learned, passing information from each layer to the next. like Figure 4As shown in the example, each layer of the GNN corresponds to information from a historical moment. This information is passed from layer (t-3) to (t-2), then to (t-1), and finally to the current moment (t), thus predicting the joint pose at the current moment t. For pose prediction, each position in the current frame is considered a node and connected to the pose joint in the previous frame of the historical tracking trajectory. The GNN model aggregates features from the positions in the current frame and estimates the pose of each position using these aggregated features to determine the joint type at that position.

[0090] like Figure 4 As shown, the feature number settings for the three multilayer perceptrons (MLPs) are as follows: MLP 类型 : 32→128→512; MLP 位置 : 3→128→512; MLP 视觉 The feature size is converted from 144 to 256 to 512, transforming all three feature types into 512-dimensional vectors to support pooling operations for a unified feature representation. In the human pose prediction step, the number of features predicted by the MLP is set from 512 to 128 to 15, finally reducing the dimensionality to 15, corresponding to the number of human joint types. The search space size in the human pose prediction step is set to 576 pixels × 576 pixels, corresponding to a 144 × 144 feature map. Each point in the feature map is considered a node. These nodes aggregate information from human poses in historical trajectories and are classified by the MLP to generate the probabilities of all 15 types of human joints.

[0091] like Figure 3 As shown in the dashed box, the latent joints in the current frame and the last tracked human pose P are used. t-1 A graph is constructed using the key points in the graph. The learned GNN model is then applied to this graph to update the joint features via message passing.

[0092] Using J as the final feature of the GNN model, another MLP is used to predict each potential joint in the current frame, i.e.

[0093] Prob = MLP pred (J)

[0094] Where Prob represents the probability distribution of all joint types of the input node. The predicted probability distribution of all potential joints in the current frame generates a predicted heatmap of all joints.

[0095] According to a preferred embodiment of the present invention, in the human pose estimation and tracking system 100, the fusion unit 130 is further configured to:

[0096] The estimation results of the posture estimation unit 110 and the prediction results of the posture prediction unit 120 are mapped one-to-one using the Hungarian algorithm, where the similarity is calculated based on the positions of all human joints.

[0097] According to a preferred embodiment of the present invention, in the human pose estimation and tracking system 100, the fusion unit 130 is further configured to:

[0098] Map the first human joint point in the prediction result to the estimation result;

[0099] When there is no corresponding human joint in the estimation result, the first human joint is mapped to the estimation result as the second human joint.

[0100] According to a preferred embodiment of the present invention, in the human pose estimation and tracking system 100, the fusion unit 130 is further configured to:

[0101] Align the center of the heatmap of the estimated result with the center of the heatmap of the predicted result, and then merge them by averaging the heatmaps.

[0102] Existing methods typically match human poses between adjacent frames to generate tracking results. This approach can lead to mismatches when human poses move rapidly and overlap. In a preferred embodiment of the present invention, the predicted human pose and the detected human pose are fused and matched within the same frame. This logic reduces the overall system's mismatch rate.

[0103] Figure 5 The diagram shows the detection results of the human posture estimation and tracking system 100 provided by the present invention for human posture detection, and compares them with the detection results obtained by the traditional top-down detection method.

[0104] According to a preferred embodiment of the present invention, such as Figure 6 As shown, the present invention also provides a method 10 for human pose estimation and tracking of video images, including steps S101-S103, which is completed by the human pose estimation and tracking system described above.

[0105] In step S101, the pose estimation unit estimates the human pose in the current frame image and outputs the estimation result.

[0106] In step S102, the pose prediction unit predicts the human pose in the current frame based on the dynamic changes of human pose in the previous multiple frames and outputs the prediction result.

[0107] In step S103, the estimation results of the attitude estimation unit and the prediction results of the attitude prediction unit are matched and fused by the fusion unit.

[0108] A preferred embodiment of the present invention provides a human pose estimation and tracking system. This system inputs a sequence of human pose tracking data from multiple consecutive frames preceding the current frame into a graph neural network, which outputs a predicted human pose for the current frame. The predicted human pose is then matched and fused with the human pose detection result of the current frame to obtain the final human pose estimation and tracking result for the current frame. This fusion method can simultaneously utilize dynamic information from the human pose and visual information from the image itself. Thanks to the pose prediction of the graph neural network, this human pose estimation and tracking method based on human dynamics modeling can produce reliable human pose estimation and tracking results even in the presence of motion blur and occlusion. The preferred embodiment of the present invention does not rely on the detection of human joints or human bounding boxes, thus solving the problem of missed detections from a more fundamental perspective.

[0109] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A human pose estimation and tracking system, characterized in that, include: The pose estimation unit is configured to estimate the human pose in the current frame image, output the estimation result, and generate the estimation result based on the estimated heatmap of human joints. The pose prediction unit is configured to predict the human pose in the current frame based on the dynamic changes of human pose in the previous multiple consecutive frames and output the prediction result. The human pose tracking sequence of the previous multiple consecutive frames is input into the graph neural network and the result of predicting the human pose in the current frame is output. The human pose tracking sequence includes a human pose tracking sequence graph, which includes: visual features, positional features, and type features of each human joint. The nodes in the human pose tracking sequence graph are human joints in multiple consecutive frames before the current frame, and the lines in the human pose tracking sequence graph are connections between human joints in each frame and connections between corresponding human joints in adjacent frames. The posture prediction unit is further configured to: average pool and merge the visual features, position features and type features of the human joints, pass them layer by layer through the graph neural network to obtain a predicted heatmap of potential human joints in the current frame, and use the predicted heatmap as the prediction result; The fusion unit is configured to match and fuse the estimation results of the attitude estimation unit and the prediction results of the attitude prediction unit in the same frame; The fusion unit is further configured to match the estimation result and the prediction result in the following manner: mapping the estimation result of the pose estimation unit and the prediction result of the pose prediction unit one-to-one using the Hungarian algorithm, wherein the similarity is calculated based on the positions of all human joints; mapping the first human joint in the prediction result to the estimation result; when there is no corresponding human joint in the estimation result, mapping the first human joint as the second human joint to the estimation result. The fusion unit is further configured to fuse the estimation result and the prediction result by aligning the center of the estimated heatmap with the center of the predicted heatmap and merging them together by averaging the heatmaps.

2. The human pose estimation and tracking system as described in claim 1, wherein the pose estimation unit is further configured to: Perform human detection in the current frame image and generate at least one human bounding box; Each human body bounding box is cropped separately, and the cropped image including each human body is scaled to a preset size; Output an estimated heatmap of human joints based on the scaled image, and use the estimated heatmap as the estimation result.

3. The human pose estimation and tracking system as described in claim 2, wherein the pose estimation unit is further configured to: The scaled image is input into the backbone network, which then outputs an estimated heatmap of the human joints.

4. The human posture estimation and tracking system as described in claim 2 or 3, wherein the number of estimated heatmaps of human joints is the number of human joints.

5. The human pose estimation and tracking system as described in claim 2, wherein, The scaled image is input into the backbone network, and the backbone network outputs the feature map of the human joint.

6. The human pose estimation and tracking system as described in claim 5, wherein the feature map of the human joint points includes: Visual, positional, and typological characteristics of each human joint.

7. The human posture estimation and tracking system as described in claim 6, wherein the visual features of the human joints are obtained through the backbone network.

8. The human posture estimation and tracking system as described in claim 6, wherein the positional features of the human joints are determined by the position of the maximum value in the estimated heatmap corresponding to each joint, the distance from the center of the human posture to the corresponding human bounding box, and the image scaling ratio.

9. The human pose estimation and tracking system of claim 6, wherein the type features are obtained through a learnable embedding feature matrix.

10. The human pose estimation and tracking system as described in claim 1, wherein the visual features and type features of the human joints are output during the estimation process of the pose estimation unit on multiple consecutive frames prior to the current frame.

11. The human pose estimation and tracking system as described in claim 1, wherein the positional features of the human joints are obtained by normalizing the center of the human pose in the last frame of the human pose tracking sequence.

12. A method for human pose estimation and tracking of video images, performed by the human pose estimation and tracking system as described in any one of claims 1-11.

Images