A method, system, and device for human pose estimation based on Transformer and diffusion models.

By combining the feature extraction-pose estimation-diffusion prediction algorithm of Transformer and diffusion model, the problems of occlusion and illumination variation in human pose estimation in video scenes are solved, achieving efficient and high-precision human pose estimation while reducing computational complexity and error.

CN118447536BActive Publication Date: 2026-11-17ZJU HANGZHOU GLOBAL SCI & TECH INNOVATION CENT
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410575937.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-10
Publication Date
2026-11-17
Estimated Expiration
2044-05-10

AI Technical Summary

Technical Problem

Existing technologies for human pose estimation in video scenes suffer from problems such as human occlusion, self-occlusion, lighting changes, and background interference, making it difficult for algorithms to accurately detect and estimate human pose, resulting in low computational complexity and efficiency.

Method used

By combining Transformer and diffusion models, a pose estimation model is constructed through frame-by-frame sampling and key point annotation. Leveraging the omniscience of Transformer and the reasoning ability of diffusion models, a feature extraction-pose estimation-diffusion prediction algorithm is designed. This algorithm only needs to observe the two support frames before and after the current frame. By utilizing the continuity of human motion and fine pose representation, it achieves efficient and high-precision human pose estimation.

Benefits of technology

It enhances the model's reasoning ability, avoids erroneous estimation, achieves a balance between high efficiency and high accuracy, reduces computational costs, and lowers attitude estimation errors caused by factors such as motion occlusion and blurring.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118447536B_ABST
    Figure CN118447536B_ABST
Patent Text Reader

Abstract

This invention discloses a human pose estimation method, system, and apparatus based on Transformer and diffusion models. The method includes: acquiring a human pose estimation dataset; extracting features from the target pose frame and support pose frames to obtain target frame pose feature maps and support frame pose feature maps; constructing a pose estimation model based on the Transformer architecture, generating a support frame pose heatmap set and a true pose heatmap set using the target frame pose feature maps, support frame pose feature maps, and keypoint data, and generating a predicted pose heatmap set based on the pose support heatmap set; constructing and training a human pose feature extraction model based on the diffusion model to obtain a human pose feature extraction model; and predicting the pose estimation images to be tested in the test set using the human pose estimation model to obtain human pose data. This method simultaneously utilizes the content omniscience of Transformer and the reasoning ability of the diffusion model to handle occlusion problems in pose estimation, enhancing the model's reasoning ability and avoiding erroneous estimations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a human pose estimation method, system, and device based on Transformer and diffusion models. Background Technology

[0002] In recent years, human pose estimation has attracted widespread attention and research from both industry and academia. Current human pose estimation techniques are mostly image-level, and performing human pose estimation in video scenes often faces numerous challenges. Among these, occlusion and self-occlusion issues prevent the accurate detection and estimation of certain human poses in videos, easily leading to errors in body part detection and pose estimation. Furthermore, in scenes with human movement, changes in lighting and background can interfere with the accurate detection and estimation of human poses, making it difficult for algorithms to accurately and effectively analyze and process image data in complex scenes. Finally, human pose estimation in video scenes requires processing large amounts of image and video data; therefore, the complexity and efficiency of the algorithm become critical issues. Designing an efficient algorithm and optimizing computational efficiency are among the major challenges in current video human pose estimation research.

[0003] Existing technologies disclose a human pose estimation method employing a spatiotemporal Transformer model. This method extracts, adjusts, interacts with, and decouples single-frame features, target frame features, spatiotemporal features, and keypoint features to recover accurate human pose from videos with noisy or incomplete information. Its significant advantages are efficiency and robustness. Another existing 2D human pose estimation method employs a diffusion model and forward and backward diffusion formulas. This method can recover clear joint heatmaps from noisy samples, thereby estimating the pose in the image. Its significant advantages are high accuracy, high generalization, and high robustness, while also being able to adjust the balance between inference speed and accuracy according to the needs of different scenarios. The above methods only consider human pose estimation using a simple Transformer or diffusion model, without considering combining an improved Transformer and an improved diffusion model for human pose estimation in video scenes. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by providing a human pose estimation method based on Transformer and diffusion models, comprising the following steps:

[0005] A human pose video is acquired, and frame-by-frame sampling and key point annotation are performed on the human pose video to obtain a human pose estimation dataset, which is then divided into a training set and a test set. The human pose estimation dataset includes pose target frames and pose support frames.

[0006] The pose feature maps of the target frame and the pose support frame are extracted from the pose target frame and pose support frame, and the corresponding key point data are generated.

[0007] A pose estimation model based on the Transformer architecture is constructed. The target frame pose feature map, the support frame pose feature map, and key point data are used to generate a support frame pose heat map set and a true pose heat map set. Based on the pose support heat map set, a predicted pose heat map set and predicted key point data are generated.

[0008] A human pose feature extraction training model based on a diffusion model is constructed, a pose constraint loss function is built, and the human pose feature extraction training model is trained based on support frame pose heatmaps, real pose heatmaps, predicted pose heatmaps and predicted key point data. The parameters of the human pose feature extraction training model are continuously updated iteratively according to the loss function until the loss function converges, thus obtaining the human pose feature extraction model.

[0009] Human pose data is obtained by predicting the pose estimation images of the test set using a human pose feature extraction model.

[0010] As one possible implementation, the human pose feature extraction training model includes a mask attention module and a diffusion recovery module, and the working steps of the human pose feature extraction training model include:

[0011] The mask attention module calculates the predicted pose heatmap and the support frame pose heatmap and converts them into the constraints of the diffusion recovery module.

[0012] The true attitude heatmap is used to obtain a noisy feature map through the forward diffusion process of the diffusion recovery module;

[0013] The noisy feature map and constraints are used as inputs to the back-diffusion process of the diffusion recovery module for n iterations of prediction learning, resulting in a more accurate predicted posture heatmap set and outputting accurate human posture estimation results.

[0014] As one possible implementation method, the feature extraction steps are as follows:

[0015] A convolutional neural network is used to extract features from the input pose target frame and pose support frame to obtain the pose feature map of the target frame and the pose feature map of the support frame.

[0016] Keypoint data is generated by using a linear projection layer and embedding absolute position vectors.

[0017] As one possible implementation, the attitude estimation steps are as follows:

[0018] Construct a pose estimation model based on the Transformer architecture;

[0019] The pose estimation module concatenates the keypoint data with the joint type data, and then inputs the concatenation result into the J2V Transformer Block.

[0020] Map the d-dimensional joint label output by the J2V Transformer Block using a linear projection. The one-dimensional feature vector is then mapped and reshaped to obtain a pose information heatmap for the two support frames.

[0021] As one possible implementation, the J2V Transformer Block includes M multi-head self-attention blocks, each block including a regularization layer L1, a multi-head self-attention mechanism layer, a regularization layer L2, and a feedforward neural network. The process of the J2V Transformer Block outputting the d-dimensional joint label is as follows:

[0022] Feedforward neural networks perform nonlinear transformations on the input vector using two linear transformation layers and activation functions.

[0023] As the network deepens, different joint labels learn visual cues and constraint relationships from the interaction of visual labels and gradually converge to some corresponding local regions. In the last few layers, they gradually locate the final position through adjacency constraints and symmetry constraints, thus obtaining d-dimensional joint labels.

[0024] As one possible implementation, the attention mechanism of the multi-head self-attention mechanism layer is shown in the following equation:

[0025]

[0026] Among them W Q W K and W V T are the learnable parameters of three linear projection layers. l-1 It is the output of the (l-1)th layer, Att(T) l-1 ) is the output of self-attention, where d is the dimension of the label. h =d,T represents transpose. Multi-head self-attention is an extension of self-attention, with h self-attention operations. Multi-head self-attention projects the query vector matrix Q, key vector matrix K, and value vector matrix V through different linear transformations and concatenates the different attention results.

[0027] As one possible implementation, the coarse heatmap acquisition step of the target frame includes:

[0028] The coarse prediction block uses linear interpolation to obtain a coarse heatmap prediction result for the target frame from the two attitude information heatmaps.

[0029] This invention also provides a human pose estimation system based on Transformer and diffusion models, comprising: a data acquisition module, a feature extraction module, a pose estimation module, a model training module, and a pose prediction module.

[0030] The data acquisition module is used to acquire human posture videos, perform frame-by-frame sampling and key point annotation based on the human posture videos to obtain a human posture estimation dataset, and divide it into a training set and a test set. The human posture estimation dataset includes posture target frames and posture support frames.

[0031] The feature extraction module is used to extract features from the attitude target frame and attitude support frame to obtain the attitude feature map of the target frame and the attitude feature map of the support frame, and generate the corresponding key point data.

[0032] The attitude estimation module is used to construct a coarse prediction model based on linear interpolation, using the target frame attitude feature map, support frame attitude feature map and key point data to generate a support frame attitude heat map set and a real attitude heat map set, a predicted key point dataset, and generate a predicted attitude heat map set and predicted key point data based on the attitude support heat map.

[0033] The model training module is used to construct a human pose feature extraction training model based on a diffusion model, build a pose constraint loss function, train the human pose feature extraction training model based on support frame pose heatmaps, real pose heatmaps, predicted pose heatmaps and predicted key point data, and continuously iterate and update the parameters of the human pose estimation training model according to the loss function until the loss function converges, thus obtaining the human pose feature extraction model.

[0034] The posture prediction module is used to predict the posture estimation images of the test set through the human posture feature extraction model and the key point prediction model to obtain human posture data.

[0035] A computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the following method:

[0036] A human pose video is acquired, and frame-by-frame sampling and key point annotation are performed on the human pose video to obtain a human pose estimation dataset, which is then divided into a training set and a test set. The human pose estimation dataset includes pose target frames and pose support frames.

[0037] The pose feature maps of the target frame and the pose support frame are extracted from the pose target frame and pose support frame, and the corresponding key point data are generated.

[0038] A pose estimation model based on the Transformer architecture is constructed. The target frame pose feature map, the support frame pose feature map, and key point data are used to generate a support frame pose heat map set and a true pose heat map set. Based on the pose support heat map set, a predicted pose heat map set and predicted key point data are generated.

[0039] A human pose feature extraction training model based on a diffusion model is constructed, a pose constraint loss function is built, and the human pose feature extraction training model is trained based on support frame pose heatmaps, real pose heatmaps, predicted pose heatmaps and predicted key point data. The parameters of the human pose feature extraction training model are continuously updated iteratively according to the loss function until the loss function converges, thus obtaining the human pose feature extraction model.

[0040] Human pose data is obtained by predicting the pose estimation images of the test set using a human pose feature extraction model.

[0041] A human pose estimation device based on Transformer and diffusion models includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the following method:

[0042] A human pose video is acquired, and frame-by-frame sampling and key point annotation are performed on the human pose video to obtain a human pose estimation dataset, which is then divided into a training set and a test set. The human pose estimation dataset includes pose target frames and pose support frames.

[0043] The pose feature maps of the target frame and the pose support frame are extracted from the pose target frame and pose support frame, and the corresponding key point data are generated.

[0044] A pose estimation model based on the Transformer architecture is constructed. The target frame pose feature map, the support frame pose feature map, and key point data are used to generate a support frame pose heat map set and a true pose heat map set. Based on the pose support heat map set, a predicted pose heat map set and predicted key point data are generated.

[0045] A human pose feature extraction training model based on a diffusion model is constructed, a pose constraint loss function is built, and the human pose feature extraction training model is trained based on support frame pose heatmaps, real pose heatmaps, predicted pose heatmaps and predicted key point data. The parameters of the human pose feature extraction training model are continuously updated iteratively according to the loss function until the loss function converges, thus obtaining the human pose feature extraction model.

[0046] Human pose data is obtained by predicting the pose estimation images of the test set using a human pose feature extraction model.

[0047] In summary, the human pose estimation method, system, and apparatus based on Transformer and diffusion models described in this application have the following advantages:

[0048] 1. This invention utilizes both the content omniscience of Tansformer and the reasoning ability of the diffusion model to address occlusion issues in pose estimation, thereby enhancing the model's reasoning ability and avoiding erroneous estimations;

[0049] 2. This invention designs a feature extraction-pose estimation-diffusion prediction algorithm, which only needs to observe two support frames before and after the current frame and utilize the continuity of human motion and fine pose representation to complete human pose estimation in video scenes, achieving a balance between high efficiency and high accuracy.

[0050] 3. This invention uses an inter-frame sampling strategy to reduce the number of video frames that need to be processed, improve the efficiency of pose estimation, reduce pose estimation errors caused by motion occlusion, blurring and other factors, and significantly reduce computational costs. Attached Figure Description

[0051] The following figures illustrate specific embodiments of this application:

[0052] Figure 1 This is a schematic diagram of the human pose estimation method based on Transformer and diffusion model of the present invention;

[0053] Figure 2 This is an overall schematic diagram of the human pose estimation system based on Transformer and diffusion model of the present invention;

[0054] Figure 3 This is a schematic diagram of human posture data obtained by estimating video images using the present invention. Detailed Implementation

[0055] This invention employs a video pose estimation method based on Transformer and diffusion models, and designs a feature extraction-pose estimation-diffusion prediction algorithm. It only requires observing two support frames before and after the current frame, and utilizing the continuity of human motion and fine pose representation, to complete human pose estimation in a video scene. The invention will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are explanations of the invention, but the invention is not limited to the following embodiments.

[0056] Example 1:

[0057] A human pose estimation method based on Transformer and diffusion models, such as Figure 1 As shown, it includes the following steps:

[0058] S100. Obtain human posture video, perform frame-by-frame sampling and key point annotation based on human posture video to obtain human posture estimation dataset, and divide it into training set and test set. The human posture estimation dataset includes posture target frame and posture support frame. The posture support frame is based on two video frames before and after the posture target frame.

[0059] S200. Extract the pose feature map of the target frame and the pose support frame from the pose target frame and the pose feature map of the support frame, and generate the corresponding key point data.

[0060] S300. Construct a pose estimation model based on the Transformer architecture, generate a support frame pose heatmap set and a true pose heatmap set using the target frame pose feature map, support frame pose feature map and key point data, and generate a predicted pose heatmap set based on the pose support heatmap set.

[0061] S400. Construct a human pose feature extraction training model based on a diffusion model, build a pose constraint loss function, train the human pose feature extraction training model based on support frame pose heatmaps, real pose heatmaps, predicted pose heatmaps and predicted key point data, and iteratively update the parameters of the human pose feature extraction training model according to the loss function until the loss function converges, thus obtaining the human pose feature extraction model.

[0062] S500: The human posture feature extraction model is used to predict the posture estimation images in the test set to obtain human posture data.

[0063] This invention utilizes both the content omniscience of Tansformer and the reasoning ability of the diffusion model to address occlusion issues in pose estimation, enhancing the model's reasoning capabilities and avoiding erroneous estimations. A feature extraction-pose estimation-diffusion prediction algorithm is designed, requiring only observation of two support frames before and after the current frame. By leveraging the continuity of human motion and refined pose representation, human pose estimation in video scenes can be completed, achieving a balance between high efficiency and high accuracy. An inter-frame sampling strategy is employed to reduce the number of video frames that need to be processed, improving the efficiency of pose estimation and reducing pose estimation errors caused by motion occlusion, blurring, and other factors, while significantly reducing computational costs.

[0064] In this embodiment, step S100 involves acquiring a human pose video, performing frame-by-frame sampling and key point annotation based on the human pose video to obtain a human pose estimation dataset, and dividing it into a training set and a test set. The human pose estimation dataset includes pose target frames and pose support frames, and specifically includes the following steps:

[0065] S110: In this embodiment, the PoseTrack2017 dataset was selected as the input dataset. This dataset includes 514 video sequences and 16,219 location annotations, including the poses of multiple human bodies and the annotations of 15 human body key points.

[0066] S120: Divide the dataset into 250, 50, and 214 video sequences for training, validation, and testing. The dataset is annotated with 15 key points and additional labels for key visibility.

[0067] S130: The training set videos provide dense pose annotations in the middle 30 frames, while the validation set videos provide pose annotations every four frames.

[0068] This embodiment uses an inter-frame sampling strategy to reduce the number of video frames that need to be processed, improve the efficiency of pose estimation, reduce pose estimation errors caused by motion occlusion, blurring, and other factors, and significantly reduce computational costs.

[0069] In step S200, the pose feature map of the target frame and the pose feature map of the support frame are extracted from the pose target frame and the pose feature map of the support frame, and the corresponding key point data is generated, including the following steps:

[0070] Step S210: Build a human pose estimation module. For each input video frame, use a convolutional neural network to process the two support frames. and Feature extraction is performed to obtain their respective feature maps. and The size is W×H×C, where W and H are 1 / 8 of the width and height of the input image, and C is the number of channels;

[0071] Step S220: Then, the extracted feature map is uniformly divided into N patches and flattened into one-dimensional vectors;

[0072] Step S230: Specify the linear projection layer and embed the absolute position vector to generate the corresponding key point data.

[0073] In step S300, a pose estimation model based on the Transformer architecture is constructed. Support frame pose feature maps, support frame pose feature maps, and keypoint data are used to generate a support frame pose heatmap set and a true pose heatmap set. A predicted pose heatmap set is then generated based on the pose support heatmap set. This includes the following steps:

[0074] Step S310: Construct a pose estimation model based on the Transformer architecture for the keypoint data sequence. The image was obtained by stitching together the annotations of 15 human body key points. Then The input is fed into a J2V Transformer Block consisting of M multi-head self-attention blocks for feature extraction and transformation;

[0075] Step S320: The J2V Transformer Block based on Transformer learns the feature representation of key points by stacking 12 blocks. Each block consists of a regularization layer L1, a multi-head self-attention mechanism layer, a regularization layer L2, and a feedforward neural network connected in sequence. The input of the regularization layer L2 is the superposition of the output of the multi-head self-attention mechanism layer and the input of the regularization layer L1. The feedforward neural network performs non-linear transformation on the input vector through two linear transformation layers and activation functions. As the network deepens, different key point data learn visual cues and constraint relationships from the interaction of human key point annotations and gradually converge to some corresponding local regions. In the last few layers, the final position is gradually located through adjacency constraints and symmetry constraints.

[0076] The Attention mechanism of the multi-head self-attention mechanism layer is shown in the following equation:

[0077]

[0078] Among them: W Q W K and W V T are the learnable parameters of three linear projection layers. l-1 It is the output of the (l-1)th layer, Att(T) l-1 ) is the output of Self-Attention, where d is the dimension of the tokens. h =d,T denotes transpose. Multi-head self-attention is an extension of self-attention, with h self-attention operations. Multi-head self-attention projects the query vector matrix Q, key vector matrix K, and value vector matrix V through different linear transformations and concatenates the different attention results.

[0079] Step S330: After the multi-head self-attention mechanism, residual connections and layer normalization are performed, followed by a feedforward neural network layer. Two linear transformations and activation functions are used to perform non-linear transformations on the vectors at each position, thereby enhancing the model's expressive power. The Dropout, Layer Normalization, and Feedforward layers are the same as the original transformer. Finally, the 15 d-dimensional human keypoint annotations output by the J2V Transformer Block are mapped using linear projection. The 1D feature vector is then mapped and reshaped into a support frame pose feature map. The same method can be used to obtain another support frame pose feature map. and target frame pose feature map

[0080] Step S340: Construct a coarse prediction block based on linear interpolation to perform preliminary recovery of the intermediate frames.

[0081] This invention utilizes the heatmap of support frame pose information obtained in the previous step. and Obtain the coarse heatmap prediction results of the target frame. To restore temporal continuity and consistency, newly inserted sampling points are estimated and filled based on the linear relationship between existing sampling points. Based on the human joint heatmap information from the preceding and following frames, the coordinate difference between each joint in the two frames is calculated, multiplied by the interpolation coefficient, to obtain the interpolation offset. This interpolation offset is then added to the joint coordinates in the first frame to obtain the joint coordinates in the inserted frame. Linear interpolation can quickly restore sparse pose sequences to a continuous state without requiring excessive computational resources, facilitating subsequent processing and analysis.

[0082] In step S400, a human pose feature extraction training model based on a diffusion model is constructed, a pose constraint loss function is established, and the human pose feature extraction training model is trained based on support frame pose heatmaps, real pose heatmaps, predicted pose heatmaps, and predicted keypoint data. The parameters of the human pose feature extraction training model are iteratively updated according to the loss function until the loss function converges, resulting in the human pose feature extraction model. This includes the following steps:

[0083] Step S410: Construct a diffusion recovery block based on a diffusion model architecture;

[0084] Step S420: Obtain the coarse heatmap prediction result of the target frame. and support frame pose information heatmap and A masking operation is performed, and the heatmap information after the masking operation is fed into the Attention module for calculation to obtain inter-frame correlation information and output the result. Then, it is summed with Q to fuse the weighted information into the original features, enhancing the features of important regions and suppressing the features of unimportant regions. Finally, it is compared with the coarse heatmap prediction result of the target frame. Perform splicing and use it as constraint X. c Input into the diffusion recovery block;

[0085] Step S430: Real-time posture heatmap The noise-added feature map is obtained through the forward diffusion process. Then add noise to the feature map and constraint X cThe predicted attitude heatmap is obtained by performing T iterations of prediction learning as input to the back-diffusion process.

[0086] Step S440: Use the MSE loss function to compare the actual pose heatmap. and predicted attitude heatmap During the reasoning process, the model incorporates the constraint X. c The true heatmap of the target frame is recovered from the random noise map, and the accurate 2D human pose estimation result is decoded by the decoder module;

[0087] Step S450: Train the model using video frames from the training set. Based on the loss function, use the optimizer to iteratively update the model parameters using gradient descent until the loss function converges. Training is then complete, and the human pose feature extraction model is obtained.

[0088] This embodiment utilizes both the content omniscience of Tansformer and the reasoning ability of the diffusion model to handle occlusion and other issues in pose estimation, thereby enhancing the model's reasoning ability and avoiding erroneous estimations.

[0089] This embodiment designs a feature extraction-pose estimation-diffusion prediction algorithm. It only needs to observe two support frames before and after the current frame and utilize the continuity of human motion and fine pose representation to complete human pose estimation in video scenes, achieving a balance between high efficiency and high accuracy.

[0090] In step S500, the human pose estimation images in the test set are predicted using a human pose feature extraction model to obtain human pose data, including the following steps:

[0091] By inputting video frames from the test set into the trained model, the pose information of the target person in the video frame, namely the position coordinates and connections of each key point, can be predicted. Figure 3 As shown.

[0092] Example 2:

[0093] A fine-grained video pose estimation system based on Transformer and diffusion models is proposed. A feature extraction-pose estimation-diffusion prediction algorithm is designed for fine-grained video pose estimation, such as... Figure 2 As shown, the module includes a data acquisition module 100, a feature extraction module 200, a pose estimation module 300, a model training module 400, and a pose prediction module 500.

[0094] The data acquisition module 100 is used to acquire human posture video, perform frame-by-frame sampling and key point annotation based on human posture video to obtain human posture estimation dataset, and divide it into training set and test set. The human posture estimation dataset includes posture target frame and posture support frame.

[0095] The feature extraction module 200 is used to extract features from the attitude target frame and attitude support frame to obtain the attitude feature map of the target frame and the attitude feature map of the support frame, and generate corresponding key point data.

[0096] The attitude estimation module 300 is used to construct a coarse prediction model based on linear interpolation, using the target frame attitude feature map, support frame attitude feature map and key point data to generate a support frame attitude heat map set and a real attitude heat map set, a prediction key point dataset, and generate a prediction attitude heat map set based on the attitude support heat map.

[0097] The model training module 400 is used to construct a human pose feature extraction training model based on a diffusion model, build a pose constraint loss function, train the human pose feature extraction training model based on support frame pose heatmaps, real pose heatmaps, predicted pose heatmaps and predicted key point data, and continuously iterate and update the parameters of the human pose estimation training model according to the loss function until the loss function converges, thus obtaining the human pose feature extraction model.

[0098] The posture prediction module 500 is used to predict the posture estimation images of the test set through the human posture feature extraction model and the key point prediction model to obtain human posture data.

[0099] Various changes and modifications made without departing from the spirit and scope of this invention, and all equivalent technical solutions, also fall within the scope of this invention.

[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0101] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This invention is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] It should be noted that:

[0106] The phrase "an embodiment" or "an embodiment" used in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Therefore, the phrase "an embodiment" or "an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0107] Furthermore, it should be noted that the shapes and names of the components in the specific embodiments described in this specification may differ. All equivalent or simple variations made to the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, as long as they do not depart from the structure of this invention or exceed the scope defined in these claims, they should all fall within the protection scope of this invention.

Claims

1. A human pose estimation method based on Transformer and diffusion models, characterized in that, Includes the following steps: A human pose video is acquired, and frame-by-frame sampling and key point annotation are performed on the human pose video to obtain a human pose estimation dataset, which is then divided into a training set and a test set. The human pose estimation dataset includes pose target frames and pose support frames. The pose feature maps of the target frame and the pose support frame are extracted from the pose target frame and pose support frame, and the corresponding key point dataset is generated. A pose estimation model based on the Transformer architecture is constructed. The target frame pose feature map, the support frame pose feature map, and key point data are used to generate a support frame pose heat map set and a true pose heat map set. Based on the support frame pose heat map set, a predicted pose heat map set and predicted key point data are generated. A human pose feature extraction training model based on a diffusion model is constructed, a pose constraint loss function is built, and the human pose feature extraction training model is trained based on support frame pose heatmaps, real pose heatmaps, predicted pose heatmaps and predicted key point data. The parameters of the human pose feature extraction training model are continuously updated iteratively according to the loss function until the loss function converges, thus obtaining the human pose feature extraction model. Human pose data is obtained by predicting the pose estimation images of the test set through a human pose feature extraction model. The human pose feature extraction training model includes a mask attention module and a diffusion recovery module. The working steps of the human pose feature extraction training model include: The mask attention module calculates the predicted pose heatmap set and the support frame pose heatmap set, and transforms them into the constraints of the diffusion recovery module; the real pose heatmap set obtains a noisy feature map through the forward diffusion process of the diffusion recovery module; The noisy feature map and constraints are used as inputs to the back-diffusion process of the diffusion recovery module for n iterations of prediction learning, resulting in a more accurate predicted posture heatmap set and outputting accurate human posture estimation results. The attitude estimation steps are as follows: Construct a pose estimation model based on the Transformer architecture; The pose estimation model concatenates the keypoint data with the human keypoint data, and then inputs the concatenation result into the J2V Transformer Block. Map the d-dimensional joint label output by the J2V Transformer Block using a linear projection. The 1D feature vector is then mapped and reshaped to obtain a pose information heatmap for the two support frames. The J2V Transformer Block comprises M multi-head self-attention blocks. Each block includes a regularization layer L1, a multi-head self-attention mechanism layer, a regularization layer L2, and a feedforward neural network. The process of the J2V Transformer Block outputting the d-dimensional joint label is as follows: Feedforward neural networks perform nonlinear transformations on the input vector through two linear transformation layers and activation functions; As the network deepens, different joint labels learn visual cues and constraint relationships from the interaction of visual labels and gradually converge to some corresponding local regions. In the last few layers, they gradually locate the final position through adjacency constraints and symmetry constraints, thus obtaining d-dimensional joint labels.

2. The human pose estimation method based on Transformer and diffusion model according to claim 1, characterized in that, The feature extraction steps are as follows: A convolutional neural network is used to extract features from the input pose target frame and pose support frame to obtain the pose feature map of the target frame and the pose feature map of the support frame. Keypoint data is generated by using a linear projection layer and embedding absolute position vectors.

3. The human pose estimation method based on Transformer and diffusion model according to claim 1, characterized in that, The attention mechanism of the multi-head self-attention mechanism layer is shown in the following equation: in , and These are the learnable parameters of three linear projection layers. It is the first The output of the layer, This is the output of self-attention, where d is the dimension of the label. T represents transpose. Multi-head self-attention is an extension of self-attention, with h self-attention operations. Multi-head self-attention projects the query vector matrix Q, key vector matrix K, and value vector matrix V through different linear transformations and concatenates the different attention results.

4. A human pose estimation system based on Transformer and diffusion models, characterized in that, include: The module includes a data acquisition module, a feature extraction module, a pose estimation module, a model training module, and a pose prediction module. The data acquisition module is used to acquire human posture videos, perform frame-by-frame sampling and key point annotation based on the human posture videos to obtain a human posture estimation dataset, and divide it into a training set and a test set. The human posture estimation dataset includes posture target frames and posture support frames. The feature extraction module is used to extract features from the pose target frame and pose support frame to obtain the pose feature map of the target frame and the pose feature map of the support frame, and generate the corresponding key point dataset. The attitude estimation module is used to construct a coarse prediction model based on linear interpolation, using the target frame attitude feature map, support frame attitude feature map and key point data to generate a support frame attitude heat map set and a real attitude heat map set, a predicted key point dataset, and based on the support frame attitude heat map set, to generate a predicted attitude heat map set and predicted key point data. The model training module is used to construct a human pose feature extraction training model based on a diffusion model, build a pose constraint loss function, train the human pose feature extraction training model based on support frame pose heatmaps, real pose heatmaps, predicted pose heatmaps and predicted key point data, and continuously iterate and update the parameters of the human pose estimation training model according to the loss function until the loss function converges, thus obtaining the human pose feature extraction model. The posture prediction module is used to predict the posture estimation images of the test set through the human posture feature extraction model and the key point prediction model to obtain human posture data. The human pose feature extraction training model includes a mask attention module and a diffusion recovery module. The working steps of the human pose feature extraction training model include: The mask attention module calculates the predicted pose heatmap set and the support frame pose heatmap set, and transforms them into the constraints of the diffusion recovery module; the real pose heatmap set obtains a noisy feature map through the forward diffusion process of the diffusion recovery module; The noisy feature map and constraints are used as inputs to the back-diffusion process of the diffusion recovery module for n iterations of prediction learning, resulting in a more accurate predicted posture heatmap set and outputting accurate human posture estimation results. The attitude estimation steps are as follows: Construct a pose estimation model based on the Transformer architecture; The pose estimation model concatenates the keypoint data with the human keypoint data, and then inputs the concatenation result into the J2V Transformer Block. Map the d-dimensional joint label output by the J2V Transformer Block using a linear projection. The 1D feature vector is then mapped and reshaped to obtain a pose information heatmap for the two support frames. The J2V Transformer Block comprises M multi-head self-attention blocks. Each block includes a regularization layer L1, a multi-head self-attention mechanism layer, a regularization layer L2, and a feedforward neural network. The process of the J2V Transformer Block outputting the d-dimensional joint label is as follows: Feedforward neural networks perform nonlinear transformations on the input vector through two linear transformation layers and activation functions; As the network deepens, different joint labels learn visual cues and constraint relationships from the interaction of visual labels and gradually converge to some corresponding local regions. In the last few layers, they gradually locate the final position through adjacency constraints and symmetry constraints, thus obtaining d-dimensional joint labels.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 3.

6. A human pose estimation device based on Transformer and diffusion models, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 3.