Video human pose propagation method based on attention mechanism

Abstract

The application discloses a video human body posture propagation method based on an attention mechanism, designs an end-to-end encoder-decoder architecture based on a pure attention mechanism, the encoder comprises a backbone network and a space-time global fusion module, and the decoder comprises a patch matching correction module; the space-time global fusion module effectively performs global space fusion on the features of multiple video frames by using a self-attention mechanism, solves the problem of missing time sequence information faced by a single-frame image method, the patch matching correction module combines multi-frame posture heat maps and performs patch matching local space correction on the posture heat maps according to the features output by the encoder, and posture in a rough heat map is more finely adjusted. The application improves posture estimation precision under complex actions and occlusion conditions, solves the posture propagation degradation problem under more sparse labeling, reduces the dependence on dense labeling, improves the efficiency of video posture labeling, and explores the feasibility of semi-automatic video posture labeling.

Description

Technical Field

[0001] This invention belongs to the field of pose estimation technology, specifically relating to a video human pose propagation method based on an attention mechanism. Background Technology

[0002] The goal of human pose estimation is to locate key points on a person's body in a given image to accurately identify their pose. This technology is a challenging and cutting-edge field in computer vision, and it is also an important foundation and prerequisite for many other visual tasks, such as medical rehabilitation and motion capture. The ultimate goal of human pose estimation is to connect the skeletal joints of various people in an image according to certain rules, thereby visually representing the person's pose.

[0003] Video-based human pose estimation techniques rely on a large number of densely labeled video frames. These methods assume that the rich correlation information between video frames can only be used for training, ignoring the orderliness and relevance of video sample annotations, i.e., the temporal continuity and spatial similarity between adjacent frames. In video, because human movement is continuous, changes in human pose between adjacent frames are often smooth and gradual. This means that if this temporal and spatial continuity can be effectively utilized, the need for annotation of every single frame can be eliminated. By analyzing the changes between consecutive frames, the pose information of intermediate frames can be inferred without annotating each frame individually; traditional dense annotation is not only time-consuming and labor-intensive but also costly.

[0004] Video pose propagation makes sparse annotation possible, and sparse annotation methods, as a technique to reduce the need for annotation, have received widespread attention. Sparse annotation refers to selecting a portion of the data for annotation, that is, not all data will be labeled. The advantage of doing so is that it can significantly reduce the workload and cost of annotation and make full use of all available data resources. Image-based pose estimation methods cannot be directly used for pose propagation tasks, primarily because these methods can only process single video frames and ignore temporal motion cues. Video-based pose estimation methods, such as those in [Gedas Bertasius, Christoph Feichtenhofer, Du Tran, Jianbo Shi, and Lorenzo Torresani. Learning temporal pose estimation from sparsely-labeled videos. In Advances in Neural Information Processing Systems, pages 3027–3038, 2019.2,3,6], suffer performance degradation in scenarios with occlusion and motion blur, and they cannot effectively utilize long-range temporal information when temporal intervals are large. Another example is [Runyang Feng, Yixing Gao, Xueqing Ma, Tze Ho EldenTse, and Hyung Jin Chang. Mutual information-based temporal difference learning for human pose estimation in video. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages...] The method proposed in 17131~17141, June 2023.1 is the best method for video, but it cannot fuse the label heatmap of the labeled frames during inference, which makes it impossible to perform pose propagation on sparse labeled video datasets.

[0005] Therefore, existing pose estimation methods cannot be directly used for pose propagation tasks. This is mainly because most of these methods are designed for pose estimation of single images and cannot effectively utilize temporal information, especially temporal information with large intervals. Furthermore, in pose propagation tasks, the model is required to fuse the label heatmaps of labeled frames, which most current methods for videos cannot achieve, and thus cannot perform pose propagation tasks well on sparsely labeled video datasets. Summary of the Invention

[0006] In view of the above, the present invention provides a video human pose propagation method based on an attention mechanism, which can solve the problem of missing temporal information faced by single-frame image methods, realize more refined adjustment of pose in coarse heatmaps, improve the accuracy of pose propagation, solve the degradation problem of pose propagation under sparser annotation, and explore the feasibility of semi-automated video pose annotation.

[0007] A video human pose propagation method based on an attention mechanism includes the following steps:

[0008] (1) Obtain and process a video dataset centered on people;

[0009] (2) Construct an end-to-end encoder-decoder model architecture based on a pure attention mechanism, including an encoder and a decoder. The encoder includes a backbone network and a spatiotemporal global fusion module, wherein:

[0010] The backbone network is used to extract human pose features from video images and output feature tokens and pose heatmaps.

[0011] The spatiotemporal global fusion module is used to fuse temporal features and perform global fusion of feature tokens in space;

[0012] The decoder is used for local spatial attitude correction. It stitches the mask heatmap and attitude heatmap together and performs attitude fusion and fine-grained attitude correction with the features output by the encoder to generate a key point heatmap.

[0013] (3) The above model architecture is trained using the video dataset to obtain the video human pose propagation model;

[0014] (4) Input the unlabeled video frame and its labeled support frame into the video human pose propagation model, and the position coordinates of each key point of the person in the video frame can be predicted and output.

[0015] Further, in step (1), for any video segment in the video dataset, n video frames are uniformly selected at intervals m as labeled frames, and the nearest labeled frames in the forward and backward directions are matched for each unlabeled frame as its supporting frames. For each labeled frame, two unlabeled frames with a forward and backward distance of m / 2 are matched as its supporting frames, where m is a natural number greater than 0 and n is a natural number greater than 1.

[0016] Furthermore, the backbone network is based on Vision Transformer, which includes a patch embedding layer, a cascaded Transformer block layer, and an upsampling detection head. The input is a video frame and its two corresponding matching support frames. First, these three frames are concatenated in the batch dimension and sent to the patch embedding layer for embedding compression and adding position information. Then, they are sent to the cascaded Transformer block layer to extract features and obtain feature tokens. Finally, the feature tokens are transformed and sent to the upsampling detection head to obtain a pose heatmap.

[0017] Furthermore, the spatiotemporal global fusion module includes a preprocessing layer, a cascaded self-attention layer, and a linear fusion layer. The input of this module is feature tokens and pose heatmaps. First, the feature tokens are weighted and filtered through the preprocessing layer. Then, the spatial global fusion is performed through the cascaded self-attention layer. Finally, the linear fusion layer is used to reduce the number of feature tokens by three times to achieve input-output correspondence, and finally, the fused features are output.

[0018] Furthermore, the specific implementation of the preprocessing layer is as follows: First, the feature tokens and pose heatmaps are separated in the batch dimension to obtain feature tokens and pose heatmaps at three time points. Then, the three pose heatmaps are reduced to a single channel using 1×1 convolutional blocks and divided into blocks of the same size as the feature tokens. The blocks are then copied along the last dimension to the same size and shape as the corresponding feature tokens, and then multiplied point by point with the corresponding feature tokens to achieve weight filtering. Finally, the three filtered feature tokens are concatenated in the channel dimension as the output.

[0019] Furthermore, the cascaded self-attention layer is used to perform global spatial computation on the features output by the preprocessing layer. This layer is composed of multiple cascaded Transformer blocks. Each Transformer block contains two normalization layers, a multi-head self-attention layer, and a multilayer perceptron. The multilayer perceptron contains a linear layer with its dimension doubled, a SiLU activation function, and a linear layer with its dimension reduced by half. The specific operation expression is as follows:

[0020] y = x + MHSA(LayerNorm(x))

[0021] z = y + MLP(LayerNorm(y))

[0022] out1 = z + x

[0023] Where: x is the input of the Transformer block, y and z are intermediate variables, out1 is the output of the Transformer block, LayerNorm() represents layer normalization processing, MHSA() represents multi-head attention computation, and MLP() represents multilayer perceptron.

[0024] Furthermore, the linear fusion layer is used to compress the features output by the cascaded self-attention layer. It first swaps the last two dimensions of the features, uses a fully connected layer to reduce the last dimension to 1 / 3, applies LayerNorm for normalization, and then swaps the last two dimensions back to output the fused features.

[0025] Furthermore, the decoder includes a mask generation module, a patch matching and correction module, and an attitude detection head. The mask generation module first calculates the residual between the attitude heatmaps output by the backbone network and performs compression and activation processing to obtain an attitude mask. Then, it merges the attitude heatmaps of the corresponding channels and calculates them with the attitude mask to obtain a mask heatmap. Subsequently, the mask heatmap and the attitude heatmap are concatenated and sent to the patch matching and correction module for attitude fusion and refined attitude correction. Finally, the attitude detection head restores the resolution to the heatmap size and outputs it.

[0026] Furthermore, the mask generation module is implemented as follows: First, the attitude heatmap output by the backbone network is divided into a forward attitude heatmap, a current attitude heatmap, and a backward attitude heatmap along the batch dimension. The three attitude heatmaps are then concatenated along the channel dimension to obtain a merged heatmap. Next, the forward and backward attitude heatmaps are subtracted from the current attitude heatmap to obtain two heatmap residuals. These two heatmap residuals are then compressed (added and merged) along the channel dimension to obtain single-channel heatmap residuals. A variant sigmoid function operation is then performed on the two sets of single-channel heatmap residuals to obtain two masks. The two masks are then added to obtain a merged mask. The merged mask is expanded to the same shape as the merged heatmap and then multiplied point-by-point with the merged heatmap to obtain a mask heatmap. Finally, the mask heatmap and the merged heatmap are concatenated along the channel dimension and then sent to the patch matching and correction module. The specific operation expression is as follows:

[0027] diff_A = htm_curr - htm_back

[0028] diff_B = htm_curr - htm_forw

[0029]

[0030] diff_A_sing_dim,diff_B_sing_dim=compress(diff_A,diff_B)

[0031] mask1=varsigmoid(diff_A_sing_dim)

[0032] mask2=varsigmoid(diff_B_sing_dim)

[0033] htm_masked=dup(mask1+mask2)⊙htm_merg

[0034]

[0035] Where: htm_curr, htm_back, and htm_forw are the current attitude heatmap, backward attitude heatmap, and forward attitude heatmap, respectively; diff_A and diff_B are the backward and forward heatmap residuals, respectively; htm_merg indicates merging heatmaps; diff_A_sing_dim and diff_B_sing_dim are two sets of single-channel heatmap residuals; compress() indicates channel compression operation; varsigmoid() indicates a variant of the sigmoid function; mask1 and mask2 are two masks; dup() indicates copying the mask to extend multi-channel operations; ⊙ indicates dot product operation; and htm_masked indicates the masked heatmap. This indicates a concatenation operation, and out2 is the output of the mask generation module.

[0036] Furthermore, the patch matching and correction module consists of multiple cascaded computational blocks for correction. Each computational block contains three LayerNorm layers, a multi-head self-attention layer, a cross-attention layer, and a feedforward network layer. The patch matching and correction module first embeds the output of the mask generation module into patches before feeding them into the computational block. The specific computational expression of the computational block is as follows:

[0037] Y = X + MHSA(LayerNorm(X))

[0038] Z = Y + CA(LayerNorm(Y)W) q FW k FW v )

[0039] out3=FFN(LayerNorm(Z))+SiLU(X)

[0040] Where: X is the input of the computation block, Y and Z are intermediate variables, out3 is the output of the computation block, and W... q W k W v There are three learnable parameter matrices, F is the fusion feature output by the spatiotemporal global fusion module, MHSA() represents multi-head attention computation, LayerNorm() represents layer normalization processing, CA() represents cross-attention computation, FFN() represents feedforward network layer, and SiLU() represents SiLU activation function.

[0041] Furthermore, in step (3), the backbone network is first trained using the video dataset, and its network parameters are saved. Then, the spatiotemporal global fusion module and decoder are trained, the parameter file of the backbone network is imported and the backbone network is frozen, the loss between the heatmap output by the decoder and the corresponding label heatmap is calculated using the MSE function, the model parameters are iteratively updated, and the optimal model parameters are saved.

[0042] Furthermore, in step (4), the unlabeled video frame and its labeled support frame are input into the video human pose propagation model, and the pose heatmap output by the backbone network is replaced with the label heatmap of the support frame to realize pose label propagation. Finally, the model predicts and outputs the position coordinates of each key point of the person in the video frame.

[0043] Based on the above technical solution, the present invention has the following beneficial technical effects:

[0044] 1. This invention effectively captures and fuses spatiotemporal information in video by employing a self-attention mechanism and an end-to-end encoder-decoder architecture, solving the problem that traditional single-frame image methods cannot fully utilize temporal information, thereby improving the accuracy and robustness of pose estimation.

[0045] 2. The spatiotemporal global fusion module in the model of this invention uses a self-attention mechanism to perform global spatial fusion of features from multiple video frames, which not only enhances the model's adaptability to dynamic changes in the video, but also improves its ability to handle complex actions and occlusions.

[0046] 3. The patch matching correction module in the model of this invention merges multiple frames of attitude heatmaps and performs patch matching local spatial correction based on the features output by the encoder, thereby achieving fine adjustment of the initial attitude estimation and significantly improving the accuracy of attitude propagation. Especially when the annotation is sparse or of low quality, it effectively avoids the degradation problem of attitude estimation.

[0047] 4. This invention introduces a semi-automated video pose annotation method, which reduces the reliance on a large amount of manual annotation, lowers annotation costs, and improves the efficiency and feasibility of the annotation process, providing a new approach for the construction of large-scale video pose datasets. Attached Figure Description

[0048] Figure 1 This is a schematic diagram illustrating the overall implementation steps of the video human posture propagation method of the present invention.

[0049] Figure 2 This is a schematic diagram of the encoder structure in the model of the present invention. Detailed Implementation

[0050] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] like Figure 1 As shown, the video human pose propagation method based on the attention mechanism of the present invention includes the following steps:

[0052] Step 1: Collect and download the human-centered video pose estimation benchmark datasets PoseTrack17, PoseTrack18, and PoseTrack21, and construct a sparse labeled dataset. Specifically, in the training set video image data, select 5 video frames at equal intervals of 7 as labeled support frames. For each unlabeled support frame, match the two nearest labeled support frames forward and backward as support frames. For each labeled support frame, match two unlabeled support frames with a forward and backward distance of 4 as support frames. In the validation set video data, select 5 frames with pose annotations at equal intervals of 7 as labeled support frames. The remaining frames are set as unlabeled frames, and two labeled support frames are assigned to generate support heatmaps. Figure 1 The frames used as input data in the verification phase are marked as supporting that they are not used as input data alone.

[0053] Step 2: Construct a Vision Transformer-based backbone network to extract human pose features from images, such as... Figure 2 As shown, the network includes a patch embedding layer, a cascaded Transformer block layer, and an upsampling detection head. The input data of the network is each video frame and two corresponding matching support frames. First, the three frames are concatenated in the batch dimension and sent to the patch embedding layer for embedding compression and adding position information. Then, they are sent to the cascaded Transformer block layer to extract features and obtain feature tokens. After the feature tokens are transformed, they are sent to the upsampling detection head to obtain a pose heatmap. The feature tokens and pose heatmaps are then output.

[0054] Step 3: Construct a spatiotemporal global fusion module to fuse temporal features, such as... Figure 2As shown, this module includes a preprocessing layer, a cascaded self-attention layer, and a linear fusion layer. The preprocessing layer first separates the feature tokens and pose heatmaps output by the backbone network in the batch dimension, obtaining feature tokens and pose heatmaps at three time steps. A 1*1 convolutional block is used to reduce the three pose heatmaps to a single channel. Then, the channels are copied along the channel dimension to the same number as the corresponding feature tokens, and multiplied channel-wise with the corresponding feature tokens to achieve weight filtering. Finally, the three filtered feature tokens are concatenated in the channel dimension to obtain the concatenated features, which are then fed into the cascaded self-attention layer for global space computation. The cascaded self-attention layer is formed by stacking multiple Transformer blocks. Each Transformer block contains two normalization layers, a multi-head self-attention layer, and an MLP. The MLP contains a linear layer with its dimension doubled, a SiLU activation function, and a linear layer with its dimension halved. The specific operation is expressed as follows:

[0055] y = x + MHSA(LayerNorm(x))

[0056] z = y + MLP(LayerNorm(y))

[0057] out = z + x

[0058] Where: x represents the input of each Transformer block, y and z represent intermediate values, out represents the output of each Transformer block, LayerNorm() represents layer normalization, and MHSA() represents multi-head attention computation; finally, the features output from the cascaded attention layer are compressed through a linear fusion layer. First, the last two dimensions of the features are swapped, and a fully connected layer is used to reduce the last dimension to 1 / 3. After applying LayerNorm for normalization, the last two dimensions are swapped back to obtain the fused feature F.

[0059] Step 4: Construct a decoder for local spatial attitude correction. The decoder includes a mask generation module, a patch matching correction module, and an attitude detection head. The mask generation module divides the attitude heatmap output by the backbone network into a forward attitude heatmap, a current attitude heatmap, and a backward attitude heatmap along the batch dimension. The three attitude heatmaps are then concatenated along the channel dimension to obtain a merged heatmap. The forward and backward attitude heatmaps are then subtracted from the current attitude heatmap to obtain two heatmap residuals, which are then added along the channel dimension and merged into a single-channel heatmap. A variant sigmoid function is applied to the two heatmaps to calculate two masks, which are then added together to obtain a merged mask. The merged mask is expanded to the same shape as the merged heatmap and then multiplied point-by-point with the merged heatmap to obtain a mask heatmap. The mask heatmap and the merged heatmap are concatenated along the channel dimension and fed into the patch matching correction module. The specific formula is as follows:

[0060] diff_A = htm_curr - htm_back

[0061] diff_B = htm_curr - htm_forw

[0062]

[0063] diff_A_sing_dim,diff_B_sing_dim=compress(diff_A,diff_B)

[0064] mask1=varsigmoid(diff_A_sing_dim)

[0065] mask2=varsigmoid(diff_B_sing_dim)

[0066] htm_masked=dup(mask1+mask2)⊙htm_merg

[0067]

[0068] varsigmoid(x) = 1 / (2+e) -|x| )

[0069] Where: htm_curr, htm_back, and htm_forw represent the current attitude heatmap, the backward attitude heatmap, and the forward attitude heatmap, respectively; diff_A and diff_B represent the backward and forward heatmap residuals, respectively; compress() represents the channel compression operation; varsigmoid() represents a variant of the sigmoid function; dup() represents the mask copying and expansion to a multi-channel operation; ⊙ represents the dot product operation; htm_merg represents the merging of heatmaps; mask1 and mask2 represent masks in two directions; and htm_masked represents the masked heatmap. `out` indicates concatenation, and `out` indicates output.

[0070] The patch matching and correction module contains multiple cascaded computation blocks for correction. Each computation block contains three LayerNorm layers, one multi-head self-attention layer, one cross-attention layer, and one feedforward network layer. The patch matching and correction module first embeds patches into the output of the mask generation module before feeding them into the computation block. The specific calculation formula for each computation block is as follows:

[0071] y = x + MHSA(LayerNorm(x))

[0072] z=y+CA(LayerNorm(y)Wq,FWk,FWv)

[0073] out=FFN(LayerNorm(z))+SiLU(x)

[0074] Where: x, y, z, and out represent the input, intermediate value, and output of the computation block, respectively, and W q W k W v Let F represent three learnable matrices, F be the fusion feature output by the spatiotemporal global fusion module, CA() represent cross-attention calculation, FFN() represent the feedforward network layer, and SiLU() represent the activation function.

[0075] Step 5: Fine-tune the backbone network for 210 epochs using the processed training sets of PoseTrack17, PoseTrack18, and PoseTrack21. The pre-loaded model of the backbone network was pre-trained on ImageNet-1k, and the optimal weight parameters of the backbone network are saved. Then train the spatiotemporal global fusion module and decoder, import the backbone network parameter file and freeze the backbone network. Use the MSE function to calculate the loss between the heatmap output by the decoder and the heatmap generated by the label, and update the network parameters. The training time is 20 epochs, the learning rate is 0.0005, the optimizer used is adamW, and the parameter weight file of the last epoch is saved.

[0076] Step 6: Import the saved weight file into the model, and feed the unlabeled frames in the validation set, the assigned labeled support frames, and the generated support heatmap into the model. Replace the attitude heatmap output by the backbone network with the assigned support heatmap to achieve attitude label propagation, and evaluate the results by calculating the average accuracy (mAP) of all relevant nodes.

[0077] Next, we evaluate the accuracy of the model of this invention on the validation sets of PoseTrack17, PoseTrack18, and PoseTrack21, and quantitatively compare the prediction results of this invention with those of other methods. The specific results are shown in Table 1.

[0078] Table 1

[0079]

[0080] It can be seen that the STDPose model of this invention has higher average accuracy than existing models on all three datasets. First, on the PoseTrack17 validation set, the average accuracy of the model of this invention is 8.9% higher than the baseline method SimpleBaseline and 2.2% higher than the previous best PoseWarper. For the difficult-to-detect wrist and ankle joints, the model of this invention improves by 2.1% and 1.5%, respectively. Second, the model of this invention also achieves the best performance on the PoseTrack18 validation set. PoseTrack18 adds more videos with motion blur and multiple occlusions to PoseTrack17, which will affect the detection accuracy of the pose estimation model. As shown in Table 1, the model of this invention achieves performance improvement, with an average accuracy 3.0% higher than the previous best-performing DCPose. Notably, for ankle joints, the model of this invention achieves an accuracy of 85.8%, which is 5.5% higher than DCPose. This demonstrates that the robustness of the model of this invention is significantly improved compared to previous models. In addition, we also validated it on PoseTrack21, which has many more human instances with small screen size than PoseTrack18. In more challenging scenarios, the model of this invention once again achieved the best performance, especially the significant improvement in the performance of the knee joint and ankle joint. This proves the advantages of the model of this invention in detecting small human instances and its robustness to problems such as pose occlusion and camera shake.

[0081] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A video human pose propagation method based on an attention mechanism, comprising the following steps: (1) Obtain and process a video dataset centered on people; (2) Construct an end-to-end encoder-decoder model architecture based on a pure attention mechanism, including an encoder and a decoder. The encoder includes a backbone network and a spatiotemporal global fusion module, wherein: The backbone network is used to extract human pose features from video images and output feature tokens and pose heatmaps. The spatiotemporal global fusion module is used to fuse temporal features and perform global fusion of feature tokens in space. This module includes a preprocessing layer, a cascaded self-attention layer, and a linear fusion layer. The input is feature tokens and pose heatmaps. First, the feature tokens are weighted and filtered through the preprocessing layer. Then, global fusion is performed in space through the cascaded self-attention layer. Finally, the linear fusion layer is used to reduce the number of feature tokens by three times to achieve input-output correspondence and finally output fused features. The decoder is used for local spatial attitude correction. It concatenates the mask heatmap and attitude heatmap and performs attitude fusion and fine-grained attitude correction with the features output by the encoder to generate a key point heatmap. The decoder includes a mask generation module, a patch matching and correction module, and an attitude detection head. The mask generation module first calculates the residual between the attitude heatmaps output by the backbone network and performs compression activation processing to obtain an attitude mask. Then, it merges the attitude heatmaps of the corresponding channels and calculates with the attitude mask to obtain a mask heatmap. Subsequently, the mask heatmap and the attitude heatmap are concatenated and sent to the patch matching and correction module for attitude fusion and fine-grained attitude correction. Finally, the attitude detection head restores the resolution to the heatmap size and outputs it. The patch matching and correction module consists of multiple cascaded computational blocks for correction. Each computational block contains three LayerNorm layers, a multi-head self-attention layer, a cross-attention layer, and a feedforward network layer. The patch matching and correction module first embeds patches into the output of the mask generation module before feeding them into the computational block. The specific computational expression of the computational block is as follows: in: X As input to the computation block, and Z As an intermediate variable, For the output of the calculation block, There are three learnable parameter matrices. The fusion features output by the spatiotemporal global fusion module. This indicates multi-head attention computation. Presentation layer normalization processing, This indicates cross-attention calculation. Indicates a feedforward network layer. Indicates the SiLU activation function; (3) The above model architecture is trained using the video dataset to obtain the video human pose propagation model; (4) Input the unlabeled video frame and its labeled support frame into the video human pose propagation model, and the position coordinates of each key point of the person in the video frame can be predicted and output.

2. The video human posture propagation method according to claim 1, characterized in that: In step (1), for any video segment in the video dataset, n video frames are uniformly selected at intervals m as labeled frames. For each unlabeled frame, the nearest labeled frame in the forward and backward directions is matched as its supporting frame. For each labeled frame, two unlabeled frames with a forward and backward distance of m / 2 are matched as its supporting frames. m is a natural number greater than 0 and n is a natural number greater than 1.

3. The video human posture propagation method according to claim 1, characterized in that: The backbone network is based on Vision Transformer, which includes a patch embedding layer, a cascaded Transformer block layer, and an upsampling detection head. The input is a video frame and its two corresponding supporting frames. First, these three frames are concatenated in the batch dimension and sent to the patch embedding layer for embedding compression and adding position information. Then, they are sent to the cascaded Transformer block layer to extract features and obtain feature tokens. Finally, the feature tokens are transformed and sent to the upsampling detection head to obtain a pose heatmap.

4. The video human posture propagation method according to claim 1, characterized in that: The specific implementation of the preprocessing layer is as follows: First, the feature tokens and pose heatmaps are separated in the batch dimension to obtain feature tokens and pose heatmaps at three time points. Then, the three pose heatmaps are reduced to a single channel using 1×1 convolutional blocks and divided into blocks of the same size as the feature tokens. The blocks are then copied along the last dimension to the same size and shape as the corresponding feature tokens, and then multiplied point by point with the corresponding feature tokens to achieve weight filtering. Finally, the three filtered feature tokens are concatenated in the channel dimension as the output.

5. The video human posture propagation method according to claim 1, characterized in that: The cascaded self-attention layer is used to perform global spatial computation on the features output by the preprocessing layer. This layer is composed of multiple cascaded Transformer blocks. Each Transformer block contains two normalization layers, a multi-head self-attention layer, and a multilayer perceptron. The multilayer perceptron contains a linear layer with its dimensions doubled, a SiLU activation function, and a linear layer with its dimensions reduced by half. The specific operation expressions are as follows: in: x For the input of the Transformer block, and z As an intermediate variable, The output of the Transformer block. Presentation layer normalization processing, This indicates multi-head attention computation. This represents a multilayer perceptron.

6. The video human posture propagation method according to claim 1, characterized in that: The linear fusion layer is used to compress the features output by the cascaded self-attention layer. It first swaps the last two dimensions of the features, uses a fully connected layer to reduce the last dimension to 1 / 3, applies LayerNorm for normalization, and then swaps the last two dimensions back to output the fused features.

7. The video human posture propagation method according to claim 1, characterized in that: The mask generation module is implemented as follows: First, the attitude heatmap output by the backbone network is divided into a forward attitude heatmap, a current attitude heatmap, and a backward attitude heatmap along the batch dimension. The three attitude heatmaps are then concatenated along the channel dimension to obtain a merged heatmap. Next, the forward and backward attitude heatmaps are subtracted from the current attitude heatmap to obtain two heatmap residuals. These two heatmap residuals are then compressed along the channel dimension to obtain single-channel heatmap residuals. A variant sigmoid function operation is then performed on the two sets of single-channel heatmap residuals to obtain two masks. The two masks are then added to obtain a merged mask. The merged mask is expanded to the same shape as the merged heatmap and then multiplied point-by-point with the merged heatmap to obtain a mask heatmap. Finally, the mask heatmap and the merged heatmap are concatenated along the channel dimension and then sent to the patch matching and correction module. The specific operation expression is as follows: in: These are the current attitude heatmap, the backward attitude heatmap, and the forward attitude heatmap, respectively. and These are the backward heatmap residuals and the forward heatmap residuals, respectively. This indicates a merged heatmap. and For the residuals of two sets of single-channel heatmaps, This indicates a channel compression operation. This represents a variant of the sigmoid function. and Two masks, This indicates that mask copying is extended to multi-channel operation. This represents the dot product operation. Represents a mask heatmap. This indicates a splicing operation. This is the output of the mask generation module.

8. The video human posture propagation method according to claim 1, characterized in that: In step (3), the backbone network is first trained using the video dataset, and its network parameters are saved. Then, the spatiotemporal global fusion module and decoder are trained, the parameter file of the backbone network is imported and the backbone network is frozen. The loss between the heatmap output by the decoder and the corresponding label heatmap is calculated using the MSE function, the model parameters are iteratively updated, and the optimal model parameters are saved.

9. The video human posture propagation method according to claim 1, characterized in that: In step (4), the unlabeled video frame and its labeled support frame are input into the video human pose propagation model. The pose heatmap output by the backbone network is replaced with the label heatmap of the support frame to achieve pose label propagation. Finally, the model predicts and outputs the position coordinates of each key point of the person in the video frame.

Images