A video-based single-stage multi-person two-dimensional human posture estimation method and device

Abstract

The present invention discloses a single-stage multi-person two-dimensional human posture estimation method and device based on video, which belongs to the field of multi-person two-dimensional human posture estimation, and efficiently models the multi-person human posture in a video sequence by introducing the temporal information fusion technology. First, the present invention comprehensively models the multi-person human body information in a single video sequence, and constructs a cross-frame human temporal information fusion module, which can effectively capture and enhance the global information of the human body in the target frame, so that multiple human bodies can still be accurately identified and located in complex scenes. Secondly, the method deeply models the information of each joint point of the human body, and designs a cross-frame joint point temporal information fusion module, which is specifically used to strengthen the information transmission and correlation of the same joint points of the same person in different frames. This design can significantly improve the difficulty of joint point recognition in video sequences due to motion blur, occlusion, and inaccurate focus, thereby making the joint point prediction of the target frame more accurate and stable.

Description

Technical Field

[0001] The present invention belongs to the field of multi-person two-dimensional human body posture estimation, and in particular relates to a video-based single-stage multi-person two-dimensional human body posture estimation method and device. Background Art

[0002] Human pose estimation is a basic and challenging task in the field of computer vision. Human pose estimation is crucial for describing human poses and human behaviors. Many computer vision tasks are based on human pose estimation, including behavior recognition, human-computer interaction, pedestrian tracking, pedestrian re-identification, etc.

[0003] Human pose estimation can be mainly divided into two subtasks: single-person pose estimation and multi-person pose estimation. Single-person pose estimation is the basis of the task. For a picture containing a person, we need to find all the joints of this person. In multi-person pose estimation, for a picture containing multiple people, we need to find the joints of all the people in this picture. For this problem, there are generally two methods: top-down and bottom-up. At present, the mainstream methods for solving multi-person pose estimation are mostly two-stage methods. In the top-down method, it is necessary to first use a human detector to detect all the people in the picture, and then perform human key point detection for each person detected. Therefore, the performance of pose estimation will be heavily dependent on the performance of the human detector, especially in complex scenes. Secondly, due to the use of two isolated detectors, the amount of calculation is large, and it increases linearly with the number of segmented human instances. For the bottom-up method, complex operations such as heuristics and manual post-processing are usually used in the joint point association stage. At present, there are also many single-stage methods to achieve multi-person pose estimation. For example, SPM proposes a posture representation of structured painting, which unifies the representation of human instances and joint point positions. FCPose and InsPose use dynamic instance-aware fully convolutional networks. These methods usually require cropping the region of interest and post-processing to group key points, achieving a good balance between performance and speed. However, they rely on heat maps and score maps, which increases the storage requirements during the calculation process.

[0004] In the task of multi-person pose estimation based on video, there are generally two ideas to solve the problem caused by the degradation of continuous frames of video. The first is to use recurrent neural networks (RNNs), such as LSTM, GRU. LSTM Pose Machines uses convolutional LSTM to capture temporal and spatial cues and directly predict the key point heat map sequence of the video. This RNN-based method is more effective when human objects are spatially sparse. However, performance is severely hampered by occlusion, which often occurs in multi-person pose estimation, and even self-occlusion in the case of a single person. The second method is to find the correlation between pixels in adjacent frames through optical flow methods, so as to find the correspondence between the previous frame and the current frame. The FlowingConvNets method calculates dense optical flow between every two frames and uses other flow-based representations to align the predictions. This method is effective when the optical flow can be calculated accurately. However, in cases involving motion blur or defocus, poor image quality leads to inaccurate optical flow, resulting in poor performance.

[0005] DCPose proposes a new method for estimating human pose from video images. The key idea of ​​DCPose is to merge continuous frames from bidirectional temporal directions to improve pose estimation in videos. The pose estimation of the target frame is optimized by using the previous and next frames of the target frame to cope with complex scenarios such as occlusion, defocus, and fast motion of the current frame. The DCPose algorithm is mainly divided into three modules: pose temporal merging, pose residual fusion, and pose correction network. First, a rectangular frame is used to frame the human body for pose estimation from the target frame, the previous frame, and the next frame, and the heat maps of various parts of the human body such as the head and left and right feet are estimated separately through HRNet. In order to prevent the parts from affecting each other, group convolution is performed after grouping them in the pose temporal merging module. The pose residual fusion is to obtain the motion residual by subtracting the target frame from the next frame and the previous frame from the target frame, so as to obtain the motion context information. Then, in the pose correction network, DCPose processes the heat map information and the motion context information together to give a pose prediction.

[0006] However, DCPose adopts a top-down approach, which requires first segmenting different people in continuous frames of the video. At the same time, the heat map used in the supervision method, although the heat map representation is very effective, has several disadvantages, such as quantization errors and high computing and storage requirements associated with maintaining high-resolution heat maps, which increases the cost of training.

[0007] FAMI-Pose believes that existing methods often directly aggregate misaligned contexts from adjacent frames, resulting in poor performance in challenging scenarios such as fast motion and pose occlusion. These spatially misaligned features may reduce the performance of the model. In addition, existing methods only use mean square error (MSE) loss to supervise the learning of heat maps, lacking effective constraints to ensure information gain from adjacent frames and supervision at the intermediate feature level. Therefore, a method for solving the problem of feature alignment in multi-frame human pose estimation is proposed. This method enhances the key frame features by supervising at the intermediate feature level and using the information in adjacent frames to improve the accuracy of pose estimation. A new information theory loss function, mutual information, is also introduced to constrain the information extracted from adjacent frames to improve the effect of feature alignment.

[0008] However, it also has the same problems as DCPose. The supervised method still uses heat maps, which cannot avoid the quantization errors and the high computing and storage requirements associated with maintaining high-resolution heat maps. Summary of the invention

[0009] The present invention aims to solve some problems caused by existing two-dimensional human pose estimation models in multi-person scenarios, such as additional computational burden and high computational and storage requirements caused by training using heat map supervision. To this end, a video-based single-stage multi-person two-dimensional human pose estimation algorithm is proposed. The algorithm can significantly reduce the high computational and storage requirements brought by existing algorithms. At the same time, by modeling different fine-grained human information and fusing temporal information, it can effectively deal with the problems of blurring, occlusion and loss of focus of target frames in video sequences, thereby further improving the accuracy of human pose estimation.

[0010] The object of the present invention is to achieve the following technical solution: a single-stage multi-person two-dimensional human posture estimation method based on video, the method comprising the following steps:

[0011] Step 1: Build a video dataset containing human body clips and divide the video frame samples into target frames and auxiliary frames;

[0012] Step 2: construct a multi-person two-dimensional human posture estimation network, which extracts human body information from the target frame and the auxiliary frame respectively, fuses spatial information and temporal information, and outputs human body posture joint coordinates;

[0013] Step 3: Construct a total loss function through classification loss and joint point regression loss to constrain the multi-person 2D human pose estimation network, and train the final multi-person 2D human pose estimation network;

[0014] Step 4: Perform human pose estimation on the input video clip based on the trained multi-person 2D human pose estimation model.

[0015] Furthermore, the video dataset containing human body clips is constructed based on PoseTrack17, and includes video data of different human body postures and different backgrounds.

[0016] Furthermore, the human posture estimation network includes a backbone network composed of ResNet-50, a downsampling layer, a Transformer encoder, a posture decoder, a posture tracker and a joint decoder;

[0017] The backbone network includes four residual blocks, which input the target frame and auxiliary frame in the video frame sample, output four feature maps of different scales for each frame, downsample the feature maps generated by the last three residual blocks through the convolution layer to obtain the downsampled feature map, and unify the number of channels of the feature maps generated by the last three residual blocks and the downsampled feature map, and the unified feature map is input into the Transformer encoder;

[0018] The Transformer encoder has a total of 6 Transformer layers, each of which consists of a multi-scale deformable attention network and a feed-forward neural network, which performs scale feature fusion on the 4 input feature maps and outputs 4 feature maps that fuse multi-scale feature information;

[0019] The posture decoder only inputs the multi-scale feature map corresponding to the target frame in the video frame sample to extract the human posture information, and includes three layers, each of which includes a self-attention network, a cross-attention network, a feedforward neural network and a multi-layer perceptron;

[0020] The posture tracker extracts human body information corresponding to the auxiliary frame, including auxiliary frame human body posture information extraction and inter-frame human body posture time series information fusion, the auxiliary frame human body posture information extraction module includes three layers, each of which includes a self-attention network, a cross-attention network, a feedforward neural network and a multi-layer perceptron, and the inter-frame human body posture time series information fusion includes three layers, each of which includes two self-attention networks, a feedforward neural network and a multi-layer perceptron;

[0021] The joint decoder further optimizes the output of the posture tracker and includes three layers, each of which includes two self-attention networks, a cross-attention network, a feedforward neural network, and a multi-layer perceptron.

[0022] Furthermore, in the posture decoder, the specific processing flow includes:

[0023] First, a linear layer and a feedforward neural network are used to process the four multi-scale features output by the Transformer encoder, and the confidence of each feature and the initial posture joint point coordinates corresponding to each feature are obtained. The corresponding k initial posture joint point coordinates are selected according to the confidence and the top-k algorithm, and the query keys of the k postures are randomly initialized; in each layer of the posture decoder, the self-attention mechanism is used to capture the correlation between different human posture query keys, and the cross-attention mechanism is used to interact with the multi-scale feature map to extract the key information related to the human posture. Then, the extracted key information is processed by the feedforward neural network, and then the multi-layer perceptron is used to regress the offset of the processed posture features relative to the initial posture joint point coordinates, and the initial posture joint is added to the relative offset to obtain the posture joint point coordinates output by this layer;

[0024] gives (i-1) th Normalized human joint point coordinates P predicted by the Transformer-layer i-1 , i-th th The Transformer-layer refines the coordinates of the human joint points to:

[0025]

[0026] Where ΔP i is the i th The relative coordinate offset of the layer prediction, and Represents sigmoid and anti-sigmoid operations;

[0027] After three layers of layer-by-layer processing, k posture features and the corresponding coarse-grained posture joint point coordinates are finally output. The characteristics, joint point coordinates and human posture query keys of the target frame corresponding to the real human joint point coordinates manually annotated in the target frame are selected through the posture joint point coordinates.

[0028] Furthermore, the processing of extracting the auxiliary frame human body posture information in the posture tracker includes:

[0029] The human joint coordinates output by the posture decoder are matched with the human posture query key and copied as the coarse-grained human joints and human posture query key of the auxiliary frame, and the coarse-grained human joint coordinates are used as the initial reference point of the auxiliary frame human posture query key, and the human posture token corresponding to the auxiliary frame is obtained by processing through the self-attention network and the cross-attention network, and the human posture token is input into the multi-layer perceptron to predict the relative offset relative to the initial reference point; the initial reference point and the relative offset are added to obtain the optimized human joint coordinates;

[0030] The optimized human body joint point coordinates are used as the initial reference points corresponding to the human body posture tokens in the next layer, and after traversing all layers, the posture feature tokens and human body posture joint point coordinates corresponding to the auxiliary frame are output.

[0031] Furthermore, the processing of the fusion of human body posture temporal information between frames in the posture tracker includes: using two self-attention networks and a feedforward neural network to fuse the human body posture spatial information and temporal information in the auxiliary frame and the target frame, obtaining the optimized human body posture token and inputting it into the multi-layer perceptron, using the multi-layer perceptron to predict the relative offset of the posture joint point coordinates of each frame before and after the fusion, adding the relative offset to the posture joint point coordinates to obtain the optimized posture joint point coordinates, and using the human body posture token and the posture joint point coordinates as the input of the next layer.

[0032] Furthermore, in the joint decoder, the joint query key is first randomly initialized and copied to obtain query keys corresponding to the auxiliary frame and the target frame respectively, the posture joint point output by the posture tracker is used as the initial reference point of the joint query key of the corresponding frame, two self-attention networks and a feedforward neural network are used to fuse the spatial information and temporal information of different joint query keys corresponding to each posture in each frame, the obtained joint query key and the multi-scale feature token are input into the cross-attention network, the joint point token is output, and then the obtained joint point token is optimized using the feedforward neural network;

[0033] A multi-layer perceptron is used to predict the relative offset of the pose joint coordinates, and the relative offset is added to the pose joint coordinates to obtain the optimized pose joint coordinates. The optimized joint tokens and pose joint coordinates are used as the input of the next layer.

[0034] Furthermore, the total loss function L loss for:

[0035] L loss = L cls + L oks + L rle

[0036] Among them, L cls is the classification loss of the pose features output by the pose tracker, L oks is the joint point coordinate regression loss output by the pose tracker, L rle The regression loss of the joint point coordinates output by the joint decoder;

[0037] The classification loss of the posture features output by the posture tracker and the regression loss of the joint point coordinates output by the posture tracker are calculated as follows: the confidence output by the posture decoder and the human joint coordinates are matched by bipartite graph using the Hungarian algorithm to calculate:

[0038]

[0039] in the formula represents the confidence of the prediction, is the true category of GT, P is the predicted joint point coordinate, P* is the true joint point coordinate of GT, i represents the i-th joint point, and vi represents whether the joint point is visible;

[0040] The formula for bipartite graph matching using the Hungarian algorithm is expressed as: ;

[0041] N=k in the formula means k Pose-tokens, y i Represents the coordinates of the i-th joint point in the prediction result, represents the coordinates of the i-th joint point in GT, Represents the loss matrix of GT and predicted value, Indicates the joint point, Indicates the minimum loss matching result between GT and prediction;

[0042] The regression loss of the joint point coordinates output by the joint decoder is expressed as:

[0043]

[0044] in is the standard normal distribution, is the predicted result distribution, f is a constant, To predict the translation coefficient, is the prediction scaling factor.

[0045] On the other hand, the present invention provides a video-based single-stage multi-person two-dimensional human posture estimation device, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, a video-based single-stage multi-person two-dimensional human posture estimation method as described is implemented.

[0046] On the other hand, the present invention specification provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the video-based single-stage multi-person two-dimensional human posture estimation method is implemented.

[0047] Beneficial effects of the present invention: The method introduces advanced temporal information fusion technology to efficiently model the postures of multiple people in video sequences, aiming to improve the accuracy and robustness of posture estimation. First, the present invention comprehensively models the information of multiple people in a single video sequence and constructs a cross-frame human temporal information fusion module, which can effectively capture and enhance the global information of the human body in the target frame, so that multiple people can still be accurately identified and located in complex scenes.

[0048] Secondly, the method deeply models the information of each joint point of the human body and designs a cross-frame joint point temporal information fusion module, which is specifically used to strengthen the information transmission and correlation of the same joint points of the same person in different frames. This design can significantly improve the difficulty of joint point recognition in video sequences caused by motion blur, occlusion, and inaccurate focus, thereby making the joint point prediction of the target frame more accurate and stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flow chart of a single-stage multi-person two-dimensional human posture estimation method based on video stream provided by an embodiment of the present invention;

[0050] Figure 2 A network structure diagram of a Pose-Tracking module provided in an embodiment of the present invention;

[0051] Figure 3 A network structure diagram of each Transformer-layer of the Joint-Decoder module provided in an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of a single-stage multi-person two-dimensional human posture estimation device based on video stream provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0055] like Figure 1 As shown, the present invention proposes a single-stage multi-person two-dimensional human posture estimation method based on video stream, which specifically includes the following steps:

[0056] Step 1: Construct the dataset required for training a video-based single-stage multi-person 2D human pose estimation model;

[0057] Constructing the data set required for training the two-dimensional human posture estimation model in step 1 includes the following steps:

[0058] This dataset is built on the existing public dataset PoseTrack17, which is a standard dataset widely used in human pose estimation research. The dataset consists of three parts: train, val, and test, which are used for model training, verification, and testing. Among them, the train (training set) and val (verification set) datasets contain video clips of multiple human poses, covering various scenes in daily life and complex background situations. These video clips not only provide rich visual information, but also contain high-quality annotated data.

[0059] In the train and val datasets, the annotation data of each video clip is carried out every 4 frames, and the annotation content includes the coordinate information of the key points of the human posture. Specifically, the annotation contains the precise coordinates of 15 joints of the human body (such as head, shoulders, elbows, knees, etc.). These coordinate data provide researchers with a reliable reference for training and evaluating human posture estimation algorithms. The test part is mainly used to evaluate the generalization ability of the model. Although it does not contain annotated data, the model prediction results can be compared with the official standard answers during the evaluation process to measure the performance of the model. Through the combination of these annotated data and video clips, the PoseTrack17 dataset provides important support for research and application in the field of human posture recognition.

[0060] Step 2: Construct a single-stage multi-person 2D human pose estimation network based on video;

[0061] The video-based single-stage multi-person two-dimensional human pose estimation network in step 2 includes:

[0062] The network as a whole consists of six major modules: backbone network Backbone, downsampling layer Neck, Transformer encoder Transformer-Encoder, pose decoder Pose-Decoder, pose tracker Pose-Tracking, and J-joint decoder oint-Decoder.

[0063] The Backbone module is mainly used to extract image features from input video frame samples (each video frame sample consists of three frames of images, namely the previous frame, the target frame, and the next frame), and the network structure used is ResNet-50. ResNet-50 is a 50-layer deep network, mainly composed of 4 residual blocks. After inputting the video frame sample, the network module will output 4 feature maps of different scales, namely F1, F2, F3 and F4. In this method, only the feature maps generated by the last three residual blocks, namely F2, F3 and F4, are used for subsequent feature processing and analysis.

[0064] The Neck module is mainly responsible for processing the feature maps F2, F3 and F4 generated by the Backbone module. First, the feature map of the F4 scale is further downsampled through a convolutional layer to generate a downsampled feature map F5. Then, in order to facilitate subsequent feature fusion and processing, another convolutional layer is used to uniformly adjust the number of channels of F2, F3, F4 and F5 to 256. Through this process, the Neck module can effectively integrate feature information from different scales.

[0065] The Transformer-Encoder module consists of 6 layers of Transformer-layers, each of which consists of a multi-scale deformable attention network (Multi-Scale DeformableAttention) and a feed-forward neural network (FFN), which is responsible for the scale feature fusion processing of the feature maps of 4 different scales generated by the Neck module. This module effectively fuses these 4 feature maps of different scales through the multi-scale deformable attention mechanism (Multi-Scale DeformableAttention Mechanism). The multi-scale deformable attention mechanism can flexibly focus on the key information in the features of each scale, thereby realizing the deep integration of features of different scales. Finally, the module outputs 4 feature maps that fuse multi-scale feature information. These feature maps retain the advantages of each scale and provide richer and more comprehensive feature expressions.

[0066] The main function of the Pose-Decoder module is to extract the human posture information of the target frame in the input video frame sample from the four multi-scale feature maps output by the Transformer-Encoder module. The input of this module only uses the four multi-scale feature maps F2 corresponding to the target frame in the video frame sample. t 、F3 t 、F4 t and F5 tThe module consists of three Transformer-layers, and each sub-module contains four core components: Self-Attention Network, Cross-Attention Network, Feedforward Neural Network (FFN), and Multi-Layer Perceptron (MLP).

[0067] In the workflow of Pose-Decoder, a linear layer is first used to process the 4 multi-scale feature tokens output by the Transformer-Encoder module. t , get each token t A confidence score (the probability of containing a human body) and an FFN is used to output each token t The corresponding initial Pose joint coordinates, and then use the top-k algorithm to select the corresponding k initial Pose joint coordinates Pose-joints according to the confidence score 0 (k=300), and then randomly initialize k human pose query keys Pose-query t , select the k initial Pose joint coordinates Pose-joints 0 With randomly initialized k Pose-queries t One-to-one correspondence, Pose-query t It will be used as the query key to extract the pose information. t First input to the first submodule for processing. In the first submodule, the self-attention mechanism is first used to capture different Pose-queries t The cross attention mechanism extracts key information related to human posture by interacting with multi-scale feature tokens. Subsequently, the feedforward neural network further processes this information to enhance the ability of feature expression. Then, a multi-layer perceptron is used to process the Pose-token output by the submodule. i Calculate the relative offset of the initial Pose joint coordinates relative to the token i , and then use the initial Pose-joints i-1 With relative offset i Add together to get refined Pose-joints i (i represents the result in the i-th Transformer-layer, starting from 1).

[0068] Standardly speaking, given (i-1) thNormalized human joint point coordinates P predicted by the Transformer-layer i-1 , i-th th The Transformer-layer refines the coordinates of the human joint points to:

[0069]

[0070] Where ΔP i is the i th The relative coordinate offset of the layer prediction, and Represents sigmoid and anti-sigmoid operations.

[0071] After layer-by-layer processing by the three submodules, Pose-Decoder finally outputs k human pose tokens Pose-Token and the corresponding coarse-grained Pose joint coordinates Pose-joints. Through Pose-joints, the Pose-token of the target frame corresponding to GT (the real human joint coordinates manually annotated in the target frame) is selected. t i , Pose-joints t i And Pose-query t i (The subscript t represents the target frame, and the superscript i is 3 in this case).

[0072] The Pose-Tracking module is mainly composed of two sub-modules, namely an auxiliary frame human pose information extraction module (AuxFramePoseExtract) and an inter-frame human pose temporal information fusion module (PoseTemporalInformationFusion).

[0073] like Figure 2 As shown in the auxiliary frame human posture information extraction module, the AuxFramePoseExtract module consists of 3 layers of Transformer-layer, each of which contains a self-attention network (Self-Attention), a cross-attention network (Cross-Attention), a feedforward neural network (FFN) and a multi-layer perceptron (MLP). The human posture query keys of the previous frame and the next frame are input, and the human posture tokens of the previous frame and the next frame are obtained after the self-attention network and the cross-attention network, which are input into the feedforward neural network and the multi-layer perceptron for further optimization.

[0074] like Figure 2As shown in the inter-frame human posture temporal information fusion module, the PoseTemporalInformationFusion module consists of three layers of Transformer-layer, where each layer of Transformer-layer contains two self-attention networks (Self-Attention), a feedforward neural network (FFN) and a multi-layer perceptron. The first layer of Transformer-layer inputs the target frame human posture token and the previous and next frames of the human posture token output by the AuxFramePoseExtract module. After the two self-attention networks, the human posture tokens of three frames are obtained and optimized by the feedforward neural network and the multi-layer perceptron, and then input into the next layer for iteration.

[0075] Specifically, in the Pose-Tracking workflow, the coarse-grained human joint coordinates obtained by the Pose-Decoder that match the GT are first matched one-to-one with the Pose-query obtained in the same way and copied twice as the coarse-grained human joint coordinates of the auxiliary frame (the previous and next frames) t-1 0 , Pose-joints t+1 0 and Pose-query t-1 0 , Pose-query t+1 0 (The subscript t-1 indicates the previous frame, and t+1 indicates the next frame). Then it is input into the AuxFramePoseExtract module. In the AuxFramePoseExtract module, it passes through the Self-Attention and Cross-Attention in each Transformer-layer. In the Cross-Attention processing flow, the coarse-grained human joint coordinates need to be used as Pose-query. t-1 i-1 , Pose-query t+1 i-1 The initial reference point is processed to obtain the previous frame human posture token Pose-token corresponding to the auxiliary frame t-1 i , the next frame human posture token Pose-token t+1 i After each layer, FFN is used to further optimize the Pose-token t-1 i , Pose-token t+1 i, then use a multi-layer perceptron to predict the relative offset relative to the initial reference point, and then add the initial reference point and the relative offset to get the optimized human joint coordinates Pose-joints t-1 i , Pose-joints t+1 i , and also serves as the Pose-token in the Cross-Attention in the next Transformer-layer t-1 , Pose-token t+1 The AuxFramePoseExtract module finally outputs the corresponding Pose-token in the auxiliary frame after three optimizations. t-1 i , Pose-token t+1 i And Pose-joints t-1 i , Pose-joints t+1 i .

[0076] Finally, Pose-token t-1 0 , Pose-token t 0 ， Pose-token t+1 0 In the PoseTemporalInformationFusion module, the first layer of Transformer-layer uses Self-Attention to fuse the spatial information between the Pose-tokens of the three frames, and then uses the second Self-Attention to fuse the temporal information between the corresponding human body posture tokens Pose-tokens between the three frames. Then, FFN is used to further optimize the Pose-token. t-1 i , Pose-token t i , Pose-token t+1 i , and then use the multi-layer perceptron to process the optimized Pose-token t-1 i , Pose-token t i , Pose-token t+1 i And regress relative to Pose-jointst-1 i , Pose-joints t i , Pose-joints t+1 i The relative offset t-1 i , offset t i , offset t+1 i Finally, the two are added together to get the optimized Pose-joints t-1 i , Pose-joints t i , Pose-joints t+1 i , output the optimized human joint point coordinates Pose-joints t-1 i , Pose-joints t i , Pose-joints t+1 i And Pose-token t-1 i , Pose-token t i , Pose-token t+1 i This is repeated three times as input to the next layer.

[0077] The Joint-Decoder module is mainly responsible for further optimizing the human joint point coordinates Pose-joints obtained by the Pose-Tracking module t-1 i , Pose-joints t i , Pose-joints t+1 i The module consists of three Transformer-layers, such as Figure 3 As shown in the figure, each Transformer-layer contains two self-attention networks (Self-Attention), a cross-attention network (Cross-Attention), a feedforward neural network (FFN) and a multi-layer perceptron (MLP).

[0078] like Figure 3As shown in the workflow of the Joint-Decoder module, firstly, the query key Joint-query of m human joint points is randomly initialized, then the m Joint-queries are copied 3xNum_GT copies to obtain the query key Joint-query of the previous frame of joint points corresponding to the three frames of images in the video frame sample. t-1 , target frame joint query key Joint-query t , the next frame joint query key Joint-query t+1 And the human joints Pose-joints obtained by the Pose-Tracking module t-1 i , Pose-joints t i , Pose-joints t+1 i As the initial reference point Pose-joints of the corresponding frame joint query key Joint-query t-1 0 , Pose-joints t 0 , Pose-joints t+1 0 , and match it to Joint-query one by one. Then, the joint query key of the previous frame is t-1 , target frame joint query key Joint-query t , the next frame joint query key Joint-query t+1 Pose-joints with initial reference point t-1 0 , Pose-joints t 0 , Pose-joints t+1 0 Input to the first layer of Transformer-layer, first perform the first Self-Attention network to fuse the spatial information between the different joint query keys corresponding to each pose in each frame of the three frames, then perform the second Self-Attention to fuse the corresponding Joint-query temporal information between the corresponding poses in the three frames, then input the obtained joint query key Joint-query and the multi-scale feature token Token into the Cross-Attention network, and output the joint token Joint-token t-1 i , Joint-tokent i , Joint-token t+1 i , and then use the feedforward neural network FFN to optimize the Joint-token t-1 i , Joint-token t i , Joint-token t+1 i , and then use the multi-layer perceptron to predict the relative pose-joints t-1 i , Pose-joints t i , Pose-joints t+1 i The relative offset t-1 i , offset t i , offset t+1 i Finally, the two are added together to get the optimized reference point coordinates Pose-joints t-1 i , Pose-joints t i , Pose-joints t+1 i , output the optimized human body posture joint coordinates Pose-joints t-1 i , Pose-joints t i , Pose-joints t+1 i Joint-token as a reference point and joint point query key t-1 i , Joint-token t i , Joint-token t+1 i As the input of the next layer, repeat the process three times to finally get the Pose-joints coordinates of multiple human body poses. t-1 i , Pose-joints t i , Pose-joints t+1 i As the estimated prediction results.

[0079] Standardly speaking, given (i-1)th Normalized human joint point coordinates J predicted by Transformer-layer i-1 , i-th th The Transformer-layer refines the coordinates of the human joint points to:

[0080]

[0081] Where ΔJ i is the i th The relative coordinate offset of the layer prediction, and Represents sigmoid and anti-sigmoid operations.

[0082] Step 3: Construct a total loss function through classification loss and joint point regression loss to constrain the multi-person 2D human pose estimation network, and train the final multi-person 2D human pose estimation model;

[0083] Furthermore, in step 3, the total loss function is constructed by classification loss and joint point regression loss to constrain the multi-person two-dimensional human posture estimation network, and the training to obtain the final multi-person two-dimensional human posture estimation model includes the following steps:

[0084] The three layers of Transformer-layer in Pose-Decoder will output the confidence of k Pose-tokens and the coordinates of human joints. This method uses the Hungarian algorithm for bipartite graph matching, that is, one-to-one correspondence between the elements of the predicted set and the true set to minimize the matching loss. The formula is:

[0085]

[0086] N=k in the formula means k Pose-tokens, y i Represents the coordinates of the i-th joint point in the prediction result, represents the coordinates of the i-th joint point in GT, Represents the loss matrix of GT and predicted value, Indicates the joint point, Indicates the minimum loss matching result between GT and prediction.

[0087] Calculate the classification loss L of k Pose-tokens after matching through the Hungarian algorithm cls And the joint point coordinate regression loss L oks , the formula is:

[0088]

[0089]

[0090] in the formula represents the confidence of the prediction, c i is the true category of GT (0 or 1), P is the predicted joint point coordinates, P* is the true joint point coordinates of GT, K is the number of joint points, i represents the i-th joint point, s represents the manual annotation error of different joint points, and v i Indicates whether the joint is visible (0 is invisible, 1 is visible).

[0091] The three Transformer-layers in the Joint-Decoder will output m Joint-token human joint coordinates and and Since m and GT are one-to-one corresponding, there is no need to calculate the classification loss, only the regression loss L of the joint point coordinates needs to be calculated rle , the formula is:

[0092]

[0093] in is the standard normal distribution, is the predicted result distribution, f is a constant, To predict the translation coefficient, is the prediction scaling factor.

[0094] The overall loss formula is expressed as: L loss = L cls + L oks + L rle .

[0095] Step 4: Estimating the human pose of the input video clip based on the trained multi-person 2D human pose estimation model;

[0096] Corresponding to the aforementioned embodiment of a video-based single-stage multi-person two-dimensional human posture estimation method, the present invention also provides an embodiment of a video-based single-stage multi-person two-dimensional human posture estimation device.

[0097] See also Figure 4 A video-based single-stage multi-person two-dimensional human posture estimation device provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a video-based single-stage multi-person two-dimensional human posture estimation method in the above embodiment.

[0098] The embodiment of a video-based single-stage multi-person two-dimensional human posture estimation device provided by the present invention can be applied to any device with data processing capabilities, and the any device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 4 As shown in FIG. 1 , a hardware structure diagram of a video-based single-stage multi-person two-dimensional human posture estimation device provided by the present invention is provided for any device with data processing capability, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0099] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0100] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0101] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a single-stage multi-person two-dimensional human posture estimation method based on video in the above embodiment is implemented.

[0102] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0103] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the video-based single-stage multi-person two-dimensional human posture estimation method is implemented.

[0104] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0105] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A single-stage multi-person two-dimensional human pose estimation method based on video, characterized in that: The method comprises the following steps: Step 1: Build a video dataset containing human body clips and divide the video frame samples into target frames and auxiliary frames; Step 2: Construct a multi-person 2D human pose estimation network, which extracts human information from the target frame and the auxiliary frame respectively, fuses spatial information and temporal information, and outputs human pose joint coordinates; the human pose estimation network includes a backbone network composed of ResNet-50, a downsampling layer, a Transformer encoder, a pose decoder, a pose tracker, and a joint decoder; The backbone network includes four residual blocks, which input the target frame and auxiliary frame in the video frame sample, output four feature maps of different scales for each frame, downsample the feature maps generated by the last three residual blocks through the convolution layer to obtain the downsampled feature map, and unify the number of channels of the feature maps generated by the last three residual blocks and the downsampled feature map, and the unified feature map is input into the Transformer encoder; The Transformer encoder has a total of 6 Transformer layers, each of which consists of a multi-scale deformable attention network and a feed-forward neural network, which performs scale feature fusion on the 4 input feature maps and outputs 4 feature maps that fuse multi-scale feature information; The posture decoder only inputs the multi-scale feature map corresponding to the target frame in the video frame sample to extract the human posture information, and includes three layers, each of which includes a self-attention network, a cross-attention network, a feedforward neural network and a multi-layer perceptron; The posture tracker extracts human body information corresponding to the auxiliary frame, including auxiliary frame human body posture information extraction and inter-frame human body posture time series information fusion, the auxiliary frame human body posture information extraction module includes three layers, each of which includes a self-attention network, a cross-attention network, a feedforward neural network and a multi-layer perceptron, and the inter-frame human body posture time series information fusion includes three layers, each of which includes two self-attention networks, a feedforward neural network and a multi-layer perceptron; The joint decoder further optimizes the output of the posture tracker and includes three layers, each of which includes two self-attention networks, a cross-attention network, a feedforward neural network, and a multi-layer perceptron; Step 3: Construct a total loss function through classification loss and joint point regression loss to constrain the multi-person 2D human pose estimation network, and train the final multi-person 2D human pose estimation network; Step 4: Perform human pose estimation on the input video clip based on the trained multi-person 2D human pose estimation model.

2. The video-based single-stage multi-person two-dimensional human posture estimation method according to claim 1, characterized in that: The video dataset containing human body clips is constructed based on PoseTrack17, and includes video data of different human body postures and different backgrounds.

3. The video-based single-stage multi-person two-dimensional human posture estimation method according to claim 1, characterized in that: In the posture decoder, the specific processing flow includes: First, a linear layer and a feedforward neural network are used to process the four multi-scale features output by the Transformer encoder, and the confidence of each feature and the initial posture joint point coordinates corresponding to each feature are obtained. The corresponding k initial posture joint point coordinates are selected according to the confidence and the top-k algorithm, and the query keys of the k postures are randomly initialized; in each layer of the posture decoder, the self-attention mechanism is used to capture the correlation between different human posture query keys, and the cross-attention mechanism is used to interact with the multi-scale feature map to extract the key information related to the human posture. Then, the extracted key information is processed by the feedforward neural network, and then the multi-layer perceptron is used to regress the offset of the processed posture features relative to the initial posture joint point coordinates, and the initial posture joint is added to the relative offset to obtain the posture joint point coordinates output by this layer; gives (i-1) th Normalized human joint point coordinates P predicted by the Transformer-layer i-1 , i-th th The Transformer-layer refines the coordinates of the human joint points to: P i =σ(σ -1 (P i-1 )+ΔP i ) Where ΔP i is the i th The relative coordinate offsets of layer predictions, σ and σ -1 Represents sigmoid and anti-sigmoid operations; After three layers of layer-by-layer processing, k posture features and the corresponding coarse-grained posture joint point coordinates are finally output. The characteristics, joint point coordinates and human posture query keys of the target frame corresponding to the real human joint point coordinates manually annotated in the target frame are selected through the posture joint point coordinates.

4. The video-based single-stage multi-person two-dimensional human posture estimation method according to claim 3, characterized in that: The processing of extracting the auxiliary frame human body posture information in the posture tracker includes: The human joint coordinates output by the posture decoder are matched with the human posture query key and copied as the coarse-grained human joints and human posture query key of the auxiliary frame, and the coarse-grained human joint coordinates are used as the initial reference point of the auxiliary frame human posture query key, and the human posture token corresponding to the auxiliary frame is obtained by processing through the self-attention network and the cross-attention network, and the human posture token is input into the multi-layer perceptron to predict the relative offset relative to the initial reference point; the initial reference point and the relative offset are added to obtain the optimized human joint coordinates; The optimized human body joint point coordinates are used as the initial reference points corresponding to the human body posture tokens in the next layer, and after traversing all layers, the posture feature tokens and human body posture joint point coordinates corresponding to the auxiliary frame are output.

5. The video-based single-stage multi-person two-dimensional human posture estimation method according to claim 3, characterized in that: The processing of the inter-frame human posture temporal information fusion in the posture tracker includes: using two self-attention networks and a feedforward neural network to fuse the human posture spatial information and temporal information in the auxiliary frame and the target frame, obtaining the optimized human posture token and inputting it into the multi-layer perceptron, using the multi-layer perceptron to predict the relative offset of the posture joint point coordinates of each frame before and after the fusion, adding the relative offset to the posture joint point coordinates to obtain the optimized posture joint point coordinates, and using the human posture token and the posture joint point coordinates as the input of the next layer.

6. The video-based single-stage multi-person two-dimensional human posture estimation method according to claim 3, characterized in that: In the joint decoder, the joint query key is first randomly initialized and copied to obtain query keys corresponding to the auxiliary frame and the target frame respectively, the posture joint point output by the posture tracker is used as the initial reference point of the joint query key of the corresponding frame, two self-attention networks and a feedforward neural network are used to fuse the spatial information and temporal information of different joint query keys corresponding to each posture in each frame, the obtained joint query key and the multi-scale feature token are input into the cross-attention network, the joint point token is output, and then the joint point token is optimized using the feedforward neural network; A multi-layer perceptron is used to predict the relative offset of the pose joint coordinates, and the relative offset is added to the pose joint coordinates to obtain the optimized pose joint coordinates. The optimized joint tokens and pose joint coordinates are used as the input of the next layer.

7. The video-based single-stage multi-person two-dimensional human posture estimation method according to claim 3, characterized in that: The total loss function L loss For: L loss =L cls +L oks +L rle ; Among them, L cls is the classification loss of the pose features output by the pose tracker, L oks is the joint point coordinate regression loss output by the pose tracker, L rle The regression loss of the joint point coordinates output by the joint decoder; The classification loss of the posture features output by the posture tracker and the regression loss of the joint point coordinates output by the posture tracker are calculated as follows: the confidence output by the posture decoder and the human joint coordinates are matched by bipartite graph using the Hungarian algorithm to calculate: in the formula represents the confidence of the prediction, c i is the true category of GT, P is the predicted joint point coordinate, P* is the true joint point coordinate of GT, K is the number of joint points, i represents the i-th joint point, s represents the manual annotation error of different joint points, and vi represents whether the joint point is visible; The formula for bipartite graph matching using the Hungarian algorithm is expressed as: N=k in the formula means k Pose-tokens, y i Represents the coordinates of the i-th joint point in the prediction result, represents the coordinates of the i-th joint point in GT, Represents the loss matrix of GT and predicted value, x represents the joint point, Indicates the minimum loss matching result between GT and prediction; The regression loss of the joint point coordinates output by the joint decoder is expressed as: Where Q(∈) is the standard normal distribution, is the predicted result distribution, f is a constant, To predict the translation coefficient, is the prediction scaling factor.

8. A video-based single-stage multi-person two-dimensional human posture estimation device, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a video-based single-stage multi-person two-dimensional human posture estimation method as described in any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a video-based single-stage multi-person two-dimensional human posture estimation method as described in any one of claims 1 to 7 is implemented.

Images