Two-stage three-dimensional human body posture estimation method based on multilevel feature compensation

Abstract

The invention discloses a two-stage three-dimensional human body posture estimation method based on multi-level feature compensation, which comprises the following steps of: 1, data preprocessing: carrying out frame sequencing processing on video data, and packaging images of all frames into a time sequence sample sequence; 2, inputting the time sequence sample sequence into a posture feature extraction branch to extract human body posture features; 3, inputting the time sequence sample sequence into an image perception branch to extract time sequence image features; 4, performing feature fusion; step 5, inputting the fusion features into a mapping network, wherein the mapping network is formed by a plurality of layers of space Transform modules and time sequence Transform modules in an alternating manner; and step 6, inputting the mapped 3D posture features into a decoder for decoding to obtain a predicted 3D human body posture. According to the method, the image sensing module is introduced, so that the information bottleneck generated by taking the two-dimensional attitude as the intermediate representation is made up, and the transmission effect of the detection error is inhibited, thereby improving the precision of three-dimensional attitude estimation.

Description

TECHNICAL FIELD

[0001] The technical solution relates to the field of computer vision, in particular to a two-stage three-dimensional human pose estimation method based on multi-level feature compensation, and is especially suitable for virtual reality (VR / AR), motion analysis, security monitoring, intelligent transportation and film and television game application scenarios. BACKGROUND

[0002] Three-dimensional human pose estimation, as a core technology of computer vision, aims to reconstruct three-dimensional pose information of human body from two-dimensional images. Existing technologies are mainly divided into two categories: one is an end-to-end method, which directly learns the mapping relationship from 2D image to 3D representation through a single neural network. This method is an end-to-end automatic processing, but has high computational complexity and requires large hardware resources, which limits the real-time performance and universality. The other is a two-stage method, which needs to first estimate the two-dimensional pose as an intermediate representation, and then map it to the three-dimensional space. This method effectively filters image noise through the intermediate representation, improving the robustness of the system, but relies heavily on the accuracy of two-dimensional pose estimation, and does not fully utilize the temporal information.

[0003] The existing two-stage method has information bottleneck effect: the two-dimensional pose as an intermediate representation has undergone excessive dimension reduction processing relative to the original image information, resulting in the loss of some key information (such as texture, lighting, depth clues, etc.), increasing the uncertainty of the two-dimensional to three-dimensional mapping. And in the occlusion scene, the depth perception of the two-dimensional pose representation degenerates, the depth clues in the original image are discarded, further increasing the uncertainty of the mapping process, and secondly, due to the error propagation effect, the detection error introduced by the detection module will be transmitted to the three-dimensional pose prediction through the mapping network, affecting the final accuracy. Since the detection error may be amplified in the propagation process, the existing method cannot completely eliminate its negative effects through simple model optimization.

[0004] The existing scheme completely entrusts the accuracy of the two-dimensional pose to the two-dimensional pose estimation task, hopes to optimize the detection error through the progress of the detection module, and ignores the negative impact of the detection error propagation effect on the prediction of the three-dimensional human pose, only focusing on model optimization, which will inevitably encounter performance bottlenecks when the model capability reaches a certain level. The reason why the existing two-stage scheme is trapped in the accuracy bottleneck is that it only uses the two-dimensional pose J as the only intermediate representation, thus inevitably leading to the occurrence of the two inherent problems of information bottleneck effect and error propagation effect, which restricts the accuracy improvement of the 3D human pose estimation scheme. SUMMARY

[0005] The present application aims to overcome the deficiencies of the prior art, and provide a two-stage three-dimensional human pose estimation method based on multi-level feature compensation, which introduces an image perception module to compensate for the information bottleneck caused by two-dimensional pose as an intermediate representation, and suppresses the transmission effect of detection errors, thereby improving the accuracy of three-dimensional pose estimation.

[0006] The purpose of the present application is achieved by the following technical solution: a two-stage three-dimensional human pose estimation method based on multi-level feature compensation, comprising the following steps:

[0007] First step, data preprocessing: frame sequence processing is performed on video data, and a human detection box is recognized from the framed image by YOLOv8, and the height and width of the human detection box are fixed to a fixed ratio of 4:3; then, the human detection box is cropped from the image, and its size is adjusted to a fixed resolution of 256x192; all frames of images are packaged as a group of time sequence sample sequences;

[0008] Second step, inputting the time sequence sample sequence into the pose feature extraction branch to extract human pose features; the pose feature extraction branch includes a 2D pose detection module, a pose encoder and a space-time encoding module; the 2D pose detection module detects two-dimensional human poses from each frame of image, and inputs the pose data into the pose encoder to encode the pose data into a feature space, obtaining a two-dimensional pose feature sequence;

[0009] The space-time encoding module is composed of a space Transformer module and a time sequence Transformer module, which introduces spatial encoding to the two-dimensional pose feature through the space Transformer module, and obtains the information of adjacent points in space; then, the time sequence Transformer module is used to introduce time sequence encoding to the two-dimensional pose, and obtain the information of adjacent poses in time sequence;

[0010] Third step, image feature extraction: inputting the time sequence sample sequence into the image perception branch to extract time sequence image features; the image perception branch is composed of an image perception module and a time sequence Transformer module, and the image perception module adopts a ViT encoder; the features output by the ViT encoder are time sequence encoded through the time sequence Transformer module, obtaining time sequence image features;

[0011] Fourth step, feature fusion: the pose feature sequence is dimensionally reorganized and expanded in the spatial dimension to obtain a skeleton point feature sequence; after the time sequence image features are spliced in the time dimension, they are reduced in dimension through a 1*1 convolution to obtain input image features; the skeleton point feature sequence and the input image features are spliced in the spatial dimension to obtain fusion features input to the mapping network;

[0012] Fifth step, input the fusion feature into the mapping network, the structure of the mapping network is composed of multiple layers of space Transformer module and time series Transformer module alternately, after passing through multiple layers of space-time alternating Transformer module, finally output the mapped 3D pose feature F o ;

[0013] Sixth step, decode the mapped feature into 3D human pose coordinates: input the mapped 3D pose feature F o into the decoder for decoding to obtain the predicted 3D human pose.

[0014] The beneficial effects of the present application are: the present application proposes a two-stage scheme architecture of image feature compensation: And according to this idea, a two-stage scheme MFCFN is proposed, which takes ViT encoder as the image perception module. The introduced image compensation branch can extract the original image feature F I , compensate the information bottleneck of , and restrict the propagation of 2D pose detection error in the mapping network, so as to break the existing precision bottleneck. The present application makes up for the information bottleneck caused by two-dimensional pose as an intermediate representation, suppresses the transmission effect of detection error, and thus improves the accuracy of three-dimensional pose estimation. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is the flow chart of the two-stage three-dimensional human pose estimation method of the present application;

[0016] Figure 2 is the structure diagram of the multi-layer feature compensation fusion network of the present application;

[0017] Figure 3 is the data flow of the self-made force mechanism, (a) is the self-attention mechanism, (b) is the space Transformer module, and (c) is the time series Transformer module. DETAILED DESCRIPTION

[0018] The technical solutions of the present application will be further described below in combination with the drawings.

[0019] As shown in Figure 1 , a two-stage three-dimensional human pose estimation method based on multi-level feature compensation of the present application comprises the following steps:

[0020] The first step, data preprocessing: the video data is frame sequenced, and the video data is divided into multiple frames of images; then the YOLOv8 is used to identify the human body detection frame from the frame-divided images, and the height and width of the human body detection frame are fixed to a fixed ratio of 4:3; then, the human body detection frame is cropped from the image, and the size is adjusted to a fixed resolution of 256x192; secondly, all frames of images are packaged into a group of time sequence sample sequences, and converted into a tensor with a size of [B, T, C, H, W] to adapt to the input format requirements of the model. The tensor is a data structure designed in the pytorch deep learning framework to facilitate the storage of vectors and matrices. The tensor ensures data format standardization and efficient calculation (the framework is automatically parallelized to speed up GPU operation). Wherein B represents the batch size, T represents the sequence length, in this embodiment, there are 243 frames of images, therefore T=243; C=3, representing the number of RGB picture channels; H represents the picture height, and W represents the picture height. In this embodiment, H=256 and W=192; the entire process maintains spatiotemporal dimension consistency, realizing a zero-copy efficient data pipeline.

[0021] The existing two-stage scheme is trapped in the accuracy bottleneck because only the two-dimensional pose J is used as the only intermediate representation, thus inevitably leading to the occurrence of the two inherent problems of information bottleneck effect and error transmission effect, which restricts the accuracy improvement of the 3D human body pose estimation scheme. To overcome the above technical obstacles, the application innovatively constructs a fusion architecture based on multi-level feature compensation, and accordingly designs a multi-layer feature compensation and fusion network (MFCFN). The MFCFN mainly consists of a pose extraction branch, an image perception branch, a multi-level image feature fusion module, and a mapping network module for restoring 3D pose features, and the complete architecture of the MFCFN is as shown in Figure 2 .

[0022] The second step is to input the time sequence sample sequence into the pose feature extraction branch to extract the human body pose feature. The pose feature extraction branch includes a 2D pose detection module a pose encoder f enc and a space-time encoding module. The 2D pose detection module can adopt several commonly used networks in the field: Stacked Hourglass Network, Cascaded Pyramid Network, and HRNet. In this embodiment, the Stacked Hourglass Network is used as the 2D pose detection module.

[0023] The input is a time sequence sample sequence (image sequence) V={I m}, where m e [1, T] and T = 243 is the length of the image sequence.

[0024] 2D pose detection module The 2D human pose will be detected from each frame of image I m to obtain a 2D pose sequence k is the number of human skeleton points; the number of skeleton points depends on the data set used, the skeleton points in the Human3.6M, MPI-INF-3DHP, COCO, etc. data sets are 17, the skeleton points in the MPII data set are 16, and the skeleton points of SMPL are 24. The present example uses Human3.6M, MPI-INF-3DHP, so the skeleton point k = 17. And J m Input pose encoder f enc encodes the pose data into a feature space to obtain the corresponding 2D pose feature sequence where d m = 512 is the dimension of the pose feature.

[0025] In order to enhance the expression ability of the pose feature, the MFCFN introduces a space-time coding module after the encoder, which is composed of a spatial Transformer module (S-MHSA) and a time series Transformer module (T-MHSA). The spatial Transformer module introduces spatial coding into the 2D pose feature and obtains the information of the adjacent points in space; then the time series Transformer module introduces time series coding into the 2D pose and obtains the information of the adjacent poses in time series. The pose feature extraction branch makes the human pose feature obtain the information of all skeleton points in the sequence through the space-time coding module, and has stronger expression ability. Its data flow can be described as:

[0026]

[0027] where T-MHSA is a multi-head time self-attention mechanism, and S-MHSA is a multi-head space self-attention mechanism, both of which are improved from the multi-head self-attention mechanism.

[0028] In the self-attention module, the input sequence information is calculated through three linear layers to obtain the Query (Q) matrix, the Key (K) matrix, and the Value (V) matrix. As shown in Figure 3 (a), the Q matrix and the K matrix corresponding to the input sequence information will calculate the vector inner product, and the inner product is scaled by Scale and activated by Softmax to obtain the attention map. The Scale value is where d mQ, K, V are the characteristic dimension numbers of Q, K, V, each element in the attention graph represents the weight of the corresponding position element of the sequence, and the attention degree of the i-th element to the j-th element is represented by the value of the [i, j] position in the matrix. The output feature is obtained by weighted sum of the attention graph and the V matrix, and the self-attention mechanism can process the mutual dependence of all elements of the sequence in parallel, greatly improving the computing efficiency. Through the back propagation optimization of the neural network, the self-attention mechanism can adaptively learn the dependence between elements. The calculation formula is as follows:

[0029]

[0030] wherein N represents the number of input tokens; is the coefficient of the attention graph, which is related to the characteristic dimension. Let the input feature be F and the characteristic dimension be d m , then Q, K, V can be calculated by the following formula:

[0031] Q = FW Q (3)

[0032] K = FW K (4)

[0033] V = FW V (5)

[0034] wherein is a learnable weight matrix; linearly transform the input information to calculate the corresponding Q, K, V three matrices respectively.

[0035] The multi-head attention mechanism is improved on the basis of the self-attention mechanism, which maps the input feature to h subspaces respectively, then calculates the above self-attention module in each subspace respectively, and finally splices the outputs of the h subspaces together, and then performs a linear transformation to obtain the final output. The calculation formula is as follows:

[0036] head i = Attention(Q i ,K i ,V i ), i ∈ h (6)

[0037] MHSA = Concat(head I ,…, head h )W O (7)

[0038] is used for linear transformation of the output information, and MHSA is the output of the multi-head attention mechanism.

[0039] T-MHSA treats each frame of the time series as a token (image word), assuming that the feature dimension of a frame is d. m Q is calculated using equations (3) to (5). t ,K t V t Subsequently, the weights of different points in the time series are calculated using the attention mechanism in equation (8). The attention map for the relevance of different frames in the input time series information is calculated as follows:

[0040]

[0041] in These represent the query, key, and value of the time series feature, respectively; L represents the length of the time series, and d represents the value of the time series. m The feature dimension represents each time point. Equation (8) yields... The matrix represents the degree of correlation between the information of each frame and the information of other frames in a time series. Through Att... t Sum V t The weighted summation yields the time series output, where the output of each frame takes into account the information of that frame and other frames in the sequence.

[0042] S-MHSA treats spatial locations (skeleton points) as tokens. Similar to Equation (8), it assumes that the number of skeleton points on a single frame of the human skeleton is J, and the token dimension of each point is d. m Q is calculated using equations (3) to (5). s K s V s Subsequently, the attention mechanism in equation (9) is used to calculate the interrelationships between different spatial locations. The attention map for the correlation between different spatial locations is calculated as follows:

[0043]

[0044] in These represent the query, key, and value for the spatial dimension, respectively; the calculated... The matrix represents the degree of correlation between the information of each point in space and the information of other points, and then through Att... s Sum V sThe output of the point in space can be obtained by the weighted sum of the above. At this time, the output contains the information of the point and the information of all other points in the frame. As with the traditional Transformer architecture, the output of S-MHSA and T-MHSA also undergoes a residual connection (connecting the input and output of S-MHS, AT-MHSA), then input LayerNorm, and then through a multi-layer perception (MLP) and a residual connection (connecting the input of LayerNorm and the output of MLP), output the intermediate table obtained by the module, and the specific process is as follows Figure 3 (b) and 3(c) are shown.

[0045] The third step is image feature extraction: the time sequence sample sequence is input into the image perception branch to extract the time sequence image feature. The image perception branch is composed of an image perception module and a time sequence Transformer module, and the image perception module adopts a ViT (Vision Transformer) encoder. The features output by the ViT encoder are time-coded through the time sequence Transformer module to obtain the time sequence image feature.

[0046] The input is a time sequence sample sequence (image sequence) V = {I m}, where m ∈ [1, T], T = 243 is the length of the image sequence. A single frame image I m is divided into multiple patches (image blocks) and converted into a token sequence, and global features are extracted through a multi-layer multi-head self-attention (T-MHSA) and a feedforward neural network (FFN), and a single frame feature (d v = 1024, which is the dimension of the single frame image feature) is output. Since F m only contains the information of the current frame, it lacks the information of other adjacent frames in time sequence and is easily affected by problems such as occlusion or motion blur, so a time sequence Transformer module is added to introduce temporal embedding, and the time attention mechanism is used to fuse the time sequence information between the sequence frames, and finally F I with time sequence robustness is output. Through the time attention mechanism of the time sequence Transformer module, the time sequence image feature not only contains the information of the current frame, but also benefits from other frames in the motion sequence. The image feature can obtain stronger robustness through the time sequence module, and has stronger anti-interference ability to problems such as occlusion and motion blur. The data flow can be described as:

[0047]

[0048] represents an image perception module, which adopts a ViT (Vision Transformer) encoder.

[0049] Fourth step, feature fusion: after the input image is processed by the posture feature extraction branch and the image perception branch, the corresponding posture feature sequence is obtained and the time sequence image feature F I , wherein m ∈ [1, T], T = 243 is the length of the image sequence. The time sequence image feature F is dimensionally reorganized and expanded in the spatial dimension (the original dimension is T × k × d m , and after reorganization, it is k × T × d m . The input skeleton point feature F is obtained by changing the expansion from the time dimension to the spatial dimension. , wherein i ∈ [1, k], k is the number of human skeleton points, d m is the feature dimension, and the skeleton point feature sequence F is obtained. The time sequence image feature F I is spliced in the time dimension, and the input image feature F C is obtained by 1*1 convolution dimension reduction. The skeleton point feature sequence F and the input image feature F C are spliced in the spatial dimension to obtain the fusion feature F f input to the mapping network; represented as:

[0050]

[0051] , wherein

[0052] Fifth step, input the fusion feature into the mapping network, and the mapping network is a plug-and-play detachable module of the model, which can be compatible with any baseline model in the two-stage work. The structure of the mapping network in the application is composed of multiple layers of spatial Transformer modules and time sequence Transformer modules, which can obtain the correlation of the input information in the time dimension and the spatial dimension multiple times. Since the image feature is introduced in the spatial dimension, when calculating the spatial correlation, the mapping network will not only consider the correlation between the skeleton points, but also consider the correlation between each skeleton point and the image feature. This information can make up for the lack of depth information in the posture feature and limit the error transmission effect of the detection error. When and F m are fused to generate the comprehensive feature F f , which is input to the mapping network, and after passing through multiple layers of space-time alternating Transformer modules, the mapped 3D posture feature F o is finally output; the data flow can be described as:

[0053] F O = T-MHSA (S-MHSA (FO-1 )) O∈[1,d] (12)

[0054] where d is the number of layers of the space-time alternating Transformer network.

[0055] The sixth step is to map the feature to decode 3D human pose coordinates: the mapped 3D pose feature F o is input into the decoder to obtain the predicted 3D human pose, and the decoder can use a single-layer fully connected layer, and the data flow thereof can be described as:

[0056]

[0057] where, is a predicted three-dimensional pose sequence, and f dec is a decoder.

[0058] In view of the problems existing in the existing two-stage scheme, the method proposes a two-stage scheme architecture of image feature compensation: According to the idea, a two-stage scheme MFCFN is proposed, in which the image compensation branch is introduced to extract the original image feature F I compensation information bottleneck, and restricts the propagation of 2D pose detection error in the mapping network, thereby breaking the existing precision bottleneck.

[0059] In line with other monocular three-dimensional human pose estimation schemes, detailed comparative experiments are carried out on Human3.6M, as shown in Table 1, which illustrates the superiority of the MFCFN scheme in estimation accuracy.

[0060] Table 1 Qualitative comparison of the average (per) joint position error (Mean Per Joint Position Error, MPJPE) in millimeters (mm) under the condition of not applying rigid alignment (MPJPE) and applying rigid alignment (P-MPJPE) on Human3.6M. CPN and SH are 2D pose detectors The best result is in bold. Among them, Ours+ represents the experimental effect of the method proposed in the application in cooperation with the mapping network module in the method (corresponding to the fifth step in the scheme),

[0061] MACs / frame represents the number of multiply-accumulate operations per frame in calculation, which is used to measure the computational complexity of the method.

[0062]

[0063]

[0064] Those skilled in the art will appreciate that the embodiments described herein are presented for purposes of illustration and that the inventive principles are not limited to these particular embodiments. Other variations and modifications can be made to the embodiments without departing from the spirit and scope of the inventive principles.

Claims

1. A two-stage 3D human pose estimation method based on multi-level feature compensation, characterized in that, The method comprises the following steps: The first step, data preprocessing: frame the video data, identify the human detection frame from the framed image through YOLOv8, and fix the height and width of the human detection frame to a fixed ratio of 4:3; Subsequently, the human detection frame is cropped from the image, and its size is adjusted to a fixed resolution of 256x192; all frame images are packaged into a set of time sequence sample sequences; The second step, input the time sequence sample sequence into the pose feature extraction branch to extract the human pose feature; The pose feature extraction branch comprises a 2D pose detection module, a pose encoder and a space-time encoding module in sequence; the 2D pose detection module detects a two-dimensional human pose from each frame of image and inputs the pose data into the pose encoder to encode the pose data into a feature space to obtain a two-dimensional pose feature sequence; The space-time encoding module is composed of a spatial Transformer module and a time sequence Transformer module; the spatial Transformer module introduces spatial coding into the two-dimensional pose feature and obtains the information of adjacent points in space; then the time sequence Transformer module introduces time sequence coding into the two-dimensional pose and obtains the information of adjacent poses in time sequence; The third step, image feature extraction: input the time sequence sample sequence into the image perception branch to extract the time sequence image feature; the image perception branch is composed of an image perception module and a time sequence Transformer module, and the image perception module adopts a ViT encoder; the time sequence Transformer module performs time sequence coding on the features output by the ViT encoder to obtain the time sequence image feature; The fourth step, feature fusion: after the pose feature sequence is dimensionally reorganized, a skeleton point feature sequence is obtained by expanding in the spatial dimension; after the time sequence image feature is spliced in the time dimension, an input image feature is obtained through 1*1 convolution dimension reduction; the skeleton point feature sequence and the input image feature are spliced in the spatial dimension to obtain a fusion feature input to the mapping network. In the fifth step, the fused features are input into a mapping network. The structure of the mapping network is composed of multiple layers of spatial Transformer modules and time-series Transformer modules. After passing through multiple layers of spatial and temporal alternating Transformer modules, the final output is the mapped 3D pose features F o ; Step 6, decode the mapped feature into 3D human pose coordinates: the mapped 3D pose feature F o decoded in the input decoder to get the predicted 3D human pose.

Images