A method and apparatus for video human pose estimation

Abstract

The application provides a video human pose estimation method, comprising the following steps: inputting a video and cutting the video into images; determining the position of each person in the video and cutting; inputting the cut images into a human pose estimation method based on a skin multi-person linear model to obtain result data, wherein the skin multi-person linear model is a naked human three-dimensional model based on a vertex; initializing a hidden space sequence, inputting data into a motion prior conditional variational autoencoder, aligning the joint nodes and calculating the square error of the generator and the observed results, and minimizing the square error; and outputting a human pose estimation result. The method has strong robustness; the method can optimize the 3D result represented by the skin multi-person linear model, solve the problem that the prior conditional variational autoencoder cannot optimize the skin multi-person linear model result, prevent the generation of depth ambiguity and the loss of 3D information, and thus generate smooth and accurate video human pose estimation results.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a video human pose estimation method and device. BACKGROUND

[0002] Human pose estimation (HPE) has been widely studied in the history of computer vision. Given an observation of a point cloud captured from a sensor, or a segment of RGB (color mode) video, or even a RGB image, it is the main task of human pose estimation to let the neural network or iterative program automatically restore the human pose.

[0003] The parameter information of the human model obtained by HPE contains the geometric information and motion information of the human body, which has been widely applied, and common applications include human-computer interaction, augmented reality (AR), virtual reality (VR), etc. In addition, HPE has a wide and key application in autonomous driving, such as pedestrian detection. Pedestrian detection achieved by HPE technology is more accurate than traditional target detection, providing a guarantee for the safety of autonomous driving. With the concept of Metaverse, human pose estimation has become a key technology, and more and more scholars are investing in the research of human pose estimation, and the popularity of HPE is about to reach a climax.

[0004] There are many ways to collect human data. We can take the projection of the human pose and shape on the camera film as an RGB image as the input of human pose estimation, or through an RGB-D camera (depth camera), we can obtain the depth information of the picture in addition to the image information. A more direct method is to collect point clouds on the surface of the human body through sensors. However, the above data is difficult to be directly applied in animation rendering programs, such as making the human body stored in the computer change a pose or move. Moreover, the collected data may have noise due to camera distortion or physical factors in the real environment, which is inevitable. Therefore, we need to establish a model to describe the human pose and shape. The current human pose model includes the linear blend skinning model (LBS Model) and the skinned multi-person linear model (SMPL Model).

[0005] After determining which model is selected to describe the human body, only the pose parameters and shape parameters of the model need to be iterated to minimize the theoretical joint position of the model and the actual joint observation. In addition to the iterative method, deep learning can also be applied to human pose estimation. In recent years, the computing power of GPU has promoted the development of deep learning. With the maturity of deep learning, more and more complex tasks can rely on the high computing power of computers to solve, such as image classification, semantic segmentation, target detection, etc. Similarly, in the task of human pose estimation, a good result can also be obtained through a deep neural network.

[0006] However, the robustness of many human pose estimation methods is not strong enough. For example, in the face of occlusion, insufficient light and other situations, the previous method is not up to the task. For observations with time sequence, if the pose estimation is simply done in frames, the resulting motion will be very stiff, and even jitter will be generated. At the same time, the error caused by the occlusion problem in the time sequence pose and motion prediction is more obvious.

[0007] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0008] The purpose of the present application is to solve the problem of improving the accuracy of video human pose estimation, and to provide a video human pose estimation method and device.

[0009] To achieve the above purpose, the present application adopts the following technical solutions:

[0010] The present application provides a video human pose estimation method, comprising the following steps:

[0011] S1: inputting a video and cutting the video into images;

[0012] S2: determining the position of each person in the video and cutting;

[0013] S3: inputting the cut image into a human pose estimation method based on a skinning multi-person linear model to obtain result data, wherein the skinning multi-person linear model is a vertex-based naked human three-dimensional model;

[0014] S4: initializing a latent space sequence, inputting data into a motion prior condition variational autoencoder, aligning the joint nodes and calculating the squared error of the generator and the observation result, and minimizing the squared error;

[0015] S5: outputting the human pose estimation result.

[0016] In some embodiments, the skinned multi-person linear model mentioned in step S3 is SMPL, and the human pose estimation method based on the skinned multi-person linear model is PARE.

[0017] In some embodiments, the action prior condition variational autoencoder in step S4 is a HuMoR.

[0018] In some embodiments, step S4, the joint alignment step includes the following transformation:

[0019] y = D(z)

[0020] y′=yy 0

[0021]

[0022] Where y is the output of the decoder in the action prior condition variational autoencoder. 0 The coordinates of the pelvic vertices of the human model are the estimation results of the action prior condition variational autoencoder. The results represent the estimation of the skinned multi-person linear model using the aforementioned human pose estimation method. The pelvic vertex coordinates of the human model are the estimation results of the skinned multi-person linear model of the human pose estimation method. D() represents the action prior condition variational self-decoder, and z is the latent space sequence.

[0023] In some embodiments, in step S4, the formula for the squared error of the generator and the observation results is as follows:

[0024] loss = (x′ - y′) 2 ;

[0025] Where loss represents the squared error between the generator and the observation results, y′ is the transformed coordinate of the pelvic vertex of the human model in the estimation result of the action prior condition variational autoencoder, and x′ is the transformed coordinate of the pelvic vertex of the human model in the estimation result of the skinned multi-person linear model of the human pose estimation method.

[0026] In some embodiments, minimizing the squared error in step S4 includes the following steps: using stochastic gradient descent, seeking a z in the latent space. t , where z t The input to the decoder is used to minimize the squared error.

[0027] In some embodiments, step S2 uses a target tracking algorithm to determine the position of each person in the video and then crop the video.

[0028] In some embodiments, the target tracking algorithm is YOLO.

[0029] In some embodiments, the video is cut into images using ffmpeg in step S1.

[0030] The application also provides a computer readable medium, which stores a computer program, and the program is executed by a processor to implement the steps of the method.

[0031] The application has the following beneficial effects:

[0032] The method of the application optimizes the results of human pose estimation of the skin multi-person linear model based on the action prior condition variational autoencoder, and combines the action prior condition variational autoencoder with timing information, so that the application has strong robustness. The method inputs data into the action prior condition variational autoencoder, aligns the joint nodes, and calculates the squared error of the results of the generator and the observation, minimizes the setting of the squared error value, can optimize the 3D results of the skin multi-person linear model, solves the problem that the prior condition variational autoencoder cannot optimize the results of the skin multi-person linear model, and can prevent the generation of depth ambiguity and the loss of 3D information, so as to generate smooth and accurate video human pose estimation results. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a video human pose estimation method flowchart in an embodiment of the application;

[0034] Figure 2 is a HuMoR flowchart in an embodiment of the application;

[0035] Figure 3 is a PARE flowchart in an embodiment of the application;

[0036] Figure 4 is a video human pose estimation method flowchart in an embodiment of the application;

[0037] Figure 5a is a human pose estimation original image in experimental example 1 of the application;

[0038] Figure 5b is a schematic diagram of the human pose estimation results of the prior art HuMoR+PARE 2D in experimental example 1 of the application;

[0039] Figure 5c is a schematic diagram of the human pose estimation results of the prior art HuMoR+PARE 2D in experimental example 1 of the application;

[0040] Figure 5dis a human pose estimation result diagram of the HuMoR+PARE3D of the present application in Experimental Example 1;

[0041] Figure 5e is a human pose estimation result extraction diagram of the HuMoR+PARE3D of the present application in Experimental Example 1;

[0042] Figure 6a is an original human pose estimation diagram in Experimental Example 2;

[0043] Figure 6b is a human pose estimation result diagram of the prior art HuMoR+PARE2D in Experimental Example 2;

[0044] Figure 6c is a human pose estimation result extraction diagram of the prior art HuMoR+PARE2D in Experimental Example 2;

[0045] Figure 6d is a human pose estimation result diagram of the HuMoR+PARE3D of the present application in Experimental Example 2;

[0046] Figure 6e is a human pose estimation result extraction diagram of the HuMoR+PARE3D of the present application in Experimental Example 2. DETAILED DESCRIPTION

[0047] The embodiments of the present application will be described in detail below. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present application and its applications.

[0048] In addition, the terms "first", "second", etc. are used only for the purpose of description and should not be construed as indicating or implying relative importance or implying the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0049] Many human pose estimation methods are not robust enough. For example, in the face of occlusion, insufficient light, and other situations, previous methods are not up to the task. For observations with timing, if the pose estimation is simply done in frames, the resulting motion will be very stiff, and even produce jitter. At the same time, the error caused by the occlusion problem in the timing pose and motion prediction is more obvious. If the timing information of the motion is considered, better results may be produced, so it is attempted to combine HuMoR with timing information. Most methods do not consider the movement of the root node of the human body (root translation), that is, most methods believe that the origin of the human body is fixed and will not translate in the real coordinate system. In the face of such results, HuMoR cannot be optimized. The only solution in the prior art is to project the SMPL result to the 2D image plane, and then optimize it with HuMoR. Since only 2D constraints are used, depth ambiguity may occur, so it is hoped to solve the problem of optimizing the SMPL result with HuMoR and retain as much 3D information as possible.

[0050] The embodiment of the application proposes a human pose estimation method combined with motion prior and suitable for SMPL of video: 1. Strong robustness; 2. All characteristics of HuMoR, combined with human motion prior; 3. Can optimize the 3D result of SMPL representation.

[0051] The prior art HuMoR is described as follows:

[0052] HuMoR is a conditional variational autoencoder (CVAE) proposed in ICCV 2019 Oral (IEEE International Conference on Computer Vision).

[0053] Referring to Figure 2 , Figure 2 x t-1 represents the past state, which is composed of past human joint coordinates, human SMPL model parameters, and their changes. x t represents the state at time t, the time unit is the frame of the video, generally 1 / 30 second. q φ is the model parameter of the encoder, which is actually composed of a plurality of matrix operations and nonlinear mappings (ReLU, etc.) in deep learning. z t is a 512-dimensional vector. Correspondingly, p θ is the parameter of the prior encoder, p θ is the parameter of the decoder. is a bool value indicating whether the 8 joints are in contact. Δθ is the change of action parameter, is the result of decoding. The meaning is that the past time point and the present state are input, they are projected into a 512-dimensional vector space, and the Prior encoder is guided to project only from the past state to this space, and the decoder can recover the change from the past state to the present from this space, that is, to reconstruct the action of the human body. If this structure is trained on a large-scale action dataset, a CVAE combined with action prior can be obtained.

[0054] The CVAE of this design learns the action prior. When training, the human action state x t-1 and x t are input, and the state change of the action is self-encoded by a deep neural network. Therefore, the VAE can instance a z (that is, an instance of the above vector space) from the distribution during training, and the action state can be obtained by decoding the z.

[0055] This CVAE is trained using AMASS (a large-scale 3D motion capture dataset), so the action sequence output by the Decoder is necessarily smooth, that is, it combines temporal information and conforms to the action prior.

[0056] Therefore, to obtain a pose estimation sequence that conforms to the action prior, a sequence of z can be found from the normal distribution to which z belongs, so that the action sequence output by the Decoder is close to the noisy action sequence , that is, the smoothing effect is achieved.

[0057] In the prior art, HuMoR can only optimize 3D absolute coordinates and 2D relative coordinates.

[0058] The prior art PARE is described as follows:

[0059] PARE is a single-frame human pose estimation method in 2021 ICCV, which has the best effect at present. It is tried to combine the results of PARE with the action prior of HuMoR to realize video human pose estimation.

[0060] Referring to Figure 3 , the input image I is represented in RGB (3X224X224), the CNN backbone can select resnet (a feature extraction network) and hrnet (a feature extraction network), and then pass through two branches. Conv represents a convolution operation, H, W, J, and C represent the length, width, number of joints, and channel dimension of the image, and P and F are tensors after convolution. After changing the shape of the tensor, we get ​σ is sigmoid (activation function of neural network) operation. After multiplication, it is F', which is called Part Attention process in Figure 3 i θ is the motion parameter of SMPL, β is the shape parameter, M is the matrix of SMPL vertex to joint (also known as key point, which is composed of 24 joints such as pelvic joint, wrist joint, etc.), ∏ is the camera projection transformation, J 2D 3D represent 2D and 3D labels of joints. L SMPL 2D 3D is the loss function necessary in deep learning. That is, the result is squared after being subtracted from the real result.

[0061] The single-frame video pose estimation of PARE is based on SMPL, and the movement of the root node coordinate (root translation) cannot be accurately estimated. Usually, the predicted root translation is set to 0, that is, the actual person is standing still. For example, input a video of a person running, in fact, the predicted human model will only do the running action in the original place, and will not really move forward, although its action pose is standard. Therefore, there are some methods to simulate the distance of the person by predicting the size of the human body shape, but this estimation is extremely unreliable and will produce severe jitter in the depth direction.

[0062] In the prior art, even if the estimation performance of PARE is very good, it cannot be optimized by HuMoR.

[0063] The embodiments of the present application are summarized as follows:

[0064] The embodiments of the present application provide a video human pose estimation method, which comprises the following steps:

[0065] S1: input a video and cut the video into images; ffmpeg (an open source project of multimedia framework) can be used to cut the video into images.

[0066] S2: determine the position of each person in the video and crop; the target tracking algorithm yolo can be used to determine the position of each person in the video and crop.

[0067] S3: input the cropped image into the human pose estimation method PARE based on the vertex-based naked human three-dimensional model skinning multi-person linear model SMPL to obtain result data;

[0068] S4: initialize the latent space sequence, input the data into the motion prior condition variational autoencoder HuMoR, align the joints, and the joint alignment step comprises the following transformation: ​​​​

[0069] y = D(z);

[0070] y' = y - y 0 ;

[0071]

[0072] where y is the output of the decoder in the action prior conditioned variational autoencoder, y 0 is the pelvis top coordinates of the human model estimated by the action prior conditioned variational autoencoder, is the estimation result of the skinning multi-person linear model of the human pose estimation method, is the pelvis top coordinates of the human model estimated by the human pose estimation method, y' is the transformed coordinates of the pelvis top coordinates of the human model estimated by the HuMoR, x' is the transformed coordinates of the pelvis top coordinates of the human model estimated by the PARE, and D() represents the action prior conditioned variational decoder, and z is the latent space sequence.

[0073] The square error of the generated result and the observation is calculated, and the formula is as follows:

[0074] loss = (x' - y') 2 ;

[0075] where loss represents the square error of the generated result and the observation, y' is the transformed coordinates of the pelvis top coordinates of the human model estimated by the HuMoR, and x' is the transformed coordinates of the pelvis top coordinates of the human model estimated by the PARE.

[0076] Minimizing the square error includes the following steps: using the stochastic gradient descent method to seek z t in the latent space t , which is the input of the decoder, so that the square error is minimized.

[0077] S5: output the human pose estimation result.

[0078] The embodiment of the present application also provides a computer readable medium, which stores a computer program, and the program is executed by a processor to realize the steps of the method.

[0079] As mentioned above, HuMoR can optimize 3D point clouds relative to the world coordinate system, while PARE can only predict the parameters of the SMPL, where the coordinates of each joint are relative to the pelvis. Therefore, how to optimize PARE results using HuMoR is a problem. One solution is to project PARE onto the 2D plane of the image and then optimize it with HuMoR, but this introduces depth ambiguity and loses some 3D information. To address this issue, a root node alignment optimization method for video human pose estimation suitable for SMPL prediction is proposed.

[0080] set up Let be the observed value, D be the decoder of CVAE, z be the latent space sequence, and y be the output of HuMoR. This is the prediction result for PARE. Normally, HuMoR cannot optimize the results of PARE because HuMoR's results are relative to world coordinates, while PARE's results are relative to the root nodes of the human pelvis.

[0081] To align the joints during the optimization process, the following transformation is performed:

[0082] y = D(z);

[0083] y′=yy 0 ;

[0084] Where y is the output of the HuMoR decoder.

[0085] And y 0 The coordinates of the pelvic apex of the human model based on the HuMoR estimation results.

[0086] Similarly, for the observation end (i.e., the PARE output represented by SMPL), the following transformation can be performed.

[0087]

[0088] in The SMPL estimation results for PARE et al. HPE are as follows.

[0089] and The coordinates of the pelvic apex of the human model as estimated by PARE;

[0090] The squared error of the generator and the observed results:

[0091] loss = (x′ - y′) 2 ;

[0092] Based on the above method, the embodiments of the present invention can achieve optimization of HuMoR+SMPL3D. Stochastic gradient descent is used to find a z in the latent space. tSo as to make the loss function as small as possible.

[0093] Embodiments:

[0094] A video human pose estimation method in an embodiment of the present application is shown in Figure 4 and includes the following steps:

[0095] S1, cutting a video into images by using ffmpeg (a multimedia framework open source project);

[0096] S2, determining the position of each person in the video and cutting by using yolo (a target detection model);

[0097] S3, inputting the cut images into PARE to obtain the SMPL result (at this time, the result is not combined with the motion prior, and the obtained result has strong jitter, and the simulated translation (relative position movement of the human body in space) is extremely unreliable);

[0098] S4, optimizing z and calculating the loss of the relative joint (i.e. the squared error loss of the generator and the observed result), and minimizing the value;

[0099] S5, the generated y at this time is the smoothed motion after denoising.

[0100] If only single-person estimation is performed, the yolo step can be replaced by openpose to circle the position of the person. With the update of deep learning algorithms, more excellent target tracking algorithms and SMPL estimation methods will be proposed, and only part of the modules of the present method need to be replaced.

[0101] Experimental Example 1

[0102] As shown in Figure 5a In the prior art, in order to solve the problem of optimizing the SMPL result by HuMoR, the 3D coordinates relative to the root node obtained by SMPL are usually projected onto the image plane to obtain 2D coordinates through a preset camera internal and external parameter matrix, and then a HuMoR+2D optimization method is adopted. Since there is a 3D to 2D projection process, depth ambiguity will be generated. For example, the legs of the human body are crossed, but after being projected onto the 2D plane, it is not clear which leg is in front of the other. The optimization method of HuMoR+PARE3D SMPL result gives a restriction in the depth dimension in the optimization process, avoiding similar ambiguity.

[0103] Experimental Example 2

[0104] As shown in Figure 6aAs shown in the figure, the optimization method of HuMoR+2D in the prior art provides limited supervised information, which may cause inaccurate results, such as the left hand not being unfolded in this example, but the method of HuMoR+PARE 3D SMPL solves this problem.

[0105] HuMoR is an optimization method of human motion prior, but it can only optimize the absolute coordinate points in 3D space and 2D key points, and has no effect on the SMPL representation method relative to the root node. PARE is one of the best methods in human pose estimation. If the two are combined, better results can be achieved.

[0106] The beneficial effects of the embodiments of the present application are as follows:

[0107] 1. PARE itself is very robust and can handle the occlusion problem, but its separate estimation result is not available. If combined with HuMoR, the motion can be smoother;

[0108] 2. Solve the problem that the SMPL prediction model such as PARE cannot be combined with HuMoR;

[0109] 3. Taking the SMPL result optimization provides more available 3D information than taking the 2D result optimization, and reduces the depth ambiguity.

[0110] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0111] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the function specified in one block or multiple blocks.

[0112] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the function specified in the flow or flows and / or blocks Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0114] The above description is further to the specific / preferred embodiments of the present application and cannot be deemed to limit the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, they can make several substitutions or variations to the described embodiments, and these substitutions or variations shall be deemed to fall within the protection scope of the present application. In the description of the present application, the description of the terms "an embodiment", "some embodiments", "a preferred embodiment", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction. Although the embodiments of the present application and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope of the patent application.

Claims

1. A method of video human pose estimation, the method comprising: The method comprises the following steps: S1: inputting a video and cutting the video into images; S2: determining the position of each person in the video and cropping; S3: inputting the cropped images into a human pose estimation method based on a skinning multi-person linear model to obtain result data, wherein the skinning multi-person linear model is a vertex-based naked human three-dimensional model; the skinning multi-person linear model is SMPL, and the human pose estimation method based on the skinning multi-person linear model is PARE; S4: initialize the latent space sequence, input the resulting data into the action prior variational autoencoder, align the nodes and compute the squared error of the generator and the observed results, seek a latent space in wherein is the input to the decoder to minimize the squared error; the action prior variational autoencoder is a HuMoR; the node alignment step comprises the following transformation: wherein is an output of a decoder in the action prior conditioned variational autoencoder, is a pelvis apex coordinate of a human body model of an estimation result of the action prior conditioned variational autoencoder, is an estimation result of a skinning multi-person linear model of the human body pose estimation method, is a pelvis apex coordinate of a human body model of an estimation result of the human body pose estimation method, D() represents an action prior conditioned variational autoencoder, and z is a latent space sequence; and a square error formula of a result of the generator and the observation is as follows: ； wherein loss denotes a squared error of the generator and the observed result, is a transformed coordinate of a pelvis vertex coordinate of a human body model estimated by the action prior conditional variational autoencoder, is a transformed coordinate of a pelvis vertex coordinate of a human body model estimated by the human pose estimation method. S5: output the human pose estimation result .

2. The method of claim 1, wherein, Minimizing the squared error in step S4 comprises the step of finding a hidden space with a stochastic gradient descent method wherein is the input to the decoder such that the squared error is minimized.

3. The method of claim 1, wherein, In step S2, a target tracking algorithm is used to determine the position of each person in the video and crop.

4. The method of claim 3, wherein, The target tracking algorithm is yolo.

5. The method of claim 1, wherein, In step S1, ffmpeg is used to cut the video into images.

6. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by a processor to implement the steps of the method according to any one of claims 1-5.

Images