A multi-view three-dimensional human posture estimation method and device and a storage medium

Abstract

The specification discloses a multi-view three-dimensional human posture estimation method, device and storage medium. In the multi-view three-dimensional human posture estimation method provided in the specification, the two-dimensional representation parameters of each human image are determined through the view-in module of the three-dimensional human posture estimation model, the three-dimensional representation parameters of each human image are determined through the intermediate features and the two-dimensional representation parameters of each human image, the three-dimensional representation parameters at least include the second shape parameters and the second posture parameters, the second shape parameters and the second posture parameters are input into a preset human model, and the estimated three-dimensional human posture is obtained. In this way, the features of each human image are fused in the parameter form, the three-dimensional representation parameters of each human image are determined, and since the three-dimensional human posture obtained through the human model is the three-dimensional human posture under the joint influence of the skeleton structure and the muscle shape, the three-dimensional human posture corresponding to each view image obtained by estimation is more accurate.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a multi-view three-dimensional human pose estimation method, device and storage medium. Background Technology

[0002] Currently, 3D human pose estimation is being used in a wider range of fields, such as posture health detection, fitness movement guidance, fall detection, sign language translation, gait analysis, and 3D special effects movies.

[0003] In existing technologies, single-view 3D human pose estimation methods suffer from inaccuracies due to the limited image acquisition from only a single perspective. Therefore, to ensure accuracy, multi-view 3D human pose estimation methods are typically employed. First, images from each perspective are acquired and then used to determine the 2D human pose at each perspective through a Pictorial Structures Model (PSM). The 2D human pose is represented as a skeleton formed by connecting multiple 2D joints. Then, the joints in each 2D human pose are mapped to 3D space to obtain the 3D human pose represented by the skeleton in 3D joints.

[0004] However, since the three-dimensional human pose obtained by PSM is represented in the form of a skeleton, and human pose is not only determined by the skeletal structure but also affected by the muscle morphology, the three-dimensional human pose obtained by PSM cannot determine the different human poses caused by different muscle morphologies under the same skeletal structure, making the three-dimensional human pose obtained based on PSM inaccurate.

[0005] Therefore, determining the accurate three-dimensional human pose is an urgent problem to be solved. Summary of the Invention

[0006] This specification provides a multi-view 3D human pose estimation method, apparatus, storage medium, and electronic device to at least partially solve the aforementioned problems existing in the prior art.

[0007] The following technical solution is adopted in this specification:

[0008] This manual provides a multi-view 3D human pose estimation method, including:

[0009] Acquire human images from different perspectives and input each human image into the in-view module of the 3D human pose estimation model;

[0010] For each human body image, the intermediate features of the human body image are obtained through the in-view module. Based on the intermediate features, the two-dimensional representation parameters corresponding to the human body image are determined. The two-dimensional representation parameters include camera pose parameters, human body pose parameters, and human body shape parameters.

[0011] Based on the posture parameters of each human body, determine the first posture parameters; based on the shape parameters of each human body, determine the first shape parameters.

[0012] The intermediate features of each human body image, the camera pose parameters of each two-dimensional representation parameter, the human pose parameters of each two-dimensional representation parameter, the first pose parameter and the first shape parameter are input into the viewpoint module of the three-dimensional human pose estimation model to perform feature fusion and determine the three-dimensional representation parameters representing each human body image. The three-dimensional representation parameters include at least the second pose parameter and the second shape parameter.

[0013] The second posture parameter and the second shape parameter are input into a preset human body model to obtain the estimated three-dimensional human posture.

[0014] Optionally, the three-dimensional human pose estimation model is trained using the following method, wherein:

[0015] Acquire sample images of the same human body in different perspectives;

[0016] Each sample image is input into the in-view module of the three-dimensional human pose estimation model to be trained, and the two-dimensional representation parameters corresponding to each sample image output by the in-view module are obtained.

[0017] The in-view annotations of each sample image are determined. For each sample image, the in-view loss of the sample image is determined based on the in-view annotations of the sample image and the two-dimensional representation parameters. The in-view module is then trained based on the in-view loss.

[0018] When the in-view module is trained, each sample image is input into the trained in-view module to determine each intermediate feature obtained by the trained in-view module and each two-dimensional representation parameter output.

[0019] Based on the posture parameters of each human body, determine the first posture parameters; based on the shape parameters of each human body, determine the first shape parameters.

[0020] The intermediate features determined by the in-view module, the camera pose parameters included in each two-dimensional representation parameter, the human pose parameters included in each two-dimensional representation parameter, the first pose parameter, and the first shape parameter are input into the inter-view module to be trained for feature fusion to obtain the three-dimensional representation parameters output by the inter-view module to be trained.

[0021] Determine the inter-view annotations, determine the inter-view loss based on the inter-view annotations and the 3D representation parameters, and train the inter-view module based on the inter-view loss.

[0022] Optionally, the in-view annotations for each sample image can be determined separately, specifically including:

[0023] For each sample image, the sample image is input into a preset neural network to determine the corresponding reference two-dimensional joint points of the sample image;

[0024] The human pose parameters and human shape parameters included in the two-dimensional representation parameters of the sample image are determined, and the preset optimization algorithm is input to obtain the first reference pose parameters and the first reference shape parameters corresponding to the image.

[0025] Each reference 2D joint, the first reference pose parameter, and the first reference shape parameter are used as the in-view annotation of the sample image.

[0026] Optionally, based on the in-view annotation of the sample image and the two-dimensional representation parameters, the in-view loss of the sample image is determined, specifically including:

[0027] The human pose parameters and human shape parameters contained in the two-dimensional representation parameters of the sample image are input into the human body model to determine the initial three-dimensional pose corresponding to the sample image and each three-dimensional joint point in the initial three-dimensional pose.

[0028] Based on the camera pose parameters included in the two-dimensional representation parameters of the sample image, the initial three-dimensional pose is projected into a two-dimensional pose diagram, and based on the camera pose parameters, each predicted two-dimensional joint of each three-dimensional joint in the two-dimensional pose diagram is determined.

[0029] For each predicted two-dimensional joint, a reference two-dimensional joint corresponding to the predicted two-dimensional joint is determined, and the probability that the predicted two-dimensional joint corresponds to the reference two-dimensional joint is determined as the matching probability of the predicted two-dimensional joint.

[0030] The first loss of the sample image is determined based on the distance between each predicted 2D joint and its corresponding baseline 2D joint, and the matching probability of each predicted 2D joint.

[0031] The second loss of the sample image is determined based on the distance between the human pose parameters included in the two-dimensional representation parameters of the sample image and the first reference pose parameters of the sample image, as well as the distance between the human shape parameters included in the two-dimensional representation parameters of the sample image and the first reference shape parameters of the sample image.

[0032] Based on the first loss and the second loss, determine the in-view loss of the sample image.

[0033] Optionally, the three-dimensional representation parameters include each camera pose parameter, each human body pose parameter, the second pose parameter, and the second shape parameter;

[0034] Determine the annotations between viewpoints, specifically including:

[0035] For each sample image, the sample image is input into a preset neural network to determine the corresponding reference two-dimensional joint points of the sample image;

[0036] The human pose parameters corresponding to each sample image included in the three-dimensional representation parameters, the second pose parameters included in the three-dimensional representation parameters, and the second shape parameters included in the three-dimensional representation parameters are determined, and respectively input into a preset optimization algorithm to obtain each reference pose parameter, the second reference pose parameter, and the second reference shape parameter.

[0037] Each reference 2D joint point, each reference pose parameter, the second reference pose parameter, and the second reference shape parameter corresponding to each sample image are used as the inter-view annotation.

[0038] Optionally, the inter-viewpoint loss is determined based on the inter-viewpoint annotations and the 3D representation parameters, specifically including:

[0039] The second pose parameter and the second shape parameter included in the three-dimensional representation parameters are input into the human body model to determine the initial three-dimensional pose and each three-dimensional joint point in the initial three-dimensional pose.

[0040] For each sample image, the initial three-dimensional pose is projected into a two-dimensional pose diagram based on the camera pose parameters corresponding to the sample image contained in the three-dimensional representation parameters. Based on the camera pose parameters corresponding to the sample image contained in the three-dimensional representation parameters, each predicted two-dimensional joint point in the two-dimensional pose diagram is determined.

[0041] For each predicted two-dimensional joint, a reference two-dimensional joint corresponding to the predicted two-dimensional joint is determined, and the probability of the two-dimensional joint corresponding to the reference two-dimensional joint is determined as the matching probability of the predicted two-dimensional joint.

[0042] The third loss of the interview module is determined based on the distance between each predicted 2D joint and its corresponding reference 2D joint, and the matching probability of each predicted 2D joint.

[0043] Determine the distance between the second posture parameter included in the three-dimensional representation parameters and the second reference posture parameter; determine the distance between the second shape parameter included in the three-dimensional representation parameters and the second reference shape parameter; determine the distance between each human posture parameter included in the three-dimensional representation parameters and its corresponding reference posture parameter;

[0044] Based on the determined distances, the fourth loss of the modules between the viewpoints is determined;

[0045] Based on the third loss and the fourth loss, the in-view loss of the sample image is determined.

[0046] Optionally, based on the posture parameters of each human body, a first posture parameter is determined; based on the shape parameters of each human body, a first shape parameter is determined, specifically including:

[0047] Determine the in-view loss of each module within the viewpoint, take the human pose parameter corresponding to the minimum in-view loss as the first pose parameter, and take the human shape parameter corresponding to the minimum in-view loss as the first shape parameter.

[0048] Optionally, each predicted two-dimensional joint in the two-dimensional pose diagram is determined, specifically including:

[0049] Determine each candidate 2D joint point in the 2D pose diagram for each 3D joint point;

[0050] For each candidate two-dimensional joint, a reference two-dimensional joint corresponding to the candidate two-dimensional joint is determined, and the probability that the candidate two-dimensional joint corresponds to the reference two-dimensional joint is determined, which is used as the matching probability of the candidate two-dimensional joint.

[0051] If the matching probability of the predicted two-dimensional joint is greater than a specified threshold, the candidate two-dimensional joint is used as the predicted two-dimensional joint. If the matching probability of the candidate two-dimensional joint is not greater than the specified threshold, the candidate two-dimensional joint is used as an invalid two-dimensional joint and is not used to predict two-dimensional joints to determine the first loss or the third loss.

[0052] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multi-view 3D human pose estimation method.

[0053] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described multi-view three-dimensional human pose estimation method.

[0054] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0055] In the multi-view 3D human pose estimation method provided in this specification, the two-dimensional representation parameters of each human image are first determined through the in-view module of the 3D human pose estimation model. Then, the three-dimensional representation parameters representing each human image are determined through the intermediate features and two-dimensional representation parameters of each human image. These three-dimensional representation parameters include at least a second shape parameter and a second pose parameter. The second shape parameter and the second pose parameter are input into a preset human model to obtain the estimated 3D human pose. In this way, feature fusion is performed on each human image in parametric form to determine the three-dimensional representation parameters representing each human image. Since the 3D human pose obtained through the human model is the 3D human pose influenced by both skeletal structure and muscle morphology, the estimated 3D human pose corresponding to each viewpoint image is more accurate. Attached Figure Description

[0056] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0057] Figure 1 This document provides a flowchart illustrating a multi-view 3D human pose estimation method.

[0058] Figure 2 A schematic diagram of the model structure of a three-dimensional human posture model provided in this specification;

[0059] Figure 3 This is a flowchart illustrating a training method for a three-dimensional human pose estimation model as described in this specification.

[0060] Figure 4 A schematic diagram of a multi-view three-dimensional human pose estimation device provided in this specification;

[0061] Figure 5 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0063] The multi-view 3D human pose estimation method provided in this manual is a parametric 3D pose estimation method based on a parametric human body model. In the parametric human body model, the 3D human body is represented by a set of parameters that determine the skeletal structure and muscle morphology of the human body. Non-parametric human body models, such as the PSM model, can only represent human pose under the influence of skeletal structure. Therefore, compared with non-parametric human body models, parametric human body models can accurately represent human pose under the combined influence of skeletal structure and muscle morphology, and the human pose represented by parametric human body models is more accurate.

[0064] Parametric human models include Skinned Multi-Person Linear (SMPL) models, Shape Completion and Animation of People (SCAPE) models, and SMPL-X models, an upgraded version of the SMPL model. The multi-view 3D human pose estimation method provided in this manual can be implemented based on any parametric human model. The following explanation uses the implementation method based on the SMPL model as an example.

[0065] The SMPL model allows for the representation of 3D human postures using a 3D skin. In the SMPL model, human posture parameters and human shape parameters control the deformation of the 3D skin to characterize different 3D human postures. The human posture parameters consist of 75 parameters representing the overall movement and pose of the human body and the relative angles of 24 joints, while the human shape parameters consist of 10 parameters representing the proportions of the human body, such as height, build, and head-to-body ratio. Using these human posture and shape parameters, the SMPL model can represent the 3D human posture influenced by both skeletal structure and muscle morphology.

[0066] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0067] Figure 1 The flowchart of a multi-view 3D human pose estimation method provided in this specification includes the following steps:

[0068] S100: Acquire human images from different perspectives and input each human image into the in-view module of the 3D human pose estimation model.

[0069] In this specification, the device used for multi-view 3D human pose estimation can be a server or an electronic device such as a desktop computer or laptop computer. For ease of description, the following description focuses on the server as the execution entity to illustrate the multi-view 3D human pose estimation method provided in this specification.

[0070] The server acquires human images from different perspectives and inputs each human image into the in-view module of the 3D human pose estimation model to convert each human image into a parametric representation and obtain the two-dimensional parametric representation of each human image.

[0071] It should be noted that the human body images acquired in the above steps can be RGB images, depth maps, pressure maps, or infrared thermograms, etc. This manual does not restrict the type of human body images acquired. The acquired human body images can be images of the same type or a combination of images of different types. This manual does not restrict the composition of the acquired human body images. All acquired human body images can be input into the 3D human pose estimation model together or sequentially. This manual does not restrict the input order of the human body images.

[0072] S102: For each human body image, the intermediate features of the human body image are obtained through the in-view module. Based on the intermediate features, the two-dimensional representation parameters of the human body image are determined. The two-dimensional representation parameters include camera pose parameters, human body posture parameters, and human body shape parameters.

[0073] The 3D human pose estimation model in this manual can be trained based on the Cross-Modality Attention Network (CMANet) framework. Figure 2 This is a schematic diagram of the structure of a three-dimensional human posture model provided in this specification. Figure 2 As shown, the three-dimensional human pose model includes an in-view module and an inter-view module. The in-view module consists of n layers of cascaded neural subnetworks, each containing an encoding layer and a regressor. The inter-view module contains a moving window attention layer (a hierarchical Transformer whose representation is computed with shifted windows, Swin Transformer) and a regressor.

[0074] Each layer of the intraview module can extract two-dimensional representation parameters of a human image. Thus, the cascaded multi-layer neural subnetworks enable the intraview module to extract two-dimensional representation parameters of human images acquired from multiple viewpoints, thereby obtaining the two-dimensional representation parameters of each viewpoint image.

[0075] For each human image, the server assigns the image to a neural subnetwork within the viewpoint module for extracting two-dimensional representation parameters. The intermediate features of the human image are obtained through the encoding layer of this subnetwork. These intermediate features are then input into the regressor of the same subnetwork, which outputs the two-dimensional representation parameters of the human image. These two-dimensional representation parameters include camera pose parameters, human pose parameters, and human shape parameters.

[0076] The two-dimensional representation parameter is the parametric representation of the human body image. The in-view module can convert each human body image into a parametric representation so that feature fusion can be performed on each image under the parametric representation.

[0077] Furthermore, within the in-view module, each human body image is converted into a parametric representation; that is, the in-view module outputs two-dimensional representation parameters of the human body image from each viewpoint. Therefore, each viewpoint human body image within the in-view module corresponds to a regressor. For each human body image, the two-dimensional representation parameters of that human body image are output through the regressor corresponding to that human body image.

[0078] S104: Determine the first posture parameter based on the posture parameters of each human body, and determine the first shape parameter based on the shape parameters of each human body.

[0079] The human pose and shape parameters in the two-dimensional representation parameters of the human image from each viewpoint can be used to determine a three-dimensional human pose using the SMPL model. In order to synthesize human images acquired from different viewpoints and obtain a more accurate three-dimensional human pose, the server needs to determine the first pose parameter based on each human pose parameter and the first shape parameter based on each human shape parameter.

[0080] There are several ways for the server to determine the first pose parameter and the first shape parameter. For example, the server can average the human pose parameters corresponding to each human image to determine the first pose parameter, and average the human shape parameters corresponding to each image to determine the first shape parameter. Alternatively, the server can apply a regression algorithm to the pose and shape parameters corresponding to each image to determine a first combination coefficient of the human pose parameters and a second combination coefficient of the human shape parameters corresponding to each human image. The server then weights the pose parameters corresponding to each human image based on the first combination coefficient to determine the first pose parameter, and weights the shape parameters corresponding to each human image based on the second combination coefficient to determine the first shape parameter. The regression algorithm can be linear regression, multinomial regression, ridge regression, etc.

[0081] Alternatively, the server can determine the in-view annotations for each human body image. For each human body image, it calculates the distance between the two-dimensional representation parameters of that human body image and the in-view annotations, using this distance as the in-view loss. The human pose parameter corresponding to the minimum in-view loss is taken as the first pose parameter, and the human shape parameter corresponding to the minimum in-view loss is taken as the first shape parameter. This specification does not restrict the specific method used to determine the first pose parameter and the first shape parameter.

[0082] In one or more embodiments of this specification, the server determines the first pose parameters and the first shape parameters based on the intra-view loss. It should be noted that the intra-view loss here is merely a metric for determining the first pose parameters and the first shape parameters and is not used to adjust the model parameters. During the training process of the intra-view module of this 3D human pose estimation model, the inter-view module is optimized based on the intra-view loss.

[0083] First, the server needs to determine the in-view annotations for each human body image.

[0084] Specifically, for each human body image, the server inputs the human body image into a preset neural network to determine the corresponding reference two-dimensional joints. The server determines the human posture parameters included in the two-dimensional representation parameters of the human body image and the human shape parameters included in the two-dimensional representation parameters of the sample image, inputs them into a preset optimization algorithm, and obtains the first reference posture parameters and the first reference shape parameters corresponding to the human body image. The reference two-dimensional joints, the first reference posture parameters, and the first reference shape parameters are used as the in-view annotations of the human body image.

[0085] Secondly, the server determines the intra-view loss of the human body image based on the intra-view annotations and the two-dimensional representation parameters of the human body image. The intra-view loss includes a first loss and a second loss.

[0086] The first loss can be determined as follows:

[0087] The server inputs the human pose parameters and human shape parameters included in the two-dimensional representation parameters of the human image into a preset human model to determine the initial three-dimensional pose corresponding to the human image and each three-dimensional joint in the initial three-dimensional pose. Based on the camera pose parameters included in the two-dimensional representation parameters of the human image, the initial three-dimensional pose is projected into a two-dimensional pose map, and based on the camera pose parameters, the predicted two-dimensional joints of each three-dimensional joint in the two-dimensional pose map are determined. That is, the camera pose parameters determine the initial three-dimensional pose and the projection direction of each three-dimensional joint.

[0088] For each predicted 2D joint, the server determines the corresponding reference 2D joint and the probability of the predicted 2D joint corresponding to the reference 2D joint, as the matching probability of the predicted 2D joint. Based on the first distance between each predicted 2D joint and its corresponding reference 2D joint, and the matching probability of each predicted 2D joint, the server determines the first loss of the human image.

[0089] Specifically, the server determines a first distance between the predicted 2D keypoint and the baseline 2D keypoint, and weights this first distance using the matching probability to determine the first part of the loss. The server sums the first part of the losses for each predicted 2D keypoint to determine the first loss for the sample image. The first loss can be determined according to the following formula:

[0090]

[0091] Where i represents the i-th human image, and M represents the number of predicted 2D keypoints. Let represent the matching probability of the m-th predicted two-dimensional joint point in the i-th human image. This represents the m-th predicted two-dimensional joint point corresponding to the i-th human image. This represents the reference 2D joint corresponding to the m-th predicted 2D joint of the i-th human image.

[0092] The second loss can be determined in the following manner:

[0093] The server determines the second loss of the sample image by summing the second and third distances between the human pose parameters included in the two-dimensional representation parameters of the human image and the first reference pose parameters of the human image, and the human shape parameters included in the two-dimensional representation parameters of the human image and the first reference shape parameters of the human image. Specifically, the second loss can be determined according to the following formula:

[0094]

[0095] Where, θ i Let represent the human pose parameters of the i-th human image. Let β represent the first reference pose parameter of the i-th human image. i This represents the human shape parameters of the i-th human image. This represents the first reference shape parameter of the i-th human body image.

[0096] After determining the first loss and the second loss, the server can sum the first loss and the second loss to determine the in-view loss corresponding to the image. To better balance the first loss and the second loss and make the in-view loss more accurate, the server can perform a weighted sum of the first loss and the second loss to determine the in-view loss. Specifically, the in-view loss can be determined according to the following formula:

[0097]

[0098] in, This represents the in-view loss for the i-th human image. α and β are weighting parameters that can be set as needed.

[0099] The first loss is the error between each predicted two-dimensional joint point and each reference two-dimensional joint point calculated based on the two-dimensional representation parameters of the human body image. The second loss is the error between the human body posture parameter and the first reference posture parameter in the two-dimensional representation parameters of the human body image, and between the human body shape parameter and the first reference shape parameter in the two-dimensional representation parameters of the human body image.

[0100] Since human pose parameters and human shape parameters can be used as inputs to the human body model to determine the three-dimensional human pose, the second loss is the error of the human body image in three-dimensional space. Since the first loss is the error of the human body image in two-dimensional space, the intra-view loss is the result obtained by combining the two-dimensional space error and the three-dimensional space error. The first pose parameters and the first shape parameters determined by the intra-view loss as the selection criterion can better characterize the human body image under each viewpoint.

[0101] Of course, when determining the first pose parameter and the first shape parameter, the server can also use either the first loss or the second loss as the selection criterion. That is, the human pose parameter and human shape parameter corresponding to the minimum first loss can be used as the first pose parameter and the first shape parameter, respectively. Alternatively, the human pose parameter and human shape parameter corresponding to the minimum second loss can be used as the first pose parameter and the first shape parameter, respectively.

[0102] In addition, to make the calculation of the first loss more accurate, the server can pre-set a specified threshold for the matching probability. First, the server determines the candidate two-dimensional joints in the two-dimensional pose graph for each three-dimensional joint. For each candidate two-dimensional joint, it determines the reference two-dimensional joint corresponding to the candidate two-dimensional joint and the probability of the candidate two-dimensional joint corresponding to the reference two-dimensional joint, which is used as the matching probability of the candidate two-dimensional joint.

[0103] Then, when calculating the first loss, the server determines whether the matching probability of the candidate 2D joint is greater than a specified threshold. If the matching probability of the candidate 2D joint is greater than the specified threshold, the candidate 2D joint is used as a predicted 2D joint to determine the first loss. If the matching probability of the candidate 2D joint is not greater than the specified threshold, the candidate 2D joint is considered an invalid 2D joint and is not used as a predicted 2D joint to determine the first loss.

[0104] It should be noted that the preset neural networks used in the above steps can be high-resolution networks (HRNet), residual neural networks (ResNet), simple vision transformer baselines for human pose estimation (ViTPose), etc., and the preset optimization algorithms can be SMPLify based on the SMPL model, SMPLify-X based on the SMPL-X model, etc. This manual does not limit the specific neural networks and optimization algorithms used in the application.

[0105] It should be further noted that since the method provided in this manual is illustrated using the parameterization method based on the SMPL model as an example, the human body model in the above steps refers to the SMPL model, and correspondingly, the optimization algorithm should be the SMPLify optimization algorithm based on the SMPL model. When the method provided in this manual is implemented using other parameterization models, the human body model is the other parameterization model used, and the optimization algorithm is also the optimization algorithm corresponding to the other parameterization model used.

[0106] S106: Input the intermediate features of each human body image, the camera pose parameters of each two-dimensional representation parameter, the human pose parameters of each two-dimensional representation parameter, the first pose parameter, and the first shape parameter into the viewpoint module of the three-dimensional human pose estimation model, perform feature fusion, and determine the three-dimensional representation parameters representing each human body image. The three-dimensional representation parameters include at least the second pose parameter and the second shape parameter.

[0107] like Figure 2 As shown, the view module contains a Swing Transformer layer and a regressor.

[0108] First, the server inputs the intermediate features of each human image, the camera pose parameters of each two-dimensional representation parameter, the human pose parameters of each two-dimensional representation parameter, the first pose parameter, and the first shape parameter into the viewpoint module. Through the SwingTransformer layer, feature fusion is performed to obtain fused features.

[0109] Then, the server inputs the fused features into a regressor, which outputs three-dimensional representation parameters that characterize each human body image. These three-dimensional representation parameters are a parameter representation of the three-dimensional space in which each human body image is located. The three-dimensional representation parameters include at least a second shape parameter and a second pose parameter.

[0110] In the interview module, each human body image undergoes feature fusion in parametric representation to obtain three-dimensional representation parameters that characterize each human body image. The second pose parameter and the second shape parameter included in these three-dimensional representation parameters are the results extracted based on the fused features. Therefore, the three-dimensional human pose determined by the second pose parameter and the second shape parameter integrates the features of human body images from various viewpoints and is a more accurate result than the three-dimensional human pose determined by the first pose parameter and the first shape parameter.

[0111] S108: Input the second posture parameters and the second shape parameters into a preset human body model to obtain the estimated three-dimensional human body posture.

[0112] The second pose parameter and second shape parameter included in the 3D representation parameters can be used to determine the 3D human pose. The server inputs the second pose parameter and second shape parameter, which are included in the 3D representation parameters output by the viewpoint module, into a preset human model to obtain the estimated 3D human pose.

[0113] In the multi-view 3D human pose estimation method provided in this specification, the two-dimensional representation parameters of each human image are first determined through the in-view module of the 3D human pose estimation model. Then, the three-dimensional representation parameters representing each human image are determined through the intermediate features and two-dimensional representation parameters of each human image. These three-dimensional representation parameters include at least a second shape parameter and a second pose parameter. The second shape parameter and the second pose parameter are input into a preset human model to obtain the estimated 3D human pose. In this way, feature fusion is performed on each human image in a parametric representation form to determine the 3D representation parameters representing each human image. Since the 3D human pose obtained through the human model is the 3D human pose influenced by both skeletal structure and muscle morphology, the estimated 3D human pose corresponding to each viewpoint image is more accurate.

[0114] This manual also provides a training method for the above-mentioned three-dimensional human pose estimation model. Figure 3 This is a flowchart illustrating a training method for a 3D human pose estimation model as described in this specification, specifically including the following steps:

[0115] S200: Acquire sample images of the same human body pose from different perspectives.

[0116] Similarly, the training method of the three-dimensional human pose estimation model provided in this manual is explained with the server as the execution subject.

[0117] The server acquires sample images of the same human pose from different perspectives for training a 3D human pose estimation model. In addition, in order to make the 3D human pose estimation model trained from each sample image more accurate in estimating the 3D human pose, each sample image should be acquired at the same time.

[0118] S202: Input each sample image into the in-view module of the three-dimensional human pose estimation model to be trained, and obtain the two-dimensional representation parameters corresponding to each sample image output by the in-view module.

[0119] For specific methods of determining the two-dimensional characterization parameters, please refer to the description of the corresponding content in S102 above, which will not be repeated here.

[0120] Similar to step S100 above, this manual does not restrict the types of images, composition, or input order of the acquired human body images.

[0121] S204: Determine the in-view annotations for each sample image. For each sample image, determine the in-view loss based on the in-view annotations and the two-dimensional representation parameters, and train the in-view module based on the in-view loss.

[0122] For specific methods of determining the annotation within the viewpoint, please refer to the corresponding content in S104 above. This manual will not elaborate on them here.

[0123] For the specific method of determining the in-view loss of the sample image, please refer to the description of the corresponding content in S104 above. This specification will not repeat it here.

[0124] After the server determines the in-view loss for each sample image, it trains the in-view module with the goal of minimizing the in-view loss. The model parameters of the in-view module are adjusted until the loss function for the in-view loss corresponding to each sample image converges. The model parameters of the in-view module are the parameters of each neural subnetwork contained within the in-view module, specifically the parameters of the coding layer and regressor in each neural subnetwork.

[0125] It should be noted that the neural subnetworks that make up the viewpoint module can use the same network structure and parameters, or different network structures and parameters, or the same network structure with different parameters. This specification does not impose any restrictions on this.

[0126] Correspondingly, if each neural subnetwork uses the same network structure and parameters, when adjusting the parameters of the intraview module based on the intraview loss, the parameters of each neural subnetwork should be adjusted synchronously to ensure that the parameters of each neural subnetwork remain consistent. If each neural subnetwork uses different network structures and parameters, or uses the same network structure but different parameters, then the parameters of each neural subnetwork can be adjusted separately based on the intraview loss corresponding to each sample image.

[0127] S206: When the in-view module training is completed, input each sample image into the trained in-view module to determine each intermediate feature obtained by the trained in-view module and each output two-dimensional representation parameter.

[0128] When the loss functions for the in-view loss corresponding to each sample image converge, the training of the in-view module is complete, and the server fixes the model parameters of the in-view module.

[0129] Then, the server inputs each sample image into the trained in-view module. Through the encoding layer of the trained in-view module, the intermediate features of each sample image are obtained. The intermediate features of each sample image are then input into the regressor corresponding to each sample image to obtain the two-dimensional representation parameters of each sample image output by each regressor.

[0130] Two-dimensional representation parameters are the parametric representation of the human body image. The in-view module can convert each human body image into a parametric representation, so as to perform feature fusion on each image under the parametric representation.

[0131] S208: Determine the first posture parameter based on the posture parameters of each human body, and determine the first shape parameter based on the shape parameters of each human body.

[0132] For the specific methods of determining the first attitude parameter and the first shape parameter, please refer to the description of the corresponding content in S104 above. This specification will not repeat them here.

[0133] S210: Input the intermediate features determined by the in-view module, the camera pose parameters included in each two-dimensional representation parameter, the human pose parameters included in each two-dimensional representation parameter, the first pose parameter, and the first shape parameter into the inter-view module to be trained, perform feature fusion, and obtain the three-dimensional representation parameters output by the inter-view module to be trained.

[0134] The server will use the intermediate features determined by the trained intra-view module, the camera pose parameters included in each two-dimensional representation parameter, the human pose parameters included in each two-dimensional representation parameter, the first pose parameter, and the first shape parameter as inputs to the inter-view module to be trained, and start training the inter-view module.

[0135] The viewpoint module comprises a Swing Transformer layer and a regressor. First, through the Swing Transformer layer, the server fuses the intermediate features input to the viewpoint module, the camera pose parameters included in each 2D representation parameter, the human pose parameters included in each 2D representation parameter, the first pose parameter, and the first shape parameter, obtaining fused features. Then, the server inputs these fused features into the regressor, which outputs 3D representation parameters that characterize each sample image. These 3D representation parameters are a parametric representation of the 3D space in which each sample image exists.

[0136] In the interview module, the parameter representation of each sample image is determined by the parameter representation of each sample image. Since each sample image corresponds to a three-dimensional space, the interview module outputs only one three-dimensional representation parameter, and there is only one regressor used for output in the interview module.

[0137] This specification does not restrict the method of feature fusion. For example, the server can directly fuse the intermediate features input from the viewpoint modules, the camera pose parameters included in each two-dimensional representation parameter, the human pose parameters included in each two-dimensional representation parameter, the first pose parameter, and the first shape parameter to obtain fused features. Alternatively, the server can first unify the dimensions of the intermediate features input from the viewpoint modules, the camera pose parameters included in each two-dimensional representation parameter, the human pose parameters included in each two-dimensional representation parameter, the first pose parameter, and the first shape parameter, and convert them into high-dimensional transition features, and then perform feature fusion on the transition features. Alternatively, the server first takes the intermediate features, camera pose parameters, human pose parameters, first pose parameters, and first shape parameters input from the viewpoint module, and uses a regressor to obtain the initial intermediate features, initial camera pose parameters, initial human pose parameters, initial first pose parameters, and initial first shape parameters, respectively. Then, it performs feature fusion on the initial intermediate features, initial camera pose parameters, initial human pose parameters, initial first pose parameters, and initial first shape parameters. The resulting fused features are the output of the viewpoint module.

[0138] This specification does not restrict the order of feature fusion. The server can simultaneously perform feature fusion on each intermediate feature input from the inter-view module, the camera pose parameters included in each 2D representation parameter, the human pose parameters included in each 2D representation parameter, the first pose parameter, and the first shape parameter. Alternatively, it can sequentially perform feature fusion on each intermediate feature input from the inter-view module, the camera pose parameters included in each 2D representation parameter, the human pose parameters included in each 2D representation parameter, the first pose parameter, and the first shape parameter.

[0139] S212: Determine the inter-view annotations, determine the inter-view loss based on the inter-view annotations and the 3D representation parameters, and train the inter-view module based on the inter-view loss.

[0140] The three-dimensional representation parameters output by the viewpoint module include the pose parameters of each camera, the pose parameters of each human body, the second pose parameters, and the second shape parameters.

[0141] For each sample image, the server inputs it into a pre-defined neural network to determine the baseline two-dimensional joints of the sample image. The server then inputs the human pose parameters corresponding to the sample image, which are included in the three-dimensional representation parameters, into a pre-defined optimization algorithm to obtain the baseline pose parameters corresponding to the sample image.

[0142] The server inputs the second pose parameters, which are included in the 3D representation parameters, into a preset optimization algorithm to obtain the second reference pose parameters. It then inputs the second shape parameters, which are also included in the 3D representation parameters, into a preset optimization algorithm to obtain the second reference shape parameters.

[0143] Each reference 2D joint point, reference pose parameter, second reference pose parameter, and second reference shape parameter corresponding to each sample image are used as inter-view annotations to calculate the inter-view loss of the pre-trained inter-view modules.

[0144] The server determines the inter-viewpoint loss using the following steps:

[0145] First, for each training round, the server determines the three-dimensional representation parameters output by the perspective module in that training round, and inputs the second pose parameter and the second shape parameter contained in the three-dimensional representation parameters into a preset human body model to obtain the initial three-dimensional pose output by the human body model, and determines each three-dimensional joint in the initial three-dimensional pose through the human body model.

[0146] For each sample image, the server projects the initial three-dimensional pose into a two-dimensional pose map based on the camera pose parameters of the sample image contained in the three-dimensional representation parameters, and determines the predicted two-dimensional joints of each three-dimensional joint in the two-dimensional pose map based on the camera pose parameters.

[0147] For each predicted 2D joint, the server determines a reference 2D joint corresponding to that predicted 2D joint, and determines the probability of the correspondence between the predicted 2D joint and the reference 2D joint, as the matching probability of the predicted 2D joint. Based on the fourth distance between each predicted 2D joint and its corresponding reference 2D joint, and the matching probability of each predicted 2D joint, the server determines the third loss of the inter-view module.

[0148] Specifically, the server determines the fourth distance between the predicted 2D keypoint and the baseline 2D keypoint, and weights this fourth distance with the matching probability to determine the second part of the loss. The server sums the second part of the loss for each predicted 2D keypoint to determine the third part of the loss for the sample image. The server sums the third part of the loss for each sample image to determine the third loss for the inter-view module. The third loss can be determined according to the following formula:

[0149]

[0150] Where N represents the number of sample images.

[0151] Secondly, the server determines the fifth distance between the second pose parameter and the second reference pose parameter included in the three-dimensional representation parameter, the sixth distance between the second shape parameter and the second reference shape parameter included in the three-dimensional representation parameter, and the seventh distance between each human pose parameter included in the three-dimensional representation parameter and its corresponding reference pose parameter. Based on the fifth distance, the sixth distance and the seventh distance, the server determines the fourth loss of the module between viewpoints.

[0152] Specifically, the server sums the seventh distances corresponding to each pose parameter included in the 3D parameter representation to determine the fourth part of the inter-view module loss. Then, it sums the fifth distance, the sixth distance, and the fourth part of the loss to determine the fourth loss of the inter-view module. The fourth loss can be determined according to the following formula:

[0153]

[0154] Where, θ b Represents the second attitude parameter, (θ) b ) opt β represents the second reference attitude parameter, and β represents the second shape parameter. opt Indicates the second reference shape parameter, This represents the human pose parameters of the i-th sample image included in the 3D representation parameters. The 3D representation parameters include the baseline pose parameters of the i-th sample image.

[0155] Then, the server sums the third and fourth losses to determine the inter-view loss. To better balance the third and fourth losses and make the inter-view loss more accurate, the server can perform a weighted sum of the third and fourth losses to determine the inter-view loss. Specifically, the inter-view loss can be determined according to the following formula:

[0156] L IEV =λL3+γL4

[0157] Here, λ and γ are weighting parameters that can be set as needed.

[0158] Finally, the server determines the inter-view loss for this training round, and adjusts the model parameters of the inter-view module with the minimum inter-view loss as the optimization objective, training the inter-view module until the loss function converges. The model parameters of the inter-view module include the parameters of the Swing Transformer layer and the parameters of the regressor.

[0159] Similarly, the third loss is the error of each sample image in two-dimensional space, and the fourth loss is the error of each sample image in three-dimensional space. Therefore, the inter-viewpoint loss is obtained by combining the errors in two-dimensional space and three-dimensional space. The inter-viewpoint module trained based on the inter-viewpoint loss can more accurately obtain the parameter representation of each sample image in three-dimensional space, that is, it can better obtain the three-dimensional representation parameters that characterize each sample image. As a result, the estimated three-dimensional human pose based on the second pose parameter and the second shape parameter included in the three-dimensional representation parameters is also more accurate.

[0160] Similar to step S104, to make the calculation of the third loss more accurate, the server can pre-set a specified threshold for the matching probability. First, the server determines the candidate two-dimensional joints in the two-dimensional pose graph for each three-dimensional joint. For each candidate two-dimensional joint, the server determines the reference two-dimensional joint corresponding to the candidate two-dimensional joint and the probability of the candidate two-dimensional joint corresponding to the reference two-dimensional joint, which is used as the matching probability of the candidate two-dimensional joint.

[0161] Then, when calculating the third loss, the server determines whether the matching probability of the candidate 2D joint is greater than a specified threshold. If the matching probability of the candidate 2D joint is greater than the specified threshold, the candidate 2D joint is used as a predicted 2D joint to determine the third loss. If the matching probability of the candidate 2D joint is not greater than the specified threshold, the candidate 2D joint is considered an invalid 2D joint and is not used as a predicted 2D joint to determine the third loss.

[0162] Because the technical solution in this specification uses a parametric human body model to determine the estimated three-dimensional human body posture, the final estimated three-dimensional human body posture is a skin structure represented by a three-dimensional mesh, rather than a skeleton. This skin structure is a three-dimensional human body posture influenced by both the skeleton structure and muscle morphology. Therefore, the estimated three-dimensional human body posture corresponding to each viewpoint image is more accurate.

[0163] In the training method of the 3D human pose estimation model provided in this manual, both intra-view and inter-view annotations are obtained through a pre-defined neural network and optimization algorithm, employing a self-supervised training approach. Alternatively, existing labeled datasets such as Human3.6M, HumanEva-I, or MPI-INF-3DHP can be used for supervised or weakly supervised training.

[0164] If the existing labeled dataset contains two-dimensional joint annotation information, then when using the existing labeled dataset as training data, the reference two-dimensional joints in the intra-view annotation and inter-view annotation can be directly obtained from the dataset. The first reference pose parameter and the first reference shape parameter in the intra-view annotation are obtained as described in step S104, and the reference pose parameters, the second reference pose parameter and the second reference shape parameter in the inter-view annotation are obtained as described in step S212.

[0165] To enhance the robustness of the 3D human pose estimation model and enable it to achieve accurate 3D pose estimation even for images with occluded parts, a certain number of images with occluded parts can be added to the sample data for training the 3D human pose estimation model.

[0166] The multi-view 3D human pose estimation method and the training method of the 3D human pose estimation model described above are all illustrated using the implementation method based on the SMPL model as an example. Therefore, the human models mentioned above are all SMPL models. In fact, the multi-view 3D human pose estimation method provided in this specification can be implemented based on any parameterized human model. When using different parameterized human models to implement the multi-view 3D human pose estimation method and the training method of the 3D human pose estimation model provided in this specification, it is only necessary to extract the two-dimensional and three-dimensional representation parameters corresponding to the parameterized human model used, according to the actual situation.

[0167] For example, in the SCAPE model, a 3D human body is represented by human shape parameters, human pose parameters, human expression parameters, and spatial parameters. Therefore, if the SCAPE model is used to implement the multi-view 3D human pose estimation method provided in this specification, in the intra-view module, the human shape parameters, human pose parameters, human expression parameters, and spatial parameters of the image from each viewpoint need to be extracted as 2D representation parameters for each viewpoint image. Then, in the inter-view module, feature fusion is performed on each 2D representation parameter to obtain the 3D representation parameters. Finally, the obtained 3D representation parameters are input into the SCAPE model to obtain the estimated 3D human pose.

[0168] The above describes the multi-view 3D human pose estimation method provided in this manual. Based on the same approach, this manual also provides a corresponding multi-view 3D human pose estimation device, such as... Figure 4 As shown.

[0169] Figure 4 This specification provides a schematic diagram of a multi-view 3D human pose estimation device, which specifically includes:

[0170] The acquisition module 300 acquires human images collected from different perspectives and inputs each human image into the in-view module of the 3D human pose estimation model.

[0171] The two-dimensional representation parameter determination module 302, for each human body image, obtains the intermediate features of the human body image through the viewpoint module, and determines the two-dimensional representation parameters corresponding to the human body image based on the intermediate features. The two-dimensional representation parameters include camera pose parameters, human body posture parameters, and human body shape parameters.

[0172] The filtering module 304 determines the first posture parameter based on the posture parameters of each human body and determines the first shape parameter based on the shape parameters of each human body.

[0173] The three-dimensional representation parameter determination module 306 inputs the intermediate features of each human body image, the camera pose parameters of each two-dimensional representation parameter, the human body pose parameters of each two-dimensional representation parameter, the first pose parameter and the first shape parameter into the viewpoint module of the three-dimensional human body pose estimation model, performs feature fusion, and determines the three-dimensional representation parameters representing each human body image. The three-dimensional representation parameters include at least the second pose parameter and the second shape parameter.

[0174] The pose estimation module 308 inputs the second pose parameters and the second shape parameters into a preset human body model to obtain the estimated three-dimensional human pose.

[0175] Optionally, the apparatus further includes: a training module 310, configured to acquire sample images of the same human pose from different viewpoints, input each sample image into the in-view module of the three-dimensional human pose estimation model to be trained, obtain the two-dimensional representation parameters corresponding to each sample image output by the in-view module, determine the in-view annotation of each sample image, determine the in-view loss of each sample image based on the in-view annotation and the two-dimensional representation parameters, train the in-view module based on the in-view loss, and when the in-view module is trained, input each sample image into the trained in-view module to determine the trained in-view module. The intermediate features and output 2D representation parameters obtained by the intra-angle module are used to determine the first pose parameter based on the human pose parameters and the first shape parameter based on the human shape parameters. The intermediate features determined by the intra-view module, the camera pose parameters included in the 2D representation parameters, the human pose parameters included in the 2D representation parameters, the first pose parameter, and the first shape parameter are input into the inter-view module to be trained for feature fusion to obtain the 3D representation parameters output by the inter-view module to be trained. The inter-view annotation is determined, and the inter-view loss is determined based on the inter-view annotation and the 3D representation parameters. The inter-view module is then trained based on the inter-view loss.

[0176] Optionally, the training module 310 is specifically configured to input the human pose parameters and human shape parameters contained in the two-dimensional representation parameters of the sample image into the human model, determine the initial three-dimensional pose corresponding to the sample image and each three-dimensional joint in the initial three-dimensional pose, project the initial three-dimensional pose into a two-dimensional pose map according to the camera pose parameters contained in the two-dimensional representation parameters of the sample image, and determine each predicted two-dimensional joint of each three-dimensional joint in the two-dimensional pose map according to the camera pose parameters, and for each predicted two-dimensional joint, determine the reference two-dimensional joint corresponding to the predicted two-dimensional joint, and determine... The probability of a predicted 2D joint point corresponding to a reference 2D joint point is defined as the matching probability of the predicted 2D joint point. Based on the distance between each predicted 2D joint point and its corresponding reference 2D joint point, and the matching probability of each predicted 2D joint point, a first loss of the sample image is determined. Based on the distance between the human pose parameters included in the 2D representation parameters of the sample image and the first reference pose parameters of the sample image, and the distance between the human shape parameters included in the 2D representation parameters of the sample image and the first reference shape parameters of the sample image, a second loss of the sample image is determined. Based on the first loss and the second loss, the in-view loss of the sample image is determined.

[0177] Optionally, the three-dimensional representation parameters include camera pose parameters, human pose parameters, the second pose parameter, and the second shape parameter. The training module 310 is specifically used to input each sample image into a preset neural network, determine each reference two-dimensional joint point corresponding to the sample image, determine the human pose parameters, the second pose parameters, and the second shape parameters included in the three-dimensional representation parameters, and input them into a preset optimization algorithm to obtain each reference pose parameter, the second reference pose parameter, and the second reference shape parameter. The reference two-dimensional joint points, reference pose parameters, the second reference pose parameter, and the second reference shape parameter corresponding to each sample image are used as inter-viewpoint annotations.

[0178] Optionally, the training module 310 is specifically configured to input the second pose parameters and the second shape parameters included in the three-dimensional representation parameters into the human body model, determine the initial three-dimensional pose and each three-dimensional joint in the initial three-dimensional pose, and for each sample image, project the initial three-dimensional pose into a two-dimensional pose diagram according to the camera pose parameters corresponding to the sample image included in the three-dimensional representation parameters, and determine each predicted two-dimensional joint in the two-dimensional pose diagram according to the camera pose parameters corresponding to the sample image included in the three-dimensional representation parameters, and for each predicted two-dimensional joint, determine the reference two-dimensional joint corresponding to the predicted two-dimensional joint, and determine the relationship between the two-dimensional joint and the... The probability between reference 2D joints is used as the matching probability of the predicted 2D joint. Based on the distance between each predicted 2D joint and its corresponding reference 2D joint, and the matching probability of each predicted 2D joint, the third loss of the viewpoint module is determined. The distance between the second pose parameter included in the 3D representation parameters and the second reference pose parameter is determined. The distance between the second shape parameter included in the 3D representation parameters and the second reference shape parameter is determined. The distance between each human pose parameter included in the 3D representation parameters and its corresponding reference pose parameter is determined. Based on the determined distances, the fourth loss of the viewpoint module is determined. Based on the third loss and the fourth loss, the intra-viewpoint loss of the sample image is determined.

[0179] Optionally, the filtering module 304 is specifically used to determine the in-view loss of each in-view module, take the human posture parameter corresponding to the minimum in-view loss as the first posture parameter, and take the human shape parameter corresponding to the minimum in-view loss as the first shape parameter.

[0180] Optionally, the training module 310 is specifically used to determine each candidate two-dimensional joint in the two-dimensional pose diagram for each three-dimensional joint. For each candidate two-dimensional joint, a reference two-dimensional joint corresponding to the candidate two-dimensional joint is determined, and the probability of the candidate two-dimensional joint corresponding to the reference two-dimensional joint is determined as the matching probability of the candidate two-dimensional joint. If the matching probability of the predicted two-dimensional joint is greater than a specified threshold, the candidate two-dimensional joint is used as the predicted two-dimensional joint. If the matching probability of the candidate two-dimensional joint is not greater than the specified threshold, the candidate two-dimensional joint is used as an invalid two-dimensional joint and is not used to predict two-dimensional joints to determine the first loss or the third loss.

[0181] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided method is a multi-view 3D human pose estimation method.

[0182] This instruction manual also provides Figure 5 The diagram shows a schematic structural representation of the electronic device. Figure 5 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The multi-view 3D human pose estimation method is described above. Of course, besides software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0183] Improvements in a technology can be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology can now be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement in methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog are the most commonly used. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0184] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0185] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0186] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0187] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0188] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0189] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0190] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0191] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0192] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0193] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0194] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0195] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0196] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0197] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0198] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.

Claims

1. A multi-view 3D human pose estimation method, characterized in that, include: Acquire human images from different perspectives and input each human image into the in-view module of the 3D human pose estimation model; For each human body image, the intermediate features of the human body image are obtained through the in-view module. Based on the intermediate features, the corresponding two-dimensional representation parameters of the human body image are determined. The two-dimensional representation parameters include camera pose parameters, human body pose parameters, and human body shape parameters. The human body pose parameters and the human body shape parameters are the parameters of a preset parametric human body model. Based on the posture parameters of each human body, determine the first posture parameters; based on the shape parameters of each human body, determine the first shape parameters. The intermediate features of each human body image, the camera pose parameters of each two-dimensional representation parameter, the human pose parameters of each two-dimensional representation parameter, the first pose parameter and the first shape parameter are input into the viewpoint module of the three-dimensional human pose estimation model to perform feature fusion and determine the three-dimensional representation parameters representing each human body image. The three-dimensional representation parameters include at least the second pose parameter and the second shape parameter. The second pose parameter and the second shape parameter are input into the parameterized human body model to obtain the estimated three-dimensional human body pose.

2. The method as described in claim 1, characterized in that, The three-dimensional human pose estimation model is trained using the following method, wherein: Acquire sample images of the same human body in different perspectives; Each sample image is input into the in-view module of the three-dimensional human pose estimation model to be trained, and the two-dimensional representation parameters corresponding to each sample image output by the in-view module are obtained. The in-view annotations of each sample image are determined. For each sample image, the in-view loss of the sample image is determined based on the in-view annotations of the sample image and the two-dimensional representation parameters. The in-view module is then trained based on the in-view loss. When the in-view module is trained, each sample image is input into the trained in-view module to determine each intermediate feature obtained by the trained in-view module and each two-dimensional representation parameter output. Based on the posture parameters of each human body, determine the first posture parameters; based on the shape parameters of each human body, determine the first shape parameters. The intermediate features determined by the in-view module, the camera pose parameters included in each two-dimensional representation parameter, the human pose parameters included in each two-dimensional representation parameter, the first pose parameter, and the first shape parameter are input into the inter-view module to be trained for feature fusion to obtain the three-dimensional representation parameters output by the inter-view module to be trained. Determine the inter-view annotations, determine the inter-view loss based on the inter-view annotations and the 3D representation parameters, and train the inter-view module based on the inter-view loss.

3. The method as described in claim 2, characterized in that, Determine the in-view annotations for each sample image, specifically including: For each sample image, the sample image is input into a preset neural network to determine the corresponding reference two-dimensional joint points of the sample image; The human pose parameters and human shape parameters included in the two-dimensional representation parameters of the sample image are determined, and the preset optimization algorithm is input to obtain the first reference pose parameters and the first reference shape parameters corresponding to the image. Each reference 2D joint, the first reference pose parameter, and the first reference shape parameter are used as the in-view annotation of the sample image.

4. The method as described in claim 3, characterized in that, Based on the in-view annotations of the sample image and the two-dimensional representation parameters, the in-view loss of the sample image is determined, specifically including: The human pose parameters and human shape parameters contained in the two-dimensional representation parameters of the sample image are input into the parameterized human body model to determine the initial three-dimensional pose corresponding to the sample image and each three-dimensional joint in the initial three-dimensional pose. Based on the camera pose parameters included in the two-dimensional representation parameters of the sample image, the initial three-dimensional pose is projected into a two-dimensional pose diagram, and based on the camera pose parameters, each predicted two-dimensional joint of each three-dimensional joint in the two-dimensional pose diagram is determined. For each predicted two-dimensional joint, a reference two-dimensional joint corresponding to the predicted two-dimensional joint is determined, and the probability that the predicted two-dimensional joint corresponds to the reference two-dimensional joint is determined as the matching probability of the predicted two-dimensional joint. The first loss of the sample image is determined based on the distance between each predicted 2D joint and its corresponding baseline 2D joint, and the matching probability of each predicted 2D joint. The second loss of the sample image is determined based on the distance between the human pose parameters included in the two-dimensional representation parameters of the sample image and the first reference pose parameters of the sample image, as well as the distance between the human shape parameters included in the two-dimensional representation parameters of the sample image and the first reference shape parameters of the sample image. Based on the first loss and the second loss, determine the in-view loss of the sample image.

5. The method as described in claim 2, characterized in that, The three-dimensional representation parameters include the pose parameters of each camera, the pose parameters of each human body, the second pose parameters, and the second shape parameters; Determine the annotations between viewpoints, specifically including: For each sample image, the sample image is input into a preset neural network to determine the corresponding reference two-dimensional joint points of the sample image; The human pose parameters corresponding to each sample image included in the three-dimensional representation parameters, the second pose parameters included in the three-dimensional representation parameters, and the second shape parameters included in the three-dimensional representation parameters are determined, and respectively input into a preset optimization algorithm to obtain each reference pose parameter, the second reference pose parameter, and the second reference shape parameter. Each reference 2D joint point, each reference pose parameter, the second reference pose parameter, and the second reference shape parameter corresponding to each sample image are used as the inter-view annotation.

6. The method as described in claim 5, characterized in that, Based on the inter-viewpoint annotations and the 3D representation parameters, the inter-viewpoint loss is determined, specifically including: The second pose parameter and the second shape parameter included in the three-dimensional representation parameters are input into the parameterized human body model to determine the initial three-dimensional pose and each three-dimensional joint point in the initial three-dimensional pose. For each sample image, the initial three-dimensional pose is projected into a two-dimensional pose diagram based on the camera pose parameters corresponding to the sample image contained in the three-dimensional representation parameters. Based on the camera pose parameters corresponding to the sample image contained in the three-dimensional representation parameters, each predicted two-dimensional joint point in the two-dimensional pose diagram is determined. For each predicted two-dimensional joint, a reference two-dimensional joint corresponding to the predicted two-dimensional joint is determined, and the probability of the two-dimensional joint corresponding to the reference two-dimensional joint is determined as the matching probability of the predicted two-dimensional joint. The third loss of the interview module is determined based on the distance between each predicted 2D joint and its corresponding reference 2D joint, and the matching probability of each predicted 2D joint. Determine the distance between the second posture parameter included in the three-dimensional representation parameters and the second reference posture parameter; determine the distance between the second shape parameter included in the three-dimensional representation parameters and the second reference shape parameter; determine the distance between each human posture parameter included in the three-dimensional representation parameters and its corresponding reference posture parameter; Based on the determined distances, the fourth loss of the modules between the viewpoints is determined; Based on the third loss and the fourth loss, the interview loss of the sample image is determined.

7. The method as described in claim 1 or 2, characterized in that, Based on the posture parameters of each individual human body, the first posture parameters are determined; based on the shape parameters of each individual human body, the first shape parameters are determined, specifically including: Determine the in-view loss of each module within the viewpoint, take the human pose parameter corresponding to the minimum in-view loss as the first pose parameter, and take the human shape parameter corresponding to the minimum in-view loss as the first shape parameter.

8. The method as described in claim 6, characterized in that, Determining each predicted 2D joint in the 2D pose diagram for each 3D joint specifically includes: Determine each candidate 2D joint point in the 2D pose diagram for each 3D joint point; For each candidate two-dimensional joint, a reference two-dimensional joint corresponding to the candidate two-dimensional joint is determined, and the probability that the candidate two-dimensional joint corresponds to the reference two-dimensional joint is determined, which is used as the matching probability of the candidate two-dimensional joint. If the matching probability of the candidate two-dimensional joint is greater than a specified threshold, the candidate two-dimensional joint is used as the predicted two-dimensional joint. If the matching probability of the candidate two-dimensional joint is not greater than the specified threshold, the candidate two-dimensional joint is used as an invalid two-dimensional joint and is not used to predict two-dimensional joints to determine the third loss.

9. The method as described in claim 4, characterized in that, Determining each predicted 2D joint in the 2D pose diagram for each 3D joint specifically includes: Determine each candidate 2D joint point in the 2D pose diagram for each 3D joint point; For each candidate two-dimensional joint, a reference two-dimensional joint corresponding to the candidate two-dimensional joint is determined, and the probability that the candidate two-dimensional joint corresponds to the reference two-dimensional joint is determined, which is used as the matching probability of the candidate two-dimensional joint. If the matching probability of the candidate two-dimensional joint is greater than a specified threshold, the candidate two-dimensional joint is used as the predicted two-dimensional joint. If the matching probability of the candidate two-dimensional joint is not greater than the specified threshold, the candidate two-dimensional joint is used as an invalid two-dimensional joint and is not used to predict two-dimensional joints to determine the first loss.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 9.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 9.

Images