Single image three-dimensional human body shape estimation method and application

Abstract

The application discloses a single-image three-dimensional human body shape estimation method, comprising the following steps: estimating a three-dimensional human body shape of a human body region contained in an input image in a non-clothed state by a three-dimensional parameterized model, and calculating error information between the obtained estimation result and a real result to describe a first loss function; reconstructing a three-dimensional reconstruction model of the human body region in a clothed state from the input image under the guidance of the estimation result and the real result, and calculating error information between the three-dimensional reconstruction models respectively guided by the two results to describe a second loss function; inputting the first loss function into the three-dimensional reconstruction model and feeding back the second loss function to the estimated three-dimensional parameterized model for iteration until the model converges, so as to obtain an optimized three-dimensional human body shape estimation result. The method can solve the problem that the existing three-dimensional human body shape estimation method is disturbed by clothes covering the human body, so that a large error occurs in shape estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer graphics and computer vision, and more particularly, to a single-image three-dimensional human body shape estimation method, a single-image three-dimensional human body shape estimation system, an electronic device and a computer readable storage medium. BACKGROUND

[0002] Three-dimensional human body shape estimation refers to estimating the three-dimensional shape and pose of a human body from a visible light image. Three-dimensional human body shape estimation has broad application prospects in fields such as sports auxiliary training and analysis, virtual fitting, film production, and health care. Generally, a parameterized three-dimensional human body model (such as SMPL, STAR, etc.) is used to describe the three-dimensional human body shape. Based on the estimated pose and shape parameters, an approximate naked three-dimensional human body mesh that matches the estimated pose and shape can be recovered through the parameterized model.

[0003] In recent years, with the rapid development of deep learning, scholars have successively proposed some single-image three-dimensional human body shape estimation methods. For example, SMPLify uses a CNN-based method to predict 2D human joint positions and joint confidence, and minimizes the objective function to penalize the error between the projected 3D model joints and the detected 2D joints, SPIN learns to reconstruct three-dimensional human poses and shapes through iterative model fitting, HMR uses a regression-based method to estimate the shape of a naked human body model and uses a GAN model to determine whether it is a real human body, GCMR uses graph convolution to predict the position of each vertex of a naked human body model, and DecoMR explicitly establishes a dense correspondence between the mesh and the local image features in the UV space (i.e., a two-dimensional space used for three-dimensional mesh texture mapping).

[0004] Although the above methods can estimate the three-dimensional human body shape from images of people wearing tight clothing, they are not effective in estimating the three-dimensional human body shape from images of people wearing casual clothing due to the lack of dense correspondence between the surface of the naked human body mesh and the image pixels. For example, a person wearing loose pants may be identified as having a bent leg, a person wearing a loose shirt may be identified as being obese, and a person wearing a long coat may be identified as having a long upper body and a short lower body. Therefore, it is an urgent problem to be solved to propose a single-image three-dimensional human body shape estimation method that can effectively reduce the shape estimation error caused by clothing covering the human body. SUMMARY

[0005] In view of at least one defect or improvement demand of the prior art, the present application provides a single-image three-dimensional human body shape estimation method, system and electronic device and computer readable storage medium, aiming to solve the problem that the existing three-dimensional human body shape estimation method is disturbed by clothing covering the human body, resulting in large errors in shape estimation.

[0006] To achieve the above-mentioned purpose, according to the first aspect of the present application, a single-image three-dimensional human body shape estimation method is provided, comprising: estimating a three-dimensional parameterized model of a human body region contained in an input image in a non-clothed state to obtain an estimated three-dimensional parameterized model, and calculating error information between the estimated three-dimensional parameterized model and a real three-dimensional parameterized model to describe a first loss function of a deep neural network; under the guidance of the estimated three-dimensional parameterized model, reconstructing a three-dimensional reconstruction model of the human body region in a clothed state according to the input image, and calculating error information between the reconstruction models guided by the estimated three-dimensional parameter model and the real parameterized model respectively to describe a second loss function of a deep neural network; inputting the first loss function into the three-dimensional reconstruction model and feeding back the second loss function to the estimated three-dimensional parameterized model respectively for iteration until the model converges, to obtain an optimized three-dimensional human body shape estimation result.

[0007] In an embodiment of the present application, the estimated three-dimensional parameterized model of the human body region contained in the input image in the non-clothed state comprises: segmenting the human body region from the input image to obtain a human body image; extracting image features from the human body image; predicting the vertex position of the human body model according to the image features; inputting the predicted human body model vertex into a fully connected layer to regress human body model parameters, and restoring the estimated three-dimensional parameterized model according to the human body model parameters.

[0008] In an embodiment of the present application, the prediction of the vertex position of the human body model according to the image features comprises: connecting the extracted image features to the corresponding human body vertex based on a human body model template, and inputting the image convolutional neural network to predict the vertex position, and the prediction formula is: Wherein, T∈R K×3 and D∈R N×K respectively represent the human body model template and the down-sampling matrix, represent the adjacency matrix of the graph, F∈R N×f is the extracted image feature, W∈R (3+f)×3 represents the weight matrix, represents the predicted vertex position of the human body model.

[0009] In an embodiment of the present application, the estimated three-dimensional parameterized model is represented in the form of a graph: M0=W(T(β, θ), J(β), θ, ω); wherein, is a standard human body model, W(·) is a fusion skin function, J(β) describes the joint displacement due to body shape changes, ω∈R N×K is a fusion weight matrix, B P (·): R |θ| →R 3N is a pose fusion function, B S (·): R |β| →R 3N is a body shape fusion function.

[0010] In an embodiment of the present application, the first loss function is: L sp =L shape +L 2D +L 3D +L θ +λL β ; wherein, wherein, L shape represents the loss of each vertex, L 3D represents the loss of the 3D joint obtained from the SMPL model, L 2D represents the loss of the 3D joint obtained from the human body model template mapped to the 2D space, L θ represents the loss of the pose parameter, and L β represents the loss of the body shape parameter.

[0011] In an embodiment of the present application, the three-dimensional reconstruction model of the human body region in a dressed state is reconstructed from the input image under the guidance of the estimated three-dimensional parameterized model and the real three-dimensional parameterized model, comprising: the estimated three-dimensional parameterized model and the real three-dimensional parameterized model are used as the prior condition of the guidance to constrain the implicit surface function, and the surface reconstruction is performed through the implicit surface function to obtain the three-dimensional reconstruction model. In an embodiment of the present application, the error information between the three-dimensional reconstruction models guided by the estimated three-dimensional parameter model and the real parameterized model respectively is used to describe the second loss function of the deep neural network, comprising:

[0012] The second loss function obtained by using the depth-blur perception reconstruction loss in the surface reconstruction process is: wherein, n p is the number of three-dimensional sampling points, F * (p i ) is the real occupancy value of the p i point, and F(p i) is p i points is the predicted occupancy value of surface reconstruction of the estimated three-dimensional parametric model, gt (p i ) is p i points is the predicted occupancy value of surface reconstruction of the real three-dimensional parametric model.

[0013] According to a second aspect of the present application, there is also provided a single-image three-dimensional human body shape estimation system, comprising: a non-clothed human body model estimation module configured to estimate a three-dimensional human body shape of a human body region contained in an input image in a non-clothed state to obtain an estimated three-dimensional parametric model, and to calculate error information between the estimated three-dimensional parametric model and a real three-dimensional parametric model to describe a first loss function of a deep neural network; a clothed human body model reconstruction module configured to reconstruct a three-dimensional reconstruction model of the human body region in a clothed state according to the input image under the guidance of the estimated three-dimensional parametric model and the real three-dimensional parametric model, and to calculate error information between the reconstruction models guided by the estimated three-dimensional parametric model and the real three-dimensional parametric model, respectively, to describe a second loss function of a deep neural network; and a model optimization module configured to input the first loss function into the three-dimensional reconstruction model and feedback the second loss function to the estimated three-dimensional parametric model, respectively, for iteration until model convergence, to obtain an optimized three-dimensional human body shape estimation result.

[0014] According to a third aspect of the present application, there is also provided an electronic device comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program which, when executed by the processing unit, causes the processing unit to perform the steps of the method according to any one of the above embodiments.

[0015] According to a fourth aspect of the present application, there is also provided a computer readable storage medium storing a computer program executable by an access authentication device, which, when running on the access authentication device, causes the access authentication device to perform the steps of the method according to any one of the above embodiments.

[0016] Overall, the above technical solutions conceived by the present application can at least achieve the following beneficial effects compared with the prior art:

[0017] 1) Estimate the three-dimensional human body shape of the human body region contained in the input image in a non-clothed state with a three-dimensional parametric model, and calculate the error information between the estimated three-dimensional parametric model and the real three-dimensional parametric model to describe the first loss function; under the guidance of the estimated three-dimensional parametric model, the three-dimensional reconstruction model of the human body region in the clothed state is reconstructed according to the input image, and the error information between the reconstruction models guided by the estimated three-dimensional parametric model and the real parametric model respectively is calculated to describe the second loss function; the first loss function is input into the three-dimensional reconstruction model and the second loss function is fed back to the estimated three-dimensional parametric model respectively to iterate until the model converges, and the optimized three-dimensional human body shape estimation result is obtained, which effectively solves the problem of inaccurate human body posture estimation caused by the complexity of clothes.

[0018] 2) The three-dimensional model using the graph representation is more flexible and accurate in the human body shape estimation process. Since the graph structure is composed of points and lines, it is suitable for accurately fitting the human body model, and the graph representation has a certain flexibility and can simulate the change of the human body vertex position. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 The execution steps of a single-image three-dimensional human body shape estimation method provided by the embodiments of the present application are shown in the figure.

[0021] Figure 2 The flowchart of non-clothed human body three-dimensional shape estimation provided by the embodiments of the present application is shown in the figure.

[0022] Figure 3 The flowchart of clothed human body three-dimensional model reconstruction provided by the embodiments of the present application is shown in the figure.

[0023] Figure 4 The network framework diagram of the loss feedback of the clothed human body three-dimensional model reconstruction provided by the embodiments of the present application is shown in the figure.

[0024] Figure 5 The effect comparison diagram of non-clothed human body three-dimensional shape estimation and combination of non-clothed human body three-dimensional shape estimation and clothed human body three-dimensional model reconstruction provided by the embodiments of the present application is shown in the figure.

[0025] Figure 6 The effect diagram of three-dimensional human body shape estimation on daily motion pictures provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0027] The terms "first", "second", "third" and the like in the specification of the present application and claims and the above-described drawings are used to distinguish different objects and are not used to describe a particular order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, product or device.

[0028] As shown in Figure 1 The first embodiment of the present application proposes a single image three-dimensional human body shape estimation method, for example, including: step P1, estimating a three-dimensional human body shape of a human body region contained in an input image in a non-clothed state to obtain an estimated three-dimensional parameterized model, and calculating error information between the estimated three-dimensional parameterized model and a real three-dimensional parameterized model to describe a first loss function of a deep neural network; step P2, under the guidance of the estimated three-dimensional parameterized model and the real three-dimensional parameterized model, reconstructing a three-dimensional reconstruction model of the human body region in a clothed state according to the input image, and calculating error information between the three-dimensional reconstruction models guided by the estimated three-dimensional parameter model and the real parameterized model, respectively, to describe a second loss function of a deep neural network; step P3, respectively inputting the first loss function into the three-dimensional reconstruction model and feeding back the second loss function to the estimated three-dimensional parameterized model for iteration until the model converges, to obtain an optimized three-dimensional human body shape estimation result.

[0029] In step P1, the three-dimensional human body shape mentioned includes shape (Shape) and pose (Pose), for example. After estimating the non-clothed three-dimensional human body shape, the error between the estimation result and the real result is calculated, and the error is used to describe a loss function L sp .

[0030] Specifically, as shown in Figure 2 P1 includes the following steps, for example:

[0031] S1: human body image segmentation: the human body region is segmented from the input image, and the segmented human body image is denoted as I.

[0032] S2: image feature extraction: extract image feature G(I) from image I using image encoder G. The image encoder G is for example a residual convolutional neural network.

[0033] S3: predict the vertex position of the human body model according to the extracted image feature. In one embodiment, the extracted image feature is connected to the corresponding human body vertex based on a human body model template such as SMPL human body model, and input into a graph convolutional neural network to predict the vertex position. Specifically, the SMPL human body model is composed of 6890 vertices, in order to construct a lightweight network, for example, the human body template of 6890 vertices is down-sampled to 1732 vertices. The formula for predicting the vertex is as follows:

[0034]

[0035] wherein T∈R K×3 and D∈R N×K respectively represent the human body model template and the down-sampling matrix, represent the adjacency matrix of the graph, F∈R N×f is the extracted image feature, W∈R (3+f)×3 represent the weight matrix, represent the predicted vertex position of the human body model.

[0036] Further, in addition to deriving the 3D rotation angle of each vertex, the graph convolutional neural network also regresses the camera parameters of the weak perspective camera model, and obtains the scaling and translation parameters [s, t] of the weakly supervised camera model through transformation projection, t∈R 2 .

[0037] S4: regress the parameters of the three-dimensional parameterized human body model. For example, the human body model vertex predicted in step S3 is input into a fully connected layer to regress the parameters (β, θ) of the parameterized human body model SMPL. The parameters (β, θ) of the three-dimensional parameterized human body model and the corresponding camera parameters [s, t] (wherein s is the scaling factor and t is the translation factor), and restore the initial three-dimensional parameterized model M0 according to the estimated parameters. It is worth mentioning that the above steps preferably use SMPL or STAR as the template of the three-dimensional parameterized human body model, of course, in other embodiments, another human body model template is also used, and the present application is not limited thereto.

[0038] Further, the three-dimensional human body model M0 is represented in the form of a graph (containing vertices, edges) for example. Specifically, the three-dimensional human body model is restored by the estimated model parameters (β, θ) as follows:

[0039] M0=W(T(β,θ),J(β),θ,ω);

[0040]

[0041] wherein, is a standard human body model, W(·) is a fusion skin function, J(β) describes the joint displacement due to body shape changes, ω ∈ R N×K is a fusion weight matrix, B P (·): R |θ| → R 3N is a pose fusion function, B S (·): R |β| → R 3N is a body shape fusion function.

[0042] In one embodiment, in order to constrain and obtain more accurate human body model parameters, this step is performed, for example, in a network, defining five loss functions combined into the first loss function to constrain:

[0043] L sp = L shape + L 2D + L 3D + L θ + λL β ;

[0044] wherein,

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] wherein, L shape represents the loss of each vertex, L 3D represents the loss of the 3D joint obtained from the SMPL model, L 2D represents the loss of the 3D joint obtained from the human body model template mapped to the 2D space, L θ represents the loss of the pose parameters, and L β represents the loss of the body shape parameters.

[0051] In step P2, as shown in Figure 3 , the clothing human three-dimensional model reconstruction comprises, for example, the following steps:

[0052] E1: data preparation. For example, the image obtained in step S1, the estimated three-dimensional parametric model M(β, θ) obtained in step S4 and the real three-dimensional parametric model M(β gt ,θ gt ) are taken as inputs.

[0053] E2: clothing human three-dimensional model reconstruction. In training, for example, the estimated three-dimensional parametric model M(β, θ) obtained in step S4 and the real three-dimensional parametric model M(β gt ,θ gt ) are used for surface detail reconstruction, and the two SMPL human models are taken as guided prior conditions to constrain the implicit surface function, and the SMPL model is reconstructed by using the implicit surface function.

[0054] Specifically, the implicit surface function is, for example:

[0055] F(p; f I ,f V ) = o, o e [0, 1];

[0056] Wherein, p is a three-dimensional sampling point, F(·) represents an implicit surface function, o is a predicted occupancy value of a vertex, which is 1 if the predicted point is inside the model, otherwise 0, f I and f V represent two-dimensional pixel features and three-dimensional voxel features of the vertex respectively, and the method is to connect the pixel features of the 2D image and the voxel features of the 3D human model SMPL, and to judge the occupancy value of each sampling vertex after decoding by using the decoder to reconstruct the details of the human surface.

[0057] E3: feedback reconstruction loss. Step E3 returns the error information between the three-dimensional reconstruction model guided by the estimated three-dimensional parametric model and the real parametric model to the forward network. Specifically, for example, a depth-blur perception reconstruction loss is used in the surface reconstruction process to judge the accuracy of the construction, and the second loss function L c is returned to the graph convolutional neural network in step S3, which is used to adjust the SMPL model prediction result. The second loss function is:

[0058]

[0059] Wherein, n p is the number of three-dimensional sampling points, F * (p i ) is the real occupancy value of p i point, F(p i ) is the predicted occupancy value of p i point for surface reconstruction of the estimated three-dimensional parametric model, and F gt (p i) is p i The predicted occupancy value for surface reconstruction using the real 3D parametric model is given.

[0060] This second loss function can be used to continuously optimize the final result of surface reconstruction. At the same time, through iteration, the second loss function can be further applied to the pose parameter θ and morphological parameter β of human pose estimation, thereby obtaining a better result for estimating the human model.

[0061] In step P3, as Figure 4 As shown, the parameters of this network are obtained by minimizing the loss function L. sp and L c The training was obtained, and the training steps are as follows:

[0062] T1: Training set sample preparation. For example, perform step S1 on the human image for each data item, so that each data item in the training dataset contains: human image, 2D joint position of human image, real parameterized SMPL model, and real 3D human model.

[0063] T2: Network training. Iteratively execute T21 and T22 to minimize the loss function L. sp and L c Until the model converges.

[0064] T21: Perform steps S2, S3, and S4 in P1 to minimize the loss function L. sp and the loss function L sp The obtained preliminary parametric human body model is then transferred to P2.

[0065] T22: Execute P2, and apply the loss penalty term L from the basic step S4, which estimates the human body model and then reconstructs the clothing human body model. c The data is fed back to step S3 to optimize the output of step S4.

[0066] In summary, the first embodiment of the present application proposes a single image three-dimensional human body shape estimation method, according to the input image, the three-dimensional human body shape of the human body region contained in a non-clothed state is estimated to obtain an estimated three-dimensional parameterized model, and the error information between the estimated three-dimensional parameterized model and the real three-dimensional parameterized model is calculated to describe the first loss function. Under the guidance of the estimated three-dimensional parameterized model, the three-dimensional reconstruction model of the human body region in the clothed state is reconstructed according to the input image, and the error information between the reconstruction models guided by the estimated three-dimensional parameter model and the real parameterized model respectively is calculated to describe the second loss function. The first loss function is input into the three-dimensional reconstruction model and the second loss function is fed back to the estimated three-dimensional parameterized model respectively for iteration until the model converges, and the optimized three-dimensional human body shape estimation result is obtained. The problem of inaccurate human body posture estimation caused by the complexity of clothes is effectively solved. In addition, the three-dimensional model represented by the graph is more flexible and accurate in the human body shape estimation process. Since the graph structure is composed of points and lines, it is suitable for accurately fitting the human body model, and the graph representation has a certain flexibility and can simulate the change of the position of the human body vertex.

[0067] In addition, the second embodiment of the present application further proposes a single image three-dimensional human body shape estimation system, for example, comprising: a non-clothed human body model estimation module, a clothed human body model reconstruction module and a model optimization module.

[0068] The non-clothed human body model estimation module is used to estimate the three-dimensional human body shape of the human body region contained in a non-clothed state according to the input image to obtain an estimated three-dimensional parameterized model, and the error information between the estimated three-dimensional parameterized model and the real three-dimensional parameterized model is calculated to describe the first loss function of the deep neural network. The clothed human body model reconstruction module is used to reconstruct the three-dimensional reconstruction model of the human body region in the clothed state according to the input image under the guidance of the estimated three-dimensional parameterized model and the real three-dimensional parameterized model, and the error information between the three-dimensional reconstruction models guided by the estimated three-dimensional parameter model and the real parameterized model is calculated to describe the second loss function of the deep neural network. The model optimization module is used to input the first loss function into the three-dimensional reconstruction model and feed back the second loss function to the estimated three-dimensional parameterized model respectively for iteration until the model converges, and the optimized three-dimensional human body shape estimation result is obtained.

[0069] Specifically, for example, the pre-processing module is used to pre-process the early image. The pre-processing module prepares the training set samples, and each data item in the training data set contains a human body image, a 2D joint position of the human body image, a real parameterized SMPL model, and a real three-dimensional human body model. At the same time, the module encapsulates the MODNet algorithm to execute the above step S1. Specifically, for example, input a picture taken by an external device, the pre-processing module uses the MODNet algorithm to segment and crop the human body region in the input image to a resolution of 512*512.

[0070] The non-clothed human body model estimation module encapsulates the non-clothed human body shape estimation model trained in the P1 process. The module is a network composed of an image encoding module, a graph convolutional neural network vertex prediction module, and a regression parameterized human body pose and shape parameter module, and executes the above steps S2, S3, and S4. Specifically, for example, input a 512*512 (background removed) human body image, obtain the image feature vector F∈R 1×2048 through step S2, and connect DT∈R 1732×3 in each vertex of the human body model downsampling template of step S3 to obtain the input feature template as Through multiplication by the adjacency matrix and the weight parameter W∈R (3+1732)×3 , the positions of the 1732 vertices are predicted, and finally up-sampled to the positions of the 6890 vertices. Then, step S4 is executed, and the 6890 vertex coordinates obtained in step S3 are connected to the human body template SMPL as input, and 85 parameters (including camera parameters c∈R 3 , pose parameters θ∈R 24×3 , and shape parameters β∈R 10 ) are output by the fully connected layer regression. The parameterized human body model SMPL obtained in this process will guide the non-clothed human body model estimation module to perform surface reconstruction on the parameterized human body model.

[0071] The clothed human body model reconstruction module encapsulates the P2 trained clothed human body three-dimensional model surface reconstruction model in the training process, and under the guidance of the estimated three-dimensional parameterized model and the real three-dimensional parameterized model, the three-dimensional reconstruction model of the human body region in the clothed state is reconstructed according to the input image. Specifically, the module includes a clothed human body three-dimensional model reconstruction and reconstruction loss feedback sub-module, and executes steps E2 and E3 to calculate the error information between the reconstruction models guided by the estimated three-dimensional parameter model and the real parameterized model, which is used to describe the second loss function L c of the deep neural network.

[0072] The loss function of the non-clothed human body model estimation module is input into the clothed human body model estimation module in the training process by the model optimization module, the loss function of the M2 clothed human body model estimation module is fed back to the non-clothed human body model estimation module for iteration until the model converges, and the optimized three-dimensional human body shape estimation result is obtained, and finally the pose and shape parameters of the SMPL model are obtained.

[0073] The use of the clothed human body model reconstruction loss back to the non-clothed human body shape estimation can obtain better estimation output. Figure 5 The results of using only P1 (non-clothed human body shape estimation) and combining P1 and P2 (non-clothed human body shape estimation and clothed human body model reconstruction) are shown in the figure. Figure 6 As shown in the figure, the use of non-clothed human body shape estimation combined with clothed human body model reconstruction to estimate human body shape parameters is not only effective in simple actions, but also accurate in complex actions.

[0074] It is worth mentioning that the method implemented by the single-image three-dimensional human body shape estimation system disclosed in the second embodiment of the present application is as described in the first embodiment, and therefore will not be described in detail here. Alternatively, each module in the second embodiment and the above-mentioned other operations or functions are respectively used to implement the single-image three-dimensional human body shape estimation method described in the first embodiment, and the beneficial effects of the present embodiment are the same as those of the first embodiment, and therefore will not be described here for the sake of brevity.

[0075] The third embodiment of the present application further provides an electronic device, for example, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the method as described in the first embodiment, and the beneficial effects of the electronic device provided by the present embodiment are the same as those of the single-image three-dimensional human body shape estimation method provided by the first embodiment.

[0076] The fourth embodiment of the present application further provides a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement the steps of the above-mentioned method, and the beneficial effects of the computer readable storage medium provided by the present embodiment are the same as those of the single-image three-dimensional human body shape estimation method provided by the first embodiment.

[0077] The computer readable storage medium can include any type of disk including floppy disks, optical disks, DVDs, CD-ROMs, micro-drives, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nano-systems (including molecular memory ICs), or any type of media or device suitable for storing instructions and / or data.

[0078] It should be noted that, for the foregoing method embodiments, the sequences of the described actions are not necessarily required to achieve the objects of the application, and certain acts can be performed in other sequences, or even at the same time. Additionally, the described embodiments are merely provided as examples, and not all of the actions and modules are necessarily required to achieve the objects of the application.

[0079] In the above embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0080] In the several embodiments provided by the present application, it should be understood that the disclosed apparatus can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electric, mechanical or other forms.

[0081] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to the actual needs to achieve the purposes of the embodiments of the present application.

[0082] In addition, each functional unit in the embodiments of the present application can be integrated in a processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software functional units.

[0083] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned memory includes: a U disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0084] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be instructed by a program to be completed by relevant hardware, and the program can be stored in a computer readable memory, which can include a flash disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0085] The above is only exemplary embodiments of the present disclosure, which cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will easily think of embodiments of the present disclosure after considering the specification and practicing the disclosure herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

[0086] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope recorded in the present disclosure.

[0087] Those skilled in the art readily understand that the above only the preferred embodiments of the present application, and not to limit the present application, any modification, equivalent replacement and improvement made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A single image three-dimensional human body shape estimation method, characterized by, The method comprises the following steps: estimating a three-dimensional parameterized model of a human body region in a non-clothed state according to an input image, calculating error information between the estimated three-dimensional parameterized model and a real three-dimensional parameterized model to describe a first loss function of a deep neural network; reconstructing a three-dimensional reconstruction model of the human body region in a clothed state according to the input image under the guidance of the estimated three-dimensional parameterized model and the real three-dimensional parameterized model, and calculating error information between the three-dimensional reconstruction models guided by the estimated three-dimensional parameterized model and the real three-dimensional parameterized model respectively to describe a second loss function of the deep neural network; inputting the first loss function into the three-dimensional reconstruction model and feeding back the second loss function to the estimated three-dimensional parameterized model respectively for iteration until the model converges, and obtaining an optimized three-dimensional human body shape estimation result.

2. The single image three-dimensional human pose estimation method of claim 1, wherein, The method of estimating a three-dimensional parameterized model of a human body region in a non-clothed state according to an input image comprises the following steps: segmenting the human body region from the input image to obtain a human body image; extracting image features from the human body image; predicting the vertex position of a human body model according to the image features; inputting the predicted vertex of the human body model into a fully connected layer to regress human body model parameters, and restoring the estimated three-dimensional parameterized model according to the human body model parameters.

3. The single image three-dimensional human pose estimation method of claim 2, wherein, The method of predicting the vertex position of a human body model according to the image features comprises the following steps: The extracted image features are connected to corresponding human body vertices based on a human body model template, and input into a graph convolutional neural network to predict the vertex positions, with a prediction formula being: ​ wherein, and denote the human model template and a down-sampling matrix, respectively, denotes an adjacency matrix of a graph, is the extracted image feature, denotes a weight matrix, denotes the predicted human model vertex positions.

4. The single image three-dimensional human pose estimation method of claim 2, wherein, The estimated three-dimensional parameterized model is represented in the form of a graph. ； ； wherein, is a standard mannequin, is a blend skin function, describes joint displacements due to body shape changes, is a blend weight matrix, is a pose blend function, is a body shape blend function.

5. The single image three-dimensional human pose estimation method of claim 4, wherein, The first loss function is: ； wherein ; ; ; ; ; wherein L represents the loss of each vertex, represents the loss of 3D joint obtained from the SMPL model, represents the loss of 3D joint obtained from the human model template mapped to 2D space, represents the loss of pose parameters, represents the loss of body shape parameters.

6. The single image three-dimensional human pose estimation method of claim 1, wherein, The method of reconstructing a three-dimensional reconstruction model of the human body region in a clothed state according to the input image under the guidance of the estimated three-dimensional parameterized model and the real three-dimensional parameterized model comprises the following steps: using the estimated three-dimensional parameterized model and the real three-dimensional parameterized model as prior conditions to constrain an implicit surface function, and performing surface reconstruction through the implicit surface function to obtain the three-dimensional reconstruction model.

7. The single image three-dimensional human pose estimation method of claim 1, wherein, The method of calculating error information between the three-dimensional reconstruction models guided by the estimated three-dimensional parameterized model and the real three-dimensional parameterized model respectively to describe a second loss function of a deep neural network comprises the following steps: The second loss function is obtained by using a depth- blur-aware reconstruction loss in a surface reconstruction process: ; wherein, is the number of three-dimensional sampling points, is is the true occupancy value of a point, is is the predicted occupancy value of a point for surface reconstruction by the estimated three-dimensional parametric model, is is the predicted occupancy value of a point for surface reconstruction by the true three-dimensional parametric model.

8. A single image three-dimensional human body shape estimation system, characterized by, The method comprises the following steps: a non-clothed human body model estimation module is configured to estimate a three-dimensional parameterized model of a human body region in a non-clothed state according to an input image, and calculate error information between the estimated three-dimensional parameterized model and a real three-dimensional parameterized model to describe a first loss function of a deep neural network; a clothed human body model reconstruction module is configured to reconstruct a three-dimensional reconstruction model of the human body region in a clothed state according to the input image under the guidance of the estimated three-dimensional parameterized model and the real three-dimensional parameterized model, and calculate error information between the three-dimensional reconstruction models guided by the estimated three-dimensional parameterized model and the real three-dimensional parameterized model respectively to describe a second loss function of the deep neural network. A model optimization module is configured to input the first loss function into the three-dimensional reconstruction model and feed back the second loss function to the estimated three-dimensional parameterized model respectively, and iterate until the model converges to obtain an optimized three-dimensional human body shape estimation result.

9. An electronic device, comprising: The computer program product comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer program product comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method in any one of claims 1-7.

Images