Three-dimensional human body model joint reconstruction method based on monocular image, electronic device and storage medium

By processing and feature extraction of monocular images, combining local feature networks and SMPLX models, the problem of difficult hand and face posture reconstruction in the existing technology is solved, and a three-dimensional mannequin reconstruction with higher accuracy and wider application is achieved. .

CN114581502BActive Publication Date: 2024-11-15XIDIAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210233442.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-10
Publication Date
2024-11-15
Estimated Expiration
2042-03-10

AI Technical Summary

Technical Problem

Existing three-dimensional human body reconstruction methods based on monocular images are difficult to accurately capture hand and facial postures, limiting the accuracy and application range of human body model reconstruction.

Method used

By processing monocular images, body, hand and face features are extracted, and the three-dimensional mannequin is combined with the SMPLX model to generate more accurate Three-dimensional mannequin.

Benefits of technology

It improves the reconstruction accuracy of the three-dimensional mannequin model, expands the scope of application, and can more accurately capture the rich and profound information of the human body, including hand and facial postures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114581502B_ABST
    Figure CN114581502B_ABST
Patent Text Reader

Abstract

The invention discloses a method for jointly reconstructing a three-dimensional human body model based on a monocular image, an electronic device and a storage medium. The method comprises: obtaining a body monocular image, a hand monocular image and a face monocular image based on an original monocular image, extracting two-dimensional key point coordinates therein to obtain a body candidate frame, a hand candidate frame and a face candidate frame, based on which the original monocular image is cropped to obtain three feature images, a local feature extraction network is used to respectively obtain features in each feature image, the features are spliced ​​to obtain cascade features, an SMPLX model is trained, and three-dimensional human body reconstruction is performed based on the cascade features. The invention extracts richer depth information, constructs a three-dimensional human body model with higher accuracy and shorter time consumption.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method for jointly reconstructing a three-dimensional human body model based on a monocular image, an electronic device and a storage medium. Background Art

[0002] In recent years, billions of daily human activities have been recorded as videos and uploaded to public Internet websites, capturing diverse human behavioral actions in various real-world scenarios. The technology of digitizing human body movements in these videos has great potential in various applications including human-computer interaction, social artificial intelligence and robotics. The three-dimensional human body reconstruction model method based on monocular images has attracted a lot of attention. There are problems such as missing depth information in monocular image reconstruction.

[0003] At present, the 3D human body reconstruction method based on monocular images only includes the human body posture and body shape, ignoring the reconstruction of hand posture and facial expression. In addition, in the reconstruction of the overall 3D human body model, it is necessary to face the problems that the hands and face account for a small proportion of the image and it is difficult to capture accurate posture. This limits the accuracy of human body model reconstruction in the posture capture system and restricts computers from understanding human movements in a more intelligent way in human-computer interaction. Summary of the invention

[0004] The purpose of an embodiment of the present invention is to provide a method for jointly reconstructing a three-dimensional human body model based on a monocular image, which combines the hand posture, eye and chin posture of the human body on the basis of body posture, covers richer depth information, and the reconstructed three-dimensional human body model has higher accuracy and a wider range of applications.

[0005] The embodiment of the present invention also aims to provide an electronic device and a storage medium.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is a method for jointly reconstructing a three-dimensional human body model based on a monocular image, comprising the following steps:

[0007] S1, processing the original monocular image to be reconstructed to obtain a body monocular image, a hand monocular image and a face monocular image, respectively predicting the coordinates of the two-dimensional key points in each monocular image to obtain a body candidate frame, a hand candidate frame and a face candidate frame;

[0008] S2, using the body candidate frame, the hand candidate frame and the face candidate frame to crop the original monocular image, and adjust the size of the cropped image to obtain a body feature image, a face feature image and a hand feature image;

[0009] S3, train three local feature extraction networks to extract body features, facial features and hand features from body feature images, facial feature images and hand feature images respectively, and concatenate body features, facial features and hand features to obtain cascade features;

[0010] S4, training SMPLX model for 3D human body reconstruction based on cascade features.

[0011] Furthermore, the candidate box Box in S1 is {(l,r)}, where l represents the coordinates of the two-dimensional key points at the upper left corner of the body, hand, and face, and r represents the coordinates of the two-dimensional key points at the lower right corner of the body, hand, and face.

[0012] Furthermore, the training process of the local feature extraction network is as follows:

[0013] The HUMBI dataset is selected to obtain the original monocular image containing the whole body, which is then cropped to obtain a local monocular image. The OpenPose model is used to predict the coordinates of the two-dimensional key points in the local monocular image. The local candidate box is obtained based on the distribution of the two-dimensional key points. The original monocular image is cropped based on the local candidate box to obtain a local feature image, and the real model parameters corresponding to each local feature image are obtained.

[0014] Use the RestNet50 network to capture local features in local feature images;

[0015] Predict camera parameters and model parameters using multi-layer perceptron based on local features;

[0016] The loss function of the feature extraction network is calculated based on the real model parameters, camera parameters and predicted model parameters, and the parameters of the feature extraction network are adjusted based on the loss function value to obtain an optimized feature extraction network.

[0017] Furthermore, the local feature extraction network includes a body feature extraction network, a hand feature extraction network and a face feature extraction network, and the loss function in the local feature extraction network is as follows:

[0018] L=L p +τ1L joint,3D +τ2L reproj

[0019] Where L p Represents the loss between the predicted model and the true model, L joint,3D represents the loss of the 3D key points extracted by the prediction model and the 3D human key points extracted by the real model, L reproj It represents the loss between the two-dimensional key points obtained by projecting the three-dimensional key points extracted by the prediction model through the camera and the two-dimensional key points extracted by the real model. τ1 and τ2 represent the weighted coefficients of the balanced loss term.

[0020] Furthermore, the process of training the SMPLX model in S4 is as follows:

[0021] Using monocular images with SMPLX model labels as training data, using the methods described in S1 to S3 to obtain local features in the training data, and concatenating the local features to obtain cascade features;

[0022] Input the cascade feature F into the multi-layer perceptron to predict the camera parameters K and the SMPLX model pose parameters Body shape parameters Empirical parameters

[0023] According to the real SMPLX model parameters and human key point coordinates corresponding to each monocular image in the training data, the projection error, 3D key point error and SMPLX model parameter error are calculated respectively, and then the loss function L′ is obtained;

[0024] Repeat the above process, calculate the loss function of each iteration, update the SMPLX model parameters based on the new loss function value, and obtain the optimized SMPLX model.

[0025] Furthermore, the loss function L′ is as follows:

[0026]

[0027]

[0028] in represents the error between the true SMPLX model parameters and the predicted SMPLX model parameters, θ smplx , β smplx , smplx Respectively represent the posture parameters, body shape parameters, and empirical parameters of the real SMPLX model, They represent the posture parameters, body shape parameters, and empirical parameters of the SMPLX model predicted by the multilayer perceptron, respectively. m represents the number of key points of the human body, M represents the total number of key points of the human body, m = 1, 2, ..., M, M = 137, and v m Indicates whether the mth body key point is visible. represents the predicted 3D keypoint coordinates extracted by the SMPLX model, (x″′ 3d ,y″′ 3d ,z″′ 3d ) represents the three-dimensional key point coordinates extracted from the real SMPLX model, L′ joint,3D express and (x″′ 3d ,y″′ 3d ,z″′3d ), express The two-dimensional key point coordinates (x′ 2d ,y′ 2d ) means (x″′ 3d ,y″′ 3d ,z″′ 3d ) The two-dimensional key point coordinates obtained by projecting the camera parameter K, L′ reproj express and (x′ 2d ,y′ 2d ), τ1 and τ2 represent the weighted coefficients of the balance loss term.

[0029] Furthermore, the construction process of the training data is as follows:

[0030] The HUMBI dataset is screened to obtain original monocular images of the human body, the SMPL model parameters and camera parameters corresponding to each original monocular image are saved, and the OpenPose model is used to obtain the two-dimensional key point coordinates in each original monocular image;

[0031] Use SMPL2SMPLX to convert the original monocular image SMPL model into the SMPLX model, and obtain the SMPLX model parameters with the same posture as the SMPL model;

[0032] The projection matrix is ​​used to project the three-dimensional key point coordinates extracted by the SMPLX model to the two-dimensional key point coordinates, and the energy function between the two-dimensional key point coordinates obtained by the projection and the two-dimensional key point coordinates obtained by the OpenPose model is calculated;

[0033] Repeat the above process, update the SMPLX model parameters based on the newly calculated energy function, and add it as a new label to the HUMBI dataset to obtain a monocular image with the SMPLX model label.

[0034] An electronic device comprises a processor, a memory and a communication bus, wherein the processor and the memory communicate with each other via the communication bus;

[0035] Memory, used to store computer programs;

[0036] The processor is used to implement the above method steps when executing the program stored in the memory.

[0037] A computer-readable storage medium stores a computer program, and the computer program implements the above method steps when executed by a processor.

[0038] The beneficial effects of the present invention are as follows: the embodiment of the present invention is based on the SMPL model of the monocular image, calculates the SMPLX model parameters that best fit it, obtains a data set with an SMPLX model label, and provides training data for subsequent three-dimensional human body model reconstruction; this embodiment performs three-dimensional human body reconstruction based on cascade features by splicing body features, hand features and facial features in the human body monocular image, and the reconstruction process combines the body posture, facial posture and hand posture of the human body, covers rich depth information, and makes the reconstructed model more accurate and has a wider range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is an overall framework for joint reconstruction of 3D human body models based on monocular images.

[0041] Figure 2 It is a flowchart of the pre-trained human feature extraction network.

[0042] Figure 3 It is the input image and the reconstructed human body model result map.

[0043] Figure 4 This is a rendering of a three-dimensional human body model reconstructed using an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] A method for joint reconstruction of a 3D human body model based on monocular images, the process is as follows Figure 1 As shown, the following steps are included:

[0046] Step S1, cropping the monocular image to be reconstructed to obtain the monocular image I of the body B , monocular hand image I H And the monocular face image I F, input them into the OpenPose model to predict the two-dimensional key point coordinates, and calculate the candidate frames of the body, hands, and face respectively;

[0047] The two-dimensional key point coordinates m represents the number of two-dimensional key points, M represents the total number of two-dimensional key points, M = 137, including 25 body key points, 2 × 21 hand key points and 70 face key points, (x, y) m represents the coordinates of the mth key point, v m is the visibility of the mth key point, with a value of 0 or 1, 0 means the key point is invisible, and 1 means the key point is visible;

[0048] The candidate box Box = {(l, r)}, where l represents the coordinates of the upper left corner of the body, hand, and face two-dimensional key point (l x ,l y ), r represents the coordinates of the two-dimensional key points in the lower right corner of the body, hand, and face (r x ,r y ), and obtain the body candidate box Box b , Hand Candidate Box h And face candidate box Box f , the candidate box is similar to a rectangle with only a skeleton, which is used to represent the minimum circumscribed interval of the target area;

[0049] Step S2: crop the monocular image using the candidate frame and adjust the image size to 224×224 to obtain the body feature image I body , facial feature image I face and hand feature image I hand ;

[0050] Step S3, training a local feature extraction network, wherein the local feature extraction network includes a body feature extraction network, a face feature extraction network, and a hand feature extraction network, each of which has the same structure but different internal parameters, and all adopts a RestNet50 network, including a multi-layer network structure of downsampling and residual blocks;

[0051] Step S4, using the local feature extraction network to extract body feature images I body , facial feature image I face and hand feature image I hand The features in , the body features, facial features, and hand features are cascaded to obtain the cascade feature F, Indicates the splicing operation; E body 、E face 、E handThey represent the body feature extraction network, the face feature extraction network, and the hand feature extraction network, respectively. body (I body ) represents the body features extracted from the body feature image by the body feature extraction network, E face (I face ) represents the facial features extracted by the facial feature extraction network from the facial feature image, E hand (I hand ) represents the hand features extracted by the hand feature extraction network from the hand feature image;

[0052] Step S5, training the SMPLX model, and using the SMPLX model to reconstruct a three-dimensional human body based on the cascade feature F.

[0053] Since the structures of the local feature extraction networks are the same, the monocular image of the body is taken as an example to illustrate the construction process of the local feature extraction network. Figure 2 As shown, the details are as follows:

[0054] Step S31, selecting the HUMBI data set to obtain original monocular images containing the entire human body, and saving the real SMPL model parameters and camera parameters K corresponding to each original monocular image;

[0055] The SMPL model parameters include the body shape parameter β smpl and posture parameter θ smpl , the camera parameters refer to the camera external parameter matrix, which represents the position and orientation of the camera in the world coordinate system, mainly represented by the rotation matrix R and the translation vector T.

[0056] The original monocular image is cropped to obtain a body monocular image, a hand monocular image, and a face monocular image;

[0057] The OpenPose model is used to predict the coordinates of the two-dimensional body key points, and the distribution of each key point is counted to determine the coordinates of the two-dimensional key points in the upper left corner and the lower right corner of the body, and then the body candidate frame is obtained. The original monocular image is cropped according to the size of the candidate frame, and the image size is adjusted to obtain the body feature image I body ;

[0058] Step S32, using RestNet50 network as the body feature extraction network to capture the body feature image I body Physical characteristics in F body =E body (I body ), where F body is a 1024-dimensional feature vector;

[0059] Step S33, for the body feature Fbody , the camera parameters K and SMPL model posture parameters corresponding to each body feature are predicted by the multi-layer perceptron and body shape parameters

[0060] The multilayer perceptron (MLP) is a forward-structured artificial neural network that maps a set of input vectors to a set of output vectors. This embodiment uses a three-layer fully connected network, with the first two layers having 1024 neurons and the last layer having neurons with the same dimension as the output result, using ReLU as the activation function;

[0061] Step S34, according to the real SMPL model parameters and body key points corresponding to each original monocular image saved in step S31, respectively calculate the projection error, the 3D key point error and the SMPL model parameter error, and then obtain the loss function L of the body feature extraction network, as shown below:

[0062] L=L p +τ1L joint,3D +τ2L reproj

[0063] In the body feature extraction network, the loss function is expressed as follows:

[0064]

[0065] Where L p Represents the loss between the predicted model and the true model, L joint,3D represents the loss of the 3D key points extracted by the prediction model and the 3D human key points extracted by the real model, L reproj represents the loss between the two-dimensional key points obtained by projecting the three-dimensional key points extracted by the prediction model through the camera and the two-dimensional key points extracted by the real model. τ1 and τ2 represent the weighted coefficients of the balance loss term, and their value range is 0-1, τ1=0.7, τ2=0.5;

[0066] j represents the number variable of body key points, J represents the total number of body key points, j=1,2,…,J, J=25, v j Indicates whether the jth body key point is visible, represents the three-dimensional key point coordinates extracted by the predicted SMPL model, (x 3d ,y 3d , z 3d ) represents the three-dimensional key point coordinates extracted from the real SMPL model, express The two-dimensional key point coordinates obtained by projecting the camera parameter K, (x 2d ,y2d ) means (x 3d ,y 3d ,z 3d ) The two-dimensional key point coordinates obtained by projecting the camera parameter K;

[0067] Step S35, repeat steps S32-S34, calculate the error between the predicted result and the actual result in each iteration, and optimize the parameters in the body feature extraction network and the multilayer perceptron through back propagation, so that the loss function value gradually decreases until it no longer decreases and tends to be stable, terminate the iteration, and obtain the final body feature extraction network and multilayer perceptron.

[0068] The process of training the hand feature extraction network and the facial feature extraction network is similar to the above steps, except that when training the hand feature extraction network, the MANO model parameters corresponding to each hand monocular image are saved in step S31, and the loss between the true MANO model parameters and the predicted MANO model parameters is calculated when calculating the loss function; when training the facial feature extraction network, the Surry model parameters corresponding to each facial monocular image are saved in step S31, and the loss between the true Surry model parameters and the predicted Surry model parameters is calculated when calculating the loss function.

[0069] Since the camera parameters cannot be obtained based on the hand features and facial features in the hand feature extraction network and the facial feature extraction network, and there is no need to derive the camera parameters, it is sufficient to extract the hand features and facial features. Therefore, the third term of the loss function in the hand feature extraction network and the facial feature extraction network is zero.

[0070] In the hand feature extraction network, the first two terms of the loss function are expressed as follows:

[0071]

[0072] in Respectively represent the posture parameters and body shape parameters of the MANO model predicted by the multilayer perceptron, θ MANO , β MANO They represent the posture parameters and body shape parameters of the real MANO model, h represents the number variable of the hand key points, H represents the total number of hand key points, h=1,2,…,H, H=2×21, v h Indicates whether the h-th hand key point is visible, (x′ 3d ,y′ 3d ,z′ 3d ) represents the three-dimensional key point coordinates extracted from the real MANO model, Represents the predicted 3D key point coordinates extracted by the MANO model, τ1 = 0.8.

[0073] In the facial feature extraction network, the first two terms of the loss function are expressed as follows:

[0074]

[0075] in They represent the empirical parameters and face shape parameters of the Surry model predicted by the multilayer perceptron, ρ Surry , β Surry They represent the empirical parameters and face shape parameters of the real Surry model, e represents the number of facial key points, E represents the total number of facial key points, e=1,2,…,E, E=70, v e Indicates whether the key point of the e-th face is visible, (x″ 3d ,y″ 3d ,z″ 3d ) represents the three-dimensional key point coordinates extracted by the real Surry model, Represents the predicted 3D key point coordinates extracted by the Surry model, τ1=0.5.

[0076] The construction process of the SMPLX model in step S5 is as follows:

[0077] Step S51, using the data set with the SMPLX model label as training data, using the local feature extraction network constructed in step S3 to respectively obtain the body features, facial features and hand features in the training data, and splicing them to obtain the cascade feature F,

[0078] Step S52: Input the cascade feature F into the multi-layer perceptron to predict the camera parameter K and the SMPLX model posture parameter Body shape parameters Empirical parameters

[0079] Step S53, respectively calculate the projection error L′ according to the real SMPLX model parameters and the coordinates of the key points of the human body corresponding to each monocular image in the training data reproj , 3D key point error L′ joint,3D and SPMLX model parameter errors Then we get the loss function L′:

[0080]

[0081]

[0082] θ smplx , β smplx , smplx They represent the real SMPLX model posture parameters, body shape parameters, and empirical parameters respectively. They represent the posture parameters, body shape parameters, and empirical parameters of the SMPLX model predicted by the multilayer perceptron, represents the predicted 3D keypoint coordinates extracted by the SMPLX model, (x″′ 3d ,y″′ 3d ,z″′ 3d ) represents the three-dimensional key point coordinates extracted from the real SMPLX model, express The two-dimensional key point coordinates (x′ 2d ,y′ 2d ) means (x″′ 3d ,y″′ 3d ,z″′ 3d ) The two-dimensional key point coordinates obtained by projecting the camera parameter K;

[0083] Step S54, repeat steps S51-S53, calculate the error between the predicted result and the actual result in each iteration, and optimize the parameters in the multilayer perceptron through back propagation until the loss function no longer decreases but tends to be stable, terminate the iteration, and obtain the final SMPLX model parameters predicted by the multilayer perceptron.

[0084] The data set with SMPLX model labels described in step S51 is constructed as follows:

[0085] Step S511: obtaining a HUMBI data set, filtering the data set to obtain original monocular images containing the entire human body, and saving the SMPL model and parameters and camera parameters corresponding to each original monocular image;

[0086] Step S512: input the filtered monocular image into the OpenPose model, detect the coordinates of the two-dimensional key points in the monocular image, and in this embodiment, generate 25 body key points, 2×21 hand key points, and 70 face key points, and save the key point coordinates in JSON format;

[0087] Step S513: converting the SMPL model of the monocular image into the SMPLX model through SMPL2SMPLX to obtain the SMPLX model and model parameters of the same posture as the SMPL model;

[0088] Step S514: Project the three-dimensional key point coordinates extracted from the SMPLX model to the two-dimensional key point coordinates using the projection matrix, as follows:

[0089] Among them J 3d is the coordinate of the three-dimensional key point, J 2d The two-dimensional key point coordinates are obtained by selecting the first two dimensions of the calculation results;

[0090] Calculate the two-dimensional key point J obtained by projection 2d The energy function E between the two-dimensional key points predicted by the OpenPose model J , update the model parameters for the next iteration until the number of iterations reaches the set value, obtain the final SMPLX model parameters, and add the SMPLX model parameters as labels to the corresponding monocular images to obtain a labeled dataset.

[0091] The energy function Where i represents the number of key points, Π K () represents the projection function, J 3d,i represents the coordinates of the i-th key point extracted by the SMPLX model, Π K (J 3d,i ) represents the effect of camera parameter K on J 3d,i The two-dimensional key point coordinates obtained by projection, J est,i Represents the coordinates of the i-th two-dimensional key point predicted by the OpenPose model;

[0092] The energy function characterizes the error between the coordinates of the key points. When the energy function is smaller, it means that the SMPLX model parameters obtained through SMPL2SMPLX are more accurate, and the SMPLX model fitted based on this is closer to the target. The LBFGS optimizer in PyTorch is used to fit the SMPLX model parameters, and the fitting parameters are set as follows: the learning rate is 0.1, the number of iterations is 30, and the iteration is terminated when the number of iterations reaches the preset value to obtain the final SMPLX model parameters.

[0093] The SMPL model only fits the posture of the body part, and the accuracy of the reconstructed human body model is low. The SMPLX model also includes the fitting of the eyes, chin, and hand postures. This embodiment uses the SMPLX model to combine the hand posture and facial expression on the basis of the body posture, and the reconstructed 3D human body model has higher accuracy. At the same time, this embodiment is based on the SMPL model, so that the body posture and camera parameters of the SMPLX model are accurate. Only the hand posture and facial expression need to be fitted. The whole process is simple and the calculation amount is small. Compared with the optimization method SMPLify-X, it has a faster processing speed and accurate reconstruction results. Figure 3 and Figure 4 As shown, the embodiment of the present invention is used to perform three-dimensional human body reconstruction on each input image, and the obtained reconstructed models all fit the expressions and postures of the human face, hands and body, accurately express the body language and emotions of the human body, etc., and have broader application prospects.

[0094] The present invention also includes an electronic device, including a memory and a processor, wherein the memory is used to store various computer program instructions, and the processor is used to execute the computer program instructions to complete all or part of the above steps; the electronic device can communicate with one or more external devices, and can also communicate with one or more devices that enable users to interact with the electronic device, and / or communicate with any device that enables the electronic device to communicate with one or more other computing devices. The electronic device can also communicate with one or more networks (such as local area networks, wide area networks and / or public networks) through a network adapter.

[0095] The present invention also includes a computer-readable storage medium storing a computer program that can be executed by a processor. The computer-readable medium may include but is not limited to magnetic storage devices, optical disks, digital versatile disks, smart cards and flash memory devices. In addition, the readable storage medium described in the present invention can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" includes but is not limited to wireless channels and various other media (and / or storage media) that can store, contain and / or carry codes and / or instructions and / or data.

[0096] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for jointly reconstructing a three-dimensional human body model based on a monocular image, characterized in that: The following steps are involved: S1, processing the original monocular image to be reconstructed to obtain a body monocular image, a hand monocular image and a face monocular image, respectively predicting the coordinates of the two-dimensional key points in each monocular image to obtain a body candidate frame, a hand candidate frame and a face candidate frame; S2, using the body candidate frame, the hand candidate frame and the face candidate frame to crop the original monocular image, and adjust the size of the cropped image to obtain a body feature image, a face feature image and a hand feature image; S3, train three local feature extraction networks to extract body features, facial features and hand features from body feature images, facial feature images and hand feature images respectively, and concatenate body features, facial features and hand features to obtain cascade features; S4, training SMPLX model for 3D human body reconstruction based on cascade features; The training process of the local feature extraction network is as follows: The HUMBI dataset is selected to obtain the original monocular image containing the whole body, which is then cropped to obtain a local monocular image. The OpenPose model is used to predict the coordinates of the two-dimensional key points in the local monocular image. The local candidate box is obtained based on the distribution of the two-dimensional key points. The original monocular image is cropped based on the local candidate box to obtain a local feature image, and the real model parameters corresponding to each local feature image are obtained. Use the RestNet50 network to capture local features in local feature images; Predict camera parameters and model parameters using multi-layer perceptron based on local features; The loss function of the feature extraction network is calculated based on the real model parameters, the camera parameters and the predicted model parameters, and the parameters of the feature extraction network are adjusted based on the loss function value to obtain an optimized feature extraction network; The loss function in the local feature extraction network is as follows: L=L p +τ1L joint,3D +τ2L reproj Where L p Represents the loss between the predicted model and the true model, L joint,3D represents the loss of the 3D key points extracted by the prediction model and the 3D human key points extracted by the real model, L reproj represents the loss between the two-dimensional key points obtained by projecting the three-dimensional key points extracted by the prediction model through the camera and the two-dimensional key points extracted by the real model, τ1 and τ2 represent the weighted coefficients of the balance loss term; The process of training the SMPLX model in S4 is as follows: Using monocular images with SMPLX model labels as training data, using the methods described in S1 to S3 to obtain local features in the training data, and concatenating the local features to obtain cascade features; Input the cascade feature F into the multi-layer perceptron to predict the camera parameters K and the SMPLX model pose parameters Body shape parameters Empirical parameters According to the real SMPLX model parameters and human key point coordinates corresponding to each monocular image in the training data, the projection error, 3D key point error and SMPLX model parameter error are calculated respectively, and then the loss function L′ is obtained; Repeat the above process, calculate the loss function of each iteration, update the SMPLX model parameters based on the new loss function value, and obtain the optimized SMPLX model.

2. The method for jointly reconstructing a three-dimensional human body model based on a monocular image according to claim 1, characterized in that: The candidate box Box={(l,r)} in S1, l represents the coordinates of the two-dimensional key points in the upper left corner of the body, hand, and face, and r represents the coordinates of the two-dimensional key points in the lower right corner of the body, hand, and face.

3. The method for jointly reconstructing a three-dimensional human body model based on a monocular image according to claim 1, characterized in that: The loss function L′ is as follows: in represents the error between the true SMPLX model parameters and the predicted SMPLX model parameters, θ smplx , β smplx , smplx Respectively represent the posture parameters, body shape parameters, and empirical parameters of the real SMPLX model, They represent the posture parameters, body shape parameters, and empirical parameters of the SMPLX model predicted by the multilayer perceptron, respectively. m represents the number of key points of the human body, M represents the total number of key points of the human body, m = 1, 2, ..., M, M = 137, and v m Indicates whether the mth body key point is visible. represents the predicted 3D keypoint coordinates extracted by the SMPLX model, (x″′ 3d ,y″′ 3d ,z″′ 3d ) represents the three-dimensional key point coordinates extracted from the real SMPLX model, L′ joint,3D express and (x″′ 3d ,y″′ 3d ,z″′ 3d ), express The two-dimensional key point coordinates (x′ 2d ,y′ 2d ) means (x″′ 3d ,y″′ 3d ,z″′ 3d ) The two-dimensional key point coordinates obtained by projecting the camera parameter K, L′ reproj express and (x′ 2d ,y′ 2d ), τ1 and τ2 represent the weighted coefficients of the balance loss term.

4. The method for jointly reconstructing a three-dimensional human body model based on a monocular image according to claim 1, characterized in that: The construction process of the training data is as follows: The HUMBI dataset is screened to obtain original monocular images of the human body, the SMPL model parameters and camera parameters corresponding to each original monocular image are saved, and the OpenPose model is used to obtain the two-dimensional key point coordinates in each original monocular image; Use SMPL2SMPLX to convert the original monocular image SMPL model into the SMPLX model, and obtain the SMPLX model parameters with the same posture as the SMPL model; The projection matrix is ​​used to project the three-dimensional key point coordinates extracted by the SMPLX model to the two-dimensional key point coordinates, and the energy function between the two-dimensional key point coordinates obtained by the projection and the two-dimensional key point coordinates obtained by the OpenPose model is calculated; Repeat the above process, update the SMPLX model parameters based on the newly calculated energy function, and add it as a new label to the HUMBI dataset to obtain a monocular image with the SMPLX model label.

5. An electronic device, characterized in that: It includes a processor, a memory and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 4 when executing a program stored in a memory.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.