Single image 3D human body reconstruction method and device based on shadow
By using the human shadow information in the image to constrain the output parameters of the deep neural network, the existing three-dimensional human body reconstruction methods are solved, and the problems of high equipment cost and dependence on three-dimensional labeled data are achieved, and efficient three-dimensional human body reconstruction with low cost and no labeled data are achieved.
Patent Information
- Application Number
- CN202210678785.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-15
- Publication Date
- 2025-02-18
- Estimated Expiration
- 2042-06-15
AI Technical Summary
The existing three-dimensional human body reconstruction methods rely on depth cameras, motion capture systems or multi-view camera groups. The equipment is costly and complex to deploy. Deep learning-based methods require three-dimensional labeled data, which is difficult to deploy to unlabeled scenes.
By using the human body shadow information in the image as auxiliary information for three-dimensional human body reconstruction, the shadow information is used to constrain the output parameters of the deep neural network, optimize the reconstruction results, and do not rely on three-dimensional labeled data during the training process.
It realizes low-cost, three-dimensional human body reconstruction without three-dimensional labeling data, get rid of the dependence on expensive equipment, and the reconstruction effect is better than existing methods, and can achieve more accurate three-dimensional human body reconstruction in real time.
Smart Images
Figure CN115115776B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the intersection field of computer vision and graphics, and in particular relates to a single-image three-dimensional human body reconstruction method and device based on shadow. Background Art
[0002] With the rise of VR / AR technology, 3D reconstruction technology has received more and more attention as an important part of VR / AR content production, including 3D human reconstruction. Traditional human reconstruction methods rely on depth cameras, motion capture systems, or strictly calibrated and synchronized multi-view camera groups. The high cost of equipment and complex deployment have greatly limited the popularity and development of this technology. With the development of deep learning technology in recent years, deep learning-based methods have achieved remarkable results in reconstructing 3D human directions from a single RGB image input, and have gotten rid of the dependence on the above equipment. Such methods usually rely on a parametric human model. The deep neural network acts on the input image and outputs the model parameters, which are then used by the model to generate a human mesh surface based on the parameters.
[0003] Existing methods usually extract the following types of human features from images: full-body image texture, human body contour, and limb image texture. Due to the inherent depth uncertainty of the image, none of the above features can provide accurate human depth information, which makes the existing methods still have a certain gap from the goal of high-precision human reconstruction. In addition, deep neural networks need to be trained on labeled data. Existing methods usually rely on expensive and time-consuming 3D human body annotation, which will cause certain difficulties when deployed to new scenes without annotations. Summary of the invention
[0004] The present invention provides a method and device for reconstructing a three-dimensional human body from a single image based on shadows, that is, using the human body shadow projected on the ground by sunlight in the image to provide auxiliary information for reconstruction. Because the illumination projection angle of the shadow source is different from the camera projection angle of the human body source in the image, the shadow can make up for the inherent depth uncertainty defect of ordinary images. The present invention uses shadow information to constrain the three-dimensional human body output parameters of the deep neural network, and ultimately achieves the purpose of optimizing the reconstruction results. At the same time, the neural network training framework based on the present invention does not rely on three-dimensional annotation data, and can work normally in any image containing human body shadows.
[0005] The technical contents of the present invention include:
[0006] A single-image three-dimensional human body reconstruction method based on shadow, the method comprising:
[0007] Acquire an image containing a shadow and environmental parameters for generating the shadow;
[0008] Extract the local image img of the shadow area in the imageshadow , and based on the local image img shadow , calculate the two-dimensional joint point coordinates S in the shadow area detect ;
[0009] Extract the local image img of the human body area in the image body , and based on the local image img body , calculate the two-dimensional joint point coordinates J in the human body area 2d ;
[0010] Based on the local image img body , calculate the initial posture parameter θ0 and the initial body shape parameter β0;
[0011] Based on the environmental parameters, the two-dimensional joint point coordinates S detect , the two-dimensional joint point coordinates J 2d Optimizing the initial posture parameter θ0 and the initial body shape parameter β0 to obtain the optimal posture parameter θ and the optimal body shape parameter β;
[0012] According to the optimal posture parameter θ and the optimal body shape parameter β, a three-dimensional human body mesh model is reconstructed.
[0013] Furthermore, the environmental parameters include: camera intrinsic parameter matrix K, illumination direction vector Ground Normal and the translation amount d.
[0014] Further, the obtaining of the environmental parameters for generating the shadow includes:
[0015] After taking photos of the calibration chessboard, perform algorithm calibration to obtain the camera intrinsic parameter matrix K;
[0016] Place the aruco Marker calibration plate on the ground and take a photo to calibrate the algorithm and obtain the ground normal vector and the translation d;
[0017] Use the mirror reflection ball for calibration to obtain the light direction vector
[0018] Furthermore, the extraction of the local image img of the shadow area in the image shadow , and based on the local image img shadow , calculate the two-dimensional joint point coordinates S of the shadow area detect ,include:
[0019] Constructing a shadow contour detection network, wherein the shadow contour detection network adopts an MTMT convolutional neural network structure and uses network model parameters trained on the SBU and ISTD datasets as initialization;
[0020] The image is input into the shadow contour detection network to obtain the local image img of the shadow area shadow ;
[0021] Constructing a shadow joint point estimation network, wherein the shadow joint point estimation network adopts a Stacked Hourglass convolutional neural network structure and uses shadow data synthesized on Human3.6M and standard values of joint point coordinates for supervised training;
[0022] The local image img shadow Input the shadow joint point estimation network to obtain the two-dimensional joint point coordinates S of the shadow area detect .
[0023] Furthermore, the local image img of the human body area in the image is extracted body , and based on the local image img body , calculate the two-dimensional joint point coordinates J in the human body area 2d ,include:
[0024] Build a human body bounding box detection network;
[0025] The image is input into the human body bounding box detection network to obtain a local image img of the human body area body ;
[0026] Construct a two-dimensional human joint point detection network;
[0027] The local image img body Input the two-dimensional human joint point detection network to obtain the two-dimensional joint point coordinates J in the human body area 2d .
[0028] Furthermore, the local image img body , calculate the initial posture parameter θ0 and the initial body shape parameter β0, including:
[0029] Constructing a human body reconstruction network, wherein the human body reconstruction network adopts a SPIN network structure and uses network model parameters trained on the Human3.6M dataset as backbone network initialization;
[0030] Based on the environmental parameters, the two-dimensional joint point coordinates S detect and the two-dimensional joint point coordinates J 2d Training a human body reconstruction network to obtain a trained human body reconstruction network;
[0031] The local image img body Input the trained human body reconstruction network to obtain the initial posture parameter θ0 and the initial body shape parameter β0.
[0032] Furthermore, the environmental parameters and the two-dimensional joint point coordinates S′ detect and the two-dimensional joint point coordinates J′ 2d Train the human body reconstruction network to obtain the trained human body reconstruction network, including:
[0033] The local image of the human body area in the image is input into the human body reconstruction network ψ τ , get the posture parameter θ τ and body shape parameter β τ , τ represents the number of first iterations;
[0034] According to the attitude parameter θ τ and body shape parameter β τ Reconstruct the 3D human body mesh model and obtain the 3D joint coordinates of the reconstructed human body
[0035] Based on the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates Constructing a constraint expression of a reprojection matching relationship, and organizing the constraint expression of the reprojection matching relationship to obtain a first non-homogeneous equation group of an offset t of a three-dimensional human body relative to a camera center;
[0036] Based on the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Constructing a constraint condition expression of the shadow consistency matching relationship, and sorting out the constraint condition expression of the shadow consistency matching relationship to obtain a second non-homogeneous equation group of the offset t of the three-dimensional human body relative to the camera center;
[0037] After the first non-homogeneous equation group and the second non-homogeneous equation group are put in parallel, the least square method is used to solve them to obtain the offset t of the three-dimensional human body relative to the camera center;
[0038] Combine the offset t and the environmental parameters to convert the three-dimensional joint point coordinates Project it to the shadow area to get the three-dimensional coordinates of the joint points in the shadow area And calculate the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Shadow consistency loss function between them;
[0039] Calculate the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates The reprojection loss function between ;
[0040] Based on the shadow consistency loss function and the reprojection loss function, the human body reconstruction network ψτ Train and obtain the human body reconstruction network ψ τ+1 ;
[0041] After the first iteration round is completed, the trained human body reconstruction network is obtained.
[0042] Furthermore, the environmental parameters and the two-dimensional joint point coordinates S detect , the two-dimensional joint point coordinates J 2d Optimizing the initial posture parameter θ0 and the initial body shape parameter β0 to obtain the optimal posture parameter θ and the optimal body shape parameter β includes:
[0043] According to the attitude parameter θ c and body shape parameter β c Reconstruct the 3D human body mesh model and obtain the 3D joint coordinates of the reconstructed human body Wherein, c is the second iteration round;
[0044] Based on the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates Constructing a constraint expression of a reprojection matching relationship, and organizing the constraint expression of the reprojection matching relationship to obtain a first non-homogeneous equation group of an offset t of a three-dimensional human body relative to a camera center;
[0045] Based on the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Constructing a constraint condition expression of the shadow consistency matching relationship, and sorting out the constraint condition expression of the shadow consistency matching relationship to obtain a second non-homogeneous equation group of the offset t of the three-dimensional human body relative to the camera center;
[0046] After the first non-homogeneous equation group and the second non-homogeneous equation group are put in parallel, the least square method is used to solve them to obtain the offset t of the three-dimensional human body relative to the camera center;
[0047] Combine the offset t and the environmental parameters to convert the three-dimensional joint point coordinates Project it to the shadow area to get the three-dimensional coordinates of the joint points in the shadow area And calculate the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Shadow consistency loss function between them;
[0048] Calculate the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates The reprojection loss function between ;
[0049] By constraining the pose parameters θc , body shape parameter β c With the attitude parameter θ c+1 , body shape parameter β c+1 The similarity of , and the regular term is obtained;
[0050] Based on the shadow consistency loss function, the reprojection loss function and the regularization term, the posture parameter θ c and body shape parameter β c Optimize to get the posture parameter θ c+1 and body shape parameter β c+1 ;
[0051] Until the second iteration round is completed, the optimal posture parameter θ and the optimal body shape parameter β are obtained.
[0052] Furthermore, the reconstructing the three-dimensional human body mesh model according to the optimal posture parameter θ and the optimal body shape parameter β includes:
[0053] Based on the SMPL human body parameter model, the optimal posture parameter θ and the optimal body shape parameter β are converted into three-dimensional human joint point coordinates J 3d ;
[0054] According to the three-dimensional human joint point coordinates J 3d , reconstruct the 3D human body mesh model.
[0055] A single-image three-dimensional human body reconstruction method based on shadow, the method comprising:
[0056] Acquire an image without a shadow, and based on the scene of the image, acquire a sample image containing a shadow and sample environmental parameters of the shadow;
[0057] Extract the local image img of the shadow area in the sample image shadow , and based on the local image img shadow , calculate the two-dimensional joint point coordinates S′ in the shadow area detect ;
[0058] Extract the local image img of the human body area in the sample image body , and based on the local image img body , calculate the two-dimensional joint point coordinates J′ in the human body area 2d ;
[0059] Constructing a human body reconstruction network, wherein the human body reconstruction network adopts a SPIN network structure;
[0060] Based on the sample environment parameters, the two-dimensional joint point coordinates S′ detect and the two-dimensional joint point coordinates J′ 2dTraining a human body reconstruction network to obtain a trained human body reconstruction network;
[0061] Inputting a local image of a human body region in the image into a trained human body reconstruction network to obtain a posture parameter θ and a body shape parameter β;
[0062] A three-dimensional human body mesh model is reconstructed according to the posture parameter θ and the body shape parameter β.
[0063] An electronic device comprises a memory and a processor, wherein a computer program is stored in the memory and the computer program is loaded and executed by the processor to implement the above method.
[0064] Compared with the existing methods, the present invention has the following advantages:
[0065] By applying shadow information to the process of 3D human body reconstruction, the spatial geometric pairing relationship between shadow and human body is used to train the human body reconstruction network, which can get rid of the dependence on 3D annotated data, realize low-cost training of 3D human body reconstruction network, and the reconstruction effect is better than the existing method. At the same time, the present invention can also use shadow information to directly optimize the reconstruction result, and realize more accurate 3D human body reconstruction in real time. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 This is a flowchart for implementing the output result optimization method.
[0067] Figure 2 This is a flowchart for implementing the network model optimization method. DETAILED DESCRIPTION
[0068] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention is described in detail below through specific implementations and drawings, but it does not constitute a limitation of the present invention.
[0069] The present invention provides a human body reconstruction method for single-image 3D human body reconstruction tasks, which uses the human body shadow projected on the ground by sunlight for optimization. According to the optimization target, it is divided into two modes: (a) based on the environmental parameters of the image usage scene, the human body reconstruction result is directly optimized on the image with shadow; (b) the network model parameters are optimized on the image with shadow. The trained model can reconstruct the human body model on the image without shadow and obtain better reconstruction results than the existing method.
[0070] Embodiment 1: Optimizing the human body reconstruction result directly on the image containing shadows.
[0071] The hardware platform of this embodiment uses an Intel i9-7900X CPU and an NVIDIA GeForce GTX1080Ti graphics card. The system program is written in Python and uses pytorch, numpy, and opencv libraries.
[0072] The single image 3D human body reconstruction method of this embodiment is as follows: Figure 1 As shown, the following steps are included.
[0073] Step 101: Calibrate environmental parameters of the usage scenario.
[0074] The environmental parameters of this embodiment include: the camera internal parameter matrix K, the direction of sunlight Ground Normal The calibration method includes camera calibration, ground calibration and light calibration, that is, using the calibration chessboard, taking a photo and then calibrating the algorithm to obtain the camera intrinsic parameter matrix K; using the aruco Marker calibration plate, placing it on the ground and taking a photo and then calibrating the algorithm to obtain the ground normal vector and translation d; the direction of sunlight is calibrated using a mirror reflection ball to obtain the light direction vector Specifically, the calibration step of the present invention includes:
[0075] 1. In the camera calibration phase, use the findChessboardCorners function of the opencv library in Python to extract the coordinates of the corner points in the chessboard image, use the calibrateCamera function to obtain the camera intrinsic parameter matrix K, and extract the camera focal length (f x , f y ) and the optical center (c x , c y );
[0076] 2. Ground calibration is divided into the following steps:
[0077] ① Use the aruco.DICT_6X6_1000 arrangement method of the aruco library in Python to print four calibration papers of different forms, then arrange them in a 2×2 manner and put them into an acrylic board to make a ground calibration board.
[0078] ②Measure the outer margin of each calibration paper and the distance between the calibration papers, and send them to the ground calibration algorithm as hyperparameters.
[0079] ③ During the calibration process, the calibration plate is placed on a flat ground to capture the calibration image.
[0080] ④ Use the camera parameters to remove the lens distortion of the calibration image. Then use the detectMarkers function of the aruco library to extract the corner points, and use the estimatePoseBoard function to obtain the rotation vector of the calibration board. and translation vectors Then the ground normal vector Ground offset Where H is the thickness of the calibration plate.
[0081] 3. Light calibration is divided into the following steps:
[0082] ① Set up the reflector ball to a fixed position, where the sun can be seen on the reflector ball. Set the camera to the standard ISO sensitivity, take an image of the mirror reflector ball, and mark the center position of the ball in the image (b u , b v );
[0083] ②Use the calibrated camera focal length (f x , f y ) and the optical center (c x , c y ), solve the normalized three-dimensional coordinates of the sphere center (b x , b y , b z ), specifically:
[0084] ③ Keep the position of the mirror reflection ball unchanged, set the camera to low sensitivity ISO = 64, and take another picture of the mirror reflection ball. At this time, the image is basically completely black, with only a few highlights coming from the refraction points of sunlight. Mark the center position of the highlight (p u , p v ), which is the highlight reflection point;
[0085] ④Use the calibrated camera focal length and optical center to solve the normalized three-dimensional coordinates of the highlighted reflection point (p x , p y , p z ), specifically:
[0086] ⑤ During the specular reflection of sunlight at a bright spot, the direction vector of the outgoing light is The normal vector to the mirror reflection plane is The direction of sunlight is the incident direction
[0087] In an example, when the camera position is fixed, the environmental parameters only need to be calibrated once. By recording the calibration time and the latitude and longitude of the scene, the sun pitch and yaw angles in the standard coordinate system can be obtained, and then converted into the reference direction of sunlight. Calculate the transformation matrix of the illumination calibration direction and reference direction in the camera coordinate system When the sun's direction changes over time, only the new reference direction at the current moment needs to be calculated. At this time, the illumination direction in the camera coordinate system is
[0088] In one example, the camera orientation is rotatable and the camera extrinsic matrix R ex When available (such as surveillance cameras), the new environment parameters are: ground normal vector The offset d remains unchanged, and the light direction
[0089] Step 102: Input a human body image containing a shadow.
[0090] The shadow described in the present invention refers to the shadow area formed by projecting a human body in three-dimensional space onto a flat ground using sunlight. The ground should be basically flat without obvious depressions or protrusions. Specifically, it is sufficient that the outline of the shadow does not show significant deformation, and the shadow should be completely visible in the image.
[0091] Step 103: Extract the local image img of the shadow area in the image shadow , and based on the local image img shadow , calculate the two-dimensional joint point coordinates S in the shadow area detect .
[0092] This step is based on the shadow contour detection network and the shadow joint point estimation network. The shadow contour detection network uses the MTMT convolutional neural network model and uses the network model parameters trained on the SBU and ISTD datasets as initialization. Binarize the original image according to the shadow area to generate a grayscale image img shadow , that is, the pixel value of the shadow area is 255, and the pixel value of the non-shadow area is 0. The shadow joint estimation network adopts the Stacked Hourglass convolutional neural network structure and uses the synthetic shadow data on Human3.6M and the standard values of the joint coordinates for supervised training. Then, the two-dimensional coordinates of the 24 joints in the shadow area are inferred in the real scene.
[0093] In one example, this step includes:
[0094] 1. Build a shadow contour detection network;
[0095] 2. Input the image into the shadow contour detection network to obtain the local image img of the shadow area shadow ;
[0096] 3. Construct a shadow joint point estimation network;
[0097] 4. The local image img shadow Input the shadow joint point estimation network to obtain the two-dimensional joint point coordinates S of the shadow area detect .
[0098] In one example, using the shadow contour detection network will perform indiscriminate shadow extraction on the entire image area. This may cause interference in some scenes with shadows of other objects. For this kind of shadow mixing situation, two solutions are given:
[0099] ① For static background environments, eliminate interference by manually dividing the background area and removing the static shadows generated by the background area;
[0100] ②For dynamic background environments, only shadows related to the human body are extracted through the SSIS shadow instance segmentation network.
[0101] Step 104: Extract the local image img of the human body area in the image body , and based on the local image img body , calculate the two-dimensional joint point coordinates J in the human body area 2d .
[0102] This step is based on the human body bounding box detection network and the two-dimensional human joint point detection network. The detected two-dimensional joint point J 2d Contains 17 human joints defined by the MPI-INF-3DHP dataset, which is a subset of the SMPL human joint collection.
[0103] In one example, this step includes:
[0104] Build a human body bounding box detection network;
[0105] The image is input into the human body bounding box detection network to obtain a local image img of the human body area body ;
[0106] Construct a two-dimensional human joint point detection network;
[0107] The local image img body Input the two-dimensional human joint point detection network to obtain the two-dimensional joint point coordinates J in the human body area 2d .
[0108] Step 105: Based on the local image imgbody , calculate the initial posture parameter θ0 and the initial body shape parameter β0.
[0109] For the input image img body The initial posture parameter θ0 and the initial body shape parameter β0 are obtained by constructing a human body reconstruction network. The human body reconstruction network adopts the SPIN network structure and uses the network model parameters trained on the Human3.6M dataset as the backbone network initialization.
[0110] In one example, the training process of the human body reconstruction network is as follows: Figure 2 As shown, including:
[0111] 1. Input the local image of the human body area in the image sample into the human body reconstruction network ψ τ , get the posture parameter θ τ and body shape parameter β τ , τ represents the number of first iterations;
[0112] 2. Based on the attitude parameter θ τ and body shape parameter β τ Reconstruct the 3D human body mesh model and obtain the 3D joint coordinates of the reconstructed human body
[0113] 3. Based on the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates Constructing the constraint expression of the reprojection matching relationship And the constraint condition expressions of the reprojection matching relationship are sorted out to obtain the first non-homogeneous equation group of the offset t of the three-dimensional human body relative to the camera center:
[0114]
[0115] The first and second rows of this system of equations contain 17 sub-equations respectively, so a total of 34 equations. Among them, Represents the two-dimensional joint point J 2d The coordinates of the nth dimension, Represents the three-dimensional joint point J 3d The coordinates of the nth dimension;
[0116] 4. Based on the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Constructing the constraint expression of the shadow consistency matching relationship make S=S detect , then the formula is simplified to
[0117] The constraint expressions of the simplified shadow consistency matching relationship are sorted out to obtain the second non-homogeneous equation group of the offset t of the three-dimensional human body relative to the camera center:
[0118]
[0119] The first and second rows of the system of equations contain 24 sub-equations respectively, thus containing 48 equations in total.
[0120] 5. After the first non-homogeneous equation group and the second non-homogeneous equation group are put in parallel, a non-homogeneous equation group of the form At = b can be obtained, and then the least squares method is used to solve the offset of the three-dimensional human body relative to the camera center t = (A T A) -1 ×A T b;
[0121] 6. Combine the offset t and the environmental parameters to convert the three-dimensional joint point coordinates Project it to the shadow area to get the three-dimensional coordinates of the joint points in the shadow area And calculate the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates The shadow consistency loss function between
[0122] 7. Calculate the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates The reprojection loss function between
[0123] 8. Based on the shadow consistency loss function and the reprojection loss function, the overall loss function L = L reproj +αL shadow , where α represents the weight coefficient;
[0124] 9. According to the overall loss function L, reconstruct the human body network ψ τ Train and obtain the human body reconstruction network ψ τ+1 ;
[0125] 910. Until the first iteration round is completed, a trained human body reconstruction network is obtained.
[0126] Step 106: Based on the environmental parameters and the two-dimensional joint point coordinates S detect , 2D joint point coordinates J 2d Optimize the initial posture parameter θ0 and the initial body shape parameter β0 to obtain the optimal posture parameter θ and the optimal body shape parameter β.
[0127] In one example, this step includes:
[0128] 1. Based on the attitude parameter θ c and body shape parameter β c Reconstruct the 3D human body mesh model and obtain the 3D joint coordinates of the reconstructed human body Wherein, c is the second iteration round;
[0129] 2. Based on the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates Constructing a constraint expression of a reprojection matching relationship, and organizing the constraint expression of the reprojection matching relationship to obtain a first non-homogeneous equation group of an offset t of a three-dimensional human body relative to a camera center;
[0130] Preferably, since the two-dimensional joint point coordinate J 2d The reconstructed 3D joint coordinates for 17 human joints There are 24 human joints, so 17 human joints need to be selected from the 24 human joints during matching.
[0131] 3. Based on the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Constructing a constraint condition expression of the shadow consistency matching relationship, and sorting out the constraint condition expression of the shadow consistency matching relationship to obtain a second non-homogeneous equation group of the offset t of the three-dimensional human body relative to the camera center;
[0132] 4. After the first non-homogeneous equation group and the second non-homogeneous equation group are put in parallel, the least square method is used to solve them to obtain the offset t of the three-dimensional human body relative to the camera center;
[0133] 5. Combine the offset t and the environmental parameters to convert the three-dimensional joint point coordinates Project it to the shadow area to get the three-dimensional coordinates of the joint points in the shadow area And calculate the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Shadow consistency loss function between them;
[0134] 6. Calculate the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates The reprojection loss function between ;
[0135] 7. By constraining the posture parameter θ c , body shape parameter β c With the attitude parameter θ c+1 , body shape parameter β c+1 The similarity of , we get the regular term L reg =||θ-θinit ||;
[0136] 8. Based on the shadow consistency loss function, the reprojection loss function and the regularization term, the overall energy term is constructed to adjust the posture parameter θ c and body shape parameter β c Optimize to get the posture parameter θ c+1 and body shape parameter β c+1 ;
[0137] 9. Until the second iteration round is completed, the optimal posture parameter θ and the optimal body shape parameter β are obtained.
[0138] Step 107: Reconstruct a three-dimensional human body mesh model according to the optimal posture parameter θ and the optimal body shape parameter β.
[0139] In one example, this step includes:
[0140] 1. Based on the SMPL human body parameter model, the optimal posture parameter θ and the optimal body shape parameter β are converted into three-dimensional human joint point coordinates J 3d .
[0141] The full name of the SMPL model is A Skinned Multi-Person Linear Model. This model uses sparse parameterization to represent dense human body meshes. The model accepts the axis angle representation parameter θ∈R of the joint rotation 24×3 And the body shape parameter β∈R 100 As input, the three-dimensional coordinates J of the joint are output according to the following formula 3d ∈R 24×3 and the surface grid coordinates V∈R 6890×3 :
[0142] J 3d =Φ(θ,T)
[0143] V=W β J 3d
[0144] Where Φ is the standard forward kinematics transformation function, which applies θ to the template posture to perform joint-by-joint rigid rotation transformation; T∈R 24×3 Defines the three-dimensional coordinates of the joints of the template posture; W β ∈R 6890×24 are the skin parameters generated from β.
[0145] 2. According to the three-dimensional human joint point coordinates J 3d , reconstruct the 3D human body mesh model.
[0146] Embodiment 2: Reconstructing a human body model on a shadow-free image.
[0147] The single-image three-dimensional human body reconstruction method of this embodiment includes the following steps.
[0148] Step 201: Acquire an image without a shadow, and based on the scene of the image, acquire a sample image containing a shadow and sample environmental parameters of the shadow.
[0149] Step 202: According to the sample image containing the shadow and the sample environment parameters of the shadow, and based on the above steps 102-105, a human body reconstruction network is trained.
[0150] Step 203: Input the image without shadow into the trained human body reconstruction network to obtain posture parameters and body shape parameters.
[0151] Step 204: Reconstruct a three-dimensional human body mesh model according to the posture parameters and body shape parameters.
[0152] In summary, the present invention gets rid of the dependence on three-dimensional labeled data and realizes low-cost training of three-dimensional human body reconstruction network.
[0153] The above implementation examples are only used to illustrate the technical solution of the present invention rather than to limit it. Those skilled in the art may modify or replace the technical solution of the present invention with equivalents without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be based on the claims.
Claims
1. A single-image three-dimensional human body reconstruction method based on shadow, the method comprising: Acquire an image containing a shadow and environmental parameters for generating the shadow; Extract the local image img of the shadow area in the image shadow , and based on the local image img shadow , calculate the two-dimensional joint point coordinates S in the shadow area detect ; wherein the extraction of the local image img of the shadow area in the image shadow , and based on the local image img shadow , calculate the two-dimensional joint point coordinates S in the shadow area detect ,include: Constructing a shadow contour detection network, wherein the shadow contour detection network adopts an MTMT convolutional neural network structure and uses network model parameters trained on the SBU and ISTD datasets as initialization; The image is input into the shadow contour detection network to obtain the local image img of the shadow area shadow ; Constructing a shadow joint point estimation network, wherein the shadow joint point estimation network adopts a Stacked Hourglass convolutional neural network structure and uses shadow data synthesized on Human3.6M and standard values of joint point coordinates for supervised training; The local image img shadow Input the shadow joint point estimation network to obtain the two-dimensional joint point coordinates S of the shadow area detect ; Extract the local image img of the human body area in the image body , and based on the local image img body , calculate the two-dimensional joint point coordinates J in the human body area 2d ; wherein the extraction of the local image img of the human body area in the image body , and based on the local image img body , calculate the two-dimensional joint point coordinates J in the human body area 2d ,include: Build a human body bounding box detection network; The image is input into the human body bounding box detection network to obtain a local image img of the human body area body ; Construct a two-dimensional human joint point detection network; The local image img body Input the two-dimensional human joint point detection network to obtain the two-dimensional joint point coordinates J in the human body area 2d ; Based on the local image img body , calculate the initial posture parameter θ0 and the initial body shape parameter β0; wherein, the local image img body , calculate the initial posture parameter θ0 and the initial body shape parameter β0, including: Constructing a human body reconstruction network, wherein the human body reconstruction network adopts a SPIN network structure and uses network model parameters trained on the Human3.6M dataset as backbone network initialization; Based on the environmental parameters, the two-dimensional joint point coordinates S detect and the two-dimensional joint point coordinates J 2d Train the human body reconstruction network to obtain a trained human body reconstruction network; wherein the human body reconstruction network based on the environmental parameters and the two-dimensional joint point coordinates S detect and the two-dimensional joint point coordinates J 2d Train the human body reconstruction network to obtain the trained human body reconstruction network, including: The local image of the human body area in the image is input into the human body reconstruction network ψ τ , get the posture parameter θ τ and body shape parameter β τ , τ represents the number of first iterations; According to the attitude parameter θ τ and body shape parameter β τ Reconstruct the 3D human body mesh model and obtain the 3D joint coordinates of the reconstructed human body Based on the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates Constructing a constraint expression of a reprojection matching relationship, and organizing the constraint expression of the reprojection matching relationship to obtain a first non-homogeneous equation group of an offset t of a three-dimensional human body relative to a camera center; Based on the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Constructing a constraint condition expression of the shadow consistency matching relationship, and sorting out the constraint condition expression of the shadow consistency matching relationship to obtain a second non-homogeneous equation group of the offset t of the three-dimensional human body relative to the camera center; After the first non-homogeneous equation group and the second non-homogeneous equation group are put in parallel, they are solved by least squares to obtain an offset t of the three-dimensional human body relative to the center of the camera; Combine the offset t and the environmental parameters to convert the three-dimensional joint point coordinates Project it to the shadow area to get the three-dimensional coordinates of the joint points in the shadow area And calculate the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Shadow consistency loss function between them; Calculate the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates The reprojection loss function between ; Based on the shadow consistency loss function and the reprojection loss function, the human body reconstruction network ψ τ Train and obtain the human body reconstruction network ψ τ+1 ; Until the first iteration round is completed, the trained human body reconstruction network is obtained; The local image img body Input the trained human body reconstruction network to obtain the initial posture parameter θ0 and the initial body shape parameter β0; Based on the environmental parameters, the two-dimensional joint point coordinates S detect , the two-dimensional joint point coordinates J 2d Optimizing the initial posture parameter θ0 and the initial body shape parameter β0 to obtain the optimal posture parameter θ and the optimal body shape parameter β; According to the optimal posture parameter θ and the optimal body shape parameter β, a three-dimensional human body mesh model is reconstructed.
2. The method according to claim 1, characterized in that The environmental parameters include: camera intrinsic parameter matrix K, light direction vector Ground Normal and translation amount d; the environmental parameters for generating the shadow include: After taking photos of the calibration chessboard, perform algorithm calibration to obtain the camera intrinsic parameter matrix K; Place the aruco Marker calibration plate on the ground and take a photo to calibrate the algorithm and obtain the ground normal vector and the translation d; Use the mirror reflection ball for calibration to obtain the light direction vector 3. The method according to claim 1, characterized in that The method is based on the environmental parameters, the two-dimensional joint point coordinates S detect , the two-dimensional joint point coordinates J 2d Optimizing the initial posture parameter θ0 and the initial body shape parameter β0 to obtain the optimal posture parameter θ and the optimal body shape parameter β includes: According to the attitude parameter θ c Reconstruct the 3D human body mesh model with the body shape parameter β to obtain the 3D joint coordinates of the reconstructed human body Wherein, c is the second iteration round; Based on the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates Constructing a constraint expression of a reprojection matching relationship, and organizing the constraint expression of the reprojection matching relationship to obtain a first non-homogeneous equation group of an offset t of a three-dimensional human body relative to a camera center; Based on the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Constructing a constraint condition expression of the shadow consistency matching relationship, and sorting out the constraint condition expression of the shadow consistency matching relationship to obtain a second non-homogeneous equation group of the offset t of the three-dimensional human body relative to the camera center; After the first non-homogeneous equation group and the second non-homogeneous equation group are put in parallel, they are solved by least squares to obtain an offset t of the three-dimensional human body relative to the center of the camera; Combine the offset t and the environmental parameters to convert the three-dimensional joint point coordinates Project it to the shadow area to get the three-dimensional coordinates of the joint points in the shadow area And calculate the two-dimensional joint point coordinates S detect With the three-dimensional joint point coordinates Shadow consistency loss function between them; Calculate the two-dimensional joint point coordinates J 2d With the three-dimensional joint point coordinates The reprojection loss function between ; By constraining the pose parameters θ c , body shape parameter β c With the attitude parameter θ c+1 , body shape parameter β c+1 The similarity of , and the regular term is obtained; Based on the shadow consistency loss function, the reprojection loss function and the regularization term, the posture parameter θ c and body shape parameter β c Optimize to get the posture parameter θ c+1 and body shape parameter β c+1 ; Until the second iteration round is completed, the optimal posture parameter θ and the optimal body shape parameter β are obtained.
4. The method according to any one of claims 1 to 3, characterized in that: The method of reconstructing a three-dimensional human body mesh model according to the optimal posture parameter θ and the optimal body shape parameter β comprises: Based on the SMPL human body parameter model, the optimal posture parameter θ and the optimal body shape parameter β are converted into three-dimensional human joint point coordinates J 3d ; According to the three-dimensional human joint point coordinates J 3d , reconstruct the 3D human body mesh model.
5. A method for reconstructing a three-dimensional human body from a single image based on shadow, the method comprising: Acquire an image without a shadow, and based on the scene of the image, acquire a sample image containing a shadow and sample environmental parameters of the shadow; Extract the local image img of the shadow area in the sample image shadow , and based on the local image img shadow , calculate the two-dimensional joint point coordinates S′ in the shadow area detect ; wherein the local image img of the shadow area in the sample image is extracted shadow , and based on the local image img shadow , calculate the two-dimensional joint point coordinates S′ in the shadow area detect ,include: Constructing a shadow contour detection network, wherein the shadow contour detection network adopts an MTMT convolutional neural network structure and uses network model parameters trained on the SBU and ISTD datasets as initialization; The sample image is input into the shadow contour detection network to obtain the local image imgshado of the shadow area w ; Constructing a shadow joint point estimation network, wherein the shadow joint point estimation network adopts a Stacked Hourglass convolutional neural network structure and uses shadow data synthesized on Human3.6M and standard values of joint point coordinates for supervised training; The local image img shadow Input the shadow joint point estimation network to obtain the two-dimensional joint point coordinates S′ of the shadow area detect ; Extract the local image img of the human body area in the sample image body , and based on the local image img body , calculate the two-dimensional joint point coordinates J′ in the human body area 2d ; wherein the local image img of the human body area in the sample image is extracted body , and based on the local image img body , calculate the two-dimensional joint point coordinates J′ in the human body area 2d ,include: Build a human body bounding box detection network; The sample image is input into the human body bounding box detection network to obtain the local image img of the human body area body ; Construct a two-dimensional human joint point detection network; The local image img body Input the two-dimensional human joint point detection network to obtain the two-dimensional joint point coordinates J′ in the human body area 2d ; Constructing a human body reconstruction network, wherein the human body reconstruction network adopts a SPIN network structure; Based on the sample environment parameters, the two-dimensional joint point coordinates S′ detect and the two-dimensional joint point coordinates J′ 2d Train the human body reconstruction network to obtain a trained human body reconstruction network; wherein the human body reconstruction network based on the sample environment parameters and the two-dimensional joint point coordinates S′ detect and the two-dimensional joint point coordinates J′ 2d Train the human body reconstruction network to obtain the trained human body reconstruction network, including: The local image of the human body area in the sample image is input into the human body reconstruction network to obtain the posture parameter θ τ and body shape parameter β τ , τ represents the number of first iterations; According to the attitude parameter θ τ and body shape parameter β τ Reconstruct the 3D human body mesh model and obtain the 3D joint coordinates of the reconstructed human body Based on the two-dimensional joint point coordinates J′ 2d With the three-dimensional joint point coordinates Constructing a constraint expression of a reprojection matching relationship, and organizing the constraint expression of the reprojection matching relationship to obtain a first non-homogeneous equation group of an offset t of a three-dimensional human body relative to a camera center; Based on the two-dimensional joint point coordinates S′ detect With the three-dimensional joint point coordinates Constructing a constraint condition expression of the shadow consistency matching relationship, and sorting out the constraint condition expression of the shadow consistency matching relationship to obtain a second non-homogeneous equation group of the offset t of the three-dimensional human body relative to the camera center; After the first non-homogeneous equation group and the second non-homogeneous equation group are put in parallel, the least square method is used to solve them to obtain the offset t of the three-dimensional human body relative to the camera center; Combine the offset t and the sample environment parameters to convert the three-dimensional joint point coordinates Project it to the shadow area to get the three-dimensional coordinates of the joint points in the shadow area And calculate the two-dimensional joint point coordinates S′ detect With the three-dimensional joint point coordinates Shadow consistency loss function between them; Calculate the two-dimensional joint point coordinates J′ 2d With the three-dimensional joint point coordinates The reprojection loss function between ; Based on the shadow consistency loss function and the reprojection loss function, the human body reconstruction network ψ τ Train and obtain the human body reconstruction network ψ τ+1 ; Until the first iteration round is completed, the trained human body reconstruction network is obtained; Inputting the local image of the human body region in the sample image into the trained human body reconstruction network to obtain the posture parameter θ and the body shape parameter β; A three-dimensional human body mesh model is reconstructed according to the posture parameter θ and the body shape parameter β.
6. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement any one of the methods of claims 1-4.