A drivable monocular multi-person reconstruction method based on human shape consistency

Abstract

The application discloses a drivable single-view multi-person reconstruction method based on human shape consistency. The application introduces a general feature voxel grid, an instance-related instance feature voxel grid and a multi-target fitting framework on the basis of volume rendering, and realizes simultaneous acquisition of multiple drivable three-dimensional digital persons in a scene in a short time. The general feature voxel grid fits the shape and texture features common between different human bodies, and the instance-related instance feature voxel grid fits the body state and texture differences between individuals. The surface regression module and prior human shape information are utilized to define the inclusion relationship between the sampling points and the human body surface, and the volume density of the points inside and outside the volume is additionally constrained through integrity. Through fusion of the general body state features and the instance-related detailed shape and texture features, and in combination with the introduction of the integrity loss based on the human shape prior, effective constraint on the human shape is realized, and the under-constrained problem of the human shape caused by insufficient visibility or sampling frequency is improved.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, and relates to human pose extraction and reconstruction, specifically to a driveable single-viewpoint multi-person reconstruction method based on human shape consistency. Background Technology

[0002] 3D digital human technology, as a convergence point of modern science and technology with information technology and computer science, has experienced rapid development in recent years. Entering the 21st century, with the rapid development of artificial intelligence technology and breakthroughs in deep learning, computer vision, and natural language processing, 3D digital human technology has ushered in unprecedented development opportunities, and demand has surged. VR / AR applications, the film industry, and game development, as leading industries heavily reliant on drivable, high-quality 3D digital humans, have extremely high demands for low-cost, mass-produced high-quality 3D digital humans. Furthermore, with the further emergence of concepts such as the metaverse, everyone will have a need for customized, exclusive 3D digital humans. Therefore, how to obtain video input from everyday devices and generate high-quality, drivable human bodies has become an important research direction.

[0003] Computer graphics employs various methods to represent 3D objects, each with specific use cases and advantages / disadvantages. These include, but are not limited to, point clouds, voxels, and polygon meshes. Polygon meshes, as the most commonly used 3D representation in the computer graphics industry, have, after extensive iterative improvements, accurately represented complex shapes and performed texture mapping, lighting calculations, and collision detection. However, they are difficult to optimize using deep learning gradient descent. Point clouds are unsuitable for direct rendering, and the representational quality of voxel representations is directly limited by the resolution of the voxel mesh. Research has found that traditional volume rendering techniques used for rendering non-solid objects such as smoke and clouds can be trained in conjunction with deep learning at a relatively low cost. Therefore, volume rendering techniques based on neural implicit representations have become the preferred technology for recent research on 3D driveable digital humans.

[0004] Existing human body rendering methods generally focus on improving the rendering quality of single-target human bodies and compressing rendering time. They impose strict limitations on the number of input viewpoints, the number of human bodies to be reconstructed, the visibility of human body parts, and the range of limb movements. Therefore, most existing methods only include a single reconstruction target with almost no obstacles, while also requiring a complete view of the human body in each frame. This limits the application of existing methods in consumer and personalized scenarios. With the decrease in the number of input viewpoints and varying degrees of occlusion on the human body, most existing methods will struggle to reconstruct continuous, high-fidelity 3D human bodies and maintain robustness in 3D digital human modeling during the driving process. In real-world scenarios, single-viewpoint video is a common and readily available input, and recovering a driveable human body representation from monocular video has always been a research hotspot. However, challenges remain regarding simultaneous reconstruction of multiple human bodies and recovery of occluded parts. This invention addresses these challenges by using sparse frames with good visibility and based on human body shape consistency. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a driveable single-viewpoint multi-person reconstruction method based on human body shape consistency.

[0006] This invention proposes a driveable single-viewpoint multi-person reconstruction framework based on volume rendering. The framework, based on the consistency of human body shape, proposes a learning process for general human body shape features and individual body posture and texture features, as well as a surface regression module with occlusion recovery and a human body integrity loss based on directed distance, thereby enabling the simultaneous acquisition of multiple driveable 3D digital humans in a monocular video. The general human body shape feature learning process is used to learn the common body posture and texture features of the human body, providing a basic human body shape for subsequent learning of non-rigid deformation of individual surfaces. The individual body posture and texture feature learning process is used to learn the individual body posture and texture features of each individual in the video, achieving detailed correction on the common body posture and texture features. The surface regression module with occlusion recovery relies on standard human body morphological information to derive the directed distance of 3D spatial points relative to the human body surface. Simultaneously, the directed distance of 3D spatial points is used to calculate the human body integrity loss to constrain the density of points inside and outside the human body, alleviating the problem of under-constraint of human body shape and texture caused by occlusion or insufficient sampling frequency.

[0007] A driveable single-viewpoint multi-person reconstruction method based on human body shape consistency, the specific steps of which are as follows:

[0008] Step S1: Establish a single-viewpoint multi-person video dataset;

[0009] Step S2: Generate sampling rays and sampling points based on the camera's intrinsic and extrinsic parameter matrices;

[0010] Step S3: Based on the Linear Blend Skinning algorithm and human pose, transform the 3D spatial sampling points from the observation space to the standard pose space (canonical space);

[0011] Step S4: Using a general feature voxel module and an individual feature voxel module, sample, interpolate, and encode the three-dimensional spatial sampling points in the standard pose space to obtain the fused feature vector;

[0012] Step S5: The fused feature vector is fed into the implicit representation decoding module to obtain the volume density and color information of the sampling points in the three-dimensional space, and the pixel color under the viewpoint to be drawn is obtained based on the volume rendering formula.

[0013] Step S6: Send the three-dimensional spatial sampling points into the surface regression module to obtain the directed distance between the three-dimensional spatial sampling points and the human body surface;

[0014] Step S7: Using the depth rendering module, the color information of different individuals is mixed into the output image of the viewpoint to be drawn;

[0015] Step S8: Constrain the generated quality, volume density of points inside and outside the human body, texture information, etc. in the model training by using the human body generation quality loss function.

[0016] The beneficial effects of this invention are as follows:

[0017] This invention provides a driveable single-viewpoint multi-person reconstruction method based on human body shape consistency.

[0018] This invention decouples the human body shape learning process into general human body shape feature learning and individual posture and texture feature learning. By introducing a general voxel grid and multiple individual voxel grids, a feature representation of a spatial sampling point is achieved, thus proposing a driveable multi-person reconstruction framework.

[0019] This invention proposes a surface regression module with occlusion recovery. By introducing human standard morphological information and the K-nearest neighbor algorithm, it calculates the directed distance of three-dimensional sampling points relative to the human standard morphological surface, thus assisting in the calculation of integrity loss.

[0020] This invention proposes a human body integrity loss function that uses directed distance constraints to make the density of points in the body tend to 1, thereby effectively reducing the occurrence of trunk cavities or limb detachment. Attached Figure Description

[0021] Figure 1 This is a flowchart of a driveable single-viewpoint multi-person reconstruction method based on human body shape consistency.

[0022] Figure 2 This is a flowchart of the surface regression module based on human priors.

[0023] Figure 3 This is a schematic diagram of the common structural and texture features of the human body in a scene learned by the universal voxel mesh of this invention.

[0024] Figure 4 This is a schematic diagram of the reconstruction result after combining individual-related voxel grid features according to the present invention.

[0025] Figure 5 This is the free viewpoint reconstruction effect in a multi-person scene according to the present invention. Detailed Implementation

[0026] The method of the present invention will be further explained and described below with reference to the accompanying drawings. Figure 1 and 2 The flowchart shown is a method for reconstructing multiple people from a single viewpoint based on human body shape consistency. The specific steps are as follows:

[0027] Step S1: Establish a single-viewpoint multi-person video dataset, the details of which are as follows:

[0028] Several panoramic cameras were deployed in an indoor scene. The cameras were calibrated to obtain the intrinsic parameter matrix (K) and extrinsic parameter matrix (E). Multiple people were positioned in the center of the panoramic cameras to record synchronized multi-viewpoint video of multiple people's actions.

[0029] Distortion correction is performed on the video from each viewpoint based on the intrinsic parameter matrix. The corrected video from each viewpoint is then input into a mask extraction network to extract the independent mask for each individual in the scene. The corrected video from each viewpoint is then input into a pose extraction network to extract the pose (θ) and body shape parameters (β) for each individual in the scene and to perform pose registration between viewpoints, thereby mitigating the jitter problem of the pose reference point caused by estimation errors to some extent.

[0030] The pose registration between viewpoints is achieved through the triangulation post-processing provided by EasyMocap.

[0031] The mask extraction networks mentioned include SegmentAnything, GroundedSAM, etc.

[0032] The pose extraction networks mentioned include VIBE, SPIN, EasyMocap, etc.

[0033] Vertex templates under the T-Pose are generated based on the SMPL human prior model and the estimated body shape parameters (β) for each individual, and are used for subsequent calculation of directed distances between spatial sampling points. The so-called T-Pose is the resting pose, which means that the rotation vectors of all joints are zero vectors.

[0034] Step S2: Generate sampling rays and sampling points based on the camera's intrinsic and extrinsic parameter matrices. Specific details are as follows:

[0035] First, the camera's intrinsic parameter matrix includes the principal point coordinates (c x ,c y ) and horizontal focal length f x and vertical focal length f y This determines the resolution of the image plane and the distance from the focal point to the image plane. The camera's extrinsic parameter matrix contains the camera's rotation (R) and displacement (T) information. Based on the camera's extrinsic parameter matrix and the world coordinate system origin determined by the calibration plate, the camera's position and orientation are determined. Starting from the camera's optical center, a sampling ray is formed by connecting specified pixels on the image plane. Uniform sampling is performed at equal intervals along this ray to generate sampling points.

[0036] This method optimizes the sampling point selection strategy to avoid unnecessary calculations as much as possible, as follows:

[0037] 1) Generate a 3D-BoundingBox based on the estimated individual pose joints, filter the sampling points, and retain only the sampling points inside the 3D-BoundingBox.

[0038] 2) Sample rays are generated only for pixels inside the effective area of ​​the individual mask and inside the projection area of ​​the 3D cube. Sampling is performed inside the individual mask with a probability of p (p = 0.9), and sampling is performed in the non-individual mask area inside the projection area of ​​the 3D cube with a probability of 1-p to reduce the impact of inaccurate individual masks.

[0039] Step S3: Based on the linear hybrid skinning algorithm and human pose, transform the 3D spatial sampling points from the observation space to the standard pose space. The specific details are as follows:

[0040] SMPL (Skinned Multi-Person Linear) is a skeleton-driven parametric human model commonly used in pose extraction. It learns a vertex-based additive human model from a large amount of human data, with the deformation of the vertex mesh controlled by shape components (β) and pose components (θ). Linear Blend Skinning (LBS) is a commonly used 3D model-driven algorithm in computer graphics, capable of transforming model vertex shapes from resting poses to arbitrary poses. In this method, the pose deformation network is denoted as F. dIn the individual pose extraction stage, rotation vectors of 24 joints under the SMPL skeleton standard are extracted with the parent node direction as the rotation axis. Based on this, the rotation matrix and displacement vector of each joint are obtained through the Rodrigues formula, thus constructing a 4×4 RT transformation matrix. In addition, the LBS algorithm needs to set a linear hybrid skinning weight value for each joint, representing the influence of the joint's transformation on the surface vertex. By applying the LBS formula in reverse, vertex transformation from any pose to a resting pose can be achieved, thus realizing motion reconstruction from any pose to a T-Pose. The inverse linear hybrid skinning (Inverse-LBS) formula is as follows: the LBS weight of the j-th joint is denoted as w. j The RT transformation matrix of the j-th joint is denoted as G. j Let x be any vertex.

[0041]

[0042] x'=F d (x,w)

[0043] Here, J represents the total number of joints, with a value of 24. By applying the skinning formula in reverse, the sampling point in any pose can be denoted as x and transformed to the corresponding point position in the resting pose, i.e., the surface vertex x'. Based on this point transformation, the appearance of the 3D digital human is reconstructed using the resting pose as the reference, thus decoupling the complete pose and appearance.

[0044] Step S4: Using a general feature voxel module and an individual feature voxel module, sample, interpolate, and encode the 3D spatial sampling points in the standard pose space to obtain the fused feature vector. Specific details are as follows:

[0045] The surface vertex x' corresponding to the resting pose obtained from the pose deformation network is normalized to the size space of the general voxel feature mesh and the individual voxel feature mesh, and then trilinear interpolation is performed to obtain the feature v. g and v i Voxel feature meshes model arbitrary 3D spatial points, enabling the encoding and conversion of features from arbitrary 3D spatial points to corresponding color and density information. Due to limitations in video memory, the resolution of voxel meshes is limited. Therefore, trilinear interpolation is introduced to interpolate low-resolution features, expanding the representation range. The voxel feature mesh is denoted as V; N. x N y N z The size of the voxel feature mesh is denoted as .

[0046]

[0047] Where v represents the feature output of the voxel feature grid, and is vg or v i ;interp represents the trilinear interpolation function, (x c ,y c ,z c () represents the three-dimensional spatial coordinates of the surface vertex x'. Represents the set of real numbers; C v N represents the number of channels in each voxel grid; x N y N z These represent the length of each dimension of the voxel grid.

[0048] This method decouples human body features into general human body features and individual features. General human body features model common shape and rough texture information that are independent of individuals in the scene, thus compensating for inaccuracies in shape and color caused by limited sampling angles or insufficient angle sampling frequency for certain individuals. Individual features model the unique posture and detailed texture information of each individual, serving as a detailed correction to the general human body features, thereby achieving rich feature representation.

[0049] The feature v obtained from sampling g and v i Sine and cosine position encoding is performed to preserve high-frequency detail information. The sine and cosine position encoding formulas are as follows:

[0050] γ(v)=(sin(2 0 v),cos(2 0 )v,...,sin(2 L-1 v),cos(2 L-1 )v)

[0051] Where γ(v) represents the sine and cosine positional encoding; v represents the input parameters, including the feature v g and v i L represents the hyperparameter; the final encoded complete feature vector v is obtained by concatenating the general human features and individual features after position encoding. f .

[0052] Step S5: The fused feature vector is fed into the implicit representation decoding module to obtain the volume density and color information of the 3D space sampling points, and the pixel color of the viewpoint to be drawn is obtained based on the volume rendering formula. The specific details are as follows:

[0053] The encoded complete feature vector v f The input is the implicit radiation field, i.e., the fully connected network decoder, denoted as Φ. r The decoding process is represented as follows:

[0054] c,σ,depth=Φ r(γ(v f ),γ(t),γ(x c ,y c ,z c ),γ(d))

[0055] Where c represents the color of the sampling point, σ represents the volume density of the sampling point, depth represents the estimated depth, and v f This represents the complete feature vector concatenated after position encoding, where t represents the frame time index in the frame sequence, (x c ,y c ,z c Let x' represent the spatial sampling point in the resting posture, and d represent the direction vector of the camera. After calculating the colors of all sampling points along a sampling ray r, the specific image plane pixel color is obtained using the volume rendering integral formula, as follows:

[0056]

[0057] in, The color of a pixel in the image plane is represented by N, the number of sampling points on ray r is represented by i, and σ represents the i-th sampling point. i δ represents the volume density of the i-th sampling point. i =x i+1 -x i c represents the distance between adjacent sampling points. i This represents the color information of the i-th sampling point.

[0058] Step S6: Input the 3D spatial sampling points into the surface regression module to obtain the directed distance between the 3D spatial sampling points and the human body surface. Specific details are as follows:

[0059] This invention introduces the vertex and normal models of the SMPL (Surface Nearest Particle Model) in the resting posture corresponding to the individual's body shape parameter (β) as a priori for human body shape. For any spatial sampling point, the K-nearest neighbor algorithm is used to determine k (k=5) the surface vertices closest to the spatial sampling point, and the normal vector of the directed distance judgment point on the surface is estimated. The specific calculation formula is as follows:

[0060]

[0061] in, v represents any spatial sampling point in a resting state. i' The i' nearest neighbor surface vertex, n i' Let i' be the normal vector of the i'-th nearest neighbor vertex on the surface. Representative vector and corresponding v i'The cosine similarity of the vertex normals. By calculating the cosine similarity as a weighted average of the nearest surface vertices and their normals, the directed distance to the surface is estimated to determine the normal vector. The directed distance d is obtained through a single vector dot product. sign The specific calculation formula is as follows:

[0062]

[0063] The obtained directed distance is used as the basis for determining the integrity loss in subsequent calculations.

[0064] Step S7: Using the depth rendering module, the color information of different individuals is blended into the output image of the viewpoint to be rendered. Specific details are as follows:

[0065] The pixel color is obtained by generating sampling rays from the corresponding image plane pixels of each individual and then performing volume rendering. And depth estimation. Due to the unsupervised nature of network depth estimation, many invalid depths appear at the edges of objects. Therefore, this method uses the bounding 3D cube of the SMPL model under the corresponding pose (P) and calculates the minimum and maximum values ​​of the effective depth based on the camera intrinsic and extrinsic parameter matrices. Invalid depths are filtered out, and the same pixels are sorted according to depth order. The images are rearranged to obtain the output image.

[0066] Step S8: By using the human body generation quality loss function, constraints are imposed on the generation quality, volume density of points inside and outside the human body, texture information, etc. during model training. Specific details are as follows:

[0067] This method uses three loss functions, including mean squared error. (Mean Squared Error), Learning-Aware Image Patch Similarity Loss (Learned Perceptual Image Patch Similarity) and loss of human integrity Effective constraints enable the model to converge effectively.

[0068] Human body integrity loss is used to constrain the volume density of sampling points within the human body to approach 1, thereby effectively improving limb dislocation and voiding phenomena. Integrity loss is denoted as... The specific definitions are as follows:

[0069]

[0070] Here, ReLU is the activation function, σ is the volume density at the corresponding point, and α (α = 10) is the hyperparameter. This method uses ReLU to truncate negative volume densities and tends to reduce the penalty coefficient for high volume densities, thereby effectively constraining the volume density within an individual.

[0071] The overall loss function used for training is as follows:

[0072]

[0073] λ1, λ2, and λ3 are set to 0.2, 1, and 5e-5, respectively. In the first 300 iterations of training, λ2 and λ3 are set to 0, taking advantage of the mean square error's ability to promote rapid convergence of most color blocks, thus accelerating the model's early convergence process. After 300 iterations, the weights of λ2 and λ3 are restored, focusing on fitting the texture details of the color blocks while minimizing the occurrence of gaps and disconnections in the limbs.

[0074] Figure 3 The common human body shape and rough texture features learned by the universal voxel mesh are shown. It can be seen that the universal voxel mesh has learned the general human body shape and general texture features, which can serve as general information for undersampling of some individuals. Figure 4 The reconstruction results, which incorporate individual voxel mesh features, are shown, and it can be observed that personalized details have been added.

[0075] Figure 5 It demonstrates the free viewpoint reconstruction effect in a multi-person scene.

[0076] As shown in the figure above, loss of human body integrity can effectively improve the reconstruction effect, and the undersampling problem caused by occlusion can be alleviated by using standard human body morphological information.

[0077] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0078] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A driveable single-viewpoint multi-person reconstruction method based on human body shape consistency, characterized in that, Includes the following steps: Step S1: Establish a monocular multi-person 3D reconstruction dataset; Step S2: Generate sampling rays and sampling points based on the camera's intrinsic and extrinsic parameter matrices; Step S3: Based on the linear hybrid skinning algorithm and pose, transform the 3D spatial sampling points from the observation space to the standard space; Step S4: Simultaneously sample and interpolate on the general feature voxel grid and the instance-related instance feature voxel grid, and then perform position encoding to obtain the fused feature vector; Step S5: Input the implicit representation decoding module to obtain the volume density and color information of the corresponding sampling point, and obtain the color of the final image plane pixel based on the volume rendering formula; Step S6: The sampling points are sent to the surface regression module to obtain the distance between the sampling points and the human body surface; Step S7: Constrain the volume density of points inside and outside the body through integrity loss; Step S8: Use the depth rendering module to blend the color information of different instances into the output image; Step S9: Constrain the color information through image loss; The specific details of step S2 are as follows: 2-1. First, the camera's intrinsic parameter matrix includes the principal point coordinates (c... x ,c y ) and horizontal focal length f x and vertical focal length f y This determines the resolution of the image plane and the distance from the focal point to the image plane. The camera's extrinsic parameter matrix contains the camera's rotation and displacement information. Based on the camera's extrinsic parameter matrix and the world coordinate system origin determined by the calibration plate, the camera's position and orientation are determined. Starting from the camera's optical center, the line connecting a certain pixel in the image plane is the direction of a sampling ray. Uniform sampling is performed on the ray at equal intervals to generate sampling points. 2-2. Optimize the sampling points as follows: 1) Generate a 3D cube of the bounding structure based on the key points of the estimated instance pose, and filter the sampling points to retain only the sampling points inside the 3D cube; 2) Only generate sampling rays for pixels inside the effective area of ​​the instance mask and the projection area of ​​the 3D cube, and sample with probability p inside the instance mask and with probability 1-p in the non-instance mask area inside the projection area of ​​the 3D cube to reduce the impact of inaccurate instance mask. The specific details of step S3 are as follows: The pose deformation network is denoted as F. d In the instance pose extraction stage, the rotation vectors of 24 joints under the SMPL skeleton standard with the parent node direction as the rotation axis are extracted, and the rotation matrix and displacement vector of each joint are obtained by Rodriguez formula, thereby constructing a 4×4 RT transformation matrix. The LBS algorithm requires setting a weight value for the linear blended skin for each joint, representing the influence of the joint's transformation on the surface vertices. By applying the LBS formula in reverse, it achieves vertex transformation from any pose to a resting pose, thus realizing motion reconstruction from any pose to a T-pose. Let w be the LBS weight of the j-th joint. j The RT transformation matrix of the j-th joint is denoted as G. j Let x be any vertex, then: x'=F d (x,w) Where J represents the total number of joints, with a value of 24; by applying the skinning formula in reverse, the sampling point under any posture is recorded as x, and transformed to the corresponding point position under the resting posture, i.e., the surface vertex x'; based on this point transformation, the appearance reconstruction of the three-dimensional digital human is carried out with the resting posture as the reference, and the complete posture and appearance are decoupled. The specific details of step S6 are as follows: We introduce the vertex and normal models of the SMPL (Surface Nearest Particle Model) in the resting pose of the corresponding instance body shape parameter β as a priori for human body shape. For any spatial sampling point, we use the K-nearest neighbor algorithm to determine the k closest surface vertices to the spatial sampling point and estimate the normal vector of the surface's directed distance decision point. The specific calculation formula is as follows: in, v represents any spatial sampling point in a resting state. i' The i' nearest neighbor surface vertex, n i' Let i' be the normal vector of the i'-th nearest neighbor vertex on the surface. Representative vector and corresponding v i' The cosine similarity of the vertex's normal vector; by calculating the cosine similarity as a weighted average of the nearest surface vertices and their normal vectors, the directed distance to the estimated surface is determined, and the normal vector is determined. The directed distance d can be easily obtained through a single vector dot product. sign The specific calculation formula is as follows: The obtained directed distance is used as the basis for determining the integrity loss in subsequent calculations; The specific details of step S7 are as follows: Integrity loss is used to constrain the volume density of sampling points within the human body to approach 1, thereby effectively improving limb dislocation and artifact phenomena. Integrity loss is denoted as... The specific definitions are as follows: Where ReLU is the activation function, σ is the volume density at the corresponding point, and α is the hyperparameter. This method uses ReLU to truncate negative volume densities and tends to reduce the penalty coefficient for high volume densities, thereby effectively constraining the volume density.

2. The method for reconstructing multiple people from a single viewpoint based on human body shape consistency according to claim 1, characterized in that... Step S1 is as follows: Several panoramic cameras were set up in an indoor scene. The cameras were calibrated to obtain the intrinsic and extrinsic parameter matrices. Several dancers were placed in the center of the panoramic cameras to record synchronous multi-viewpoint multi-person action videos. Based on video data instances, pose, body shape parameters, and independent masks are extracted. Based on the intrinsic parameter matrix, distortion correction is performed on the video of each viewpoint. The corrected video of each viewpoint is input into the mask extraction network to extract the independent mask of each instance in the scene. The corrected video of each viewpoint is then input into the pose extraction network to extract the pose θ and body shape parameter β of each instance in the scene and to perform pose registration between viewpoints. Based on the SMPL human prior model and the estimated body shape parameter β for each instance, a vertex template under T-Pose is generated for subsequent calculation of the directed distance of spatial sampling points.

3. The method for reconstructing multiple people from a single viewpoint based on human body shape consistency according to claim 2, characterized in that, The specific details of step S4 are as follows: The surface vertex x' corresponding to the resting pose obtained from the pose deformation network is normalized to the size space of the general voxel feature mesh and the instance voxel feature mesh, and then trilinear interpolation is performed to obtain the feature v. g and v i Voxel feature meshes model arbitrary 3D spatial points, realizing the encoding conversion from arbitrary 3D spatial points to corresponding color and density information features. Trilinear interpolation is introduced to interpolate low-resolution features, thereby expanding the expression range; where the voxel feature mesh is denoted as V. Where v represents the feature output of the voxel feature grid, and is v g or v i ;interp represents the trilinear interpolation function, (x c ,y c ,z c () represents the three-dimensional spatial coordinates of the surface vertex x'. Represents the set of real numbers; C v N represents the number of channels in each voxel grid; x N y N z These represent the length of each dimension of the voxel grid; The feature v obtained from sampling g and v i Sine and cosine position encoding is performed to preserve high-frequency detail information. The sine and cosine position encoding formulas are as follows: γ(v)=(sin(2 0 v),cos(2 0 )v,...,sin(2 L-1 v),cos(2 L-1 )v) Where γ(v) represents the sine and cosine positional encoding; v represents the input parameters, including the feature v g and v i L represents the hyperparameter; the final encoded complete feature vector v is obtained by concatenating the general human features encoded at location and the instance-related human features. f .

4. The method for reconstructing multiple people from a single viewpoint based on human body shape consistency according to claim 3, characterized in that, The specific details of step S5 are as follows: The encoded complete feature vector v f The input is the implicit radiation field, i.e., the fully connected network decoder, denoted as Φ. r The decoding process is represented as follows: c,σ,depth=Φ r (c(v f ),γ(t),γ(x c ,y c ,z c ),c(d)) Where c represents the color of the sampling point, σ represents the volume density of the sampling point, depth represents the estimated depth, and v f This represents the complete feature vector concatenated after position encoding, where t represents the frame time index in the frame sequence, (x c ,y c ,z c () represents the spatial sampling point x' in the resting posture, and d represents the direction vector of the camera orientation; after calculating the colors of all sampling points of a certain sampling ray r, the specific image plane pixel color is obtained through the volume rendering integral formula, as follows: in, The color of a pixel in the image plane is represented by N, the number of sampling points on ray r is represented by i, and σ represents the i-th sampling point. i δ represents the volume density of the i-th sampling point. i =x i+1 -x i c represents the distance between adjacent sampling points. i This represents the color information of the i-th sampling point.

5. The method for reconstructing multiple people from a single viewpoint based on human body shape consistency according to claim 4, characterized in that, The specific details of step S8 are as follows: The corresponding image plane pixels of each human body instance generate sampling rays, which are then processed by volume rendering to obtain... And depth estimation; due to the unsupervised nature of network depth estimation, many invalid depths will appear at the edges of objects; therefore, the bounding 3D cube of the SMPL model under the corresponding pose P is used, and the minimum and maximum values ​​of the effective depth are calculated based on the camera intrinsic and extrinsic parameters. Invalid depths are filtered out, and the same pixels are sorted according to depth. The images are rearranged to obtain the output image.

6. The method for reconstructing multiple people from a single viewpoint based on human body shape consistency according to claim 5, characterized in that, The specific details of step S9 are as follows: Three loss functions were used, including mean squared error. Learning perceptual image patch similarity loss and integrity loss The effective constraint model achieves effective convergence, and the specific loss function obtained is as follows: λ1, λ2, and λ3 are set to 0.2, 1, and 5e-5, respectively. In the first 300 iterations of training, λ2 and λ3 are set to 0 to take advantage of the characteristic that mean square error can promote the rapid convergence of most color blocks and accelerate the early convergence process of the model. After 300 iterations, the weight values ​​of λ2 and λ3 are restored.

Images