Drivable single-viewpoint multi-person reconstruction method based on human body shape consistency

Abstract

The invention discloses a drivable single-viewpoint multi-person reconstruction method based on human body shape consistency. According to the method, on the basis of volume rendering, a general feature voxel grid, an instance feature voxel grid related to an instance and a multi-target fitting framework are introduced, and a plurality of drivable three-dimensional digital humans in a scene can be obtained at the same time in a short time. The general feature voxel grids fit common shape and texture features among different human bodies, and the instance feature voxel grids related to instances fit posture and texture differences among individuals. A surface regression module and prior human body shape information are utilized to define an inclusion relation between sampling points and a human body surface, and the volume density of in-vivo and in-vitro points is subjected to additional constraint through integrity. By fusing general posture features and detail shapes and texture features related to examples, integrity loss is introduced in combination with human body shape prior to realize effective constraint on the human body shape, and the problem that the human body shape is under-constrained due to insufficiency of visibility or sampling frequency is solved.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional reconstruction, relates to human body posture extraction and reconstruction, and specifically relates to a drivable single-viewpoint multi-person reconstruction method based on human body shape consistency. Background Art

[0002] As the intersection of modern science and technology, information technology and computer science, 3D digital human technology has developed rapidly in recent years. Entering the 21st century, with the rapid development of artificial intelligence technology, with the help of breakthroughs in deep learning, computer vision and natural language processing, 3D digital human technology has ushered in unprecedented development opportunities and demand has also shown a blowout growth. As leading industries that rely heavily on high-quality 3D digital humans, VR / AR applications, film industry, game production and other fields have extremely high demands for low-cost, batch-generated high-quality 3D digital humans. And with the further birth of concepts such as the metaverse, everyone will have the need to generate customized exclusive 3D digital humans. Therefore, how to obtain video input from everyday devices and generate high-quality drivable human bodies has become an important research direction.

[0003] There are many ways to represent three-dimensional objects in computer graphics, each with specific usage scenarios and advantages and disadvantages, including but not limited to point clouds, voxels, and polygonal meshes. Among them, polygonal meshes are the most commonly used three-dimensional representation method in the computer CG industry. After a long period of iterative improvement, they can accurately represent complex shapes and perform texture mapping, lighting calculations, and collision detection, but they are difficult to optimize using deep learning gradient descent. The point cloud method is not suitable for direct rendering, and the representation effect of the voxel representation method is directly limited by the resolution size of the voxel grid. Studies have found that traditional volume rendering technology used to render non-solid objects such as smoke and clouds can be combined with deep learning for training in a relatively low-cost manner. Therefore, volume rendering technology based on neural implicit representation has become the preferred technical solution for recent three-dimensional driveable digital human research.

[0004] Existing human body rendering methods generally focus on how to improve the quality of single-target human body rendering and compress rendering time. There are strict restrictions on the number of input viewpoints, the number of human bodies to be reconstructed, the visibility of human body parts, and the amplitude of limb movement. Therefore, most existing methods only contain a single reconstruction target and almost no obstacles. At the same time, they also require that a complete human body view can be seen in each frame, which limits the application of existing methods in consumer and personalized scenarios. With the decrease in the number of input viewpoints and the varying degrees of occlusion of the human body, most existing methods will find it difficult to reconstruct a continuous, high-fidelity three-dimensional human body and ensure the robustness of three-dimensional digital human modeling during the driving process. In real scenes, single-viewpoint video is a common and easy-to-obtain input. Restoring a drivable human body representation from a monocular video has always been a current research hotspot. However, it faces the challenges of simultaneous reconstruction of multiple people and recovery of occluded parts. The present invention solves the above challenges based on the consistency of human body shape and using sparse frames with good visibility. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a drivable single-viewpoint multi-person reconstruction method based on human body shape consistency.

[0006] The present invention proposes a drivable single-viewpoint multi-person reconstruction framework based on volume rendering. According to the consistency of human body shape, the framework proposes a learning process of general human body shape features and individual posture and texture features, as well as a surface regression module with occlusion recovery and human body integrity loss based on directed distance, so as to achieve simultaneous acquisition of multiple drivable three-dimensional digital people in monocular video. The general human body shape feature learning process is used to learn the common posture and common texture features of the human body, providing a basic human body shape for the subsequent learning of non-rigid deformation of individual surface. The individual posture and texture feature learning process is used to learn the individual posture and individual texture features of each individual in the video, and realize the detail correction on the common posture and common texture features. The surface regression module (Surface Regressor) with occlusion recovery relies on the standard morphological information of the human body to derive the directed distance of the three-dimensional space point relative to the human body surface. At the same time, the directed distance of the three-dimensional space point is used to calculate the human body integrity loss to constrain the density of the internal and external points of the human body, and alleviate the problem of under-constraint of human body shape and texture caused by occlusion or insufficient sampling frequency.

[0007] A drivable single-viewpoint multi-person reconstruction method based on human body shape consistency, the specific steps are as follows:

[0008] Step S1: Establish a single-viewpoint multi-person video dataset;

[0009] Step S2: Generate sampling rays and sampling points based on the camera intrinsic parameter and extrinsic parameter matrix;

[0010] Step S3: transforming the three-dimensional space sampling points from the observation space to the canonical space based on the linear blend skinning algorithm and the human body posture;

[0011] Step S4: using a general feature voxel module and an individual feature voxel module, sampling, interpolating and position encoding the three-dimensional space sampling points in the standard posture space to obtain a fused feature vector;

[0012] Step S5: sending the fused feature vector to the implicit representation decoding module to obtain the volume density and color information of the three-dimensional space sampling point, and obtaining the color of the pixel point under the viewpoint to be drawn based on the volume rendering formula;

[0013] Step S6: sending the three-dimensional space sampling points to the surface regression module to obtain the directed distance between the three-dimensional space sampling points and the human body surface;

[0014] Step S7: Mixing the color information of different individuals into the output image of the viewpoint to be drawn through the depth rendering module;

[0015] Step S8: The mass loss function of the human body is generated to realize the constraints on the generated mass, the body density of the internal and external points of the human body, the texture information, etc. in the model training.

[0016] The beneficial effects of the present invention are as follows:

[0017] The invention provides a drivable single-viewpoint multi-person reconstruction method based on human body shape consistency.

[0018] The present invention decouples the human body shape learning process into general human body shape feature learning and individual posture and texture feature learning. By introducing a general voxel grid and multiple individual voxel grids, the feature expression of a certain spatial sampling point is jointly realized, and a drivable multi-person reconstruction framework is proposed.

[0019] The present invention proposes a surface regression module with occlusion recovery, which calculates the directed distance of three-dimensional space sampling points relative to the standard human body morphology surface by introducing human body standard morphology information and K nearest neighbor algorithm, and assists in integrity loss calculation.

[0020] The present invention proposes a human body integrity loss function, which uses a directed distance constraint to make the point density in the body tend to 1, thereby effectively reducing the occurrence of trunk holes or limb separation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Flowchart of the drivable single-viewpoint multi-person reconstruction method based on human shape consistency.

[0022] Figure 2 Flowchart of the surface regression module based on human body prior.

[0023] Figure 3 This is a schematic diagram of the common structural features and texture features of the human body in the scene learned by the universal voxel grid of the present invention.

[0024] Figure 4 It is a schematic diagram of the reconstruction result after combining the individual related voxel grid features of the present invention.

[0025] Figure 5 This is the free viewpoint reconstruction effect in a multi-person scene of the present invention. DETAILED DESCRIPTION

[0026] The method of the present invention is further explained below with reference to the accompanying drawings. Figure 1 and 2 As shown in FIG. 1 , a flowchart of a drivable single-viewpoint multi-person reconstruction method based on human body shape consistency is shown in FIG. 1 . The specific steps are as follows:

[0027] Step S1: Establish a single-viewpoint multi-person video dataset. The details are as follows:

[0028] Several ring-shot synchronous cameras are arranged in indoor scenes, and the cameras are calibrated to obtain the intrinsic parameter matrix (K) and the extrinsic parameter matrix (E). Multiple people are located in the center of the ring-shot camera to record synchronous multi-viewpoint multi-person action videos.

[0029] Based on the intrinsic parameter matrix, the video of each viewpoint is distorted and corrected, and the corrected video of each viewpoint is input into the mask extraction network to extract the independent mask of each individual in the scene. The corrected video of each viewpoint is then input into the posture extraction network to extract the posture (θ) and body shape parameters (β) of each individual in the scene and perform posture registration between viewpoints, thereby avoiding the problem of posture reference point jitter caused by estimation errors to a certain extent.

[0030] The posture registration between viewpoints is achieved through the triangulation post-processing process provided by EasyMocap.

[0031] The mask extraction network includes SegmentAnything, GroundedSAM, etc.

[0032] The posture extraction network includes VIBE, SPIN, EasyMocap and the like.

[0033] Based on the SMPL human prior model and the estimated body shape parameters (β) of each individual, a vertex template under T-Pose is generated for the subsequent calculation of the directed distance of the spatial sampling points. The so-called T-Pose is a resting pose, which means that the rotation vectors of all joints are 0 vectors.

[0034] Step S2: Generate sampling rays and sampling points based on the camera intrinsic and extrinsic parameter matrices. The details are as follows:

[0035] First, the camera's intrinsic parameter matrix contains the principal point coordinates (c x ,c y ) and horizontal focal length f x and vertical focal length f y , in order to determine the resolution of the image plane and the distance of the focus from the image plane. The camera's extrinsic matrix contains the camera's rotation (R) and displacement (T) information. The camera's position and orientation are determined based on the camera's extrinsic matrix and the origin of the world coordinate system determined by the calibration plate. Taking the camera's optical center as the starting point, the line connecting the pixels specified in the image plane is the direction of a sampling ray, and uniform sampling is performed on the ray at equal intervals to generate sampling points.

[0036] This method avoids unnecessary calculations as much as possible for the optimized sampling point screening strategy, as follows:

[0037] 1) Generate a 3D cube (3D-BoundingBox) of the bounding structure based on the estimated joint points of the individual posture, filter the sampling points, and only retain the sampling points inside the 3D cube.

[0038] 2) Sampling rays are generated only for pixels inside the valid area of ​​the individual mask within the 3D cube and inside the projection area of ​​the 3D cube, and sampling is performed inside the individual mask with probability p (p=0.9), and sampling is performed in the non-individual mask area inside the projection area of ​​the 3D cube with probability 1-p to reduce the impact of inaccurate individual masks.

[0039] Step S3: Based on the linear hybrid skinning algorithm and the human body posture, the three-dimensional space sampling points are transformed from the observation space to the standard posture space. The specific details are as follows:

[0040] SMPL (Skinned Multi-Person Linear) is a parametric human body model driven by a skeleton commonly used in pose extraction. It learns a vertex-based additive human body model from a large amount of human body data, and controls the deformation of the vertex mesh by shape parameters (shape components, β) and pose parameters (pose components, θ). Linear Blend Skinning is a three-dimensional model-driven algorithm commonly used in the field of computer graphics. It can realize the transformation of model vertices from resting pose to arbitrary pose, which is denoted as LBS algorithm. In this method, the pose deformation network (Pose Deformation Network) is denoted as F dIn the individual posture extraction stage, the rotation vectors of the 24 joints under the SMPL skeleton standard with the parent node direction as the rotation axis are extracted, and based on this, the rotation matrix and displacement vector of each joint are further obtained through the Rodrigues formula (Rodrigues), thereby constructing a 4×4 RT transformation matrix. In addition, the LBS algorithm needs to set the weight value of the linear blend skinning for each joint point, which represents the weight of the influence of the transformation of the joint point on the surface vertex. By applying the LBS formula in reverse, the vertex transformation from any posture to the resting posture can be achieved, thereby realizing the motion restoration from any posture to T-Pose. The formula for inverse linear blend skinning (Inverse-LBS) is as follows: The LBS weight of the j-th joint point is denoted as w j , the RT transformation matrix of the jth joint point is recorded as G j . Let x be an arbitrary vertex.

[0041]

[0042] x'=F d (x,w)

[0043] Among them, J represents the total number of joint points, which is 24. By using the skinning formula in reverse, the sampling point in any posture can be recorded as x and transformed to the corresponding point position in the resting posture, that is, the surface vertex x'. Based on this point transformation, the appearance of the 3D digital human is reconstructed with the resting posture as the benchmark, and the posture and appearance are completely decoupled.

[0044] Step S4: Using the general feature voxel module and the individual feature voxel module, the three-dimensional space sampling points in the standard posture space are sampled, interpolated and position-encoded to obtain a fused feature vector. The specific details are as follows:

[0045] The surface vertex x' corresponding to the resting posture obtained by the posture deformation network part is normalized to the size space of the general voxel feature grid and the individual voxel feature grid, and trilinear interpolation sampling is performed to obtain the feature v g and v i The voxel feature grid models any 3D space point and realizes the encoding conversion from any 3D space point to the corresponding color and density information. Due to the limited video memory space problem, the resolution of the voxel grid is limited. The trilinear interpolation method is introduced to interpolate the low-resolution features to expand the expression range. The voxel feature grid is denoted as V; N x 、N y 、N z Denote as the size of the voxel feature grid.

[0046]

[0047] Where v represents the feature of the voxel feature grid output, which is vg or v i ; interp represents the trilinear interpolation function, (x c ,y c ,z c ) represents the three-dimensional space coordinate value of the surface vertex x', represents the set of real numbers; C v Indicates the number of channels per voxel grid; N x ,N y ,N z Represents the length of each dimension of the voxel grid.

[0048] This method decouples human features into universal human features and individual features. Universal human features model the common shapes and rough texture information that are irrelevant to individuals in the scene, so as to make up for the inaccurate shapes and colors of some individuals due to limited sampling angles or insufficient angle sampling frequency. Individual features model the unique posture and detailed texture information of individuals as a detailed correction to universal human features, thereby achieving rich feature expression.

[0049] Sampling to get feature v g and v i Sin and cosine position encoding is performed to retain high-frequency detail information. The sin and cosine position encoding formula is as follows:

[0050] γ(v)=(sin(2 0 v),cos(2 0 )v,...,sin(2 L-1 v),cos(2 L-1 )v)

[0051] Among them, γ(v) represents the sine and cosine position encoding; v represents the input parameters, including the feature v g and v i , L represents a hyperparameter; the general human features and individual features after position encoding are concatenated to obtain the final encoded complete feature vector v f .

[0052] Step S5: Send the fused feature vector to the implicit representation decoding module to obtain the volume density and color information of the three-dimensional space sampling point, and obtain the pixel color under the viewpoint to be drawn based on the volume rendering formula. The specific details are as follows:

[0053] The encoded complete feature vector v f Send it to the implicit radiation field (Radiance Field), that is, the fully connected network decoder, denoted as Φ r , the decoding process is expressed as:

[0054] c,σ,depth=Φ r(γ(v f ),γ(t),γ(x c ,y c ,z c ),γ(d))

[0055] Among them, c represents the color of the sampling point, σ represents the volume density of the sampling point, depth represents the estimated depth, and v f represents the complete feature vector after position encoding, t represents the frame time sequence in the frame sequence, (x c ,y c ,z c ) represents the spatial sampling point x' in the resting posture, and d represents the direction vector of the camera. After calculating the colors of all sampling points of a sampling ray r, the specific image plane pixel color is obtained through the volume rendering integral formula. The specific formula is as follows:

[0056]

[0057] in, represents the color of the pixel on the image plane, N represents the number of sampling points on the ray r, i represents the i-th sampling point, σ i represents the volume density of the i-th sampling point, δ i =x i+1 -x i Represents the distance between adjacent sampling points, c i Represents the color information of the i-th sampling point.

[0058] Step S6: Send the three-dimensional space sampling points to the surface regression module to obtain the directed distance between the three-dimensional space sampling points and the human body surface. The specific details are as follows:

[0059] The present invention introduces the vertex model and normal model of SMPL in the resting posture corresponding to the individual body shape parameter (β) as the human body shape prior. For any spatial sampling point, the K nearest neighbor algorithm is used to determine the k (k=5) surface vertices closest to the spatial sampling point, and the normal vector of the surface directed distance judgment point is estimated. The specific calculation formula is as follows:

[0060]

[0061] in, represents any spatial sampling point in the resting posture, v i' The i'th nearest neighbor surface vertex, n i' represents the normal vector of the i'th nearest neighbor surface vertex, Representation vector and corresponding to v i'The cosine similarity of the normal vector of the vertex. By calculating the cosine similarity as the weighted weight of the neighboring surface vertices and normal vectors, the estimated surface directed distance determination normal vector is obtained. The directed distance d is obtained by a vector dot product sign , the specific calculation formula is as follows:

[0062]

[0063] The obtained directed distance is used as the basis for determining the subsequent calculation of integrity loss.

[0064] Step S7: Through the depth rendering module, the color information of different individuals is mixed into the output image of the viewpoint to be drawn. The specific details are as follows:

[0065] The corresponding image plane pixels of each individual generate sampling rays and then get pixel colors after volume rendering. As well as estimating the depth depth. Due to the unsupervised nature of the network depth estimation, there will be a lot of invalid depths at the edge of the object. Therefore, this method uses the bounding 3D cube of the SMPL model under the corresponding pose (P) and calculates the minimum and maximum values ​​of the effective depth based on the camera intrinsic parameter matrix and the extrinsic parameter matrix. It filters out the invalid depth and sorts the same pixels in the order of depth depth. Rearrange to obtain the output image.

[0066] Step S8: The human body is generated through a mass loss function to realize constraints on the generated mass, body density of points inside and outside the human body, texture information, etc. in model training. The specific details are as follows:

[0067] This method uses three loss functions including mean square error (Mean Squared Error), learning-aware image patch similarity loss (Learned Perceptual Image Patch Similarity) and loss of human integrity Effectively constrain the model for efficient convergence.

[0068] The human body integrity loss is used to constrain the density of sampling points in the human body to approach 1, thereby effectively improving the limb separation and cavity phenomenon. The integrity loss is recorded as The specific definitions are as follows:

[0069]

[0070] Among them, ReLU is the activation function, σ is the volume density of the corresponding point, and α (α = 10) is a hyperparameter. This method uses ReLU to truncate negative volume density and tends to reduce the penalty coefficient for high volume density, thereby effectively constraining the volume density within an individual.

[0071] The comprehensive loss function used in training is as follows:

[0072]

[0073] λ1, λ2 and λ3 are set to 0.2, 1, and 5e-5 respectively. In the first 300 iterations of training, λ2 and λ3 are set to 0. The mean square error can promote the rapid convergence of most color blocks and accelerate the early convergence process of the model. After 300 iterations, the weight values ​​of λ2 and λ3 are restored, focusing on fitting the texture details of the color blocks, while avoiding the appearance of limb holes and disconnection as much as possible.

[0074] Figure 3 The common human body shape and rough texture features learned by the universal voxel grid are shown. It can be seen that the universal voxel grid has learned the universal human body shape and universal texture features, which can be used as universal information for undersampling of some individuals. Figure 4 The reconstruction results after combining the individual voxel grid features are shown, and it can be observed that the personalized details are supplemented.

[0075] Figure 5 The free viewpoint reconstruction effect in multi-person scenes is demonstrated.

[0076] It can be seen from the above figure that the loss of human body integrity can effectively improve the reconstruction effect and alleviate the undersampling problem caused by occlusion through the standard human morphological information.

[0077] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0078] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A drivable single-viewpoint multi-person reconstruction method based on human body shape consistency, characterized in that: The steps include: Step S1: Establish a monocular multi-person 3D reconstruction dataset; Step S2: Generate sampling rays and sampling points based on the camera intrinsic parameter and extrinsic parameter matrix; Step S3: transforming the three-dimensional space sampling points from the observation space to the standard space based on the linear hybrid skinning algorithm and the posture; Step S4: sampling and interpolating the general feature voxel grid and the instance feature voxel grid related to the instance at the same time, and then performing position encoding to obtain a fused feature vector; Step S5: sending the implicit representation decoding module to obtain the volume density and color information of the corresponding sampling point, and obtaining the color of the final image plane pixel based on the volume rendering formula; Step S6: the sampling points are sent to the surface regression module to obtain the distance between the sampling points and the human body surface; Step S7: constraining the volume density of points inside and outside the body through integrity loss; Step S8: Mixing the color information of different instances into the output image through the depth rendering module; Step S9: Constrain the color information through image loss.

2. The method for drivable single-viewpoint multi-person reconstruction based on human body shape consistency according to claim 1, characterized in that Step S1 is specifically as follows: Several ring-shot synchronous cameras are arranged in the indoor scene, and the cameras are calibrated to obtain the intrinsic parameter matrix and the extrinsic parameter matrix. Several dancers are placed in the center of the ring-shot cameras to record synchronous multi-viewpoint multi-person action videos. Based on the extracted posture, body shape parameters and independent masks of the video data instance, the video of each viewpoint is distorted and corrected based on the internal parameter matrix, and the corrected videos of each viewpoint are input into the mask extraction network to extract the independent mask of each instance in the scene; the corrected videos of each viewpoint are further input into the posture extraction network to extract the posture θ and body shape parameters β of each instance in the scene and perform posture registration between viewpoints; Based on the SMPL human prior model and the estimated body shape parameter β of each instance, a vertex template under T-Pose is generated for subsequent calculation of the directed distance of the spatial sampling points.

3. The method for drivable single-viewpoint multi-person reconstruction based on human body shape consistency according to claim 1, characterized in that: The specific details of step S2 are as follows: 2-1. First, the camera's intrinsic parameter matrix contains the principal point coordinates (c x ,c y ) and horizontal focal length f x and vertical focal length f y , in order to determine the resolution of the image plane and the distance of the focal point from the image plane; the camera's extrinsic matrix contains the camera's rotation and displacement information. According to the camera's extrinsic matrix, the camera's position and orientation are determined based on the origin of the world coordinate system determined by the calibration plate. The camera's optical center is used as the starting point, and the line connecting a certain pixel on the image plane is used as the direction of a sampling ray. Evenly spaced sampling is performed on the ray to generate sampling points. 2-2. Optimize the sampling points as follows: 1) Generate a 3D cube of the enclosing structure based on the joint points of the estimated instance pose, filter the sampling points, and only retain the sampling points inside the 3D cube; 2) Generate sampling rays only for pixels inside the instance mask valid area within the 3D cube and inside the 3D cube projection area, and sample inside the instance mask with probability p, and sample the non-instance mask area inside the 3D cube projection area with probability 1-p to reduce the impact of inaccurate instance mask.

4. The method for drivable single-viewpoint multi-person reconstruction based on human body shape consistency according to claim 3, characterized in that: The specific details of step S3 are as follows: The pose deformation network is denoted as F d In the instance posture extraction stage, the rotation vectors of the 24 joint points under the SMPL skeleton standard with the parent node direction as the rotation axis are extracted, and based on this, the rotation matrix and displacement vector of each joint point are obtained through the Rodriguez formula, thereby constructing a 4×4 RT transformation matrix; The LBS algorithm needs to set a linear blend skinning weight value for each joint point, which represents the weight of the transformation of the joint point on the surface vertex. By applying the LBS formula in reverse, the vertex transformation from any pose to the rest pose is realized, thereby realizing the motion restoration from any pose to T-Pose. Let the LBS weight of the j-th joint point be w j , the RT transformation matrix of the jth joint point is recorded as G j , x is an arbitrary vertex, then: x'=F d (x,w) Among them, J represents the total number of joint points, which is 24. By using the skinning formula in reverse, the sampling point in any posture is recorded as x and transformed to the corresponding point position in the resting posture, that is, the surface vertex x'. On the basis of this point transformation, the appearance of the three-dimensional digital human is reconstructed with the resting posture as the benchmark, and the complete posture and appearance are decoupled.

5. The method for drivable single-viewpoint multi-person reconstruction based on human body shape consistency according to claim 4, characterized in that: The specific details of step S4 are as follows: The surface vertex x' corresponding to the resting posture obtained by the posture deformation network part is normalized to the size space of the general voxel feature grid and the instance voxel feature grid, and trilinear interpolation sampling is performed to obtain the feature v g and v i ; The voxel feature grid models any three-dimensional space point, realizes the encoding conversion from any three-dimensional space point to the feature of corresponding color and density information, and introduces trilinear interpolation method to interpolate low-resolution features to expand the expression range; the voxel feature grid is denoted as V; N x 、N y 、N z Denote as the size of the voxel feature grid; Where v represents the feature of the voxel feature grid output, which is v g or v i ; interp represents the trilinear interpolation function, (x c ,y c , z c ) represents the three-dimensional space coordinate value of the surface vertex x′, represents the set of real numbers; C v Indicates the number of channels per voxel grid; N x , N y , N z Represents the length of each dimension of the voxel grid; Sampling to get feature v g and v i Sin and cosine position encoding is performed to retain high-frequency detail information. The sin and cosine position encoding formula is as follows: γ(v)=(sin(2 0 v), cos(2 0 )v,...,sin(2 L-1 v), cos(2 L-1 )v) Among them, γ(v) represents the sine and cosine position encoding; v represents the input parameters, including the feature v g and v i , L represents a hyperparameter; the general human features after position encoding and the instance-related instance human features are concatenated to obtain the final encoded complete feature vector v f .

6. The method for drivable single-viewpoint multi-person reconstruction based on human body shape consistency according to claim 5, characterized in that: The specific details of step S5 are as follows: The encoded complete feature vector v f Send it to the implicit radiation field, that is, the fully connected network decoder, denoted by Φ r , the decoding process is expressed as: c, σ, depth=Φ r (c(v f ),γ(t),γ(x c ,y c ,z c ),γ(d)) Among them, c represents the color of the sampling point, σ represents the volume density of the sampling point, depth represents the estimated depth, and v f represents the complete feature vector after position encoding, t represents the frame time sequence in the frame sequence, (x c ,y c , z c ) represents the spatial sampling point x′ in the resting posture, and d represents the direction vector of the camera. After calculating the colors of all sampling points of a sampling ray r, the specific image plane pixel color is obtained through the volume rendering integral formula. The specific formula is as follows: in, represents the color of the pixel on the image plane, N represents the number of sampling points on the ray r, i represents the i-th sampling point, σ i represents the volume density of the i-th sampling point, δ i =x i+1 -x i Represents the distance between adjacent sampling points, c i Represents the color information of the i-th sampling point.

7. The method for drivable single-viewpoint multi-person reconstruction based on human body shape consistency according to claim 6, characterized in that: The specific details of step S6 are as follows: The vertex model and normal model of SMPL in the resting posture corresponding to the instance body shape parameter β are introduced as the human body shape prior. For any spatial sampling point, the k surface vertices closest to the spatial sampling point are determined by the K nearest neighbor algorithm, and the normal vector of the surface directed distance judgment point is estimated. The specific calculation formula is as follows: in, represents any spatial sampling point in the resting posture, v i′ The i′th nearest neighbor surface vertex, n i′ represents the normal vector of the i′th nearest neighbor surface vertex, Representation vector and corresponding to v i′ The cosine similarity of the normal vector of the vertex; by calculating the cosine similarity as the weighted weight of the neighboring surface vertices and normal vectors, the estimated surface directed distance determination normal vector is obtained The directed distance d can be easily obtained by a vector dot product sign , the specific calculation formula is as follows: The obtained directed distance is used as the basis for determining the subsequent calculation of integrity loss.

8. The method for drivable single-viewpoint multi-person reconstruction based on human body shape consistency according to claim 7, characterized in that: The specific details of step S7 are as follows: The integrity loss is used to constrain the density of sampling points in the human body to approach 1, thereby effectively improving the limb separation and artifact phenomena. The integrity loss is recorded as The specific definitions are as follows: Among them, ReLU is the activation function, σ is the volume density of the corresponding point, and α is the hyperparameter; this method uses ReLU to truncate the negative volume density, and tends to reduce the penalty coefficient for high volume density to achieve effective constraints on the volume density.

9. The method for drivable single-viewpoint multi-person reconstruction based on human body shape consistency according to claim 7, characterized in that: The specific details of step S8 are as follows: The corresponding image plane pixels of each human body instance generate sampling rays and then are rendered. And estimate the depth depth; Due to the unsupervised nature of network depth estimation, more invalid depths will appear at the edge of the object; Therefore, this method uses the bounding 3D cube of the SMPL model under the corresponding posture (P) and calculates the minimum and maximum values ​​of the effective depth based on the camera intrinsic parameter matrix and the extrinsic parameter matrix, and filters out the invalid depth while sorting the same pixels in the order of depth depth. Rearrange to obtain the output image.

10. The method for drivable single-viewpoint multi-person reconstruction based on human body shape consistency according to claim 7, characterized in that: The specific details of step S9 are as follows: Use 3 loss functions including mean square error Learning to Perceive Patch Similarity Loss and loss of integrity The effective constraint model converges effectively, and the specific loss function is as follows: λ1, λ2 and λ3 are set to 0.2, 1, and 5e-5, respectively. In the first 300 iterations of training, λ2 and λ3 are set to 0. The mean square error can promote the rapid convergence of most color blocks and accelerate the early convergence process of the model. The weight values ​​of λ2 and λ3 are restored after 300 iterations.

Images