Human body three-dimensional reconstruction method based on conical ray nerve radiation field

Abstract

The invention discloses a human body three-dimensional reconstruction method based on a conical ray neural radiation field. The human body three-dimensional reconstruction method comprises the following steps: firstly, preprocessing a human body video image; initializing a cone ray and sampling to obtain a sampling point, and expressing a cone section by using a multi-dimensional Gaussian function; a potential code and an SMPL human body model are initialized; carrying out SMPL human body model potential code diffusion and SMPL human body model density and color regression; and finally, performing three-dimensional reconstruction on the human body through SMPL human body model rendering. The conic ray concept is introduced, and more light information can be captured. In this way, the overall color of the reconstructed model can be expressed more accurately, and the reconstructed human body model is more real and natural in color. By means of the method, the edge transition problem can be better solved, the sawtooth phenomenon is reduced, the reconstructed human body model is smoother and finer on the edge contour, and the visual quality is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image, video processing and three-dimensional reconstruction, and in particular relates to a Gaussian distribution calculation model method and a method for three-dimensional reconstruction of the human body through deep learning. Background Art

[0002] Human 3D reconstruction is a hot topic in computer vision and graphics. It aims to create a 3D digital model of the human body from various types of 2D data sources, such as photos, videos, or scanner data. This technology has a wide range of applications, from medical imaging to entertainment, fashion, sports science, augmented reality, and virtual reality. Human 3D modeling is a complex process that can be achieved through a variety of different technologies and methods. These methods can be roughly divided into two categories: scanning-based methods and image-based methods. Among them, scanning-based methods generally include laser scanning, which uses a laser scanner to accurately scan the human body, capture subtle surface details, generate point cloud data, and then convert it into a 3D mesh model. In image-based methods, two or more cameras are used simultaneously to shoot the human body from different perspectives, imitating human binocular vision to determine depth and shape, and reconstruct the 3D structure by tracking and matching feature points. With the vigorous development of deep learning, the field of human 3D reconstruction has reached new heights, and there have been great breakthroughs in image and visual processing. The traditional method of using laser radar to generate point clouds and mimicking human binocular vision to measure human body shape to complete human body 3D reconstruction is highly complex and computationally intensive. In addition, since human body 3D modeling usually needs to consider many changes such as clothing, hairstyle, and posture, it is difficult for traditional modeling algorithms to achieve high model accuracy. Therefore, a more efficient and accurate human body 3D modeling method based on deep learning has gained continuous research attention.

[0003] Existing methods for human 3D reconstruction based on deep learning mainly focus on the relationship between joints and bones, use depth information as prior knowledge, and combine human models (such as SMPL) to reconstruct accurate human posture and shape. Although the existing technology pays attention to information such as joint and bone point depth, it lacks the information about how light interacts with the human body. In the same situation, it can pay more attention to the feature expression of the edge of the human model. For the task of human 3D reconstruction, the problem of algorithm perception of changes in the edge of the human model is an important and challenging task. Summary of the invention

[0004] In view of the shortcomings of the existing three-dimensional reconstruction methods of the human body, the present invention provides a three-dimensional reconstruction method of the human body based on cone ray nerve radiation field.

[0005] Technical problem solved: Since the rays in the traditional neural radiation field model are designed based on pixels, the Neural Radiance Fields (NeRF) projects a single infinitesimal and narrow ray for each pixel, which will cause excessive blurring and aliasing problems. The present invention adds a radius feature to the ray and converts the ray sampling into conical ray sampling.

[0006] A method for three-dimensional reconstruction of a human body based on cone-ray nerve radiation field comprises the following steps:

[0007] Step (1) preprocessing of human body video images;

[0008] Step (2) Initialize the conic ray and sample it, obtain the sampling points and use the multi-dimensional Gaussian function to express the conic section;

[0009] Step (3) Initialize the latent code and SMPL human body model;

[0010] Step (4) SMPL human model latent code diffusion;

[0011] Step (5) SMPL human body model density and color regression;

[0012] Step (6) performs three-dimensional reconstruction of the human body through SMPL human body model volume rendering.

[0013] Furthermore, the specific method of step (1) is as follows;

[0014] First, the videos captured by cameras with different serial numbers at the same time are divided into picture groups according to the camera serial number and frame serial number: c=1,..., ; f = 1, ..., }; where P is the picture group, c is the camera number, is the total number of cameras, f is the frame number, is the total number of frames. Each image in the divided image group, i.e. each frame f, is processed, and each image is segmented into human parts (Instance-level Human Parsing) to separate the human body contour from the background, and the obtained human body contour features are saved according to the image group. At the same time, the position features of the human body skeleton points in the image are calculated and saved according to the image group for the posture control of the SMPL human body model later.

[0015] Furthermore, the specific method of step (2) is as follows;

[0016] Get the width W, height H, and camera internal parameters (K, D, R, T) of each picture in the picture group, calculate the ray ray_o and ray direction ray_d corresponding to each pixel in the picture, and then calculate the angle between point t on ray ray_o ( ) is the radius dt of the circle with ) as the center.

[0017] For each pixel x in the image set, a cone is projected from the camera's ray center o along the ray direction ray_d passing through the pixel center. The vertex of the cone is located at o, and the cone radius from point o to point t in the image plane is parameterized as .Will Set to the width in pixels in world coordinates, according to the radius formula = , produces a cone whose set of pixels (x) in the area intersecting the image plane has a normal distribution on the x-axis and y-axis that matches the normal distribution of the pixel footprint. , The set of pixels (x) within the frustum of the cone between ] is:

[0018] ;

[0019] in {·} is an indicator function: a pixel (x) is positive if and only if it is in (ray_o, ray_d, , , ) is within the conic frustum defined by , F(x, )=1.

[0020] In NeRF, positional encoding is used to encode each point sampled on the ray, and the feature information of the sampling point is discretized by position encoding.

[0021] ;

[0022] in, is the position code corresponding to point x, is the number of layers used to expand the position encoding features.

[0023] Since the single sampling point is replaced by a sampling cone section, the position encoding method used is also changed:

[0024] ;

[0025] Where o and d represent the ray (ray_o) and the ray direction (ray_d). Since the integral in the numerator has no closed-form solution, using ordinary position coding to encode the sampling points in the conic section cannot obtain exact results. Therefore, it is necessary to use multivariate Gaussian functions to represent the sampling points and then perform position coding on them.

[0026] To approximate a cone with a multivariate Gaussian, we need to compute the mean and covariance of F(x, ·). Assuming each cone is circular, and since cones are symmetric about their axis, such a multivariate Gaussian is completely characterized by three values: the average distance along the ray , the variance along the ray and the variance perpendicular to the ray :

[0027]

[0028] in, =( + ) / 2, =( − ) / 2.

[0029] Convert the cone's coordinate axes to world coordinate axes. The formula is as follows, which is the final multivariate Gaussian:

[0030] ;

[0031] in, is the mean of the multivariate Gaussian, is the covariance matrix of the multivariate Gaussian and I is the identity matrix.

[0032] Afterwards, according to the expectation of the position encoding coordinates of the above multivariate Gaussian ( ) for position encoding, first Rewritten as Fourier characteristic:

[0033] ,

[0034] ;

[0035] The P matrix is ​​a power sequence cascade matrix obtained by doubling a 3X3 unit matrix L-1 times and merging L matrices with 3 rows.

[0036] Using the fact that the covariance of a linear transformation of a variable is a linear transformation of the covariance of the variable, the mean and covariance after lifting the multivariate Gaussian to the PE (E is the identity matrix) basis P can be expressed as:

[0037] ;

[0038] We then compute the expectations of the lifted multivariate Gaussians, dominated by the sines and cosines of the positions, which have simple closed-form expressions:

[0039] ;

[0040] The final positional encoding is computed as the mean and expected sine and cosine of the diagonal of the covariance matrix based on the expectation of the lifted multivariate Gaussian:

[0041] ;

[0042] in represents element-wise multiplication. Since the positional encoding encodes each dimension independently, this expected encoding depends only on the positional encoding The marginal distribution of The size of is relatively large, so directly calculating The diagonal of:

[0043] ;

[0044] The vector Depends only on the covariance matrix of the multivariate Gaussian The diagonal of is calculated as follows:

[0045] ;

[0046] In one embodiment, Set to 16 layers.

[0047] Furthermore, the specific method of step (3) is as follows:

[0048] SMPL is a model based on the vertices of the skin surface. It is a function of shape parameters, pose parameters and a rigid transformation relative to the SMPL coordinate system. The function outputs a 3D mesh of 6890 skin surface vertices. The latent code uses a neural network to represent the local geometry and appearance of humans. Define a set of latent codes Z = {z1, z2, ..., z6890} and fix it to the vertices of the SMPL model. For each frame in the image set , SMPL parameters A group of images containing images from different perspectives { =1, ..., } related human skeleton point features and human contour features. Then according to the SMPL parameters The spatial position of the latent code is transformed to obtain a structured latent code with the geometric characteristics of the human body.

[0049] Furthermore, the specific method of step (4) is as follows:

[0050] The 6890 latent codes on the surface of the SMPL model are diffused using a sparse convolutional network (SparseConvNet), where the sparse convolutional network generates a specific 3D sparse convolution based on the feature information of each image in the image group (photo human model border, camera rotation matrix, etc.). After that, the human model is voxelized, the sampling point obtained by sampling is used as the center point of the voxel, and the volume block with a length, width and height of 5 mm in the three directions of x, y and z is segmented as a voxel, and finally the human pixel coordinate system is converted into the human voxel coordinate system. After that, the obtained human voxels containing the latent code are input into the specific 3D sparse convolution generated by the sparse convolutional network according to the human feature information unique to each image, and the voxels containing human features and latent codes are obtained after 2×, 4×, 8×, and 16× downsampling.

[0051] Through convolution and downsampling, the input latent code is diffused to the space near the human body surface to obtain voxels containing human features and latent codes. Since code diffusion should not be affected by the position and direction of the human body in the real world coordinate system, the code position is converted to the SMPL coordinate system, which is the latent code voxel coordinate system. For any point of a human body in a photo in 3D space , can be described by a multivariate Gaussian, and its potential code can be obtained from the potential code voxel coordinate system. First, point Convert from the world coordinate system to the SMPL coordinate system, which aligns the point in 3D space with the potential code group Z. In addition, since the conic rays and the conic sections of the image are approximated by multivariate Gaussians, the mean and covariance of the multivariate Gaussians can be used to convert the points Transform the position in the SMPL coordinate system to obtain new points in the 8 quadrants , then use trilinear interpolation to calculate the point and 8 other new points The potential code for SMPL parameters , the original point and 8 other new points The potential code at is represented as , Finally, the 9 potential codes are concatenated together according to their features to obtain a new potential code Passed as input to the density model and color model to predict the point density and color.

[0052] Furthermore, the specific method of step (5) is as follows:

[0053] Density model: For frames ,point The volume density at is predicted as the potential code The function is defined as: ,in Represents an MLP network with four layers.

[0054] Color model: The potential code and ray direction ray_d as input to color regression. To model the position-dependent incident light, the color model also As input. For each video frame Assign a latent embedding To encode the time variation factor. Specifically, for the frame ,point The color at is predicted as the potential code , observation direction d, point and potential embedding The function of The color model of a frame is defined as:

[0055] ;

[0056] Where Mc represents an MLP network with two layers, and are the observation direction and point position encoding functions respectively. The dimension is set to 128.

[0057] Furthermore, the specific method of step (6) is as follows:

[0058] Given a pixel point, use the calculated camera ray r to sample along the camera ray r between the near boundary and the far boundary Conic sections The scene boundary is estimated based on the SMPL human body model. Then, NeRF is used to predict the volume density and color of the pixel. After all the pixels are predicted, the obtained volume density and color are reflected in the 3D space, and the 3D model is projected to generate the volume rendering model.

[0059] For video frames , the volume rendering color of the corresponding pixel Given by:

[0060] ,

[0061] ;

[0062] in is the distance between adjacent sampling points.

[0063] The beneficial effects of the present invention are as follows:

[0064] 1. Improve overall color expression: Traditional 3D reconstruction methods usually only focus on the color of a single pixel, but this invention introduces the concept of conical rays, which can capture more light information. This can more accurately express the overall color of the reconstructed model, making the reconstructed human body model more realistic and natural in color.

[0065] 2. Enhanced anti-aliasing capability of edge contours: Since conical rays can cover a larger spatial area, they can better handle edge transition issues and reduce aliasing compared to single-pixel rays. This makes the edge contours of the reconstructed human body model smoother and more delicate, improving visual quality.

[0066] 3. Improve the expression of model details: Using more light information captured by conical rays can better express the details of the model, making the reconstructed model richer in morphology and texture details.

[0067] 4. Optimize the efficiency of the reconstruction algorithm: The use of conical rays can reduce the need to process individual pixels one by one, thereby simplifying and accelerating the calculation process of the reconstruction algorithm and improving the reconstruction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a photograph of a human body segmentation image (grayscale) according to an embodiment of the present invention.

[0069] Figure 2 FIG. 4 is a schematic diagram of a mask difference (latent embedding) according to an embodiment of the present invention.

[0070] Figure 3 This is an overall flow chart of three-dimensional reconstruction of a human body according to an embodiment of the present invention.

[0071] Figure 4(a) is the original image, and Figure 4(b) is the reconstructed image.

[0072] Figure 5 The three-dimensional model of the human body reconstructed according to the embodiment of the present invention.

[0073] Figure 6 , Figure 7 , Figure 8 They are respectively embodiments of the present invention. DETAILED DESCRIPTION

[0074] The technical solution of the present invention is further described below with reference to the preset embodiments of the drawings.

[0075] like Figure 3 As shown, a method for 3D reconstruction of a human body based on cone ray nerve radiation field comprises the following steps:

[0076] Step (1) preprocessing of human body video images;

[0077] First, the videos captured by cameras with different serial numbers at the same time are divided into picture groups according to the camera serial number and frame serial number: c=1,..., ; f = 1, ..., }; where P is the picture group, c is the camera number, is the total number of cameras, f is the frame number, is the total number of frames. Each image in the divided image group, i.e. each frame f, is processed, and each image is segmented into human parts (Instance-level Human Parsing) to separate the human body contour from the background. The obtained human body contour features are saved according to the image group, such as Figure 1 At the same time, the existing tool mocap is used to calculate the position features of the human skeleton points in the image and save them according to the image group for the subsequent posture control of the SMPL human model.

[0078] Step (2) Initialize the conic ray and sample it, obtain the sampling points and use the multi-dimensional Gaussian function to express the conic section;

[0079] Get the width (W), height (H), and camera internal parameters (K, D, R, T) of each picture in the picture group, calculate the ray (ray_o) and ray direction (ray_d) corresponding to each pixel in the picture, and then calculate the angle between the point (t, ) is the radius of the circle with its center at (dt).

[0080] For each pixel (x) in the image group (here we consider that pixels also have height and width), a cone is projected from the camera's ray center o along the ray direction ray_d passing through the pixel center. The vertex of the cone is located at o, and the cone radius from point o to point t in the image plane is parameterized as .Will Set to the width in pixels in world coordinates, according to the radius formula = , produces a cone whose set of pixels (x) in the area intersecting the image plane has a normal distribution on the x-axis and y-axis that matches the normal distribution of the pixel footprint. , The set of pixels (x) within the frustum of the cone between ] is:

[0081] ;

[0082] in {·} is an indicator function: a pixel (x) is positive if and only if it is in (ray_o, ray_d, , , ) is within the conic frustum defined by , F(x, )=1.

[0083] In NeRF, positional encoding is used to encode each point sampled on the ray, and the feature information of the sampling point is discretized by position encoding.

[0084] ;

[0085] in, is the position code corresponding to point x, In order to expand the number of layers of position coding features, it is set to 16 layers in this embodiment.

[0086] Since the single sampling point is replaced by a sampling cone section, the position encoding method used is also changed:

[0087] ;

[0088] Where o and d represent the ray (ray_o) and the ray direction (ray_d). Since the integral in the numerator has no closed-form solution, using ordinary position coding to encode the sampling points in the conic section cannot obtain exact results. Therefore, it is necessary to use multivariate Gaussian functions to represent the sampling points and then perform position coding on them.

[0089] To approximate a cone with a multivariate Gaussian, we must compute the mean and covariance of F(x, ·). Since we assume that each cone is circular, and since cones are symmetric about the cone axis, such a multivariate Gaussian is completely characterized by three values ​​(in addition to o and d): the average distance along the ray , the variance along the ray and the variance perpendicular to the ray :

[0090]

[0091] in =( + ) / 2, =( − ) / 2.

[0092] Convert the cone's coordinate axes to world coordinate axes. The formula is as follows, which is the final multivariate Gaussian:

[0093] ;

[0094] in, is the mean of the multivariate Gaussian, is the covariance matrix of the multivariate Gaussian and I is the identity matrix.

[0095] Afterwards, according to the expectation of the position encoding coordinates of the above multivariate Gaussian ( ) for position encoding, first Rewritten as Fourier characteristic:

[0096] ,

[0097] ;

[0098] The P matrix is ​​a power sequence concatenated matrix obtained by doubling a 3X3 identity matrix L-1 times and merging L matrices with a row count of 3. This reparameterization operation allows us to derive a closed form for positional encoding.

[0099] Using the fact that the covariance of a linear transformation of variables is a linear transformation of the covariance of variables (Cov[Ax, By] = ACov[x, y] ), after lifting the multivariate Gaussian to the PE (E is the identity matrix) basis P (i.e., and pass The mean and covariance after lifting to the new coordinate system of PE basis P under Fourier characteristics can be expressed as:

[0100] ;

[0101] We then compute the expectations of the lifted multivariate Gaussians, dominated by the sines and cosines of the positions, which have simple closed-form expressions:

[0102] ;

[0103] We see that this expected sine or cosine is just the sine or cosine of the mean decayed by the Gaussian function of the variance. Based on the expectation of the lifted multivariate Gaussian, we can compute the final positional encoding as the expected sine and cosine of the mean and diagonal of the covariance matrix:

[0104] ;

[0105] in represents element-wise multiplication. Since the positional encoding encodes each dimension independently, this expected encoding depends only on the positional encoding The marginal distribution of The size of is relatively large, so it is very expensive to calculate. We directly calculate The diagonal of:

[0106] ;

[0107] The vector Depends only on the covariance matrix of the multivariate Gaussian The diagonal of is calculated as follows:

[0108] ;

[0109] Step (3) Initialize the latent code and SMPL human body model;

[0110] SMPL is a model based on the vertices of the skin surface. It is a function of shape parameters, pose parameters and a rigid transformation relative to the SMPL coordinate system. The function outputs a 3D mesh of 6890 skin surface vertices. The latent code uses a neural network to represent the local geometry and appearance of humans. Define a set of latent codes Z = {z1, z2, ..., z6890} and fix it to the vertices of the SMPL model. For each frame in the image set , SMPL parameters A group of images containing images from different perspectives { =1, ..., } related human skeleton point features and human contour features. Then according to the SMPL parameters The spatial position of the latent code is transformed to obtain a structured latent code with the geometric characteristics of the human body.

[0111] Step (4) SMPL human model latent code diffusion;

[0112] The 6890 latent codes on the surface of the SMPL model are diffused using a sparse convolutional network (SparseConvNet), where the sparse convolutional network generates a specific 3D sparse convolution based on the feature information of each image in the image group (photo human model border, camera rotation matrix, etc.). After that, the human model is voxelized, the sampling point obtained by sampling is used as the center point of the voxel, and the volume block with a length, width and height of 5 mm in the three directions of x, y and z is segmented as a voxel, and finally the human pixel coordinate system is converted into the human voxel coordinate system. After that, the obtained human voxels containing the latent code are input into the specific 3D sparse convolution generated by the sparse convolutional network according to the human feature information unique to each image, and the voxels containing human features and latent codes are obtained after 2×, 4×, 8×, and 16× downsampling.

[0113] Through convolution and downsampling, the input latent code is diffused to the space near the human body surface to obtain voxels containing human features and latent codes. Since code diffusion should not be affected by the position and direction of the human body in the real world coordinate system, the code position is converted to the SMPL coordinate system, which is the latent code voxel coordinate system. For any point of a human body in a photo in 3D space , can be described by a multivariate Gaussian, and its potential code can be obtained from the potential code voxel coordinate system. First, point Convert from world coordinates to SMPL coordinates, which aligns points in 3D space with the potential code group (Z). In addition, since we approximate the conic rays and conic sections of the image based on multivariate Gaussians, we can use the mean and covariance of the multivariate Gaussians to convert points Transform the position in the SMPL coordinate system to obtain new points in the 8 quadrants , then use trilinear interpolation to calculate the point and 8 other new points The potential code for SMPL parameters , the original point and 8 other new points The potential code at is represented as , Finally, the 9 potential codes are concatenated together according to their features to obtain a new potential code Passed as input to the density model and color model to predict the point density and color.

[0114] Step (5) SMPL human body model density and color regression;

[0115] Density model: For frames ,point The volume density at is predicted as the potential code The function is defined as: ,in Represents an MLP network with four layers.

[0116] Color model: The potential code and ray direction ray_d as input to color regression. To model the position-dependent incident light, the color model also As input, we observe that temporal variations affect the appearance of the hand, such as double exposure and self-shadowing. For each video frame, we Assign a latent embedding To encode the time variation factor. Specifically, for the frame ,point The color at is predicted as the potential code , observation direction d, point and potential embedding The function of The color model of a frame is defined as:

[0117] ;

[0118] Where Mc represents an MLP network with two layers, and are the observation direction and point position encoding functions respectively. The dimension is set to 128.

[0119] Step (6) reconstructing the human body in three dimensions by using SMPL human body model volume rendering;

[0120] Given a pixel point, use the calculated camera ray r to sample along the camera ray r between the near boundary and the far boundary Conic sections The scene boundaries (i.e., near boundaries and far boundaries) are estimated based on the SMPL human body model. Then, NeRF is used to predict the volume density and color of the pixel. After predicting all the pixels, the obtained volume density and color are reflected in the 3D space to project a 3D model, i.e., the generated volume rendering model. This volume rendering model is a 3D human body model.

[0121] For video frames , the volume rendering color of the corresponding pixel Given by:

[0122] ,

[0123] ;

[0124] in is the distance between adjacent sampling points. Set to 64.

[0125] Figure 4(a) is the original image, and Figure 4(b) is the reconstructed image. Figure 5 The three-dimensional model of the human body reconstructed according to the embodiment of the present invention.

[0126] Experimental results:

[0127] We call our own model Ours, and compare it with another SOA model NeuraolBody, hereinafter referred to as NB. We will use the PSNR scores of the models for comparison. The higher the PSNR score, the better the restoration effect.

[0128] Experimental effect 1, such as Figure 6 As shown:

[0129] Ours PSNR: 27.611 (first from left); NB PSNR: 27.274 (second from left); original image (third from left).

[0130] Experimental effect 2, such as Figure 7 As shown:

[0131] Ours PSNR: 31.400 (first from left); NB PSNR: 30.663 (second from left); original image (third from left).

[0132] Experimental effect 3, such as Figure 8 As shown:

[0133] Ours PSNR: 33.525 (first from left); NB PSNR: 32.88 (second from left); original image (third from left).

[0134] Based on the above experimental results 1-3, it can be seen that the method of the present invention significantly reduces the black spots in the generated pictures compared with the NB model, and has a stronger perception of the edge of the human body. The method of the present invention is effective in processing black spots, mainly due to the conical ray sampling, which makes the model much better in controlling the overall color. At the same time, the method of the present invention is better than the NB model in controlling the brightness of local areas, especially in the legs of the character model, the lighting is more natural.

[0135] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0136] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A method for three-dimensional reconstruction of the human body based on cone ray neural radiation field, characterized in that: The steps include: Step (1) preprocessing of human body video images; Step (2) Initialize the conic ray and sample it, obtain the sampling points and use the multi-dimensional Gaussian function to express the conic section; Step (3) initialize the latent code and SMPL human model; Step (4) SMPL human model latent code diffusion; Step (5) SMPL human body model density and color regression; Step (6) performs three-dimensional reconstruction of the human body through SMPL human body model volume rendering.

2. The method for three-dimensional reconstruction of a human body based on cone-ray neural radiation field according to claim 1, characterized in that: The specific method of step (1) is as follows; First, the videos captured by cameras with different serial numbers at the same time are divided into picture groups according to the camera serial number and frame serial number: c=1,..., ; f=1,..., }; where P is the picture group, c is the camera number, is the total number of cameras, f is the frame number, is the total number of frames; each picture in the divided picture group, that is, each frame f, is processed, and each picture is segmented into human body parts to separate the human body contour from the background, and the obtained human body contour features are saved according to the picture group; at the same time, the position features of the human body skeleton points in the picture are calculated and saved according to the picture group for the subsequent posture control of the SMPL human body model.

3. The method for three-dimensional reconstruction of a human body based on cone ray neural radiation field according to claim 1, characterized in that: The specific method of step (2) is as follows; Get the width W, height H, and internal parameters of the camera for each image in the image group, calculate the ray ray_o and ray direction ray_d corresponding to each pixel in the image, and then calculate the radius dt of the circle with point t on the ray ray_o as the center based on the angle formed by the ray ray_o and the straight-line distance between the camera origin and the pixel point x; For each pixel x in the image set, a cone is projected from the camera's ray center o along the ray direction ray_d passing through the pixel center; the vertex of the cone is located at o, and the cone radius from point o to point t in the image plane is parameterized as ;Will Set to the width in pixels in world coordinates, according to the radius formula = , produces a cone whose pixel set in the area intersecting the pixel on the image plane conforms to a normal distribution on the x-axis and y-axis, which matches the normal distribution of the pixel footprint; the two t values ​​[ , The set of pixels (x) within the frustum of the cone between ] is: ； in {·} is an indicator function: a pixel (x) is positive if and only if it is in (ray_o, ray_d, , , ) is within the conical frustum defined by, F(x, ) = 1; In NeRF, position coding is used to encode each point sampled on the ray, and the feature information of the sampling point is discretized by position coding. ； in, is the position code corresponding to point x, is the number of layers used to expand the position encoding features; Since the single sampling point is replaced by a sampling cone section, the position encoding method used is also changed: ； Where o and d represent the ray (ray_o) and the ray direction (ray_d). Since the integral in the numerator has no closed form solution, using ordinary position coding to encode the sampling points in the conic section cannot obtain exact results. Therefore, it is necessary to use multivariate Gaussian functions to represent the sampling points and then perform position coding on them. To approximate a cone with a multivariate Gaussian, we need to compute the mean and covariance of F(x, ·); assuming that each cone is circular, and since cones are symmetric about their axis, such a multivariate Gaussian is completely characterized by three values: the average distance along the ray , the variance along the ray and the variance perpendicular to the ray : ； in, =( + ) / 2, =( − ) / 2; Convert the cone's coordinate axes to world coordinate axes. The formula is as follows, which is the final multivariate Gaussian: ； in, is the mean of the multivariate Gaussian, is the covariance matrix of the multivariate Gaussian, I is the identity matrix; Afterwards, according to the expectation of the position encoding coordinates of the above multivariate Gaussian ( ) for position encoding, first Rewritten as Fourier characteristics: ， ； Among them, the P matrix is ​​a power sequence cascade matrix obtained by doubling a 3X3 unit matrix L-1 times and merging L matrices with 3 rows; Using the fact that the covariance of a linear transformation of variables is a linear transformation of the covariance of variables, the mean and covariance after lifting the multivariate Gaussian to the PE basis P can be expressed as: ； We then compute the expectations of the lifted multivariate Gaussians, dominated by the sines and cosines of the positions, which have simple closed-form expressions: ； The final positional encoding is computed as the mean and expected sine and cosine of the diagonal of the covariance matrix based on the expectation of the lifted multivariate Gaussian: ； in represents element-wise multiplication; since the positional encoding encodes each dimension independently, this expected encoding depends only on the positional encoding The marginal distribution of The size of is relatively large, so directly calculating The diagonal of: ； The vector Depends only on the covariance matrix of the multivariate Gaussian The diagonal of is calculated as follows: 。 4. The method for three-dimensional reconstruction of a human body based on cone ray nerve radiation field according to claim 3, characterized in that: Set to 16 layers.

5. A method for three-dimensional reconstruction of a human body based on cone ray nerve radiation field according to claim 1 or 3, characterized in that: The specific method of step (3) is as follows: SMPL is a model based on the vertices of the skin surface. It is a function of shape parameters, pose parameters, and a rigid transformation relative to the SMPL coordinate system. The function outputs a 3D mesh of 6890 skin surface vertices. The latent code uses a neural network to represent the local geometry and appearance of humans. Define a set of latent codes Z = {z1, z2, ..., z6890} and fix them on the vertices of the SMPL model; for each frame in the image group , SMPL parameters A group of images containing images from different perspectives { =1, ..., }Related human skeleton point features and human contour features; then according to the SMPL parameters The spatial position of the latent code is transformed to obtain a structured latent code with the geometric characteristics of the human body.

6. The method for three-dimensional reconstruction of a human body based on cone-ray neural radiation field according to claim 5, characterized in that: The specific method of step (4) is as follows: The 6890 potential codes on the surface of the SMPL model are diffused using a sparse convolutional network, where the sparse convolutional network generates a specific 3D sparse convolution based on the feature information of each image in the image group; then, the human body model is voxelized, the sampling point obtained by sampling is used as the center point of the voxel, and the volume block with a length, width and height of 5 mm in the three directions of x, y and z is segmented as a voxel, and finally the human body pixel coordinate system is converted into the human body voxel coordinate system; then, the obtained human body voxels containing potential codes are input into the specific 3D sparse convolution generated by the sparse convolutional network according to the human body feature information unique to each image, and the voxels containing human body features and potential codes are obtained after 2×, 4×, 8×, and 16× downsampling output in turn; Through convolution and downsampling, the input latent code is diffused to the space near the human body surface to obtain voxels containing human features and latent codes; since code diffusion should not be affected by the position and direction of the human body in the real world coordinate system, the code position is converted to the SMPL coordinate system, which is the latent code voxel coordinate system; for any point of a human body in a photo in 3D space , can be described by multivariate Gaussian, and its potential code can be obtained from the potential code voxel coordinate system; first, the point Convert from the world coordinate system to the SMPL coordinate system, which aligns the point in 3D space with the potential code group Z. In addition, since the conic rays and the conic sections of the image are approximated by multivariate Gaussians, the mean and covariance of the multivariate Gaussians can be used to convert the points Transform the position in the SMPL coordinate system to obtain new points in the 8 quadrants , then use trilinear interpolation to calculate the point and 8 other new points The potential code for ; for the SMPL parameters , the original point and 8 other new points The potential code at is represented as , ; Finally, the 9 potential codes are spliced ​​together according to the characteristics to obtain a new potential code Passed as input to the density model and color model to predict the point density and color.

7. The method for three-dimensional reconstruction of a human body based on cone-ray neural radiation field according to claim 6, characterized in that: The specific method of step (5) is as follows: Density model: For frames ,point The volume density at is predicted as the potential code The function is defined as: ,in Represents an MLP network with four layers; Color model: The potential code and ray direction ray_d as input to color regression; in order to model the position-dependent incident light, the color model also As input; for each video frame Assign a latent embedding to encode the temporal variation factor; specifically, for the frame ,point The color at is predicted as the potential code , observation direction d, point and potential embedding The function of The color model of a frame is defined as: ； Where Mc represents an MLP network with two layers, and are the observation direction and point position encoding functions respectively; The dimension is set to 128.

8. The method for three-dimensional reconstruction of a human body based on cone-ray neural radiation field according to claim 7, characterized in that: The specific method of step (6) is as follows: Given a pixel point, use the calculated camera ray r to sample along the camera ray r between the near boundary and the far boundary Conic sections ; The scene boundary is estimated based on the SMPL human body model; then, NeRF is used to predict the volume density and color of the pixel; after predicting all the pixels, the obtained volume density and color are reflected in the 3D space, and the 3D model is projected to generate the volume rendering model; For video frames , the volume rendering color of the corresponding pixel Given by: ， ； in is the distance between adjacent sampling points.

Images