Digital human reconstruction method based on focus joint point perception

Abstract

The invention discloses a digital human reconstruction method based on focus joint point perception. The method comprises the following steps: taking a human body grid model of a first frame in an RGB image frame sequence as a standard human body grid model; performing Gaussian field initialization on the corresponding standard human body mesh model; constructing a current human body grid model of the current RGB image frame according to the second information; calculating a new parameter of a Gaussian primitive corresponding to the current RGB image frame; rendering according to the new parameters of the Gaussian primitives corresponding to the current RGB image frame and the pose of the human body camera to obtain a human body rendering picture, calculating the loss between the human body rendering picture and the current RGB image frame, iteratively optimizing the parameters of the Gaussian primitives of all the RGB image frames according to a training strategy for minimizing the loss, and combining all the optimized Gaussian primitives to obtain a human body image. And outputting the reconstructed digital human. According to the invention, the digital human reconstruction precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human reconstruction, and in particular to a digital human reconstruction method focusing on joint point perception. Background Art

[0002] In recent years, 3D human reconstruction technology has made rapid progress in the fields of computer vision and graphics, and has shown broad application potential in the fields of virtual reality, augmented reality, film and television production, and game development. The purpose of digital human reconstruction is to accurately estimate the geometric shape and appearance of the human body through computers to support high-fidelity presentation in virtual scenes. It is widely used in fields such as film and television production and virtual reality. Among them, high-quality rendering of movable digital human faces is the key to achieving emotional interaction and providing a real and immersive user experience.

[0003] In order to achieve a more natural human animation rendering effect, a high-precision human model needs to be reconstructed to make the rendering effect of the digital human in the virtual environment more realistic and delicate. However, in digital human animation, unseen postures may cause the generated animation model to be blurred or distorted, especially in complex postures. For example, the digital human reconstruction method based on Nerf achieves fast training and rendering of digital human reconstruction, but the quality of their full-body digital human reconstruction is low.

[0004] Therefore, the applicant has developed a digital human reconstruction method focusing on joint point perception to solve the above problems. Summary of the invention

[0005] The present invention proposes a digital human reconstruction method focusing on joint point perception to solve the problem that the existing digital human reconstruction method may cause blur or distortion in the generated animation model for unseen postures.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] The present invention provides a digital human reconstruction method focusing on joint point perception, comprising:

[0008] Acquire first information, wherein the first information includes a sequence of RGB image frames of a dynamic human body;

[0009] Acquire second information according to the first information, where the second information includes shape parameters, posture parameters, and human camera pose of the SMPL human body model corresponding to each predicted RGB image frame;

[0010] Constructing a human body mesh model corresponding to each of the RGB image frames based on a preset SMPL human body model according to the second information, and taking the human body mesh model of the first frame in the RGB image frame sequence as a standard human body mesh model;

[0011] Based on the preset triangle sampling rule, the human joint point area of ​​the standard human mesh model is sampled to obtain corresponding sampling results;

[0012] Performing Gaussian field initialization on the standard human body mesh model according to the sampling result to obtain initialization parameters of Gaussian basis elements corresponding to the standard human body mesh model;

[0013] Constructing a current human body mesh model of the current RGB image frame according to the second information;

[0014] New parameters of the Gaussian primitives corresponding to the current RGB image frame are calculated based on the initialization parameters of the Gaussian primitives corresponding to the standard human mesh model, the rotation changes and scaling changes of the triangular faces corresponding to the current human mesh model and the standard human mesh model;

[0015] A human body rendering image is obtained according to the new parameters of the Gaussian primitives corresponding to the current RGB image frame and the human body camera pose rendering, the loss between the human body rendering image and the current RGB image frame is calculated, the parameters of the Gaussian primitives of all the RGB image frames are iteratively optimized according to a training strategy that minimizes the loss, all the optimized Gaussian primitives are combined, and the reconstructed digital human is output.

[0016] Specifically, obtaining first information, and obtaining second information according to the first information, includes:

[0017] Get the RGB image frame sequence of the dynamic human body captured by the monocular camera;

[0018] The RGB image frame sequence of the dynamic human body captured by the monocular camera is input into the pre-trained human body parameter prediction model, and the shape parameters, posture parameters and human body camera posture of the SMPL human body model corresponding to each RGB image frame are output.

[0019] Furthermore, based on the preset triangle sampling rule, the human joint point area of ​​the standard human mesh model is sampled to obtain corresponding sampling results, including:

[0020] Constructing a point cloud detection model of human joints according to the human mesh model corresponding to the RGB image frame;

[0021] Dense sampling is performed on the human joint point area in the point cloud detection model to obtain a dense sampling result;

[0022] Random sampling is performed on other areas except the human body joint point area in the point cloud detection model to obtain a random sampling result;

[0023] The dense sampling result and the random sampling result are integrated to obtain the sampling result.

[0024] Specifically, the standard human body mesh model is initialized with a Gaussian field according to the sampling result to obtain initialization parameters of the Gaussian basis element corresponding to the standard human body mesh model, including:

[0025] Randomly initialize weights of three vertices of each triangular face in the sampling result to obtain random initialization weights of the sampling points;

[0026] Performing interpolation processing on the three vertices according to the randomly initialized weights to obtain the coordinates of the sampling points;

[0027] Interpolating the normals of the three vertices according to the randomly initialized weights to obtain a normal vector of the sampling point;

[0028] Randomly offset the sampling point along the direction of the normal vector to obtain the position of the corresponding Gaussian primitive;

[0029] According to the position of the Gaussian primitive, a distance between the Gaussian primitive and its nearest Gaussian primitive is calculated and the distance is used as an initial scaling value;

[0030] The initial rotation, spherical harmonics, and opacity of the Gaussian primitive are randomly initialized.

[0031] Specifically, performing Gaussian field initialization on the standard human body mesh model according to the sampling result includes:

[0032] Use the coordinates of the three vertices on each of the triangles The weights are randomly initialized to , use the initialization weights to interpolate the vertices to get the sampling points The coordinates of the vertex are interpolated using the initialization weights to obtain the normal vector ,in:

[0033] Among them, n1, n2, n3 are the normals of the three vertices;

[0034] The initial scaling value The calculation formula is as follows:

[0035]

[0036]

[0037] in is the position of the Gaussian basis element, is the position of the Gaussian primitive closest to the Gaussian primitive, and d is the distance of the random offset;

[0038] The initial rotation value of the Gaussian primitive Initialize to quaternion , the spherical harmonic function h is initialized to The tensor of dimension and opacity α are initialized to 0.1;

[0039] Wherein, N is the number of Gaussian basis elements, the order of the spherical harmonic function is d=3, the value of the tensor at the position (N, 3, 0) is 0.5, and the rest are 0.

[0040] Specifically, the new parameters of the Gaussian primitive corresponding to the current RGB image frame are calculated based on the initialization parameters of the Gaussian primitive corresponding to the standard human mesh model, the rotation change and scaling change of the triangular surface corresponding to the current human mesh model and the standard human mesh model, including:

[0041] Calculate the final position of the Gaussian primitive corresponding to the current RGB image frame according to the initialization weight of the standard human body mesh model, the distance of the random offset, the coordinates of the three vertices of the sampling triangle, and the normal vector;

[0042] Calculating the rotation changes of corresponding triangular faces in the standard human body mesh model and the current human body mesh model;

[0043] Calculate the final rotation value of the Gaussian basis element corresponding to the current RGB image frame according to the rotation change and the initial rotation value;

[0044] Calculating the area change of the corresponding triangular faces in the standard human body mesh model and the current human body mesh model;

[0045] A final scaling value of a Gaussian primitive corresponding to the current RGB image frame is calculated according to the area change and the initial scaling value.

[0046] Specifically, the calculation formula of the initialization parameter of the Gaussian primitive corresponding to the current RGB image frame includes:

[0047]

[0048]

[0049]

[0050] in is the final position, is the final scaling value, is the final rotation value, Respectively represent the coordinates of the three vertices of the sampling triangle of the current RGB image frame, represents the rotation change, represents the area change.

[0051] Furthermore, adding geometric constraints to the Gaussian primitives in the human body joint point region includes:

[0052] Constructing a geometric boundary constraint, wherein the geometric boundary constraint includes forcibly scaling the Gaussian primitive to a boundary content if the Gaussian primitive exceeds a boundary threshold during the change process;

[0053] A Gaussian primitive scaling constraint is constructed, wherein the Gaussian primitive scaling constraint includes a given scaling factor, finding a corresponding surface index of a point in the surface information in the SMPL model, and locking the Gaussian primitive on the surface according to the surface index.

[0054] Furthermore, the training loss function adopted by the training strategy is:

[0055]

[0056] The target rendered image is compared with the RGB image of the standard frame. loss, is the mean square error loss, is the perceptual similarity loss, is the Gaussian basis scaling loss, is the normal consistency loss, where are all constants, among which:

[0057] The loss is:

[0058]

[0059] In the formula, H, W, and C are the height, width, and number of channels of the image, respectively. is the pixel value of the rendered image, The pixel value of the picture taken by the camera;

[0060] Mean square error loss for:

[0061]

[0062] In the formula, is the mean of the squared errors of all pixels, is the pixel value of the rendered image, The pixel value of the picture taken by the camera;

[0063] Perceptual Similarity Loss The loss is:

[0064]

[0065] In the formula, represents the feature map of the i-th layer of image I in the VGG deep network, N is the number of feature layers for calculating the loss, To render the image, Take pictures for the camera;

[0066] Gaussian basis scaling loss :

[0067]

[0068] In the formula, is the maximum value of the Gaussian basis element scaling, is the minimum value of the Gaussian basis element scaling, To specify the maximum scaling value, The maximum value of the ratio of the maximum scaling value to the minimum scaling value is specified;

[0069] in ,and is the Gaussian primitive scaling loss in the joint region, where , , is the Gaussian primitive scaling loss in the non-joint region, , :

[0070] Normal consistency loss :

[0071]

[0072] Where N is the normal map derived from the rendered depth map, is the normal of the Gaussian primitive inside the camera frustum, is the weight of α of the i-th Gaussian basis element. This loss enables Gaussian to achieve a smoother surface.

[0073] Furthermore, in the iterative optimization process:

[0074] Determine whether there is a negative number in the initialization weight of the sampling point;

[0075] If yes, then determining that the vertex of the triangular face corresponding to the initialization weight is located in another triangular face outside the triangular face corresponding to the initialization weight;

[0076] The new initialization weight is calculated using the three vertices of the other triangular face, and the sampling point is updated to the sampling point on the other triangular face.

[0077] The beneficial effects of the present invention are:

[0078] The present invention proposes a digital human reconstruction method that focuses on joint point perception, constructs a high-precision digital human reconstruction model, makes the rendering effect of the digital human in the subsequent virtual environment more realistic and delicate, avoids the problem of blurring or distortion for postures that have never been seen, and solves the problem that the existing digital human reconstruction method may cause blurring or distortion of the generated animation model for postures that have never been seen. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a method flow chart of a digital human reconstruction method focusing on joint point perception in an embodiment of the present application;

[0080] Figure 2 This is a schematic diagram of the principle of optimizing triangle A in an embodiment of the present application;

[0081] Figure 3 It is a schematic diagram of scaling constraints to points near SMPL arm joints in an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0083] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0085] In the description of the present invention, it should be understood that the terms "upper", "lower", "inside", "outside", "left", "right", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use, or are directions or positional relationships commonly understood by those skilled in the art. These directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0086] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0087] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0088] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0089] like Figure 1 As shown, a digital human reconstruction method focusing on joint point perception includes:

[0090] Acquire first information, wherein the first information includes a sequence of RGB image frames of a dynamic human body;

[0091] The second information is obtained according to the first information, wherein the second information includes shape parameters, posture parameters and human camera posture of the SMPL human body model corresponding to each predicted RGB image frame, wherein the shape parameter β is a 10-dimensional vector, and the posture parameter θ is a (24,3) dimension, representing the rotation of 24 joint points.

[0092] Constructing a human body mesh model corresponding to each of the RGB image frames based on a preset SMPL human body model according to the second information, and taking the human body mesh model of the first frame in the RGB image frame sequence as a standard human body mesh model;

[0093] Based on the preset triangle sampling rule, the human joint point area of ​​the standard human mesh model is sampled to obtain corresponding sampling results;

[0094] Performing Gaussian field initialization on the standard human body mesh model according to the sampling result to obtain initialization parameters of Gaussian basis elements corresponding to the standard human body mesh model;

[0095] Constructing a current human body mesh model of the current RGB image frame according to the second information;

[0096] New parameters of the Gaussian primitives corresponding to the current RGB image frame are calculated based on the initialization parameters of the Gaussian primitives corresponding to the standard human mesh model, the rotation changes and scaling changes of the triangular faces corresponding to the current human mesh model and the standard human mesh model;

[0097] According to the new parameters of the Gaussian primitives corresponding to the current RGB image frame and the human camera pose rendering, a human body rendering picture is obtained, the loss between the human body rendering picture and the current RGB image frame is calculated, and the parameters of the Gaussian primitives of all the RGB image frames are iteratively optimized according to the training strategy that minimizes the loss, and all the optimized Gaussian primitives are combined to output the reconstructed digital human. Optimize the parameters of the Gaussian field model to gradually improve the model's ability to express human body shape, posture and details.

[0098] The present invention realizes real-time rendering of human body animation based on the reconstructed human body model, presenting a smooth and realistic human body animation effect.

[0099] In some embodiments, the obtaining of information includes:

[0100] Acquiring first information, and acquiring second information according to the first information, including:

[0101] Get the RGB image frame sequence of the dynamic human body captured by the monocular camera;

[0102] The RGB image frame sequence of the dynamic human body captured by the monocular camera is input into the pre-trained human body parameter prediction model, and the shape parameters, posture parameters and human camera posture of the SMPL human body model corresponding to each RGB image frame are output. The human body parameter prediction model is an existing parameter prediction model.

[0103] In some embodiments, sampling is performed on the human joint point area of ​​the standard human mesh model based on a preset triangle surface sampling rule to obtain corresponding sampling results, including:

[0104] Constructing a point cloud detection model of human joints according to the human mesh model corresponding to the RGB image frame;

[0105] Dense sampling is performed on the human joint point area in the point cloud detection model to obtain a dense sampling result;

[0106] Random sampling is performed on other areas except the human body joint point area in the point cloud detection model to obtain a random sampling result;

[0107] The dense sampling result and the random sampling result are integrated to obtain the sampling result.

[0108] Generally speaking, the total number of samples of a human body model is about 13,000 triangles, of which 60% are distributed in the joint area and 40% are distributed in other areas. A point cloud detection model for human joints is constructed, and dense sampling is performed in the joint area of ​​the human body, and random sampling is performed in other areas. Compared with the traditional method based on random sampling, the reconstruction accuracy of the key point area is higher and the generated animation is more natural.

[0109] In some embodiments, the standard human body mesh model is initialized with a Gaussian field according to the sampling result to obtain initialization parameters of Gaussian primitives corresponding to the standard human body mesh model, including:

[0110] Randomly initialize weights of three vertices of each triangular face in the sampling result to obtain random initialization weights of the sampling points;

[0111] Performing interpolation processing on the three vertices according to the randomly initialized weights to obtain the coordinates of the sampling points;

[0112] Interpolating the normals of the three vertices according to the randomly initialized weights to obtain a normal vector of the sampling point;

[0113] Randomly offset the sampling point along the direction of the normal vector to obtain the position of the corresponding Gaussian primitive;

[0114] According to the position of the Gaussian primitive, a distance between the Gaussian primitive and its nearest Gaussian primitive is calculated and the distance is used as an initial scaling value;

[0115] The initial rotation, spherical harmonics, and opacity of the Gaussian primitive are randomly initialized.

[0116] In some embodiments, performing Gaussian field initialization on the standard human body mesh model according to the sampling result includes:

[0117] Use the coordinates of the three vertices on each of the triangles The weights are randomly initialized to , use the initialization weights to interpolate the vertices to get the sampling points The coordinates of the vertex are interpolated using the initialization weights to obtain the normal vector ,in:

[0118]

[0119]

[0120] Among them, n1, n2, n3 are the normals of the three vertices;

[0121] The initial scaling value The calculation formula is as follows:

[0122]

[0123]

[0124] in is the position of the Gaussian basis element, is the position of the Gaussian primitive closest to the Gaussian primitive, and d is the distance of the random offset;

[0125] The initial rotation value of the Gaussian primitive Initialized to quaternion (1,0,0,0), spherical harmonic function h is initialized to The tensor of dimension and opacity α are initialized to 0.1;

[0126] Wherein, N is the number of Gaussian basis elements, the order of the spherical harmonic function is d=3, the value of the tensor at the position (N, 3, 0) is 0.5, and the rest are 0.

[0127] The initial rotation value is the initial rotation parameter of the Gaussian primitive in the Gaussian field, used to indicate the initial direction of the Gaussian primitive. The spherical harmonic function h is a parameter describing the color characteristics of the Gaussian primitive, used to indicate the color of the Gaussian primitive observed from different directions. The opacity α is a scalar value describing the transparency of the Gaussian primitive in the Gaussian field, with a value range of [0,1], where 0 indicates complete transparency and 1 indicates complete opacity.

[0128] In some embodiments, the new parameters of the Gaussian primitives corresponding to the current RGB image frame are calculated based on the initialization parameters of the Gaussian primitives corresponding to the standard human mesh model, the rotation change and the scaling change of the triangular surface corresponding to the current human mesh model and the standard human mesh model, including:

[0129] Calculate the final position of the Gaussian primitive corresponding to the current RGB image frame according to the initialization weight of the standard human body mesh model, the distance of the random offset, the coordinates of the three vertices of the sampling triangle, and the normal vector;

[0130] Calculating the rotation changes of corresponding triangular faces in the standard human body mesh model and the current human body mesh model;

[0131] Calculate the final rotation value of the Gaussian basis element corresponding to the current RGB image frame according to the rotation change and the initial rotation value;

[0132] Calculating the area change of the corresponding triangular faces in the standard human body mesh model and the current human body mesh model;

[0133] A final scaling value of a Gaussian primitive corresponding to the current RGB image frame is calculated according to the area change and the initial scaling value.

[0134] In some embodiments, the calculation formula of the initialization parameter of the Gaussian primitive corresponding to the current RGB image frame includes:

[0135]

[0136]

[0137]

[0138] in is the final position, is the final scaling value, is the final rotation value, Respectively represent the coordinates of the three vertices of the sampling triangle of the current RGB image frame, represents the rotation change, represents the area change.

[0139] In some examples, the above steps are repeated 30,000 times to output a digital human model with good quality.

[0140] In some embodiments, determining whether there is a negative number in the initialization weight of the sampling point;

[0141] If yes, then determining that the vertex of the triangular face corresponding to the initialization weight is located in another triangular face outside the triangular face corresponding to the initialization weight;

[0142] The new initialization weight is calculated using the three vertices of the other triangular face, and the sampling point is updated to the sampling point on the other triangular face.

[0143] For example, Figure 2 As shown, when optimizing the u and v of triangle A, if If a negative number appears in , then the vertex obtained through the triangle element will appear in the triangle B outside the triangle A, and the three vertices of triangle B Calculating weights Update point P to the sample point on triangle B.

[0144] In some embodiments, adding geometric constraints to the Gaussian primitives in the human joint point region includes:

[0145] Constructing a geometric boundary constraint, wherein the geometric boundary constraint includes forcibly scaling the Gaussian primitive to a boundary content if the Gaussian primitive exceeds a boundary threshold during the change process;

[0146] A Gaussian primitive scaling constraint is constructed, wherein the Gaussian primitive scaling constraint includes a given scaling factor, finding a corresponding surface index of a point in the surface information in the SMPL model, and locking the Gaussian primitive on the surface according to the surface index.

[0147] like Figure 3As shown, the scaling constraint is to a point near the SMPL arm joint (i.e. Figure 3 ), find the face index f corresponding to the point in the face element information in the SMPL model. i , through f i It can lock to the GS primitive on this surface element, f i The GS primitive on ,Will The scaling constraint is ;

[0148] The training loss function used by the training strategy is:

[0149]

[0150] The target rendered image is compared with the RGB image of the standard frame. loss, is the mean square error loss, is the perceptual similarity loss, is the Gaussian basis scaling loss, is the normal consistency loss. are all constants, among which:

[0151] The loss is:

[0152]

[0153] In the formula, H, W, and C are the height, width, and number of channels of the image, respectively. is the pixel value of the rendered image, The pixel value of the picture taken by the camera;

[0154] Mean square error loss for:

[0155]

[0156] In the formula, is the mean of the squared errors of all pixels, is the pixel value of the rendered image, The pixel value of the picture taken by the camera;

[0157] Perceptual Similarity Loss The loss is:

[0158]

[0159] In the formula, represents the feature map of the i-th layer of the VGG deep network for image I. N is the number of feature layers for calculating the loss. To render the image, Take pictures for the camera;

[0160] Gaussian basis scaling loss :

[0161]

[0162] In the formula, is the maximum value of the Gaussian basis element scaling, is the minimum value of the Gaussian basis element scaling, To specify the maximum scaling value, The maximum value of the ratio of the maximum scaling value to the minimum scaling value is specified;

[0163] in ,and is the Gaussian primitive scaling loss in the joint region, where , . is the Gaussian primitive scaling loss in the non-joint region, , :

[0164] Normal consistency loss :

[0165]

[0166] Where N is the normal map derived from the rendered depth map, is the normal of the Gaussian primitive inside the camera frustum, is the weight of α of the i-th Gaussian basis element. This loss enables Gaussian to achieve a smoother surface.

[0167] The present invention also includes human body animation image rendering, and the human body animation image rendering includes:

[0168] Acquire a human body Gaussian field model according to the reconstructed digital human;

[0169] A human body image is obtained by rendering according to the human body Gaussian field model and the predicted camera pose of the human body.

[0170] Specifically, by inputting the posture parameters of the human body , animate the basic SMPL model into a human mesh model in the current posture, and calculate the rotation difference between the corresponding triangles of the human mesh in the current posture and the human mesh in the standard posture and scale difference , the Gaussian primitive is animated by the following formula:

[0171]

[0172]

[0173] The remaining parameters such as parameter weights u, v, offset distance d, spherical harmonics h and opacity α remain unchanged to obtain the final animated Gaussian field. , through the predicted human camera pose and You can get the human body image of the current animation posture .

[0174] The main advantages of the present invention compared to the prior art are as follows:

[0175] 1. It does not require the traditional method of multi-view training. This method only uses a single-purpose video to reconstruct a high-fidelity digital human and achieve full-body rendering;

[0176] 2. Faster rendering speed compared to existing technologies.

[0177] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A digital human reconstruction method focusing on joint point perception, characterized in that: include: Acquire first information, wherein the first information includes a sequence of RGB image frames of a dynamic human body; Acquire second information according to the first information, where the second information includes shape parameters, posture parameters, and human camera pose of the SMPL human body model corresponding to each predicted RGB image frame; Constructing a human body mesh model corresponding to each of the RGB image frames based on a preset SMPL human body model according to the second information, and taking the human body mesh model of the first frame in the RGB image frame sequence as a standard human body mesh model; Based on the preset triangle sampling rule, the human joint point area of ​​the standard human mesh model is sampled to obtain corresponding sampling results; Performing Gaussian field initialization on the standard human body mesh model according to the sampling result to obtain initialization parameters of Gaussian basis elements corresponding to the standard human body mesh model; Constructing a current human body mesh model of the current RGB image frame according to the second information; New parameters of the Gaussian primitives corresponding to the current RGB image frame are calculated based on the initialization parameters of the Gaussian primitives corresponding to the standard human mesh model, the rotation changes and scaling changes of the triangular faces corresponding to the current human mesh model and the standard human mesh model; A human body rendering image is obtained according to the new parameters of the Gaussian primitives corresponding to the current RGB image frame and the human body camera pose rendering, the loss between the human body rendering image and the current RGB image frame is calculated, the parameters of the Gaussian primitives of all the RGB image frames are iteratively optimized according to a training strategy that minimizes the loss, all the optimized Gaussian primitives are combined, and a reconstructed digital human is output.

2. A digital human reconstruction method focusing on joint point perception according to claim 1, characterized in that: Acquiring first information, and acquiring second information according to the first information, including: Get the RGB image frame sequence of the dynamic human body captured by the monocular camera; The RGB image frame sequence of the dynamic human body captured by the monocular camera is input into the pre-trained human body parameter prediction model, and the shape parameters, posture parameters and human body camera posture of the SMPL human body model corresponding to each RGB image frame are output.

3. The method for reconstructing a digital human focusing on joint point perception according to claim 1, characterized in that: Based on the preset triangle sampling rule, the human joint point area of ​​the standard human mesh model is sampled to obtain corresponding sampling results, including: Constructing a point cloud detection model of human joints according to the human mesh model corresponding to the RGB image frame; Dense sampling is performed on the human joint point area in the point cloud detection model to obtain a dense sampling result; Random sampling is performed on other areas except the human body joint point area in the point cloud detection model to obtain a random sampling result; The dense sampling result and the random sampling result are integrated to obtain the sampling result.

4. A digital human reconstruction method focusing on joint point perception according to claim 1 or 3, characterized in that: The standard human body mesh model is initialized with a Gaussian field according to the sampling result to obtain initialization parameters of the Gaussian basis element corresponding to the standard human body mesh model, including: Randomly initialize weights of three vertices of each triangular face in the sampling result to obtain random initialization weights of the sampling points; Performing interpolation processing on the three vertices according to the randomly initialized weights to obtain the coordinates of the sampling points; Interpolating the normals of the three vertices according to the randomly initialized weights to obtain a normal vector of the sampling point; Randomly offset the sampling point along the direction of the normal vector to obtain the position of the corresponding Gaussian primitive; According to the position of the Gaussian primitive, a distance between the Gaussian primitive and its nearest Gaussian primitive is calculated and the distance is used as an initial scaling value; The initial rotation, spherical harmonics, and opacity of the Gaussian primitive are randomly initialized.

5. The method for reconstructing a digital human focusing on joint point perception according to claim 4, characterized in that: Performing Gaussian field initialization on the standard human body mesh model according to the sampling result includes: Use the coordinates of the three vertices on each of the triangles The weights are randomly initialized to , use the initialization weights to interpolate the vertices to get the sampling points The coordinates of the vertex are interpolated using the initialization weights to obtain the normal vector ,in: ， ， Among them, n1, n2, n3 are the normals of the three vertices; The initial scaling value The calculation formula is as follows: ， ， in is the position of the Gaussian basis element, is the position of the Gaussian primitive closest to the Gaussian primitive, and d is the distance of the random offset; The initial rotation value of the Gaussian primitive Initialized to quaternion (1,0,0,0), spherical harmonic function h is initialized to The tensor of dimension and opacity α are initialized to 0.1; Where N is the number of Gaussian basis elements, the order of spherical harmonics is d=3, and the tensor is ( ) is 0.5, and the rest are 0.

6. The method for reconstructing a digital human focusing on joint point perception according to claim 5, characterized in that: The new parameters of the Gaussian primitive corresponding to the current RGB image frame are calculated according to the initialization parameters of the Gaussian primitive corresponding to the standard human mesh model, the rotation change and the scaling change of the triangular surface corresponding to the current human mesh model and the standard human mesh model, including: Calculate the final position of the Gaussian primitive corresponding to the current RGB image frame according to the initialization weight of the standard human body mesh model, the distance of the random offset, the coordinates of the three vertices of the sampling triangle, and the normal vector; Calculating the rotation changes of corresponding triangular faces in the standard human body mesh model and the current human body mesh model; Calculate the final rotation value of the Gaussian basis element corresponding to the current RGB image frame according to the rotation change and the initial rotation value; Calculating the area change of the corresponding triangular faces in the standard human body mesh model and the current human body mesh model; A final scaling value of a Gaussian primitive corresponding to the current RGB image frame is calculated according to the area change and the initial scaling value.

7. The method for reconstructing a digital human focusing on joint point perception according to claim 6, characterized in that: The calculation formula for the initialization parameters of the Gaussian primitives corresponding to the current RGB image frame includes: ， ， ， in is the final position, is the final scaling value, is the final rotation value, Respectively represent the coordinates of the three vertices of the sampling triangle of the current RGB image frame, represents the rotation change, represents the area change.

8. The method for reconstructing a digital human focusing on joint point perception according to claim 7, characterized in that: Adding geometric constraints to the Gaussian primitives in the human joint point region includes: Constructing a geometric boundary constraint, wherein the geometric boundary constraint includes forcibly scaling the Gaussian primitive to a boundary content if the Gaussian primitive exceeds a boundary threshold during the change process; Constructing a Gaussian primitive scaling constraint, the Gaussian primitive scaling constraint includes a given scaling factor, finding a corresponding surface index of a point in the surface element information in the SMPL model, and locking the Gaussian primitive on the surface element according to the surface element index.

9. The method for reconstructing a digital human focusing on joint point perception according to claim 8, characterized in that: The training loss function used by the training strategy is: ， The target rendered image is compared with the RGB image of the standard frame. loss, is the mean square error loss, is the perceptual similarity loss, is the Gaussian basis scaling loss, is the normal consistency loss, where are all constants, among which: The loss is: ， In the formula, H, W, and C are the height, width, and number of channels of the image, respectively. is the pixel value of the rendered image, The pixel value of the picture taken by the camera; Mean square error loss for: ， In the formula, is the mean of the squared errors of all pixels, is the pixel value of the rendered image, The pixel value of the picture taken by the camera; Perceptual Similarity Loss The loss is: ， In the formula, represents the feature map of the i-th layer of image I in the VGG deep network, N is the number of feature layers for calculating the loss, To render the image, Take pictures for the camera; Gaussian basis scaling loss : ， In the formula, is the maximum value of the Gaussian basis element scaling, is the minimum value of the Gaussian basis element scaling, To specify the maximum scaling value, The maximum value of the ratio of the maximum scaling value to the minimum scaling value is specified; in ,and is the Gaussian primitive scaling loss in the joint region, where , , is the Gaussian primitive scaling loss in the non-joint region, , : Normal consistency loss : ， Where N is the normal map derived from the rendered depth map, is the normal of the Gaussian primitive inside the camera frustum, is the weight of α of the i-th Gaussian basis element. This loss enables Gaussian to achieve a smoother surface.

10. The method for reconstructing a digital human focusing on joint point perception according to claim 9, characterized in that: In the iterative optimization process: Determine whether there is a negative number in the initialization weight of the sampling point; If yes, then determining that the vertex of the triangular face corresponding to the initialization weight is located in another triangular face outside the triangular face corresponding to the initialization weight; The new initialization weight is calculated using the three vertices of the other triangular face, and the sampling point is updated to the sampling point on the other triangular face.

Images