A gaussian digital human reconstruction method, device and apparatus

Abstract

The application discloses a Gaussian digital human reconstruction method, device and apparatus, and belongs to the field of artificial intelligence, and aims to solve the problem of rough hairstyle in the 3D GS digital human reconstruction process. The method comprises the following steps: performing segmentation processing on multi-view images of a target object to extract multi-view hairstyle images; performing implicit modeling on 3D face key points of the target object and the multi-view hairstyle images to construct a hairstyle SDF field; performing isosurface extraction on the hairstyle SDF field based on a DMT algorithm to obtain a hairstyle mesh; splicing a body mesh of the target object and the hairstyle mesh to obtain an initial 3D model of the target object; and rendering a digital human by sampling Gaussian particles on the surface of the initial 3D model through a 3D Gaussian sputtering technology. The hairstyle region can be finely reconstructed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a Gaussian digital human reconstruction method, device and apparatus. BACKGROUND

[0002] In recent years, digital human reconstruction technology based on 3D Gaussian Splatting (3DGS) has attracted widespread attention due to its high-quality rendering effect and real-time performance. The existing mainstream scheme usually takes the SMPL or SMPLX model as the human geometry prior, and optimizes through multi-view images to realize the reconstruction of human posture, shape and texture.

[0003] However, the SMPLX model only contains the geometry and topology information of the human torso, limbs and face, but does not contain the geometry and topology information of the hair, resulting in a lack of reasonable geometric prior in the Gaussian Splatting initialization stage. The hairstyle area usually only relies on sparse point clouds or random initialization, and the reconstruction result is fuzzy, lacks details, and the edges are not clear, which further causes the hairstyle of the digital human to be rough. SUMMARY

[0004] To solve the above problems, the present application provides a Gaussian digital human reconstruction method, comprising:

[0005] segmenting the multi-view images of the target object to extract multi-view hairstyle images;

[0006] implicitly modeling the 3D face key points of the target object and the multi-view hairstyle images to construct a hairstyle SDF field;

[0007] extracting the isosurface of the hairstyle SDF field based on a DMT algorithm to obtain a hairstyle mesh;

[0008] splicing the body mesh of the target object and the hairstyle mesh to obtain an initial 3D model of the target object;

[0009] rendering a digital human by sampling Gaussian particles on the surface of the initial 3D model through 3D Gaussian Splatting technology.

[0010] In one example, after the multi-view images of the target object are segmented to extract multi-view hairstyle images, the method further comprises:

[0011] extracting body key points of the multi-view segmentation images;

[0012] matching the key points of different multi-view segmentation images according to the internal and external parameters of the shooting device to establish a cross-view alignment relationship between the multi-view segmentation images;

[0013] The key points of different multi-view segmentation images are converted into 3D key points according to the cross-view alignment relationship between multi-view segmentation images.

[0014] In one example, the 3D face key points of the target object and the multi-view hairstyle image are implicitly modeled to construct a hairstyle SDF field, specifically including:

[0015] The 3D face key points of the target object and the expression parameters are input into a geometric MLP network to obtain an initial SDF field and vertex features; the geometric MLP network is used to constrain the 3D face key points on the surface of the SDF field;

[0016] The vertex features are input into a color MLP network to obtain vertex colors;

[0017] The initial SDF field and the vertex colors are rendered into 2D images to obtain multi-view rendered face images;

[0018] Loss calculation is performed between the multi-view rendered face images and the multi-view hairstyle images to obtain a hairstyle SDF field.

[0019] In one example, the DMT algorithm is used to extract an isosurface from the hairstyle SDF field to obtain a hairstyle mesh, specifically including:

[0020] According to the SDF field, the SDF values of the tetrahedral mesh are sampled;

[0021] According to the SDF values of the four vertices of the tetrahedron, the intersection points of the isosurface and the tetrahedral boundary are calculated using a differentiable linear interpolation formula;

[0022] The intersection points of different tetrahedrons are connected into triangular patches to obtain an initial hairstyle mesh;

[0023] The vertex positions of the initial hairstyle mesh are optimized according to the differentiable rendering gradient to obtain a hairstyle mesh.

[0024] In one example, the 3D Gaussian sputtering technique is used to sample Gaussian particles on the surface of the initial 3D model to render a digital human, specifically including:

[0025] Gaussian particles are sampled on the surface of the initial 3D model to determine the initial mesh positions of each Gaussian particle;

[0026] The initial attribute parameters of each Gaussian particle are predicted by a GSMLP network according to the pose parameters of the target object and the initial mesh positions;

[0027] The initial 3D model is rendered and loss calculated according to the initial attribute parameters of each Gaussian particle to obtain the target attributes of each Gaussian particle.

[0028] render the initial 3D model according to the target attribute of each Gaussian particle to obtain the digital human of the target object.

[0029] In one example, the initial attribute parameter of each Gaussian particle is predicted by the GSMLP network according to the pose parameter of the target object and the initial grid position, specifically including:

[0030] The pose parameter of the target object and the initial grid position of all Gaussian particles are input into a pose MLP network to obtain the vertex feature of each Gaussian particle;

[0031] The vertex feature of each Gaussian particle is input into a color MLP network to obtain the vertex color of each Gaussian particle;

[0032] The vertex feature and the vertex color of all Gaussian particles are input into a Gaussian attribute MLP network to obtain the initial attribute parameter of each Gaussian particle.

[0033] In one example, the initial 3D model is rendered and loss calculation is performed according to the initial attribute parameter of each Gaussian particle to obtain the target attribute of each Gaussian particle, specifically including:

[0034] The initial grid position is adjusted according to the position offset in the initial attribute parameter to obtain the initial adjusted grid position of the Gaussian particle;

[0035] The initial 3D model is rendered according to the initial adjusted grid position and the initial attribute parameter except the position offset to obtain a rendered multi-view image;

[0036] The rendered multi-view image and the multi-view image are subjected to loss calculation, so that the GSMLP network parameters are iteratively optimized by a back propagation algorithm until the loss value converges to a predetermined threshold, and the target attribute of each Gaussian particle is obtained.

[0037] In one example, the initial 3D model is rendered according to the target attribute of the Gaussian particle to obtain the digital human, specifically including:

[0038] The initial 3D model is rendered according to the target attribute of the Gaussian particle to obtain an initial 3D Gaussian splatting rendering image;

[0039] The initial 3D Gaussian splatting rendering image is subjected to super-resolution reconstruction by a pre-trained lightweight attention up-sampling network to obtain a 3D Gaussian splatting rendering image with higher resolution.

[0040] In another aspect, the embodiments of the present application provide a Gaussian digital human reconstruction device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above-mentioned Gaussian digital human reconstruction methods.

[0041] In another aspect, the embodiments of the present application provide a Gaussian digital human reconstruction device, comprising:

[0042] The segmentation unit performs segmentation processing on the multi-view images of the target object to extract multi-view hairstyle images;

[0043] The modeling unit implicitly models the 3D face key points of the target object and the multi-view hairstyle images to construct a hairstyle SDF field;

[0044] The extraction unit extracts an isosurface of the hairstyle SDF field based on a DMT algorithm to obtain a hairstyle mesh;

[0045] The splicing unit splices the body mesh of the target object and the hairstyle mesh to obtain an initial 3D model of the target object;

[0046] The rendering unit samples Gaussian particles on the surface of the initial 3D model by 3D Gaussian sputtering technology to render a digital human.

[0047] The above-mentioned at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects:

[0048] In the multi-view image segmentation processing process, not only the face region is segmented, but also the hairstyle region is segmented, so that the 3D hairstyle is implicitly represented by using the 3D face key points (providing geometric constraints of the head) and the multi-view hairstyle images, and the 3D face key points are introduced as prior knowledge to ensure that the reconstructed hairstyle is grown on the head and perfectly matches the proportion and shape of the head, avoiding unreasonable phenomena such as hairstyle floating or penetrating the scalp. Further, an explicit triangular mesh model is extracted from the learned SDF field, thereby overcoming the problem of blank initialization of the hairstyle region after 3D GS reconstruction, splicing the body mesh of the target object and the hairstyle mesh to form a complete initial geometric model of the body and hair, and finally realizing fine reconstruction of the hairstyle region in the full-body 3D GS digital human reconstruction process. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the present application, some embodiments of the present application will be described in detail below in combination with the accompanying drawings, in which:

[0050] Figure 1 A Gaussian digital human reconstruction method provided by an embodiment of the present application can be applied to a system architecture schematic diagram in which the method is applied.

[0051] Figure 2 A flowchart of a Gaussian digital human reconstruction method provided by an embodiment of the present application is shown in FIG. 4.

[0052] Figure 3 A flowchart of a hairstyle SDF field construction method provided by an embodiment of the present application is shown in FIG. 5.

[0053] Figure 4 A flowchart of a hairstyle mesh extraction method provided by an embodiment of the present application is shown in FIG. 6.

[0054] Figure 5 A flowchart of a digital human rendering method provided by an embodiment of the present application is shown in FIG. 7.

[0055] Figure 6 A flowchart of a rendering image super-resolution reconstruction method provided by an embodiment of the present application is shown in FIG. 8.

[0056] Figure 7 A structural schematic diagram of a Gaussian digital human reconstruction device provided by an embodiment of the present application is shown in FIG. 9.

[0057] Figure 8 A structural schematic diagram of a Gaussian digital human reconstruction device provided by an embodiment of the present application is shown in FIG. 9. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with specific embodiments and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0059] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0060] A Gaussian digital human reconstruction method provided by an embodiment of the present application can be applied to an application environment as shown in FIG. 1. Figure 1 A Gaussian digital human reconstruction method provided by an embodiment of the present application can be applied to an application environment as shown in FIG. 1. Figure 1As shown, the application environment can include: a client terminal 101, a communication network 102, a server 103, a database 104. The communication network 102 can serve as a channel for data transmission, providing a communication link for communication between the client terminal 101 and the server 103, so that after the client terminal 101 inputs the multi-view video image of the target object, it is transmitted to the server 103 based on the communication network 102, and the Gaussian digital human image processed by the server 103 can also be fed back to the client terminal 101 for digital human display. The server 103 is a server that can provide various services, and in order to improve processing efficiency, the server 103 is connected with multiple servers 103 to form a master-slave server cluster, so that after receiving the information uploaded by the client terminal 101 based on the communication network 102, it can be sent to the corresponding server 103 in the server cluster for multi-view image segmentation processing of the target object, so as to extract the multi-view hairstyle image. Then, the 3D face key points of the target object and the multi-view hairstyle image are implicitly modeled to construct a hairstyle SDF field. Then, based on the DMT algorithm, the isosurface of the hairstyle SDF field is extracted to obtain the hairstyle mesh. Then, the body mesh of the target object is spliced with the hairstyle mesh to obtain the initial 3D model of the target object. Finally, for the initial 3D model, Gaussian particles are sampled on the surface through 3D Gaussian sputtering technology, and a digital human is rendered. The Gaussian digital human image is fed back to the client terminal through the communication network 102.

[0061] Among them, the client terminal 101 can be a computer, a mobile phone and other hardware that can display digital humans, etc., or it can be software installed in the above-mentioned electronic devices. The communication network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0062] Among them, the database 104 is connected with each server 103 respectively, and can be used to store and manage the data of the preset first neural network and the preset second neural network. The database 104 can be integrated on the server, or placed on the cloud or other network servers.

[0063] As shown in Figure 2 The embodiment of the application provides a Gaussian digital human reconstruction method, which comprises the following steps:

[0064] S201: performing segmentation processing on the multi-view image of the target object to extract a multi-view hairstyle image.

[0065] The target object refers to the core subject that needs to be reconstructed into a digital human. The core subject can be a real person or a virtual person without life.

[0066] When setting up the collection environment, a multi-view visual system can be set up based on the all-around shooting. For example, in a professional motion capture studio, 16 high-definition cameras are arranged on a 360-degree ring-shaped support to ensure that the hairstyle and clothing details on the front, side, back, and top of the target object can be captured.

[0067] The multi-view image of the target object can be obtained based on shooting dynamic materials. For example, Xiao Ming is asked to perform stage actions such as singing, turning around, and waving hands, and 16 cameras synchronously shoot continuous videos while recording the intrinsic parameters (such as lens focal length and pixel size) and extrinsic parameters (such as placement position and camera orientation) of each camera. For example, the camera is oriented to shoot the front or left side of the target object. The extrinsic parameters determine the position and angle from which each camera shoots, which is a key to subsequent cross-view alignment.

[0068] In this example, the multi-view image is segmented using an image segmentation network to obtain a multi-view segmented image. The segmented regions include the head, torso, limbs, and hairstyle region. The hairstyle region can be segmented based on the texture and color features of the hair. For example, the image segmentation network can be an HRNet and BodyPix network model.

[0069] Based on this, the multi-view hairstyle image is extracted from the hairstyle region in the multi-view segmented image.

[0070] S202: Implicit modeling of the 3D face key points of the target object and the multi-view hairstyle image to construct a hairstyle SDF field.

[0071] In this example, the body key points of the multi-view segmented image are extracted to obtain the 3D key points of the target object, and then the 3D face key points are obtained.

[0072] Since the multi-view images are shot from different angles, cross-view alignment is needed between the multi-view segmented images.

[0073] Based on this, the process of extracting body key points is as follows:

[0074] First, the body key points of the multi-view segmented image are extracted.

[0075] The SIFT algorithm can be used to extract the key points of each image, including head and face key points (such as the tip of the nose, ears, eyes, and mouth), torso key points (such as the neck, shoulders, and hips), and limb key points (such as the elbows, wrists, knees, and ankles).

[0076] It should be noted that each key point records three pieces of information. Position: pixel coordinates in the image.

[0077] Scale: the size of the detail corresponding to the key point (e.g., the tip of the nose is larger than the corner of the eye, and the scale value is larger). Direction: the orientation of the key point (e.g., the direction of the lip peak is upward, and the direction angle is 30 degrees).

[0078] Since the core advantage of the SIFT algorithm is scale invariance and rotation invariance (e.g., the tip of Xiao Ming's nose is a circular protrusion in the front view and an elliptical protrusion in the 45-degree right front view), but the common features of both are extracted by the SIFT algorithm to ensure that the subsequent matching can recognize that both points are the tip of Xiao Ming's nose.

[0079] Then, according to the internal and external parameters of the shooting device, the key points of different multi-view segmentation images are matched to establish the cross-view alignment relationship between the multi-view segmentation images.

[0080] Among them, the SIFT algorithm generates a 128-dimensional feature descriptor (equivalent to the identity card of the key point) for each key point. By comparing the distance (such as Euclidean distance) between two descriptors, the closer the distance, the higher the similarity.

[0081] Specifically, the feature similarity of each key point of the multi-view segmentation image of the first shooting device and each key point of the multi-view segmentation image of the second shooting device is calculated, and the key point matching pair is filtered based on the feature similarity between the key points and the key points. Cross-view, accurate point-to-point correspondence is achieved.

[0082] For example, the feature descriptor distance between the tip of the nose key point shot by the first shooting device and the side of the nose key point shot by the second shooting device is the smallest (the highest similarity), and these two points are marked as a matching point pair.

[0083] After completing the key point matching between different shooting devices, the cross-view alignment relationship between the multi-view segmentation images of different shooting devices is obtained. For example, it is equivalent to knowing that each detailed point of Xiao Ming's head corresponds to which pixel position in the multi-view segmentation image.

[0084] Finally, using multi-view stereo vision technology, according to the cross-view alignment relationship between the multi-view segmentation images, the key points of different multi-view segmentation images are converted into 3D key points. That is, the 3D key points are dense key points.

[0085] Among them, the multi-view stereo vision technology is MVS technology, and its role is to calculate the 3D spatial coordinates of the target object based on the multi-view segmentation image, the internal and external parameters of the shooting device, and the feature matching result (cross-view pixel correspondence).

[0086] Specifically, based on the matched matching points, the internal and external parameters of the shooting device are combined to calculate the 3D coordinates of each matching point using the triangulation method. That is, it is equivalent to converting 2D matching points to 3D sparse points.

[0087] For example, the intrinsic and extrinsic parameters of camera 1 and camera 6 are known, as well as the pixel coordinates of the nose tip matching points in the two images. Then, the intersection of the two lines of sight (straight lines from the camera positions to the pixel coordinates) is calculated by triangulation, and this intersection is the coordinate of Xiao Ming's nose tip in 3D space.

[0088] Further, the above sparse reconstruction can only obtain the 3D coordinates of the feature points, and the dense reconstruction needs to be extended to all pixels, from sparse to dense, by the following steps to supplement more 3D points:

[0089] Image block matching: divide each multi-view image into multiple image blocks of a predetermined size (such as 16x16 pixels), and for each image block, find the corresponding image block in other views (based on the previous feature matching relationship and camera geometric constraints).

[0090] Depth calculation: calculate the depth of each image block in 3D space. For example, the right ear image block of camera 6, by comparing the parallax (the difference in pixel coordinates of the same 3D point in different view images, such as left image x coordinate - right image x coordinate) of the corresponding image block in camera 3 (left rear 45 degrees), combined with the camera parameters (camera baseline length, camera focal length) of the two, the depth of the right ear image block is calculated to be 1.45 meters. Wherein, by calculating the product of the camera baseline length and the camera focal length, the depth result is obtained by taking the ratio of the product result and the parallax. The baseline is the distance parameter between the two cameras, and the focal length is the common intrinsic parameter of the two cameras.

[0091] Generate dense points: for each pixel of each multi-view segmented image, calculate the 3D coordinates by the above method, and finally generate a dense point cloud composed of 3D points.

[0092] In summary, the information complementation of multiple views and the geometric constraints of the shooting device parameters are used to reversely calculate the details in the 2D image into the positions in the 3D space, and finally realize the conversion from the plane to the solid.

[0093] In this example, SDF is a method of implicitly representing three-dimensional geometric shapes. It does not directly store points, lines, and surfaces (such as meshes), but describes the distance from each point in space to the surface of the object through a mathematical function. The SDF value is positive for points outside the object, zero for points exactly on the surface, and negative for points inside the object.

[0094] Implicit representation means that it does not directly store the object surface, but stores a rule (function), which is the above-mentioned SDF. The object surface is implicitly defined as the set of all points that satisfy SDF(x, y, z) = 0.

[0095] Based on this, the implicit hairstyle SDF field refers to a complex mathematical function (usually represented by a neural network) stored in the computer. When the coordinates (x, y, z) of a point in three-dimensional space are input, the function can output the signed distance value of the point to the hair surface.

[0096] That is, the distance of a spatial point to the hair surface is described by a mathematical function, and a mathematical model of the hairstyle is established. For example, the distance of a certain point in space to Xiaoming's hair surface is described by a mathematical function: the SDF value of a point 1 mm away from the hair surface is +1, the SDF value of a point exactly on the hair surface is 0, and the SDF value of a point inside the hair is -1, which is equivalent to drawing an invisible contour boundary for the hair.

[0097] S203: Based on the DMT algorithm, the isosurface of the hairstyle SDF field is extracted to obtain the hairstyle mesh.

[0098] In this example, the DMT algorithm is an algorithm for extracting an explicit 3D mesh from an implicit SDF field. The core is to convert the abstract mathematical function into a visual triangular mesh through tetrahedral subdivision and isosurface tracking. Compared with the traditional Marching Cubes (cubic subdivision), tetrahedral subdivision is more flexible and can capture small-scale and irregular hairstyle details such as hair tips and curls more finely, reducing mesh jaggedness.

[0099] The hairstyle mesh is an explicit geometric model: a 3D structure composed of a large number of triangular faces, which can intuitively present the outline (such as the hair top curvature and hair tail drooping shape) and details (such as the delicate feeling of the hair stubble) of the hairstyle.

[0100] Based on this, the core of isosurface extraction is to find the surface with SDF value = 0 in the SDF field (because SDF = 0 corresponds to the hair surface, which is the basis for selecting the isosurface), and then convert the invisible surface into a visible triangular mesh through the DMT algorithm.

[0101] S204: The body mesh of the target object is spliced with the hairstyle mesh to obtain the initial 3D model of the target object.

[0102] In this example, the SMPLX model is projected onto the multi-view segmentation image, and the optimal model parameters are back calculated through an optimization algorithm (such as gradient descent) to obtain the expression parameters of the target object and the body mesh.

[0103]

[0104] Among them, the SMPLX model controls a highly realistic 3D body mesh through a set of compact parameters.

[0105] ​The shape, pose, and expression of the mesh. In simple terms, SMPLX is like a very advanced digital clay man. You don't need to manually pinch each vertex, but by adjusting some knobs (parameters), it can automatically become the shape you want, pose the way you want, and even make a rich facial expression.

[0106] The input of SMPLX is a set of parameters that together determine the appearance of the output mesh, mainly including: shape parameters, pose parameters, expression parameters, etc. The output of SMPLX is a triangular mesh. SMPLX is essentially a function that maps input parameters to output 3D mesh vertex coordinates, enabling the closest fitting of input parameters.

[0107] Based on this, the body mesh generated by SMPLX is spatially aligned and fused with the hairstyle mesh generated by DMT to form a complete initial geometric model of the body and hair.

[0108] It should be noted that the body mesh is composed of a large number of triangular faces, covering the entire body from the hairline to the soles of the feet, with high geometric accuracy and no hair topology.

[0109] The essence of splicing is to solve two problems: accurately attaching the hairstyle mesh to the correct position of the body mesh (such as aligning the hairline and not suspending the hair); eliminating the redundant overlap of the two meshes (such as the facial skin covered by the hair), forming a conflict-free, continuous and complete geometric structure.

[0110] Based on this, the body face key points (such as the tip of the nose, the brow bone, and the tragus) can be used as coordinate reference points to unify the coordinate systems of the body mesh and the hair mesh, ensuring that the attachment relationship between the hair and the body conforms to the real physiological structure.

[0111] For example, reference positioning: taking the tip of the nose as the coordinate origin, the brow bone as the horizontal axis, and the tip of the nose to the top of the head as the vertical axis, a unified 3D coordinate system is established. Hair mesh alignment: adjust the position and rotation angle of the hair mesh to make the hairline (the intersection line between the hair and the face) of the hair mesh completely coincide with the face hairline of the body mesh (such as the forehead hairline of the body mesh is 3 cm above the brow bone, then the forehead boundary of the hair mesh also needs to be aligned with this position, to avoid the hair growing in the middle of the forehead or the hair suspending above the top of the head). Detail fine-tuning: for special hairstyles (such as ponytails and hairpins), adjust the local position of the hair mesh to ensure that the hair root of the ponytail is aligned with the back of the neck of the body mesh, and the base of the hairpin is in line with the contour of the top of the head.

[0112] Further, after spatial alignment, a continuous and complete geometric model is formed through topological integration.

[0113] For example, vertex connection: for the junction area of the hairstyle mesh and the body mesh (such as the hairline), the edge vertices of the hairstyle mesh are soft-connected (through a small triangular transition) with the edge vertices of the body mesh to avoid the problem of gaps at the hairline. Uniform format: save the merged model in a standard 3D format to ensure that the vertex, triangular face, color, and other information of the 3D model can be normally read by subsequent steps (Gaussian particle sampling, rendering).

[0114] In summary, by introducing the SDF+DMT generated hair geometry prior, the problem of blank initialization of the hairstyle area after 3D GS reconstruction is overcome, providing a complete geometric reference for subsequent 3D Gaussian digital human reconstruction. 3D Gaussian particles need to be sampled on the surface of the 3D model (such as sampling Gaussian particles on the surface of the hair to simulate hair, and sampling on the surface of the body to simulate skin / clothing), and if there is no complete geometric model, the Gaussian particles will be initialized in chaos due to the lack of uniform geometric reference (such as hair particles floating outside the body).

[0115] S205: For the initial 3D model, sample Gaussian particles on the surface by 3D Gaussian sputtering technology, and render to obtain a digital human.

[0116] In this example, the core of the 3D Gaussian sputtering technology is to represent a 3D object with a large number of 3D Gaussian particles, and to generate a 2D image by particle projection superposition. Gaussian particles are a kind of 3D data unit with attributes, which can simulate different materials (skin opacity, hair semi-transparency, clothing texture).

[0117] Step 1: The essence of sampling is to select points on the surface of the model, so that the Gaussian particles are attached to the correct position, and the sampling strategy needs to be designed in combination with the detailed requirements of the 3D model (to avoid particle waste or detail loss).

[0118] For example, the sampling position is based on the principle of closely following the model surface: traverse all the triangular faces of the initial 3D model (the shirt of the body, the face, and the hair tips), and uniformly sample 3D points on the surface and the adjacent area (such as less than or equal to 0.5mm from the surface) of each triangular face. These points are the initial 3D positions of the Gaussian particles, ensuring that the particles do not deviate from the model geometry (such as hair particles only on the surface of the hair mesh, not floating outside the body).

[0119] The sampling density is based on the principle of more details and less roughness. The number of particles is dynamically adjusted according to the detail requirements of different areas of the model (avoiding excessive particles leading to computational pressure): high detail areas (hair tips, face, clothing texture): high sampling density, such as 2-3 particles per square millimeter for hair tips (to restore the thinness of hair), 1-2 particles per square millimeter for face (to capture fine lines on the nose and eye corners), and 1 particle per millimeter for shirt grid texture (to restore the grid boundary); rough areas (torso, back): low sampling density, such as 5-10 particles per square centimeter for torso (without excessive details, efficiency is considered).

[0120] Step 2: The obtained particles only have initial positions, and their size (control particle size), rotation (control particle orientation), color (control particle appearance), and opacity (control particle transparency) need to be optimized through 3D GS technology, so that the particles can simulate the visual effect of real materials.

[0121] For example, match the detail scale of the model area: hair particle size is set to 0.2-0.3mm (thin), face particle size is set to 0.3-0.5mm (delicate), and torso particle size is set to 1-2mm (coarse), to avoid excessive hair particle size leading to adhesion or excessive face particle size leading to computational redundancy.

[0122] Match the texture direction of the model surface: hair particles are rotated along the growth direction of the hair (such as short hair rotated along the arc of the top of the head, long hair rotated along the vertical direction), and shirt grid particles are rotated along the grid texture direction, to ensure that the texture is not skewed during rendering.

[0123] Extract from the vertex color of the initial 3D model: skin particles take the skin color of the face of the model (such as light skin color RGB: 255, 224, 189), hair particles take the hair color of the model (such as black RGB: 30, 30, 30), and shirt particles take the grid color (black or white).

[0124] Match the material characteristics: skin particles have an opacity of 0.9-1.0 (completely opaque), hair particles have an opacity of 0.7-0.9 (semi-transparent, reflecting the hair hierarchy), and shirt particles have an opacity of 0.9 (slightly transparent, more realistic).

[0125] Step 3: 3D Gaussian sputtering rendering, the core of sputtering is to project 3D Gaussian particles to the 2D camera plane, generate an image by particle superposition, and project the particles into a visual digital person.

[0126] In summary, although some traditional head avatars can achieve high-fidelity animation, it is extremely difficult to reconstruct a high-definition full body. It is usually necessary to reconstruct and track accurate geometry from dense multi-view video, and the rendering effect after reconstruction is often not clear, with fuzzy hairstyles, and the Figure 2The method combines the SMPLX human body model and the Gaussian sputtering digital human full-body reconstruction method of the implicit surface reconstruction technology, and performs fine reconstruction on the hairstyle area.

[0127] Specifically, in the multi-view image segmentation process, not only the face area is segmented, but also the hairstyle area is segmented, so as to implicitly represent the 3D hairstyle by using the 3D face key points (providing geometric constraints of the head) and the multi-view hairstyle images, and the 3D face key points are introduced as prior knowledge to ensure that the reconstructed hairstyle is grown on the head and perfectly matches the proportion and shape of the head, avoiding unreasonable phenomena such as hairstyle floating or penetrating the scalp. Further, an explicit triangular mesh model is extracted from the learned SDF field, and the SDF+DMT generated hair geometry prior model surface is introduced to initialize the Gaussian position and vertex color, so as to overcome the problem of blank initialization of the hairstyle area after 3DGS reconstruction and speed up the subsequent network fitting. The body mesh of the target object is spliced with the hairstyle mesh to form a complete initial geometric model of the body and the hair, and finally the hairstyle area is finely reconstructed in the full-body 3DGS digital human reconstruction process.

[0128] In one example, the 3D face key points of the target object and the multi-view hairstyle images are implicitly modeled to construct a hairstyle SDF field, as shown in Figure 3 The method includes the following steps:

[0129] S301: input the 3D face key points of the target object and the expression parameters into a geometric MLP network to obtain an initial SDF field and vertex features; the geometric MLP network is used to constrain the 3D face key points on the surface of the SDF field.

[0130] In this example, the expression parameters of the target object are obtained based on the SMPLX model. The vertex features refer to attribute information of each vertex in the SDF field, such as position, direction, color, texture information, etc.

[0131] The model class MeshMain includes two main multi-layer perception machine (MLP) networks: Geomlp (responsible for building geometry) and Colormlp (responsible for coloring).

[0132] The model class MeshMain includes two main multi-layer perception machine (MLP) networks: Geomlp (responsible for building geometry) and Colormlp (responsible for coloring).

[0133] 3D face landmarks can only provide sparse local constraints (e.g. the position of eye corners, mouth corners), but cannot fully describe the subtle deformation of the whole face. As a low-dimensional vector, expression parameters provide global expression information, helping Geomlp fill in reasonable geometric details between landmarks, ensuring that the whole face surface (including cheeks, forehead, etc.) changes naturally with expressions. In addition, since the segmentation loss and RGB loss are based on 2D images during loss optimization, these images contain expression information. If the SDF field cannot change with expressions, the model will have difficulty optimizing effectively through backpropagation, and expression parameters provide a clear guide to help the network quickly converge to the correct geometry.

[0134] Specifically, after receiving the coordinates of 3D space points and expression parameters, for each query point, Geomlp adjusts its internal weights according to the expression parameters and outputs the SDF value and vertex features of the point. This means that the same 3D point will have different SDF values under different expressions. For example, when smiling, the points around the mouth may be more outward (SDF value changes), while in a neutral expression, they may be more inward.

[0135] When training the Geomlp geometry network, the coordinates of sample 3D face landmarks are input into Geomlp, and it is forced to output an SDF value of 0. The network learns the mapping relationship between human head geometry and SDF field through a large amount of data, which anchors the basic structure of the face, ensuring the stability of the basic structure of the face and ensuring that the generated SDF field does not deviate from the true outline of the face.

[0136] It should be noted that the initial SDF field is used to represent the 3D face geometry shape and does not include hair.

[0137] S302: input the vertex features into the shading MLP network to obtain vertex colors.

[0138] In this example, the MLP is also a trained network that reads the vertex features provided by the geometry MLP and outputs the color that the vertex should present, achieving accurate color matching.

[0139] S303: render the initial SDF field and the vertex color into a 2D image to obtain a multi-view rendered face image.

[0140] In this example, the rendering process can be as follows:

[0141] For a given camera pose, the system will emit a ray through each pixel of the image.

[0142] Using the initial SDF field, the intersection point of each ray and the SDF surface (SDF = 0) is efficiently found through the Sphere Tracing algorithm.

[0143] At each intersection, the vertex feature vector from Geomlp is queried and input into Colormlp to compute the color of that point.

[0144] All the pixel colors are composited to get a predicted RGB face image.

[0145] S304: Loss computation between the multi-view rendered face image and the multi-view hairstyle image to get the hairstyle SDF field.

[0146] In this example, the RGB reconstruction loss: compare the predicted RGB image with the input ground truth RGB image (using L1 or L2 loss, for example). This loss forces the model to learn the correct appearance color.

[0147] Segmentation loss: this is the key to let the model learn the hairstyle geometry. Compare the predicted face image with the ground truth hairstyle segmentation map (usually using cross-entropy loss).

[0148] Since the hairstyle image explicitly indicates which regions in the image are hair, when the predicted face image does not match the ground truth segmentation map (for example, the model incorrectly classifies a certain pixel as face, but the ground truth label is hair), the gradient will be backpropagated through the differentiable renderer to Colormlp and Geomlp.

[0149] Geomlp adjusts its internal S field to change the geometry of the hair, so that the next time the rendering is done, that misclassified pixel will be projected onto the correct (hair) surface region. The supervision signal from the 2D segmentation map is successfully translated into a correction of the 3D geometry, thus sculpting the SDF model of the hairstyle.

[0150] That is, when the rendered face image does not match the ground truth segmentation map, the loss value will be backpropagated through the differentiable renderer to Geomlp and Colormlp. For the hairstyle region, the loss signal will force Geomlp to adjust its parameters so that the SDF value becomes zero (i.e., a surface is formed) at the 3D spatial location corresponding to the hairstyle.

[0151] If a ray passes through the hairstyle region in the hairstyle image, but the surface point rendered by the current SDF field is occupied by the face feature (i.e., misclassified), the loss will instruct Geomlp to push or pull the surface until the ray hits a point with hairstyle features.

[0152] Geomlp gradually defines a new surface region (hair) in 3D space by updating its weights, so that the zero level surface of the SDF field extends to the hair surface.

[0153] In summary, through the supervised loss of the segmentation map, the model can indirectly learn the 3D geometry of the hairstyle, even without direct 3D hairstyle data. Geomlp is forced to define the hairstyle surface in the SDF field through back propagation.

[0154] In one example, based on the DMT algorithm, the isosurface of the hairstyle SDF field is extracted, as Figure 4 As shown, comprising the following steps:

[0155] S401: According to the hairstyle SDF field, sample SDF values of the tetrahedral mesh.

[0156] In this example, a preset regular cubic mesh is generated, and each cubic unit in the cubic mesh is subdivided into multiple tetrahedrons to obtain a tetrahedral mesh.

[0157] A tetrahedron refers to a solid composed of 4 vertices and 4 triangular faces. Because the hairstyle mesh has many fine structures (such as hair tips of short hair), the cubic subdivision block is too large and is likely to miss details; and the tetrahedron has a small volume and flexible splicing, which can fit the fine and sharp shape such as hair tips (for example, the edge length of the tetrahedron in the hair tip area is set to 0.5 mm, which is finer than the hair, ensuring that no details are lost).

[0158] S402: According to the SDF values of the four vertices of the tetrahedron, use a differentiable linear interpolation formula to calculate the intersection points of the isosurface and the tetrahedral boundary.

[0159] In this example, the SDF values of the 4 vertices of each tetrahedron are calculated, and the SDF values are used to determine whether the tetrahedron crosses the hair surface (SDF=0).

[0160] Specifically, among the 4 vertices of a tetrahedron, there are points with negative SDF values (inside, such as in the hair) and points with positive SDF values (outside, such as in the air), indicating that the tetrahedron crosses the hair surface.

[0161] For the tetrahedron that crosses the surface, the intersection line where SDF=0 is found through mathematical calculation, that is, the intersection point of the edge of the tetrahedron and the hair surface is found.

[0162] S403: Connect the intersection points of different tetrahedrons into triangular facets to obtain an initial hairstyle mesh.

[0163] In this example, the intersection points of different tetrahedrons are connected into triangular facets, and the triangular facets are spliced to obtain an initial rough hairstyle mesh model.

[0164] For example, the hair tip region of a hairstyle, two vertices of a tetrahedron are inside the hair (SDF negative), two are outside (SDF positive), find the four intersections of these four edges with the hair tip surface (SDF = 0), connect them to form two small triangles, these small triangles are the grid outline of the hair tip.

[0165] S404: optimize the vertex positions of the initial hairstyle grid according to the differentiable rendering gradient, to obtain a hairstyle grid.

[0166] In this example, the vertex positions are optimized using the differentiable rendering gradient. First, render the initial rough hairstyle grid into an image using a certain camera perspective, and then compare it with the hairstyle image corresponding to the perspective. According to this correction direction, adjust the position of the grid vertex to make the shape of the grid closer to the real object.

[0167] Specifically, the vertex colors output by Colormlp are used with the initial hairstyle grid to generate hairstyle images at different perspectives through the differentiable renderer from different shooting device internal and external parameters.

[0168] The differentiable renderer is differentiable, not only can generate images, but also can calculate the difference (gradient) between the generated image and the real image for each pixel, and know which vertex position, color or normal change caused the difference.

[0169] Then, the multi-perspective rendered hairstyle image and the multi-perspective hairstyle image are used for loss calculation to optimize the initial hairstyle grid and vertex color in reverse.

[0170] The optimization process can be: through the differentiable renderer, the gradient of the vertex color and position is propagated to each vertex. The gradient of the vertex color is transmitted to Colormlp to guide it to better predict the color. The gradient of the vertex position is transmitted to the DMT module. The DMT module uses its differentiable characteristics to further propagate the gradient to the SDF field (i.e. how should the SDF value of each point change for the vertex position to be more accurate).

[0171] After the process of repeated rendering, comparison and optimization, the difference (loss value) between the multi-perspective rendered hairstyle image and the real multi-perspective hairstyle image is minimized, and the hairstyle grid model and vertex color at this time are accurate enough.

[0172] For example, the first rendered target object has shorter hair on the left side that is more fluffy than the real photo (high loss value), and the vertex of the short hair edge is moved 0.3 mm inward through the differentiable rendering gradient calculation, and the Colormlp is adjusted to make the edge vertex color darker, and then re-rendered. After several iterations, the fluffiness and color of the virtual short hair are almost the same as the real image.

[0173] In summary, through multi-view rendering and loss calculation, it is ensured that the digital person looks like the target object from any angle (such as the short hair arc on the front and side looks like).

[0174] In one example, for the initial 3D model, Gaussian particles are sampled on the surface by 3D Gaussian sputtering technology, and the digital person is rendered as shown in Figure 5 The method comprises the following steps:

[0175] S501: Sample Gaussian particles on the surface of the initial 3D model to determine the initial grid position of the Gaussian particles.

[0176] In this example, sampling (such as random sampling or uniform sampling) is performed on the surface of the initial 3D model to generate a large number of sampling points, and the 3D coordinates of the sampling points are taken as the initial positions of the Gaussian particles.

[0177] In this example, based on the consistency of the size in the spatial dimension, isotropic 3D Gaussian particles can be used to ensure isotropy between all 3D Gaussian functions by maintaining a uniform size in the dimension, and to speed up the attribute learning of the Gaussian. At the same time, isotropic Gaussian is more stable and is not prone to produce undesirable shapes due to poor rotation and scaling cooperation.

[0178] Isotropy: The size of each Gaussian particle in the x, y, and z directions is consistent (such as a particle of 0.3mm in the x, y, and z directions, which is a sphere).

[0179] If the particle is anisotropic (such as 0.5mm in the x direction and 0.2mm in the y direction, which is an ellipsoid), more parameters need to be learned (the size of three directions needs to be adjusted), and the training is slower. And isotropic only needs to learn one size parameter, which can significantly speed up the training speed while ensuring the consistency of the particle shape (especially suitable for hair, skin and other relatively uniform materials).

[0180] S502: Predict the initial attribute parameters of each Gaussian particle according to the pose parameters of the target object and the initial grid position through the GSMLP network.

[0181] In this example, the GSMLP class includes Posemlp (pose MLP network), Colormlp (color MLP network), and Gsattmlp (Gaussian attribute MLP network).

[0182] Wherein, Posemlp is responsible for adjusting the position of Gaussian particles according to the body posture. For example, when the target object turns its head, the Gaussian particles of the hair need to be offset according to the head posture to avoid the problem of not moving the hair. Colormlp is responsible for coloring the Gaussian particles. Gsattmlp is responsible for predicting all the detailed attributes (position fine-tuning, rotation direction, size, color) of the Gaussian particles, so that each particle is more consistent with the real material (such as the fine rotation of the hair and the luster of the skin).

[0183] Through the strategy of decomposing the task first and then fusing it, each sub-network is easier to train and more focused, and the overall model converges faster, better, and more stably.

[0184] Based on this, the pose parameters of the target object and the initial grid position of all Gaussian particles are input into Posemlp, and the vertex feature of each Gaussian particle is output. This feature encodes the appearance and geometric characteristics of the position point at a specific pose.

[0185] Then, the vertex feature of each Gaussian particle is input into the Colormlp network to obtain the vertex color of each Gaussian particle.

[0186] For example, the hair tip particle outputs light black, and the cheek particle outputs light skin color.

[0187] It should be noted that during the training of Colormlp, the parameters trained in the implicit modeling process are loaded and fine-tuned, which plays a role in knowledge transfer.

[0188] Then, the vertex feature and vertex color of all Gaussian particles are input into the Gaussian attribute MLP network to obtain the initial attribute parameters of each Gaussian particle.

[0189] The initial attribute parameters include xyz offset, rotation, color, and scale. Among them, the xyz offset refers to the amount of offset of the Gaussian center point from the initial grid position. The rotation refers to the rotation of the Gaussian ellipsoid, the color refers to the color of the Gaussian particle itself, and the scale refers to the size of the Gaussian particle in each axis.

[0190] In summary, the initial attributes (position, color, shape) provided are already very close to the final ideal state, rather than being randomly initialized, so that the optimization process does not need to start from scratch and only needs to be fine-tuned, greatly reducing the number of iterations, thereby significantly speeding up the entire 3D Gaussian model training process, and avoiding the optimization process from falling into a local optimal solution, which helps to generate clearer and more detailed rendering results and improve the final quality.

[0191] S503: Render the initial 3D model according to the initial attribute parameters of each Gaussian particle to obtain the target attribute of each Gaussian particle.

[0192] In this example, the initial mesh position is adjusted according to the position offset to obtain the initial adjusted mesh position of the Gaussian particle.

[0193] According to the initial adjusted mesh position and the initial attribute parameters other than the position offset, the initial 3D model is rendered to obtain a rendered multi-view image.

[0194] Wherein, the attributes (initial adjusted mesh position, rotation, color, scaling) of all Gaussian particles can be rendered into a 2D image using a standard 3D GS differentiable renderer.

[0195] The rendered multi-view image is loss calculated with the multi-view image to iteratively optimize the GSMLP network parameters by a back propagation algorithm until the loss value converges to a predetermined threshold to obtain the target attribute of each Gaussian particle.

[0196] It should be noted that the final mesh position of the Gaussian particle refers to the adjusted mesh position generated in the last iteration plus the offset in the target attribute.

[0197] Thus, it is ensured that the particles are basically on the mesh surface (initial coordinates) and can be dynamically adjusted (xyz offset) according to the body posture to capture more detailed geometric details. For example, when the target object bends down, the hair particles on the top of the head will move slightly downward with the head to avoid the hair floating.

[0198] In this example, two key loss functions are set to constrain the optimization process: color loss and position loss.

[0199] Specifically, Lossrgb (color loss): compare the difference between each pixel of the rendered image and the real multi-view image (such as L1 loss). Thus, the system can learn the correct appearance, lighting and details to make the rendering result more realistic and optimize the parameters of the GSMLP network.

[0200] Lossposition (position loss): constrain the xyz offset predicted by Gsattmlp to be not too large (such as L2 regularization), that is, constrain the position deviation of the Gaussian center coordinates relative to the body and hair model surface.

[0201] Thus, it is prevented that the Gaussian particles move too far and it is ensured that the center position of the Gaussian is still near the Mesh grid. This allows the model to capture details and ensures that the overall structure is stable according to the above geometric prior, avoiding training divergence or artifacts.

[0202] For example, if a Gaussian particle is too far from the mesh surface (e.g. 0.8mm), the value of lossposition will become large, adjusting the xyz offset output by Gsattmlp to pull the particle back near the mesh surface (e.g. within 0.1mm), ensuring that the particle is always attached to the model and rendering errors such as penetration, floating, etc. do not occur.

[0203] S504: render the initial 3D model according to the target attributes of each Gaussian particle to obtain a digital human.

[0204] Wherein, a standard 3DGS differentiable renderer can be used to render the attributes (final mesh position, rotation, color, scale) of all Gaussian particles into a 2D image.

[0205] In summary, Gaussian particles are sampled on the surface of the initial 3D model including the hairstyle mesh, serving as the initial positions of the Gaussian particles. Through this initialization, the edge, level and volume of the hair strands are effectively restored, and the problem of random initialization of Gaussian particles by 3DGS algorithm, which may result in hollow or hole in the reconstructed hair, is avoided.

[0206] That is, the initialization difficulty of 3DGS for thin structures (such as hair) is solved, and the hollow or blur in the hairstyle area is avoided. The training is faster and more stable, the Gaussian starting point is good, and the attributes are predicted by the network, which is more efficient and more structured than directly randomly optimizing thousands of independent Gaussian attributes.

[0207] In one example, the initial 3D model is rendered according to the target attributes of the Gaussian particles to obtain a digital human, as shown in Figure 6 The method comprises the following steps:

[0208] S601: render the initial 3D model according to the target attributes of the Gaussian particles to obtain an initial 3D Gaussian sputtering rendering image.

[0209] Wherein, the initial 3D Gaussian sputtering rendering image is rendered by a 3D Gaussian sputtering (3DGS) algorithm. Although 3DGS is fast, it may produce small flaws or blur at low resolution or edges. Gaussian attribute rendering is a low-resolution image, such as 512x512x3.

[0210] S602: perform super-resolution reconstruction on the initial 3D Gaussian sputtering rendering image by a pre-trained lightweight attention upsampling network to obtain a 3D Gaussian sputtering rendering image with higher resolution.

[0211] In this example, the main reason for the blurriness in the 3DGS full-body digital human rendering is that Gaussian properties are rendered as low-resolution images. Furthermore, the number of Gaussian distributions is too small relative to the head, failing to provide sufficient detail for complex details such as facial features, hairstyles, and clothing textures. Therefore, while increasing the number of Gaussian distributions can improve some detail, increasing the number of Gaussian distributions for the entire body render would only increase computational complexity, and would still be limited by the overall image sharpness. Therefore, to address the blurriness problem in the 3DGS full-body digital human rendering, a lightweight attention-based upsampling network for 3D Gaussian sputtering rendered images is designed, with the SRnet network as its basic structure. Customized for the characteristics of 3DGS rendered images (such as natural color transitions and details presented in particle form, like hair strands and skin textures), the repair effect is better than general-purpose tools.

[0212] Since 3DGS is a real-time rendering technology, its lightweight design allows the network to be integrated into real-time applications as a highly efficient post-processing module.

[0213] Furthermore, the attention mechanism can automatically guide the network to focus on difficult-to-repair details, thereby generating sharper and more realistic high-resolution images. For example, details such as a person's eyes, hair texture, and the edges of objects require the model to allocate more computing resources for fine restoration; while large areas of solid-color sky or walls can be handled simply.

[0214] In addition, upsampling convolutional networks extract image features (such as color, texture, and contour) through a series of convolutional layers, and then enlarge the feature maps to the target size through upsampling layers (such as subpixel convolution, PixelShuffle, transposed convolution, etc.), finally reconstructing a high-resolution image.

[0215] In addition, the dual loss function for training the network is set as follows:

[0216] L1 color loss: Calculates the absolute difference of each pixel's color value between the high-resolution image output by the lightweight network and the real multi-view image, and then averages the results. This ensures color fidelity and the accuracy of the overall structure. It ensures that the generated image is as close as possible to the real target at the pixel level, avoiding severe color casts or distortions. Compared to the more commonly used L2 loss, L1 loss is less sensitive to outliers, and the trained images are generally sharper.

[0217] Perceptual loss: Instead of comparing pixel values, compare high-level features. Both the rendered generated image and the real image of the corresponding view are input into a pre-trained large network (such as VGGNet), extract the feature map of a certain layer in the middle, and then calculate the difference (such as L2 loss) between the two feature maps. Thus, visual similarity is ensured. For example, the overall fluffy feeling of short hair, the three-dimensional feeling of shirt grid, and the overall contour of the face. The human eye is more sensitive to details, textures, and style consistency than absolute pixel values. Perceptual loss encourages the network to generate images that are true in feeling, even if the pixel values do not completely match, but the texture and style look more natural and comfortable.

[0218] In summary, by extracting color and texture features, the overall display effect is further improved after upsampling. 3DGS can only render low-resolution images for real-time performance, and SRnet fills the gap in high-definition repair, making 3DGS digital people fast and clear. Without significantly increasing the rendering time of 3DGS, an efficient post-processing network significantly improves the resolution and visual quality of the final output image.

[0219] Based on the same idea, some embodiments of the present application also provide a device and a non-volatile computer storage medium corresponding to the above method.

[0220] Figure 7 A structural schematic diagram of a Gaussian digital human reconstruction device provided by an embodiment of the present application includes:

[0221] at least one processor; and

[0222] a memory in communication connection with the at least one processor; wherein

[0223] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the Gaussian digital human reconstruction method of any one of the above.

[0224] Figure 8 A structural schematic diagram of a Gaussian digital human reconstruction device provided by an embodiment of the present application includes:

[0225] The segmentation unit 801 performs segmentation processing on the multi-view image of the target object to extract the multi-view hairstyle image;

[0226] The modeling unit 802 implicitly models the 3D face key points of the target object and the multi-view hairstyle image to construct a hairstyle SDF field;

[0227] The extraction unit 803 extracts the isosurface of the hairstyle SDF field based on the DMT algorithm to obtain a hairstyle mesh;

[0228] The splicing unit 804 splices the body mesh of the target object with the hairstyle mesh to obtain an initial 3D model of the target object.

[0229] The rendering unit 805 samples Gaussian particles on a surface by using a 3D Gaussian sputtering technique for the initial 3D model, and renders to obtain a digital human.

[0230] Some embodiments of the present application provide a non-volatile computer storage medium storing computer executable instructions, which can execute the Gaussian digital human reconstruction method described in any of the above.

[0231] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts of each of the embodiments can be referred to each other. Each of the embodiments mainly describes the difference from other embodiments. In particular, the device, equipment and medium embodiments are basically similar to the method embodiments, and thus the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0232] The device, equipment and medium provided by the embodiments of the present application are one-to-one corresponding to the method, and thus the device, equipment and medium also have the similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device, equipment and medium will not be described here.

[0233] The above only describes the embodiments of the present application and is not used to limit the present application. The present application can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the technical principles of the present application shall fall within the protection scope of the present application.

Claims

1. A method of Gaussian digital human reconstruction, characterized by, The method comprises: segmenting a multi-view image of a target object to extract a multi-view hairstyle image; performing implicit modeling on 3D face key points of the target object and the multi-view hairstyle image to construct a hairstyle SDF field, specifically comprising: inputting the 3D face key points of the target object and expression parameters into a geometric MLP network to obtain an initial SDF field and vertex features; the geometric MLP network is used to constrain the 3D face key points on the surface of the SDF field; inputting the vertex features into a color MLP network to obtain vertex colors; rendering the initial SDF field and the vertex colors into a 2D image to obtain a multi-view rendered face image; performing loss calculation between the multi-view rendered face image and the multi-view hairstyle image to obtain a hairstyle SDF field; based on a DMT algorithm, extracting an isosurface of the hairstyle SDF field to obtain a hairstyle mesh; splicing a body mesh of the target object and the hairstyle mesh to obtain an initial 3D model of the target object; rendering a digital human by sampling Gaussian particles on the surface of the initial 3D model through a 3D Gaussian sputtering technology.

2. The method of claim 1, wherein, After the segmentation processing of the multi-view image of the target object to extract the multi-view hairstyle image, the method further comprises: extracting body key points of the multi-view segmentation image; matching the key points of different multi-view segmentation images according to the internal and external parameters of the shooting device to establish a cross-view alignment relationship between the multi-view segmentation images; using multi-view stereo vision technology, converting the key points of different multi-view segmentation images into 3D key points according to the cross-view alignment relationship between the multi-view segmentation images.

3. The method of claim 1, wherein, The method for extracting an isosurface of the hairstyle SDF field based on the DMT algorithm to obtain a hairstyle mesh specifically comprises: sampling SDF values of a tetrahedral mesh according to the hairstyle SDF field; calculating the intersection points of the isosurface and the tetrahedral boundary using a differentiable linear interpolation formula according to the SDF values of the four vertices of the tetrahedron; connecting the intersection points of different tetrahedrons into triangular patches to obtain an initial hairstyle mesh; optimizing the vertex positions of the initial hairstyle mesh according to the differentiable rendering gradient to obtain a hairstyle mesh.

4. The method of claim 1, wherein, The method for rendering a digital human by sampling Gaussian particles on the surface of the initial 3D model through a 3D Gaussian sputtering technology specifically comprises: sampling Gaussian particles on the surface of the initial 3D model to determine the initial mesh positions of each Gaussian particle; predicting the initial attribute parameters of each Gaussian particle through a GSMLP network according to the pose parameters of the target object and the initial mesh positions; rendering and loss calculation on the initial 3D model according to the initial attribute parameters of each Gaussian particle to obtain the target attributes of each Gaussian particle; rendering the initial 3D model according to the target attributes of each Gaussian particle to obtain a digital human of the target object.

5. The method of claim 4, wherein, The method for predicting the initial attribute parameters of each Gaussian particle through a GSMLP network according to the pose parameters of the target object and the initial mesh positions specifically comprises: inputting the pose parameter of the target object and initial grid positions of all Gaussian particles into a pose MLP network to obtain vertex features of each Gaussian particle; inputting the vertex features of each Gaussian particle into a color MLP network to obtain vertex colors of each Gaussian particle; inputting the vertex features and vertex colors of all Gaussian particles into a Gaussian attribute MLP network to obtain initial attribute parameters of each Gaussian particle.

6. The method of claim 4, wherein, The initial 3D model is rendered and loss is calculated according to the initial attribute parameters of each Gaussian particle to obtain target attributes of each Gaussian particle, and the method specifically comprises: adjusting the initial grid positions according to the position offset in the initial attribute parameters to obtain initial adjusted grid positions of the Gaussian particles; rendering the initial 3D model according to the initial adjusted grid positions and the initial attribute parameters except the position offset to obtain a rendered multi-view image; loss is calculated between the rendered multi-view image and the multi-view image to iteratively optimize the GSMLP network parameters through a back propagation algorithm until the loss value converges to a predetermined threshold, and the target attributes of each Gaussian particle are obtained.

7. The method of claim 4, wherein, The initial 3D model is rendered according to the target attributes of the Gaussian particles to obtain a digital human of the target object, and the method specifically comprises: rendering the initial 3D model according to the target attributes of the Gaussian particles to obtain an initial 3D Gaussian splatting rendering image; performing super-resolution reconstruction on the initial 3D Gaussian splatting rendering image through a pre-trained lightweight attention up-sampling network to obtain a 3D Gaussian splatting rendering image with higher resolution.

8. A Gaussian digital human reconstruction device, characterized by, comprise: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the Gaussian digital human reconstruction method of any one of claims 1-7.

9. A Gaussian digital human reconstruction apparatus, characterized by, comprise: a segmentation unit that performs segmentation processing on a multi-view image of a target object to extract a multi-view hairstyle image; a modeling unit that inputs 3D face key points and expression parameters of the target object into a geometry MLP network to obtain an initial SDF field and vertex features; the geometry MLP network is used to constrain the 3D face key points on the surface of the SDF field; the vertex features are input into a color MLP network to obtain vertex colors; the initial SDF field and the vertex colors are rendered into a 2D image to obtain a multi-view rendered face image; loss is calculated between the multi-view rendered face image and the multi-view hairstyle image to obtain a hairstyle SDF field; an extraction unit that extracts an isosurface of the hairstyle SDF field based on a DMT algorithm to obtain a hairstyle grid; a splicing unit that splices a body grid of the target object and the hairstyle grid to obtain an initial 3D model of the target object; a rendering unit that samples Gaussian particles on the surface of the initial 3D model through a 3D Gaussian splatting technique to render a digital human.

Images