Editable digital human modeling method and device based on neural texture and three-dimensional Gaussian

Abstract

The invention provides an editable digital human modeling method and device based on neural texture and three-dimensional Gaussian, and the method comprises the steps: carrying out the modeling of the expression, posture and appearance of a digital human through a three-dimensional Gaussian splashing model according to the motion sequence video data of a target object, and obtaining a Gaussian model of the digital human; modeling the expression, posture and appearance of the digital human through the neural texture model to obtain a neural texture model of the digital human; pruning the Gaussian model, and performing hybrid rendering according to a rendered image output by the pruned Gaussian model and a rendered image output by the neural texture model of the digital human to obtain a hybrid rendered image; and optimizing a Gaussian model of the digital human based on the mixed rendered image and the action sequence video data to obtain an editable digital human model. According to the method, the storage overhead and the structural complexity of the digital human model are reduced, and the high-frequency details and editing efficiency of the skin surface of the digital human model are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an editable digital human modeling method and device based on neural texture and three-dimensional Gaussian. Background Art

[0002] 3D Gaussian is based on each point in the sparse point cloud obtained by Structure from Motion (SFM), and introduces 3D Gaussian distribution as scene representation. Each 3D Gaussian distribution consists of parameters that can be used as learning, such as position, rotation component, and perspective-related color. It is combined with Gaussian splattering technology to achieve fast differentiable rendering. Therefore, 3D Gaussian becomes the optimal choice for 3D representation in terms of training time, image quality, and real-time performance.

[0003] In the related technology, the Gaussian is bound to the grid through a mixed representation model of the grid and Gaussian to achieve effective driving of the three-dimensional Gaussian, and then build a digital human model; however, the reconstructed digital human has problems such as large storage space overhead, lack of high-frequency details on the skin surface, and difficulty in optimizing complex structures such as the hands, and is difficult to edit effectively, so that the reconstructed digital human is often limited to specific scenes and has weak generalization. Summary of the invention

[0004] The present invention provides an editable digital human modeling method and device based on neural texture and three-dimensional Gaussian, which are used to solve the defects of digital human models constructed by the prior art, such as large storage space overhead, missing high-frequency details on the skin surface, difficult optimization and effective editing of complex structures such as hands, resulting in poor generalization of the reconstructed digital human, and improve the generalization of three-dimensional Gaussian modeling in different scenarios.

[0005] The present invention provides an editable digital human modeling method based on neural texture and three-dimensional Gaussian, comprising: Modeling the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object by using a three-dimensional Gaussian splash model to obtain the Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters and appearance parameters of the target object; Modeling the expression, posture and appearance of the digital human according to the action sequence video data through a neural texture model to obtain a neural texture model of the digital human; The Gaussian model is pruned, and a mixed rendering is performed based on a rendered image output by the pruned Gaussian model and a rendered image output by the neural texture model of the digital human to obtain a mixed rendering image; the Gaussian model of the digital human is optimized based on the mixed rendering image and the action sequence video data to obtain an editable digital human model.

[0006] According to an editable digital human modeling method based on neural texture and three-dimensional Gaussian provided by the present invention, the action sequence video data is collected by a monocular camera; The method of modeling the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object by using the three-dimensional Gaussian splash model to obtain the Gaussian model of the digital human includes: Acquire SMPL-X model parameters, frame mask and camera parameters of the monocular camera of the target video frame from the action sequence video data; the SMPL-X model parameters include the expression parameters, the posture parameters and the appearance parameters; Initialize local three-dimensional Gaussian parameters of the SMPL-X model according to the SMPL-X model parameters; the local three-dimensional Gaussian parameters include position parameters, rotation parameters, scale parameters, color parameters, opacity parameters, index parameters and semantic labels; Calculate the global three-dimensional Gaussian parameters of the SMPL-X model based on the local three-dimensional Gaussian parameters and the SMPL-X model parameters; Rendering the target video frame according to the global three-dimensional Gaussian parameters and the camera parameters based on a Gaussian renderer to obtain a Gaussian rendered image, and calculating a color loss according to the target video frame and the target video frame; A first comprehensive loss is obtained based on the color loss, the perceptual loss, the structural similarity loss, the Sobel operator and the regularization term. Based on the first comprehensive loss, the three-dimensional Gaussian splash model is iteratively trained by gradient backpropagation to obtain the Gaussian model.

[0007] According to the method for modeling an editable digital human based on neural texture and three-dimensional Gaussian provided by the present invention, the expression, posture and appearance of the digital human are modeled according to the action sequence video data by using the neural texture model to obtain the neural texture model of the digital human, which includes: Initialize the differentiable rasterizer based on the triangle mesh information, vertex coordinates and index of the SMPL-X model; Based on the initialized differentiable rasterizer, the target video frame is rasterized according to the triangle mesh information corresponding to the target video frame in the SMPL-X model, the vertex coordinates, the index and the optimized neural texture to obtain a neural texture rendered image, and the color loss is calculated according to the neural texture rendered image and the target video frame; A second comprehensive loss is obtained based on the color loss, perceptual loss, structural similarity loss and total variation algorithm, and the neural texture model is iteratively trained by gradient backpropagation based on the second comprehensive loss to obtain the neural texture model.

[0008] According to an editable digital human modeling method based on neural texture and three-dimensional Gaussian provided by the present invention, pruning the Gaussian model includes: Determine the opacity threshold of the Gaussian renderer according to the depth image, and determine the semantic label of the target to be pruned based on the frame mask of the target video frame; wherein the depth map is obtained by rasterizing the target video frame according to the triangle mesh information, vertex coordinates, indexes and optimized neural texture corresponding to the target video frame in the SMPL-X model through the initialized differentiable rasterizer; the frame mask is obtained through the action sequence video data; Based on the Gaussian renderer, the Gaussian points inside the triangular mesh corresponding to the Gaussian model and the hands and arms are pruned according to the opacity threshold and the semantic label of the target to be pruned, so as to obtain the pruned Gaussian model.

[0009] According to an editable digital human modeling method based on neural texture and three-dimensional Gaussian provided by the present invention, the Gaussian model of the digital human is optimized based on the mixed rendering image to obtain the editable digital human model, which includes: Calculating a loss value according to the mixed rendered image and the target video frame in the action sequence video data based on a third comprehensive loss, and updating the parameters of the Gaussian model according to the loss value, and obtaining the editable digital human model when the Gaussian model converges; The third comprehensive loss is determined based on the first comprehensive loss and the 2-norm of Gaussian opacity, and the first comprehensive loss is determined based on color loss, perceptual loss, structural similarity loss, Sobel operator and regularization term.

[0010] According to the method for modeling an editable digital human based on neural texture and three-dimensional Gaussian provided by the present invention, after obtaining the editable digital human model, the method further comprises: Based on the editable digital human model, the target three-dimensional Gaussian representation is edited according to the user input instruction to render the three-dimensional virtual scene of the digital human.

[0011] The present invention also provides an editable digital human modeling device based on neural texture and three-dimensional Gaussian, comprising: The first training module is used to model the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object through a three-dimensional Gaussian splash model to obtain the Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters and appearance parameters of the target object; A second training module is used to model the expression, posture and appearance of the digital human according to the action sequence video data through a neural texture model to obtain a neural texture model of the digital human; The third training module is used to prune the Gaussian model, and perform mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; based on the mixed rendered image and the action sequence video data, the Gaussian model of the digital human is optimized to obtain an editable digital human model.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, an editable digital human modeling method based on neural texture and three-dimensional Gaussian as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described editable digital human modeling methods based on neural texture and three-dimensional Gaussian.

[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned editable digital human modeling methods based on neural texture and three-dimensional Gaussian.

[0015] The editable digital human modeling method and device based on neural texture and three-dimensional Gaussian provided by the present invention model the expression, posture and appearance of the digital human corresponding to the target object through a three-dimensional Gaussian splash model, and then model the expression, posture and appearance of the digital human through a neural texture model, then prune the modeled Gaussian model and perform mixed rendering according to the rendered image output by the model and the rendered image output by the neural texture model to obtain a mixed rendered image, and finally optimize the Gaussian model of the digital human using the mixed rendered image and action sequence video data to obtain an editable digital human model, thereby reducing the storage overhead and structural complexity of the digital human model, and improving the high-frequency details and editing efficiency of the skin surface of the digital human model, thereby improving the generalization of three-dimensional Gaussian modeling in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the editable digital human modeling method based on neural texture and three-dimensional Gaussian provided by the present invention.

[0018] Figure 2 This is one of the structural schematic diagrams of the editable digital human modeling device based on neural texture and three-dimensional Gaussian provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of an editing device for an editable digital human model provided by the present invention.

[0020] Figure 4 This is the second structural schematic diagram of the editable digital human modeling device based on neural texture and three-dimensional Gaussian provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Combine the following Figure 1-Figure 4 Describe the editable digital human modeling method and device based on neural texture and three-dimensional Gaussian of the present invention Figure 1 is a flow chart of the editable digital human modeling method based on neural texture and three-dimensional Gaussian provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110: Model the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object through a three-dimensional Gaussian splash model to obtain a Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters and appearance parameters of the target object.

[0024] In this step, the target object includes a moving target, such as a person, an animal, an aircraft, or a robotic device.

[0025] Taking the target object as a person as an example, video data of a specific person is collected through a video acquisition device (such as a camera, mobile phone, etc.); during the acquisition process, the camera shooting angle and shooting parameter settings remain unchanged, for example, the video resolution is set to 1920 × 1080 pixels and the frame rate is 30 frames per second.

[0026] In this embodiment, the person being collected can make large body movements such as kicking, dancing, bending, etc. In addition, the movements can also include complex hand movements such as counting, playing musical instruments, and using tools.

[0027] In this embodiment, the data collection time is 1 minute or more (number of frames N ≥ 1800 frames). The longer the time, the more perspectives of the subject are captured, the richer the virtual human movements and expressions contained therein, and the better the digital human effect that can be reconstructed in the end.

[0028] Furthermore, the action sequence video data is collected by a monocular camera; the expression, posture and appearance of the digital human corresponding to the target object are modeled according to the action sequence video data of the target object by a three-dimensional Gaussian splash model, and the Gaussian model of the digital human includes: (1) Obtain the SMPL-X model parameters, frame mask, and camera parameters of the monocular camera of the target video frame from the action sequence video data; the SMPL-X model parameters include expression parameters, posture parameters, and appearance parameters.

[0029] In this embodiment, the action sequence video data of the acquisition object in the natural scene is collected and processed to obtain a monocular digital human training data set.

[0030] Specifically, the corresponding image of the video frame t is extracted frame by frame from the collected action sequence video data , and use the following formula according to the set field of view (for example, the field of view angle ), the length of the picture w and width h Calculate the intrinsic parameters of the monocular camera: ; Among them, the external parameters of the monocular camera Set to the 4×4 identity matrix.

[0031] Specifically, for the video frame image data in the action sequence video data, this embodiment uses an open source posture estimation method to identify the SMPL-X model parameters of the captured object in each video frame. ,in, , , They respectively represent the posture parameters, shape parameters and expression parameters of the corresponding video frame of the acquisition object.

[0032] In this embodiment, the posture parameters It can be further divided into overall posture , head posture , joint hand posture and other poses , and combined with the open source object segmentation model SAM2 to obtain the mask of the object collected in each video frame , The mask corresponding to the background, Masks corresponding to the head, arms and clothes respectively.

[0033] In this embodiment, the video frame image data, camera parameters, SMPL-X model parameters, and image mask are matched one by one according to the sequence of video frames, and the acquisition data of the tth frame of the video can be obtained, which is expressed as: ;in, Represents a two-dimensional image of a video frame, Represents the SMPL-X human action posture expression parameters corresponding to time t. Indicates the mask corresponding to the video frame image; They represent the extrinsic and intrinsic parameters of a globally consistent monocular camera, respectively.

[0034] (2) Initialize the local three-dimensional Gaussian parameters of the SMPL-X model according to the SMPL-X model parameters; the local three-dimensional Gaussian parameters include position parameters, rotation parameters, scale parameters, color parameters, opacity parameters, index parameters and semantic labels.

[0035] In this embodiment, the SMPL-X parameters of the 0th frame obtained by (1) above are Perform linear blending skinning to obtain the vertex and mesh representation of the corresponding SMPL-X model , and They respectively represent the set of all mesh vertex coordinates and mesh indexes at the current moment.

[0036] In this embodiment, the SMPL-X model includes multiple triangular meshes, and the corresponding first The local three-dimensional Gaussian parameters of a triangular mesh are as follows: The local three-dimensional Gaussian parameters are expressed as: ;in, is the position parameter of the three-dimensional Gaussian, initialized to the origin of the local coordinate system ; is the rotation parameter of the three-dimensional Gaussian, expressed in quaternion form, initialized to ; is the scale parameter of the three-dimensional Gaussian, initialized to 0.0000001; is the color parameter of the three-dimensional Gaussian, initialized to a random value; is the opacity parameter of the three-dimensional Gaussian, initialized to 1; is the index parameter of the three-dimensional Gaussian, indicating the index of the attached triangle mesh, initialized to ; is the semantic label of the three-dimensional Gaussian, initialized to 0.

[0037] (3) Calculate the global three-dimensional Gaussian parameters of the SMPL-X model based on the local three-dimensional Gaussian parameters and the SMPL-X model parameters.

[0038] In this embodiment, based on the local three-dimensional Gaussian parameters obtained in (2) above, the SMPL-X parameters at the tth moment are subjected to linear mixed skinning to obtain the representation of the SMPL-X vertices and mesh at the current moment: ; Then for the current moment triangular mesh, calculate vertex centers , quaternion rotation , and scale , transform the local three-dimensional Gaussian parameters into global three-dimensional Gaussian parameters .

[0039] in, = , = , = .

[0040] (4) Based on the Gaussian renderer, the target video frame is rendered according to the global three-dimensional Gaussian parameters and the camera parameters to obtain a Gaussian rendered image, and the color loss is calculated based on the target video frame and the target video frame.

[0041] In this embodiment, a Gaussian renderer is used according to the global three-dimensional Gaussian parameters and the camera parameters. Render the current frame image to get the rendered image of the current frame , combined with the target video frame Compute color loss over the masked region.

[0042] For example, first obtain a binary mask with the same size as the rendered image and the original image, multiply the mask by the image element by element to extract the region of interest in the rendered image and the original image, and for the extracted region of interest, calculate the color difference of each pixel between the rendered image and the original image, and then convert the color difference into a loss value; the color loss function is calculated by the following formula: = * ; in, is the color loss value, is the mask corresponding to the background ( A mask used to separate a person as foreground from the background. The foreground area represents the person and is usually white, while the background is usually black).

[0043] (5) The first comprehensive loss is obtained by combining color loss, perceptual loss, structural similarity loss, Sobel operator and regularization term. Based on the first comprehensive loss, the three-dimensional Gaussian splash model is iteratively trained by gradient backpropagation to obtain a Gaussian model.

[0044] In this embodiment, after obtaining the above color loss, combined with the perception loss , structural similarity loss (used to monitor image quality), (corresponding to the Sobel operator, used to enhance high-frequency details) and regularization term (Used to address the problem of non-smooth Gaussian rendering results and constrain the standard deviation of adjacent Gaussian attributes) to calculate the loss of the first-order derivative between the rendered image and the reference image, that is, the first comprehensive loss; the final first comprehensive loss can be expressed as: ; Among them, the hyperparameters , , , .

[0045] In this embodiment, the first comprehensive loss is used The attribute parameters of the three-dimensional Gaussian are optimized by gradient backpropagation, and the three-dimensional Gaussian is split and cloned using the densification operation in the three-dimensional Gaussian method, so that the three-dimensional Gaussian is adaptive to the geometry and appearance distribution, and a trained Gaussian model can be obtained.

[0046] In some embodiments, after the Gaussian model is optimized and generated by the step corresponding to step (5) above, the index parameters of the three-dimensional Gaussian model are updated by introducing a Gaussian walking strategy on the triangular mesh surface.

[0047] Specifically, if the position of the Gaussian during the optimization process exceeds the range of the triangle mesh to which it is attached, the index parameters of the three-dimensional Gaussian are updated Index for the new triangle mesh .

[0048] Step 120: Model the expression, posture and appearance of the digital human according to the action sequence video data through the neural texture model to obtain the neural texture model of the digital human.

[0049] In this step, the neural texture model uses a neural network to learn and store the appearance information of the object surface for high-quality texture rendering and editing.

[0050] In this embodiment, the expression, posture and appearance of the digital human are modeled according to the action sequence video data by using the neural texture model, and the neural texture model of the digital human is obtained, including: (1) Initialize the differentiable rasterizer renderer based on the triangle mesh information, vertex coordinates and indices of the SMPL-X model.

[0051] In this embodiment, after freezing and disabling the three-dimensional Gaussian model obtained in step 110, the 0th frame triangle mesh according to the 0th frame SMPL-X parameters obtained in step 110 is Set the parameters of the differentiable rasterizer Nvdiffrast , and read the initialized neural texture ;in, Used to store the triangle mesh information of the SMPL-X model. Storage of SMPL-X model vertices coordinate, Storage of SMPL-X models Index, used for subsequent texture mapping operations; and Corresponding to the width and height of the image.

[0052] (2) Based on the initialized differentiable rasterizer renderer, the target video frame is rasterized according to the triangle mesh information, vertex coordinates, index and optimized neural texture corresponding to the target video frame in the SMPL-X model to obtain a neural texture rendered image, and the color loss is calculated based on the neural texture rendered image and the target video frame.

[0053] In this embodiment, SMPL-X triangle mesh at the moment With the above and The two images are sent to the differentiable rasterizer Nvdiffrast together, and the rendered image is obtained after rasterization. With Depth Map .

[0054] In this embodiment, according to the rendered image and the target video frame Calculate the color loss in the masked area, as described above based on the rendered image and the target video frame The color loss is calculated in the mask area in the same way, and will not be described in detail in this embodiment.

[0055] Specifically, the color loss can be calculated using the following formula: = * .

[0056] (3) A second comprehensive loss is obtained based on color loss, perceptual loss, structural similarity loss and total variation algorithm. Based on the second comprehensive loss, the neural texture model is iteratively trained by gradient backpropagation to obtain a neural texture model.

[0057] In this embodiment, after obtaining the above color loss, combined with the perception loss , structural similarity loss and (corresponding to the total variation algorithm, used to maintain the smoothness of the texture map and reduce noise) to calculate the loss of the first-order derivative between the rendered image and the reference image, that is, the second comprehensive loss; the final loss function can be expressed as: = ; Among them, the hyperparameters Keep consistent with the first stage optimization Gaussian, =0.2.

[0058] In this embodiment, based on the second comprehensive loss Continuously iteratively optimize the neural texture model by gradient backpropagation , a neural texture model of the digital human is obtained by training, which is used to model the corresponding appearance features of the digital human's skin surface.

[0059] Step 130: prune the Gaussian model, and perform mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; optimize the Gaussian model of the digital human based on the mixed rendered image and the action sequence video data to obtain an editable digital human model.

[0060] In this step, deep supervision and semantic labels can be used to prune the three-dimensional Gaussians to reduce the number of Gaussians.

[0061] In this embodiment, the output result of the pruned three-dimensional Gaussian splash model is mixed with the output result of the neural texture model of the digital human for rendering (for example, image superposition or fusion processing) to obtain a mixed rendered image, and finally the training loss of the Gaussian model is calculated based on the mixed rendered image to achieve deep optimization of the Gaussian model.

[0062] Specifically, the Gaussian model of the digital human is optimized based on the mixed rendering image, and the editable digital human model is obtained, including: Based on the third comprehensive loss, a loss value is calculated according to the target video frame in the mixed rendered image and the action sequence video data, and the parameters of the Gaussian model are updated according to the loss value. When the Gaussian model converges, an editable digital human model is obtained; wherein, the third comprehensive loss is determined based on the first comprehensive loss and the 2-norm of the Gaussian opacity, and the first comprehensive loss is determined based on the color loss, the perceptual loss, the structural similarity loss, the Sobel operator and the regularization term.

[0063] In this embodiment, in the process of optimizing the Gaussian model of the digital human obtained in step 110, the depth map As a depth control condition sent to the Gaussian renderer, if the The depth of a Gaussian , then the corresponding Gaussian opacity Set to 0; then prune the Gaussian points inside the triangle mesh according to the opacity threshold, so that the number of Gaussians can be reduced without losing the effect, and the storage space of the Gaussian model can be optimized; among them, are the pixel coordinates of the three-dimensional Gaussian splash in the image space.

[0064] Specifically, the Gaussian model of the digital human obtained in step 110 is optimized through the following steps: (1) Pruning the Gaussian model: (1.1) Determine the opacity threshold of the Gaussian renderer based on the depth image, and determine the semantic label of the target to be pruned based on the frame mask of the target video frame; wherein the depth map is obtained by rasterizing the target video frame according to the triangle mesh information, vertex coordinates, indexes and optimized neural texture corresponding to the target video frame in the SMPL-X model through an initialized differentiable rasterizer; the frame mask is obtained through the action sequence video data.

[0065] In this embodiment, the objects to be pruned include parts such as hands and arms.

[0066] In this embodiment, the frame mask for the target video frame , convert the masks of different parts into corresponding semantic labels, and superimpose them to get the final mask map , The value is the semantic label corresponding to each part ; For example, background, head, bare arms, and clothes correspond to 0, 1, 2, and 3 respectively.

[0067] (1.2) Based on the Gaussian renderer, the Gaussian points inside the triangular mesh corresponding to the Gaussian model and the hands and arms are pruned according to the opacity threshold and the semantic label of the target to be pruned to obtain the pruned Gaussian model.

[0068] In this embodiment, for the above mask image , initialize the all-zero 2D matrix , Indicates the number of semantic parts divided, and N indicates the number of Gaussians. In the process of Gaussian rendering, [ ]= ; Accumulate all the time ,get , , and perform majority voting to calculate the semantic label of the three-dimensional Gaussian , = [ ]; then The opacity of the three-dimensional Gaussian of 2 is set to 0, and the three-dimensional Gaussian fitting the arm part is pruned according to the opacity threshold, further reducing the number of Gaussians without losing the effect.

[0069] The three-dimensional Gaussian semantic label acquisition process in the above embodiment does not require training, so that the target area to be edited can be accurately selected through semantic labels, which provides the possibility for multi-scenario applications.

[0070] In this embodiment, the pruned three-dimensional Gaussian is sent to the Gaussian renderer to obtain a rendered image. , and the corresponding Gaussian opacity map , and The rasterized rendered image obtained in step 120 according to Overlay to get the final rendering, expressed as = By combining the advantages of the 3D Gaussian splatter renderer and the differentiable rasterizer renderer in flexibly processing high-frequency details and skin texture, the modeling effect of the 3D Gaussian splatter model for clothing, hair, etc. is improved, and the modeling effect of the neural texture model for skin and hands with complex topological structures is improved, and the appearance of digital humans is effectively decoupled based on semantic masks.

[0071] In this embodiment, the third comprehensive loss is calculated by combining the above-mentioned final rendered image with the target video frame: ; in, For the first stage, the loss function of the three-dimensional Gaussian is optimized separately. is the 2-norm of the Gaussian opacity, which regularizes the opacity of the Gaussian to prune as many Gaussians as possible and reduce space storage.

[0072] In this embodiment, the third comprehensive loss The opacity of the Gaussian is iteratively optimized by gradient backpropagation, and finally a trained editable digital human model is obtained.

[0073] The editable digital human modeling method based on neural texture and three-dimensional Gaussian provided by the present invention models the expression, posture and appearance of the digital human corresponding to the target object through a three-dimensional Gaussian splash model, and then models the expression, posture and appearance of the digital human through a neural texture model, then prunes the modeled Gaussian model and performs mixed rendering according to the rendered image output by the model and the rendered image output by the neural texture model to obtain a mixed rendering image, and finally optimizes the Gaussian model of the digital human using the mixed rendering image and action sequence video data to obtain an editable digital human model, thereby reducing the storage overhead and structural complexity of the digital human model, and improving the high-frequency details and editing efficiency of the skin surface of the digital human model, thereby improving the generalization of three-dimensional Gaussian modeling in different scenarios.

[0074] Figure 2 This is one of the structural schematic diagrams of the editable digital human modeling device based on neural texture and three-dimensional Gaussian provided by the present invention. Figure 2 In the embodiment shown, an editable digital human modeling device based on neural texture and three-dimensional Gaussian includes a segmentation module, an action posture parameter estimation module, a three-dimensional Gaussian splash model, a neural texture model, a Gaussian pruning module and a hybrid rendering module; wherein the segmentation module is used to obtain the semantic mask of the image, and the action posture parameter estimation module is used to obtain the human body action posture expression parameters. The training stage of the model is divided into the training of the three-dimensional Gaussian splash model and the training of the neural texture model. The two models are optimized separately, and then the three-dimensional Gaussian is pruned through the deep supervision and semantic mask in the Gaussian pruning module, so that the skin surface of the arm and the like is well modeled by the neural texture model while reducing the storage overhead of the model; finally, the rendering results of the three-dimensional Gaussian splash model and the neural texture model are superimposed by opacity in the hybrid rendering module.

[0075] In some embodiments, after obtaining the editable digital human model, the editable digital human modeling method based on neural texture and three-dimensional Gaussian further includes: editing the target three-dimensional Gaussian representation based on the editable digital human model according to user input instructions to render the three-dimensional virtual scene of the digital human.

[0076] In this embodiment, the user input instruction can be represented by a three-dimensional Gaussian corresponding to the editing target area, and then a three-dimensional virtual scene corresponding to the target area is rendered.

[0077] Taking the scenario where the user inputs a message for buying clothes online as an example, the corresponding 3D virtual scene is rendered through the following steps: (1) Based on the 3D attribute tags of the editable digital human model =3 Filter out the target Gaussian of the clothing area to be edited, and render to get the corresponding mask of the area to be edited and the rendered image to be edited .

[0078] (2) The user enters the prompt word for the specified editing content , mask the area to be edited and the rendered image to be edited , a given reference clothing image At the same time, it is input into the two-dimensional image completion model IDM-VTON to obtain the corresponding two-dimensional completed image and depth map .

[0079] (3) Based on the open source image-generated 3D method Wonder3d, complete the image from two dimensions Get a rough 3D grid.

[0080] (4) Initialize the 3D Gaussian grid based on the above rough 3D grid , and through the above depth map The edited three-dimensional Gaussian Align with the original 3D Gaussian to get a new 3D Gaussian .

[0081] (6) Iteratively optimize the above three-dimensional Gaussian , complete the editing of three-dimensional Gaussian, and render the video of digital human virtual trying on clothes under a given perspective, which can provide services for users to buy clothes online.

[0082] The editable digital human modeling method based on neural texture and three-dimensional Gaussian provided by the present invention edits the target three-dimensional Gaussian representation by editing the digital human model according to the user input instruction to render the three-dimensional virtual scene of the digital human, thereby realizing dynamic rendering of the current scene.

[0083] Figure 3 is a schematic diagram of the structure of the editing device for the editable digital human model provided by the present invention. Figure 3In the illustrated embodiment, an editing device for an editable digital human model includes: a two-dimensional image completion module, an image-derived three-dimensional grid module, a three-dimensional Gaussian initialization module, a depth alignment module, and an iterative optimization module; wherein, for a mixed rendering image of a three-dimensional Gaussian splash model and a neural texture model, a reference clothing image is given, a two-dimensional virtual fitting effect is obtained through the two-dimensional image completion module, a three-dimensional representation of the corresponding clothing is obtained through the image-derived three-dimensional grid module, and the three-dimensional Gaussian initialization module is used to convert it into a three-dimensional Gaussian representation, and then the alignment and splicing with the original three-dimensional Gaussian is completed through the depth alignment module, and finally the edited three-dimensional Gaussian is iteratively optimized through the iterative optimization module, and mixed rendering with the neural texture model is used to achieve a three-dimensional virtual fitting effect.

[0084] The following describes the editable digital human modeling device based on neural texture and three-dimensional Gaussian provided by the present invention. The editable digital human modeling device based on neural texture and three-dimensional Gaussian described below and the editable digital human modeling method based on neural texture and three-dimensional Gaussian described above can be referenced to each other.

[0085] Figure 4 This is the second structural schematic diagram of the editable digital human modeling device based on neural texture and three-dimensional Gaussian provided by the present invention, such as Figure 4 As shown, the editable digital human modeling device based on neural texture and three-dimensional Gaussian includes: a first training module 410, a second training module 420 and a third training module 430.

[0086] The first training module 410 is used to model the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object through a three-dimensional Gaussian splash model to obtain a Gaussian model of the digital human; the action sequence video data includes expression parameters, posture parameters and appearance parameters of the target object; The second training module 420 is used to model the expression, posture and appearance of the digital human according to the action sequence video data through the neural texture model to obtain the neural texture model of the digital human; The third training module 430 is used to prune the Gaussian model, and perform mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; based on the mixed rendered image and the action sequence video data, the Gaussian model of the digital human is optimized to obtain an editable digital human model.

[0087] The editable digital human modeling device based on neural texture and three-dimensional Gaussian provided by the present invention models the expression, posture and appearance of the digital human corresponding to the target object through a three-dimensional Gaussian splash model, and then models the expression, posture and appearance of the digital human through a neural texture model, then prunes the modeled Gaussian model and performs mixed rendering according to the rendered image output by the model and the rendered image output by the neural texture model to obtain a mixed rendered image, and finally optimizes the Gaussian model of the digital human using the mixed rendered image and action sequence video data to obtain an editable digital human model, thereby reducing the storage overhead and structural complexity of the digital human model, and improving the high-frequency details and editing efficiency of the skin surface of the digital human model, thereby improving the generalization of three-dimensional Gaussian modeling in different scenarios.

[0088] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute an editable digital human modeling method based on neural texture and three-dimensional Gaussian, which includes: using a three-dimensional Gaussian splash model to model the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object, to obtain a Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters and appearance parameters of the target object; using the neural texture model to model the expression, posture and appearance of the digital human according to the action sequence video data, to obtain a neural texture model of the digital human; pruning the Gaussian model, and mixing and rendering the rendered image output by the pruned Gaussian model with the rendered image output by the neural texture model of the digital human to obtain a mixed rendering image; optimizing the Gaussian model of the digital human based on the mixed rendering image and the action sequence video data to obtain an editable digital human model.

[0089] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the editable digital human modeling method based on neural texture and three-dimensional Gaussian provided by the above-mentioned methods, the method including: modeling the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object through a three-dimensional Gaussian splash model to obtain a Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters and appearance parameters of the target object; modeling the expression, posture and appearance of the digital human according to the action sequence video data through a neural texture model to obtain a neural texture model of the digital human; pruning the Gaussian model, and mixing the rendered image output by the pruned Gaussian model with the rendered image output by the neural texture model of the digital human to obtain a mixed rendering image; optimizing the Gaussian model of the digital human based on the mixed rendering image and the action sequence video data to obtain an editable digital human model.

[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the editable digital human modeling method based on neural texture and three-dimensional Gaussian provided by the above-mentioned methods, the method comprising: modeling the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object through a three-dimensional Gaussian splash model to obtain a Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters and appearance parameters of the target object; modeling the expression, posture and appearance of the digital human according to the action sequence video data through a neural texture model to obtain a neural texture model of the digital human; pruning the Gaussian model, and performing mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; optimizing the Gaussian model of the digital human based on the mixed rendered image and the action sequence video data to obtain an editable digital human model.

[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An editable digital human modeling method based on neural texture and three-dimensional Gaussian, characterized in that: include: Modeling the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object by using a three-dimensional Gaussian splash model to obtain the Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters and appearance parameters of the target object; Modeling the expression, posture and appearance of the digital human according to the action sequence video data using a neural texture model to obtain a neural texture model of the digital human; The Gaussian model is pruned, and a mixed rendering is performed based on a rendered image output by the pruned Gaussian model and a rendered image output by the neural texture model of the digital human to obtain a mixed rendering image; the Gaussian model of the digital human is optimized based on the mixed rendering image and the action sequence video data to obtain an editable digital human model.

2. The editable digital human modeling method based on neural texture and three-dimensional Gaussian according to claim 1 is characterized in that: The action sequence video data is collected by a monocular camera; The method of modeling the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object by using the three-dimensional Gaussian splash model to obtain the Gaussian model of the digital human includes: Acquire SMPL-X model parameters, frame mask and camera parameters of the monocular camera of the target video frame from the action sequence video data; the SMPL-X model parameters include the expression parameters, the posture parameters and the appearance parameters; Initialize local three-dimensional Gaussian parameters of the SMPL-X model according to the SMPL-X model parameters; the local three-dimensional Gaussian parameters include position parameters, rotation parameters, scale parameters, color parameters, opacity parameters, index parameters and semantic labels; Calculate the global three-dimensional Gaussian parameters of the SMPL-X model based on the local three-dimensional Gaussian parameters and the SMPL-X model parameters; Rendering the target video frame according to the global three-dimensional Gaussian parameters and the camera parameters based on a Gaussian renderer to obtain a Gaussian rendered image, and calculating a color loss according to the target video frame and the target video frame; A first comprehensive loss is obtained based on the color loss, the perceptual loss, the structural similarity loss, the Sobel operator and the regularization term. Based on the first comprehensive loss, the three-dimensional Gaussian splash model is iteratively trained by gradient backpropagation to obtain the Gaussian model.

3. The editable digital human modeling method based on neural texture and three-dimensional Gaussian according to claim 1 is characterized in that: The step of modeling the expression, posture and appearance of the digital human according to the action sequence video data by using the neural texture model to obtain the neural texture model of the digital human comprises: Initialize the differentiable rasterizer based on the triangle mesh information, vertex coordinates and index of the SMPL-X model; Based on the initialized differentiable rasterizer, the target video frame is rasterized according to the triangle mesh information corresponding to the target video frame in the SMPL-X model, the vertex coordinates, the index and the optimized neural texture to obtain a neural texture rendered image, and the color loss is calculated according to the neural texture rendered image and the target video frame; A second comprehensive loss is obtained based on the color loss, perceptual loss, structural similarity loss and total variation algorithm, and the neural texture model is iteratively trained by gradient backpropagation based on the second comprehensive loss to obtain the neural texture model.

4. The editable digital human modeling method based on neural texture and three-dimensional Gaussian according to claim 1, characterized in that: The pruning of the Gaussian model comprises: Determine the opacity threshold of the Gaussian renderer according to the depth image, and determine the semantic label of the target to be pruned based on the frame mask of the target video frame; wherein the depth map is obtained by rasterizing the target video frame according to the triangle mesh information, vertex coordinates, indexes and optimized neural texture corresponding to the target video frame in the SMPL-X model through the initialized differentiable rasterizer; the frame mask is obtained through the action sequence video data; Based on the Gaussian renderer, the Gaussian points inside the triangular mesh corresponding to the Gaussian model and the hands and arms are pruned according to the opacity threshold and the semantic label of the target to be pruned, so as to obtain the pruned Gaussian model.

5. The editable digital human modeling method based on neural texture and three-dimensional Gaussian according to claim 1 is characterized in that: The optimizing the Gaussian model of the digital human based on the mixed rendering image to obtain an editable digital human model comprises: Calculating a loss value according to the mixed rendered image and the target video frame in the action sequence video data based on a third comprehensive loss, and updating the parameters of the Gaussian model according to the loss value, and obtaining the editable digital human model when the Gaussian model converges; The third comprehensive loss is determined based on the first comprehensive loss and the 2-norm of Gaussian opacity, and the first comprehensive loss is determined based on color loss, perceptual loss, structural similarity loss, Sobel operator and regularization term.

6. The editable digital human modeling method based on neural texture and three-dimensional Gaussian according to claim 1, characterized in that: After obtaining the editable digital human model, the method further includes: Based on the editable digital human model, the target three-dimensional Gaussian representation is edited according to the user input instruction to render the three-dimensional virtual scene of the digital human.

7. An editable digital human modeling device based on neural texture and three-dimensional Gaussian, characterized in that: include: The first training module is used to model the expression, posture and appearance of the digital human corresponding to the target object according to the action sequence video data of the target object through a three-dimensional Gaussian splash model to obtain the Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters and appearance parameters of the target object; A second training module is used to model the expression, posture and appearance of the digital human according to the action sequence video data through a neural texture model to obtain a neural texture model of the digital human; The third training module is used to prune the Gaussian model, and perform mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; based on the mixed rendered image and the action sequence video data, the Gaussian model of the digital human is optimized to obtain an editable digital human model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the editable digital human modeling method based on neural texture and three-dimensional Gaussian is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the editable digital human modeling method based on neural texture and three-dimensional Gaussian is implemented as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the editable digital human modeling method based on neural texture and three-dimensional Gaussian is implemented as described in any one of claims 1 to 6.

Images