Neural texture and three-dimensional gaussian-based editable digital human modeling method and device

Abstract

This invention provides a method and apparatus for editing digital humans based on neural texture and 3D Gaussian models. The method includes: modeling the expression, posture, and appearance of the digital human using a 3D Gaussian splash model based on video data of a target object's motion sequence, obtaining a Gaussian model of the digital human; modeling the expression, posture, and appearance of the digital human using a neural texture model, obtaining a neural texture model of the digital human; pruning the Gaussian model, and performing a blended rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human, obtaining a blended rendered image; and optimizing the Gaussian model of the digital human based on the blended rendered image and the motion sequence video data, obtaining an editable digital human model. The method of this invention reduces the storage overhead and structural complexity of the digital human model, and improves the high-frequency details and editing efficiency of the digital human model's skin surface.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an editable digital human modeling method and apparatus based on neural texture and three-dimensional Gaussian. Background Technology

[0002] 3D Gaussian is based on each point in the sparse point cloud obtained by Structure from Motion (SFM). It introduces a 3D Gaussian distribution as the scene representation. Each 3D Gaussian distribution consists of parameters such as position, rotation components, and view-dependent color that can be learned. Combined with Gaussian splashing technology, it achieves fast differentiable rendering. Therefore, 3D Gaussian has become the optimal choice for 3D representation in terms of training time, image quality, and real-time performance.

[0003] In related technologies, a hybrid representation model of mesh and Gaussian is used to bind Gaussian to the mesh, thereby achieving effective driving of 3D Gaussian and constructing a digital human model. However, the reconstructed digital human has problems such as large storage space overhead, loss of high-frequency details on the skin surface, and difficulty in optimizing complex structures such as hands. It is also difficult to edit effectively, so the reconstructed digital human is often limited to specific scenarios and has weak generalization. Summary of the Invention

[0004] This invention provides an editable digital human modeling method and apparatus based on neural texture and 3D Gaussian to solve the defects of existing digital human models, such as large storage space overhead, loss of high-frequency details on the skin surface, difficulty in optimizing complex structures such as hands and other parts, and difficulty in effective editing, resulting in poor generalization of the reconstructed digital human. This invention improves the generalization of 3D Gaussian modeling in different scenarios.

[0005] This invention provides a method for modeling editable digital humans based on neural textures and 3D Gaussian, comprising:

[0006] The expression, posture, and appearance of the digital human corresponding to the target object are modeled using a three-dimensional Gaussian splash model based on the action sequence video data of the target object, thus obtaining the Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters, and appearance parameters of the target object;

[0007] The digital human's facial expressions, postures, and appearance are modeled using a neural texture model based on the action sequence video data, thus obtaining the digital human's neural texture model.

[0008] The Gaussian model is pruned, and the rendered image output by the pruned Gaussian model is mixed with the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; the Gaussian model of the digital human is optimized based on the mixed rendered image and the action sequence video data to obtain an editable digital human model.

[0009] According to the present invention, an editable digital human modeling method based on neural texture and three-dimensional Gaussian is provided, wherein the motion sequence video data is acquired through a monocular camera;

[0010] The process of modeling the expression, posture, and appearance of the digital human corresponding to the target object based on the action sequence video data of the target object using a three-dimensional Gaussian splash model to obtain the Gaussian model of the digital human includes:

[0011] The SMPL-X model parameters, frame mask, and camera parameters of the monocular camera for the target video frame are obtained from the action sequence video data; the SMPL-X model parameters include the expression parameters, the pose parameters, and the appearance parameters.

[0012] The local 3D Gaussian parameters of the SMPL-X model are initialized based on the SMPL-X model parameters; the local 3D Gaussian parameters include position parameters, rotation parameters, scale parameters, color parameters, opacity parameters, index parameters, and semantic labels;

[0013] The global three-dimensional Gaussian parameters of the SMPL-X model are calculated based on the local three-dimensional Gaussian parameters and the parameters of the SMPL-X model.

[0014] The Gaussian renderer renders the target video frame based on the global 3D Gaussian parameters and the camera parameters to obtain a Gaussian rendered image, and calculates the color loss based on the target video frame and the target video frame.

[0015] The first comprehensive loss is obtained based on the color loss, perceptual loss, structural similarity loss, Sobel operator and regularization term. The three-dimensional Gaussian splash model is iteratively trained based on the first comprehensive loss through gradient backpropagation to obtain the Gaussian model.

[0016] According to the present invention, an editable digital human modeling method based on neural texture and 3D Gaussian is provided. The method involves modeling the digital human's facial expressions, postures, and appearance using a neural texture model based on the action sequence video data to obtain the digital human's neural texture model, which includes:

[0017] Initialize a differentiable rasterizer renderer based on triangle mesh information, vertex coordinates, and indexes from the SMPL-X model;

[0018] Based on the initialized differentiable rasterizer renderer, the target video frame is rasterized according to the triangle mesh information corresponding to the target video frame in the SMPL-X model, the vertex coordinates, the index, and the optimized neural texture to obtain a neural texture rendering image, and the color loss is calculated based on the neural texture rendering image and the target video frame.

[0019] The second comprehensive loss is obtained based on the color loss, perceptual loss, structural similarity loss, and total variation algorithm. The neural texture model is then iteratively trained using gradient backpropagation based on the second comprehensive loss to obtain the neural texture model.

[0020] According to the present invention, an editable digital human modeling method based on neural texture and 3D Gaussian model is provided, wherein pruning the Gaussian model includes:

[0021] The opacity threshold of the Gaussian renderer is determined based on the depth image, and the semantic labels of the target video frame to be pruned are determined based on the frame mask of the target video frame. The depth image is obtained by rasterizing the target video frame using an initialized differentiable rasterizer based on the triangular mesh information, vertex coordinates, index, and optimized neural texture corresponding to the target video frame in the SMPL-X model. The frame mask is obtained from the action sequence video data.

[0022] The Gaussian renderer prunes the Gaussian points inside the triangular mesh corresponding to the Gaussian model, as well as the Gaussian points of the hand and arm, based on the opacity threshold and the semantic label of the target to be pruned, to obtain the pruned Gaussian model.

[0023] According to the present invention, an editable digital human modeling method based on neural texture and 3D Gaussian is provided, wherein optimizing the Gaussian model of the digital human based on the hybrid rendering image to obtain an editable digital human model includes:

[0024] The loss value is calculated based on the third comprehensive loss according to the target video frame in the hybrid rendered image and the action sequence video data, and the parameters of the Gaussian model are updated according to the loss value. When the Gaussian model converges, the editable digital human model is obtained.

[0025] The third comprehensive loss is determined based on the first comprehensive loss and the 2-norm of Gaussian opacity. The first comprehensive loss is determined based on color loss, perceptual loss, structural similarity loss, Sobel operator, and regularization term.

[0026] According to the present invention, an editable digital human modeling method based on neural texture and 3D Gaussian is provided. After obtaining the editable digital human model, the method further includes:

[0027] Based on the editable digital human model, the target 3D Gaussian representation is edited according to user input instructions to render a 3D virtual scene of the digital human.

[0028] The present invention also provides an editable digital human modeling device based on neural texture and three-dimensional Gaussian, comprising:

[0029] The first training module is used to model the expression, posture, and appearance of the digital human corresponding to the target object based on the action sequence video data of the target object using a three-dimensional Gaussian splash model, thereby obtaining the Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters, and appearance parameters of the target object;

[0030] The second training module is used to model the digital human's facial expressions, postures, and appearance based on the action sequence video data using a neural texture model, thereby obtaining the digital human's neural texture model.

[0031] The third training module is used to prune the Gaussian model and perform a mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; and optimize the Gaussian model of the digital human based on the mixed rendered image and the action sequence video data to obtain an editable digital human model.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the editable digital human modeling method based on neural texture and three-dimensional Gaussian as described above.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the editable digital human modeling method based on neural texture and three-dimensional Gaussian as described above.

[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the editable digital human modeling method based on neural texture and three-dimensional Gaussian as described above.

[0035] The present invention provides an editable digital human modeling method and apparatus based on neural texture and 3D Gaussian modeling. The method and apparatus model the expression, posture, and appearance of the digital human corresponding to the target object using a 3D Gaussian splash model, followed by modeling the expression, posture, and appearance of the digital human using a neural texture model. Next, the Gaussian model is pruned, and the rendered image output by the model is mixed with the rendered image output by the neural texture model to obtain a mixed rendered image. Finally, the Gaussian model of the digital human is optimized using the mixed rendered image and motion sequence video data to obtain an editable digital human model. This reduces the storage overhead and structural complexity of the digital human model, improves the high-frequency details and editing efficiency of the digital human model's skin surface, and thus enhances the generalization of 3D Gaussian modeling in different scenarios. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the editable digital human modeling method based on neural texture and 3D Gaussian provided by the present invention.

[0038] Figure 2 This is one of the structural schematic diagrams of the editable digital human modeling device based on neural texture and three-dimensional Gaussian provided by the present invention.

[0039] Figure 3 This is a schematic diagram of the editing device for the editable digital human model provided by the present invention.

[0040] Figure 4 This is the second schematic diagram of the editable digital human modeling device based on neural texture and three-dimensional Gaussian provided by the present invention.

[0041] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0043] The following is combined Figures 1-4 This invention describes the method and apparatus for creating editable digital humans based on neural textures and 3D Gaussian.

[0044] Figure 1 This is a flowchart illustrating the editable digital human modeling method based on neural texture and 3D Gaussian provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0045] Step 110: Using a 3D Gaussian splash model, model the expression, posture, and appearance of the digital human corresponding to the target object based on the action sequence video data of the target object, and obtain the Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters, and appearance parameters of the target object.

[0046] In this step, the target object includes moving objects, such as people, animals, aircraft, or robotic devices.

[0047] Taking a person as the target, video data of a specific person is collected through video capture devices (such as cameras, mobile phones, etc.). During the collection process, the camera shooting angle and shooting parameter settings remain unchanged. For example, the video resolution is set to 1920 × 1080 pixels and the frame rate is 30 frames / second.

[0048] In this embodiment, the subject can perform large-amplitude body movements such as kicking, dancing, and bending over. In addition, the movements can also include complex hand movements such as counting, playing musical instruments, and using tools.

[0049] In this embodiment, the data collection time is 1 minute or more (N ≥ 1800 frames). The longer the time, the more perspectives of the subject are captured, the richer the virtual human's movements and expressions are, and the better the digital human effect can be reconstructed.

[0050] Furthermore, the motion sequence video data is acquired using a monocular camera; based on the motion sequence video data of the target object, a 3D Gaussian splash model is used to model the expression, posture, and appearance of the corresponding digital human, resulting in a Gaussian model of the digital human including:

[0051] (1) Obtain the SMPL-X model parameters, frame mask and camera parameters of the target video frame from the action sequence video data; the SMPL-X model parameters include expression parameters, pose parameters and appearance parameters.

[0052] In this embodiment, a monocular digital human training dataset is obtained by collecting and processing video data of action sequences of the subject in a natural scene.

[0053] Specifically, the images corresponding to video frames t are extracted frame by frame from the acquired motion sequence video data. And use the following formula based on the set field of view (e.g., field of view angle) ), length of the image w and width h Calculate the intrinsic parameters of a monocular camera:

[0054] ;

[0055] Among them, the extrinsic parameters of a monocular camera Set it to a 4×4 identity matrix.

[0056] Specifically, for video frame image data in action sequence video data, this embodiment uses an open-source pose estimation method to identify the SMPL-X model parameters of the acquired object in each video frame. ,in, , , These represent the pose parameters, shape parameters, and facial expression parameters of the corresponding video frame of the captured object, respectively.

[0057] In this embodiment, attitude parameters It can be further divided into overall posture Head posture Joint hand posture Other postures The mask of the captured object in each video frame is obtained by combining the open-source object segmentation model SAM2. , The mask corresponding to the background, These are masks corresponding to the head, arms, and clothing, respectively.

[0058] In this embodiment, by mapping the video frame image data, camera parameters, SMPL-X model parameters, and image mask one-to-one according to the video frame sequence, the acquired data of the t-th frame of the video can be obtained, represented as: ;in, Represents a two-dimensional image of a video frame. This represents the SMPL-X human body posture and facial expression parameters at time t. This represents the mask corresponding to a video frame image. These represent the extrinsic and intrinsic parameters of a globally consistent monocular camera, respectively.

[0059] (2) Initialize the local 3D Gaussian parameters of the SMPL-X model according to the SMPL-X model parameters; the local 3D Gaussian parameters include position parameters, rotation parameters, scale parameters, color parameters, opacity parameters, index parameters and semantic labels.

[0060] In this embodiment, the SMPL-X parameters of frame 0 obtained in (1) above are... Perform linear blending skinning to obtain the vertex and mesh representations of the corresponding SMPL-X model. , and These represent the set of all grid vertex coordinates and the set of grid indices at the current moment, respectively.

[0061] In this embodiment, the SMPL-X model includes multiple triangular meshes. The corresponding triangular meshes are initialized by defining a local coordinate system (the origin of which is the center of the vertex of each triangular mesh). The local 3D Gaussian parameters of a triangular mesh are determined by the following steps:

[0062] The local three-dimensional Gaussian parameters are expressed as follows: ;in, The position parameters of the 3D Gaussian are initialized to the origin of the local coordinate system. ; The rotation parameters of the three-dimensional Gaussian are represented in quaternion form and initialized to . ; The scaling parameter for the 3D Gaussian is initialized to 0.0000001; The color parameters for a 3D Gaussian color system are initialized to random values. The opacity parameter for a 3D Gaussian shape is initialized to 1. The index parameter of the 3D Gaussian is used to represent the index of the attached triangular mesh, initialized to... ; The semantic label for a 3D Gaussian is initialized to 0.

[0063] (3) Calculate the global three-dimensional Gaussian parameters of the SMPL-X model based on the local three-dimensional Gaussian parameters and the SMPL-X model parameters.

[0064] In this embodiment, based on the local three-dimensional Gaussian parameters obtained in (2) above, linear blending skinning is performed on the SMPL-X parameters at time t to obtain the representation of the SMPL-X vertices and mesh at the current time. ; and then regarding the current moment A triangular mesh, calculate the vertex center. Quaternion rotation and scale Transform local 3D Gaussian parameters into global 3D Gaussian parameters. .

[0065] in, = , = , = .

[0066] (4) The target video frame is rendered based on the global three-dimensional Gaussian parameters and camera parameters by the Gaussian renderer to obtain the Gaussian rendered image, and the color loss is calculated based on the target video frame.

[0067] In this embodiment, a Gaussian renderer is used based on global 3D Gaussian parameters and camera parameters. Render the current frame image to obtain the rendered image of the current frame. Combined with target video frames Calculate the color loss in the masked area.

[0068] For example, first obtain a binary mask with the same size as the rendered image and the original image. Multiply the mask element-wise with the image to extract the region of interest (ROI) in both the rendered and original images. For each extracted ROI, calculate the color difference between the rendered and original images for each pixel, and then convert the color difference into a loss value. Specifically, the color loss function is calculated using the following formula:

[0069] = * ;

[0070] in, This represents the color loss value. For the corresponding background mask ( A mask used to separate people as foreground and background, where the foreground area represents the person (usually white) and the background (usually black).

[0071] (5) The first comprehensive loss is obtained by color loss, perception loss, structural similarity loss, Sobel operator and regularization term. Based on the first comprehensive loss, the three-dimensional Gaussian splash model is iteratively trained by gradient backpropagation to obtain the Gaussian model.

[0072] In this embodiment, after obtaining the aforementioned color loss, it is then combined with the perceptual loss. Structural similarity loss (Used to monitor image quality) (Corresponding to the Sobel operator, used to improve performance for high-frequency details) and regularization terms (Used to address the issue of unsmooth Gaussian rendering results, by constraining the standard deviation of adjacent Gaussian properties) to calculate the loss of the first derivative between the rendered image and the reference image, i.e., the first comprehensive loss; the final first comprehensive loss can be expressed as:

[0073] ;

[0074] Among them, hyperparameters , , , .

[0075] In this embodiment, the first comprehensive loss described above is utilized. By optimizing the attribute parameters of the 3D Gaussian model through gradient backpropagation, and using the density operation in the 3D Gaussian method to split and clone the 3D Gaussian model, the 3D Gaussian model can be made to adapt to the geometric and appearance distribution, thus obtaining the trained Gaussian model.

[0076] In some embodiments, after optimizing the Gaussian model through the steps corresponding to step (5) above, the index parameters of the three-dimensional Gaussian model are updated by introducing a Gaussian walking strategy on the surface of the triangular mesh.

[0077] Specifically, if the Gaussian's position during the optimization process exceeds the range of its attached triangular mesh, the index parameters of the 3D Gaussian are updated. For the new triangle grid index .

[0078] Step 120: Model the digital human's facial expressions, postures, and appearance based on the action sequence video data using a neural texture model to obtain the digital human's neural texture model.

[0079] In this step, the neural texture model uses neural networks to learn and store the appearance information of the object's surface for high-quality texture rendering and editing.

[0080] In this embodiment, the digital human's facial expressions, postures, and appearance are modeled based on action sequence video data using a neural texture model, resulting in the following neural texture model for the digital human:

[0081] (1) Initialize the differentiable rasterizer renderer based on the triangle mesh information, vertex coordinates and index of the SMPL-X model.

[0082] In this embodiment, after freezing and disabling the three-dimensional Gaussian model obtained in step 110 above, the 0th frame triangular mesh is generated based on the 0th frame SMPL-X parameters obtained in step 110. Configure the parameters of the differentiable rasterizer Nvdiffrast. and read the initialized neural texture ;in, Used to store the triangular mesh information of the SMPL-X model. Storing the vertices of the SMPL-X model coordinate, Storing SMPL-X models Index, used for subsequent texture mapping operations; and The width and height of the corresponding image.

[0083] (2) Based on the initialized differentiable rasterizer, the target video frame is rasterized according to the triangular mesh information, vertex coordinates, index and optimized neural texture of the target video frame in the SMPL-X model to obtain the neural texture rendering image, and the color loss is calculated based on the neural texture rendering image and the target video frame.

[0084] In this embodiment, SMPL-X triangle mesh at time 1 With the above and The images are fed together into the differentiable rasterizer Nvdiffrast, and after rasterization, the rendered image is obtained. With depth map .

[0085] In this embodiment, based on the rendered image and target video frames Calculate the color loss in the masked region, as described above based on the rendered image. and target video frames The calculation of color loss in the mask area is the same, and will not be repeated in this embodiment.

[0086] Specifically, the color loss can be calculated using the following formula:

[0087] = * .

[0088] (3) The second comprehensive loss is obtained based on color loss, perceptual loss, structural similarity loss and total variation algorithm. The neural texture model is iteratively trained based on the second comprehensive loss through gradient backpropagation to obtain the neural texture model.

[0089] In this embodiment, after obtaining the aforementioned color loss, it is then combined with the perceptual loss. Structural similarity loss and (Corresponding to the total variation algorithm, used to preserve the smoothness of the texture map and reduce noise) to calculate the loss of the first derivative between the rendered image and the reference image, i.e., the second comprehensive loss; the final loss function can be expressed as:

[0090] = ;

[0091] Among them, hyperparameters Consistent with the first stage of Gaussian optimization, =0.2.

[0092] In this embodiment, based on the aforementioned second comprehensive loss The neural texture model is continuously optimized through gradient backpropagation. The neural texture model of the digital human is trained and used to model the appearance features corresponding to the skin surface of the digital human.

[0093] Step 130: Prune the Gaussian model, and perform a mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; optimize the Gaussian model of the digital human based on the mixed rendered image and action sequence video data to obtain an editable digital human model.

[0094] In this step, deep supervision and semantic labeling can be used to prune the 3D Gaussian to reduce the number of Gaussians.

[0095] In this embodiment, the output of the pruned 3D Gaussian splash model is mixed with the output of the digital human's neural texture model for rendering (e.g., image overlay or fusion processing) to obtain a mixed rendering image. Finally, the training loss of the Gaussian model is calculated based on the mixed rendering image to achieve deep optimization of the Gaussian model.

[0096] Specifically, the Gaussian model of the digital human is optimized based on the hybrid rendered image to obtain an editable digital human model, including:

[0097] The loss value is calculated based on the target video frame in the mixed rendered image and action sequence video data, and the parameters of the Gaussian model are updated according to the loss value. If the Gaussian model converges, an editable digital human model is obtained. The third comprehensive loss is determined based on the first comprehensive loss and the 2-norm of Gaussian opacity. The first comprehensive loss is determined based on color loss, perceptual loss, structural similarity loss, Sobel operator and regularization term.

[0098] In this embodiment, during the optimization of the Gaussian model of the digital human obtained in step 110, the aforementioned depth map is... As a depth control condition, it is fed into the Gaussian renderer if the first Depth of Gauss Then the corresponding Gaussian opacity will be... Set it to 0; then prune Gaussian points inside the triangular mesh according to the opacity threshold, so as to reduce the number of Gaussians without losing effect and optimize the storage space of the Gaussian model; among which, The pixel coordinates are the three-dimensional Gaussian splashes applied to the image space.

[0099] Specifically, the Gaussian model of the digital human obtained in step 110 is optimized through the following steps:

[0100] (1) Pruning the Gaussian model:

[0101] (1.1) Determine the opacity threshold of the Gaussian renderer based on the depth image, and determine the target semantic label to be pruned based on the frame mask of the target video frame; wherein, the depth image is obtained by rasterizing the target video frame through the initialized differentiable rasterizer renderer based on the triangle mesh information, vertex coordinates, index and optimized neural texture of the target video frame in the SMPL-X model; the frame mask is obtained through action sequence video data.

[0102] In this embodiment, the target to be pruned includes parts such as hands and arms.

[0103] In this embodiment, the frame mask for the target video frame The mask of different parts is converted into corresponding semantic tags, and then superimposed to obtain the final mask image. , The value is the semantic tag corresponding to each part. For example, the background, head, bare arms, and clothes correspond to 0, 1, 2, and 3 respectively.

[0104] (1.2) Based on the Gaussian renderer, the Gaussian points inside the corresponding triangle mesh of the Gaussian model, as well as the hands and arms, are pruned according to the opacity threshold and the semantic labels of the target to be pruned, to obtain the pruned Gaussian model.

[0105] In this embodiment, regarding the above-mentioned mask image Initialize an all-zero two-dimensional matrix , This represents the number of semantic parts in the division, and N represents the Gaussian number. During Gaussian rendering, [ ]= ; Accumulate all moments ,get , Then, a majority vote is performed to calculate the semantic labels of the three-dimensional Gaussian. , = [ ]; then The 3D Gaussian opacity of 2 is set to 0. Based on the opacity threshold, the 3D Gaussian of the fitted arm part is pruned to further reduce the number of Gaussians without losing the effect.

[0106] The above-described embodiment's training-free 3D Gaussian semantic label acquisition process enables precise selection of the target region to be edited through semantic labels, providing possibilities for multi-scenario applications.

[0107] In this embodiment, the pruned 3D Gaussian image is fed into a Gaussian renderer to obtain the rendered image. And the corresponding Gaussian opacity map and will Compared with the rasterized rendering image obtained in step 120 according to The images are overlaid to obtain the final rendered image, represented as follows: = By combining the advantages of a 3D Gaussian splatter renderer and a differentiable rasterizer renderer in flexibly handling high-frequency details and skin textures, the 3D Gaussian splatter model improves the modeling effect of clothing, hair, etc., and the neural texture model improves the modeling effect of skin and hands with complex topological structures. Furthermore, it effectively decouples the appearance of digital humans based on semantic masks.

[0108] In this embodiment, a third comprehensive loss is calculated by combining the final rendered image with the target video frame:

[0109] ;

[0110] in, The loss function for the first stage is optimized separately for the three-dimensional Gaussian. The 2-norm of the Gaussian opacity is used to regularize the Gaussian opacity, thereby pruning as many Gaussians as possible and reducing storage space.

[0111] In this embodiment, through the aforementioned third comprehensive loss The opacity of the Gaussian is iteratively optimized using gradient backpropagation, ultimately resulting in a trained, editable digital human model.

[0112] The present invention provides an editable digital human modeling method based on neural texture and 3D Gaussian. This method models the expression, posture, and appearance of the digital human corresponding to the target object using a 3D Gaussian splash model, and then models the expression, posture, and appearance of the digital human using a neural texture model. Next, the modeled Gaussian model is pruned, and the rendered image output by the model is blended with the rendered image output by the neural texture model to obtain a blended rendered image. Finally, the blended rendered image and motion sequence video data are used to optimize the Gaussian model of the digital human to obtain an editable digital human model. This method reduces the storage overhead and structural complexity of the digital human model, improves the high-frequency details and editing efficiency of the digital human model's skin surface, and thus improves the generalization of 3D Gaussian modeling in different scenarios.

[0113] Figure 2 This is one of the structural schematic diagrams of the editable digital human modeling device based on neural texture and 3D Gaussian provided by the present invention. Figure 2In the illustrated embodiment, an editable digital human modeling device based on neural texture and 3D Gaussian includes a segmentation module, a motion pose parameter estimation module, a 3D Gaussian splash model, a neural texture model, a Gaussian pruning module, and a hybrid rendering module. The segmentation module is used to obtain the semantic mask of the image, and the motion pose parameter estimation module is used to obtain human motion pose and expression parameters. The model training phase is divided into training the 3D Gaussian splash model and training the neural texture model. These two models are optimized separately. Then, the 3D Gaussian model is pruned using depth supervision and semantic masking in the Gaussian pruning module, ensuring that skin surfaces such as arms are well modeled by the neural texture model while reducing the model's storage overhead. Finally, the rendering results of the 3D Gaussian splash model and the neural texture model are superimposed using opacity in the hybrid rendering module.

[0114] In some embodiments, after obtaining the editable digital human model, the editable digital human modeling method based on neural texture and 3D Gaussian further includes: editing the target 3D Gaussian representation based on the editable digital human model according to user input instructions to render a 3D virtual scene of the digital human.

[0115] In this embodiment, user input commands can be represented by a three-dimensional Gaussian symbol corresponding to the target area, thereby rendering a three-dimensional virtual scene corresponding to the target area.

[0116] Taking a user input scenario for purchasing clothes online as an example, the corresponding 3D virtual scene is rendered through the following steps:

[0117] (1) Based on the three-dimensional attribute labels of the editable digital human model =3 filters out the target Gaussian mask for the clothing area to be edited, and renders the corresponding mask for the area to be edited. And the rendered image to be edited .

[0118] (2) Prompt words for users to input specified editing content Mask the area to be edited. And the rendered image to be edited Given a reference clothing image Simultaneously input into the 2D image completion model IDM-VTON, the corresponding 2D completed image is obtained. and depth map .

[0119] (3) Using the open-source graph-based 3D method Wonder3d to complete images from two dimensions A rough 3D mesh was obtained.

[0120] (4) Initialize the three-dimensional Gaussian grid based on the above rough three-dimensional mesh. And through the above depth map The edited 3D Gaussian Align with the original 3D Gaussian to obtain a new 3D Gaussian. .

[0121] (6) Iteratively optimize the above three-dimensional Gaussian The process involves completing 3D Gaussian editing and rendering a video of a digital human virtually trying on clothes from a given perspective, which can then be used to provide services for users to purchase clothes online.

[0122] The present invention provides an editable digital human modeling method based on neural texture and 3D Gaussian. By editing the digital human model according to the user's input instructions, the target 3D Gaussian representation is edited to render the 3D virtual scene of the digital human, thus realizing dynamic rendering of the current scene.

[0123] Figure 3 This is a schematic diagram of the editing device for the editable digital human model provided by the present invention. Figure 3 In the illustrated embodiment, an editing device for an editable digital human model includes: a two-dimensional image completion module, a graph-generated 3D mesh module, a 3D Gaussian initialization module, a depth alignment module, and an iterative optimization module. Specifically, for a hybrid rendering image of a 3D Gaussian splatter model and a neural texture model, given a reference clothing image, the two-dimensional image completion module obtains a two-dimensional virtual try-on effect, the graph-generated 3D mesh module obtains a 3D representation of the corresponding clothing, the 3D Gaussian initialization module converts it into a 3D Gaussian representation, the depth alignment module aligns and stitches it with the original 3D Gaussian representation, and finally the iterative optimization module iteratively optimizes the edited 3D Gaussian representation, which is then hybridized with the neural texture model to achieve a 3D virtual try-on effect.

[0124] The editable digital human modeling device based on neural texture and three-dimensional Gaussian provided by the present invention will be described below. The editable digital human modeling device based on neural texture and three-dimensional Gaussian described below can be referred to in correspondence with the editable digital human modeling method based on neural texture and three-dimensional Gaussian described above.

[0125] Figure 4 This is the second schematic diagram of the structure of the editable digital human modeling device based on neural texture and 3D Gaussian provided by the present invention, as shown below. Figure 4 As shown, the editable digital human modeling device based on neural texture and 3D Gaussian includes: a first training module 410, a second training module 420 and a third training module 430.

[0126] The first training module 410 is used to model the expression, posture and appearance of the digital human corresponding to the target object based on the action sequence video data of the target object using a three-dimensional Gaussian splash model, so as to obtain the Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters and appearance parameters of the target object;

[0127] The second training module 420 is used to model the digital human's facial expressions, postures, and appearance based on action sequence video data using a neural texture model, thereby obtaining the digital human's neural texture model.

[0128] The third training module 430 is used to prune the Gaussian model and perform a mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; the Gaussian model of the digital human is optimized based on the mixed rendered image and action sequence video data to obtain an editable digital human model.

[0129] The present invention provides an editable digital human modeling device based on neural texture and 3D Gaussian. It models the expression, posture and appearance of the digital human corresponding to the target object using a 3D Gaussian splash model, and then models the expression, posture and appearance of the digital human using a neural texture model. Next, the Gaussian model is pruned and the rendered image output by the model is mixed with the rendered image output by the neural texture model to obtain a mixed rendered image. Finally, the Gaussian model of the digital human is optimized using the mixed rendered image and action sequence video data to obtain an editable digital human model. This reduces the storage overhead and structural complexity of the digital human model, and improves the high-frequency details and editing efficiency of the skin surface of the digital human model, thereby improving the generalization of 3D Gaussian modeling in different scenarios.

[0130] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can call logic instructions in the memory 530 to execute an editable digital human modeling method based on neural texture and 3D Gaussian. This method includes: modeling the expression, posture, and appearance of the digital human corresponding to the target object using a 3D Gaussian splash model based on the action sequence video data of the target object, obtaining a Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters, and appearance parameters of the target object; modeling the expression, posture, and appearance of the digital human using a neural texture model based on the action sequence video data, obtaining a neural texture model of the digital human; pruning the Gaussian model and performing a blended rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human, obtaining a blended rendered image; and optimizing the Gaussian model of the digital human based on the blended rendered image and the action sequence video data, obtaining an editable digital human model.

[0131] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the editable digital human modeling method based on neural texture and 3D Gaussian provided by the above methods. The method includes: modeling the expression, posture, and appearance of the digital human corresponding to the target object using a 3D Gaussian splash model based on the action sequence video data of the target object to obtain a Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters, and appearance parameters of the target object; modeling the expression, posture, and appearance of the digital human using a neural texture model based on the action sequence video data to obtain a neural texture model of the digital human; pruning the Gaussian model and performing a mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; and optimizing the Gaussian model of the digital human based on the mixed rendered image and the action sequence video data to obtain an editable digital human model.

[0133] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the editable digital human modeling method based on neural texture and 3D Gaussian provided by the above methods. The method includes: modeling the expression, posture, and appearance of a digital human corresponding to a target object using a 3D Gaussian splash model based on video data of the action sequence of the target object, to obtain a Gaussian model of the digital human; the action sequence video data includes expression parameters, posture parameters, and appearance parameters of the target object; modeling the expression, posture, and appearance of the digital human using a neural texture model based on the video data of the action sequence, to obtain a neural texture model of the digital human; pruning the Gaussian model and performing a mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human, to obtain a mixed rendered image; and optimizing the Gaussian model of the digital human based on the mixed rendered image and the video data of the action sequence, to obtain an editable digital human model.

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for modeling editable digital humans based on neural textures and 3D Gaussian, characterized in that, include: The expression, posture, and appearance of the digital human corresponding to the target object are modeled using a three-dimensional Gaussian splash model based on the action sequence video data of the target object, thus obtaining the Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters, and appearance parameters of the target object; The digital human's facial expressions, postures, and appearance are modeled using a neural texture model based on the action sequence video data, thus obtaining the digital human's neural texture model. The Gaussian model is pruned, and the rendered image output by the pruned Gaussian model is mixed with the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; the Gaussian model of the digital human is optimized based on the mixed rendered image and the action sequence video data to obtain an editable digital human model. The hybrid rendered image is obtained through the following steps: Pruning the Gaussian model: The opacity threshold of the Gaussian renderer is determined based on the depth image, and the semantic labels of the targets to be pruned are determined based on the frame mask of the target video frame. The depth image is obtained by rasterizing the target video frame using an initialized differentiable rasterizer based on the triangular mesh information, vertex coordinates, index, and optimized neural texture of the target video frame in the SMPL-X model. The frame mask is obtained from action sequence video data. The targets to be pruned include hands and arms. Based on the Gaussian renderer, the Gaussian points inside the corresponding triangle mesh of the Gaussian model, as well as the Gaussian points of the hand and arm, are pruned according to the opacity threshold and the semantic labels of the target to be pruned, to obtain the pruned Gaussian model. The pruned 3D Gaussian model is input into the Gaussian renderer to obtain a rendered image and a corresponding Gaussian opacity map. The rendered image is then superimposed with the rasterized rendered image output by the neural texture model of the digital human according to the Gaussian opacity map to obtain a hybrid rendered image. The hybrid rendered image is represented by the following formula: = ； in, For the hybrid rendered image, This is the rendered image output by the pruned Gaussian model. For Gaussian opacity, The rasterized rendering image output by the neural texture model of the digital human, i.e., the neural texture rendering image.

2. The editable digital human modeling method based on neural texture and 3D Gaussian as described in claim 1, characterized in that, The action sequence video data was acquired using a monocular camera; The process of modeling the expression, posture, and appearance of the digital human corresponding to the target object based on the action sequence video data of the target object using a three-dimensional Gaussian splash model to obtain the Gaussian model of the digital human includes: The SMPL-X model parameters, frame mask, and camera parameters of the monocular camera for the target video frame are obtained from the action sequence video data; the SMPL-X model parameters include the expression parameters, the pose parameters, and the appearance parameters. The local 3D Gaussian parameters of the SMPL-X model are initialized based on the SMPL-X model parameters; the local 3D Gaussian parameters include position parameters, rotation parameters, scale parameters, color parameters, opacity parameters, index parameters, and semantic labels; The global three-dimensional Gaussian parameters of the SMPL-X model are calculated based on the local three-dimensional Gaussian parameters and the parameters of the SMPL-X model. The Gaussian renderer renders the target video frame based on the global 3D Gaussian parameters and the camera parameters to obtain a Gaussian rendered image, and calculates the color loss based on the target video frame and the target video frame. The first comprehensive loss is obtained based on the color loss, perceptual loss, structural similarity loss, Sobel operator and regularization term. The three-dimensional Gaussian splash model is iteratively trained based on the first comprehensive loss through gradient backpropagation to obtain the Gaussian model.

3. The editable digital human modeling method based on neural texture and 3D Gaussian as described in claim 1, characterized in that, The step of modeling the digital human's facial expressions, postures, and appearance based on the action sequence video data using a neural texture model to obtain the digital human's neural texture model includes: Initialize a differentiable rasterizer renderer based on triangle mesh information, vertex coordinates, and indexes from the SMPL-X model; Based on the initialized differentiable rasterizer renderer, the target video frame is rasterized according to the triangle mesh information corresponding to the target video frame in the SMPL-X model, the vertex coordinates, the index, and the optimized neural texture to obtain a neural texture rendering image, and the color loss is calculated based on the neural texture rendering image and the target video frame. The second comprehensive loss is obtained based on the color loss, perceptual loss, structural similarity loss, and total variation algorithm. The neural texture model is then iteratively trained using gradient backpropagation based on the second comprehensive loss to obtain the neural texture model.

4. The editable digital human modeling method based on neural texture and 3D Gaussian as described in claim 1, characterized in that, The pruning of the Gaussian model includes: The opacity threshold of the Gaussian renderer is determined based on the depth image, and the semantic label of the target to be pruned is determined based on the frame mask of the target video frame; wherein, the depth image is obtained by rasterizing the target video frame using an initialized differentiable rasterizer based on the triangle mesh information, vertex coordinates, index, and optimized neural texture corresponding to the target video frame in the SMPL-X model; the frame mask is obtained from the action sequence video data; The Gaussian renderer prunes the Gaussian points inside the triangular mesh corresponding to the Gaussian model, as well as the Gaussian points of the hand and arm, based on the opacity threshold and the semantic label of the target to be pruned, to obtain the pruned Gaussian model.

5. The editable digital human modeling method based on neural texture and 3D Gaussian as described in claim 1, characterized in that, The step of optimizing the Gaussian model of the digital human based on the hybrid rendered image to obtain an editable digital human model includes: The loss value is calculated based on the third comprehensive loss according to the target video frame in the hybrid rendered image and the action sequence video data, and the parameters of the Gaussian model are updated according to the loss value. When the Gaussian model converges, the editable digital human model is obtained. The third comprehensive loss is determined based on the first comprehensive loss and the 2-norm of Gaussian opacity. The first comprehensive loss is determined based on color loss, perceptual loss, structural similarity loss, Sobel operator, and regularization term.

6. The editable digital human modeling method based on neural texture and 3D Gaussian as described in claim 1, characterized in that, After obtaining the editable digital human model, the method further includes: Based on the editable digital human model, the target 3D Gaussian representation is edited according to user input instructions to render a 3D virtual scene of the digital human.

7. An editable digital human modeling device based on neural texture and 3D Gaussian, employing the editable digital human modeling method based on neural texture and 3D Gaussian as described in claim 1, characterized in that, include: The first training module is used to model the expression, posture, and appearance of the digital human corresponding to the target object based on the action sequence video data of the target object using a three-dimensional Gaussian splash model, thereby obtaining the Gaussian model of the digital human; the action sequence video data includes the expression parameters, posture parameters, and appearance parameters of the target object; The second training module is used to model the digital human's facial expressions, postures, and appearance based on the action sequence video data using a neural texture model, thereby obtaining the digital human's neural texture model. The third training module is used to prune the Gaussian model and perform a mixed rendering based on the rendered image output by the pruned Gaussian model and the rendered image output by the neural texture model of the digital human to obtain a mixed rendered image; and optimize the Gaussian model of the digital human based on the mixed rendered image and the action sequence video data to obtain an editable digital human model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the editable digital human modeling method based on neural texture and three-dimensional Gaussian as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the editable digital human modeling method based on neural texture and three-dimensional Gaussian as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the editable digital human modeling method based on neural texture and three-dimensional Gaussian as described in any one of claims 1 to 6.

Images