Neural radiance field digital human generation method, system and device

Abstract

The application discloses a kind of neural radiation field digital person generation method, system and device, comprising: S1, constructing action and texture editing neural radiation field model;S2, training action and texture editing neural radiation field model;S3, based on the action and texture editing neural radiation field model that is trained to obtain action and texture editing custom digital person two-dimensional image.The application can realize neural radiation field digital person generation method, system and device.

Description

Technical Field

[0001] This invention relates to the field of digital human generation, and in particular to a method, system, and apparatus for generating digital humans using neural radiation fields. Background Technology

[0002] Digital twins are an important research direction in computer vision, especially in technologies related to digital human figures. They utilize digital technology to generate computer images of real-world people, which can then be edited. In short video scenarios, users want to be able to edit the viewing perspective; in virtual anchor scenarios, staff need to edit the virtual anchor's movements. However, existing digital human technologies still have certain limitations in meeting user needs.

[0003] Currently, the creation and editing of digital characters mainly employs methods such as Signed Distance Function (SDM), Generative Adversarial Networks (GANs), and Neural Radiance Field (NNF). However, these methods all have limitations and drawbacks. SDM relies on 3D data, but acquiring 3D datasets requires expensive specialized equipment and manpower, and the generated digital characters have low resolution and may lack the contours of the human body surface. While GANs only require 2D images and can generate high-resolution digital characters, they often produce effects that do not match the real world when faced with uncommon information. Furthermore, this method has limited viewpoint editing capabilities, and achieving ideal convergence during model training is difficult. NNF is implemented by training a neural network on a series of 2D images. It has strong fitting capabilities and can generate new viewpoint images of scenes with a certain degree of complexity. NNF algorithms can generate high-resolution digital characters and possess viewpoint editing capabilities. However, existing algorithms can only generate discrete, motion-editable digital characters and cannot generate smooth, high-resolution digital characters. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and apparatus for generating digital humans using neural radiation fields, aiming to solve the problem.

[0005] This invention provides a method for generating a digital human using neural radiation fields, comprising:

[0006] S1. Construct a neural radiation field model for motion and texture editing;

[0007] S2, a neural radiation field model for training actions and texture editing;

[0008] S3. Based on the trained neural radiation field model for motion and texture editing, obtain a customized 2D image of a digital human with motion and texture editing.

[0009] The present invention also provides a neural radiation field digital human generation system, comprising:

[0010] Builder modules: used to build neural radiation field models for motion and texture editing;

[0011] The training module is used to train neural radiation field models for motion and texture editing;

[0012] Application module: Used to obtain customized 2D images of digital humans with motion and texture editing based on a trained neural radiation field model of motion and texture editing.

[0013] This invention also provides a neural radiation field digital human generation device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above method.

[0014] This invention also provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the above-described method.

[0015] Using the embodiments of the present invention, digital human generation of neural radiation fields can be realized.

[0016] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the neural radiation field digital human generation method according to an embodiment of the present invention;

[0019] Figure 2 This is a detailed flowchart of the neural radiation field digital human generation method according to an embodiment of the present invention;

[0020] Figure 3This is a schematic diagram of the neural radiation field digital human generation system according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the neural radiation field digital human generation device according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Method Implementation Examples

[0024] According to an embodiment of the present invention, a method for generating a digital human using neural radiation fields is provided. Figure 1 This is a flowchart of the neural radiation field digital human generation method according to an embodiment of the present invention, such as... Figure 1 As shown, it specifically includes:

[0025] S1. Construct a neural radiation field model for motion and texture editing;

[0026] S2, a neural radiation field model for training actions and texture editing;

[0027] S3. Based on the trained neural radiation field model for motion and texture editing, obtain a customized 2D image of a digital human with motion and texture editing.

[0028] S1 specifically includes: Given the sampling point positions in space, the actions of a digital human in a T-pose represented by a neural radiation field are edited by inputting human motion information p. Combined with the input human texture encoding, the color c and volume density σ of the sampling points are calculated to customize and generate the digital human's actions and textures, as defined below:

[0029] F(x,d,p,α)=(c,σ)

[0030] Where x is the three-dimensional coordinate of the sampling point, d is the camera parameter or viewpoint parameter, p is the human motion information, and α is the texture encoding; after predicting the color c and volume density σ of the sampling point, a two-dimensional image of the digital human is generated using traditional volume rendering methods.

[0031] The editing of digital human motion specifically includes: transforming the sampled points based on a linear hybrid skinning algorithm, that is:

[0032]

[0033] Where K is the number of bones, w iLet R be the bone weight of the i-th bone. i ,t i ) is the action basis, i.e., the deformation matrix M i The deformation matrix is ​​calculated from the human motion information p; the skeletal weights w i It is represented by a three-dimensional convolutional neural network. For a sampling point x, the skeletal weights can be obtained by sampling at the spatial position corresponding to the three-dimensional convolutional neural network. The human motion information p is used to perform motion-based transformation on point x in space, so that the T-pose digital human is transformed into the specified motion pose.

[0034] Texture editing specifically includes: binding a human photograph with a specific texture to a corresponding texture code α, where each texture corresponds to a unique texture code, and editing the texture of the digital human by inputting different texture codes;

[0035] The digital human definition after motion and texture editing is as follows:

[0036] F(T s (x,p),d,α)=(c,σ)

[0037] V(F(T s (x,p),d,α))=I

[0038] Where T s (x,p) represents motion editing; color c and volume density σ are predicted by neural radiation field, and V is a traditional volume rendering method, generating a 2D image I of a digital human with motion and texture editing.

[0039] S2 specifically includes: a given input image, human motion information p, texture encoding α, and viewpoint d. i (i = 1…N), the training of neural radiation fields with editable actions and textures is achieved by optimizing the following equations, and the optimization formula is as follows:

[0040]

[0041] Where N is the number of input images, Loss function, V-volume rendering, I i For the specific d input i The image is viewed from a specific perspective, where p represents human motion information and α represents texture encoding.

[0042] S3 specifically includes: inputting customized motion information p and texture encoding α to generate a customized 2D image of a digital human after motion and texture editing.

[0043] The specific implementation method is as follows:

[0044] By generating a standard action, namely the T-pose, represented by a neural radiation field, a digital human is created. This is combined with linear blending skinning to achieve effects such as motion and perspective editing. The process takes a complete 2D image of the human body, or each frame of a video stream, as input. Through masking operations and a 3D pose estimation model, the camera imaging parameters of the target human body in the image are obtained, namely the camera imaging matrix M. c Human motion information p (including bone position J and bone orientation Ω) is collected for subsequent linear blending skinning operations. During the generation of 2D images using traditional graphics-based volume rendering methods with neural radiation fields, linear blending skinning based on human joints is performed on the sampled points. Furthermore, texture encoding is used to control the surface texture of the digital human, enabling the editing of both motion and texture. After training the digital human using neural radiation fields, we can generate a high-resolution digital human with editable motion and texture, and smooth movements, based on the input motion information, texture encoding, and camera imaging parameters.

[0045] 1. Preprocessing stage

[0046] Masking the input 2D image, i.e., removing redundant background information, can effectively improve the accuracy of subsequent 3D pose estimation. Through this operation, we can extract key information about the target human body from the image. The masked 2D image can then be used in the subsequent 3D pose estimation model to obtain the camera imaging parameters of the target human body in the image, i.e., the camera imaging matrix M. c Human motion information p (including bone position J and bone orientation Ω) is also crucial. Accurate motion and camera information are essential for subsequent linear blending skinning operations and the generation of 2D images from neural radiation fields.

[0047] 2. Digital Human Generation Based on Neural Radiation Fields

[0048] The neural radiation field is a neural network trained on a series of two-dimensional images. It optimizes a scene represented by a five-dimensional function from the input two-dimensional images, combining traditional volumetric rendering methods in computer graphics to generate new perspective two-dimensional images of the scene. It leverages the powerful fitting ability of neural networks to generate new perspective images of scenes with a certain degree of complexity. Its definition is as follows:

[0049] F(x,θ)=(c,σ)

[0050] Where x is the three-dimensional coordinate of the sampling point, θ is the coordinate of the camera imaging matrix M. c The relevant ray axis angle parameters, color value c, and transparency σ are the parameters required for volume rendering to generate a two-dimensional image.

[0051] We train a standard motion digital human represented by a neural radiation field, based on our input camera imaging matrix M. cThis allows for editing from the perspective of a digital human.

[0052] 3. Digital Human Motion and Texture Editing

[0053] By combining the linear blending skinning algorithm, we can achieve motion editing of digital humans. Linear blending skinning is an algorithm for deforming human meshes based on skeletal structure. It is one of the most commonly used motion-driven algorithms in current human animation, characterized by fast computation speed and good deformation effects. In the linear blending skinning algorithm, each mesh vertex is controlled by multiple bones. Bone weights represent the degree of influence of each bone on the vertex. When a bone moves, its associated vertex also moves accordingly. By interpolating the positions of mesh vertices, we can create smooth animation effects. The definition is as follows:

[0054]

[0055] Where T s The image shows the deformed human body mesh, where x represents the number of vertices, K represents the number of bones, and w represents the number of bones. i Let R be the bone weight of the i-th bone. i ,t i ) is the action basis, i.e., the deformation matrix M i This can be calculated from human motion information p. In the process of digital human generation, skeletal weights w... i It is represented by a three-dimensional convolutional neural network. For a sampling point x, we sample in the three-dimensional convolutional neural network to obtain the bone weights.

[0056] Our method combines neural radiation fields and linear blending skin to achieve motion editing effects on a standard-motion digital human represented by a neural radiation field. Specifically, we use sampling points from a 2D image generated by volume rendering of the neural radiation field as vertices of the human body mesh in the linear blending skin. Simultaneously, we combine this with motion information p estimated from the 2D image to deform the digital human, thereby achieving motion editing effects.

[0057] Furthermore, we introduce the concept of texture encoding. We compare human photographs with the same texture to a learnable texture encoding α. i The data is bound together, with each texture corresponding to a unique texture code. By inputting different texture codes, we can edit the texture of the digital human and adjust its appearance.

[0058] The digital human definition after motion and texture editing is as follows:

[0059] F(T s (x,p),θ,α)=(c,σ)

[0060] Where x is the three-dimensional coordinate of the sampling point, θ is the coordinate of the camera imaging matrix M. c The relevant ray axis angle parameters, p represents motion information, and T represents... s (x,p) represents a linear blending skinning operation with α-texture encoding. The color value c and transparency σ are predicted using a neural radiation field, which are the parameters required for volume rendering to generate a 2D image.

[0061] The pixel values ​​of a two-dimensional image can be obtained through volume rendering from (c,σ) predicted by the neural radiation field, as defined below:

[0062]

[0063] in

[0064] For each pixel, we correspond to a ray r and sample G points along that ray. The spacing between these sampling points is Δt. i We input the coordinates of each sampling point and the ray axis angle parameter θ, and pass them to the neural radiation field to return the corresponding color value c and transparency σ. Using volume rendering techniques, we accumulate the color value c and transparency σ to obtain the final pixel value C(r) for that pixel. In this way, by performing ray sampling, neural radiation field processing, and volume rendering on each pixel, we are able to generate two-dimensional images with a novel perspective.

[0065] 4. Generation of Editable Digital Humans Based on Neural Radiation Fields

[0066] V(F(T s (x,p),θ,α))=I

[0067] Where V represents volume rendering, we use volume rendering techniques to compare the 2D human body image I after motion and texture editing with the 2D human body image I in the dataset. T We calculate the loss for each pixel value. We apply mean squared error (MSE) loss and perceptual similarity (LPIPS) loss to the network, combining them to obtain the total loss function L, i.e., L = L MSE +λL LPIPS , where λ is a hyperparameter.

[0068] After training, we obtained a standard-motion digital human represented by a neural radiation field. Next, based on the camera imaging matrix M of the specified input viewpoint... c Action information p and texture encoding α iThis allows us to generate digital humans with target movements and textures, achieving editability of viewpoint and action, and producing high-resolution digital humans. Furthermore, by inputting continuous motion flow information (i.e., continuous motion information p), we can also generate digital humans with smooth motion and high resolution. This effect is difficult to achieve with existing digital human technologies.

[0069] The input to this method is a complete 2D image of the human body or each frame of a video stream. First, redundant background information is removed from the image through a masking operation. Then, a 3D pose estimation model is used to obtain the camera imaging parameters of the target human body in the image, i.e., the camera imaging matrix M. c Human motion information p (including skeletal position J and skeletal orientation Ω). Based on the camera imaging matrix M c The system generates sampling points required for volume rendering and uses a linear blending skinning algorithm to deform these sampling points, enabling motion editing of the digital human. Furthermore, it combines texture information with a learnable texture encoding α. i Binding is performed to enable texture editing of the digital human. Finally, traditional volume rendering methods in computer graphics are used to generate the final two-dimensional image of the digital human.

[0070] 2. After training a digital human to perform a standard action (T-pose) represented by a neural radiation field, the input camera imaging matrix M is used as the basis for the training. c Human motion information p and the texture encoding α of the corresponding texture i We can generate high-resolution digital humans with editable viewpoints, movements, and textures. If the input human motion information p is a continuous motion stream, we can generate digital humans with smooth movements, enabling fluid editing of the digital human's movements.

[0071] This invention is a method for generating digital humans with editable viewpoints and movements. Our approach leverages the advantages of both neural radiation fields and linear blending skins. On one hand, neural radiation fields offer high-resolution generation and superior viewpoint editing capabilities, features not found in other methods; on the other hand, linear blending skins combine to offer fast computation speed and good deformation effects in human animation; simultaneously, a texture encoding system controls the texture of the digital human, thereby generating high-resolution digital humans with smooth movements while maintaining editable viewpoints, movements, and textures.

[0072] System Implementation Examples

[0073] According to embodiments of the present invention, a neural radiation field digital human generation system is provided. Figure 3 This is a schematic diagram of the neural radiation field digital human generation system according to an embodiment of the present invention, as shown below. Figure 3 As shown, it specifically includes:

[0074] Builder modules: used to build neural radiation field models for motion and texture editing;

[0075] The training module is used to train neural radiation field models for motion and texture editing;

[0076] 2D Image Module: Used to obtain customized 2D images of digital humans with motion and texture editing based on a trained neural radiation field model of motion and texture editing.

[0077] The module is specifically used to: given the location of sampling points in space, edit the movements of a digital human in a T-pose represented by a neural radiation field by inputting human motion information p; combine the input human texture encoding to calculate the color c and density σ of the sampling points; and customize and generate the movements and textures of the digital human, as defined below:

[0078] F(x,p,α)=(c,σ)

[0079] Where x is the three-dimensional coordinate of the sampling point, p is the human motion information, and α i Texture encoding is performed; after predicting the color c and density σ of the sampling points, a two-dimensional image of the digital human is generated using traditional volume rendering methods.

[0080] The editing of digital human motion specifically includes: transforming the sampled points based on a linear hybrid skinning algorithm, that is:

[0081]

[0082] Where K is the number of bones, w i Let R be the bone weight of the i-th bone. i ,t i ) is the action basis, i.e., the deformation matrix M i The deformation matrix is ​​calculated from the human motion information p; the skeletal weights w i It is represented by a three-dimensional convolutional neural network, and the skeletal weights are obtained by sampling in the three-dimensional convolutional neural network; the human motion information p is used to perform motion-based transformation on point x in space, so that the T-pose digital human is transformed into the specified motion pose.

[0083] Texture editing specifically includes: associating a human photograph with a specific texture with a texture code α. i Binding is performed, where each texture corresponds to a unique texture code, and the texture of the digital human can be edited by inputting different texture codes;

[0084] The digital human definition after motion and texture editing is as follows:

[0085] F(T s (x,p),α)=(c,σ)

[0086] V(F(T s (x,p),α))=I

[0087] Where T s (x,p) represents motion editing; color value c and transparency σ are predicted through neural radiation fields, and V represents a traditional volume rendering method, generating a 2D image I of a digital human with motion and texture editing.

[0088] The training module is specifically used for: given an input image, human motion information p, and texture encoding α i Regarding the perspective, the training of editable neural radiation fields for motion and texture is achieved by optimizing the equations. The optimized formula is as follows:

[0089]

[0090] Where N is the number of input images, Loss function, V-volume rendering, I i The input image is p, where p represents human motion information and α represents texture encoding.

[0091] The 2D image module is specifically used for: the input customized motion information p and texture encoding α i Generate customized 2D images of digital humans with motion and texture editing.

[0092] The embodiments of the present invention are system embodiments corresponding to the above method embodiments. The specific operation of each module can be understood by referring to the description of the method embodiments, and will not be repeated here.

[0093] Device Example 1

[0094] This invention provides a method for generating digital humans using neural radiation fields, such as... Figure 4 As shown, it includes: a memory 40, a processor 42, and a computer program stored on the memory 40 and executable on the processor 42. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0095] Device Example 2

[0096] This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor 42, implements the steps described in the above method embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions to the technical solutions of the embodiments of the present invention do not cause the essence of the corresponding technical solutions to deviate from the scope of the present solution.

Claims

1. A method for generating a digital human using neural radiation fields, characterized in that, include: S1. Construct a neural radiation field model for action and texture editing, specifically including: Texture editing specifically includes: associating a human photograph with a specific texture with a corresponding texture code. Binding is performed, where each texture corresponds to a unique texture code, and the texture of the digital human can be edited by inputting different texture codes; The digital human definition after motion and texture editing is as follows: in For motion editing, The three-dimensional coordinates of the sampling point For human motion information, For camera parameters or viewpoint parameters; predict color using neural radiation fields. and volume density , Generate 2D images of digital humans with motion and texture editing, using traditional volume rendering methods. ; S2, a neural radiation field model for training actions and texture editing, specifically including: Given an input image and human motion information Texture encoding and perspective The training of neural radiation fields with editable motion and texture is achieved by optimizing the following equations: in This refers to the number of images entered. loss function Volume rendering, For specific input Images from a specific perspective For human motion information, For texture encoding, For the ray axis angle parameter; S3. Based on the trained neural radiation field model for motion and texture editing, obtain a customized 2D image of a digital human with motion and texture editing.

2. The method according to claim 1, characterized in that, S1 specifically includes: given the location of a sampling point in space, and inputting human motion information. This method edits the motion of a digital human in a T-pose represented by a neural radiation field, and calculates the color of the sampled points by combining the input human texture encoding. and volume density Customize the generated digital human's movements and textures, as defined below: in, The three-dimensional coordinates of the sampling point For camera parameters or viewpoint parameters, For human motion information, Encode the texture; predict the color of the sampled points. and volume density Then, a two-dimensional image of the digital human is generated using traditional volume rendering methods; The editing of digital human motion specifically includes: transforming the sampled points based on a linear hybrid skinning algorithm, that is: in, For the number of bones, For the first Bone weights of block bones For action basis, i.e., deformation matrix The deformation matrix is ​​composed of human motion information. Calculated; Bone weights Represented by a three-dimensional convolutional neural network, for sampling points Skeletal weights can be obtained by sampling at the spatial location corresponding to the 3D convolutional neural network; through human motion information Point Perform motion-based transformations in space to transform a T-pose digital human into a specified motion pose.

3. The method according to claim 2, characterized in that, S3 specifically includes: input customized action information. and texture encoding Generate customized 2D images of digital humans with motion and texture editing.

4. A neural radiance field digital human generation system, characterized in that, include: Builder modules: used to build neural radiation field models for action and texture editing, specifically for: Texture editing specifically includes: associating a human photograph with a specific texture with a texture code. Binding is performed, where each texture corresponds to a unique texture code, and the texture of the digital human can be edited by inputting different texture codes; The digital human definition after motion and texture editing is as follows: in For motion editing, The three-dimensional coordinates of the sampling point For human motion information, For camera parameters or viewpoint parameters; predict color using neural radiation fields. and volume density , Generate 2D images of digital humans with motion and texture editing, using traditional volume rendering methods. ; The training module is used to train the neural radiation field model for action and texture editing, specifically for: Given an input image and human motion information Texture encoding Regarding the perspective, the training of editable neural radiation fields for motion and texture is achieved by optimizing the equations. The optimized formula is as follows: in This refers to the number of images entered. loss function Volume rendering, For specific input Images from a specific perspective For human motion information, For texture encoding, For the ray axis angle parameter; Application module: Used to obtain customized 2D images of digital humans with motion and texture editing based on a trained neural radiation field model of motion and texture editing.

5. The system according to claim 4, characterized in that, The construction module is specifically used for: given the location of sampling points in space, and inputting human motion information. This method edits the motion of a digital human in a T-pose represented by a neural radiation field, and calculates the color of the sampled points by combining the input human texture encoding. and volume density Customize the generated digital human's movements and textures, as defined below: in, The three-dimensional coordinates of the sampling point For camera imaging parameters, For human motion information, Encode the texture; predict the color of the sampled points. and volume density Then, a two-dimensional image of the digital human is generated using traditional volume rendering methods; The editing of digital human motion specifically includes: transforming the sampled points based on a linear hybrid skinning algorithm, that is: in, For the number of bones, For the first Bone weights of block bones For action basis, i.e., deformation matrix The deformation matrix is ​​composed of human motion information. Calculated; Bone weights Represented by a three-dimensional convolutional neural network, for sampling points Skeletal weights can be obtained by sampling at the spatial location corresponding to the 3D convolutional neural network; through human motion information Point Perform motion-based transformations in space to transform a T-pose digital human into a specified motion pose.

6. The system according to claim 5, characterized in that, The application module is specifically used for: inputting customized action information. and texture encoding Generate customized 2D images of digital humans with motion and texture editing.

7. A neural radiation field digital human generation device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the neural radiation field digital human generation method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the neural radiation field digital human generation method as described in any one of claims 1 to 3.

Images