Relightable portrait generation method based on Gaussian three-dimensional splashing

Abstract

The invention provides a reilluminable portrait generation method based on Gaussian three-dimensional splashing, which decouples albedo attributes and illumination, so that the albedo attributes tend to express essential attributes irrelevant to visual angles, and three-dimensional inconsistency during visual angle switching is reduced; in addition, one group of spherical harmonic coefficients is used for representing incident illumination, and the other group of spherical harmonic coefficients is used for representing radiation transmission; therefore, the illumination rendering equation can be simplified into vector dot multiplication of the radiation transmission vector and the spherical harmonic illumination coefficient, so that the emergent light and the incident light are in a linear relation. And finally, multiplying the inner product by the albedo attribute to form a color attribute, and displaying stable three-dimensional portrait relighting information in the form of color-lacking geometric parameters. Therefore, the efficient rendering speed of three-dimensional Gaussian splashing is inherited while a physical rendering equation is met, and the efficiency and quality of a portrait relighting rendering technology are improved.

Description

Technical Field

[0001] This invention relates to the field of computer graphics, specifically to the field of 3D human image generation technology in computer vision, and more specifically, to a method for generating re-illuminated human images based on Gaussian 3D splashing. Background Technology

[0002] Relighting of 3D-perceptual portraits is an important research topic in the fields of computer graphics and computer vision. Relighting and synthesizing portraits from new perspectives has important applications in cultural media, portrait photography, film and television games, short video production, and VR / AR.

[0003] Existing 3D perceptual human relighting methods mainly use adversarial generative neural radiation fields as 3D human priors. However, the necessary upsampling and volumetric rendering components limit 3D consistency and generation speed. They mainly use real data captured by expensive and difficult-to-obtain light stages as supervision signals, which limits data diversity and model generalization ability. They mainly rely on neural networks as renderers, and each frame of rendering requires neural network inference, which limits their real-time rendering capabilities.

[0004] There has been previous work on using generative adversarial networks trained on human portrait datasets as priors, neural radiation fields as 3D representations, and lighting information for conditional generation to achieve 3D-perceptive human portrait relighting. However, when the inventors were researching 3D human portrait relighting rendering technology, they found that almost all existing technologies use neural radiation fields as 3D representations, and their upsampling modules and volumetric rendering components inevitably lead to 3D inconsistencies and low rendering speeds.

[0005] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0006] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a method for generating re-illuminated human figures based on Gaussian three-dimensional splashing.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] According to a first aspect of the present invention, a method for generating a portrait based on Gaussian 3D splashing is provided, comprising: acquiring Gaussian noise of a target, spherical harmonic illumination coefficients for relighting the target person, and camera parameters; acquiring a pre-trained 3D Gaussian portrait model to generate geometric parameters for relighting the target person, the model comprising: a generator for extracting Gaussian features from the input Gaussian noise and camera parameters; an albedo decoder for extracting albedo attributes independent of the viewpoint from the Gaussian features; a geometry decoder for generating color-deficient geometric parameters of the person based on the Gaussian features, which are a set of Gaussian spheres without color attributes; a radiative transfer decoder for modeling diffuse and specular reflection based on the Gaussian features and the camera parameters to obtain a radiative transfer vector related to the viewpoint expressed in spherical harmonic functions; a fusion module for multiplying the inner product of the radiative transfer vector and the spherical harmonic illumination coefficients by the albedo attributes to obtain a set of color attributes, and incorporating them into the color-deficient geometric parameters to obtain the geometric parameters for relighting the target person; and performing Gaussian splashing processing based on the geometric parameters for relighting the target person to obtain a rendered image of the target person. This scheme achieves at least the following beneficial technical effects: Firstly, by decoupling the albedo attribute from lighting, the albedo attribute tends to express essential properties independent of the viewpoint, reducing 3D inconsistencies when switching viewpoints. Secondly, the scheme uses one set of spherical harmonic coefficients to represent incident lighting and another set to represent radiative transmission; thus, the lighting rendering equation can be simplified to a vector dot product (inner product) of the radiative transmission vector and the spherical harmonic lighting coefficients, resulting in a linear relationship between outgoing and incident light. Furthermore, the inner product is multiplied by the albedo to form the color attribute, which, together with the color-deficient geometric parameters, forms a stable 3D portrait relighting information. Therefore, while conforming to the physically based rendering equation, it inherits the high-efficiency rendering speed of 3D Gaussian splashing, improving the efficiency and quality of portrait relighting rendering technology.

[0009] Optionally, during inference, the spherical harmonic illumination coefficient is a three-channel illumination coefficient, and the albedo attribute is a vector containing three elements. The fusion module is configured to calculate the inner product of the spherical harmonic illumination coefficient and the three channels of the radiative transfer vector, and multiply the inner product of the three channels by the albedo to obtain the RGB three-channel color attribute. This scheme can achieve at least the following beneficial technical effects: during inference, three-channel illumination coefficients are used for rendering, thereby allowing the application of any desired color lighting effect during relighting.

[0010] Optionally, when training a 3D Gaussian portrait model, the fusion module is configured to: receive single-channel illumination coefficients; calculate the inner product of the single-channel illumination coefficients and the radiative transfer vector for each single channel; duplicate the single-channel inner product twice to form an isomorphic three-channel inner product; and multiply the isomorphic three-channel inner product by the albedo to obtain the three-channel color attributes during training. This scheme can achieve at least the following beneficial technical effects: during training, simply generating conditions using three-channel illumination input makes it difficult to decouple the color ambiguity between albedo and illumination. Therefore, during training, it is assumed that all illumination is single-channel light. The single-channel light assumption is proposed to be applied during training so that color is controlled only by albedo. The contribution of color attributes is attributed entirely to the material rather than the illumination, thereby better decoupling illumination and albedo. This allows the albedo decoder to better learn to extract albedo knowledge that is independent of illumination and viewpoint. Furthermore, during training, the data specification consistency is maintained by forming isomorphic three-channel inner products by copying the inner product of the single channels. However, during inference, three-channel illumination coefficients are used, which can be applied to generate any desired color illumination effect during re-illumination.

[0011] Optional methods for obtaining color attributes include:

[0012]

[0013] in, Represents a Gaussian sphere in a set of Gaussian spheres Color attributes, Indicates the use of albedo decoder Compressed Gaussian embedding based on Gaussian features Extracted albedo attribute, This indicates the order of the spherical harmonic function of illumination. Represents the spherical harmonic coefficient of illumination. Individual harmonic coefficients Indicates the use of a radiative transmission decoder Based on compressed Gaussian embedding and perspective encoding Extracting the first from the radiative transfer vector This scheme achieves at least the following beneficial technical effects: it uses pre-computed compressed Gaussian embedding to obtain color attributes, which can be cached for real-time rendering using standard 3D Gaussian splashing.

[0014] Optionally, the radiative transfer decoder includes: a compression layer for linearly compressing Gaussian features to obtain compressed Gaussian embeddings; a viewpoint encoding layer for extracting viewpoint codes from camera parameters; and a radiative transfer decoding layer for extracting radiative transfer vectors based on the compressed Gaussian embeddings and viewpoint codes. During training, the generator and compression layer compute the compressed Gaussian embeddings for the viewpoints corresponding to the input camera parameters on the fly. During inference, the generator and compression layer pre-compute and cache the compressed Gaussian embeddings for multiple viewpoints. When the user adjusts the viewpoint, the corresponding compressed Gaussian embedding is obtained from the cached embeddings and input to the radiative transfer decoding layer. This scheme achieves at least the following beneficial technical effects: Based on this scheme, during training, the compressed Gaussian embeddings for the required viewpoints can be computed on the fly to complete rendering, without calculating the compressed Gaussian embeddings for multiple viewpoints, thus ensuring training efficiency. During inference, since users typically adjust their viewpoints to observe the rendering effect, the generator and compression layer pre-compute and cache the compressed Gaussian embeddings for multiple viewpoints. Utilizing the cached results allows for faster response to rendering requests from different viewpoints, improving the user experience.

[0015] Optionally, a pre-trained 3D Gaussian portrait model is obtained after multiple iterations of training as follows: Multiple random Gaussian noises, spherical harmonic illumination coefficients used in training, and camera parameters are acquired; the 3D Gaussian portrait model is then used to obtain the geometric parameters of the person's relighting corresponding to the random Gaussian noises; Gaussian splashing processing is performed based on the geometric parameters of the person's relighting corresponding to the random Gaussian noises to obtain the rendered image of the generated person; generated data pairs and real data pairs are obtained, where each data pair includes samples and labels, and the samples of the generated data pairs include: the rendered image of the person generated in this training, ... The training uses spherical harmonics and camera parameters, with labels indicating fake samples. Real data pairs consist of real portrait images of people, estimated spherical harmonics and camera parameters based on those images, with labels indicating real samples. Based on the generated and real data pairs, the parameters of the 3D Gaussian portrait model and the discriminator are updated using a generative adversarial network (GAN) and a GAN loss function. Through adversarial training, the model is continuously optimized, enabling it to generate higher-quality, relit portraits that conform to the relevant spherical harmonics and camera parameters. This scheme achieves at least the following beneficial technical effects: It can easily and efficiently train a 3D Gaussian portrait model to generate higher-quality, relit portraits that conform to the relevant spherical harmonics and camera parameters using the constructed GAN; this can be used to improve the viewing experience in scenarios requiring virtual characters (such as games or virtual streamers) and reduce the difficulty of character creation.

[0016] Optionally, the pre-trained 3D Gaussian portrait model can be fine-tuned using pivot fine-tuning technology, enabling it to generate the geometric parameters of relighting for a specified person from a relevant viewpoint based on Gaussian noise in an image. This scheme can achieve at least the following beneficial technical effects: Utilizing pivot fine-tuning technology, this scheme can efficiently generate 3D portraits and perform relighting processing on specified persons in an image, allowing users to quickly complete the relighting processing of a specific portrait based on a single image.

[0017] According to a second aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the method described in the first aspect by executing the executable instructions. Attached Figure Description

[0018] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0019] Figure 1 This is a schematic diagram of the training architecture corresponding to the Gaussian 3D splash-based portrait generation method according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of the rendering result based on white spherical harmonic illumination according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the rendering result based on color spherical harmonic illumination according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of a rendering result based on an ambient light map according to an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram illustrating another rendering result based on ambient light mapping according to an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram comparing the ambient light mapping rendering results of the method according to an embodiment of the present invention with those of other methods. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0026] As mentioned in the background section, the inventors discovered during their research on 3D portrait relighting rendering technology that almost all existing technologies use neural radiation fields as the 3D representation. Their upsampling modules and volumetric rendering components inevitably lead to 3D inconsistencies and low rendering speeds. The inventors believe that this deficiency can be addressed by changing the 3D portrait representation from neural radiation fields to 3D Gaussian splashing, inheriting its inherent high-efficiency rendering capabilities, and constructing a physically based lighting model using unified neural radiative transfer. This model satisfies the physical laws of linear light transfer while learning lighting patterns through data-driven methods, thus achieving both efficient rendering and high-quality relighting results. In illustrative terms, the method decouples the albedo attribute from lighting, allowing the albedo attribute to tend to express essential attributes independent of the viewpoint, reducing 3D inconsistencies when switching viewpoints. Furthermore, the method uses one set of spherical harmonic coefficients to represent incident lighting and another set to represent radiative transfer. Therefore, the lighting rendering equation can be simplified to a vector dot product (inner product) of the radiative transfer vector and the spherical harmonic lighting coefficients, resulting in a linear relationship between the outgoing and incident light. Furthermore, the inner product and albedo are multiplied to form the color attribute, which, together with the color-deficient geometric parameters, forms a stable 3D portrait relighting information. Therefore, while conforming to the physical rendering equation, it inherits the high-efficiency rendering speed of 3D Gaussian splashing, improving the efficiency and quality of portrait relighting rendering technology.

[0027] To better illustrate the present invention, the following is a schematic introduction from the aspects of rendering equation derivation, model training architecture, training process and application scenarios.

[0028] I. Derivation of Rendering Equations

[0029] Traditional pre-calculated radiative transfer has a significant advantage in calculating global illumination, assuming incident light from a distance... From a certain point The emitted light Calculated by the following rendering equation:

[0030]

[0031] in, It is the surface normal. It is the direction of the incident light. It's the perspective direction. It is the bidirectional reflection distribution function (BRDF). It is a light transfer function that considers self-occlusion and mutual reflection. Indicates the direction of incident light. The incident light, Indicates the direction of incident light. Integral differential elements.

[0032] To efficiently compute the integrals in equation (1), the illumination and radiative transfer functions are represented using spherical harmonic functions of order λ. For view-dependent specular surfaces, radiative transfer is represented by a transfer matrix containing BRDF and cosine terms. describe:

[0033]

[0034] in, Represents Gaussian sphere The first emitted light spherical harmonic coefficients, Represents the transmission matrix The (i,j)th element, The incident light's first Sphere harmonic coefficients of order 1. However, due to the transfer matrix... The dimension is proportional to Rank (referring to 0- The square of the number of channels of a spherical harmonic function (order, the same below), that is It has a large computational cost and can usually only handle low-order spherical harmonics.

[0035] For a view-independent diffuse surface, radiative transfer can be represented by a radiative transfer vector. describe:

[0036]

[0037] in, Represents the radiative transfer vector The spherical harmonic coefficients, The incident light's first Spherical harmonic coefficients.

[0038] Radiative transfer is primarily used to describe the diffuse component of illumination. By incorporating the viewpoint direction into the generative network, a unified neural radiative transfer framework can describe both diffuse and low-frequency specular reflection terms. Revisiting equation (1), optical transfer, BRDF, and cosine terms are absorbed into the unified radiative transfer function. In the middle, using spherical harmonics as basis functions, the rendering equation becomes:

[0039]

[0040] in, It is the incident light's first Individual harmonic coefficients It is the radiative transfer function of the 1st generation. Individual harmonic coefficients This represents the radiative transfer function. Specifically, it represents the albedo attribute. from Separation is used to decouple appearance from lighting. After symbol replacement, the view-dependent outgoing light is formalized as:

[0041]

[0042] This unified radiative transfer framework, by making the radiative transfer vector a function of the viewing direction, can unify the rendering of diffuse reflection and low-frequency specular reflection terms or viewing-dependent terms. Furthermore, due to the linearity of the unified radiative transfer function, the model of this invention can achieve colored lighting rendering by applying equation (5) channel-by-channel during inference.

[0043] II. Model Training Architecture

[0044] According to one embodiment of the present invention, see Figure 1 The architecture of generative adversarial networks is used to train the generative network (i.e., the 3D Gaussian human portrait model) and the discriminator.

[0045] Preferably, the 3D Gaussian human portrait model includes: a generator, an albedo decoder, a geometry decoder, a radiative transfer decoder, and a fusion module. Wherein:

[0046] A generator is used to extract Gaussian features from input Gaussian noise and camera parameters. The generator may include convolutional layers, residual connection layers, attention layers, and pooling layers, or combinations thereof. For example, the generator may be a StyleGAN generator, or a generator of StyleGAN may be followed by a linear layer for adjusting dimensions to obtain the generator of this embodiment. Alternatively, implementers may design other generator structures of their own.

[0047] An albedo decoder is used to extract viewpoint-independent albedo attributes from Gaussian features. The albedo decoder is preferably implemented using a fully connected layer.

[0048] A geometric decoder used to generate the missing color geometric parameters of a person based on Gaussian features. It is a set of Gaussian spheres that do not have color attributes. The geometry decoder is preferably implemented using fully connected layers.

[0049] A radiative transfer decoder is used to model diffuse and specular reflections based on the Gaussian features and camera parameters, obtaining a view-dependent radiative transfer vector expressed as a spherical harmonic function. The radiative transfer decoder is preferably implemented using a fully connected layer. Preferably, the radiative transfer decoder includes: a compression layer for linearly compressing Gaussian features to obtain a compressed Gaussian embedding; a viewpoint encoding layer for extracting viewpoint codes from camera parameters; and a radiative transfer decoding layer for extracting the radiative transfer vector based on the compressed Gaussian embedding and the viewpoint code. The radiative transfer decoding layer is a fully connected layer.

[0050] The fusion module is used to multiply the inner product of the radiative transfer vector and the spherical harmonic illumination coefficient with the albedo attribute to obtain a set of color attributes, and then incorporate them into the color-deficient geometric parameters to obtain the geometric parameters of the target person's relighting. Preferably, when training a 3D Gaussian portrait model, the fusion module is configured to: receive single-channel illumination coefficients, calculate the inner product of the single-channel illumination coefficients and the single-channel radiative transfer vector; duplicate the single-channel inner product twice to form an isomorphic three-channel inner product; and multiply the isomorphic three-channel inner product with the albedo to obtain the three-channel color attributes during training. The radiative transfer vector and the spherical harmonic illumination coefficients are processed by the fusion module (unifying the radiative transfer framework) to obtain an inner product (shadow component), which is then multiplied with the albedo attribute to obtain the final color attributes. For visualization purposes, Figure 1 It also provides shadow rendering effects based on inner product and albedo rendering effects based on albedo properties.

[0051] Of the components listed above, the generator, albedo decoder, geometry decoder, and radiative transfer decoder are neural networks that require training. The fusion module is a non-neural network; it only needs to perform predetermined calculations and fuse color attributes and missing color geometric parameters, and requires no training.

[0052] The discriminator D consists of a neural network-based feature extractor and a classifier. The feature extractor extracts discriminative features from the samples, and the classifier determines whether a sample is true or false based on these discriminative features. The classifier can be a sigmoid or softmax classifier. The feature extractor needs to be trained to improve its discriminative ability.

[0053] Based on the above generative adversarial network architecture, it is equivalent to each Gaussian sphere in a set of Gaussian spheres. Assign an additional albedo attribute And a radiative transfer vector (or radiative transfer coefficient). The original view-independent color values ​​are replaced with color attributes calculated based on albedo properties, radiative transfer vectors, and spherical harmonic illumination coefficients. It is the spherical harmonic order. (Illustrative example, such as...) Figure 1 As shown, the generator generates a Gaussian feature for each Gaussian sphere. Similar to the original Gaussian geometric properties, the albedo property... It is a Gaussian feature Input albedo decoder Obtained:

[0054]

[0055] Radiative transfer vector It is achieved by Gaussian embedding With viewpoint direction encoding Pieced together and input into the radiative transfer decoder Obtained:

[0056]

[0057] in, This indicates compressed Gaussian embedding. This indicates the viewpoint encoding.

[0058] According to one embodiment of the present invention, substituting formulas (6) and (7) into formula (5) yields the formula for calculating the emitted light (i.e., the color attribute of the Gaussian sphere) of each Gaussian sphere:

[0059]

[0060] in, Represents a Gaussian sphere in a set of Gaussian spheres Color attributes, Indicates the use of albedo decoder Compressed Gaussian embedding based on Gaussian features Extracted albedo attribute, This indicates the order of the spherical harmonic function of illumination. Represents the spherical harmonic coefficient of illumination. Individual harmonic coefficients Indicates the use of a radiative transmission decoder Based on compressed Gaussian embedding and perspective encoding Extracting the first from the radiative transfer vector Each element.

[0061] III. Training Process

[0062] This invention employs an adversarial training strategy. In response to the ambiguity of illumination and material properties, it proposes to apply a single-channel light assumption during adversarial training, attributing the contribution of color attributes entirely to the material rather than illumination, thereby decoupling illumination and albedo.

[0063] According to one embodiment of the present invention, a pre-trained 3D Gaussian portrait model is obtained after multiple iterations of training as follows: multiple random Gaussian noises, spherical harmonic illumination coefficients used in training, and camera parameters are acquired, and the 3D Gaussian portrait model is used to obtain the geometric parameters of the person's relighting corresponding to the random Gaussian noises; Gaussian splashing processing is performed based on the geometric parameters of the person's relighting corresponding to the random Gaussian noises to obtain the rendered image of the generated person; generated data pairs and real data pairs are acquired, wherein each data pair includes a sample and a label, and the sample of the generated data pair includes: the rendered image of the person generated in this training. The generated data pairs consist of a 3D Gaussian portrait model and a discriminator, along with their estimated spherical harmonics and camera parameters. The generated data pairs are labeled as fake, while the real data pairs consist of real portrait images, estimated spherical harmonics, and camera parameters. Based on these real data pairs, the parameters of the generative adversarial network (GAN), composed of the 3D Gaussian portrait model and the discriminator, are updated using a GAN loss function. Through adversarial training, the model is continuously optimized to generate higher-quality, relit portraits that conform to the relevant spherical harmonics and camera parameters. During training, since only one viewpoint is needed, the generator and compression layer simultaneously calculate the compressed Gaussian embedding for the viewpoint corresponding to the input camera parameters. In the real data pairs, the spherical harmonics estimated from the real portrait images can be obtained using the DPR illumination estimation method, and the camera parameters estimated from the real portrait images can be obtained using the Deep3DFaceRecon 3D face reconstruction method.

[0064] During training, the discriminator receives real data pairs and generated data pairs, while also using relevant camera parameters and spherical harmonic illumination coefficients as conditional inputs to the discriminator. This allows the discriminator to generate a supervisory signal based on the generative adversarial loss function, guiding the generator to learn the correct geometric and illumination priors.

[0065] Furthermore, without explicit supervision of the albedo attribute, ambiguity between the albedo attribute and the radiative transfer vector is unavoidable. Simple adversarial training leads to an unreasonable contribution ratio between the albedo and radiative transfer vectors to color attributes in real images, directly resulting in very poor quality albedo attributes. Therefore, a single-channel light assumption is imposed during model training. The incident light and radiative transfer vector are thus simplified to a single channel, i.e. This approach eliminates the contribution of the radiative transfer vector to the color distribution, resolving color ambiguity. Experiments showed that adversarial training averages out the influence of the actual color of the illumination on the albedo.

[0066] IV. Application Scenarios

[0067] Scenario 1: Generating arbitrary virtual characters based on random Gaussian noise.

[0068] According to an embodiment of the present invention, a method for generating a portrait based on Gaussian 3D splashing includes: acquiring randomly generated Gaussian noise of a target, spherical harmonic illumination coefficients for relighting the target person, and camera parameters; acquiring a pre-trained 3D Gaussian portrait model to generate geometric parameters for relighting the target person, the model including: a generator for extracting Gaussian features from the input Gaussian noise and camera parameters; an albedo decoder for extracting viewpoint-independent albedo attributes from the Gaussian features; a geometry decoder for generating color-deficient geometric parameters of the person based on the Gaussian features, which are a set of Gaussian spheres without color attributes; a radiative transfer decoder for modeling diffuse and specular reflection based on the Gaussian features and the camera parameters to obtain a viewpoint-dependent radiative transfer vector represented by a spherical harmonic function; a fusion module for multiplying the inner product of the radiative transfer vector and the spherical harmonic illumination coefficients with the albedo attributes to obtain a set of color attributes, and incorporating them into the color-deficient geometric parameters to obtain the geometric parameters for relighting the target person; and performing Gaussian splashing processing based on the geometric parameters for relighting the target person to obtain a rendered image of the target person. The Gaussian splatter processing can employ standard Gaussian rasterization techniques. During inference, the spherical harmonic illumination coefficient is a three-channel illumination coefficient, and the albedo attribute is a vector containing three elements. In the fusion module, it is configured to calculate the inner product of the spherical harmonic illumination coefficient and the three channels of the radiative transfer vector, and multiply the inner product of the three channels by the albedo to obtain the RGB three-channel color attribute.

[0069] During inference, the RGB values ​​required for relighting can be converted into spherical harmonics, and then input into the model to achieve relighting processing for any color. For example... Figure 2 As shown, this displays the rendering results of white spherical harmonic lighting. The first column shows the generated albedo rendering results (i.e., the result of rendering by incorporating only the albedo attribute as a color attribute into the missing color geometry parameters). Columns two through five show the rendering results corresponding to different white spherical harmonic lighting. The spherical harmonic lighting visualization is displayed in the lower left corner of the image. Spherical harmonic lighting of different intensities and directions can be rendered correctly. Or, as... Figure 3 As shown, this displays the results of color spherical harmonic lighting rendering. The first column shows the generated albedo rendering results, and the second to fifth columns show the rendering results from different viewpoints corresponding to different color spherical harmonic lighting. The spherical harmonic lighting visualization is displayed in the lower left corner of the image. Spherical harmonic lighting with different color distributions can be rendered correctly at different viewpoints.

[0070] Additionally, if a user wants to use the lighting effect from a reference image, they can estimate its spherical harmonic illumination coefficients and input the estimated spherical harmonic illumination coefficients into the model to achieve a similar lighting effect to the reference image.

[0071] Furthermore, the ambient light map input by the user can be converted into spherical harmonic lighting coefficients, and the converted spherical harmonic lighting coefficients can be input into the model to achieve the lighting effect corresponding to the ambient light map. For example... Figure 4 and Figure 5 The image shows the rendering results of the color ambient octane map. The first column displays the generated albedo rendering results, and the remaining columns show the rendering results of different color ambient octane maps from different viewpoints. The ambient octane map is displayed in the lower right corner of the image. Different ambient octane maps can be rendered correctly from different viewpoints.

[0072] Finally, the inventors made a simple comparison of the relighting effect with some other existing methods. Figure 6 The results show a comparison between the method of this invention and other methods in 3D portrait lighting rendering. As can be seen from the figures, existing methods can either only handle white light rendering or are limited in rendering quality and resolution, while the method of this invention can complete colored light rendering with higher quality at high resolution.

[0073] Scenario 2: Based on the scenario of generating a specified character

[0074] The difference between this scenario and scenario 1 is that the Gaussian noise of the target is not randomly generated, but is Gaussian noise extracted from a person's image, which contains the head structure features of that person.

[0075] According to one embodiment of the present invention, a method for generating human portraits based on Gaussian 3D splashing utilizes pivotal tuning technology (refer to Pivotal Tuning in GAN Inversion) to fine-tune a pre-trained 3D Gaussian portrait model, enabling it to generate geometric parameters for relighting a specified person at a relevant viewpoint based on Gaussian noise of a specified person in an image, thus obtaining a fine-tuned 3D Gaussian portrait model; acquiring Gaussian noise of the specified person, spherical harmonic illumination coefficients for relighting the specified person, and camera parameters; acquiring the pre-trained 3D Gaussian portrait model to generate geometric parameters for relighting the specified person, the model comprising: a generator for extracting Gaussian features of the specified person from the input Gaussian noise and camera parameters; and an albedo decoder for extracting albedo attributes of the specified person independent of viewpoint from the Gaussian features of the specified person. The system consists of: a geometry decoder for generating the missing color geometric parameters of a specified character based on Gaussian features, which are a set of Gaussian spheres without color attributes; a radiative transfer decoder for modeling diffuse and specular reflection based on the Gaussian features and camera parameters, obtaining a radiative transfer vector of the specified character that is related to the viewing angle and expressed in spherical harmonic functions; a fusion module for multiplying the inner product of the radiative transfer vector and the spherical harmonic illumination coefficient of the specified character with the albedo attribute to obtain a set of color attributes, and incorporating them into the missing color geometric parameters of the specified character to obtain the relighting geometric parameters of the specified character; and Gaussian splashing processing based on the relighting geometric parameters of the specified character to obtain the rendered image of the specified character.

[0076] In summary, some embodiments of the present invention can achieve at least one of the following beneficial effects:

[0077] 1) This invention achieves physically based lighting modeling of human figures using 3D Gaussian splashing and radiative transfer. Based on a generative 3D Gaussian human figure model, it binds albedo and radiative transfer vector attributes to a Gaussian sphere, and models the physically based lighting model using pre-calculated radiative transfer. Lighting is modeled using spherical harmonic functions, with one set of spherical harmonic coefficients representing incident light and another set representing radiative transfer. The lighting rendering equation is simplified to vector dot product, resulting in a linear response between outgoing and incident light. Therefore, while conforming to the physically based rendering equation, it inherits the high rendering speed of 3D Gaussian splashing, ensuring high-quality and efficient lighting rendering.

[0078] 2) This invention implements a view-dependent unified neural radiative transport framework. Radiative transport only supports modeling diffuse reflection terms, but through the derivation of the rendering equation, modeling the radiative transport vector as a function of the viewpoint allows for the modeling of view-dependent or specular reflection terms. Modeling the generation of the radiative transport vector as view-dependent forms a unified neural radiative transport framework, effectively modeling diffuse reflection and low-frequency specular reflection terms.

[0079] 3) This invention is based on the single-channel light assumption to decouple albedo from illumination. If illumination is simply used as the input for conditional generation, it will be difficult to decouple the color ambiguity between albedo and illumination. It is assumed that the illumination in the natural portrait image dataset is all single-channel light, so that the color is controlled only by the albedo, thus decoupling the color ambiguity.

[0080] 4) During training, this invention assumes that the incident light is a single-channel light to ensure the linear response capability of the physical lighting model to the incident light; during inference, RGB channel-by-channel rendering is performed to accurately render colored lighting.

[0081] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0082] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0083] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0084] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating human figures based on Gaussian 3D splashing, characterized in that, include: Acquire the Gaussian noise of the target, the spherical harmonic illumination coefficient for relighting the target figure, and the camera parameters; A pre-trained 3D Gaussian portrait model is obtained to generate the geometric parameters for relighting the target subject. This model includes: A generator is used to extract Gaussian features from the input Gaussian noise and camera parameters; Albedo decoder, used to extract viewpoint-independent albedo attributes from Gaussian features; The geometry decoder is used to generate the missing color geometric parameters of a person based on Gaussian features, which are a set of Gaussian spheres without color attributes; A radiative transfer decoder is used to model diffuse and specular reflections based on the Gaussian features and the camera parameters to obtain a view-dependent radiative transfer vector expressed in spherical harmonics. The fusion module is used to multiply the inner product of the radiative transfer vector and the spherical harmonic illumination coefficient with the albedo attribute to obtain a set of color attributes, and then incorporate them into the color-deficient geometric parameters to obtain the geometric parameters of the target character's relighting. Gaussian splashing is performed on the geometric parameters of the target person's relighting to obtain the rendered image of the target person.

2. The method according to claim 1, characterized in that, During inference, the spherical harmonic illumination coefficient is a three-channel illumination coefficient, and the albedo attribute is a vector containing three elements. In the fusion module, it is configured to calculate the inner product of the spherical harmonic illumination coefficient and the three channels of the radiative transfer vector, and multiply the inner product of the three channels by the albedo to obtain the RGB three-channel color attribute.

3. The method according to claim 2, characterized in that, When training a 3D Gaussian portrait model, the fusion module is configured as follows: Receive the single-channel illumination coefficient and calculate the single-channel inner product of the single-channel illumination coefficient and the radiative transfer vector. The inner product of a single channel is copied twice to form an isomorphic three-channel inner product. The isomorphic three-channel inner product is then multiplied by the albedo to obtain the color attributes of the three channels during training.

4. The method according to claim 3, characterized in that, The ways to obtain color attributes include: in, Represents a Gaussian sphere in a set of Gaussian spheres Color attributes, Indicates the use of albedo decoder Compressed Gaussian embedding based on Gaussian features Extracted albedo attribute, This represents the order of the spherical harmonic function of illumination. The spherical harmonic coefficient of illumination represents the first... Individual harmonic coefficients, Indicates the use of a radiated transmission decoder Based on compressed Gaussian embedding and perspective encoding Extracting the first radiative transfer vector Each element.

5. The method according to any one of claims 1-4, characterized in that, The radiative transfer decoder includes: The compression layer is used to linearly compress Gaussian features to obtain compressed Gaussian embeddings; Viewpoint coding layer, used to extract viewpoint codes from camera parameters; A radiative transfer decoding layer is used to extract radiative transfer vectors based on compressed Gaussian embedding and viewpoint coding; During training, a generator and a compression layer are used to compute compressed Gaussian embeddings of the viewpoints corresponding to the input camera parameters on the fly. During inference, a generator and a compression layer are used to pre-compute and cache compressed Gaussian embeddings for multiple viewpoints. When the user adjusts the viewpoint, the compressed Gaussian embedding for the corresponding viewpoint is obtained based on the cached compressed Gaussian embedding and input into the radiative transfer decoding layer.

6. The method according to any one of claims 1-4, characterized in that, The pre-trained 3D Gaussian portrait model was obtained after multiple iterations of training in the following manner: We obtain multiple random Gaussian noises, spherical harmonic illumination coefficients used for training, and camera parameters, and use a 3D Gaussian portrait model to obtain the geometric parameters of the relighting of the person corresponding to the random Gaussian noise. Gaussian splashing is performed based on the geometric parameters of the character's relighting corresponding to random Gaussian noise to obtain the rendered image of the generated character. Obtain generated data pairs and real data pairs. Each data pair includes a sample and a label. The sample of the generated data pair includes: the rendered image of the person generated in this training, the spherical harmonic illumination coefficient and camera parameters used in the training, and its label indicates that the sample is fake. The sample of the real data pair includes a real portrait image of a person, the spherical harmonic illumination coefficient and camera parameters estimated based on the image, and its label indicates that the sample is real. Based on generated data pairs and real data pairs, the parameters of the 3D Gaussian portrait model and the discriminator are updated using a generative adversarial network and a generative adversarial loss function. Through adversarial training, the model is continuously trained and optimized, enabling it to generate higher-quality relit portraits that conform to relevant spherical harmonic illumination coefficients and camera parameters.

7. The method according to claim 6, characterized in that, Pivot fine-tuning technique is used to fine-tune a pre-trained 3D Gaussian portrait model so that it can generate the geometric parameters of the re-illumination of a specified person from a relevant viewpoint based on the Gaussian noise of a specified person in an image.

8. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1-7.

10. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1-7 by executing the executable instructions.

Images