High-fidelity three-dimensional digital clothing re-illumination method and device
By improving the tensor radiation field model and introducing gradient guided smoothing terms, combined with the multi-factor incident light field model, the texture distortion and light distortion problems in three-dimensional digital clothing re-illumination are solved, and the high-fidelity clothing re-illumination effect is achieved.
Patent Information
- Application Number
- CN202411220175.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-12-13
AI Technical Summary
The existing three-dimensional digital clothing re-illumination methods have problems in texture extraction and lighting model adaptation, resulting in texture distortion and lighting distortion, making it difficult to achieve high-fidelity re-illumination on flexible objects such as clothing.
The tensor radiation field model was improved to build a feature space that is more suitable for digital clothing, and a gradient-guided smoothing term was introduced. Combined with the multi-factor incident light field model and the BRDF parameter estimation network, the texture characteristics and lighting information were accurately extracted, and the BRDF parameter regression process was optimized. .
Improve the ability to extract texture features, reduce texture noise, render re-illumination results that are closer to reality, forming a high-fidelity three-dimensional digital clothing material library.
Smart Images
Figure CN119131224A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to computer graphics and computer vision technology, in particular to the fields of digital fashion, digital entertainment, virtual reality, and more particularly to a high-fidelity three-dimensional digital clothing relighting method and device, which can efficiently and automatically generate high-fidelity three-dimensional digital clothing relighting images and generate a clothing material library under different ambient lights. Background Art
[0002] Clothing, as the core carrier of fashion elements, not only reflects fashion trends, but also demonstrates people's inner desire for personality and self-expression. With the improvement of the quality of life, people's requirements for the beauty of clothing are becoming more and more refined. For example, before trying on clothes, people often need to understand the appearance of clothes in various occasions and lighting conditions, and be able to examine them from different perspectives in order to make a more comprehensive evaluation. Therefore, how to allow users to conveniently and realistically preview the appearance of clothing under various lighting conditions, that is, the re-lighting effect display of digital clothing, is of great significance.
[0003] Traditional image relighting methods mainly focus on the processing of two-dimensional images. However, these traditional methods often fail to achieve the desired effect when dealing with multi-view relighting tasks because the input two-dimensional images lack three-dimensional information. To overcome this limitation, it is particularly important to introduce a network model containing three-dimensional information. The Neural Radiance Fields technology meets this demand better. It uses a series of target object or scene images taken by a synchronized camera to construct a radiation field, and uses multi-layer perceptron (MLP), voxel grid, multi-tensor field and other methods to represent the target object or scene, and regards the rigid body as a mutually occluded light source in space (i.e., particles that emit and absorb light), thereby optimizing the entire space. In addition, Tensorial Radiance Fields, as a radiation field model based on multi-tensor field representation, decomposes high-dimensional tensors into multiple low-dimensional tensors through a tensor decomposition algorithm, which not only significantly improves the convergence speed of the model, but also can more accurately decode the texture, depth, normal vector and other features of the target object in the scene feature space it constructs.
[0004] In recent years, many studies have taken the neural radiance field model as the cornerstone, combined with advanced neural networks and illumination models, to achieve the task of multi-view relighting of simple scenes. Nrf incorporates the reflectivity equation into the Nerf model, abandons the assumption that objects are luminous particles, regards them as particles with reflective properties, and considers that they are illuminated by external light sources. Nerv has made further improvements on the basis of Nrf, breaking the flash assumption and optimizing the modeling of environmental lighting: it considers single-bounce indirect lighting and introduces the concept of visibility to represent the proportion of energy particles that can reflect light, thereby improving the flexibility of the reflectivity equation. NeRFactor adopts a two-stage strategy to decouple geometric modeling and color rendering in Nerf: first, Nerf is used to reconstruct the geometry of the target object or scene, and then multiple multi-layer perceptrons are used to regress the BRDF results and relighting results. In addition, methods such as Physg and InvRender further utilize the spherical Gaussian function, a spherical approximation method that is closer to the real physical world. By mapping the environment map data to a sphere, these methods better capture the lighting and reflection characteristics of the environment.
[0005] However, current research is mainly applied to large scenes without boundaries or some rigid objects with high-gloss curved surfaces, such as skulls, glass balls, etc. When it is applied to three-dimensional digital clothing data, it often faces some problems. The first is the texture noise problem. Since the surface of clothing usually has complex textures, the model constructed using Nerf as the basis of feature space often cannot accurately extract the texture information of clothing from the image, resulting in the loss of texture in the relighting result. The second is the relighting distortion problem. As a flexible object, clothing has differences in the adaptation of the lighting model compared to rigid objects. It is difficult to directly use multiple spherical Gaussians or neural networks to establish an accurate lighting model, so the existing light field model easily leads to the distortion of the light brightness of the clothing relighting result. Summary of the invention
[0006] In view of the deficiencies of the prior art, the present invention proposes a digital clothing relighting method based on feature space. In view of the problem of texture distortion in the relighting result, the present invention improves the tensor radiation field model and uses it to construct the feature space of digital clothing. This not only improves the ability of the feature space to obtain texture features from the image, but also effectively reduces texture noise. At the same time, in order to solve the problem of relighting distortion, the present invention constructs an incident light field that is more suitable for the digital clothing model. In addition, during the training process of the BRDF model, the present invention uses a gradient-guided smoothing term to regress more accurate digital clothing surface roughness and albedo. This enables the model of the present invention to render relighting results that are closer to real samples, and can efficiently form a digital clothing material library under different ambient lights.
[0007] The digital clothing relighting method constructed by the method calculates and outputs through the reflectivity rendering equation, and its input consists of three core parts: digital clothing surface geometry, ambient lighting information and digital clothing material characteristics. The calculation model of each part is summarized as follows: First, it is necessary to extract the geometric information of the digital clothing surface, which can be obtained by reconstructing the normal of the digital clothing surface. To this end, the present invention inputs the feature data of the digital clothing surface into the normal decoding network to accurately extract the normal field of the digital clothing surface. Secondly, in order to simulate the real ambient lighting, the present invention constructs a multi-factor incident light field model, which integrates the spherical Gaussian function, indirect lighting, and the reception rate of direct light and indirect light, so that the complex lighting conditions in the real environment can be highly realistically reproduced. Finally, in order to characterize the material characteristics of the digital clothing, the present invention estimates the parameters of the bidirectional reflectance distribution function and embeds a parameter decoding network, which can accurately extract key BRDF parameters such as albedo and roughness from the surface characteristics of the digital clothing. In addition, the present invention also introduces a gradient-guided smoothing term to further constrain and optimize the regression process of the BRDF parameters.
[0008] The objective of the present invention is achieved through the following technical solution: A high-fidelity three-dimensional digital clothing relighting method provided by the present invention comprises the following steps:
[0009] Step 1: Acquire and construct a multi-view image dataset of real clothing, and shoot HDRi panoramic environment maps in the corresponding environment: shoot real clothing from multiple perspectives in different environments or shoot videos around the clothing, extract multi-view images from them, and then use COLMAP software to calculate the corresponding camera pose of the clothing image.
[0010] Step 2: Construct a three-dimensional digital clothing feature space based on a single tensor radiation field model. Use the single tensor radiation field model to train the multi-view images in the above steps, and extract the features of the digital clothing based on the feature space as the input of the subsequent network. Extract the grid data and the normal field data under the corresponding view from the radiation field;
[0011] Step three, construct a normal prediction network based on a deep neural network, and perform regression training on the digital clothing features based on the normal field data in step two to obtain the normal geometry information of the digital clothing model.
[0012] Step 4: Construct a multi-factor incident light field model: According to the characteristics of human eyes capturing ambient light in reality, the incident light field model is divided into several relatively independent modules, namely the direct light of the environment, which is modeled by multiple spherical Gaussian functions; the indirect reflected light of digital clothing, which is modeled by a deep neural network; further, the opacity formula and multiple spherical Gaussian functions are used to fit the reception rate of direct light and indirect light, thereby obtaining the lighting information of digital clothing.
[0013] Step 5: Construct a BRDF parameter estimation network, and use a multi-layer fully connected neural network combined with gradient-guided terms to constrain the material parameters of each digital clothing to obtain the material information of the digital clothing.
[0014] Step six, combine the normal geometry information, lighting information, and material information in steps three, four, and five with the reflectivity equation to render the re-illuminated image of the digital clothing, and use the spherical Gaussian function that fits different environment maps to generate high-fidelity three-dimensional digital clothing images under different ambient lighting.
[0015] Furthermore, in step one, the multi-view clothing images and the re-illuminated real sample images are both 512*512 pixels, RGB three-channel color images, and the environment map can be 2k or 4k resolution. COLMAP is used to reconstruct the camera pose from the image set. If it is video data, the image needs to be obtained by averaging frames.
[0016] Further, in step 2, the tensor field can be interpreted as a four-dimensional tensor, the first three dimensions store the three-dimensional coordinates corresponding to the three-dimensional model, and the fourth dimension stores the features of the digital clothing under the corresponding three-dimensional coordinates. The present invention fuses the volume density, color and BRDF parameter features of the digital clothing model into hybrid features, and uniformly stores them in a single tensor field. The convergence speed of the model is accelerated, and the overall quality of the relighting result is also improved. Finally, the data set constructed in step 1 is used for training to obtain a tensor radiation field model storing the three-dimensional features of the digital clothing. The three-dimensional mesh model of the corresponding digital clothing can be obtained by mesh extraction using the volume density values stored therein. The normal field under different viewing angles can be extracted using Blender software, and the normal field stores the vector of each point on a given surface. In the normal field, the color value of each pixel corresponds to the normal direction of the surface at that point. Specifically, the normal field uses the RGB color space to represent the components of the normal on the spatial coordinate axis.
[0017] Furthermore, in step 2, tensor decomposition technology is applied to accelerate model convergence, and the characteristic formula of digital clothing at a certain point can be expressed as:
[0018]
[0019] Among them, R is the set tensor decomposition amount. Represents the weight vector and matrix factor of the feature component on the rth corresponding X, Y and Z coordinate axes respectively. The digital clothing model is at the surface point The feature vector at is the sum of the R component features. The present invention defines the first element of the feature vector as the volume density feature value of the surface point, recorded as F[0], and the remaining elements constitute the appearance feature vector of the digital clothing, recorded as: F(1,l), where l represents the length of the digital clothing surface feature vector.
[0020] Furthermore, in step 3, the digital clothing appearance feature vector F(1, l) mentioned in step 2 is input into the normal decoding network (NMLP) for decoding, so as to obtain the normal vector of the digital clothing surface. This process can be expressed by the following formula:
[0021] n=NMLP(F(1,l))
[0022] Furthermore, in step 4, based on the single tensor radiation field described in step 2, a multi-factor incident light field model is constructed, which integrates three key factors: first, the direct light L of the environment in which the digital clothing is located. d (ω i ); secondly, the light indirectly reflected from other surface points of the digital clothing to this point Finally, the point's reception rate for direct light and indirect light N spherical Gaussian functions are used to fit the direct light in the environment, and the fitting result is recorded as:
[0023]
[0024] where ω i Represents the light angle sampled from the environment map according to the viewing direction input by the user, and the parameters of the spherical Gaussian function include the amplitude ξ k , standard deviation λ k and the center position μ k The digital clothing appearance feature vector described in step 2 is input into the indirect light decoding network (SCMLP), and the digital clothing surface base color obtained by decoding approximates the indirect reflected light of the digital clothing surface. This process can be expressed by the following formula:
[0025]
[0026] The final incident light collection formula can be written as follows:
[0027]
[0028] in The opacity calculation result proposed in Nerf is used to approximate the direct light reception rate, T ind (ω i ) represents the reception rate of indirect light, which is approximated by multiple spherical Gaussian functions.
[0029] Furthermore, in step 5, a BRDF formula is constructed and the intensity of the reflected light is calculated according to the incident light set constructed in step 4. The formula is recorded as:
[0030]
[0031] It includes:
[0032] The normal distribution term D is represented by a spherical Gaussian function, denoted as SG(h; n, λ, μ); where h = (ω0+ω i ) / ||ω0+ω i || represents the half-angle vector of the viewing direction and the incident light direction. n is the normal vector of the geometric surface of the digital clothing. In addition, λ and μ are both trainable parameters.
[0033] The Fresnel term Fr is expressed as follows:
[0034] Fr(ω0,h,s)=Fr0+(1-Fr0)(1-(ω0·h) 5 )
[0035] Fr0=((1-s) / (1+s)) 2
[0036] Where s represents the albedo value extracted from the feature space.
[0037] The geometric terms are approximated by the GGX function:
[0038] G(ω i ,ω0,n,R n )=GGX(ω i ·n)GGX(ω0·n)
[0039]
[0040] Where R n Represents the roughness value extracted from the feature space.
[0041] The input of this formula includes four parameters ω0, ω i ,s,R n , respectively represent the reflection angle of light, the incident angle, the albedo and roughness of the digital clothing surface point. Among them, the albedo and roughness are obtained by inputting the digital clothing surface features mentioned in step 2 into the BRDF parameter decoding network (BMLP). This process is recorded as:
[0042] [s,Rn]=BMLP(F(1,l))
[0043] The output of BMLP contains 4 channels, of which the first three channels represent albedo and the last channel represents roughness. Furthermore, step 5 includes a gradient-guided smoothing strategy, which combines the gradient of the albedo and roughness of each pixel of the digital clothing image with the pixel value gradient to guide the albedo and roughness of the generated image. The specific formula is as follows:
[0044]
[0045] Where P n Indicates that the image I is at a certain viewing angle d The number of pixels sampled on and Represents the pixel coordinates X P The gradient of albedo and roughness at . These gradient information can be obtained by back propagation algorithm. Represents the gradient of the image grayscale value.
[0046] Furthermore, in step 6, the normal geometry information in step 3, the illumination information in step 4, and the material information in step 5 are combined and applied to the reflectivity equation to render the digital clothing color value with ambient light attributes. The relevant formula is as follows:
[0047]
[0048] Where X represents the three-dimensional coordinate vector of the surface point of the digital clothing model, ω0 represents the viewing direction, ω i Indicates the light angle, S L represents the incident light set, L num represents the amount of incident light. This formula converts the digital clothing at point The color at ω0 is regarded as the sum of the components of the reflected light from multiple incident lights on the micro plane at the point under the viewing angle ω0. By combining the normal information n and the ambient light information L mentioned in the previous part of the present invention, tol And surface material information b f , the color of the point can be calculated. By minimizing the loss between the calculated value and the actual clothing surface point color, the feature space and decoding network can be optimized. Thus, the training of the entire relighting model is completed.
[0049] In a second aspect, the present invention also provides a high-fidelity three-dimensional digital clothing re-lighting device, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the high-fidelity three-dimensional digital clothing re-lighting method is implemented.
[0050] In a third aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the high-fidelity three-dimensional digital clothing re-lighting method is implemented.
[0051] The beneficial effects of the present invention are as follows: a digital clothing relighting method based on feature space is proposed. In view of the problem of texture distortion in the relighting result, the present invention improves the tensor radiation field model and makes full use of it to construct the feature space of digital clothing. This improves the ability of the feature space to obtain texture features from the image and effectively reduces texture noise. At the same time, in order to solve the relighting distortion problem, the present invention constructs an incident light field that is more suitable for the digital clothing model. In addition, in the training process of the BRDF model, the present invention introduces a gradient-guided smoothing term, so that a more accurate surface roughness and albedo of the digital clothing can be regressed. This enables the model of the present invention to render a relighting result that is closer to the real sample. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 It is an overall schematic diagram of the method of the present invention.
[0053] Figure 2 Implementing a flow chart for digital garment relighting models.
[0054] Figure 3 This is an example of Gaussian fitting of ambient lighting. (a) is a real ambient lighting sample, and (b) is an example of SG fitting ambient map.
[0055] Figure 4 Schematic diagram of the incident light field model.
[0056] Figure 5 Example results for relighting a digital garment.
[0057] Figure 6 A structural diagram of a high-fidelity three-dimensional digital clothing re-lighting device provided by the present invention DETAILED DESCRIPTION
[0058] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0059] like Figure 1 and Figure 2 As shown, the present invention provides a high-fidelity three-dimensional digital clothing re-lighting method, and the specific implementation process is as follows:
[0060] (1) Create a training dataset
[0061] The camera is used to take surround shots of clothing, and pictures of clothing with no ambient light and multiple perspectives and pictures of clothing under different ambient lighting are taken. At the same time, corresponding panoramas are taken. Each clothing picture is compressed to 512*512 pixels, RGB three-channel color images, and the environmental map can be 2k or 4k resolution. The COLORMAP software is used to reconstruct the camera posture from the image set. If it is video data, the image needs to be obtained by averaging frames. In this implementation case, 300 sets of pictures under different perspectives are taken and extracted. The first 200 sets are used as test sets to test the effect after the model training is completed, and the last 100 sets are used as training sets to train the model of the present invention.
[0062] (2) Build and train the relighting network model
[0063] The relighting network model is composed of a single tensor radiation field, a normal prediction network, an incident light field module, a BRDF module, and a reflectivity equation. The tensor field can be interpreted as a four-dimensional tensor, the first three dimensions store the three-dimensional coordinates corresponding to the three-dimensional model, and the fourth dimension stores the features of the digital clothing under the corresponding three-dimensional coordinates. The present invention fuses the volume density, color and BRDF parameter features of the digital clothing model into hybrid features and stores them uniformly in a single tensor field. The convergence speed of the model is accelerated, and the overall quality of the relighting result is also improved. Finally, the data set constructed in step 1 is used for training to obtain a tensor radiation field model storing the three-dimensional features of the digital clothing. The three-dimensional mesh model of the corresponding digital clothing can be obtained by mesh extraction using the volume density value stored therein. The normal field under different viewing angles can be extracted using Blender software, and the normal field stores the vector of each point on a given surface. In the normal field, the color value of each pixel corresponds to the normal direction of the surface at that point. Specifically, the normal field uses the RGB color space to represent the components of the normal on the spatial coordinate axis.
[0064] The single tensor radiation field contains a decomposable tensor. Suppose the coordinates of a surface point of the digital clothing model are Then the digital clothing surface feature F at this point can be specifically expressed as:
[0065]
[0066] Among them, 16 is the tensor decomposition amount set in the example. Represents the weight vector and matrix factor of the feature component on the rth corresponding X, Y and Z coordinate axes respectively. The digital clothing model is at the surface point The feature vector at is the sum of the 48 component features. For more specific VM decomposition algorithm related content, please refer to the article TensoRF (AC; ZX). Regarding the training of the radiation field, rays are emitted according to the input camera perspective, and multiple sampling points are sampled on the rays. The rays are converged into a pixel color value according to the color of the sampling point and the volume density value. The color value is a pixel point of the image at the observation perspective. Therefore, the radiation field contains a 4*128 fully connected network CMLP, whose input is the feature F(1,28) extracted from the four-dimensional tensor digital clothing feature space, combined with the volume density value F[0] extracted from the feature space. The color value of the corresponding surface point is calculated by inputting the volume rendering formula. The volume rendering formula is as follows:
[0067]
[0068] Where 32 represents the number of sampling points on the ray extending from the surface point, σ i ,δ i Respectively represent the volume density and the sampling point spacing, T i Indicates the cumulative transmittance of light. The calculated color value is compared with the actual sample color value for loss calculation. Finally, back-propagation training regresses the digital clothing feature space and the CMLP network.
[0069] After training the single tensor radiation field, the three-dimensional mesh of the digital clothing can be extracted using SDF sampling according to the volume density value in the radiation field. Then, the normal field data of the digital clothing under different perspectives can be rendered using Blender for subsequent training of the normal network.
[0070] The present invention defines the first element of the feature vector as the volume density feature value of the surface point, recorded as F[0], and the remaining elements constitute the appearance feature vector of the digital clothing, recorded as: F(1,l). The normal prediction network is a 4*128 fully connected network NMLP. The features extracted from the four-dimensional tensor digital clothing feature space are input into the network, and the normal field under the corresponding viewing angle can be predicted: n=NMLP(F(1,l)). In the normal field, the color value of each pixel corresponds to the normal direction of the surface at that point. Specifically, the normal field uses the RGB color space to represent the components of the normal on the spatial coordinate axis. This value will finally be input into the reflectivity equation for calculation, and the equation will also be used to back-propagate and train the regression network.
[0071] like Figure 3 and Figure 4 As shown in the figure, the incident light field is jointly modeled by the direct light of the environment, the indirect reflected light, the direct light reception rate and the reflected light reception rate, which integrates three key factors: the first is the direct light L of the environment where the digital clothing is located. d (ω i); secondly, the light indirectly reflected from other surface points of the digital clothing to this point Finally, the point's reception rate for direct light and indirect light The direct light is calculated by summing 128 spherical Gaussian functions and the formula is as follows:
[0072]
[0073] where ω i Indicates the light angle of the light sampled from the environment map according to the user input viewing angle parameter. You can choose uniform sampling or importance sampling strategy to sample the corresponding light from the environment map. For more specific implementation methods, please refer to the article PhySG (KZ; FL). Amplitude ξ k , standard deviation λ k and the center position μ k There are three trainable parameters. Based on the user input, the color value of direct light at the corresponding viewing angle and the angle of direct light can be calculated and inferred. The indirect reflected light prediction network is a 4*128 fully connected network SCMLP. The features extracted from the four-dimensional tensor digital clothing feature space are input into the network to predict the color value of indirect reflected light at the corresponding viewing angle. The direct light reception rate and the reflected light reception rate are two weight values. The former is calculated by the opacity part number in the volume density function, that is, The latter is fitted by 8 spherical Gaussian functions, namely T ind (ω i )=SG(ω i ξ j ,λ j ,μ j ). Finally, the incident light field module is integrated with these items. This module can output the incident light set under the corresponding perspective according to the user input perspective. This set will eventually output the reflectivity equation. Finally, the various parameters are trained by back propagation regression based on the reflectivity equation result. This process can be expressed by the following formula:
[0074]
[0075] The final incident light collection formula can be written as follows:
[0076]
[0077] in The opacity calculation result proposed in Nerf is used to approximate the direct light reception rate, T ind (ω i ) represents the reception rate of indirect light, which is approximated by multiple spherical Gaussian functions.
[0078] The BRDF module contains a parameter decoding network, which is a 4*128 fully connected network BMLP. The features extracted from the four-dimensional tensor digital clothing feature space are input into the network, and the albedo parameter s and roughness parameter R of the digital clothing under the corresponding viewing angle can be predicted. n , these two parameters are input into the BRDF model, which includes the normal distribution term D, the Fresnel term Fr, and the geometric term G. The specific formula is as follows:
[0079]
[0080] D=SG(h;n,λ,μ),h=(ω0+ω i ) / ||ω0+ω i ||
[0081] Fr(ω0,h,s)=Fr0+(1-Fr0)(1-(ω0·h) 5 ),Fr0=((1-s) / (1+s)) 2
[0082]
[0083] Where ω0 represents the user input viewing angle, ω i represents the light angle; h represents the half-angle vector between the viewing direction and the incident light direction; and n is the normal vector of the geometric surface of the digital clothing. In addition, λ and μ are both trainable parameters; s represents the albedo value extracted from the feature space, R n Represents the roughness value extracted from the feature space. Among them, albedo and roughness are obtained by inputting the digital clothing surface features into the BRDF parameter decoding network (BMLP). This process is recorded as:
[0084] [s,Rn]=BMLP(F(1,l))
[0085] The output of BMLP contains 4 channels, of which the first three channels represent albedo, and the last channel represents roughness. Furthermore, based on a gradient-guided smoothing strategy, the gradients of the albedo and roughness of each pixel of the digital clothing image are combined with the pixel value gradient to guide the albedo and roughness of the generated image. The specific formula is as follows:
[0086]
[0087] Where P n Indicates that the image I is at a certain viewing angle d The number of pixels sampled on and Represents the pixel coordinates X PThe gradient of albedo and roughness at . These gradient information can be obtained by back propagation algorithm. d (X P ) represents the gradient of the image grayscale value.
[0088] The BRDF model can calculate the intensity of reflected light based on the symmetry of the incident light angle and the reflected light angle about the normal line and the reflected light angle, which is the viewing angle. Finally, it will combine the incident light set and calculate the digital clothing color value containing ambient light information through the reflectivity equation. The reflectivity equation formula is as follows:
[0089]
[0090] in represents the three-dimensional coordinate vector of the surface point of the digital clothing model, ω0 represents the viewing direction, S L represents the incident light set, L num represents the amount of incident light. This formula converts the clothing at point The color at is regarded as the sum of the components of the reflected light from multiple incident lights on the microplane at the point ω0. The result rendered by this formula can be used to calculate the loss with the images in the training set, and the complete network model can be trained by back propagation regression. The feature space of the digital clothing model is jointly constrained and optimized by the reflectivity equation rendering loss, volume rendering loss, normal loss, and gradient smoothing loss. Therefore, the total loss function can be expressed as:
[0091]
[0092] in, Represents the loss between the rendering result of the reflectivity equation and the real pixel value of the corresponding point, Represents the loss between the volume rendering result and the true pixel value of the corresponding point, Represents the loss between the normal prediction result and the true normal value of the corresponding point, l s Represents the result of the gradient-guided smoothing term. λ1~λ4 are custom loss weights. Before a certain number of iterations is reached (that is, before the scene surface is obtained), the value of λ1 is set to zero. In this example, when the number of iterations reaches the specified value, λ1 is set to 0.005, λ2 is set to 1, λ3 is set to 0.05, and λ4 is set to 1. The batch_size is set to 4096 and the network is trained in batches each time. After 80,000 iterations, the total loss of the model is close to convergence, and a trained relighting model is obtained.
[0093] (3) Relight the image using the trained relighting model.
[0094] like Figure 5As shown, the environment image can be input into the model. When the specified viewing angle is input, the model can render the digital clothing re-lighting image corresponding to the environment lighting and the corresponding viewing angle.
[0095] Corresponding to the aforementioned embodiment of a high-fidelity three-dimensional digital clothing re-lighting method, the present invention also provides an embodiment of a high-fidelity three-dimensional digital clothing re-lighting device.
[0096] See also Figure 6 A high-fidelity three-dimensional digital clothing re-lighting device provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a high-fidelity three-dimensional digital clothing re-lighting method in the above embodiment.
[0097] The embodiment of a high-fidelity three-dimensional digital clothing re-lighting device provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 6 As shown, it is a hardware structure diagram of a high-fidelity three-dimensional digital clothing re-lighting device provided by the present invention, in which any device with data processing capability is located, except Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.
[0098] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0099] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.
[0100] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a high-fidelity three-dimensional digital clothing re-lighting method in the above embodiment is implemented.
[0101] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.
[0102] The present invention also provides a computer program product, comprising a computer program, and when the computer program is executed by a processor, the high-fidelity three-dimensional digital clothing re-lighting method is implemented.
[0103] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Although the present invention has been illustrated and described by referring to certain preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and detail without departing from the spirit and scope of the present invention. Any modifications and changes made to the present invention fall within the scope of protection of the present invention.
Claims
1. A high-fidelity three-dimensional digital clothing relighting method, characterized in that: The following steps are involved: Step 1: Obtain and construct a multi-view image dataset of real clothing, and shoot an HDRi panoramic environment map in the corresponding environment; Step 2: construct a three-dimensional clothing feature space based on a single tensor radiation field model; The single tensor radiation field model is used to train the multi-view images in step 1, and the features of the digital clothing are extracted based on the feature space as the input of the subsequent network; the grid data and the normal field data under the corresponding view are extracted from the radiation field; Step 3: construct a normal prediction network based on a deep neural network, and perform regression training on the digital clothing features based on the normal field data in step 2, so as to obtain the normal geometry information of the digital clothing model; Step 4: construct a multi-factor incident light field model; use multiple spherical Gaussian functions to model the direct light of the environment, and use a deep neural network to model the indirect light indirectly reflected by the digital clothing; further use the opacity formula and multiple spherical Gaussian functions to fit the reception rate of direct light and indirect light, so as to obtain the lighting information of the digital clothing; Step 5: construct a BRDF parameter estimation network; use a multi-layer fully connected neural network combined with a gradient-guided term to constrain the material parameters of each digital clothing, thereby obtaining the material information of the digital clothing; Step six, combine the normals in steps three, four, and five with the geometric information, lighting information, and material information and the reflectivity equation to render the re-illuminated image of the digital clothing, and use the spherical Gaussian function that fits different environment maps to generate high-fidelity three-dimensional digital clothing images under different ambient lighting.
2. The high-fidelity three-dimensional digital clothing relighting method according to claim 1, characterized in that: In step 1, multi-view shooting of real clothing in different environments or shooting of videos around the clothing is performed, and multi-view images are extracted from them. Then, the COLMAP software is used to calculate the corresponding camera pose of the clothing image.
3. The high-fidelity three-dimensional digital clothing relighting method according to claim 1, characterized in that: In step 2, the normal field data is extracted using Blender software. The normal field stores the vector perpendicular to each point on a given surface. In the normal field, the color value of each pixel corresponds to the normal direction of the surface at that point. Specifically, the normal field uses the RGB color space to represent the components of the normal on the spatial coordinate axis.
4. The high-fidelity three-dimensional digital clothing relighting method according to claim 1, characterized in that: In the step 2, the tensor field can be interpreted as a four-dimensional tensor, the first three dimensions store the three-dimensional coordinates corresponding to the three-dimensional model, and the fourth dimension stores the mixed features of the clothing under the three-dimensional coordinates corresponding to the clothing. The mixed features are the body density, color and BRDF parameters of the clothing model obtained by feature fusion through the same tensor field mixed training.
5. The high-fidelity three-dimensional digital clothing relighting method according to claim 1, characterized in that: In the step 3, the clothing feature vector F(1, l) is input into the normal decoding network NMLP for decoding, so as to obtain the normal vector of the clothing surface, that is, the normal geometric information. This process is expressed by the following formula: n=NMLP(F(1,l)) Where F(1,l) represents the feature vector of digital clothing, and l represents the length of the feature vector of digital clothing.
6. The high-fidelity three-dimensional digital clothing relighting method according to claim 1, characterized in that: In step 4, a multi-factor incident light field model is constructed, which integrates three key factors: first, the direct light L of the environment in which the digital clothing is located. d (ωi); followed by indirect reflection from digital clothing Finally, the point's reception rate for direct light and indirect light Tind(ω i ); N spherical Gaussian functions are used to fit the direct light in the environment, and the fitting result is recorded as: where ω i represents the viewing direction input by the user, and the parameters of the spherical Gaussian function include the amplitude ξ k , standard deviation λ k and the center position μ k ; The digital clothing feature vector F(1, l) described in step 3 is input into the indirect light decoding network SCMLP. The digital clothing surface base color obtained by decoding approximates the indirect reflected light of the digital clothing surface. This process is expressed by the following formula: The final incident light collection formula can be written as follows: T ind (ω)=SG(ω i ,x j ,l j ,m j ) in Represents the opacity calculation result, which is used to approximate the direct light reception rate, T ind (ω i ) represents the reception rate of indirect light, which is approximated by the sum of multiple spherical Gaussians.
7. The high-fidelity three-dimensional digital clothing relighting method according to claim 1, characterized in that: In step 5, a BRDF parameter decoding network BMLP is used to decode and obtain the four parameters of the BRDF formula ω0, ω1, s, R n , which represent the reflection angle of light, the incident angle, the albedo and roughness of the surface points of the digital clothing respectively. This process is recorded as: [s,Rn]=BMLP(F(1,l)) The output of BMLP contains 4 channels, the first three channels represent the albedo s(albedo), and the last channel represents the roughness Rn(roughness).
8. The high-fidelity three-dimensional digital clothing relighting method according to claim 1, characterized in that: The step 5 includes a gradient-guided smoothing strategy, which combines the gradient of the albedo and roughness of each pixel of the clothing image with the normal gradient to guide the albedo and roughness of the generated image. The specific formula is as follows: Where P n Indicates that the image I is at a certain viewing angle d The number of pixels sampled on the image, P represents the pixel subscript number, and and Represents the pixel coordinates X P The gradients of albedo and roughness at the location are obtained by back propagation. P ) represents the gradient of the normal value, which is obtained directly from the normal field.
9. A high-fidelity three-dimensional digital clothing re-lighting device, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a high-fidelity three-dimensional digital clothing relighting method as described in any one of claims 1-8 is implemented.
10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a high-fidelity three-dimensional digital clothing re-lighting method as described in any one of claims 1-8 is implemented.