Face material reconstruction method based on multiple illumination images and micro-renderable

Abstract

The invention discloses a face material reconstruction method based on multiple illumination images and capable of micro-rendering, and the method comprises the steps: firstly calculating an optical flow based on a full-bright sequence in the shot multiple illumination images, carrying out the alignment of the faces of the whole shooting sequence, and then carrying out the diffuse reflection and highlight separation of the shot face images, thereby achieving the reconstruction of the face material. The method comprises the following steps of: separating a human face, optimizing a material parameter of the human face according to a separation result to obtain a high-quality PBR material parameter, and finally splicing multi-view solving results to obtain a final material map. And a multi-layer pyramid calculation method is adopted, and a result of optimization on low-resolution data is used as an initial value of optimization on high-resolution data, so that the optimization process is accelerated. In addition, solution of material parameters is accelerated through a method of sampling light sources in small batches during optimization. According to the method, the used multi-illumination image is not limited by polarization and a far-field light source, the solving process is accelerated, the face texture map can be solved conveniently and quickly, and the method has high practical value.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics, and in particular relates to a method for reconstructing human face materials based on multi-light images and differentiable rendering. Background Technology

[0002] With the development of computer graphics technology, the construction of high-fidelity digital humans to provide a medium for emotional expression in the virtual world is a technology with huge demand and application scenarios.

[0003] Reconstructing facial textures is a crucial step in building high-fidelity digital humans. Methods for reconstructing high-quality facial textures for specific objects can be divided into two categories: One involves using specialized acquisition equipment, such as Lightstage, to capture a large number of images under varying lighting conditions and then fitting material parameters. This method places high demands on both hardware and software. For example, to facilitate subsequent material parameter calculations, the assumption of far-field light sources is often required, leading to significant equipment space requirements. Furthermore, a polarization system is typically needed for later separation of diffuse and specular reflections, necessitating the use of numerous cameras and further increasing data storage and processing costs. Additionally, the large volume of raw data requires substantial computational resources for material calculation. The other method uses deep learning models to infer facial textures. This method can use calibrated images or images from uncontrolled environments as input. However, to obtain high-quality results, the deep learning models used in this method often require high-quality reconstructed material maps acquired using specialized equipment as training data. Therefore, how to efficiently reconstruct high-quality facial textures from acquired data is an important and practically significant problem. Summary of the Invention

[0004] This invention addresses the shortcomings of existing methods by designing a face material reconstruction method based on multi-illumination images and differentiable rendering. This method uses multi-illumination images without polarization or far-field light source limitations as input, solves for the face material map through a two-stage differentiable rendering method, and accelerates the solution process by constructing a multi-layer pyramid and performing mini-batch sampling of light sources, enabling convenient and rapid generation of high-quality face material maps.

[0005] This invention is achieved through the following technical solution: According to a first aspect of the present invention, a method for reconstructing facial materials based on multi-light images and differentiable rendering is provided, comprising the following steps: S1. Calculate the optical flow based on the image sequence of fully lit lights in the multi-illumination image. The optical flow between images of fully lit lights is obtained by interpolating the optical flow of adjacent fully lit light images. The entire image sequence is aligned with the first fully lit image through the optical flow. S2. Based on images captured under multi-view light source conditions, decouple the reflectivity of each surface point and decompose it into a diffuse reflection component that is independent of the viewpoint and a specular component that is dependent on the viewpoint. S3. Based on the separation of diffuse and specular components, the material parameters at each location are fitted using a differentiable rendering method; an image pyramid is introduced as a multi-scale optimization framework, and a mini-batch strategy is adopted for light source sampling to accelerate computation; in order to obtain the final material parameters. S4. Stitch together the material textures obtained from each viewpoint and eliminate the seams caused by stitching the results from different viewpoints to obtain the final complete material texture.

[0006] Further, step S1 includes the following sub-steps: S11. First, the images in the entire shooting sequence are aligned. The images captured under full-brightness illumination are called tracking frames. The optical flow of adjacent tracking frames is calculated. S12. Assuming the human motion between two adjacent tracking frames is linear, obtain the optical flow of the image between adjacent tracking frames through linear interpolation; align the image between two adjacent tracking frames with the previous tracking frame in these two tracking frames using the optical flow. S13. Calculate the optical flow between each fully illuminated image and the first fully illuminated image, and then combine it with the optical flow obtained by linear interpolation in step S12 to align all the images in the sequence to the first fully illuminated image.

[0007] Further, step S2 includes the following sub-steps: S21. Calculate the total reflectance under incident light from different directions based on the images of light incident from different directions. S22. Remove the two directions with an incident angle greater than 80 degrees and those with self-blocking; S23. Based on the results remaining after removing the two directions in step S22, calculate the diffuse reflectance under the Lambertian model, and then remove directions whose diffuse reflectance values ​​differ from this value by more than 0.3. S24. Remove the top 20% of the remaining reflectance values ​​after sorting them from smallest to largest; S25. Take the average of the 10 smallest values ​​of the remaining reflectance as the diffuse reflectance at that location; S26. Based on the diffuse reflectance calculated in step S25, the captured result is decomposed into diffuse reflectance component and highlight component, where the minimum value of the highlight component is truncated to 0; then the diffuse reflectance part is obtained again based on the truncated highlight. S27. Recalculate the diffuse reflectance based on the diffuse reflectance component obtained in step S26, and the maximum difference between the recalculated diffuse reflectance and the diffuse reflectance calculated in step S25 shall not exceed 10%.

[0008] Further, step S3 includes the following sub-steps: S31. The image is downsampled through multiple layers to obtain a three-layer image pyramid with the length and width pixels being 10%, 40%, and 100% of the original image. The material parameters are first solved on the low-resolution image and then used as the initial values ​​for solving the higher-resolution image. S32. Solve for material parameters using the differentiable rendering method. The optimized loss function is the difference between the rendering result and the shooting result, plus three regularization terms: the difference between the roughness of the constraint solution and the given value, the deviation between the diffuse normal of the constraint solution and the initial normal direction of the model, and the deviation between the specular normal of the constraint solution and the initial normal direction of the model. S33. Each time, select 4% of the images in the entire shooting sequence and optimize them according to the loss function defined in step S32 until the result converges.

[0009] Furthermore, step S32 is specifically as follows: Material parameters and position are solved using differentiable rendering methods. Location, lighting mode The rendering result below diffuse component and highlight component The sum, specifically expressed as follows: ; in, ; ; ; in For all lighting modes, The intensity of the light source, The position of the light source, This is the position of the current solution point. For the current solution, the visibility function from the point to the light source is being calculated. For the Fresnel term of transmission, For refractive index, The direction of the incident light. Diffuse reflectance The direction of the emitted light. For diffuse reflection normal, Normal to the light source; For reflectivity, For roughness, For high light reflectivity, For the specular normal; Let GGX be the normal distribution function. For half-range vectors, For Fresnel terms, For GGX geometric functions; The loss function during optimization is: ; in It's the result of the filming. Given a roughness constraint value, let's set it to 0.35. The initial normal direction of the model. , , , Let the weights of each part of the loss function be 1, 0.1, 3, and 3; the parameters to be solved include diffuse reflectance. roughness High light reflectivity diffuse normal Specular normal MSE is the mean squared error function, and L1 is the L1 norm.

[0010] Specifically, step S4 includes the following sub-steps: S41. Expand the material results obtained from each viewpoint into the UV space; S42. Based on the orientation of each triangle in the 3D face model and its distance to the camera, assign each triangle to a certain viewpoint, and use the material result solved under that viewpoint as the material result of that region. S43. Obtain the stitched material map based on the result allocated in step S42, and then solve an additional function to eliminate the seams caused by stitching results from different perspectives; add the stitched material map and the additional function to obtain the final material map.

[0011] Further, step S42 assigns each triangle to a specific viewpoint, using the material result obtained from that viewpoint as the material result for that region; specifically: Based on the orientation of each triangle in the 3D face model and its distance from the camera, calculate the quality of each triangle for each viewpoint: ; in Let be the normal vector of the triangular face; Let be the direction vector from the triangle face to the camera; It is an inverse cosine function. Let L2 be the norm of the vector; Each triangle is assigned to the viewpoint with the smallest quality value, and the material result obtained from that viewpoint is used as the material result for that region.

[0012] Furthermore, the additional function in step S43 needs to satisfy the condition that its value at the splice can cancel out the abrupt change at the splice, while the sum of the squares of the gradients of the additional function over its domain other than the splice remains minimal; specifically as follows: Let f be the concatenated result. Solve for the additional function g according to the following objective: minimize the additional function across the entire domain except for the gradient at the seam, and make the value at the seam equal to the negative of the value of the jump at the seam. The expression is as follows: ; ; in Let g be the gradient of g, S be the seam of the splicing, and M be the complete domain of g. Then, the solved g is added to f, and the difference in value changed during the addition is limited to no more than 10% of f to obtain the final texture.

[0013] According to a second aspect of the present invention, an electronic device is provided, comprising: one or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the face material reconstruction method based on multi-light images and differentiable rendering.

[0014] According to a third aspect of the present invention, a computer-readable storage medium is provided having computer instructions stored thereon, which, when executed by a processor, implement the steps of the face material reconstruction method based on multi-light images and differentiable rendering.

[0015] The beneficial effects of this invention are as follows: This invention uses multi-illumination images as input, without the limitations of polarization and far-field light sources. Compared to devices requiring polarization and far-field light sources, it saves equipment space and camera costs when setting up the corresponding acquisition equipment, while also conserving data storage and computational resources. This invention designs a two-stage differentiable rendering method to separate diffuse reflection and specular highlights, and to solve for material parameters. In the process of solving for material parameters, a multi-layer pyramid calculation process and small-batch sampling of light sources accelerate the solution. Using the method of this invention, high-quality facial material can be obtained conveniently and quickly. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the main process of the present invention; Figure 2 This is a real-world example of obtaining the multi-illumination image data used in this invention; Figure 3 This is an example of multi-illumination image data used in this invention; Figure 4 This is an example of the results of the present invention in separating diffuse reflection and specular components; Figure 5 This is an example of the material reconstruction result of the present invention; Figure 6 This is a comparison image of the material texture map reconstructed and rendered by this invention and the captured image. Detailed Implementation

[0017] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0018] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0019] The core of this invention lies in the design of a face material reconstruction method based on multi-lighting images and differentiable rendering, such as... Figure 1 As shown, a two-stage differentiable rendering method is used. In the first stage, diffuse and specular reflections of the captured face image are separated. In the second stage, the material parameters of the face are optimized based on the separation results. Furthermore, a multi-layer pyramid and mini-batch sampling of the light source are used to accelerate the solution of the material parameters. The specific steps are as follows: S1. Data Preparation

[0020] This algorithm requires a sequence of images under multiple lighting conditions as input, such as Figure 2 As shown, this sequence should contain images of a face illuminated by light sources from different directions. The light sources do not need to satisfy the far-field assumption, but for the location to be solved, the light source should be illuminated from approximately the same direction each time, and an image with all the lights on should be inserted every few different lighting patterns. Additionally, reconstructed face geometry and calibrated camera and light source parameters are required. The light source parameters provide the information needed for light source modeling during the differentiable rendering stage. For LED light panel sources, the required light source parameter is the position of the light source.

[0021] S2, Aligning Face Sequences Because slight shaking is unavoidable during filming, the images in the entire shooting sequence must first be aligned. For example... Figure 3As shown, the image captured under full illumination is called a tracking frame. Assuming the human motion between two adjacent tracking frames is linear, the method for aligning the image between the i-th tracking frame and the (i+1)-th tracking frame to the target frame is as follows: First, calculate the optical flow from the (i+1)-th tracking frame to the i-th tracking frame, and obtain the optical flow from the image between these two tracking frames to the i-th tracking frame through linear interpolation. Then, calculate the optical flow from the i-th tracking frame to the target image, and combine it with the optical flow calculated in the previous step to obtain the optical flow from the image between the i-th tracking frame and the (i+1)-th tracking frame to the target image, and align them with the target image.

[0022] S3, Separation of Diffuse Reflection and Specular Reflection Because the input contains many images captured under different incident directions, such as Figure 4 As shown, assuming that at certain incident angles, the observed results only include the diffuse reflection component, the diffuse reflectance at each location is obtained through decoupling. First, the total reflectance at each location is calculated. And assuming that reflectance is entirely provided by diffuse reflection. The diffuse reflectance at each location is then calculated using the following formula: ; in This indicates taking the minimum value found among all incident directions i. This is the position of the current solution point. The direction of the emitted light. The direction of the incident light. For the Fresnel term of transmission, is the refractive index.

[0023] Consider using an LED light panel as the light source during shooting. Record the illumination status of the light source during each shot as a lighting mode. Each actual lighting mode includes multiple LED beads. Therefore, the position... The rendering process is modeled as follows: ; in ; For visibility, Let be the light source brightness. The diffuse reflectance at each location is: ; in This is the shooting result corresponding to lighting mode i.

[0024] The above expression can be used to calculate the diffuse reflectance of each location under each lighting mode, assuming that only diffuse reflectance exists. Assuming there are s lighting modes, each location will have s diffuse reflectances. Next, we need to calculate the final diffuse reflectance of this location based on these s diffuse reflectances.

[0025] First, diffuse reflectance values ​​obtained under lighting conditions with an incident angle greater than 80 degrees need to be removed, as well as diffuse reflectance values ​​obtained under lighting conditions where there is occlusion between the incident light and the target position. Then, the diffuse reflectance under the Lambertian model is calculated based on the remaining results, and results with a difference greater than 0.3 from this diffuse reflectance value are removed. From the remaining diffuse reflectance results, the smallest 20% are removed after sorting by value from smallest to largest, and then the average of the remaining 10 minimum values ​​is calculated and denoted as . .

[0026] according to The captured image can be decomposed into diffuse and highlight components. Highlight values ​​less than 0 are truncated to 0. The remaining diffuse component is then recalculated based on the truncated highlights, and the final diffuse reflectance is recalculated based on the remaining diffuse component, denoted as [missing information]. In calculation When it is necessary to restrict and The difference cannot exceed a given numerical proportion.

[0027] S4, Material Parameter Solving Material parameters and position are solved using differentiable rendering methods. Location, lighting mode The rendering result below It can be represented as diffuse reflection component and highlight component sum: ; in, ; ; ; in For all lighting modes, The intensity of the light source, The position of the light source, This is the position of the current solution point. For the current solution, the visibility function from the point to the light source is being calculated. For the Fresnel term of transmission, For refractive index, The direction of the incident light. Diffuse reflectance The direction of the emitted light. For diffuse reflection normal, This is the normal to the light source. For reflectivity, For roughness, For high light reflectivity, This is the specular normal. Let GGX be the normal distribution function. For half-range vectors, For Fresnel terms, For GGX geometric functions.

[0028] The loss function during optimization is: ; in It's the result of the filming. Given a roughness constraint value, let's set it to 0.35. The initial normal direction of the model. , , , Let the weights of each part of the loss function be 1, 0.1, 3, and 3; the parameters to be solved include diffuse reflectance. roughness High light reflectivity diffuse normal Specular normal MSE is the mean squared error function, and L1 is the L1 norm.

[0029] To accelerate the solution of material parameters, the image is downsampled through multiple layers to obtain image pyramids of different resolutions. Material parameters are first solved on the low-resolution images, and the results are then used as the initial values ​​for solving higher-resolution images. Furthermore, in each round of material parameter optimization, it is not necessary to calculate all lighting modes; instead, a subset of lighting modes is selected for optimization (e.g., 4% of the images in the entire shooting sequence are selected as the training set and optimized according to the loss function defined above; the number of optimization attempts for each image at each resolution level from low to high is 25, 5, and 1 respectively), thus accelerating the optimization process to obtain the final material parameters. Examples of material reconstruction results for each material parameter are shown below. Figure 5 As shown.

[0030] S5, Multi-view Material Splicing The material results obtained from each viewpoint are unfolded into UV space. Based on the orientation of each triangle in the 3D face model and its distance from the camera, the quality of the triangles for each viewpoint is calculated. ; in Let be the normal vector of the triangular face; Let be the direction vector from the triangle face to the camera; It is an inverse cosine function. Let L2 be the L2 norm of the vector.

[0031] Each triangle is assigned to the viewpoint with the smallest quality value, and the material result obtained from that viewpoint is used as the material result for that region. This yields a stitched material map. Then, an additional function is solved to eliminate seams caused by stitching results from different viewpoints. The additional function needs to satisfy the condition that the value at the stitching point can offset the abrupt change at the stitching point, while keeping the sum of the squares of the gradients of the additional function in the domain other than the stitching point to be minimized.

[0032] Let f be the concatenated result. Solve for the additional function g according to the following objective: make the additional function as small as possible across the entire domain except for the gradient value at the seam, and make the value at the seam equal to the negative of the value of the jump at the seam: ; ; in Let g be the gradient of g, S be the seam of the stitching, and M be the complete domain of g. Then, the solved g is added to f to obtain the final texture. To avoid the influence of noise and other extreme cases, the difference in value during addition is limited to no more than 10% of f.

[0033] Implementation Example 1: This application implements an embodiment of the invention on a desktop computer equipped with one Intel i9-13900K central processing unit and one Nvidia RTX 3080 graphics processor. The input multi-lighting images include nine camera viewpoints, each capturing 301 images. Of these, 224 are images under different lighting modes, one is a redundant image not used in the calculation, and 76 are images with all lights on. The fully lit images are inserted every three images under different lighting modes. Each image has a resolution of 3700*2900. Figure 6 As shown, using the method of the present invention, the reconstruction of the face material was completed in 1.5 hours.

[0034] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the face material reconstruction method based on multi-light images and differentiable rendering as described above.

[0035] This application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the face material reconstruction method based on multi-light images and differentiable rendering as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0036] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0037] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for reconstructing facial materials based on multi-illumination images and differentiable rendering, characterized in that, Includes the following steps: S1. Calculate the optical flow based on the image sequence of fully lit lights in the multi-illumination image. The optical flow between images of fully lit lights is obtained by interpolating the optical flow of adjacent fully lit light images. The entire image sequence is aligned with the first fully lit image through the optical flow. S2. Based on images captured under multi-view light source conditions, decouple the reflectivity of each surface point and decompose it into a diffuse reflection component that is independent of the viewpoint and a specular component that is dependent on the viewpoint. S3. Based on the separation of diffuse and specular components, the material parameters at each location are fitted using a differentiable rendering method. An image pyramid was introduced as a multi-scale optimization framework, and a mini-batch strategy was adopted for light source sampling to accelerate computation in order to obtain the final material parameters. S4. Stitch together the material textures obtained from each viewpoint and eliminate the seams caused by stitching the results from different viewpoints to obtain the final complete material texture.

2. The method for reconstructing facial materials based on multi-illumination images and differentiable rendering according to claim 1, characterized in that, Step S1 includes the following sub-steps: S11. First, the images in the entire shooting sequence are aligned. The images captured under full-brightness illumination are called tracking frames. The optical flow of adjacent tracking frames is calculated. S12. Assuming the human motion between two adjacent tracking frames is linear, obtain the optical flow of the image between adjacent tracking frames through linear interpolation; align the image between two adjacent tracking frames with the previous tracking frame in these two tracking frames using the optical flow. S13. Calculate the optical flow between each fully illuminated image and the first fully illuminated image, and then combine it with the optical flow obtained by linear interpolation in step S12 to align all the images in the sequence to the first fully illuminated image.

3. The method for reconstructing facial materials based on multi-illumination images and differentiable rendering according to claim 1, characterized in that, Step S2 includes the following sub-steps: S21. Calculate the total reflectance under incident light from different directions based on the images of light incident from different directions. S22. Remove the two directions with an incident angle greater than 80 degrees and those with self-blocking; S23. Based on the results remaining after removing the two directions in step S22, calculate the diffuse reflectance under the Lambertian model, and then remove directions whose diffuse reflectance values ​​differ from this value by more than 0.

3. S24. Remove the top 20% of the remaining reflectance values ​​after sorting them from smallest to largest; S25. Take the average of the 10 smallest values ​​of the remaining reflectance as the diffuse reflectance at that location; S26. Based on the diffuse reflectance calculated in step S25, the captured result is decomposed into diffuse reflectance component and highlight component, where the minimum value of the highlight component is truncated to 0; then the diffuse reflectance part is obtained again based on the truncated highlight. S27. Recalculate the diffuse reflectance based on the diffuse reflectance component obtained in step S26, and the maximum difference between the recalculated diffuse reflectance and the diffuse reflectance calculated in step S25 shall not exceed 10%.

4. The method for reconstructing facial materials based on multi-illumination images and differentiable rendering according to claim 1, characterized in that, Step S3 includes the following sub-steps: S31. The image is downsampled through multiple layers to obtain a three-layer image pyramid with the length and width pixels being 10%, 40%, and 100% of the original image. The material parameters are first solved on the low-resolution image and then used as the initial values ​​for solving the higher-resolution image. S32. Solve for material parameters using the differentiable rendering method. The optimized loss function is the difference between the rendering result and the shooting result, plus three regularization terms: the difference between the roughness of the constraint solution and the given value, the deviation between the diffuse normal of the constraint solution and the initial normal direction of the model, and the deviation between the specular normal of the constraint solution and the initial normal direction of the model. S33. Each time, select 4% of the images in the entire shooting sequence and optimize them according to the loss function defined in step S32 until the result converges.

5. The method for reconstructing facial materials based on multi-illumination images and differentiable rendering according to claim 4, characterized in that, The specific steps of S32 are as follows: Material parameters and position are solved using differentiable rendering methods. Location, lighting mode The rendering result below diffuse component and highlight component The sum, specifically expressed as follows: ; in, ； ； ； in For all lighting modes, The intensity of the light source, The position of the light source, This is the position of the current solution point. For the current solution, the visibility function from the point to the light source is being calculated. For the Fresnel term of transmission, For refractive index, The direction of the incident light. Diffuse reflectance The direction of the emitted light. For diffuse reflection normal, The normal to the light source; For reflectivity, For roughness, For high light reflectivity, For the specular normal; Let GGX be the normal distribution function. For half-range vectors, For Fresnel terms, For GGX geometric functions; The loss function during optimization is: ； in It's the result of the filming. Given a roughness constraint value, let's set it to 0.

35. The initial normal direction of the model. , , , Let the weights of each part of the loss function be 1, 0.1, 3, and 3; the parameters to be solved include diffuse reflectance. roughness High light reflectivity diffuse normal Specular normal MSE is the mean squared error function, and L1 is the L1 norm.

6. The method for reconstructing facial materials based on multi-illumination images and differentiable rendering according to claim 1, characterized in that, Step S4 includes the following sub-steps: S41. Expand the material results obtained from each viewpoint into the UV space; S42. Based on the orientation of each triangle in the 3D face model and its distance to the camera, assign each triangle to a certain viewpoint, and use the material result solved under that viewpoint as the material result of that region. S43. Obtain the stitched material map based on the result allocated in step S42, and then solve an additional function to eliminate the seams caused by stitching results from different perspectives; add the stitched material map and the additional function to obtain the final material map.

7. The method for reconstructing facial materials based on multi-illumination images and differentiable rendering according to claim 6, characterized in that, Step S42 assigns each triangle to a specific viewpoint and uses the material result obtained from that viewpoint as the material result for that region; specifically: Based on the orientation of each triangle in the 3D face model and its distance from the camera, calculate the quality of each triangle for each viewpoint: ； in Let be the normal vector of the triangular face; Let be the direction vector from the triangle face to the camera; It is an inverse cosine function. Let L2 be the norm of the vector; Each triangle is assigned to the viewpoint with the smallest quality value, and the material result obtained from that viewpoint is used as the material result for that region.

8. The method for reconstructing facial materials based on multi-illumination images and differentiable rendering according to claim 6, characterized in that, The additional function in step S43 needs to satisfy the condition that its value at the concatenation point can cancel out the abrupt change at the concatenation point, while minimizing the sum of squared gradients of the additional function over its domain other than the concatenation point; specifically as follows: Let f be the concatenated result. Solve for the additional function g according to the following objective: minimize the additional function across the entire domain except for the gradient at the seam, and make the value at the seam equal to the negative of the value of the jump at the seam. The expression is as follows: ； ； in Let g be the gradient of g, S be the seam of the splicing, and M be the complete domain of g. Then, the solved g is added to f, and the difference in value changed during the addition is limited to no more than 10% of f to obtain the final texture.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the face material reconstruction method based on multi-light images and differentiable rendering as described in any one of claims 1-8.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the face material reconstruction method based on multi-light images and differentiable rendering as described in any one of claims 1-8.

Images