A face highlight and roughness reconstruction method

Abstract

The application discloses a face highlight and roughness reconstruction method, and proposes a method of first calculating a regional mean value, and then taking the regional mean value as a constraint to calculate a pixel value in the region. When automatically dividing a face region, the application uses a kmeans algorithm to divide the face into blocks, and uses data with different attributes as classification parameters, so that points with similar material values are divided into the same block. Theoretical verification and experimental results prove that the application solves the under-constraint problem faced when solving the pixel material, and the obtained result can withstand artificial data experiment verification. The application ingeniously increases the constraint, and creatively solves the under-constraint problem when solving the highlight and roughness map, can obtain the highlight and roughness of an object through consumer-level hardware, and can be applied to the reconstruction of the highlight and roughness of various different complex objects, and is applied in a virtual reality scene, has a huge application space, and has high universality.

Description

Technical Field

[0001] This invention relates to the field of face material reconstruction in computer graphics, and more particularly to a method for face material reconstruction using optimization methods. Background Technology

[0002] Facial texture reconstruction is in high demand in fields such as virtual reality and digital entertainment. However, there is currently no fully automated method for reconstructing facial highlights and roughness parameters that can pass accuracy tests.

[0003] While current methods for solving facial texture problems can reconstruct images that align with human intuition, the results often fail to reflect real-world facial textures. Existing reconstruction methods cannot pass synthetic experiments (i.e., using an image rendered with the desired texture as the target image for texture reconstruction fails to reproduce the texture map used in the rendering), and significant differences between the reconstructed image and the real facial texture can be observed in the rendering results. These limitations stem from the inclusion of excessive average material constraints and smoothness constraints. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for reconstructing facial highlights and roughness.

[0005] The objective of this invention is achieved through the following technical solution: a method for reconstructing facial highlights and roughness, comprising the following steps:

[0006] (1) Data acquisition: The acquisition environment consists of a polarized light source and 6 SLR cameras. The polarized light source and 6 SLR cameras are distributed on a semicircle centered on the subject. The polarized light source is acquired by covering the searchlight with a polarizing film. Among the 6 SLR cameras, 4 cameras have polarizing filters parallel to the polarizing film of the searchlight in front of their lenses, while the other 2 do not. The delay parameters of each camera are adjusted so that each camera can capture images of the subject from different angles at the same time.

[0007] (2) Reconstructing the diffuse map: The loss function used in this step is as shown in equation (1), where albedo is the diffuse map parameter, and the function... This represents the Lambert BRDF rendering formula, where `img` is the target image, and the difference between the rendered image and the target image is used as the energy during optimization. (1)

[0008] (3) Solving for the mean values ​​of specular highlights and roughness: This step has two states:

[0009] The first state is when the face has not yet been segmented during the first execution of this step, and the loss function for calculating the mean is shown in equation (2): (2)

[0010] in This represents the average roughness of the region. This represents the average highlight value of the region, img is the value of the corresponding region in the target image, and F represents the image rendered using roughness and highlight parameters.

[0011] The second state is when this step is not being executed for the first time, and the face has already been segmented. The mean of each sub-region after segmentation is calculated: using the mean of the region to which this region belonged during the previous execution of this step as a constraint, the mean of the highlight and roughness of this region is calculated; the loss function is shown in equations (3), (4), (5), and (6); the loss function consists of three parts, the first part being the energy E1, which is the difference between the rendered image and the target image, where... This represents the average roughness of the region. The first part represents the mean highlight value of this region, where img is the target image and F represents the cook-torrance BSDF rendering function; the second part, energy E2, is the difference between the roughness of this region and the roughness of the next higher-level region to which this region belongs. The first part represents the average roughness of the region to which this pixel belongs; the third part, energy E3, is the difference between the highlight of this region and the highlight of the region to which this region belongs. This represents the average highlight value of the region to which this pixel belongs. (3) (4) (5) (6);

[0012] (4) Segmenting the face: After obtaining the coordinate values ​​of the image region, index these coordinate points, add the classification attributes to the corresponding space of the index and standardize and normalize them; then multiply by the classification weight of each attribute; finally divide each block into smaller classes through the kmeans function and record the index of the coordinate points of this class; before the face segmentation algorithm, all attributes need to be standardized and normalized. The standardization and normalization algorithm used in this invention is shown in equations (7) and (8), where mean(x) is the mean of array x, var(x) is the variance of array x, min(x) and max(x) are the minimum and maximum values ​​of the array respectively; the data is first processed by equation (3) and then processed by equation (4) to make the data evenly distributed in the closed interval [0,1]; (7) (8)

[0013] The face segmentation algorithm uses the k-means clustering algorithm, with the three-dimensional normal vector values ​​of the face geometry surface and the two-dimensional coordinate values ​​of the image space as clustering parameters, as shown in equation (9), where normal_r, normal_g, and normal_b represent the values ​​stored in the normal vector. Figure 3 The normal vector values ​​in each channel are three dimensions, where x and y are the coordinates of the point in image space, and err_r, err_g, and err_b represent the color temperature of the point stored in the previous reconstruction error. Figure 3 The values ​​in each channel, normal_r0, normal_g0, and normal_b0, represent the cluster center points stored in the normal vector. Figure 3 The normal vector values ​​of the three dimensions in each channel, x0, y0 are the coordinates of the cluster center points in the image space, and err_r0, err_g0, err_b0 represent the color temperature of the cluster center points stored in the previous reconstruction error. Figure 3 The values ​​in each channel; this classification allows points with similar facial attributes to be grouped into one category; (9);

[0014] (5) Repeat steps (3) and (4) until the last segmented area is small enough to be less than 400 pixels;

[0015] (6) Using the solutions obtained by the face segmentation algorithm and the face mean solving algorithm as constraints, the specular texture map and diffuse texture map are solved pixel by pixel by the optimization algorithm. The loss function is shown in equations (10), (11), (12) and (13). The loss function consists of three parts. The first part, energy E1, is the difference between the rendered image and the target image, where roughness is the roughness value of the pixel, specular is the specular value of the pixel, img is the corresponding pixel value of the target image, and F represents the image rendered with roughness and specular parameters. The second part, energy E2, is the difference between the roughness value roughness of the pixel and the roughness mean value roughness0 of the region to which the pixel belongs. The third part, energy E3, is the difference between the specular value specular of the pixel and the specular mean value specular0 of the region to which the pixel belongs. (10) (11) (12) (13).

[0016] The beneficial effects of this invention are that it uses low-cost, consumer-grade hardware and solves the problem of under-constraint in facial highlight and roughness reconstruction by first calculating the mean of a region and then using that mean as a constraint to calculate the pixel-by-pixel value within that region. This method does not require the addition of average material constraints or smoothness constraints during the reconstruction process. The correctness of the reconstruction results is proven through synthetic experiments (i.e., using an image rendered with the material as the target image for material reconstruction can reconstruct the material map used for rendering). The rendering results demonstrate that the reconstruction accurately reflects the real facial material of the user, and the error between the reconstructed image and the real image is significantly reduced compared to previous methods. Attached Figure Description

[0017] Figure 1 This is a flowchart of the reverse rendering process;

[0018] Figure 2 Flowchart for face segmentation;

[0019] Figure 3 This is a schematic diagram of a face that has not yet been segmented in the embodiment;

[0020] Figure 4 This is a schematic diagram of a face after being segmented in the embodiment;

[0021] Figure 5 This is a schematic diagram of a face after processing using the method of the present invention in an embodiment. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0023] The general process of this invention is as follows: first, a material texture map is reconstructed from the target image and the geometric model; then, the specular and roughness maps are optimized simultaneously. The specific process of this invention will now be described in detail.

[0024] Step 1: Data Collection

[0025] Before performing high-sensitivity and roughness reconstruction of the face, it is necessary to first acquire photographs of the subject. The photographic acquisition environment in this paper consists of a polarized light source and six SLR cameras, distributed in a semicircle centered on the subject. The polarized light source is obtained by covering a spotlight with a polarizing film. Four of the six SLR cameras have polarizing filters parallel to the spotlight's polarizing film in front of their lenses, while the other two do not. This is done to eliminate specular reflection, allowing the reconstruction of the diffuse map to remove the effects of roughness and specular properties, ensuring that these effects do not affect the diffuse map. This allows for more accurate reconstruction of the diffuse, specular, and roughness maps. During acquisition, the delay parameters of each camera should be adjusted to ensure that each camera captures images of the subject from different angles simultaneously. Furthermore, it is important to ensure that there are no other light sources indoors to guarantee that the indoor light source is primarily polarized.

[0026] Step 2: Reconstruct the diffuse map

[0027] The diffuse texture is reconstructed using an optimization algorithm based on the target image and the geometric model. The loss function used in this step is shown in Equation (1), where albedo represents the diffuse texture parameters, function F represents the Lambert BRDF rendering formula, and img is the target image. This paper uses the difference between the rendered image and the target image as the energy during optimization.

[0028] (1)

[0029] Step 3: Calculate the mean values ​​of specular highlights and roughness.

[0030] After initializing the relevant variables, considering memory limitations, the image is segmented into pieces and each piece is reverse-rendered separately. The reverse-rendering process uses the difference between the forward-rendered image and the target image as the loss function to calculate the gradient. If there are external constraints, the gradient of the constraint portion is obtained. The two gradients are then summed, and the target parameters are updated based on the gradient and step size. This process stops when the loss function converges.

[0031] This step has two states:

[0032] The first state is when this step is executed for the first time, such as... Figure 3 The face shown has not yet been segmented. Note that the segmentation here is not based on memory limitations, but rather on the segmentation algorithm. The loss function for calculating the mean during the first execution of this step is shown in equation (2). This represents the average roughness of the region. This represents the average highlight value of the region, img is the target image, and F represents the cook-torrance BSDF rendering function.

[0033] (2)

[0034] in This represents the average roughness of the region. is the average highlight value of the region, img is the value of the corresponding region in the target image, and F represents the image rendered using roughness and highlight parameters.

[0035] The second state is when this step is not being executed for the first time; the face has already been segmented in the fourth step, such as... Figure 4 As shown. When calculating the mean of a sub-region within a region, this algorithm uses the mean of the region it belonged to during the last execution of this step as a constraint to solve for the mean of specular highlights and roughness in this region. The loss function is shown in equations 3, 4, 5, and 6. The loss function consists of three parts. The first part, energy E1, is the difference between the rendered image and the target image, where... This represents the average roughness of the region. The first part represents the mean highlight value of this region, where `img` is the target image and `F` represents the cook-torrance BSDF rendering function. The second part, energy E2, is the difference between the roughness of this region and the roughness of its parent region. This represents the average roughness of the region to which this pixel belongs. The third part, energy E3, is the difference between the highlight of this region and the highlight of the region to which it belongs in the next higher level. This is the average highlight value of the region to which this pixel belongs.

[0036] (3) (4) (5) (6)

[0037] Step 4: Segment the face

[0038] After obtaining the coordinates of the image region, indexes are created for these coordinate points. The categorical attributes are then added to the corresponding index space and standardized / normalized. Next, the values ​​are multiplied by the categorical weight of each attribute. Finally, each region is divided into smaller classes using the k-means function, and the indices of the coordinate points for that class are recorded.

[0039] Before the face segmentation algorithm, all attributes need to be standardized and normalized. The standardization and normalization algorithms used in this invention are shown in equations (7) and (8), where mean(x) is the mean of array x, var(x) is the variance of array x, and min(x) and max(x) are the minimum and maximum values ​​of the array, respectively. The data is first processed by equation (3), and then processed by equation (4) to make the data evenly distributed in the closed interval [0,1].

[0040] (7) (8)

[0041] The face segmentation algorithm uses the k-means clustering algorithm, with the three-dimensional normal vector values ​​of the face geometry surface and the two-dimensional coordinate values ​​of the image space as clustering parameters, as shown in equation (9), where normal_r, normal_g, and normal_b represent the values ​​stored in the normal vector. Figure 3 The normal vector values ​​in each channel are three dimensions, where x and y are the coordinates of the point in image space, and err_r, err_g, and err_b represent the color temperature of the point stored in the previous reconstruction error. Figure 3 The values ​​in each channel, normal_r0, normal_g0, and normal_b0, represent the cluster center points stored in the normal vector. Figure 3 The normal vector values ​​of the three dimensions in each channel, x0, y0 are the coordinates of the cluster center points in the image space, and err_r0, err_g0, err_b0 represent the color temperature of the cluster center points stored in the previous reconstruction error. Figure 3 The values ​​in each channel. This classification group points with similar facial attributes into the same category.

[0042] (9)

[0043] Repeat steps three and four until the final segmented area is small enough, less than 400 pixels, then stop.

[0044] Finally, using the solutions obtained by the face segmentation algorithm and the face mean averaging algorithm as constraints, the specular and diffuse texture maps are solved pixel-by-pixel through an optimization algorithm, and the results are as follows. Figure 5 As shown. The loss function is shown in equations (10), (11), (12), and (13). The loss function consists of three parts. The first part, energy E1, is the difference between the rendered image and the target image, where roughness is the roughness value of the pixel, specular is the specular value of the pixel, img is the corresponding pixel value of the target image, and F represents the image rendered using the roughness and specular parameters. The second part, energy E2, is the difference between the roughness value of the pixel, roughness, and the mean roughness value, roughness0, of the region to which the pixel belongs. The third part, energy E3, is the difference between the specular value of the pixel, specular, and the mean specular value, specular0, of the region to which the pixel belongs. (10) (11) （12） （13）。

Claims

1. A method for reconstructing facial highlights and roughness, characterized in that, Includes the following steps: (1) Data acquisition: The acquisition environment consists of a polarized light source and 6 SLR cameras. The polarized light source and 6 SLR cameras are distributed on a semicircle centered on the subject. The polarized light source is acquired by covering the searchlight with a polarizing film. Among the 6 SLR cameras, 4 cameras have polarizing filters parallel to the polarizing film of the searchlight in front of their lenses, while the other 2 do not. The delay parameters of each camera are adjusted so that each camera can capture images of the subject from different angles at the same time. (2) Reconstructing the diffuse map: The loss function used in this step is as shown in equation (1), where albedo is the diffuse map parameter, and the function... This represents the Lambert BRDF rendering formula, where `img` is the target image, and the difference between the rendered image and the target image is used as the energy during optimization. （1） (3) Solving for the mean values ​​of specular highlights and roughness: This step has two states: The first state is when the face has not yet been segmented during the first execution of this step, and the loss function for calculating the mean is shown in equation (2): （2） in This represents the average roughness of the region. This represents the average highlight value of the region, img is the value of the corresponding region in the target image, and F represents the image rendered using roughness and highlight parameters. The second state is when this step is not being executed for the first time, and the face has already been segmented. The mean of each sub-region after segmentation is calculated: using the mean of the region to which this region belonged during the previous execution of this step as a constraint, the mean of the highlight and roughness of this region is calculated; the loss function is shown in equations (3), (4), (5), and (6); the loss function consists of three parts, the first part being the energy E1, which is the difference between the rendered image and the target image, where... This represents the average roughness of the region. The first part represents the mean highlight value of this region, where img is the target image and F represents the cook-torrance BSDF rendering function; the second part, energy E2, is the difference between the roughness of this region and the roughness of the next higher-level region to which this region belongs. The first part represents the average roughness of the region to which this pixel belongs; the third part, energy E3, is the difference between the highlight of this region and the highlight of the region to which this region belongs. This represents the average highlight value of the region to which this pixel belongs. （3） （4） （5） （6） (4) Segmenting the face: After obtaining the coordinate values ​​of the image region, index these coordinate points, add the classification attributes to the corresponding space of the index and standardize and normalize them; then multiply by the classification weight of each attribute; finally divide each block into smaller classes through the kmeans function and record the index of the coordinate points of this class; before the face segmentation algorithm, all attributes need to be standardized and normalized. The standardization and normalization algorithm used in this invention is shown in equations (7) and (8), where mean(x) is the mean of array x, var(x) is the variance of array x, min(x) and max(x) are the minimum and maximum values ​​of the array respectively; the data is first processed by equation (3) and then processed by equation (4) to make the data evenly distributed in the closed interval [0,1]; （7） （8） The face segmentation algorithm uses the k-means clustering algorithm, taking the three-dimensional normal vector values ​​of the geometric surface of the face and the two-dimensional coordinate values ​​of the image space as clustering parameters, as shown in Equation (9). Here, normal_r, normal_g, and normal_b represent the normal vector values ​​of the three dimensions stored in the three channels of the normal vector map, x and y are the coordinates of the point in the image space, err_r, err_g, and err_b represent the values ​​of the point stored in the three channels of the previous reconstruction error color temperature map, normal_r0, normal_g0, and normal_b0 represent the normal vector values ​​of the three dimensions stored in the three channels of the normal vector map of the cluster center point, x0 and y0 are the coordinates of the cluster center point in the image space, and err_r0, err_g0, and err_b0 represent the values ​​of the cluster center point stored in the three channels of the previous reconstruction error color temperature map. This classification makes points with similar facial attributes grouped into one class. （9）； (5) Repeat steps (3) and (4) until the last segmented area is less than 400 pixels; (6) Using the solutions obtained by the face segmentation algorithm and the face mean solving algorithm as constraints, the specular texture map and diffuse texture map are solved pixel by pixel by the optimization algorithm. The loss function is shown in equations (10), (11), (12) and (13). The loss function consists of three parts. The first part, energy E1, is the difference between the rendered image and the target image, where roughness is the roughness value of the pixel, specular is the specular value of the pixel, img is the corresponding pixel value of the target image, and F represents the image rendered with roughness and specular parameters. The second part, energy E2, is the difference between the roughness value roughness of the pixel and the roughness mean value roughness0 of the region to which the pixel belongs. The third part, energy E3, is the difference between the specular value specular of the pixel and the specular mean value specular0 of the region to which the pixel belongs. （10） （11） （12） （13）。

Images