Face reconstruction method and device based on key point recognition

Abstract

The application relates to the technical field of three-dimensional reconstruction, and provides a face reconstruction method and equipment based on key point identification, which corrects the color of key points in remaining face images locally according to the mapping relationship between the key points in the front face image and the remaining face images, and then migrates the color of each face key point to the whole image, so that the problem of subsequent feature point matching errors caused by different illuminations under different viewing angles is solved, the number of matched feature point pairs is enriched, the number of viewing angles can be reduced, a high-precision face model can be reconstructed under sparse view points, and the hardware cost is reduced; meanwhile, when target camera parameters of multiple cameras are determined, face key point pairs with mapping relationship are added into a projection error energy function as regular terms, so that the semantic matching between the face key points can be taken as a constraint condition, the matching accuracy is improved, and the reconstruction precision of multi-view 3D face model reconstruction is improved.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and provides a face reconstruction method and device based on key point recognition. Background Technology

[0002] With the rise of the concept of "metaverse", the reconstruction of lifelike and high-precision digital humans has attracted great attention. As one of the core technologies of high-precision digital human reconstruction, face reconstruction has become a research hotspot in academia and industry.

[0003] The widespread use of RGB cameras has promoted the development of vision-based face reconstruction. Currently, face reconstruction methods based on multiple view stereo (MVS) are one of the main methods for face reconstruction. They take RGB images of faces from multiple dense viewpoints as input, automatically extract feature points from the images, and complete face reconstruction through feature point matching.

[0004] Because the traditional MVS algorithm is a highly generalized reconstruction algorithm and was not specifically designed for face reconstruction, it is easily affected by lighting conditions when performing feature matching on RGB face images from different viewpoints. This can lead to incorrect or failed feature point matching, reducing the accuracy of face reconstruction. Furthermore, to ensure the density of the reconstruction results, the traditional MVS algorithm requires a sufficient amount of overlap between the RGB face images from different viewpoints to ensure that enough matching feature points can be found. In other words, the traditional MVS algorithm needs to deploy a very dense number of viewpoints (dozens or even hundreds) to ensure that a high-precision face model can be reconstructed. This greatly increases the hardware cost (tens or even millions of dollars), making it difficult to apply on a large scale. Summary of the Invention

[0005] This application provides a face reconstruction method and device based on key point recognition, which is used to improve the reconstruction accuracy of face models under sparse viewpoints.

[0006] On the one hand, this application provides a face reconstruction method based on key point recognition, including:

[0007] Acquire face images captured by the camera from multiple perspectives, and use the face image from the frontal view as the reference image and the face images from other perspectives as neighboring images;

[0008] Identify the facial key points in the face image from each viewpoint;

[0009] For each neighboring image, based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring images, the color values ​​of pixels in the reference image are used to correct the color values ​​of pixels in the neighboring images.

[0010] Based on the reference image and each neighboring image, camera parameters of multiple cameras are determined to align each neighboring image with the reference image;

[0011] Based on the reference image and each of the neighboring images, a 3D face model is reconstructed using a multi-view stereo vision algorithm.

[0012] On the other hand, this application provides a reconstruction device, including a processor, a memory, a communication interface, and a display screen, wherein the communication interface, the display screen, the memory, and the processor are connected via a bus;

[0013] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0014] The communication interface is used to acquire face images captured by the camera from multiple perspectives, and the face image from the frontal perspective is used as the reference image, while the face images from other perspectives are used as neighboring images.

[0015] Identify the facial key points in the face image from each viewpoint;

[0016] For each neighboring image, based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring images, the color values ​​of pixels in the reference image are used to correct the color values ​​of pixels in the neighboring images.

[0017] Based on the reference image and each neighboring image, camera parameters of multiple cameras are determined to align each neighboring image with the reference image;

[0018] Based on the reference image and each of the neighboring images, a 3D face model is reconstructed using a multi-view stereo vision algorithm and displayed on the display screen.

[0019] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to execute a face reconstruction method based on key point recognition provided in embodiments of this application.

[0020] The face reconstruction method and device based on key point recognition provided in this application uses a frontal view face image as a reference image and the rest as neighboring images. Facial key points are extracted from each image. Based on the mapping relationship between the reference image and each neighboring image, the color values ​​of pixels in the reference image are used to correct the color values ​​of pixels in the corresponding neighboring images. By transferring the color values ​​of facial key points in the reference image to each neighboring image, the colors of facial key points in each neighboring image are made closer to the colors of facial key points in the reference image. This solves the problem of subsequent feature point matching errors caused by different lighting conditions under different viewpoints, enriches the number of matched feature point pairs, thereby reducing the number of viewpoints and ensuring that a high-precision face model can be reconstructed under sparse viewpoints (less than ten), thus reducing hardware costs. Furthermore, based on the corrected neighboring images and reference image, and the camera parameters of multiple cameras, multi-view 3D face model reconstruction is achieved. By introducing facial key points as auxiliary information, the high-precision characteristics of pixel-by-pixel reconstruction by multi-view stereo vision algorithms can be preserved under sparse viewpoints. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a face reconstruction method based on key point recognition provided in this application embodiment;

[0023] Figure 2 A schematic diagram of 98 facial key points identified in an embodiment of this application;

[0024] Figure 3 Comparison of facial key points extracted from facial images from different perspectives provided in the embodiments of this application;

[0025] Figure 4 This is a flowchart of a method for color correction of adjacent images provided in an embodiment of this application;

[0026] Figure 5 This is a flowchart of a method for local color correction of facial key points in adjacent images provided in an embodiment of this application;

[0027] Figure 6 A flowchart illustrating a method for determining the color values ​​of facial key points provided in an embodiment of this application;

[0028] Figure 7This is a flowchart of a method for global color correction of pixels in neighboring images provided in an embodiment of this application;

[0029] Figure 8 An image showing the effect of adjusting the colors of neighboring images based on a reference image, provided as an embodiment of this application.

[0030] Figure 9 A flowchart illustrating a method for determining camera parameters of a multi-view camera, as provided in an embodiment of this application.

[0031] Figure 10 The reconstruction effect of the face geometry model provided in the embodiments of this application;

[0032] Figure 11 A textured 3D face model rendering provided for an embodiment of this application;

[0033] Figure 12 This is a hardware structure diagram of the reconstruction device provided in the embodiments of this application;

[0034] Figure 13 This is a functional structure diagram of the reconstruction device provided in the embodiments of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0036] Currently, the mainstream methods for 3D face reconstruction include: face reconstruction methods based on parametric models, face reconstruction methods based on deep learning, and face reconstruction methods based on the MVS algorithm. Among them:

[0037] Face reconstruction methods based on parametric models use a single RGB image of a face as input to guide the optimization of a standard parametric model (such as 3DMM, FLAME, DECA, etc.) to obtain a personalized 3D face model. However, this method is an approximate optimization algorithm. Because the input single RGB image of a face contains limited information, the reconstructed 3D face model can only achieve a superficial resemblance to the real person, making high-precision reconstruction difficult.

[0038] Among deep learning-based face reconstruction methods, the ToFU (Topologically Consistent Multi-View Face Inference Using Volumetric Sampling) model achieves relatively good reconstruction results. This model takes sparse RGB face images as input and uses deep learning algorithms to recover high-precision geometric models and texture maps with the same topology. However, this method requires precise calibration relationships between RGB cameras arranged on sparse viewpoints, and the calibration process is quite complex, limiting its application.

[0039] The face reconstruction method based on the MVS algorithm takes multiple densely packed RGB images of faces as input, automatically extracts feature points from the images, and matches them by calculating the illumination consistency of feature points between two images, thereby completing face reconstruction. Specifically, for each feature point, the pixel intensity, mean, variance, and other information of the feature point and its surrounding neighboring pixels are calculated, and this information is used as the feature of that feature point. Then, the feature difference between feature points in the two images is calculated. When the difference is less than a certain threshold, the two feature points are considered to have matched successfully. However, since the traditional MVS algorithm is a highly generalized reconstruction algorithm and was not specifically designed for face reconstruction, the same feature captured from different viewpoints often results in incorrect or failed matching of feature points due to different lighting conditions, thus reducing the accuracy of face reconstruction. Furthermore, in order to ensure the density of the reconstruction results, the traditional MVS algorithm requires a sufficient amount of overlap between the RGB images of the face from different viewpoints to ensure that enough matching feature points can be found. In other words, the traditional MVS algorithm needs to deploy a very dense number of viewpoints (dozens or even hundreds) to ensure that a high-precision face model can be reconstructed. This greatly increases the hardware cost (tens or even millions of dollars), making it difficult to apply on a large scale.

[0040] In view of this, this application provides a face reconstruction method and device based on key point recognition, based on the traditional MVS algorithm. By introducing facial key points as auxiliary information for color transfer, the number of matching point pairs is increased, reducing the MVS algorithm's requirement for high overlap of multiple views. At the same time, it reduces the MVS algorithm's requirement for the number of viewpoints, enabling the reconstruction of high-precision 3D face models even with sparse viewpoint input (within ten views). Furthermore, by using semantic matching between facial key points in multiple views as constraints to directly calculate the camera parameters of the multi-view RGB camera, this method can directly use RGB face images captured by mobile devices (such as mobile phones, tablets, TVs, VR glasses, etc.) as input for face reconstruction, reducing deployment difficulty and expanding the scope of application.

[0041] See Figure 1The following is a flowchart of a face reconstruction method based on key point recognition provided in an embodiment of this application. The process mainly includes the following steps:

[0042] S101: Acquire face images captured by the camera from multiple perspectives, and use the face image from the frontal view as the reference image and the face images from other perspectives as neighboring images.

[0043] Typically, face images viewed from the front contain the most facial details and have the most facial key points identified. Therefore, face images viewed from the front are used as reference images, while face images viewed from other perspectives are used as neighboring images.

[0044] S102: Identify facial key points in the face image from each viewpoint.

[0045] In the field of computer vision, most facial landmark recognition algorithms can accurately identify facial landmarks with semantic features, and the number of landmarks has increased from the initial 5 to hundreds. These facial landmarks mainly include multiple points such as the inner and outer corners of the eyes, upper and lower eyelids, corners of the mouth, lips, nose, facial contours, and eyebrows.

[0046] For example, such as Figure 2 The image shown is a schematic diagram of 98 facial key points identified in an embodiment of this application.

[0047] Because face images viewed from the front contain the most complete facial area, the number of facial landmarks identified is greater than that identified in face images viewed from other perspectives.

[0048] like Figure 3 The image shown is a comparison of facial landmarks extracted from face images from different viewpoints. Figure 3 As can be seen, the face image from the frontal view can be used as a reference image to detect all facial key points, while the face images from other views can be used as neighboring images of the reference image to extract some facial key points.

[0049] S103: For each neighboring image, based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring images, use the color values ​​of the pixels in the reference image to correct the color values ​​of the pixels in the neighboring images.

[0050] Each facial landmark identified by the facial landmark recognition algorithm contains its corresponding semantic information (such as inner and outer corners of the eyes, upper and lower eyelids, corners of the mouth, lips, nose, facial contours, eyebrows, etc.). This semantic information can be uniquely identified by the feature point number, such as... Figure 1The semantic information of facial landmark number 96 is the right eyeball. Therefore, based on the semantic information of facial landmarks, it is easy to establish a one-to-one mapping relationship between facial landmarks in the reference image and facial landmarks in neighboring images. For example, facial landmarks with the same number in the reference image and neighboring images can be directly mapped.

[0051] When executing S103, based on the mapping relationship between facial key points in the reference image and facial key points in neighboring images, color space correction is performed on each neighboring image to the reference image.

[0052] It should be noted that since the identified facial key points include eyebrow key points, and eyebrows are hair material, not skin material, when performing color correction on each neighboring image, the eyebrow key points in the reference image should not be used as a reference for color correction. The color information of the eyebrows needs to be removed. That is to say, in S103, the color correction of the corresponding neighboring images is performed based on the one-to-one mapping relationship between the skin texture facial key points in the reference image and the neighboring images.

[0053] An optional implementation is that, in S103, when performing color correction on each neighboring image, a color transfer algorithm with good correction results can be used. A color transfer algorithm can be understood as: based on a reference image and neighboring images, a new target image is synthesized, so that the color-corrected target image learns the overall color tone of the reference image without changing the color information expressed by the neighboring images. Therefore, the process of the color transfer algorithm can be regarded as a color transfer synthesis process of the image.

[0054] Traditional color transfer algorithms convert the entire reference image and neighboring images from the RGB color space to the Lab color space. Then, through a linear transformation, they adjust the mean and variance of all pixels in the neighboring images in the Lab color space to match those of all pixels in the reference image. Finally, they convert back to the RGB color space, thus completing the color transfer of the entire image. Because traditional color transfer algorithms are a forced global color transfer algorithm, the color transfer effect will be poor if there is a significant color difference between the two images.

[0055] To address the aforementioned issues, since the embodiments of this application have established a mapping relationship between facial key points in the reference image and facial key points in neighboring images based on semantic information, in S103, when calculating the linear equation for color migration, it is only necessary to consider the local regions where the facial key point pairs with mapping relationships are located, thereby reducing the impact of global color difference.

[0056] For the specific color correction process for each neighboring image, please refer to [link / reference]. Figure 4 It mainly includes the following steps:

[0057] S1031: Convert the reference image and neighboring images from the RGB color space to the Lab color space.

[0058] A color space, also known as a color model, color space, or color system, is used to describe color in a generally acceptable way under certain standards. Commonly used color spaces include RGB, CMY, HSV, HIS, and Lab.

[0059] RGB (Red, Green, Blue) is a color space defined based on human visual perception. It can represent most colors and consists of a red channel (R), a green channel (G), and a blue channel (B). It is a widely used hardware-oriented color model, such as for color monitors and a large class of color video cameras. However, the RGB color space is generally not used in scientific research because it represents hue, brightness, and saturation together, making them difficult to separate and digitally adjust details.

[0060] The Lab color space is used for tone adjustment and color correction. It's a device-independent CIE color model used to map device values ​​to the model and the model's own color distribution quality changes. It's a device-independent color system and also a physiologically based color system, meaning it uses a digital method to describe human visual perception. The Lab color space consists of a lightness (L) channel and two color channels (a, b). The L channel is specifically responsible for the overall brightness of the image; simply put, it's a black and white version of the entire image. The a and b channels are only responsible for the amount of color. The a channel represents the range from magenta (white in the channel) to dark green (black in the channel); b represents the range from yellowish-brown (white in the channel) to bluish-green (black in the channel). 50% neutral gray in the a and b channels represents no color, so the closer to gray, the less color there is. Furthermore, the colors in the a and b channels have no brightness. An example of this in visual phenomena is that the outline of a red T-shirt is very clear in the a and b components because red is composed of magenta and yellowish-brown.

[0061] Based on the characteristics of the Lab color space, under any monochromatic background, extracting parts with obvious color differences using component analysis can be completed quickly in the Lab color space. Furthermore, any adjustments to the L channel in the Lab color space (such as sharpening, blurring, etc.) will not affect the hue. Therefore, in S1031, the reference image and neighboring images can be converted from the RGB color space to the Lab color space.

[0062] S1032: Based on the mapping relationship between facial key points in the reference image and facial key points in neighboring images, determine the color attributes of each channel of the facial key point pairs with mapping relationships in their respective set neighborhoods, and correct the color values ​​of the corresponding facial key points in each channel in neighboring images according to each color attribute.

[0063] In the embodiments of this application, considering that each facial key point corresponds to a pixel, and the color value of the facial key point is easily interfered with by noise, in order to improve robustness, in S1023, a local neighborhood of a set size (such as a set neighborhood size of 9*9 pixels) is used to replace the facial key point. Then, according to the mapping relationship between facial key points, the color attributes (including color mean and standard deviation) of the facial key point pairs in their respective set neighborhoods are determined to correct the color values ​​of each facial key point in the neighboring images.

[0064] For ease of description, facial landmarks in neighboring images are designated as first facial landmarks, and facial landmarks in the reference image are designated as second facial landmarks. In practice, the color correction process for each first facial landmark in the neighboring images is described in [reference needed]. Figure 5 It mainly includes the following steps:

[0065] s1032_1: Based on the mapping relationship between facial key points in the reference image and facial key points in neighboring images, determine the second facial key point corresponding to the first facial key point in the reference image.

[0066] In S1032_1, the second facial key point, which has a mapping relationship with the first facial key point, has the same number and shares the same semantic information.

[0067] S1032_2: Determine the first color mean and first standard deviation of each pixel in each channel within the set neighborhood of the first facial key point, and determine the second color mean and second standard deviation of each pixel in each channel within the set neighborhood of the second facial key point.

[0068] In S1032_2, the setting range of the first and second facial key points contains multiple pixels. Therefore, in the Lab color space, the color mean and variance of each channel in their respective setting neighborhoods can be calculated.

[0069] Assume that within the defined neighborhood of the first facial landmark, the mean values ​​of the first color of each pixel in the L, a, and b channels are denoted as ml, ma, and mb, respectively, and the first standard deviations are denoted as nl, na, and nb, respectively. Similarly, within the defined neighborhood of the second facial landmark, the mean values ​​of the second color of each pixel in the L, a, and b channels are denoted as ml', ma', and mb', respectively, and the first standard deviations are denoted as nl', na', and nb', respectively.

[0070] S1032_3: Correct the color value of the first facial landmark based on the first color mean, second color mean, first standard deviation, and second standard deviation of each channel.

[0071] For specific implementation details, please refer to the color value correction process for the first facial key point in S1032_3. Figure 6 It mainly includes the following steps:

[0072] S1032_31: Obtain the initial color values ​​of the first face key points in the L channel, a channel and b channel respectively.

[0073] Suppose that the initial color values ​​of the first facial landmark in the L channel, a channel, and b channel are denoted as L0, a0, and b0, respectively.

[0074] S1032_32: Subtract the mean first color value of the corresponding channel from the initial color values ​​of the L channel, a channel, and b channel to obtain the intermediate color value of the first facial key point in the L channel, a channel, and b channel.

[0075] The formula for the intermediate color value is expressed as follows:

[0076]

[0077] In Formula 1, L1, a1, and b1 represent the intermediate color values ​​of the first facial landmark in the L, a, and b channels, respectively.

[0078] S1032_33: Scale the intermediate color values ​​of the corresponding channels according to the first and second standard deviations of the L, a, and b channels.

[0079] In S1032_33, the scaling factor for each channel is the ratio of the first standard deviation and the second standard deviation of the first and second facial key points within a defined neighborhood. The scaling formula is expressed as follows:

[0080]

[0081] In Formula 2, L2, a2, and b2 represent the color values ​​of the first facial key points after scaling in the L, a, and b channels, respectively.

[0082] S1032_34: Add the average second color value of the corresponding channel to the scaled color values ​​of the L channel, a channel, and b channel to obtain the corrected color values ​​of the first facial landmark in the L channel, a channel, and b channel.

[0083] The formula for the corrected color value is expressed as follows:

[0084]

[0085] In Formula 2, L′, a′, and b′ represent the color values ​​of the first facial landmarks after correction in the L, a, and b channels, respectively.

[0086] s1033: For each pixel in the neighboring images, determine the target color value of the pixel based on the color values ​​of the facial key points in the corrected neighboring images.

[0087] After local color value correction of each facial key point in the neighboring image, in s1033, the linear transformation equation from the neighboring image to the reference image can be solved based on the corrected color values ​​of each facial key point, and then applied to the entire neighboring image to complete the color correction of the entire neighboring image.

[0088] For specific implementation, please refer to Figure 7 For each pixel in the neighboring images, perform the following operations:

[0089] S1033_1: Determine the average color value of each channel corresponding to the pixel within its own defined neighborhood.

[0090] In S1033_1, the set neighborhood size of this pixel is the same as the set neighborhood size of the facial key points. Similarly, in order to reduce noise interference, the color value of this pixel is replaced with the average color value of its own set neighborhood.

[0091] S1033_2: Calculate the distance from this pixel to each facial key point in the neighboring images.

[0092] In face reconstruction, the lighting environment is fixed, and it's generally assumed that the lighting won't change drastically. In other words, the lighting changes for a face within the same image can be considered linear. Therefore, after determining the color transfer relationship between keypoints in neighboring and reference images, linear interpolation can be performed on pixels in other locations in neighboring images using a distance factor, thus making the color transformation of the entire neighboring image closer to the reference image. Specifically, the farther a keypoint is from a given pixel, the smaller its influence on that pixel's color value. Thus, different influence weights can be assigned based on the distance between the pixel and other facial keypoints; the greater the distance, the greater the weight.

[0093] S1033_3: Determine the target color value of the pixel based on the average color value of the pixel in each channel, the weight corresponding to each distance, and the color value of each facial key point in each channel after correction.

[0094] The formula for the target color value is expressed as follows:

[0095]

[0096] In Formula 4, ml, ma, and mb represent the average color values ​​of the L, a, and b channels of the pixel within its defined neighborhood, respectively, and i represents the i-th facial landmark, i = 1, 2, 3, ..., n, L. i ′、a i ′、b i '' represents the color values ​​of the L channel, a channel, and b channel after the facial landmark correction for the i-th person, respectively, and ω' represents the color values ​​of the L channel, a channel, and b channel. i The weights corresponding to the distance from the pixel to the i-th facial key point are L″, a″, and b″, which are the target color values ​​of the L, a, and b channels of the pixel.

[0097] After color correction of each neighboring image, all neighboring images have the same Lab distribution as the reference image, thus eliminating the problem of inconsistent light intensity of the same feature point under different viewing angles due to different lighting conditions.

[0098] S1034: Convert the reference image and the adjusted neighboring image from the Lab color space to the RGB color space.

[0099] In face reconstruction, RGB images are generally used for texture mapping. Therefore, after the colors of the neighboring images are made similar to those of the reference image through a color transfer algorithm, the reference image and the neighboring images are then converted from the Lab color space to the RGB color space.

[0100] For example, such as Figure 8 As shown, this is an example of the effect of adjusting the color of a neighboring image based on a reference image according to an embodiment of this application. The entire neighboring image after color adjustment retains the original color information it expresses, while also retaining the overall color tone of the reference image, making the neighboring image closer to the reference image.

[0101] In the embodiments of this application, the mapping relationship between facial key points in neighboring images and reference images is utilized to locally transfer the color values ​​of key points in the reference image to the corresponding facial key points in neighboring images. This achieves local color correction of facial key points in neighboring images, making the colors of facial key points in neighboring images closer to those in the reference image, thus reducing the impact of global color difference. Since the environment in which the face is captured is a typical daylight environment, there will be no sudden changes in color or brightness. Therefore, the stretching and deformation of each facial key point in the color space in the neighboring image can also be applied to pixels at other non-facial key point locations in the neighboring image, thereby achieving the purpose of color correction for the entire neighboring image. This solves the problem of subsequent feature point matching errors caused by different lighting conditions under different viewing angles during multi-view stereo vision face reconstruction, greatly enriching the number of feature points that can be used for matching, thus enabling high-precision face reconstruction even under sparse viewpoints.

[0102] S104: Based on the reference image and each neighboring image, determine the camera parameters of multiple cameras to align each neighboring image with the reference image.

[0103] The process of determining the intrinsic and extrinsic parameters of a camera is called camera calibration. Generally, the intrinsic parameters of each camera are pre-calibrated, and during face reconstruction, only the extrinsic parameters reflecting the relative pose between the cameras need to be determined.

[0104] In the embodiments of this application, after color correction of each adjacent image, the face images captured from each viewpoint have very similar color distributions. Furthermore, in S104, the face images can be aligned using the traditional Structure from Motion (SFM) method and Bundle Adjustment (BA) method, thereby obtaining the intrinsic and extrinsic parameters between the two cameras from each viewpoint.

[0105] The SFM algorithm determines matching 2D feature point pairs directly through the illumination consistency among multiple images without any prior calibration information. Then, based on these matching 2D feature point pairs, it initially estimates the extrinsic parameters of multiple cameras and obtains a 3D point cloud. Finally, it optimizes the camera's extrinsic parameters using the BA algorithm. The BA algorithm can also be understood as the minimum projection error algorithm; its idea is to optimize a projection error energy function to minimize the projection error, thereby obtaining the optimal camera extrinsic parameters.

[0106] Since there is no prior calibration information, the accuracy of the 2D feature point pair calculation requires high illumination consistency, resulting in a small number of matchable 2D feature points. This leads to sparse 3D point cloud data and low face reconstruction accuracy. To improve face reconstruction accuracy, the number of cameras must be increased, which increases the difficulty of camera deployment and hardware costs. Furthermore, relying solely on illumination consistency to determine the 2D feature point pairs may result in incorrectly matched 2D feature point pairs. If there are many incorrectly matched feature point pairs, it will directly affect the optimization results of camera extrinsic parameters.

[0107] To address the problems in the traditional SFM+BA algorithm, this embodiment adds the face key point pairs with mapping relationships as regularization terms to the projection error energy function when aligning face images from different viewpoints in S104, thereby improving the robustness of the calibration and obtaining more accurate camera extrinsic parameters.

[0108] In S104, by unifying the camera coordinate systems of face images acquired from multiple viewpoints into a single coordinate system, alignment of multi-view face images can be achieved. Specifically, in S104, for each neighboring image, the following steps are performed: Figure 9 The operation shown:

[0109] S104l: Determine the initial camera parameters and the first 3D point cloud corresponding to the 2D feature point pairs based on multiple 2D feature point pairs matched in the reference image and neighboring images.

[0110] In S1041, 2D feature points are extracted from the reference image and neighboring images respectively, and the intensity, mean, variance and other illumination information of each 2D feature point are calculated. Based on the illumination information, the difference between the 2D feature points in the reference image and neighboring images is calculated, and the 2D feature points in two images with a difference of less than a certain threshold are taken as a pair of matched 2D feature points. Then, based on the multiple matched 2D feature point pairs, the initial camera parameters and the first 3D point cloud corresponding to the 2D feature point pairs are determined.

[0111] S1042: Based on multiple face key point pairs that have a mapping relationship in the reference image and neighboring images, determine the second 3D point cloud corresponding to the face key point pairs.

[0112] Considering that the first 3D point cloud obtained based on 2D feature point pairs is relatively sparse, in s1042, a second 3D point cloud corresponding to the face key point pairs is obtained based on multiple face key point pairs that have a mapping relationship in the reference image and neighboring images, so as to serve as a matching constraint for 2D feature points and improve the accuracy of matching.

[0113] S1043: Construct the projection error energy function of the reference image and the neighboring image based on the initial camera parameters, the first 3D point cloud and the second 3D point cloud.

[0114] In S1043, the second 3D point cloud obtained from face keypoint pairs with mapping relationships is added as a regularization term to the traditional projection error energy function, and a weight is assigned to the regularization term to adjust its impact on the overall projection error energy function. The projection error energy function formula is expressed as follows:

[0115]

[0116] Among them, M j Let V(j) represent the j-th 3D point in the first 3D point cloud, and let V(j) represent the set of all cameras that can see the j-th 3D point in the first 3D point cloud. i This represents the initial camera parameters of the i-th camera. M represents the pixel corresponding to the j-th 3D point in the first 3D point cloud in the i-th image. n Let V(n) represent the nth 3D point in the second 3D point cloud, and let V(n) represent the set of all cameras that can see the nth 3D point in the second 3D point cloud. Let λ represent the pixel corresponding to the nth 3D point in the second 3D point cloud in the i-th image, λ represent the weight of the regularization term, and E(P,M) represent the projection error energy function.

[0117] The projection error energy function shown in Equation 5, when optimizing the initial camera parameters of the i-th camera, considers not only the sparse first 3D point cloud obtained by the SFM algorithm, but also the semantic matching added by facial key points, thus improving the matching accuracy. Here, the larger λ is, the greater the impact of the semantic matching result of facial key points on camera parameter optimization.

[0118] S1044: Optimize the projection error energy function to reduce the projection error, and use the camera parameters corresponding to the minimum projection error as the target camera parameters.

[0119] In S1044, the projection error energy function is optimized to reduce the projection error, thereby obtaining the target camera parameters that minimize the projection error.

[0120] s105: Based on the reference image and each neighboring image, a multi-view stereo vision algorithm is used to reconstruct a 3D face model.

[0121] After aligning the reference image and each neighboring image, the MVS algorithm is used to reconstruct the 3D face based on the target camera parameters of each camera, the color-corrected neighboring images, and the reference image. The MVS algorithm is mainly divided into PMVSNet (Point-Based Multi-View Stereo Network), which directly generates point clouds, and MVSNet (Depth Inference for Unstructured Multi-view Stereo), which generates depth maps, based on the different generated results.

[0122] In S105, since the colors of each neighboring image are corrected, the luminosity consistency of the same feature points under different viewpoints can be guaranteed. When performing MVS reconstruction, the number of matching point pairs can be greatly increased, and a dense 3D point cloud can be obtained, thereby improving the reconstruction accuracy and density of the 3D face model.

[0123] like Figure 10 and Figure 11 The image shown is a reconstruction result achieved using the face reconstruction method provided in this application. Figure 10 For the reconstructed facial geometry model, Figure 11 A 3D face model after texture mapping of the facial geometry. From Figure 10 As can be seen, even with sparse viewpoints, a high-precision 3D face model can be reconstructed, reducing the reconstruction cost and difficulty of multi-view stereo vision.

[0124] The face reconstruction method based on keypoint recognition provided in this application uses a frontal view face image as a reference image and the rest as neighboring images. Facial keypoints are extracted from each image. Based on the mapping relationship between the reference image and the facial keypoints in each neighboring image, the local color of the corresponding facial keypoints in the neighboring images is corrected using the local color of the facial keypoints in the reference image. The colors of each facial keypoint in the neighboring images are then transferred to the entire image. This solves the problem of subsequent feature point matching errors caused by different lighting conditions under different viewpoints, enriches the number of matched feature point pairs, and thus reduces the number of viewpoints. This ensures the reconstruction of high-precision face models under sparse viewpoints (up to ten), reducing hardware costs. Furthermore, when determining the target camera parameters for multiple cameras, face keypoint pairs with mapping relationships are added as regularization terms to the projection error energy function. This allows semantic matching between face keypoints to be used as a constraint, improving matching accuracy and obtaining optimal target camera parameters. By combining color-corrected neighboring images and reference images, multi-view 3D face model reconstruction is achieved. During the reconstruction process, face keypoints are introduced as auxiliary information, thus preserving the high-precision pixel-by-pixel reconstruction characteristics of multi-view stereo vision algorithms under sparse viewpoints.

[0125] Based on the same technical concept, embodiments of this application provide a reconstruction device. This reconstruction device can be a client device with data processing capabilities, such as a laptop, desktop computer, smartphone, tablet, VR glasses, or AR glasses. It can also be a server used to implement the interaction process, including but not limited to microservers, cloud servers, and server clusters. When the reconstruction device is a client, its own camera can be used directly. When the reconstruction device is a server, multiple RGB cameras can be deployed independently.

[0126] The reconstruction device described above can implement the steps of the face reconstruction method based on key point recognition in the above embodiments and achieve the same technical effect, which will not be elaborated here.

[0127] See Figure 12 The reconstruction device includes a processor 1201, a memory 1202, a communication interface 1203, and a display screen 1204. The display screen 1204, the communication interface 1203, the memory 1202, and the processor 1201 are connected via a bus 1205.

[0128] The memory 1202 stores a computer program, and the processor 1201 performs the following operations according to the computer program:

[0129] The communication interface 1203 is used to acquire face images captured by the camera from multiple perspectives, and the face image from the frontal perspective is used as the reference image, while the face images from other perspectives are used as neighboring images.

[0130] Identify the facial key points in the face image from each viewpoint;

[0131] For each neighboring image, based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring images, the color values ​​of pixels in the reference image are used to correct the color values ​​of pixels in the neighboring images.

[0132] Based on the reference image and each neighboring image, camera parameters of multiple cameras are determined to align each neighboring image with the reference image;

[0133] Based on the reference image and each of the neighboring images, a 3D face model is reconstructed using a multi-view stereo vision algorithm and displayed on the display screen 1204.

[0134] Optionally, the processor 1201 corrects the color values ​​of pixels in the neighboring images using the color values ​​of pixels in the reference image, based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring images. Specifically, the operation is as follows:

[0135] Convert the reference image and the neighboring image from the RGB color space to the Lab color space;

[0136] Based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring image, the color attributes of each channel of the facial key point pairs with mapping relationship are determined in their respective set neighborhoods, and the color values ​​of the corresponding facial key points in each channel in the neighboring image are corrected according to each color attribute.

[0137] For each pixel in the neighboring image, the target color value of the pixel is determined based on the color values ​​of each facial key point in the corrected neighboring image.

[0138] The reference image and the adjusted neighboring image are converted from the Lab color space to the RGB color space.

[0139] Optionally, the color attributes include color mean and standard deviation. The processor 1201 determines the color attributes of each channel in its respective defined neighborhood for face keypoint pairs with mapping relationships based on the mapping relationship between face keypoints in the reference image and face keypoints in the neighboring images. Based on these color attributes, the processor corrects the color values ​​of the corresponding face keypoints in each channel of the neighboring images. Specifically, the processor 1201 performs the following operations:

[0140] For each first facial landmark in the neighboring images, perform the following operations:

[0141] Based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring image, determine the second facial key point corresponding to the first facial key point in the reference image;

[0142] The first color mean and first standard deviation of each pixel in each channel within the defined neighborhood of the first facial key point are determined, and the second color mean and second standard deviation of each pixel in each channel within the defined neighborhood of the second facial key point are determined.

[0143] The color values ​​of the first facial landmarks are corrected based on the first color mean, the second color mean, the first standard deviation, and the second standard deviation for each channel.

[0144] Optionally, the processor 1201 corrects the color values ​​of the first facial key points based on the first color mean, the second color mean, the first standard deviation, and the second standard deviation of each channel. Specifically, the operation is as follows:

[0145] Obtain the initial color values ​​of the first facial key points in the L channel, a channel, and b channel, respectively;

[0146] The first color mean of the corresponding channel is subtracted from the initial color values ​​of the L channel, a channel and b channel respectively to obtain the intermediate color value of the first facial key point in the L channel, a channel and b channel;

[0147] Based on the first and second standard deviations of the L, a, and b channels, the intermediate color values ​​of the corresponding channels are scaled.

[0148] The scaled color values ​​of the L, a, and b channels are added to the second color mean of the corresponding channels to obtain the corrected color values ​​of the first facial key points in the L, a, and b channels.

[0149] Optionally, for each pixel in the neighboring images, the processor 120l adjusts the color value of the pixel based on the color values ​​of each facial key point in the corrected neighboring images. Specifically, the operation is as follows:

[0150] Determine the average color value of each channel corresponding to the pixel within its own defined neighborhood;

[0151] Calculate the distance from the pixel to each facial key point in the neighboring image;

[0152] The target color value of the pixel is determined based on the average color value of the pixel in each channel, the weight corresponding to each distance, and the color value of each facial key point in each channel after correction.

[0153] Optionally, the formula for calculating the target color value of the pixel is:

[0154]

[0155] Where L″, a″, and b″ represent the target color values ​​of the L, a, and b channels of the pixel, respectively. i ′、a i ′、b i ' represents the color values ​​of the L, a, and b channels of the i-th facial landmark after correction, respectively; ml, ma, and mb represent the average color values ​​of the L, a, and b channels of the pixel within its defined neighborhood, respectively; and i represents the number of facial landmarks, i = 1, 2, 3, ..., n, ω. i The weight is the distance between the pixel and the i-th facial key point.

[0156] Optionally, the processor 1201 determines camera parameters for multiple cameras based on the reference image and each neighboring image, specifically by:

[0157] For each neighboring image, perform the following operations:

[0158] Based on the reference image and the multiple 2D feature point pairs matched in the neighboring images, the initial camera parameters and the first 3D point cloud corresponding to the 2D feature point pairs are determined.

[0159] Based on multiple facial key point pairs that have a mapping relationship in the reference image and the neighboring images, determine the second 3D point cloud corresponding to the facial key point pairs;

[0160] Based on the initial camera parameters, the first 3D point cloud, and the second 3D point cloud, construct the projection error energy function of the reference image and the neighboring image;

[0161] The projection error energy function is optimized to reduce the projection error, and the camera parameters corresponding to the minimum projection error are used as the target camera parameters.

[0162] Optionally, the projection error energy function formula is as follows:

[0163]

[0164] Among them, M j Let V(j) represent the j-th 3D point in the first 3D point cloud, and let V(j) represent the set of all cameras that can see the j-th 3D point in the first 3D point cloud.i This represents the initial camera parameters of the i-th camera. Let Mn represent the pixel corresponding to the j-th 3D point in the first 3D point cloud in the i-th image, Mn represent the n-th 3D point in the second 3D point cloud, and V(n) represent the set of all cameras that can see the n-th 3D point in the second 3D point cloud. Let λ represent the pixel point corresponding to the nth 3D point in the second 3D point cloud in the i-th image, λ represent the weight of the regularization term, and E(P,M) represent the projection error energy function.

[0165] Optionally, the mapping relationship between facial key points in the reference image and facial key points in the neighboring image is established based on the semantic information of the facial key points.

[0166] It should be noted that, Figure 12 This is merely an example illustrating the hardware necessary for a reconstruction device to implement the face reconstruction method based on key point recognition provided in this application embodiment. As not shown, the reconstruction device may include conventional components such as microphones, speakers, audio / video processors, and power supplies.

[0167] Examples of this application Figure 12 The processor involved can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0168] See Figure 13 This is a functional structure diagram of the reconstruction device provided in the embodiments of this application. The reconstruction device mainly includes an image acquisition module 1301, a key point recognition module 1302, a color correction module 1303, an image alignment module 1304, and a reconstruction module 1305, wherein:

[0169] The image acquisition module 1301 is used to acquire face images captured by the camera from multiple perspectives, and to use the face image from the frontal perspective as the reference image and the face images from other perspectives as neighboring images.

[0170] Key point recognition module 1302 is used to identify facial key points in the face image from each viewpoint;

[0171] The color correction module 1303 is used to correct the color value of the pixels in the neighboring images based on the mapping relationship between the facial key points in the reference image and the facial key points in the neighboring images, using the color value of the pixels in the reference image.

[0172] The image alignment module 1304 is used to determine the camera parameters of multiple cameras based on the reference image and each neighboring image in order to align each neighboring image with the reference image.

[0173] The reconstruction module 1305 is used to reconstruct a 3D face model based on the reference image and each of the neighboring images using a multi-view stereo vision algorithm.

[0174] The specific implementation of each of the above functional modules can be found in the aforementioned embodiments, and will not be described again here.

[0175] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform a face reconstruction method based on key point recognition as described in the foregoing embodiments.

[0176] This application also provides a computer program product for storing a computer program for executing a face reconstruction method based on key point recognition as described in the foregoing embodiments.

[0177] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0179] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0180] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0181] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A face reconstruction method based on key point recognition, characterized in that, include: Acquire face images captured by the camera from multiple perspectives, and use the face image from the frontal view as the reference image and the face images from other perspectives as neighboring images; Identify the facial key points in the face image from each viewpoint; For each neighboring image, based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring images, the color values ​​of pixels in the reference image are used to correct the color values ​​of pixels in the neighboring images. Based on the reference image and each neighboring image, camera parameters of multiple cameras are determined to align each neighboring image with the reference image; Based on the reference image and each of the neighboring images, a 3D face model is reconstructed using a multi-view stereo vision algorithm. The step of correcting the color values ​​of pixels in the neighboring images using the color values ​​of pixels in the reference image based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring images includes: Convert the reference image and the neighboring image from the RGB color space to the Lab color space; Based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring image, the color attributes of each channel of the facial key point pairs with mapping relationship are determined in their respective set neighborhoods, and the color values ​​of the corresponding facial key points in each channel in the neighboring image are corrected according to each color attribute. For each pixel in the neighboring image, determine the average color value of each channel corresponding to the pixel in its own set neighborhood, calculate the distance from the pixel to each facial key point in the neighboring image, and determine the target color value of the pixel based on the average color value of the pixel in each channel, the weight corresponding to each distance, and the corrected color value of each facial key point in each channel. The reference image and the adjusted neighboring image are converted from the Lab color space to the RGB color space.

2. The method as described in claim 1, characterized in that, The color attributes include color mean and standard deviation. The step of determining the color attributes of each channel in a defined neighborhood of facial keypoint pairs with a mapping relationship between facial keypoints in the reference image and facial keypoints in neighboring images, and correcting the color values ​​of corresponding facial keypoints in each channel in the neighboring images based on these color attributes, includes: For each first facial landmark in the neighboring images, perform the following operations: Based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring image, determine the second facial key point corresponding to the first facial key point in the reference image; The first color mean and first standard deviation of each pixel in each channel within the defined neighborhood of the first facial key point are determined, and the second color mean and second standard deviation of each pixel in each channel within the defined neighborhood of the second facial key point are determined. The color values ​​of the first facial landmarks are corrected based on the first color mean, the second color mean, the first standard deviation, and the second standard deviation for each channel.

3. The method as described in claim 2, characterized in that, The step of correcting the color values ​​of the first facial landmarks based on the first color mean, the second color mean, the first standard deviation, and the second standard deviation of each channel includes: Obtain the initial color values ​​of the first facial key points in the L channel, a channel, and b channel, respectively; The first color mean of the corresponding channel is subtracted from the initial color values ​​of the L channel, a channel and b channel respectively to obtain the intermediate color value of the first facial key point in the L channel, a channel and b channel; Based on the first and second standard deviations of the L, a, and b channels, the intermediate color values ​​of the corresponding channels are scaled. The scaled color values ​​of the L, a, and b channels are added to the second color mean of the corresponding channels to obtain the corrected color values ​​of the first facial key points in the L, a, and b channels.

4. The method as described in claim 1, characterized in that, The formula for calculating the target color value of the pixel is: in, , , These represent the target color values ​​of the L channel, a channel, and b channel of the pixel, respectively. , , These represent the color values ​​of the L channel, a channel, and b channel after the facial landmarks of the i-th person are corrected. , , These represent the average color values ​​of the L, a, and b channels of the pixel within its defined neighborhood, respectively, where i represents the number of facial landmarks, i = 1, 2, 3, ..., n. The weight is the distance between the pixel and the i-th facial key point.

5. The method as described in claim 1, characterized in that, The step of determining camera parameters for multiple cameras based on the reference image and each neighboring image includes: For each neighboring image, perform the following operations: Based on the reference image and the multiple 2D feature point pairs matched in the neighboring images, the initial camera parameters and the first 3D point cloud corresponding to the 2D feature point pairs are determined. Based on multiple facial key point pairs that have a mapping relationship in the reference image and the neighboring images, determine the second 3D point cloud corresponding to the facial key point pairs; Based on the initial camera parameters, the first 3D point cloud, and the second 3D point cloud, construct the projection error energy function of the reference image and the neighboring image; The projection error energy function is optimized to reduce the projection error, and the camera parameters corresponding to the minimum projection error are used as the target camera parameters.

6. The method as described in claim 5, characterized in that, The formula for the projection error energy function is as follows: in, This represents the j-th 3D point in the first 3D point cloud. This indicates the set of all cameras that can see the j-th 3D point in the first 3D point cloud. This represents the initial camera parameters of the i-th camera. This represents the pixel corresponding to the j-th 3D point in the first 3D point cloud in the i-th image. This represents the nth 3D point in the second 3D point cloud. This indicates the set of all cameras that can see the nth 3D point in the second 3D point cloud. This represents the pixel corresponding to the nth 3D point in the second 3D point cloud in the i-th image. This represents the weight of the regularization term. This represents the projection error energy function.

7. The method according to any one of claims 1-6, characterized in that, The mapping relationship between facial key points in the reference image and facial key points in the neighboring images is established based on the semantic information of the facial key points.

8. A reconstruction device, characterized in that, It includes a processor, a memory, a communication interface, and a display screen, wherein the communication interface, the display screen, the memory, and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: The communication interface is used to acquire face images captured by the camera from multiple perspectives, and the face image from the frontal perspective is used as the reference image, while the face images from other perspectives are used as neighboring images. Identify the facial key points in the face image from each viewpoint; For each neighboring image, based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring images, the color values ​​of pixels in the reference image are used to correct the color values ​​of pixels in the neighboring images. Based on the reference image and each neighboring image, camera parameters of multiple cameras are determined to align each neighboring image with the reference image; Based on the reference image and each of the neighboring images, a 3D face model is reconstructed using a multi-view stereo vision algorithm and displayed on the display screen. Specifically, the processor, based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring images, uses the color values ​​of pixels in the reference image to correct the color values ​​of pixels in the neighboring images. The specific operation is as follows: Convert the reference image and the neighboring image from the RGB color space to the Lab color space; Based on the mapping relationship between facial key points in the reference image and facial key points in the neighboring image, the color attributes of each channel of the facial key point pairs with mapping relationship are determined in their respective set neighborhoods, and the color values ​​of the corresponding facial key points in each channel in the neighboring image are corrected according to each color attribute. For each pixel in the neighboring image, determine the average color value of each channel corresponding to the pixel in its own set neighborhood, calculate the distance from the pixel to each facial key point in the neighboring image, and determine the target color value of the pixel based on the average color value of the pixel in each channel, the weight corresponding to each distance, and the corrected color value of each facial key point in each channel. The reference image and the adjusted neighboring image are converted from the Lab color space to the RGB color space.

Images