A face image face filling method, system and storage medium

Abstract

The application discloses a face image face filling method and system and a storage medium, and the method comprises the following steps: acquiring an input image, performing face feature extraction, and obtaining two-dimensional face features; obtaining two-dimensional face texture feature information according to the two-dimensional face features, and calculating the illumination coefficient of the input image; performing three-dimensional face reconstruction based on the two-dimensional face features and the two-dimensional face texture feature information, combining a preset three-dimensional face model, obtaining a first three-dimensional face model; performing morphological filling processing on the first three-dimensional model, obtaining a second three-dimensional face model, and then adding the illumination coefficient to obtain a third three-dimensional face model; and mapping the third three-dimensional face model to a two-dimensional plane to obtain a face image face filling effect diagram. Through the three-dimensional face reconstruction and morphological filling processing based on the preset three-dimensional face model, the filling effect diagram is more realistic, natural and stereoscopic, and the user can comprehensively evaluate the risk of medical beauty filling.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system and storage medium for facial image filling. Background Technology

[0002] The face is a highly complex and precise area of ​​tissue structure. Its surface can be divided into dozens of cosmetic subunits, while its interior is mainly supported by dozens of fat compartments, forming the overall appearance of the face. As we age, the volume of fat in these fat compartments changes continuously, and a person's appearance changes accordingly. If the fat volume in certain areas is smaller, these areas will appear sunken, making a person look older and less attractive.

[0003] Facial depressions are mainly categorized into the forehead, brow bone, eye sockets, temples, tear troughs, cheekbones, nasal base, and chin. Therefore, filling these areas—cheekbones, forehead, brow bone, and chin—has a significant effect on improving facial appearance. If these areas are sunken, it creates a hollowed-out appearance, making a person look older, and even with otherwise good features, it's difficult to appear beautiful. Secondly, filling the temples to make them fuller also has an immediate effect on improving appearance. Furthermore, aging causes the body's fat to atrophy, which often occurs at the nasal base and eye sockets. Therefore, filling these areas with fat can significantly reduce the appearance of aging, making one look younger and more energetic.

[0004] Cosmetic procedures can fill in sunken areas to make users appear younger. Similarly, facial filler in images can serve as an effective simulation of cosmetic procedures, allowing users to more comprehensively assess the risks. Furthermore, in the field of image enhancement, beauty enthusiasts also hope to fill in facial features in photo images to make their faces appear fuller and younger in photos.

[0005] However, existing facial filling methods mainly use filtering and other techniques to fill faces in two-dimensional images, resulting in a lack of three-dimensionality and poor final presentation. This makes it impossible for users to fully assess the risks of cosmetic fillers. Summary of the Invention

[0006] The main objective of this invention is to provide a facial image filling method, system, and storage medium, which aims to solve the technical problem that existing facial image filling methods lack three-dimensionality, resulting in poor final presentation effects and failing to provide users with a comprehensive assessment of the risks of cosmetic filling.

[0007] To achieve the above objectives, the present invention provides a face filling method for facial images, comprising the following steps: acquiring an input image and extracting facial features to obtain two-dimensional facial features; obtaining two-dimensional facial texture feature information based on the two-dimensional facial features; calculating the illumination coefficient of the input image based on the two-dimensional facial features; reconstructing a three-dimensional face based on the two-dimensional facial features and the two-dimensional facial texture feature information, combined with a preset three-dimensional facial model, to obtain a first three-dimensional facial model; performing deformation filling processing on the first three-dimensional model to obtain a second three-dimensional facial model; adding illumination coefficients to the second three-dimensional model to obtain a third three-dimensional facial model; and mapping the third three-dimensional facial model onto a two-dimensional plane to obtain a face filling effect image.

[0008] Optionally, based on two-dimensional facial features and two-dimensional facial texture features, and combined with a preset three-dimensional facial model, a three-dimensional facial reconstruction is performed to obtain a first three-dimensional facial model. Specifically, this includes the following steps: based on two-dimensional facial features and two-dimensional facial texture features, alignment and registration processing are performed, and a three-dimensional facial reconstruction is performed in combination with the preset three-dimensional facial model to obtain an initial three-dimensional facial model; wherein, the preset three-dimensional facial model is obtained through facial feature extraction or is a standard three-dimensional facial model; after performing shape optimization and texture optimization processing on the initial three-dimensional facial model, the first three-dimensional facial model is obtained.

[0009] Optionally, an input image is acquired and facial features are extracted to obtain two-dimensional facial features. Specifically, the following steps are included: acquiring the input image and processing it to obtain a processed image; inputting the processed image into a deep learning model to extract facial features to obtain two-dimensional facial features, wherein the two-dimensional facial features include the bounding box of the face, facial key points, and the confidence level of the face region.

[0010] Optionally, image processing includes image conversion processing and image adjustment processing. Specifically, image conversion processing involves converting the input image into a grayscale image, and image adjustment processing involves adjusting the brightness and contrast of the grayscale image.

[0011] Optionally, based on two-dimensional facial feature points, two-dimensional facial texture feature information is obtained. Specifically, based on the facial bounding box and facial key points, the pixels within the facial bounding box are sampled and interpolated using an image interpolation algorithm to obtain two-dimensional facial texture feature information. The calculation process of the illumination coefficient of the input image specifically includes the following steps: based on the facial key points, the minimum bounding rectangle of the face is calculated, and the extracted facial region is projected onto a unit hemisphere to generate a spherical harmonic function. The order of the spherical harmonic function is determined according to the required precision. The extracted facial region is divided into several blocks, the RGB values ​​of the pixels in each block are calculated, and they are converted into function values ​​in the spherical coordinate system. Then, the spherical harmonic function coefficients are fitted using the least squares method to obtain the illumination coefficient β.

[0012] Optionally, after calculating the illumination coefficient of the input image based on the two-dimensional facial features, the input image is then subjected to illumination filtering before three-dimensional face reconstruction. Specifically, illumination filtering of the input image involves calculating the illumination intensity under different light source directions based on the spherical harmonic function and the illumination coefficient β. Specifically, for each pixel, its function value in the spherical coordinate system is calculated, and the spherical harmonic function and the illumination coefficient β are summed to obtain the illumination intensity. The shadow value of the pixel is obtained by calculating the difference between the illumination intensity of each pixel in the input image and the average illumination intensity value of the surrounding pixels. This shadow value is then added back to the RGB value of the pixel to achieve illumination filtering.

[0013] Optionally, the first 3D model is deformed and filled to obtain the second 3D face model. This involves the following steps: The first 3D face model, Face3D, is represented as: Face3D = M + S * A + T * B, where M is the average shape of the 3D face model, S is the shape vector, A is the coefficient of the shape vector, T is the 3D face texture vector, and B is the coefficient of the 3D face texture vector, specifically the average value obtained through principal component analysis of the preset 3D face model; Based on the 3D feature points of the first 3D model, the 3D coordinates of each feature point are calculated and arranged into a vector to obtain the shape vector S; Based on the 3D feature points of the first 3D model, the RGB values ​​of each feature point are obtained and arranged into a vector to obtain the texture vector T; The standard shape vector ShapeVector_Model of the standard 3D face model is obtained, and the difference between the shape vector S and the standard shape vector ShapeVector_Model is calculated, with the difference used as the user's face filling degree value ΔS; Based on the face region extracted from the face features, a preset face feature point region is further divided; Based on the preset face feature point region, the user's face is filled... The fill level value ΔS is subdivided into the first subdivision value to obtain the fill level value corresponding to the preset facial feature point region. Based on the preset facial feature point region, the shape vector S is subdivided into the second subdivision value to obtain the first shape vector corresponding to the preset facial feature point region. The fill level value corresponding to the required fill region is found in the fill level value ΔS of the user's face and added to the shape vector of the required fill region to obtain the set of deformed feature shape vectors S_reshape. The second shape vector of the corresponding region after adding the fill level value is set as the constraint feature shape vector of the set of deformed feature shape vectors S_reshapede. The constraint feature shape vector is interpolated and fitted based on the deformation method of third-order Laplacian coordinates to obtain the deformed shape vector. According to the feature shape vector S_reshape obtained by interpolation and fitting, the second three-dimensional face model is represented as: Face3D_reshape=M+S_reshape*A+T*B, thus obtaining the second three-dimensional face model.

[0014] Optionally, the preset facial feature point regions include at least the forehead region, brow bone region, cheekbone region, chin region, eye socket region, temple contour region, tear trough region, nasal base region, and other regions; the preset facial feature point regions' corresponding fill values ​​include at least the forehead region fill value ΔS_forehead, the brow bone region fill value ΔS_browArch, the cheekbone region fill value ΔS_plumpcheeks, and the chin region fill value ΔS_chi n, the filling degree values ​​for the eye socket region (ΔS_eyesocket), temple contour region (ΔS_temple), tear trough region (ΔS_tearTrough), nasal base region (ΔS_smileline), and other regions (ΔS_other); the first shape vector includes at least the first forehead feature shape vector S_foreHead, the first brow arch feature shape vector S_browArch, the first cheekbone feature shape vector S_plumpcheeks, the first chin feature shape vector S_chin, the first eye socket feature shape vector S_eyesocket, the first temple feature shape vector S_temple, the first tear trough feature shape vector S_teartrough, the first nasal base feature shape vector S_smileline, and the first other shape vector S_other; the second shape vector includes at least the second forehead feature shape vector S'_foreHead, the second brow arch feature shape vector S'_browArch, the second cheekbone feature shape vector S'_plumpcheeks, the second chin feature shape vector S'_chin, and the second eye socket feature shape vector S'_eyesocke t, second temple feature shape vector S'_temple, second tear trough feature shape vector S'_teartrough, second nasal base feature shape vector S'_smileline, second other shape vector S'_other.

[0015] Corresponding to the aforementioned face image facial filling method, this invention provides a face image facial filling system, comprising: a face feature extraction module for acquiring an input image and extracting face features to obtain two-dimensional face features; a two-dimensional face texture feature information acquisition module for obtaining two-dimensional face texture feature information based on the two-dimensional face features; an illumination coefficient calculation module for calculating the illumination coefficient of the input image based on the two-dimensional face features; a three-dimensional face reconstruction module for performing three-dimensional face reconstruction based on the two-dimensional face features and two-dimensional face texture feature information, combined with a preset three-dimensional face model, to obtain a first three-dimensional face model; a deformation filling processing module for performing deformation filling processing on the first three-dimensional model to obtain a second three-dimensional face model; adding illumination coefficients to the second three-dimensional model to obtain a third three-dimensional face model; and a mapping module for mapping the third three-dimensional face model onto a two-dimensional plane to obtain a face image facial filling effect diagram.

[0016] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a face image face filling program, which, when executed by a processor, implements the steps of the face image face filling method described above.

[0017] The beneficial effects of this invention are:

[0018] (1) Compared with the prior art, the present invention combines a preset three-dimensional face model to perform three-dimensional face reconstruction, and the three-dimensional face reconstruction result is more accurate. Combined with deformation filling processing, the filling effect image is more realistic, natural and three-dimensional. It can help users to comprehensively assess the risks of medical aesthetic filling and help users to comprehensively assess the expected effect of medical aesthetic filling.

[0019] (2) Compared with the prior art, the present invention uses alignment and registration processing to map the input image to a preset three-dimensional face model, performs three-dimensional face reconstruction, and then performs three-dimensional face reconstruction again, so that it can more accurately match the actual face shape of the input image.

[0020] (3) Compared with the prior art, the present invention can improve the robustness and accuracy of facial feature extraction through image conversion processing and image adjustment processing;

[0021] (4) Compared with the prior art, the present invention can improve the accuracy and precision of three-dimensional face reconstruction by calculating the illumination coefficient β and performing illumination filtering on the two-dimensional image;

[0022] (5) Compared with the prior art, the present invention performs deformation filling processing in different areas, which makes the filling effect more natural and realistic, and can obtain a full and three-dimensional human face image, allowing users to comprehensively assess the risks of medical aesthetic filling. Moreover, the operation process is simple and more efficient. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0024] Figure 1 This is a simplified flowchart of the facial image filling method of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] like Figure 1 As shown, a facial image filling method of the present invention includes the following steps: acquiring an input image and extracting facial features to obtain two-dimensional facial features; obtaining two-dimensional facial texture feature information based on the two-dimensional facial features; calculating the illumination coefficient of the input image based on the two-dimensional facial features; reconstructing a three-dimensional face based on the two-dimensional facial features and the two-dimensional facial texture feature information, combined with a preset three-dimensional facial model, to obtain a first three-dimensional facial model; performing deformation filling processing on the first three-dimensional model to obtain a second three-dimensional facial model; adding illumination coefficients to the second three-dimensional model to obtain a third three-dimensional facial model; and mapping the third three-dimensional facial model onto a two-dimensional plane to obtain a facial image filling effect diagram.

[0027] This invention combines a preset 3D face model with 3D face reconstruction, resulting in more accurate 3D face reconstruction results. Combined with deformation filling processing, the filling effect image is more realistic, natural, and has a stronger three-dimensional effect, enabling users to comprehensively assess the risks of cosmetic filling.

[0028] In this embodiment, a first three-dimensional face model is obtained by reconstructing a three-dimensional face based on two-dimensional facial features and two-dimensional facial texture features, combined with a preset three-dimensional face model. Specifically, the steps include: performing alignment and registration processing based on two-dimensional facial features and two-dimensional facial texture features, and then reconstructing a three-dimensional face based on the preset three-dimensional face model to obtain an initialized three-dimensional face model; wherein, the preset three-dimensional face model is obtained through facial feature extraction or is a standard three-dimensional face model; after performing shape optimization and texture optimization processing on the initialized three-dimensional face model, the first three-dimensional face model is obtained.

[0029] Preferably, the three-dimensional model initialization in this invention uses a method based on 3D Morphable Model (3DMM). 3DMM is a statistical model used to describe the shape and texture of a face, which can be learned from a large amount of three-dimensional face scanning data. Through alignment and registration processing based on two-dimensional face features, the input image is mapped to the 3DMM to generate an initial three-dimensional face model.

[0030] Preferably, shape optimization specifically utilizes an Iterative Closest Point (ICP) algorithm to optimize the shape of the 3D model. This algorithm achieves optimization by finding the minimum distance between the 3D model and the input image. Specifically, shape regularization methods, weight balancing methods, and other techniques can be used to balance shape variations and maintain stability, making it more accurately match the user's actual face shape.

[0031] Preferably, texture optimization refers to optimizing the texture information of the 3D model to more accurately match the user's actual facial texture. This embodiment uses a nonlinear optimization method based on gradient descent to achieve texture optimization by minimizing the difference between the 3D model and the input image, combined with local optimization strategies to improve the optimization effect.

[0032] This invention uses alignment and registration processes to map the input image to a preset 3D face model, performs 3D face reconstruction, and then performs another 3D face reconstruction, so that it can more accurately match the actual face shape of the input image.

[0033] In this embodiment, the process of acquiring an input image and extracting facial features to obtain two-dimensional facial features includes the following steps: acquiring an input image and processing it to obtain a processed image; inputting the processed image into a deep learning model for facial feature extraction to obtain two-dimensional facial features, wherein the two-dimensional facial features include the bounding box of the face, facial key points, and the confidence level of the face region.

[0034] Preferably, the deep learning model is RetinaFace, which is a face detection algorithm based on a single-stage object detector. Its main idea is to directly regress the bounding box, facial landmarks, and confidence of the face region by using a fully convolutional neural network.

[0035] For each input image, the RetinaFace algorithm generates a set of predictions, including the bounding box of each detected face, facial landmarks, and the confidence score of the face region. Preferably, based on the confidence score, the face detection results with higher confidence scores are selected and retained.

[0036] The RetinaFace model structure mainly consists of a base network and three sub-networks. The base network uses the ResNet network structure to extract features from the input image. The three sub-networks are used to predict the bounding box of the face, facial landmarks, and the confidence score of the face region, respectively.

[0037] In this embodiment, image processing includes image conversion processing and image adjustment processing. Specifically, image conversion processing involves converting the input image into a grayscale image, and image adjustment processing involves adjusting the brightness and contrast of the grayscale image.

[0038] This invention improves the robustness and accuracy of facial feature extraction through image conversion and image adjustment processing.

[0039] In this embodiment, two-dimensional facial texture feature information is obtained based on two-dimensional facial feature points. Specifically, based on the facial bounding box and facial key points, the pixels within the facial bounding box are sampled and interpolated using an image interpolation algorithm to obtain two-dimensional facial texture feature information. The calculation process of the illumination coefficient of the input image specifically includes the following steps: Based on the facial key points, the minimum bounding rectangle FaceRect is calculated, and the extracted facial region is projected onto a unit hemisphere for illumination estimation on a spherical harmonic function; a spherical harmonic function is generated, which is a set of basis functions used to describe the function distribution on a sphere. Predefined spherical harmonic function coefficients can be used to represent the illumination intensity under different light source directions; the order of the spherical harmonic function is determined according to the required accuracy; the extracted facial region is divided into several blocks, the RGB values ​​of the pixels in each block are calculated, and they are converted into function values ​​in a spherical coordinate system. Then, the spherical harmonic function coefficients are fitted using the least squares method to obtain the illumination coefficient β.

[0040] Spherical harmonic functions are mathematical functions used to describe phenomena such as illumination, reflection, and radiation in three-dimensional space. Higher orders of spherical harmonic functions represent more detailed illumination or radiation distributions, but also require more computational resources and memory. Therefore, the required precision is usually determined by weighing and selecting based on the specific application requirements and the availability of computational resources.

[0041] Generally, lower spherical harmonic order (e.g., 1 to 4th order) can be used for coarse approximations or fast calculations, suitable for real-time or high-performance scenarios. Higher spherical harmonic order (e.g., 5th order and above) can be used for more refined lighting or radiation simulations, suitable for scenarios with high requirements for lighting or radiation distribution.

[0042] In practical applications, experiments and optimizations can be conducted as needed. By observing the performance of the generated spherical harmonic function on a sphere, and considering the availability of computing resources and performance requirements, an appropriate order of the spherical harmonic function can be selected to achieve the required level of precision.

[0043] Since this invention requires more precise lighting and minimizes processing time, the order of the spherical harmonic function is preferably 5.

[0044] In this embodiment, based on the detected bounding box and key point positions of the face, an image interpolation algorithm is used to sample and interpolate the pixels within the face bounding box, thereby obtaining high-resolution facial texture feature information. Simultaneously, 2D feature point information can be used for the segmentation and recognition of key facial regions, including the eye region, mouth region, nose region, forehead region, cheek region, and chin region.

[0045] In this embodiment, after calculating the illumination coefficient of the input image based on the two-dimensional face features, the input image is then subjected to illumination filtering before three-dimensional face reconstruction. The illumination filtering of the input image specifically involves calculating the illumination intensity under different light source directions based on the spherical harmonic function and the illumination coefficient β. Specifically, for each pixel, its function value in the spherical coordinate system is calculated, and the illumination intensity is obtained by summing the spherical harmonic function and the illumination coefficient β.

[0046] Preferably, for each pixel, its function value in spherical coordinates is calculated, and the illumination intensity is obtained by summing the spherical harmonic function and the illumination coefficient β. This further includes the following steps:

[0047] S1. For each pixel, calculate the function value of the spherical harmonic function in spherical coordinates;

[0048] Spherical harmonics are typically represented using SH (Spherical Harmonics) coefficients, which can be obtained either pre-computed or in real-time. The order of the spherical harmonic is determined by the required accuracy; generally, higher orders offer higher accuracy but also greater computational cost.

[0049] S2. Multiply the value of the spherical harmonic function by the illumination coefficient β;

[0050] The illumination coefficient β typically represents information such as the intensity and color of the light source, and is set according to the specific scene and requirements;

[0051] S3. At each pixel, sum the calculation results obtained in step S2; this takes into account the contribution of different light source directions to the illumination and obtains the final illumination intensity.

[0052] By calculating the difference between the light intensity of each pixel in the input image and the average light intensity of the surrounding pixels, the shadow value of that pixel is obtained; then, the shadow value is added back to the RGB value of that pixel, and the shadow can be removed, thus achieving light filtering.

[0053] Preferably, the surrounding pixels are the other pixels within a circle centered on each pixel and with a radius of 7 pixels. That is, the shadow value of a pixel is obtained by calculating the difference between the average illumination intensity value of each pixel and the average illumination intensity value of the pixels within a radius of 7 pixels.

[0054] This invention improves the accuracy and precision of 3D face reconstruction by calculating the illumination coefficient β and applying illumination filtering to 2D images.

[0055] In this embodiment, the first 3D model is deformed and filled to obtain the second 3D face model, specifically including the following steps:

[0056] The first 3D face model, Face3D, is represented as: Face3D = M + S * A + T * B, where M is the average shape of the 3D face model, S is the shape vector, A is the coefficient of the shape vector, T is the 3D face texture vector, and B is the coefficient of the 3D face texture vector, specifically the average value obtained by performing principal component analysis (PCA) on the preset 3D face model;

[0057] It should be noted that Face3D = M + S * A + T * B. In this formula, M and A are the parameters of the preset 3D face model (or standard face model), which can be obtained through existing technology.

[0058] For example, M typically collects a set of representative face data, aligns and registers these face data, and then calculates the average shape vector of all faces in this set of face data.

[0059] Preferably, a dataset of known 3D face models is used, from which representative faces are selected as the training set. These face data are aligned to a reference coordinate system, and then the face shape vectors of these aligned faces are averaged. A is the coefficient calculated based on the average shape vector, and B is the coefficient of the 3D texture vector calculated based on the preset 3D face model.

[0060] Based on the three-dimensional feature points of the first three-dimensional model, the three-dimensional coordinates of each feature point are calculated and arranged into a vector to obtain the shape vector S. Specifically, assuming there are m facial feature points, the three-dimensional coordinates of each feature point are (X... i ,Y i Z i If ), and 1 ≤ i ≤ m, then the shape vector S can be represented as a 3m-dimensional vector: S = [X1, Y1, Z1, X2, Y2, Z2, ..., X...]. m ,Y m Z m ];

[0061] Based on the 3D feature points of the first 3D model, the RGB values ​​of each feature point are obtained and arranged into a vector to obtain the texture vector T. Specifically, assuming there are m facial feature points, the color value of each feature point is (R... i G i B i If ), and 1 ≤ i ≤ m, then the texture vector T can be represented as a 3m-dimensional vector: T = [R1, G1, B1, R2, G2, B2, ..., R...]. m G m B m ];

[0062] Obtain the standard shape vector ShapeVector_Model of a standard 3D face model, calculate the difference between the shape vector S and the standard shape vector ShapeVector_Model, and use the difference as the user's face filling degree value ΔS; preferably, the standard 3D face model is a 3D face model of a model with a full face.

[0063] Based on the facial region extracted from facial features, a preset facial feature point region is further divided. Based on the preset facial feature point region, the user's facial fill level value ΔS is subjected to a first subdivision process to obtain the fill level value corresponding to the preset facial feature point region. Specifically, the preset facial feature point region is gridded or divided into multiple sub-regions. Then, based on the position, shape, and other features of the preset facial feature point region, the user's facial fill level value ΔS is assigned to the corresponding sub-region to obtain the fill level value corresponding to the preset facial feature point region.

[0064] If the same deviation is added to all the user's 3D face shape vectors, it will eventually lead to a large difference between the user's face contour and the initial face contour. Therefore, based on the preset face feature point region, the shape vector S is subjected to a second subdivision process to obtain the first shape vector corresponding to the preset face feature point region.

[0065] Find the fill level value corresponding to the area to be filled in the user's face fill level value ΔS, and add it to the shape vector of the area to be filled to obtain the set of feature shape vectors after deformation S_reshape;

[0066] The second shape vector of the corresponding region after the fill level value is added is set as the constraint feature shape vector of the set of deformed feature shape vectors S_reshapede. The constraint feature shape vector is then interpolated and fitted based on the deformation method of third-order Laplacian coordinates to obtain the deformed shape vector. According to the feature shape vector S_reshape obtained by the interpolation and fitting process, the second three-dimensional face model is represented as: Face3D_reshape=M+S_reshape*A+T*B, thus obtaining the second three-dimensional face model.

[0067] Preferably, the preset facial feature point regions include at least the forehead region, brow bone region, cheekbone region, chin region, eye socket region, temple contour region, tear trough region, nasal base region, and other regions; the other regions are the remaining regions.

[0068] Preferably, the fill level values ​​corresponding to the preset facial feature point regions include at least the forehead region fill level value ΔS_forehead, the brow arch region fill level value ΔS_browArch, the cheekbone region fill level value ΔS_plumpcheeks, the chin region fill level value ΔS_chin, the eye socket region fill level value ΔS_eyesocket, the temple contour region fill level value ΔS_temple, the tear trough region fill level value ΔS_tearTrough, the nasal base region fill level value ΔS_smileline, and other regions fill level values ​​ΔS_other.

[0069] Preferably, the first shape vector includes at least the first forehead feature shape vector S_foreHead, the first brow arch feature shape vector S_browArch, the first cheekbone feature shape vector S_plumpcheeks, the first chin feature shape vector S_chin, the first eye socket feature shape vector S_eyesocket, the first temple feature shape vector S_temple, the first tear trough feature shape vector S_teartrough, the first nasal base feature shape vector S_smileline, and the first other shape vector S_other; the second shape vector includes at least the second forehead feature shape vector S'_foreHead, the second brow arch feature shape vector S'_browArch, the second cheekbone feature shape vector S'_plumpcheeks, the second chin feature shape vector S'_chin, the second eye socket feature shape vector S'_eyesocket, the second temple feature shape vector S'_temple, the second tear trough feature shape vector S'_teartrough, the second nasal base feature shape vector S'_smileline, and the second other shape vector S'_other.

[0070] This invention employs a region-by-region deformation filling process. Using a standard 3D human face as a benchmark, it performs deformation filling on areas such as the forehead, brow bone, eye sockets, nasal base, cheekbones, corners of the mouth, and chin, adapting to different facial depressions. This results in a natural overall facial contour and a more natural and realistic filling effect. By adding a lighting coefficient after facial retouching in 3D space, a third 3D facial model is obtained, achieving a full and three-dimensional effect. This allows users to comprehensively assess the risks of cosmetic fillers. Furthermore, the operation is simple; users can achieve a natural, non-puffy, three-dimensional, full, and youthful face with a single click on their mobile devices, obtaining medical-grade facial filler and retouching effects with greater efficiency.

[0071] Corresponding to the aforementioned face image facial filling method, this invention provides a face image facial filling system, comprising: a face feature extraction module for acquiring an input image and extracting face features to obtain two-dimensional face features; a two-dimensional face texture feature information acquisition module for obtaining two-dimensional face texture feature information based on the two-dimensional face features; an illumination coefficient calculation module for calculating the illumination coefficient of the input image based on the two-dimensional face features; a three-dimensional face reconstruction module for performing three-dimensional face reconstruction based on the two-dimensional face features and two-dimensional face texture feature information, combined with a preset three-dimensional face model, to obtain a first three-dimensional face model; a deformation filling processing module for performing deformation filling processing on the first three-dimensional model to obtain a second three-dimensional face model; adding illumination coefficients to the second three-dimensional model to obtain a third three-dimensional face model; and a mapping module for mapping the third three-dimensional face model onto a two-dimensional plane to obtain a face image facial filling effect diagram.

[0072] Preferably, the system further includes an image processing module for processing the input image to obtain a processed image. Image processing includes image conversion processing and image adjustment processing. Specifically, image conversion processing involves converting the input image to a grayscale image, and image adjustment processing involves adjusting the brightness and contrast of the grayscale image.

[0073] The illumination filtering module is used to filter the illumination of the input image.

[0074] Furthermore, the present invention also provides a computer-readable storage medium storing a face image face filling program, which, when executed by a processor, implements the steps of the face image face filling method described above. The computer-readable storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0075] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments.

[0076] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0077] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for facial filling in a human face image, characterized in that, Includes the following steps: The process of acquiring an input image and extracting facial features to obtain two-dimensional facial features includes the following steps: acquiring an input image and processing it to obtain a processed image; inputting the processed image into a deep learning model to extract facial features to obtain two-dimensional facial features, wherein the two-dimensional facial features include the bounding box of the face, facial key points, and the confidence level of the face region; Based on two-dimensional facial features, two-dimensional facial texture feature information is obtained. Specifically, based on the facial bounding box and facial key points, the pixels within the facial bounding box are sampled and interpolated using an image interpolation algorithm to obtain two-dimensional facial texture feature information. The calculation process of the illumination coefficient of the input image includes the following steps: based on the facial key points, the minimum bounding rectangle of the face is calculated, and the extracted facial region is projected onto a unit hemisphere to generate a spherical harmonic function. The order of the spherical harmonic function is determined according to the required precision. The extracted facial region is divided into several blocks, the RGB values ​​of the pixels in each block are calculated, and they are converted into function values ​​in the spherical coordinate system. Then, the spherical harmonic function coefficients are fitted using the least squares method to obtain the illumination coefficient β. Calculate the illumination coefficients of the input image based on the two-dimensional facial features; Based on the 2D facial features, after calculating the illumination coefficient of the input image, the input image is first filtered for illumination before 3D face reconstruction. The illumination filtering of the input image is specifically as follows: based on the spherical harmonic function and the illumination coefficient β, the illumination intensity under different light source directions is calculated. Specifically, for each pixel, its function value in the spherical coordinate system is calculated, and the illumination intensity is obtained by summing the spherical harmonic function and the illumination coefficient β. The shadow value of the pixel is obtained by calculating the difference between the illumination intensity of each pixel in the input image and the average illumination intensity value of the surrounding pixels. Then add the shadow value back into the RGB value of that pixel; Based on two-dimensional facial features and texture information, a three-dimensional face is reconstructed using a preset three-dimensional facial model to obtain a first three-dimensional facial model. The specific steps include: alignment and registration processing are performed based on the two-dimensional facial features and texture information; then, a three-dimensional face is reconstructed using the preset three-dimensional facial model to obtain an initial three-dimensional facial model. The preset three-dimensional facial model is obtained through facial feature extraction or is a standard three-dimensional facial model. After shape and texture optimization processing of the initial three-dimensional facial model, the first three-dimensional facial model is obtained. The first 3D model is deformed and filled to obtain the second 3D face model; Adding lighting coefficients to the second 3D model yields the third 3D face model; The third-dimensional face model is mapped onto a two-dimensional plane to obtain the face filling effect map of the face image.

2. The facial filling method for human face images according to claim 1, characterized in that: Image processing includes image conversion processing and image adjustment processing. Image conversion processing specifically involves converting the input image into a grayscale image, while image adjustment processing specifically involves adjusting the brightness and contrast of the grayscale image.

3. The facial filling method for human face images according to claim 1, characterized in that: The first 3D model is deformed and filled to obtain the second 3D face model, which includes the following steps: The first 3D face model, Face3D, is represented as: Face3D = M + S A+T B, where M is the average shape of the 3D face model, S is the shape vector, A is the coefficient of the shape vector, T is the 3D face texture vector, and B is the coefficient of the 3D face texture vector, specifically the average value obtained by principal component analysis of the preset 3D face model; Based on the three-dimensional feature points of the first three-dimensional model, calculate the three-dimensional coordinates of each feature point and arrange them into a vector to obtain the shape vector S; Based on the three-dimensional feature points of the first three-dimensional model, the RGB values ​​of each feature point are obtained and arranged into a vector to obtain the texture vector T; Obtain the standard shape vector ShapeVector_Model of a standard 3D face model, calculate the difference between shape vector S and the standard shape vector ShapeVector_Model, and use the difference as the user's face fill level value. S; Based on the facial regions extracted from facial features, the preset facial feature point regions are further divided. Based on preset facial feature point regions, the degree of facial fill is determined. S performs the first subdivision process to obtain the fill degree value corresponding to the preset facial feature point region; Based on the preset facial feature point region, the shape vector S is further subdivided to obtain the first shape vector corresponding to the preset facial feature point region. The degree of facial fill in the user's face. Find the fill degree value corresponding to the area to be filled in S, and add it to the shape vector of the area to be filled to obtain the set of feature shape vectors after deformation, S_reshape; The second shape vector of the corresponding region after the fill degree value is added is set as the constraint feature shape vector of the set of deformed feature shape vectors S_reshapede, and the constraint feature shape vector is interpolated and fitted based on the deformation method of third-order Laplacian coordinates to obtain the deformed shape vector. Based on the feature shape vector S_reshape obtained through interpolation fitting, the second 3D face model can be represented as: Face3D_reshape = M + S_reshape A+T B, thus obtaining the second three-dimensional face model.

4. The facial filling method for human face images according to claim 3, characterized in that: The preset facial feature point areas include at least the forehead area, brow bone area, cheekbone area, chin area, eye socket area, temple contour area, tear trough area, nasal base area, and other areas; The preset fill level values ​​for facial feature point regions include at least the fill level values ​​for the forehead region. S_forehead, fill level value for the brow bone area S_browArch, cheekbone area fill level value S_plumpcheeks, chin area fill level value S_chin, Eye socket area filling level value S_eyesocket, temple contour area fill level value S_temple, tear trough area fill level value S_tearTrough, Nasal base area filling level value S_smileline, fill level values ​​for other regions S_other; The first shape vector includes at least the first forehead feature shape vector S_foreHead, the first brow arch feature shape vector S_browArch, the first apple cheek feature shape vector S_plumpcheeks, the first chin feature shape vector S_chin, the first eye socket feature shape vector S_eyesocket, the first temple feature shape vector S_temple, the first tear trough feature shape vector S_teartrough, the first nasal base feature shape vector S_smileline, and the first other shape vector S_other; The second shape vector includes at least the second forehead feature shape vector S'_foreHead, the second brow arch feature shape vector S'_browArch, the second cheekbone feature shape vector S'_plumpcheeks, the second chin feature shape vector S'_chin, the second eye socket feature shape vector S'_eyesocket, the second temple feature shape vector S'_temple, the second tear trough feature shape vector S'_teartrough, the second nasal base feature shape vector S'_smileline, and the second other shape vector S'_other.

5. A facial image filling system, using the facial image filling method according to any one of claims 1-4, characterized in that, include: The face feature extraction module is used to acquire the input image and extract face features to obtain two-dimensional face features; The 2D face texture feature information acquisition module is used to obtain 2D face texture feature information based on 2D face features; The illumination coefficient calculation module is used to calculate the illumination coefficient of the input image based on the two-dimensional facial features. The 3D face reconstruction module is used to reconstruct a 3D face based on 2D face features and 2D face texture features, combined with a preset 3D face model, to obtain a first 3D face model. The deformation filling processing module is used to perform deformation filling processing on the first 3D model to obtain the second 3D face model. Adding lighting coefficients to the second 3D model yields the third 3D face model; The mapping module is used to map the third-dimensional face model onto a two-dimensional plane to obtain a facial filling effect map of the face image.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a face image face filling program, which, when executed by a processor, implements the steps of the face image face filling method as described in any one of claims 1 to 4.

Images