Method and device for beautifying a side of a human face

Abstract

This invention relates to a method and apparatus for beautifying the profile of a human face. The method includes determining facial key points in an original image; estimating the facial orientation in three-dimensional space to obtain a facial orientation vector; performing skin segmentation on the original image to obtain a skin segmentation texture; combining the original image, facial key points, facial orientation vector, skin segmentation texture, and a preset jawline shadow texture to obtain a first texture; combining the first texture, facial key points, facial orientation vector, skin segmentation texture, and a preset protruding mouth beautification value to obtain a second texture; combining the second texture, facial key points, facial orientation vector, and a preset double chin correction value to obtain a third texture; and combining the third texture, facial key points, facial orientation vector, and a preset chin correction value to obtain and output a fourth texture. The technical solution provided by this invention can correct protruding mouths, chins, and double chins in video images and add jawline editing functions.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method and apparatus for beautifying the profile of a human face. Background Technology

[0002] The demand for beautifying portrait photos has been long-standing, with a clear need for editing profile pictures to correct protruding mouths, chins, double chins, and enhance jawlines. However, since these functions mainly involve the mouth, chin, and neck areas, there are relatively few key points that can be identified and utilized. Furthermore, when the profile angle is large, key points may overlap or be lost.

[0003] In related technologies, existing facial reshaping and beautification functions generally do not perform special processing on side-view images. This means that when functions originally designed for frontal faces are applied to side-view images, the application area and angle often do not work as ideally, resulting in side-view distortion and jagged or broken edges in parts of the nose and mouth that extend beyond the contours, failing to achieve the desired effect for the user.

[0004] In addition, since the angle of a person's face in a video changes every moment, there are moments when the face is viewed from the front and from the side. Existing applications cannot provide side face editing functions while ensuring that these functions do not produce strange distortions when viewed from the front. Therefore, existing applications do not support functions such as correcting protruding mouths, chins, double chins, or adding jawline editing functions to videos, resulting in low user satisfaction. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a method and apparatus for beautifying the side profile of a human face, so as to solve the problem of low user satisfaction caused by the limited functions when editing the side profile of a human face in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for beautifying the profile of a human face, comprising:

[0007] Acquire the original image and determine the facial key points in the original image; wherein, the original image is a single-frame image;

[0008] Based on the facial key points, the facial orientation in three-dimensional space is estimated to obtain the facial orientation vector;

[0009] The original image is segmented to obtain skin segmentation texture;

[0010] The first texture is obtained by combining the original image, facial key points, facial orientation vector, skin segmentation texture, and preset jawline shadow texture;

[0011] By combining the first texture, facial key points, facial orientation vector, skin segmentation texture, and preset protruding mouth beauty value, a second texture is obtained;

[0012] The third texture is obtained by combining the second texture, facial key points, facial orientation vector, and preset double chin correction value;

[0013] The fourth texture is obtained and output by combining the third texture, facial key points, facial orientation vector, and preset chin correction value.

[0014] Furthermore, based on the facial key points, the facial orientation in three-dimensional space is estimated to obtain a facial orientation vector, including:

[0015] The positive X-axis direction is defined as the right direction directly facing the original image, the positive Y-axis direction is the upward direction, and the positive Z-axis direction is the direction directly facing the observer.

[0016] The facial key points are input into a pre-constructed 3D head model to obtain the corresponding points of 3D coordinates and 2D coordinates, and the camera extrinsic parameters are obtained through the corresponding points;

[0017] The rotation vector extracted from the camera extrinsic parameters is applied to vector (0, 0, 1) to obtain the face orientation vector.

[0018] Furthermore, by combining the original image, facial key points, facial orientation vector, skin segmentation texture, and a preset jawline shadow texture, a first texture is obtained, including:

[0019] Filter the facial key points that relate to the facial contour to obtain a point sequence;

[0020] The center point and first step length of the facial contour are calculated using the point sequence.

[0021] The point sequence is expanded inward in a direction close to the center point in a first preset layer, and then the point sequence is expanded outward in a direction away from the center point in a second preset layer. All the points then form a first point set; wherein the spacing between each layer is 1 step length.

[0022] Using the triangulation method with edge constraints, the first set of points is constructed into a first mesh, and a preset strength value is assigned to each vertex on the first mesh;

[0023] Calculate the first shadow intensity of the left mandibular line and the second shadow intensity of the right mandibular line based on the face orientation vector;

[0024] Multiply the intensity value of the left vertex in the first grid by the first shadow intensity, and multiply the intensity value of the right vertex in the first grid by the second shadow intensity to obtain the shadow texture;

[0025] Based on the first grid, the shadow texture is applied to the original image using a blending method to obtain the first texture.

[0026] Furthermore, by combining the first texture, facial key points, facial orientation vector, skin segmentation texture, and preset protruding mouth beauty value, a second texture is obtained, including:

[0027] Based on the facial key points, calculate the distance between the tip of the nose and the upper lip in the original image;

[0028] The offset is calculated based on the face orientation vector, distance, and the aesthetic value of a protruding mouth; the offset is used to represent the direction and magnitude of the mouth region's offset.

[0029] Based on the facial key points, determine the key points of the mouth region, and based on the key points of the mouth region, determine the convex hull;

[0030] Determine the center point of the convex hull, and expand the convex hull outward by a preset number of layers through the center point to obtain a second set of points;

[0031] Using the triangulation method with edge constraints, the second set of points is constructed into a second mesh;

[0032] For each layer in the second grid, offset according to the corresponding offset to obtain the third grid;

[0033] The first texture is subjected to mesh deformation based on the third network to obtain the second texture.

[0034] Furthermore, by combining the second texture, facial key points, facial orientation vector, and a preset double chin correction value, a third texture is obtained, including:

[0035] Filter the facial key points that relate to the facial contour to obtain a point sequence;

[0036] The center point of the facial contour is calculated using the point sequence;

[0037] Based on the aforementioned facial key points, calculate the distance from the lower lip to the tip of the chin to obtain the second step length;

[0038] The point sequence is expanded inward in a direction closer to the center point by a third preset layer, and then the point sequence is expanded outward in a direction farther from the center point by a fourth preset layer, so that all points are combined to obtain the third point set;

[0039] Using the triangulation method with edge constraints, the third set of points is constructed into a fourth mesh;

[0040] Calculate the direction vector of the nose in two-dimensional space based on the key points of the nose;

[0041] The offset is calculated based on the direction vector, the second compensation, and the double chin correction value.

[0042] The first intensity coefficient and the second intensity coefficient are calculated based on the face orientation vector;

[0043] Calculate the deformation weights for each contour layer;

[0044] Based on the deformation weights and offsets, the contours from the innermost layer to the outermost layer are traversed, and the contour points of each layer are offset by a corresponding distance to obtain the fourth set of points.

[0045] Based on the fourth set of points, construct the fifth grid;

[0046] The second texture is deformed based on the fourth and fifth grids to obtain the third texture.

[0047] Furthermore, by combining the third texture, facial key points, facial orientation vector, and a preset chin correction value, a fourth texture is obtained and output, including:

[0048] Based on the aforementioned facial key points, the distance from the lower lip to the tip of the chin is calculated to obtain the liquefaction radius;

[0049] The third step length is calculated based on the chin correction value, the face orientation vector, and the liquefaction radius.

[0050] Calculate the two-dimensional stretching direction based on the face orientation vector;

[0051] The liquefaction point is calculated based on the two-dimensional stretching direction and the third step length; wherein, the liquefaction point includes multiple elements;

[0052] Iterate through the influence of each element on each pixel in the original image to obtain the new coordinates after offset. Then, assign the pixel color magnitude of the new coordinates to the original pixel to obtain the fourth texture.

[0053] Furthermore, after determining the facial key points in the original image, the process also includes:

[0054] A Kalman filter is used to smooth facial landmarks.

[0055] Furthermore, the facial key points include the two-dimensional coordinates of facial features and facial contours in the image.

[0056] This application provides a device for beautifying the profile of a human face, comprising:

[0057] A facial landmark detection module is used to acquire an original image and determine facial landmarks in the original image; wherein the original image is a single-frame image;

[0058] The face orientation detection module is used to estimate the face orientation in three-dimensional space based on the facial key points, and obtain the face orientation vector;

[0059] The skin segmentation module is used to perform skin segmentation on the original image to obtain skin segmentation texture;

[0060] The jawline beautification module is used to combine the original image, facial key points, facial orientation vector, skin segmentation texture, and preset jawline shadow texture to obtain the first texture;

[0061] The protruding mouth beautification module is used to combine the first texture, facial key points, facial orientation vector, skin segmentation texture, and preset protruding mouth beautification value to obtain the second texture;

[0062] The double chin beautification module is used to combine the second texture, facial key points, facial orientation vector and preset double chin correction value to obtain the third texture;

[0063] The chin beautification module is used to combine the third texture, facial key points, facial orientation vector, and preset chin correction value to obtain and output the fourth texture.

[0064] The beneficial effects that can be achieved by adopting the above technical solution in this invention include:

[0065] This invention provides a method and apparatus for beautifying the profile of a human face. The technical solution provided in this application supports editing functions such as correcting protruding mouths, chins, double chins, and adding jawline editing to videos. Based on public demand for profile enhancement, this application redesigns the effects of protruding mouths, double chins, and jawlines, and adds a chin-lengthening function, allowing users to achieve a perfect profile with a straight line between the nose, lips, and chin. Simultaneously, it redefines the processing range for protruding mouths, double chins, jawlines, and chin tips according to editing needs, making the superposition of these functions more natural. Furthermore, it supports profile editing of videos, accurately identifying and processing profile angles without causing facial distortion from a frontal view. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram illustrating the steps of the method for beautifying the profile of a human face according to the present invention;

[0068] Figure 2 This is a flowchart illustrating the method for beautifying the profile of a human face according to the present invention.

[0069] Figure 3 This is a schematic diagram of the structure of the device for beautifying the profile of a human face according to the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0071] The following describes a specific method and apparatus for beautifying the profile of a human face, provided in an embodiment of this application, with reference to the accompanying drawings.

[0072] like Figure 1 As shown in the embodiments of this application, the method for beautifying the profile of a face includes:

[0073] S101, acquire the original image and determine the facial key points in the original image; wherein, the original image is a single-frame image;

[0074] It is understandable that, such as Figure 2 As shown, this application receives a single-frame image I as input. The single-frame image I can be a frame in a video or a simple two-dimensional image texture. After performing face landmark detection on the input single-frame image I, a set of face landmarks FaceLandmarkList is output as subsequent input data.

[0075] It should be noted that currently, the most effective face detection algorithms in industry are all based on machine learning or deep learning, and both open-source and closed-source algorithms are very mature. Therefore, the specific algorithm used can be flexibly chosen by the implementer without limitation, and will not have a substantial impact on the algorithmic flow of subsequent processing. Preferably, this application uses a self-developed deep learning-based network for detection.

[0076] Furthermore, most detection models do not perform in-depth optimization for continuous frame detection. Therefore, due to perturbations and errors, there will be some jitter in the detection points between consecutive frames of a video / selfie. If this jitter is not addressed, it will affect the beautification editing effect of subsequent videos / selfies.

[0077] In some embodiments, this application employs a Kalman filter or other post-processing procedures to smooth key points.

[0078] S102, Estimate the face orientation in three-dimensional space based on the facial key points to obtain the face orientation vector;

[0079] In some embodiments, the facial orientation in three-dimensional space is estimated based on the facial key points to obtain a facial orientation vector, including:

[0080] The positive X-axis direction is defined as the right direction directly facing the original image, the positive Y-axis direction is the upward direction, and the positive Z-axis direction is the direction directly facing the observer.

[0081] The facial key points are input into a pre-constructed 3D head model to obtain the corresponding points of 3D coordinates and 2D coordinates, and the camera extrinsic parameters are obtained through the corresponding points;

[0082] The rotation vector extracted from the camera extrinsic parameters is applied to vector (0, 0, 1) to obtain the face orientation vector.

[0083] Specifically, this application estimates the approximate orientation of the face in 3D space using FaceLandmarkList, and outputs a 3D vector FaceDir. Specifically, as one implementation method, this application defines the right-hand direction when the observer is facing the image as the positive X-axis, the top direction as the positive Y-axis, and the direction directly facing the observer as the positive Z-axis. Using a predefined general 3D head model and the input face landmarks, a set of corresponding 3D-2D points is found. The Perspective-n-Point (PnP) algorithm is used to obtain the camera extrinsic parameters, and then the rotation vector is extracted and applied to the vector (0, 0, 1) to obtain the face orientation.

[0084] There are various approaches to solving the PnP algorithm. For the technical solution in this application, due to the inherent errors of the head model and face itself, as well as the existence of detection errors, it is best to use a two-stage solution. That is, first use algorithms such as EPnP or SQPnP to perform a rough solution and obtain better initial values, and then use an iterative method to perform a precise solution.

[0085] Due to the inherent error between the head model and the input image, and the fact that real human faces may make various expressions, when calculating 3D-2D point pairs, not all detection key points should be used. The best practice is to use key points that are relatively less prone to deformation, such as the corners of the eyes, the tip of the nose, the corners of the mouth, and the chin, which can greatly improve detection robustness.

[0086] S103, Perform skin segmentation on the original image to obtain skin segmentation texture;

[0087] It is understandable that a single frame image I can be a frame in a video or a simple two-dimensional image texture. After segmenting the single frame image I, the skin segmentation texture FaceSeg is output as the output.

[0088] Specifically, skin segmentation algorithms currently available for industrial use are relatively mature. Based on implementation, they can be broadly categorized into color statistics-based classification methods and machine learning-based classification methods. Therefore, implementers can flexibly choose the specific algorithm based on their own scenario and equipment performance without substantially affecting the algorithmic flow of subsequent processing. Preferably, this invention uses a self-developed deep learning-based network for detection.

[0089] S104, combine the original image, facial key points, facial orientation vector, skin segmentation texture, and preset jawline shadow texture to obtain the first texture;

[0090] In some embodiments, a first texture is obtained by combining the original image, facial key points, facial orientation vector, skin segmentation texture, and a preset jawline shadow texture, including:

[0091] Filter the facial key points that relate to the facial contour to obtain a point sequence;

[0092] The center point and first step length of the facial contour are calculated using the point sequence.

[0093] The point sequence is expanded inward in a direction close to the center point in a first preset layer, and then the point sequence is expanded outward in a direction away from the center point in a second preset layer. All the points then form a first point set; wherein the spacing between each layer is 1 step length.

[0094] Using the triangulation method with edge constraints, the first set of points is constructed into a first mesh, and a preset strength value is assigned to each vertex on the first mesh;

[0095] Calculate the first shadow intensity of the left mandibular line and the second shadow intensity of the right mandibular line based on the face orientation vector;

[0096] Multiply the intensity value of the left vertex in the first grid by the first shadow intensity, and multiply the intensity value of the right vertex in the first grid by the second shadow intensity to obtain the shadow texture;

[0097] Based on the first grid, the shadow texture is applied to the original image using a blending method to obtain the first texture.

[0098] Specifically, the original image I, the face landmarks FaceLandmarkList, the face orientation FaceDir, the skin segmentation texture FaceSeg, and the pre-designed jawline shadow texture Shadow are used as inputs to obtain the processed first texture S1 as output.

[0099] The specific steps are as follows:

[0100] 1. Filter out the points related to the facial contour from FaceLandmarkList to obtain the point sequence FaceContourLandmarkList;

[0101] 2. Calculate the center point of the contour, FaceContourCenterPoint, using the following formula:

[0102] FaceContourCenterPoint=(FaceContourLandmarkList[first]+

[0103] FaceContourLandmarkList[last])*0.5;

[0104] 3. Calculate the length of the first step, FaceContourStep, using the following formula:

[0105] FaceContourStep=distance(FaceContourLandmarkList[first],FaceContourLandmarkList[last])*0.08;

[0106] Here, distance(x,y) is a function that calculates the Euclidean distance between vectors x and y.

[0107] 4. Expand the point sequence FaceContourLandmarkList inward by one layer along the direction closest to the center point FaceContourCenterPoint, with the spacing between each layer being the length of the first step, FaceContourStep. Then expand the point sequence FaceContourLandmarkList outward by eight layers along the direction furthest from the center point FaceContourCenterPoint, with the spacing between each layer being the length of the first step, FaceContourStep, to form the first point set AllPoints.

[0108] 5. Using triangulation with edge constraints, construct the first set of points AllPoints into the first mesh Mesh1, and assign an intensity value of 1.0 to each vertex on the first mesh Mesh.

[0109] 6. Calculate the shadow intensity of the left and right jawlines based on the face orientation vector FaceDir. The calculation formula is:

[0110] leftStrength=clamp(1.0+FaceDir.x*1.3,0.0,1.0)

[0111] rightStrength = clamp(1.0 - FaceDir.x * 1.3, 0.0, 1.0)

[0112] Among them, for the clamp(x, minValue, maxValue) function, if x < minValue, then output minValue; if x > maxValue, then output maxValue; otherwise, output x.

[0113] 7. Multiply the intensity values of all the left vertices in the first mesh Mesh1 by the first shadow intensity leftStrength. Multiply the intensity values of all the right vertices in the first mesh Mesh1 by the second shadow intensity rightStrength to obtain the shadow texture Shadow.

[0114] 8. Fit the shadow texture Shadow back to the original image I according to the first mesh Mesh1 using the multiply blend mode. Among them, when processing each pixel, the blend intensity is determined by linearly interpolating the intensity values of the vertices of the first mesh Mesh1 to obtain the first texture S1.

[0115] S105. Combine the first texture, facial key points, face orientation vector, skin segmentation texture, and a preset protruding mouth beauty value to obtain a second texture;

[0116] In some embodiments, combining the first texture, facial key points, face orientation vector, skin segmentation texture, and a preset protruding mouth beauty value to obtain a second texture includes:

[0117] Calculate the distance between the tip of the nose and the upper lip in the original image according to the facial key points;

[0118] Calculate an offset according to the face orientation vector, the distance, and the protruding mouth beauty value; the offset is used to represent the direction and magnitude of the offset of the mouth area;

[0119] Determine the key points of the mouth area according to the facial key points, and determine the convex hull according to the key points of the mouth area;

[0120] Determine the center point of the convex hull, and expand the convex hull outward by a preset number of layers through the center point to obtain a second point set;

[0121] Use the triangulation method with edge constraints to construct the second point set into a second mesh;

[0122] For each layer in the second mesh, perform an offset according to the corresponding offset to obtain a third mesh; [[ID=The first texture is subjected to mesh deformation based on the third network to obtain the second texture.

[0124] Specifically, the first texture S1, the face landmarks FaceLandmarkList, the face orientation vector FaceDir, and a user-inputted protruding mouth correction value Strength (range [-1, 1]) are used as inputs. The processed second texture S2 is then output.

[0125] Specifically, the following steps are included:

[0126] 1. Based on the FaceLandmarkList, calculate the distance D between the tip of the nose and the upper lip in the image.

[0127] 2. Obtain the offset bias using the formula FaceDir.xy*D*Strength. The offset bias indicates the direction and magnitude of the mouth region offset.

[0128] 3. Based on the FaceLandmarkList, filter out the key points in the mouth area and solve for the convex hull of the filtered key points.

[0129] 4. Find the center point C of the convex hull, and expand the convex hull outward in 4 layers using the center point C and the convex hull. The spacing between each layer is D*0.35, resulting in the second set of points, AllPoints.

[0130] 5. Using triangulation with edge constraints, construct the second mesh Mesh2 from the second set of points AllPoints.

[0131] 6. For the innermost contour in the second mesh (Mesh2), offset by Bias. For the next innermost contour, offset by Bias * 0.75. For the next innermost and outermost contours, offset by Bias * 0.5. For the next outermost contour, offset by Bias * 0.25. The outermost contour is not offset. This results in the third mesh (Mesh3).

[0132] 7. Based on the third mesh (Mesh3), perform ordinary mesh deformation on the input first texture (S1) to obtain the second texture (S2).

[0133] S106, combine the second texture, facial key points, facial orientation vector and preset double chin correction value to obtain the third texture;

[0134] In some embodiments, a third texture is obtained by combining the second texture, facial key points, facial orientation vector, and a preset double chin correction value, including:

[0135] Filter the facial key points that relate to the facial contour to obtain a point sequence;

[0136] The center point of the facial contour is calculated using the point sequence;

[0137] Based on the aforementioned facial key points, calculate the distance from the lower lip to the tip of the chin to obtain the second step length;

[0138] The point sequence is expanded inward in a direction closer to the center point by a third preset layer, and then the point sequence is expanded outward in a direction farther from the center point by a fourth preset layer, so that all points are combined to obtain the third point set;

[0139] Using the triangulation method with edge constraints, the third set of points is constructed into a fourth mesh;

[0140] Calculate the direction vector of the nose in two-dimensional space based on the key points of the nose;

[0141] The offset is calculated based on the direction vector, the second compensation, and the double chin correction value.

[0142] The first intensity coefficient and the second intensity coefficient are calculated based on the face orientation vector;

[0143] Calculate the deformation weights for each contour layer;

[0144] Based on the deformation weights and offsets, the contours from the innermost layer to the outermost layer are traversed, and the contour points of each layer are offset by a corresponding distance to obtain the fourth set of points.

[0145] Based on the fourth set of points, construct the fifth grid;

[0146] The second texture is deformed based on the fourth and fifth grids to obtain the third texture.

[0147] Specifically, this step takes the second texture S2, the face landmarks FaceLandmarkList, the face orientation vector FaceDir, and a double chin correction value Strength (range [0, 1]) input by the user as input. The processed texture S3 is then output.

[0148] Specifically, the following steps are included:

[0149] 1. Filter out the points related to the facial contour from FaceLandmarkList to obtain the point sequence FaceContourLandmarkList.

[0150] 2. Calculate the center point of the contour, FaceContourCenterPoint, using the following formula:

[0151] FaceContourCenterPoint=(FaceContourLandmarkList[first]+

[0152] FaceContourLandmarkList[last])*0.5

[0153] 3. Calculate the distance from the lower lip to the tip of the chin using FaceLandmarkList, and multiply this distance by a coefficient of 0.36 to obtain the second step length D.

[0154] 4. Move the point sequence FaceContourLandmarkList along the path closest to the center point.

[0155] Expand the FaceContourCenterPoint inwards by 2 layers, with a spacing of D between each layer. Then expand the point sequence FaceContourLandmarkList outwards by 4 layers along the direction away from the center point FaceContourCenterPoint, with a spacing of the second step size between each layer, and combine all 7 layers of points into a third point set AllPoints.

[0156] 5. Using triangulation with edge constraints, construct the fourth mesh Mesh4 from the third set of points AllPoints.

[0157] 6. Based on the key points of the nose, calculate the direction of the nose in two-dimensional space, Dir1.

[0158] 7. Obtain the offset Bias using the formula Dir1*D*0.88*Strength.

[0159] 8. Calculate the intensity coefficients p1 and p2 based on the face orientation vector FaceDir. The formula is:

[0160] p1=0.4+0.6*MIN(1.0,ABS(FaceDir.x) / 0.6);

[0161] p2=0.2+0.8*MIN(1.0,MAX(0.0,1.0–

[0162] FaceDir.y));

[0163] in:

[0164] The MIN function takes the minimum value of the input parameters as its output.

[0165] The MAX function takes the maximum value of the input parameters as its output.

[0166] The ABS function takes the absolute value of the input parameter and outputs it.

[0167] 9. Calculate the deformation weights for each contour layer. The deformation amplitude decreases from the center to both sides. Let N be the number of points between a point i and the center point of the contour. The calculation process for weights[i] is as follows:

[0168] Define SideList = [0.0, 0.30, 0.50, 0.66, 0.80, 0.88, 0.92, 0.92, 0.92, 0.84, 0.70, 0.54]

[0169] p = 1.0 – (N – 5) / 7.0

[0170] p = clamp(p, 0.0, 1.0)

[0171] p = p * 0.3

[0172] weights[i]=SideList[i]*clamp(1.0+FaceDir.x*1.2*p,0.0,1.0)*p1*p2.

[0173] 10. Based on the deformation weights[i] and the offset bias, traverse the contours from the innermost to the outermost layer (a total of 5 layers), and offset the contour points of each layer by the corresponding distance. Assuming we are currently processing the j-th node of the i-th layer contour, we obtain the fourth point set AllPoints[i][j]. The calculation process for AllPoints[i][j] is as follows:

[0174] Define OffsetWeightList = [0.5, 1.2, 1.6, 2.0, 1.6]

[0175] AllPoints[i][j]=AllPoints[i][j]+Bias*OffsetWeightList[i]*weights[j]

[0176] 11. Based on the fourth set of points AllPoints, obtain the fifth grid Mesh5.

[0177] 12. Based on the fourth mesh Mesh4 and the fifth mesh Mesh5, transform the second texture S2 into the third texture S3.

[0178] S107, combining the third texture, facial key points, facial orientation vector, and preset chin correction value, a fourth texture is obtained and output.

[0179] In some embodiments, a fourth texture is obtained and output by combining the third texture, facial key points, facial orientation vector, and a preset chin correction value, including:

[0180] Based on the aforementioned facial key points, the distance from the lower lip to the tip of the chin is calculated to obtain the liquefaction radius;

[0181] The third step length is calculated based on the chin correction value, the face orientation vector, and the liquefaction radius.

[0182] Calculate the two-dimensional stretching direction based on the face orientation vector;

[0183] The liquefaction point is calculated based on the two-dimensional stretching direction and the third step length; wherein, the liquefaction point includes multiple elements;

[0184] Iterate through the influence of each element on each pixel in the original image to obtain the new coordinates after offset. Then, assign the pixel color magnitude of the new coordinates to the original pixel to obtain the fourth texture.

[0185] Specifically, the third texture S3, the face key points FaceLandmarkList, the face orientation FaceDir, and a chin correction value Strength (range [-1,1]) input by the user are used as inputs to obtain the processed fourth texture S4 as the final output.

[0186] Specifically, the following steps are included:

[0187] 1. Calculate the liquefaction radius R based on the distance from the lower lip to the tip of the chin.

[0188] 2. Calculate the third step length Step, the formula is: step = Strength * (1.0 – FaceDir.z) * (0.12 * R);

[0189] 3. Calculate the two-dimensional stretching direction Dir based on the face orientation vector FaceDir. The calculation formula is: Dir = Normalize(FaceDir.x, FaceDir.y);

[0190] The Normalize function combines multiple input numbers into a single vector and normalizes it.

[0191] 4. Calculate the liquefaction points LiquidPointList. Let P be the key point of the chin tip, and the elements in the LiquidPointList array be:

[0192] LiquidPointList[0]=P+0*Dir*step*0.3;

[0193] LiquidPointList[1]=P+1*Dir*step*0.3;

[0194] LiquidPointList[2]=P+2*Dir*step*0.3;

[0195] LiquidPointList[3]=P+3*Dir*step*0.3;

[0196] LiquidPointList[4]=P+4*Dir*step*0.3;

[0197] LiquidPointList[5]=P+5*Dir*step*0.3;

[0198] LiquidPointList[6]=P+6*Dir*step*0.3;

[0199] LiquidPointList[7]=P+7*Dir*step*0.3;

[0200] 5. For each pixel in the image, calculate the impact of each element in LiquidPointList on it. Assuming the coordinates of a point on the texture are TexCoord, then a certain element LiquidPointList[i] will shift it to a new coordinate LiquidPointList[i]':

[0201] infect=clamp(1.0-distance(TexCoord,LiquidPointList[i]) / R,0.0,1.0)

[0202] offset = Dir * Step * Infect

[0203] LiquidPointList[i]'=LiquidPointList[i]–offset

[0204] After iteratively calculating the influence of all elements in LiquidPointList, a new coordinate with the final offset is obtained. The pixel color at the new coordinate is then assigned to the original pixel. After traversing and processing all pixels in the image, the fourth texture S4 is obtained.

[0205] The working principle of the facial profile beautification method is as follows: This application acquires an original image and determines the facial key points in the original image; wherein, the original image is a single-frame image; the facial orientation in three-dimensional space is estimated based on the facial key points to obtain a facial orientation vector; skin segmentation is performed on the original image to obtain a skin segmentation texture; the first texture is obtained by combining the original image, facial key points, facial orientation vector, skin segmentation texture, and a preset jawline shadow texture; the second texture is obtained by combining the first texture, facial key points, facial orientation vector, skin segmentation texture, and a preset protruding mouth beautification value; the third texture is obtained by combining the second texture, facial key points, facial orientation vector, and a preset double chin correction value; the fourth texture is obtained and output by combining the third texture, facial key points, facial orientation vector, and a preset chin correction value.

[0206] This application redesigns the effects of protruding mouth, double chin, and jawline based on the public's demand for side profiles, and adds a chin-lengthening function, allowing users to achieve a perfect side profile with the nose, lips, and chin in a straight line. It also redefines the processing range for protruding mouth, double chin, jawline, and chin-lengthening based on editing needs, making the combination of these functions more natural. Furthermore, it supports side profile editing in videos, accurately identifying and processing side profile angles without causing facial distortion from a frontal view.

[0207] In some embodiments, such as Figure 3 As shown, this application provides a device for beautifying the profile of a human face, comprising:

[0208] The facial landmark detection module 201 is used to acquire an original image and determine the facial landmarks in the original image; wherein the original image is a single-frame image;

[0209] The face orientation detection module 202 is used to estimate the face orientation in three-dimensional space based on the face key points to obtain the face orientation vector;

[0210] Skin segmentation module 203 is used to perform skin segmentation on the original image to obtain skin segmentation texture;

[0211] The jawline beautification module 204 is used to combine the original image, facial key points, facial orientation vector, skin segmentation texture and preset jawline shadow texture to obtain a first texture;

[0212] The protruding mouth beautification module 205 is used to combine the first texture, facial key points, facial orientation vector, skin segmentation texture and preset protruding mouth beautification value to obtain the second texture;

[0213] The double chin beautification module 206 is used to combine the second texture, facial key points, facial orientation vector and preset double chin correction value to obtain the third texture;

[0214] The chin-refining module 207 is used to combine the third texture, facial key points, facial orientation vector, and preset chin correction value to obtain and output the fourth texture.

[0215] The working principle of the facial profile beautification device provided in this application is as follows: A facial key point detection module 201 acquires an original image and determines the facial key points in the original image; wherein, the original image is a single-frame image; a facial orientation detection module 202 estimates the facial orientation in three-dimensional space based on the facial key points, obtaining a facial orientation vector; a skin segmentation module 203 performs skin segmentation on the original image to obtain skin segmentation texture; and a jawline beautification module 204 combines the original image, facial key points, facial orientation vector, and skin... The skin segmentation texture and the preset jawline shadow texture are used to obtain the first texture; the protruding mouth beautification module 205 combines the first texture, facial key points, facial orientation vector, skin segmentation texture and the preset protruding mouth beautification value to obtain the second texture; the double chin beautification module 206 combines the second texture, facial key points, facial orientation vector and the preset double chin correction value to obtain the third texture; the chin beautification module 207 combines the third texture, facial key points, facial orientation vector and the preset chin correction value to obtain the fourth texture and output it.

[0216] It is understood that the method embodiments provided above correspond to the device embodiments described above, and the specific details can be referred to each other, which will not be repeated here.

[0217] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0218] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0219] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction methods implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0220] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0221] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for beautifying the profile of a human face, characterized in that, include: Acquire the original image and determine the facial key points in the original image; wherein, the original image is a single-frame image; Based on the facial key points, the facial orientation in three-dimensional space is estimated to obtain the facial orientation vector; The original image is segmented to obtain skin segmentation texture; The first texture is obtained by combining the original image, facial key points, facial orientation vector, skin segmentation texture, and preset jawline shadow texture; By combining the first texture, facial key points, facial orientation vector, skin segmentation texture, and preset protruding mouth beauty value, a second texture is obtained; The third texture is obtained by combining the second texture, facial key points, facial orientation vector, and preset double chin correction value; By combining the third texture, facial key points, facial orientation vector, and preset chin correction value, a fourth texture is obtained and output; Based on the facial landmarks, the facial orientation in three-dimensional space is estimated to obtain a facial orientation vector, including: The positive X-axis direction is defined as the rightward direction facing the original image, the positive Y-axis direction is the upward direction, and the positive Z-axis direction is the direction facing the observer. The facial key points are input into a pre-constructed 3D head model to obtain the corresponding points of 3D coordinates and 2D coordinates, and the camera extrinsic parameters are obtained through the corresponding points; The rotation vector in the camera extrinsic parameters is extracted and applied to the vector (0, 0, 1) to obtain the face orientation vector; Combining the first texture, facial key points, facial orientation vector, skin segmentation texture, and preset protruding mouth beauty value, a second texture is obtained, including: Based on the facial key points, calculate the distance between the tip of the nose and the upper lip in the original image; The offset is calculated based on the face orientation vector, distance, and the aesthetic value of a protruding mouth; the offset is used to represent the direction and magnitude of the mouth region's offset. Based on the facial key points, determine the key points of the mouth region, and based on the key points of the mouth region, determine the convex hull; Determine the center point of the convex hull, and expand the convex hull outward by a preset number of layers through the center point to obtain a second set of points; Using the triangulation method with edge constraints, the second set of points is constructed into a second mesh; For each layer in the second grid, offset according to the corresponding offset to obtain the third grid; The first texture is deformed according to the third grid to obtain the second texture.

2. The method according to claim 1, characterized in that, Combining the original image, facial landmarks, facial orientation vectors, skin segmentation textures, and a preset jawline shadow texture, a first texture is obtained, including: Filter the facial key points that relate to the facial contour to obtain a point sequence; The center point and first step length of the facial contour are calculated using the point sequence. The point sequence is expanded inward in a direction close to the center point in a first preset layer, and then the point sequence is expanded outward in a direction away from the center point in a second preset layer. All the points then form a first point set; wherein the spacing between each layer is 1 step length. Using the triangulation method with edge constraints, the first set of points is constructed into a first mesh, and a preset strength value is assigned to each vertex on the first mesh; Calculate the first shadow intensity of the left mandibular line and the second shadow intensity of the right mandibular line based on the face orientation vector; Multiply the intensity value of the left vertex in the first grid by the first shadow intensity, and multiply the intensity value of the right vertex in the first grid by the second shadow intensity to obtain the shadow texture; Based on the first grid, the shadow texture is applied to the original image using a blending method to obtain the first texture.

3. The method according to claim 1, characterized in that, Combining the second texture, facial key points, facial orientation vector, and preset double chin correction value, a third texture is obtained, including: Filter the facial key points that relate to the facial contour to obtain a point sequence; The center point of the facial contour is calculated using the point sequence; Based on the aforementioned facial key points, calculate the distance from the lower lip to the tip of the chin to obtain the second step length; The point sequence is expanded inward in a direction closer to the center point by a third preset layer, and then the point sequence is expanded outward in a direction farther from the center point by a fourth preset layer, so that all points are combined to obtain the third point set; Using the triangulation method with edge constraints, the third set of points is constructed into a fourth mesh; Calculate the direction vector of the nose in two-dimensional space based on the key points of the nose; The offset is calculated based on the direction vector, the second compensation, and the double chin correction value. The first intensity coefficient and the second intensity coefficient are calculated based on the face orientation vector; Calculate the deformation weights for each contour layer; Based on the deformation weights and offsets, the contours from the innermost layer to the outermost layer are traversed, and the contour points of each layer are offset by a corresponding distance to obtain the fourth set of points. Based on the fourth set of points, construct the fifth grid; The second texture is deformed based on the fourth and fifth grids to obtain the third texture.

4. The method according to claim 1, characterized in that, Combining the third texture, facial key points, facial orientation vector, and preset chin correction value, a fourth texture is obtained and output, including: Based on the aforementioned facial key points, the distance from the lower lip to the tip of the chin is calculated to obtain the liquefaction radius; The third step length is calculated based on the chin correction value, the face orientation vector, and the liquefaction radius. Calculate the two-dimensional stretching direction based on the face orientation vector; The liquefaction point is calculated based on the two-dimensional stretching direction and the third step length; wherein, the liquefaction point includes multiple elements; Iterate through the influence of each element on each pixel in the original image to obtain the new coordinates after offset. Then, assign the pixel color magnitude of the new coordinates to the original pixel to obtain the fourth texture.

5. The method according to claim 1, characterized in that, After determining the facial landmarks in the original image, the process also includes: A Kalman filter is used to smooth facial landmarks.

6. The method according to claim 5, characterized in that, The facial key points include the two-dimensional coordinates of facial features and facial contours in the image.

7. A device for beautifying the profile of a human face, characterized in that, include: A facial landmark detection module is used to acquire an original image and determine facial landmarks in the original image; wherein the original image is a single-frame image; The face orientation detection module is used to estimate the face orientation in three-dimensional space based on the facial key points, and obtain the face orientation vector; The skin segmentation module is used to perform skin segmentation on the original image to obtain skin segmentation texture; The jawline beautification module is used to combine the original image, facial key points, facial orientation vector, skin segmentation texture, and preset jawline shadow texture to obtain the first texture; The protruding mouth beautification module is used to combine the first texture, facial key points, facial orientation vector, skin segmentation texture, and preset protruding mouth beautification value to obtain the second texture; The double chin beautification module is used to combine the second texture, facial key points, facial orientation vector and preset double chin correction value to obtain the third texture; The chin beautification module is used to combine the third texture, facial key points, facial orientation vector and preset chin correction value to obtain and output the fourth texture; The second texture is obtained by combining the first texture, facial key points, facial orientation vector, skin segmentation texture, and a preset protruding mouth beauty value, including: Based on the facial key points, calculate the distance between the tip of the nose and the upper lip in the original image; The offset is calculated based on the face orientation vector, distance, and the aesthetic value of a protruding mouth; the offset is used to represent the direction and magnitude of the mouth region's offset. Based on the facial key points, determine the key points of the mouth region, and based on the key points of the mouth region, determine the convex hull; Determine the center point of the convex hull, and expand the convex hull outward by a preset number of layers through the center point to obtain a second set of points; Using the triangulation method with edge constraints, the second set of points is constructed into a second mesh; For each layer in the second grid, offset according to the corresponding offset to obtain the third grid; The first texture is deformed according to the third grid to obtain the second texture.

Images