A method for facial feature correction based on global optical flow and neural network

Abstract

The present invention discloses a method for correcting facial features based on global optical flow and neural network, and the method belongs to the field of facial image processing. The method trains original facial training images, mirrored facial training images and a first optical flow through a neural network, and uses the trained model for correcting facial features. The purpose of the present invention is to solve the problem that the authenticity of the face beautification process in the prior art is lacking and the traditional optical flow method directly processes the facial image and easily causes the unevenness of the processed image. Thereby, the present invention can avoid the problem that only the sparse matrix is ​​calculated in the traditional optical flow method, resulting in uneven regressed facial features, thereby achieving the effect of face beautification.

Description

Technical Field

[0001] The present invention relates to the field of face image processing, and in particular to a method for correcting facial features based on global optical flow and neural network. Background Art

[0002] In the field of face image processing, in order to beautify the original face image, many beauty functions such as skin smoothing, whitening, face slimming and eye enlargement have emerged. These functions adopt a set of inherent patterns in the implementation process, and the effects after beautification are also stereotyped. Ordinary faces after these processing are usually quite different from the original ones, lacking the characteristics that each individual should have, especially in some occasions with high requirements for authenticity, excessive beauty effects may be counterproductive; some scholars have previously proposed using learning methods for face beautification, such as first extracting 84 feature points of the face, and then using SVR and convolutional neural networks to optimize these feature points, but these feature point-based methods only use sparse features and rely on the accuracy of feature point detection. The traditional optical flow method is only applied to sparse matrices, and directly calculating face images faces the problem of extremely roughness, so it has great defects. Summary of the invention

[0003] In order to solve the problem that the above-mentioned prior art lacks authenticity in the face beautification process and the traditional optical flow method directly processes the face image and easily produces unevenness, resulting in distorted processed images, the present invention proposes a facial feature correction method based on global optical flow and neural network, which uses a neural network to train the optical flow, outputs the corresponding label, and then uses the optical flow information to correct the image through interpolation, so that the correction of the facial features is smoother.

[0004] In view of the above situation, the present invention proposes a method for correcting facial features based on global optical flow and neural network, which specifically includes the following steps:

[0005] S1: Obtain the face image to be processed, parse out 68 feature points of the face image to be processed, and obtain the coordinates of the outer corners of the left and right eyes respectively, mark the left corner of the left eye as (x1, y1), and the right corner of the right eye as (x2, y2), and find the midpoint coordinates of the outer corners of the left and right eyes, that is, ((x1+x2) / 2, (y1+y2) / 2), record it as the center point of the face, take the center point as the starting point, take 1.6 times the distance between the two eyes up and down, and take 1.2 times the distance between the two eyes to the left and right, and crop the image. If the up-down distance or the left-right distance is too small, padding is used to obtain a cropped image to obtain a face area image, and then the cropped face area image is proportionally reduced to a size of 448*336;

[0006] S2: Calculate the global left-right optical flow of the reduced face area image and its horizontal flip, and obtain the corresponding matrix of the reduced face area image through the face parsing method for dot multiplication to obtain the optimized global left-right optical flow. Face parsing is to perform semantic segmentation on the face image, mark the face as 1, and mark the rest as O;

[0007] S3: Flip the reduced face area image to obtain a flipped face image, and concatenate it with the face area image before flipping and the global left and right optical flows optimized in S2, input them into the neural network, obtain the optical flow prediction, and resize the optical flow prediction to the size before cropping to obtain the predicted optical flow;

[0008] S4: Use the interpolation function to apply the predicted optical flow to the cropped image to obtain a corrected face image;

[0009] S5: Replace the original face image with the rectified face image.

[0010] Preferably, the neural network training method in S3 specifically includes the following steps:

[0011] S301: Obtain facial organs on either side of a training sample and mirror them to obtain a symmetrical face image, simulate the asymmetry of facial features in practice, and perform random PS on the symmetrical face image to obtain a distorted face image, and mark the symmetrical face image and the distorted face image as an image pair;

[0012] S302: Calculate the distance and direction from each pixel of the face distortion image in the image pair in S301 to the corresponding pixel of the face symmetry image, and record it as the first optical flow;

[0013] S303: performing face analysis on the face distortion image, obtaining feature point coordinates, and using the feature point coordinates to respectively establish a first grid area covering the eyebrow range and a second grid area covering the mouth and nose range;

[0014] S304: Obtain the value (v) of the corresponding position of each grid point (x, y) in the two grid areas under the action of the first optical flow x , v y ), and then add them together to get the coordinates of the grid point pixels of the face symmetry map (x+v x ,y+v y );

[0015] S305: Calculating a TPS function F that reflects the correspondence between the coordinates of the grid points of the face distortion map and the coordinates of the grid points of the face symmetry map;

[0016] S306: Establish a dense grid of points with the same size as the face distortion map. Each grid point is the pixel coordinate of the distortion sample corresponding to the face distortion map. After inputting into the TPS function F, the coordinates of the pixel points corresponding to the training sample are calculated.

[0017] Thin Plate Spline Interpolation (TPS) is a commonly used 2D interpolation method. Its matching is to give some corresponding control points between two images and minimize the overall curvature.

[0018] The loss function during the entire interpolation process is: ε = ε Φ +λε d , where ε Φ is the fitting term, which measures the distance from the source point to the target point after deformation, ε d is the distortion term, which measures the distortion of the surface, and λ is the weight coefficient, which controls the degree to which non-rigid deformation is allowed.

[0019] Among them are

[0020]

[0021]

[0022] Where N is the number of control points. After derivation, the solution of the function is:

[0023]

[0024] Where p is any point on the surface, p = (x, y)T, M = (m1, m2), U is the radial basis function;

[0025] S307: Subtract the coordinates of the distorted sample pixel points from the coordinates of the training sample pixel points in S306 to obtain an optical flow after the first optical flow is corrected, which is recorded as a second optical flow;

[0026] S308: Calculate the TPS function from the grid points of the face symmetry map to the grid points of the face distortion map to obtain the inverse function F′ of F. Since the second optical flow is obtained based on the TPS function trained on 200 sparse grid points, it cannot fully and accurately reflect the relationship between the face distortion map and the face symmetry map. Although the first optical flow directly calculates the corresponding relationship between the face distortion map and the face symmetry map, due to the limitations of the existing optical flow algorithm, the first optical flow obtained is not smooth in the airspace, which will greatly affect the imaging quality. Therefore, we calculate the TPS function from 200 grid points of the face symmetry map to 200 grid points of the face distortion map to obtain the inverse function F′ of F.

[0027] S309: inputting the face symmetry map into F′ to obtain a corrected distortion map having the same distortion form as the face distortion map;

[0028] S310: The corrected distortion map, the image of the face after the corrected distortion map is flipped to any side, and the global left and right optical flows are used as inputs, and the second optical flow is used as output to train the neural network. In order to minimize the local deformation of the trained image, the loss function of the neural network is defined as:

[0029]

[0030] d i =y i -y i * ,

[0031] where y i To predict the optical flow value, y i * is the actual optical flow value 2

[0032] Preferably, the pixel matrix size of the face image to be processed, the training sample, the face distortion map and the corrected distortion map is 1000*800, the pixel matrix size of the first grid area is 15*8, and the pixel matrix size of the second grid area is 8*10.

[0033] Preferably, the upper boundary of the first grid area is the horizontal line corresponding to the top of the eyebrow, the lower boundary is the middle line of the horizontal line where the bottom of the eye is located and the horizontal line where the top of the nose is located, the left boundary is the vertical line where the leftmost pixel point of all the pixels of the eyebrow and the eye is located, and the right boundary is the vertical line where the rightmost pixel point of all the pixels of the eyebrow and the eye is located.

[0034] Preferably, the upper boundary of the second grid area is the horizontal line where the top of the nose is located, the lower boundary is the horizontal line where the bottom of the mouth is located, the left boundary is the vertical line where the leftmost pixel of all pixels of the nose and mouth is located, and the right boundary is the vertical line where the rightmost pixel of all pixels of the nose and mouth is located.

[0035] Compared with the prior art, the present invention has the following beneficial effects: the first optical flow is calculated for the original training face image and the mirror training face image obtained after mirroring, and then the first optical flow is trained by using a convolutional neural network and a recurrent neural network, and the image is corrected by the trained first optical flow; since the first grid area and the second grid area are used to calculate the TPS function in the process of training the first optical flow, the trained first optical flow can directly regress the moving direction of each pixel point of the face in the process of correcting the face image, avoiding the problem of only calculating the sparse matrix in the traditional optical flow method resulting in uneven regressed face features, thereby achieving the effect of face beautification. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Attached Figure 1Flowchart of the face correction method proposed in this method. DETAILED DESCRIPTION

[0037] In order to make the technical means, creative features, objectives and effects of the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0038] like Figure 1 The present invention proposes a method for correcting facial features based on global optical flow and neural network, which specifically includes the following steps:

[0039] S1: Obtain a face image to be processed with a pixel matrix size of 1000*800, parse out 68 feature points of the face image to be processed, and obtain the coordinates of the outer corners of the left and right eyes respectively, mark the left corner of the left eye as (x1, y1), and the right corner of the right eye as (x2, y2), and find the midpoint coordinates of the outer corners of the left and right eyes, that is, ((x1+x2) / 2, (y1+y2) / 2), record it as the center point of the face, take the center point as the starting point, take 1.6 times the distance between the two eyes up and down, and take 1.2 times the distance between the two eyes to the left and right, and crop the image. If the up-down distance or the left-right distance is too small, padding is used to obtain a cropped image to obtain a face area image, and then the cropped face area image is proportionally reduced to a size of 448*336;

[0040] S2: Calculate the global left-right optical flow of the reduced face area image and its horizontal flip, and obtain the corresponding matrix of the reduced face area image through the face parsing method for dot multiplication to obtain the optimized global left-right optical flow. Face parsing is to perform semantic segmentation on the face image, mark the face as 1, and mark the rest as O;

[0041] S3: Flip the reduced face area image to obtain a flipped face image, and splice it with the face area image before flipping and the global left and right optical flows optimized in S2, input it into the neural network, obtain the optical flow prediction, and resize the optical flow prediction to the size before cropping to obtain the predicted optical flow.

[0042] Specifically, the training of the neural network includes the following steps:

[0043] S301: Obtaining a training sample with a pixel matrix size of 1000*800 and mirroring the facial organs on any left and right sides to obtain a symmetrical face image, performing random PS on the symmetrical face image to simulate the asymmetry of facial features in reality to obtain a distorted face image, and marking the symmetrical face image and the distorted face image as an image pair;

[0044] S302: Calculate the distance and direction from each pixel of the face distortion image in the image pair in S301 to the corresponding pixel of the face symmetry image, and record it as the first optical flow;

[0045] S303: Perform face analysis on the face distortion image to obtain feature point coordinates, and use the feature point coordinates to establish a first grid area with a pixel matrix size of 15*8 covering the eyebrow range and a second grid area with a pixel matrix size of 8*10 covering the mouth and nose range.

[0046] The upper boundary of the first grid area is the horizontal line corresponding to the top of the eyebrow, the lower boundary is the middle line of the horizontal line where the bottom of the eye is located and the horizontal line where the top of the nose is located, the left boundary is the vertical line where the leftmost pixel of all the pixels of the eyebrow and the eye is located, and the right boundary is the vertical line where the rightmost pixel of all the pixels of the eyebrow and the eye is located. The upper boundary of the second grid area is the horizontal line where the top of the nose is located, the lower boundary is the horizontal line where the bottom of the mouth is located, the left boundary is the vertical line where the leftmost pixel of all the pixels of the nose and the mouth is located, and the right boundary is the vertical line where the rightmost pixel of all the pixels of the nose and the mouth is located;

[0047] S304: Obtain the value (v) of the corresponding position of each grid point (x, y) in the two grid areas under the action of the first optical flow x , v y ), and then add them together to get the coordinates of the grid point pixels of the face symmetry map (x+v x ,y+v y );

[0048] S305: Calculate a thin plate spline interpolation (TPS) function F that reflects the correspondence between the coordinates of the grid points of the face distortion map and the coordinates of the grid points of the face symmetry map;

[0049] S306: Establish a dense grid of points with the same size as the face distortion map, where each grid point is the pixel coordinate of the distortion sample corresponding to the face distortion map, and input it into the TPS function F to calculate the coordinate of the pixel point corresponding to the training sample;

[0050] S307: Subtract the coordinates of the distorted sample pixel points from the coordinates of the training sample pixel points in S306 to obtain an optical flow after the first optical flow is corrected, which is recorded as a second optical flow;

[0051] S308: Calculate the TPS function from the grid points of the face symmetry map to the grid points of the face distortion map to obtain the inverse function F′ of F. Since the second optical flow is obtained based on the TPS function trained on 200 sparse grid points, it cannot fully and accurately reflect the relationship between the face distortion map and the face symmetry map. Although the first optical flow directly calculates the corresponding relationship between the face distortion map and the face symmetry map, due to the limitations of the existing optical flow algorithm, the first optical flow obtained is not smooth in the airspace, which will greatly affect the imaging quality. Therefore, we calculate the TPS function from 200 grid points of the face symmetry map to 200 grid points of the face distortion map to obtain the inverse function F′ of F.

[0052] S309: inputting the face symmetry map into F′ to obtain a corrected distortion map with a pixel matrix size of 1000*800 and the same distortion form as the face distortion map;

[0053] S310: using the corrected distortion map, the image of the face in the corrected distortion map flipped to any side, and the global left-right optical flow as input, and the second optical flow as output, to train the neural network;

[0054] S4: Use the interpolation function to apply the predicted optical flow to the cropped image to obtain a corrected face image;

[0055] S5: Replace the original face image with the rectified face image.

[0056] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A face correction method based on global optical flow and neural network, characterized by: The following steps are involved: S1: Obtain a face image to be processed, parse out 68 feature points of the face image to be processed, and obtain the coordinates of the outer corners of the left and right eyes respectively, calculate the average value of the coordinate positions of the outer corners of the left and right eyes, record it as the center point of the face, take the center point as the starting point, take 1.6 times the distance between the two eyes upward and downward, and take 1.2 times the distance between the two eyes to the left and right, crop the image, and obtain a face area image, and then reduce the cropped face area image to a size of 448*336 in equal proportion; S2: Calculate the global left-right optical flow of the reduced face area image and its horizontal flip, and obtain the corresponding matrix of the reduced face area image through the face parsing method, and perform dot multiplication to obtain the optimized global left-right optical flow; S3: Flip the reduced face area image to obtain a flipped face image, and concatenate it with the face area image before flipping and the global left and right optical flows optimized in S2, input them into the neural network, obtain the optical flow prediction, and resize the optical flow prediction to the size before cropping to obtain the predicted optical flow; S4: Use the interpolation function to apply the predicted optical flow to the cropped image to obtain a corrected face image; S5: Replace the original face image with the rectified face image.

2. The face correction method based on global optical flow and neural network according to claim 1, characterized in that: The training method of the neural network in S3 specifically includes the following steps: S301: Obtain facial organs on either side of a training sample and mirror them to obtain a symmetrical face image, simulate the asymmetry of facial features in practice, and perform random PS on the symmetrical face image to obtain a distorted face image, and mark the symmetrical face image and the distorted face image as an image pair; S302: Calculate the distance and direction from each pixel of the face distortion image in the image pair in S301 to the corresponding pixel of the face symmetry image, and record it as the first optical flow; S303: performing face analysis on the face distortion image, obtaining feature point coordinates, and using the feature point coordinates to respectively establish a first grid area covering the eyebrow range and a second grid area covering the mouth and nose range; S304: Obtain the value (v) of the corresponding position of each grid point (x, y) in the two grid areas under the action of the first optical flow x ,v y ), after adding, the coordinates of the grid point pixels of the face symmetry map are obtained; S305: Calculating a TPS function F that reflects the correspondence between the coordinates of the grid points of the face distortion map and the coordinates of the grid points of the face symmetry map; S306: Establish a dense grid of points with the same size as the face distortion map, where each grid point is the pixel coordinate of the distortion sample corresponding to the face distortion map, and input it into the TPS function F to calculate the coordinate of the pixel point corresponding to the training sample; S307: Subtract the coordinates of the distorted sample pixel points from the coordinates of the training sample pixel points in S306 to obtain an optical flow after the first optical flow is corrected, which is recorded as a second optical flow; S308: Calculate the TPS function from the grid points of the face symmetry map to the grid points of the face distortion map to obtain the inverse function F ’ ; S309: Input the face symmetry map into F ’ A corrected distortion image with the same distortion form as the face distortion image is obtained; S310: The corrected distortion map, the image of the face after the corrected distortion map is flipped to any side, and the global left-right optical flow are used as input, and the second optical flow is used as output to train the neural network.

3. The face correction method based on global optical flow and neural network according to claim 2, characterized in that: The pixel matrix size of the face image to be processed, the training sample, the face distortion map and the corrected distortion map is 1000*800, the pixel matrix size of the first grid area is 15*8, and the pixel matrix size of the second grid area is 8*10.

4. The face correction method based on global optical flow and neural network according to claim 2, characterized in that: The upper boundary of the first grid area is the horizontal line corresponding to the top of the eyebrow, the lower boundary is the middle line between the horizontal line where the bottom of the eye is located and the horizontal line where the top of the nose is located, the left boundary is the vertical line where the leftmost pixel point of all the pixels of the eyebrow and the eye is located, and the right boundary is the vertical line where the rightmost pixel point of all the pixels of the eyebrow and the eye is located.

5. The face correction method based on global optical flow and neural network according to claim 2, characterized in that: The upper boundary of the second grid area is the horizontal line where the top of the nose is located, the lower boundary is the horizontal line where the bottom of the mouth is located, the left boundary is the vertical line where the leftmost pixel of all the pixels of the nose and mouth is located, and the right boundary is the vertical line where the rightmost pixel of all the pixels of the nose and mouth is located.

Images