Face alignment method and device

Abstract

The application is suitable for the technical field of image processing, and provides a face alignment method, which comprises the following steps: obtaining an initial face image, and obtaining a face component attention feature map corresponding to the initial face image; determining face feature information according to the initial face image and the face component attention feature map corresponding to the initial face image; inputting the face feature information into a preset face alignment network for processing to obtain a first face feature parameter vector output by the preset face alignment network; and obtaining a target face image after alignment of the initial face image according to the first face feature parameter vector. The above method excludes the interference of irrelevant information in face alignment, and improves the overall alignment accuracy of the network. The performance of the face alignment method is no longer limited to a face base model or template defined by a 3D model, and a large amount of training data is not required, so that the error is further reduced and the face alignment accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a face alignment method and device. BACKGROUND

[0002] Face alignment is a basic and concerned research hotspot in the field of computer vision, which is the alignment process of key feature points of a face, such as eyes, nose, mouth and contour. Existing face alignment methods mainly include: using CNN to regress 3D face information, training a relatively complex network to regress a single face to obtain 68 2D feature points; and reconstructing the corresponding 3D information of a single face by estimating 3DMM (3D Morphable Model) coefficients. However, the performance of the above methods is limited by the face base model or template defined by the 3D model, and is limited by the acquisition of a large amount of training data, so that the above methods are prone to errors when used, and cannot accurately align faces. SUMMARY

[0003] The embodiments of the present application provide a face alignment method and device, which can solve the problem that the existing face alignment method is prone to errors when used and cannot accurately align faces.

[0004] In a first aspect, the embodiments of the present application provide a face alignment method, comprising:

[0005] obtaining an initial face image and obtaining a face part attention feature map corresponding to the initial face image;

[0006] determining face feature information according to the initial face image and the face part attention feature map corresponding thereto;

[0007] inputting the face feature information into a preset face alignment network for processing to obtain a first face feature parameter vector output by the preset face alignment network;

[0008] obtaining a target face image aligned from the initial face image according to the first face feature parameter vector.

[0009] Further, the obtaining of the face part attention feature map corresponding to the initial face image comprises:

[0010] segmenting the initial face image to obtain a face image segmentation map;

[0011] obtaining a face image grayscale map of the initial face image;

[0012] determining a face part attention feature map corresponding to the face image segmentation map according to the face image segmentation map and the face image grayscale map.

[0013] Further, the determining the face component attention feature map corresponding to the initial face image according to the face image segmentation map and the face image grayscale map comprises:

[0014] multiplying the face image segmentation map and the face image grayscale map to obtain a grayscale face component attention feature map;

[0015] processing the grayscale face component attention feature map according to a preset Gaussian smoothing algorithm to obtain the face component attention feature map corresponding to the face image segmentation map.

[0016] Further, the face part contained in the face image segmentation map is a face part in the face image except hair, ears and background.

[0017] Further, the preset face alignment network comprises a first convolutional layer, a first max-pooling layer, a second convolutional layer, a first fully connected layer and a second fully connected layer.

[0018] The processing of the face feature information by inputting the face feature information into the preset face alignment network to obtain the first face feature parameter vector output by the preset face alignment network comprises:

[0019] extracting first feature information corresponding to the face feature information by inputting the face feature information into the first convolutional layer for feature extraction;

[0020] extracting second feature information from the first feature information by inputting the first feature information into the first max-pooling layer for processing;

[0021] extracting a position-aware feature from the second feature information by inputting the second feature information into the second convolutional layer for processing;

[0022] obtaining a first parameter vector by inputting the position-aware feature into the first fully connected layer for regression processing;

[0023] obtaining the first face feature parameter vector output by the preset face alignment network by inputting the first parameter vector into the second fully connected layer for regression processing.

[0024] Further, before the processing of the face feature information by inputting the face feature information into the preset face alignment network to obtain the first face feature parameter vector output by the preset face alignment network, the method further comprises:

[0025] obtaining a plurality of sample face feature information and corresponding vector labels; wherein the vector labels are used to identify face feature parameter vectors corresponding to the sample face feature information;

[0026] The face alignment network is obtained by training a preset deep learning network based on the facial feature information of multiple samples and their corresponding vector labels.

[0027] Furthermore, the face alignment network is obtained by training a preset deep learning network based on the facial feature information of multiple samples and their corresponding vector labels, including the following steps:

[0028] Multiple sample facial feature information are input into a preset deep learning network to obtain the second facial feature parameter vector output by the deep learning network;

[0029] Based on the second facial feature parameter vector, the vector label, and the preset loss function, the training error of the preset deep learning network is obtained;

[0030] When the training error does not meet the preset termination condition, the preset deep learning network is trained iteratively based on the training error until the training error meets the preset termination condition, thus obtaining the face alignment network.

[0031] Furthermore, the preset loss functions include: weighted parameter distance loss function, vertex distance loss function, and symmetry loss function.

[0032] Furthermore, when the loss function is a weighted parametric distance loss function, obtaining the training error of the preset deep learning network based on the second face feature parameter vector, the vector label, and the preset loss function includes:

[0033] The training error of the preset deep learning network is obtained based on the second facial feature parameter vector, the vector label, and the weighted parameter distance loss function.

[0034] Further, when the loss function is a vertex distance loss function, the step of obtaining the training error of the preset deep learning network based on the second face feature parameter vector, the face feature parameter vector corresponding to the sample face feature information identified by the vector label, and the preset loss function includes:

[0035] Based on the second face feature parameter vector, the face feature parameter vector corresponding to the sample face feature information identified by the vector label, and the vertex distance loss function, the training error of the preset deep learning network is obtained.

[0036] Further, when the loss function is a symmetric loss function, the step of obtaining the training error of the preset deep learning network based on the output face feature parameter vector, the face feature parameter vector corresponding to the sample face feature information identified by the vector label, and the preset loss function includes:

[0037] Based on the second facial feature parameter vector, the vector label, and the symmetry loss function, the training error of the preset deep learning network is obtained.

[0038] Furthermore, the first facial feature parameter vector includes one or more of the following: shape coefficient, expression coefficient, scale factor, rotation matrix, and displacement parameter.

[0039] Further, obtaining the target face image aligned with the initial face image based on the first face feature parameter vector includes:

[0040] Based on the first face feature parameter vector and the preset calculation rules, the target face vertex information after the initial face image is aligned is obtained;

[0041] The target face image is obtained based on the target face vertex information.

[0042] Further, obtaining the target face vertex information after alignment of the initial face image based on the first face feature parameter vector includes:

[0043] A 3D face model corresponding to the initial face image is constructed based on at least one of the preset 3D face construction rules, the shape coefficient, and the expression coefficient;

[0044] Based on at least one of the scale factor, the rotation matrix, and the displacement parameter, the target face vertex information after alignment with the initial face image is obtained using the 3D face model.

[0045] Secondly, embodiments of this application provide a face alignment device, comprising:

[0046] The first acquisition unit is used to acquire an initial face image and acquire the face component attention feature map corresponding to the initial face image;

[0047] The first determining unit is used to determine facial feature information based on the initial face image and its corresponding face component attention feature map;

[0048] The first processing unit is used to input the facial feature information into a preset face alignment network for processing, and obtain the first facial feature parameter vector output by the preset face alignment network.

[0049] The second processing unit is used to obtain the target face vertex information after the initial face image is aligned based on the first face feature parameter vector.

[0050] Furthermore, the first acquisition unit is specifically used for:

[0051] The initial face image is segmented to obtain the face image segmentation map;

[0052] Obtain the grayscale image of the face from the initial face image;

[0053] Based on the face image segmentation map and the face image grayscale map, determine the face component attention feature map corresponding to the face image segmentation map.

[0054] Furthermore, the first acquisition unit is specifically used for:

[0055] Multiply the face image segmentation map and the face image grayscale map to obtain a grayscale face component attention feature map;

[0056] The grayscale face component attention feature map is processed using a preset Gaussian smoothing algorithm to obtain the face component attention feature map corresponding to the face image segmentation map.

[0057] Furthermore, the face regions included in the face image segmentation map are the face regions in the face image excluding hair, ears, and background.

[0058] Furthermore, the preset face alignment network includes a first convolutional layer, a first max pooling layer, a second convolutional layer, a first fully connected layer, and a second fully connected layer;

[0059] Further, the first processing unit is specifically used for:

[0060] The facial feature information is input into the first convolutional layer for feature extraction to obtain the first feature information corresponding to the facial feature information;

[0061] The first feature information is input into the first max pooling layer for processing, and the second feature information is extracted from the first feature information.

[0062] The second feature information is input into the second convolutional layer for processing, and position-aware features are extracted from the second feature information.

[0063] The location-aware features are input into the first fully connected layer for regression processing to obtain the first parameter vector;

[0064] The first parameter vector is input into the second fully connected layer for regression processing to obtain the first face feature parameter vector output by the preset face alignment network.

[0065] Furthermore, the face alignment device also includes:

[0066] The second acquisition unit is used to acquire multiple sample face feature information and their corresponding vector labels; wherein, the vector labels are used to identify the face feature parameter vector corresponding to the sample face feature information.

[0067] The first training unit is used to train a preset deep learning network based on the facial feature information of multiple samples and their corresponding vector labels to obtain the face alignment network.

[0068] Furthermore, the first training unit includes:

[0069] The third acquisition unit is used to input multiple sample facial feature information into a preset deep learning network to obtain the second facial feature parameter vector output by the deep learning network.

[0070] The fourth acquisition unit is used to acquire the training error of the preset deep learning network based on the second face feature parameter vector, the vector label and the preset loss function.

[0071] The second training unit, when the training error does not meet the preset termination condition, iteratively trains the preset deep learning network based on the training error until the training error meets the preset termination condition, thereby obtaining the face alignment network.

[0072] Furthermore, the preset loss functions include: weighted parameter distance loss function, vertex distance loss function, and symmetry loss function.

[0073] Furthermore, when the loss function is a weighted parameter distance loss function, the fourth acquisition unit is specifically used for:

[0074] The training error of the preset deep learning network is obtained based on the second facial feature parameter vector, the vector label, and the weighted parameter distance loss function.

[0075] Furthermore, when the loss function is a vertex distance loss function, the fourth acquisition unit is specifically used for:

[0076] The training error of the preset deep learning network is obtained based on the second face feature parameter vector, the vector label, and the vertex distance loss function.

[0077] Furthermore, when the loss function is a symmetric loss function, the fourth acquisition unit is specifically used for:

[0078] Based on the second facial feature parameter vector, the vector label, and the symmetry loss function, the training error of the preset deep learning network is obtained.

[0079] Furthermore, the first facial feature parameter vector includes one or more of the following: shape coefficient, expression coefficient, scale factor, rotation matrix, and displacement parameter.

[0080] Further, the second processing unit includes:

[0081] The third processing unit is used to obtain the target face vertex information after the initial face image is aligned, based on the first face feature parameter vector and the preset calculation rules.

[0082] The fourth processing unit is used to obtain the target face image based on the target face vertex information.

[0083] Furthermore, the third processing unit is specifically used for:

[0084] A 3D face model corresponding to the initial face image is constructed based on at least one of the preset 3D face construction rules, the shape coefficient, and the expression coefficient;

[0085] Based on at least one of the scale factor, the rotation matrix, and the displacement parameter, the target face vertex information after alignment with the initial face image is obtained using the 3D face model.

[0086] Thirdly, embodiments of this application provide a face alignment device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the face alignment method as described in the first aspect above.

[0087] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the face alignment method described in the first aspect above.

[0088] In this embodiment, an initial face image is acquired, and a corresponding face component attention feature map is obtained. Face feature information is determined based on the initial face image and its corresponding face component attention feature map. This face feature information is then input into a preset face alignment network for processing, resulting in a first face feature parameter vector output by the preset face alignment network. Finally, a target face image aligned with the initial face image is obtained based on the first face feature parameter vector. This method extracts the face component attention feature map and fuses it with the face image, eliminating interference from irrelevant information during face alignment and improving the overall alignment accuracy of the network. The preset face alignment network accurately obtains the face feature parameter vector, further improving the final face alignment accuracy. The performance of this face alignment method is no longer limited to defining a basic face model or template for a 3D model, and it does not require acquiring a large amount of training data. This further reduces errors and improves the accuracy of face alignment during use. Attached Figure Description

[0089] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 This is a schematic flowchart of a face alignment method provided in the first embodiment of this application;

[0091] Figure 2 This is a schematic flowchart of steps S1011 to S1013 in a face alignment method provided in the first embodiment of this application;

[0092] Figure 3 This is a schematic diagram of the structure of the preset face alignment network in a face alignment method provided in the first embodiment of this application;

[0093] Figure 4 This is a detailed flowchart of S104 in the face alignment method provided in the first embodiment of this application;

[0094] Figure 5 This is a detailed schematic flowchart of step S1041 in the face alignment method provided in the first embodiment of this application;

[0095] Figure 6 This is a schematic diagram of the face alignment device provided in the second embodiment of this application;

[0096] Figure 7 This is a schematic diagram of the face alignment device provided in the third embodiment of this application. Detailed Implementation

[0097] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0098] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0099] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0100] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0101] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0102] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0103] Please see Figure 1 , Figure 1This is a schematic flowchart illustrating a face alignment method provided in the first embodiment of this application. In this embodiment, the execution subject of the face alignment method is a device with face alignment functionality, such as a desktop computer, server, mobile device, etc. Figure 1 The face alignment methods shown may include:

[0104] S101: Obtain an initial face image and obtain the face component attention feature map corresponding to the initial face image.

[0105] The device acquires an initial face image. This initial face image includes the face itself. The initial face image can be acquired through the device's built-in image acquisition unit, or it can be acquired by directly receiving an initial face image sent from another device; there are no restrictions on this.

[0106] After acquiring the initial face image, the device obtains the corresponding facial component attention feature map. Here, facial components can be understood as parts of the face, such as the eyes, mouth, and nose. The facial component attention feature map is the attention feature map of the facial component image, which can be used to identify facial features.

[0107] The device can obtain the facial component attention feature map corresponding to the initial facial image by pre-setting facial component attention feature map acquisition rules. For example, the device can pre-set a facial component attention feature map acquisition model. The device inputs the initial facial image into the pre-set facial component attention feature map acquisition model to obtain the facial component attention feature map corresponding to the initial facial image.

[0108] In one embodiment, the device can obtain the facial component attention feature map corresponding to the initial facial image using the methods in S1011 to S1013, such as... Figure 2 As shown, S1011 to S1013 are as follows:

[0109] S1011: Segment the initial face image to obtain the face image segmentation map.

[0110] The device segments an initial face image to obtain a face image segmentation map. The device can have a pre-set segmentation size, and the device segments the initial face image according to the segmentation size to obtain the face image segmentation map; alternatively, the device can have pre-set segmentation rules, for example, the device can be preset to first identify the facial features in the face image, and then obtain the face image segmentation map based on the identified facial features.

[0111] Understandably, since the attention feature map of the face component image is used to identify facial features, the resulting face image segmentation map should include only the distinctive facial components. Specifically, it is generally difficult to obtain facial features from images such as hair, ears, and background. Therefore, to obtain a high-quality face component attention feature map and thus improve the accuracy of face alignment, the face image segmentation map includes all facial components in the face image except for hair, ears, and background.

[0112] S1012: Obtain the grayscale image of the face from the initial face image.

[0113] The device performs grayscale processing on the initial face image to obtain a grayscale image of the face. This grayscale image has only one channel with 256 gray levels, where 255 represents pure white and 0 represents pure black. The specific grayscale processing method can utilize existing grayscale image conversion techniques, which will not be elaborated upon here.

[0114] S1013: Based on the face image segmentation map and the face image grayscale map, determine the face component attention feature map corresponding to the face image segmentation map.

[0115] The device determines the facial component attention feature map corresponding to the facial image segmentation map based on the facial image segmentation map and the facial image grayscale map. Specifically, the device can multiply the facial image segmentation map and the facial image grayscale map to obtain the facial component attention feature map corresponding to the facial image segmentation map.

[0116] In one embodiment, to suppress errors caused by face segmentation, the device multiplies the face image segmentation map and the face image grayscale map to obtain a grayscale face component attention feature map; the grayscale face component attention feature map is then processed according to a preset Gaussian smoothing algorithm to obtain the face component attention feature map corresponding to the face image segmentation map. In this embodiment, the final face component attention feature map obtained is a smoothed grayscale face component attention feature map.

[0117] S102: Determine facial feature information based on the initial face image and its corresponding face component attention feature map.

[0118] The device determines facial feature information based on an initial face image and its corresponding facial component attention feature map. Specifically, the device can stitch together the initial face image and its corresponding facial component attention feature map to obtain a multi-dimensional data block, which is the facial feature information. This facial feature information serves as the input to a pre-defined face alignment network.

[0119] S103: Input the facial feature information into a preset face alignment network for processing to obtain the first facial feature parameter vector output by the preset face alignment network.

[0120] The device inputs facial feature information into a preset face alignment network for processing, obtaining a first facial feature parameter vector output by the preset face alignment network. The device pre-stores the preset face alignment network, and the training process of the preset face alignment network is as follows:

[0121] Multiple sample facial feature information and their corresponding vector labels are obtained; the vector labels are used to identify the facial feature parameter vectors corresponding to the sample facial feature information. Then, a preset deep learning network is trained based on the multiple sample facial feature information and their corresponding vector labels to obtain the face alignment network.

[0122] Here, the preset deep learning network refers to an untrained network. After training this deep learning network, the resulting face alignment network is obtained. The preset deep learning network can be any type of deep learning network, and no specific limitation is made here.

[0123] Understandably, the preset face alignment network can be pre-trained by the device, or it can be pre-trained by another device and then the corresponding file is ported to that device. Specifically, when the other device trains the deep learning network, it freezes the model parameters of the deep learning network and then ports the preset face alignment network file corresponding to the frozen deep learning network to the device.

[0124] In one implementation, to improve the accuracy of the preset face alignment network and obtain accurate face feature parameter vectors, thereby improving the accuracy of face alignment, a loss function can be used to calculate the training error during training. The specific training process using the loss function is as follows:

[0125] Based on the second facial feature parameter vector, the vector label, and the preset loss function, the training error of the preset deep learning network is obtained;

[0126] When the training error does not meet the preset termination condition, the preset deep learning network is trained iteratively based on the training error until the training error meets the preset termination condition, thus obtaining the face alignment network.

[0127] The preset termination condition is pre-set in the device and can be reasonably set based on the accuracy requirements of the face alignment network.

[0128] The loss function in this preset can include: weighted parametric distance loss function, vertex distance loss function, and symmetry loss function.

[0129] Since the goal of face alignment is to make the reconstructed 3D face coincide with the real face in 3D space, the face feature parameter vector is optimized by minimizing the error between the reconstructed vertices and the ground truth 3D face.

[0130] Specifically, when the loss function is the vertex distance loss function, the device can obtain the training error of the preset deep learning network based on the second face feature parameter vector, the vector label, and the vertex distance loss function.

[0131] The vertex distance loss function is:

[0132] L vdc =||V(p)-V(p) g )|| 2

[0133] Where V(.) represents the coordinates of the vertex of the face reconstructed from the face feature parameter vector and projected onto the image plane. The parameter p is the face feature parameter vector, p g For L vdc The vertex distance loss function, compared to directly minimizing the error of the regressed face feature parameter vector, converts the regressed face feature parameter vector into semantic information, which can more accurately express the fitting error. However, the vertex distance loss function itself is not a convex function, so directly using the vertex distance loss function cannot guarantee that the network will converge completely.

[0134] Therefore, in this embodiment, the loss function may further include a weighted parametric distance loss function. When the loss function is a weighted parametric distance loss function, the device can obtain the training error of the preset deep learning network based on the second face feature parameter vector, the vector label, and the weighted parametric distance loss function.

[0135] The weighted distance loss is:

[0136] L wpdc =(p g -p) T diag(w)(p g -p)

[0137] Where w represents the parameter importance vector, defined as follows:

[0138] w = (w1, w2, ..., w i ,...,w p )

[0139] w i =||V(p) d,i )-V(p g )|| / Z

[0140]

[0141] Where, parameter p is the face feature parameter vector, p d,i This indicates that the i-th parameter in parameter p is the value predicted by the network, while parameters at other positions are obtained from the ground truth. Z is the largest element of the regularization parameter w. represents the vertex distance error caused by the network's prediction of the i-th parameter, i.e., the importance of the i-th parameter. During training, the network first fits parameters with large vertex distance errors, such as rotation and translation parameters. As the parameters with large vertex distances approach the ground truth, their weights begin to converge, and the network starts optimizing other important parameters while maintaining the accuracy of high-priority parameters. Compared to vertex error loss, weighted parameter distance loss ensures that parameters continuously approach the ground truth and makes the network easier to train.

[0142] When the loss function is a symmetric loss function, the device obtains the training error of the preset deep learning network based on the output face feature parameter vector, the face feature parameter vector corresponding to the sample face feature information identified by the vector label, and the symmetric loss function.

[0143] The symmetry loss function is as follows:

[0144] L sym =||D(V(p) l ,N)-D(V(p) r ,N)|| 2

[0145] Where D(.) represents the distance from the face vertex to the central plane N, and l and r represent the left and right vertices, respectively. The symmetry loss mainly utilizes the approximate central symmetry of the face to improve the accuracy of face alignment.

[0146] The device inputs facial feature information into a preset face alignment network for feature extraction, feature scaling, and other processing to obtain a facial feature parameter vector.

[0147] Specifically, the preset face alignment network may include a first convolutional layer, a first max pooling layer, a second convolutional layer, a first fully connected layer, and a second fully connected layer.

[0148] Understandably, the pre-defined face alignment network can include more convolutional and pooling layers, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the pre-defined face alignment network, where Conv represents a convolutional layer, FC represents a fully connected layer, and ReLU is the activation function. Figure 3The preset face alignment network includes four convolutional layers. The first three convolutional layers are each followed by a max pooling layer to reduce the size of the feature maps and thus reduce computation. The last convolutional layer is used to extract position-aware features and then regress them into a 256-dimensional feature vector using a fully connected layer.

[0149] The specific process by which the device inputs facial feature information into a preset face alignment network for processing, and obtains the first facial feature parameter vector output by the preset face alignment network, is as follows:

[0150] The device inputs facial feature information into the first convolutional layer for feature extraction, obtaining the first feature information corresponding to the facial feature information. In the preset face alignment network, each convolutional layer consists of several convolutional units, and the parameters of each convolutional unit are optimized using the backpropagation algorithm. The first convolutional layer is used to extract features from the input facial feature information, extracting the first feature information corresponding to the facial feature information. The first feature information may include facial expression features, shape features, pose features, etc.

[0151] The device inputs the first feature information into the first max pooling layer for processing, extracting the second feature information from it. The first pooling layer further reduces the size of the first feature information, decreasing computational complexity and extracting effective feature information to obtain the second feature information. Pooling, also known as downsampling, is primarily used for feature dimensionality reduction, compressing the amount of data and parameters, reducing overfitting, and improving the model's fault tolerance. Pooling layers typically apply to each input feature individually and reduce its size. The most commonly used form of pooling layer currently involves dividing the image into 2x2 blocks every two elements and then taking the maximum value of the four numbers in each block. This reduces the amount of data by 75%.

[0152] The device inputs the second feature information into the second convolutional layer for processing, extracting position-aware features from the second feature information; the position-aware features are then input into the first fully connected layer for regression processing to obtain the first parameter vector. The fully connected layer acts as a "classifier" in the entire convolutional neural network. Each node in the fully connected layer is connected to all nodes in the previous layer, used to synthesize the previously extracted features. Due to its fully connected nature, the fully connected layer generally has the most parameters. The first parameter vector can be a multi-dimensional parameter vector.

[0153] Finally, the device inputs the first parameter vector into the second fully connected layer for regression processing to obtain the first face feature parameter vector output by the preset face alignment network.

[0154] The purpose of convolution is to extract different features from the input. The first convolutional layer may only extract some low-level features such as edges, lines, and corners. A network with more convolutional layers can iteratively extract more complex features from low-level features. Therefore, in one possible implementation, the preset face alignment network may include more convolutional layers. For example, the preset face alignment network may include 4 convolutional layers, 3 max pooling layers, and 2 fully connected layers. Each of the first three convolutional layers is followed by a max pooling layer, and the fourth convolutional layer is followed by a fully connected layer. The final output of the preset face alignment network is the face feature parameter vector regressed by a fully connected layer.

[0155] In one embodiment, the first facial feature parameter vector includes one or more of the following: shape coefficients, expression coefficients, scale factors, rotation matrices, and translation parameters. The shape parameters and expression parameters are coefficients used in constructing the 3D facial model, while the scale factors, rotation matrices, and translation parameters are used to construct the vertex information of the target face.

[0156] For example, the face feature parameter vector output by the pre-defined face alignment network can be a multi-dimensional parameter vector, which includes a 7-dimensional pose vector, a scale factor, a rotation matrix (three rotation angles on the x, y, and z axes), displacement parameters in three directions, a shape coefficient, and an expression coefficient.

[0157] S104: Obtain the target face image after alignment of the initial face image based on the first face feature parameter vector.

[0158] The device obtains the target face image aligned with the initial face image based on the first face feature parameter vector. Specifically, the device can store preset calculation rules, which are used to obtain the target face image aligned with the initial face image. In this embodiment, there are no specific restrictions on the preset calculation rules, as long as the target face image aligned with the initial face image can be determined based on the first face feature parameter vector.

[0159] In one implementation, such as Figure 4 As shown, S1041 to S1042 are a specific method for obtaining the target face image after face image alignment. S1041 to S1042 are as follows:

[0160] S1041: Based on the first face feature parameter vector and the preset calculation rules, obtain the target face vertex information after the initial face image is aligned.

[0161] The device stores preset calculation rules. The device first obtains the target face vertex information after the initial face image is aligned based on the face feature parameter vector and the preset calculation rules.

[0162] In one implementation, such as Figure 5 As shown, S10411 to S10412 are specific methods for obtaining the vertex information of the target face after the initial face image is aligned. S10411 to S10412 are as follows:

[0163] S10411: Construct a 3D face model corresponding to the initial face image according to at least one of the preset 3D face construction rules, the shape coefficient, and the expression coefficient.

[0164] The device stores preset 3D face construction rules. Based on at least one of the preset 3D face construction rules, shape parameters, and expression parameters, the device constructs a 3D face model corresponding to the initial face image. The preset 3D face construction rules can be a preset 3D face construction model. This model can be understood as adding facial expression and shape features to preset baseline face information to obtain the 3D face model corresponding to the initial face image. A specific preset 3D face construction model can be as follows:

[0165]

[0166] in, Represented as the preset average face shape, A id α id For facial shape feature information, A exp α exp This refers to facial expression features. A id α represents the shape components of a 3D face under neutral facial expressions captured by a 3D facial scanner. id A represents the shape factor. exp α represents the facial expression components of a three-dimensional face captured by a 3D facial scanner. exp This represents the expression coefficient.

[0167] Preset average face shape and shape component A id It can be obtained from the Basel face model (BFM), with expression components A. exp It can be obtained from the 3D facial expression database (FaceWarehouse).

[0168] In this embodiment, 199 shape components A can be obtained. id ={1, 2, ..., 199} to represent identity characteristics, such as tall / short, light / heavy, male / female, etc.; 29 facial expression components A exp = {1, 2, ..., 29} to represent facial features, such as opening the mouth, smiling, blinking, etc.

[0169] S10412: Based on at least one of the scale factor, the rotation matrix, and the displacement parameter, obtain the target face vertex information after alignment with the initial face image using the 3D face model.

[0170] The device obtains the target face vertex information aligned with the initial face image based on at least one of the scale factor, rotation matrix, and translation parameters, and uses the 3D face model. Specifically, the device can set calculation rules corresponding to the target face vertex information, and calculate the target face vertex information aligned with the initial face image according to the calculation rules.

[0171] The calculation rule for the target face vertex information can be:

[0172] V(p) = f * R * S + t 3d

[0173] Where V(p) are the 3D face vertices, f is the scale factor, R is the rotation matrix, and t 3d Let S be the displacement parameter, and S be the 3D face model.

[0174] S1042: Based on the target face vertex information, obtain the target face image.

[0175] After obtaining the target face vertex information, the device marks each face vertex in the initial face image based on the target face vertex information, and obtains the target face image after the initial face image is aligned.

[0176] In this embodiment, an initial face image is acquired, and a corresponding face component attention feature map is obtained. Face feature information is determined based on the initial face image and its corresponding face component attention feature map. This face feature information is then input into a preset face alignment network for processing, resulting in a first face feature parameter vector output by the preset face alignment network. Finally, a target face image aligned with the initial face image is obtained based on the first face feature parameter vector. This method extracts the face component attention feature map and fuses it with the face image, eliminating interference from irrelevant information during face alignment and improving the overall alignment accuracy of the network. The preset face alignment network accurately obtains the face feature parameter vector, further improving the final face alignment accuracy. The performance of this face alignment method is no longer limited to defining a basic face model or template for a 3D model, and it does not require acquiring a large amount of training data. This further reduces errors and improves the accuracy of face alignment during use.

[0177] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0178] Please see Figure 6 , Figure 6 This is a schematic diagram of the face alignment device provided in the second embodiment of this application. The included units are used to perform... Figures 1-2 , Figures 3-5 The steps in the corresponding embodiments. Please refer to the details. Figures 1-2 , Figures 3-5 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 6 The face alignment device 6 includes:

[0179] The first acquisition unit 610 is used to acquire an initial face image and acquire the face component attention feature map corresponding to the initial face image;

[0180] The first determining unit 620 is used to determine facial feature information based on the initial face image and its corresponding face component attention feature map;

[0181] The first processing unit 630 is used to input the facial feature information into a preset face alignment network for processing, and obtain the first facial feature parameter vector output by the preset face alignment network.

[0182] The second processing unit 640 is used to obtain the target face image after alignment of the initial face image based on the first face feature parameter vector.

[0183] Furthermore, the first acquisition unit 610 is specifically used for:

[0184] The initial face image is segmented to obtain the face image segmentation map;

[0185] Obtain the grayscale image of the face from the initial face image;

[0186] Based on the face image segmentation map and the face image grayscale map, determine the face component attention feature map corresponding to the face image segmentation map.

[0187] Furthermore, the first acquisition unit 610 is specifically used for:

[0188] Multiply the face image segmentation map and the face image grayscale map to obtain a grayscale face component attention feature map;

[0189] The grayscale face component attention feature map is processed using a preset Gaussian smoothing algorithm to obtain the face component attention feature map corresponding to the face image segmentation map.

[0190] Furthermore, the face regions included in the face image segmentation map are the face regions in the face image excluding hair, ears, and background.

[0191] Furthermore, the preset face alignment network includes a first convolutional layer, a first max pooling layer, a second convolutional layer, a first fully connected layer, and a second fully connected layer;

[0192] Furthermore, the first processing unit 630 is specifically used for:

[0193] The facial feature information is input into the first convolutional layer for feature extraction to obtain the first feature information corresponding to the facial feature information;

[0194] The first feature information is input into the first max pooling layer for processing, and the second feature information is extracted from the first feature information.

[0195] The second feature information is input into the second convolutional layer for processing, and position-aware features are extracted from the second feature information.

[0196] The location-aware features are input into the first fully connected layer for regression processing to obtain the first parameter vector;

[0197] The first parameter vector is input into the second fully connected layer for regression processing to obtain the first face feature parameter vector output by the preset face alignment network.

[0198] Furthermore, the face alignment device 6 also includes:

[0199] The second acquisition unit is used to acquire multiple sample facial feature information and their corresponding vector labels; wherein, the vector labels are used to identify the facial feature parameter vector corresponding to the sample facial feature information.

[0200] The first training unit is used to train a preset deep learning network based on the facial feature information of multiple samples and their corresponding vector labels to obtain the face alignment network.

[0201] Furthermore, the first training unit includes:

[0202] The third acquisition unit is used to input multiple sample facial feature information into a preset deep learning network to obtain the second facial feature parameter vector output by the deep learning network.

[0203] The fourth acquisition unit is used to acquire the training error of the preset deep learning network based on the second face feature parameter vector, the vector label and the preset loss function.

[0204] The second training unit, when the training error does not meet the preset termination condition, iteratively trains the preset deep learning network based on the training error until the training error meets the preset termination condition, thereby obtaining the face alignment network.

[0205] Furthermore, the preset loss functions include: weighted parameter distance loss function, vertex distance loss function, and symmetry loss function.

[0206] Furthermore, when the loss function is a weighted parameter distance loss function, the fourth acquisition unit is specifically used for:

[0207] The training error of the preset deep learning network is obtained based on the second facial feature parameter vector, the vector label, and the weighted parameter distance loss function.

[0208] Furthermore, when the loss function is a vertex distance loss function, the fourth acquisition unit is specifically used for:

[0209] The training error of the preset deep learning network is obtained based on the second face feature parameter vector, the vector label, and the vertex distance loss function.

[0210] Furthermore, when the loss function is a symmetric loss function, the fourth acquisition unit is specifically used for:

[0211] Based on the second facial feature parameter vector, the vector label, and the symmetry loss function, the training error of the preset deep learning network is obtained.

[0212] Furthermore, the first facial feature parameter vector includes one or more of the following: shape coefficient, expression coefficient, scale factor, rotation matrix, and displacement parameter.

[0213] Further, the second processing unit 640 includes:

[0214] The third processing unit is used to obtain the target face vertex information after the initial face image is aligned, based on the first face feature parameter vector and the preset calculation rules.

[0215] The fourth processing unit is used to obtain the target face image based on the target face vertex information.

[0216] Furthermore, the third processing unit is specifically used for:

[0217] A 3D face model corresponding to the initial face image is constructed based on at least one of the preset 3D face construction rules, the shape coefficient, and the expression coefficient;

[0218] Based on at least one of the scale factor, the rotation matrix, and the displacement parameter, the target face vertex information after alignment with the initial face image is obtained using the 3D face model.

[0219] Figure 7 This is a schematic diagram of the face alignment device provided in the third embodiment of this application. Figure 7 As shown, the face alignment device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a face alignment program. When the processor 70 executes the computer program 72, it implements the steps in the various face alignment method embodiments described above, for example... Figure 1 Steps 101 to 104 are shown. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 6 The functions of modules 610 to 640 are shown.

[0220] For example, the computer program 72 can be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 72 in the face alignment device 7. For example, the computer program 72 can be divided into a first acquisition unit, a first determination unit, a first processing unit, and a second processing unit, with the specific functions of each unit as follows:

[0221] The first acquisition unit is used to acquire an initial face image and acquire the face component attention feature map corresponding to the initial face image;

[0222] The first determining unit is used to determine facial feature information based on the initial face image and its corresponding face component attention feature map;

[0223] The first processing unit is used to input the facial feature information into a preset face alignment network for processing, and obtain the first facial feature parameter vector output by the preset face alignment network.

[0224] The second processing unit is used to obtain the target face image after alignment of the initial face image based on the first face feature parameter vector.

[0225] The face alignment device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that... Figure 7This is merely an example of the face alignment device 7 and does not constitute a limitation on the face alignment device 7. It may include more or fewer components than shown, or combine certain components, or different components. For example, the face alignment device may also include input / output devices, network access devices, buses, etc.

[0226] The processor 70 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0227] The memory 71 can be an internal storage unit of the face alignment device 7, such as a hard drive or memory of the face alignment device 7. The memory 71 can also be an external storage device of the face alignment device 7, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the face alignment device 7. Furthermore, the face alignment device 7 can include both internal and external storage units. The memory 71 is used to store the computer program and other programs and data required by the face alignment device. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0228] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0229] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0230] This application also provides a face alignment device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, it implements the steps in any of the above method embodiments.

[0231] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0232] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0233] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0234] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0235] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0236] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0237] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0238] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A face alignment method, characterized in that, include: Obtain an initial face image, and obtain the face component attention feature map corresponding to the initial face image; Based on the initial face image and its corresponding face component attention feature map, facial feature information is determined; Obtain multiple sample facial feature information and their corresponding vector labels; wherein, the vector labels are used to identify the facial feature parameter vector corresponding to the sample facial feature information; A face alignment network is obtained by training a pre-defined deep learning network based on the facial feature information of multiple samples and their corresponding vector labels. The facial feature information is input into a preset face alignment network for processing to obtain the first facial feature parameter vector output by the preset face alignment network; The target face image is obtained by aligning the initial face image with the first face feature parameter vector.

2. The face alignment method as described in claim 1, characterized in that, The step of obtaining the facial component attention feature map corresponding to the initial facial image includes: The initial face image is segmented to obtain a face image segmentation map; Obtain the grayscale image of the face from the initial face image; Based on the face image segmentation map and the face image grayscale map, determine the face component attention feature map corresponding to the face image segmentation map.

3. The face alignment method as described in claim 2, characterized in that, The step of determining the facial component attention feature map corresponding to the initial facial image based on the facial image segmentation map and the facial image grayscale image includes: Multiply the face image segmentation map and the face image grayscale map to obtain a grayscale face component attention feature map; The grayscale face component attention feature map is processed using a preset Gaussian smoothing algorithm to obtain the face component attention feature map corresponding to the face image segmentation map.

4. The face alignment method as described in claim 2, characterized in that, The face segmentation map includes the face parts in the face image excluding hair, ears, and background.

5. The face alignment method as described in claim 1, characterized in that, The preset face alignment network includes a first convolutional layer, a first max pooling layer, a second convolutional layer, a first fully connected layer, and a second fully connected layer; The step of inputting the facial feature information into a preset face alignment network for processing to obtain a first facial feature parameter vector output by the preset face alignment network includes: The facial feature information is input into the first convolutional layer for feature extraction to obtain the first feature information corresponding to the facial feature information; The first feature information is input into the first max pooling layer for processing, and the second feature information is extracted from the first feature information. The second feature information is input into the second convolutional layer for processing, and position-aware features are extracted from the second feature information. The location-aware features are input into the first fully connected layer for regression processing to obtain the first parameter vector; The first parameter vector is input into the second fully connected layer for regression processing to obtain the first face feature parameter vector output by the preset face alignment network.

6. The face alignment method according to claim 1, characterized in that, The face alignment network is obtained by training a preset deep learning network based on the facial feature information of multiple samples and their corresponding vector labels, including the following steps: Multiple sample facial feature information are input into a preset deep learning network to obtain the second facial feature parameter vector output by the deep learning network; Based on the second facial feature parameter vector, the vector label, and the preset loss function, the training error of the preset deep learning network is obtained; When the training error does not meet the preset termination condition, the preset deep learning network is trained iteratively based on the training error until the training error meets the preset termination condition, thus obtaining the face alignment network.

7. The face alignment method as described in claim 6, characterized in that, The preset loss functions include: weighted parameter distance loss function, vertex distance loss function, and symmetry loss function.

8. The face alignment method as described in claim 7, characterized in that, When the loss function is a weighted parametric distance loss function, obtaining the training error of the preset deep learning network based on the second face feature parameter vector, the vector label, and the preset loss function includes: The training error of the preset deep learning network is obtained based on the second facial feature parameter vector, the vector label, and the weighted parameter distance loss function.

9. The face alignment method as described in claim 7, characterized in that, When the loss function is a vertex distance loss function, the step of obtaining the training error of the preset deep learning network based on the second face feature parameter vector, the vector label, and the preset loss function includes: The training error of the preset deep learning network is obtained based on the second face feature parameter vector, the vector label, and the vertex distance loss function.

10. The face alignment method as described in claim 7, characterized in that, When the loss function is a symmetric loss function, obtaining the training error of the preset deep learning network based on the output face feature parameter vector, the vector label, and the preset loss function includes: Based on the second facial feature parameter vector, the vector label, and the symmetry loss function, the training error of the preset deep learning network is obtained.

11. The face alignment method as described in claim 1, characterized in that, The first facial feature parameter vector includes one or more of the following: shape coefficient, expression coefficient, scale factor, rotation matrix, and translation parameter.

12. The face alignment method as described in claim 11, characterized in that, The step of obtaining the target face image aligned with the initial face image based on the first face feature parameter vector includes: Based on the first face feature parameter vector and the preset calculation rules, the target face vertex information after the initial face image is aligned is obtained; The target face image is obtained based on the target face vertex information.

13. The face alignment method as described in claim 12, characterized in that, The step of obtaining the target face vertex information after alignment of the initial face image based on the first face feature parameter vector includes: A 3D face model corresponding to the initial face image is constructed based on at least one of the preset 3D face construction rules, the shape coefficient, and the expression coefficient; Based on at least one of the scale factor, the rotation matrix, and the displacement parameter, the target face vertex information after alignment with the initial face image is obtained using the 3D face model.

14. A face alignment device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 13.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 13.

Images