Facial aesthetics evaluation and enhancement method based on structural reasoning

Abstract

The present invention discloses a method for facial aesthetic evaluation and enhancement based on structural reasoning. The steps of the present invention are as follows: Step (1) constructing a facial aesthetic evaluation network, including a convolutional neural network structure and a key point detection network, fusing the facial comprehensive features and facial structural features output by the two networks, respectively inputting them into two multi-layer perceptrons with different functions for prediction, and respectively outputting aesthetic evaluation results and facial attribute information. Step (2) constructing a facial aesthetic enhancement network, including extracting facial content features using a content encoder and a deformable convolution; including facial structural features II obtained using a key point detection network and a graph attention network; fusing facial content features and facial structural features II, inputting them into a decoder, and predicting an enhanced facial image. The present invention can significantly improve the effect of picture beautification on the basis of improving the model's extraction of facial structural information.

Description

Technical Field

[0001] The present invention provides a method for evaluating and enhancing facial aesthetics based on structural reasoning. Background Art

[0002] Facial aesthetic quality, also known as facial beauty or attractiveness, has a wide impact on human social activities. Facial aesthetic quality evaluation and enhancement aims to evaluate or improve the aesthetics of the face, thereby obtaining a better social experience. Related algorithms have been widely used in various fields.

[0003] Existing facial aesthetic quality evaluation mainly focuses on three aspects: binary classification, score prediction, and distribution prediction. There is still a lack of relevant data for related research on aesthetic factors and aesthetic description. In early studies, researchers designed various manual features based on heuristic rules (such as average symmetry, facial geometry, and golden section). However, these rules and hand-crafted features lack universality. There is still a lack of reliable learning mechanisms to explore the mechanism of facial aesthetic evaluation, which makes the existing methods have low interpretability and difficult to use to guide the aesthetic quality enhancement process of facial images. Therefore, how to construct a self-learning mechanism for facial aesthetic factors is a fundamental problem that needs to be solved in the field of facial aesthetic evaluation.

[0004] Facial aesthetic enhancement, also known as face beautification, aims to improve the attractiveness of the face by adjusting the skin color, face structure or automatic makeup. Existing face beautification methods are generally data-driven, requiring examples of beautified faces as references, and then using mathematical models to predict the fusion parameters between the instance information and the input face information. Overall, existing work only considers a single aspect of face beautification, and there is still a lack of effective modeling methods for the beautification of facial structure, and the impact of face beautification on face recognition is not considered. Therefore, how to achieve modeling and migration of facial structure, and based on a completely model-driven approach, study multi-factor face automatic beautification methods that preserve identity to improve the diversity and reliability of face beautification are the key and difficulty of the theoretical development and application promotion of related technologies. Summary of the invention

[0005] The purpose of the present invention is to provide a method for facial aesthetic evaluation and enhancement based on structural reasoning in view of the deficiencies of the prior art. The present invention includes two parts: evaluation and enhancement. In order to improve the model's ability to represent non-grid structures of the face, it is proposed to use deformable convolution to extract comprehensive visual features from face images; combine face key points and graph models to model the face structure, and realize migration and reasoning in the face beautification process based on the graph model; in order to improve the rationality of structural deformation, deformation consistency constraints are constructed for deformable convolution and graph model reasoning processes; finally, identity discrimination networks and other distribution discrimination networks are used to realize multiple rationality constraints of the beautification process to improve the beautification effect.

[0006] The technical solution adopted by the present invention comprises the following steps:

[0007] Step (1) Constructing a facial aesthetic evaluation network:

[0008] The facial aesthetic evaluation network consists of two branches. Branch one adopts a convolutional neural network structure to extract aesthetic-related comprehensive facial features from the input face image; branch two adopts a key point detection network to extract preliminary structural features of the face, and uses a graph attention network with a DiffPool pooling module to transform the preliminary structural features of the face to obtain aesthetic-related facial structural features I; the comprehensive facial features and facial structural features output by the two branches are fused and input into two multi-layer perceptrons with different functions for prediction, and the aesthetic evaluation results and facial attribute information are output respectively.

[0009] Step (2) Constructing a face aesthetic enhancement network:

[0010] The face aesthetic enhancement network consists of two branches. Branch one uses content encoder and deformable convolution to extract face content features from the input face image; branch two uses key point detection network to extract preliminary face structural features, and uses graph attention network to transform the preliminary face structural features to obtain aesthetic-related face structural features II; the face content features and face structural features II are fused and input into the decoder to predict the enhanced face image.

[0011] Furthermore, the facial aesthetics evaluation network described in step (1) is:

[0012] 1-1 In order to improve the model's ability to represent facial structures, it is proposed to introduce a deformable convolution module and global pooling in the first branch. Compared with traditional standard convolution, deformable convolution no longer uses a strict grid-like local neighborhood to calculate the activation response, but automatically learns an offset vector Δp ​​for each position p in the convolution window, and uses the offset position (p+p n +Δp) elements are calculated. n Represents the grid neighborhood offset in the standard convolution (assuming the convolution window is 3, it is {(-1,-1),...,(1,1)}). The total offset corresponding to the entire feature map is called the offset field. The prediction of the offset field is implemented using one or more layers of convolution.

[0013] 1-2 In the second branch, we plan to use a graph attention network with a DiffPool pooling module, labeled GAT / Pool. In each layer of the graph attention network, a multi-way self-attention mechanism is used. For each current node, the representation of the current node is updated using the features of its adjacent nodes. The pooling module plans to use the DiffPool mechanism, which is not based on global maximum / minimum / average pooling methods, but automatically learns the weights of each node for pooling to improve the performance of the model and accelerate convolution operations.

[0014] 1-3 In the prediction stage, in order to take into account the correlation between facial aesthetics and attribute information such as makeup and expression, and to improve the representation ability of the model, a multi-task-multi-label learning model is proposed to simultaneously predict the facial aesthetics evaluation results and facial attribute information. Among them, for the facial aesthetics evaluation results, it is proposed to simultaneously predict the binary classification label, average score and score distribution of facial aesthetics. The three use cross entropy loss, L2 loss and EMD loss respectively. For facial attribute information, multi-label cross entropy loss is proposed, that is, each label may have different types, and the cross entropy is calculated separately, and then integrated as the objective function.

[0015] Furthermore, the face aesthetic enhancement network described in step (2) is:

[0016] 2-1 Face structure migration based on graph generation adversarial network:

[0017] It is proposed to use a graph-generated adversarial network for the representation and transfer of facial structure. The input face image is input into the key point detection network to extract the preliminary structural features of the face. The preliminary structural features of the face include the corresponding key point map A and feature matrix X; the key point map A and feature matrix X are mapped to the key point map corresponding to the beautified face image through the graph attention network. and the feature matrix Key point diagram and the feature matrix Constitute the structural features of the human face II.

[0018] Similar to the Generative Adversarial Network, GGAN contains a generator and an encoder. The generator adopts a graph autoencoder structure. The generator is proposed to adopt a graph attention network structure, in which the output of the multi-layer generation attention module in the graph attention network structure is the feature matrix corresponding to the beautified face. Afterwards, based on Predict the key points of the face after beautification Its mathematical form is:

[0019]

[0020] In practical applications, and are all unknown and may be diverse (the beautification result is not unique). Therefore, it is impossible to use the error between the predicted result and the corresponding real result to calculate the loss. It is assumed that the key point map and feature matrix of all beautiful faces conform to a certain distribution and have certain differences from non-beautiful faces. In order to map the input face to a more beautiful face, a generative adversarial mechanism is proposed to make The distribution of key point maps corresponding to beautiful faces. The corresponding discriminant network is a structural discriminant network, which is based on the adversarial learning mechanism. The distance between the distribution and the corresponding distribution of beautiful faces is the smallest.

[0021] 2-2 Face content migration based on deformable convolution:

[0022] In the process of facial content migration, the structure of the face has certain changes. It is difficult to accurately describe the mapping relationship between input image information and output image information based on traditional convolution. In order to describe this geometric deformation, a residual module based on deformable convolution is proposed in the middle area of ​​the network. The offset field prediction module in the deformable convolution can automatically learn the information source area of ​​each pixel of the beautified image (corresponding to the non-grid receptive field), thereby improving its prediction accuracy. In order to improve the rationality of the displacement field, it is proposed to input the face image and its coordinate matrix at the input end at the same time; and in branch two, for the key point position, the key point offset before and after passing through the graph attention network GAT is calculated, and the deformation consistency constraint is constructed between the displacement field. In addition, in order to improve the performance of the enhanced network, a direct channel is added to the deformable convolution based on the residual network idea to achieve accurate migration of facial content information.

[0023] 2-3 Discriminant network based on multi-distribution constraints:

[0024] In order to improve the performance of the enhanced network, a multi-distribution constrained discriminant network is proposed, which consists of four sub-networks:

[0025] 1) Identity discrimination subnetwork: determines whether the synthesized face image retains the identity information of the input face image. It uses the pre-trained SphereFace to extract features and calculates the cosine distance between the feature maps of the input face image and the synthesized face image as the identity discrimination loss.

[0026] 2) Unconditional discriminant sub-network, using CNN structure, to judge whether the synthesized face image is attractive (Y / N?) and whether the synthesized face image is real or fake (R / F?). L2 distance is used as unconditional adversarial loss;

[0027] 3) Structure discrimination sub-network: It uses a graph attention network structure with DiffPool pooling (GAT / Pool) to predict whether the synthesized face image is attractive and real, and plans to use L2 loss;

[0028] 4) The conditional adversarial sub-network predicts whether the synthesized face image matches the input face image (R / F?), and whether the input face image is attractive (Y / N?). It is planned to use the same structure and L2 loss as the face aesthetic evaluation network. During the training process, the loss function of the above discriminant network is weighted to optimize the discriminant network and the enhancement network.

[0029] The beneficial effects of the present invention are as follows:

[0030] Existing facial aesthetics work is either based on key points and manual features to describe structural information, or uses convolutional neural networks without considering structural information. The former is limited by the expressive power of features, and the latter is limited by the grid receptive field of convolution operations, and both cannot accurately and efficiently extract facial structural information. The present invention uses a convolutional neural network to extract the features of each key point, and then constructs a graph network model to describe the facial structure, which can effectively overcome the limitations of both. In addition, based on the update and iteration of the graph network model, the changes in the key point attribute information (position, feature expression, etc.) during the face beautification process can be inferred to achieve a progressive migration of the facial structure. The adversarial learning mechanism can cause the migration results to approach the distribution of beautiful faces. Overall, the present invention can significantly improve the effect of picture beautification on the basis of improving the model's extraction of facial structural information. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of facial aesthetic evaluation and enhancement based on graphical models; DETAILED DESCRIPTION

[0032] The present invention will be further described below in conjunction with the accompanying drawings.

[0033] like Figure 1 (a) shows the face aesthetic evaluation network. Figure 1 (b) shows the face aesthetic enhancement network. Figure 1 (c) in the figure shows the discriminant network. Figure 1 (d) in the figure shows a deformable convolution module. Figure 1 (e) in the figure shows a graph attention network with a DiffPool pooling module, which includes the following steps:

[0034] Step (1) Constructing a facial aesthetics evaluation network

[0035] The facial aesthetic evaluation network consists of two branches. Branch one adopts a convolutional neural network structure to extract aesthetic-related comprehensive facial features from the input face image; branch two adopts a key point detection network to extract preliminary structural features of the face, and uses a graph attention network with a DiffPool pooling module to transform the preliminary structural features of the face to obtain aesthetic-related facial structural features I; the comprehensive facial features and facial structural features output by the two branches are fused and input into two multi-layer perceptrons with different functions for prediction, and the aesthetic evaluation results and facial attribute information are output respectively.

[0036] Step (2) Constructing a facial aesthetic enhancement network

[0037] The face aesthetic enhancement network consists of two branches. Branch one uses content encoder and deformable convolution to extract face content features from the input face image; branch two uses key point detection network to extract preliminary face structural features, and uses graph attention network to transform the preliminary face structural features to obtain aesthetic-related face structural features II; the face content features and face structural features II are fused and input into the decoder to predict the enhanced face image.

[0038] Furthermore, the facial aesthetics evaluation network described in step (1) is:

[0039] 1-1 In order to improve the model's ability to represent facial structures, it is proposed to introduce a deformable convolution module and global pooling in the first branch. Compared with traditional standard convolution, deformable convolution no longer uses a strict grid-like local neighborhood to calculate the activation response, but automatically learns an offset vector Δp ​​for each position p in the convolution window, and uses the offset position (p+p n +Δp) elements are calculated. n Represents the grid neighborhood offset in the standard convolution (assuming the convolution window is 3, it is {(-1,-1),...,(1,1)}). The total offset corresponding to the entire feature map is called the offset field. The prediction of the offset field is implemented using one or more layers of convolution. The calculation process of the deformable convolution is as follows Figure 1 As shown in (d) in .

[0040] 1-2 In the second branch, we plan to use a graph attention network with a DiffPool pooling module, labeled GAT / Pool. Figure 1 As shown in (d) in the figure, the network model structure of branch 2 is as follows Figure 1As shown in (e) in the figure. In each layer of the graph attention network, a multi-way self-attention mechanism is used. For each current node, the features of the nodes adjacent to it are used to update the representation of the current node. The pooling module is proposed to adopt the DiffPool mechanism, that is, it is not based on the global maximum / minimum / average pooling method, but automatically learns the weights of each node for pooling to improve the performance of the model and accelerate the convolution operation.

[0041] 1-3 In the prediction stage, in order to take into account the correlation between facial aesthetics and attribute information such as makeup and expression, and to improve the representation ability of the model, a multi-task-multi-label learning model is proposed to simultaneously predict the facial aesthetics evaluation results and facial attribute information. Among them, for the facial aesthetics evaluation results, it is proposed to simultaneously predict the binary classification label, average score and score distribution of facial aesthetics. The three use cross entropy loss, L2 loss and EMD loss respectively. For facial attribute information, multi-label cross entropy loss is proposed, that is, each label may have different types, and the cross entropy is calculated separately, and then integrated as the objective function.

[0042] Furthermore, the face aesthetic enhancement network described in step (2) is:

[0043] 2-1 Face structure migration based on graph generation adversarial network:

[0044] It is proposed to use a graph-generated adversarial network for the representation and transfer of facial structure. The input face image is input into the key point detection network to extract the preliminary structural features of the face. The preliminary structural features of the face include the corresponding key point map A and feature matrix X; the key point map A and feature matrix X are mapped to the key point map corresponding to the beautified face image through the graph attention network. and the feature matrix Key point diagram and the feature matrix Constitute the structural features of the human face II.

[0045] Similar to the Generative Adversarial Network, GGAN contains a generator and an encoder. The generator adopts a graph autoencoder structure. The generator is proposed to adopt a graph attention network structure, which is as mentioned above. Figure 1 As shown in (b) in the figure. The output of the multi-layer generation attention module in the attention network structure is the feature matrix corresponding to the beautified face Afterwards, based on Predict the key points of the face after beautification Its mathematical form is:

[0046]

[0047] In practical applications, and are all unknown and may be diverse (the beautification result is not unique). Therefore, it is impossible to use the error between the predicted result and the corresponding real result to calculate the loss. It is assumed that the key point map and feature matrix of all beautiful faces conform to a certain distribution and have certain differences from non-beautiful faces. In order to map the input face to a more beautiful face, a generative adversarial mechanism is proposed to make The distribution of key point maps corresponding to beautiful faces. The corresponding discriminant network is a structural discriminant network, which is based on the adversarial learning mechanism. The distance between the distribution and the corresponding distribution of beautiful faces is the smallest.

[0048] 2-2 Face content migration based on deformable convolution:

[0049] In the process of facial content migration, the structure of the face has certain changes. It is difficult to accurately describe the mapping relationship between input image information and output image information based on traditional convolution. In order to describe this geometric deformation, a residual module based on deformable convolution is proposed in the middle area of ​​the network. The offset field prediction module in the deformable convolution can automatically learn the information source area of ​​each pixel of the beautified image (corresponding to the non-grid receptive field), thereby improving its prediction accuracy. In order to improve the rationality of the displacement field, it is proposed to input the face image and its coordinate matrix at the input end at the same time; and in branch two, for the key point position, the key point offset before and after passing through the graph attention network GAT is calculated, and the deformation consistency constraint is constructed between the displacement field. In addition, in order to improve the performance of the enhanced network, a direct channel is added to the deformable convolution based on the residual network idea to achieve accurate migration of facial content information.

[0050] 2-3 Discriminant network based on multi-distribution constraints:

[0051] In order to improve the performance of the enhanced network, a multi-distribution constrained discriminant network is proposed, which consists of four sub-networks:

[0052] 1) Identity discrimination subnetwork: determines whether the synthesized face image retains the identity information of the input face image. It uses the pre-trained SphereFace to extract features and calculates the cosine distance between the feature maps of the input face image and the synthesized face image as the identity discrimination loss.

[0053] 2) Unconditional discriminant sub-network, using CNN structure, to judge whether the synthesized face image is attractive (Y / N?) and whether the synthesized face image is real or fake (R / F?). L2 distance is used as unconditional adversarial loss;

[0054] 3) Structure discrimination sub-network: It uses a graph attention network structure with DiffPool pooling (GAT / Pool) to predict whether the synthesized face image is attractive and real, and plans to use L2 loss;

[0055] 4) The conditional adversarial sub-network predicts whether the synthesized face image matches the input face image (R / F?), and whether the input face image is attractive (Y / N?). It is planned to use the same structure and L2 loss as the face aesthetic evaluation network. During the training process, the loss function of the above discriminant network is weighted to optimize the discriminant network and the enhancement network.

Claims

1. A facial aesthetic evaluation and enhancement method based on structural reasoning, characterized by The steps include: Step (1) Constructing a facial aesthetic evaluation network: The facial aesthetic evaluation network consists of two branches. Branch 1 uses a convolutional neural network structure to extract aesthetic-related comprehensive facial features from the input face image. Branch 2 uses a key point detection network to extract preliminary structural features of the face, and uses a graph attention network with a DiffPool pooling module to transform the preliminary structural features of the face to obtain aesthetic-related facial structural features I; the comprehensive facial features and facial structural features output by the two branches are fused and input into two multi-layer perceptrons with different functions for prediction, and the aesthetic evaluation results and facial attribute information are output respectively; Step (2) Constructing a face aesthetic enhancement network: The face aesthetic enhancement network consists of two branches. Branch 1 uses content encoder and deformable convolution to extract facial content features from input face images. Branch 2 uses a key point detection network to extract the preliminary structural features of the face, and uses a graph attention network to transform the preliminary structural features of the face to obtain aesthetically relevant facial structural features II; The facial content features and facial structure features II are fused and input into the decoder to predict the enhanced facial image; The facial aesthetic evaluation network described in step (1) is specifically as follows: 1-1 In order to improve the model's ability to represent facial structures, a deformable convolution module and global pooling are introduced in the first branch; the deformable convolution automatically learns an offset vector Δp ​​for each position p in the convolution window, and uses the offset position (p+p n +Δp) elements are calculated; p n Represents the grid neighborhood offset in the standard convolution; the total offset corresponding to the entire feature map is called the offset field; the prediction of the offset field is realized using one or more layers of convolution; 1-2 In the second branch, a graph attention network with a DiffPool pooling module is used, labeled as GAT / Pool; in each layer of the graph attention network, a multi-way self-attention mechanism is adopted. For each current node, the representation of the current node is updated using the features of its adjacent nodes; the pooling module is proposed to adopt the DiffPool mechanism; In the 1-3 prediction stage, a multi-task-multi-label learning model is used to simultaneously predict the facial aesthetic evaluation results and facial attribute information; for the facial aesthetic evaluation results: the binary classification label, average score and score distribution of facial aesthetics are predicted simultaneously; the three are respectively predicted using cross entropy loss, L2 loss and EMD loss; for facial attribute information: multi-label cross entropy loss is used, that is, each label may have different types, and the cross entropy is calculated separately, and then integrated as the objective function; The face aesthetic enhancement network described in step (2): 2-1 Face structure migration based on graph generation adversarial network: The graph generation adversarial network is used to complete the representation and transfer of facial structure; the input face image is input into the key point detection network to extract the preliminary structural features of the face, which include the corresponding key point map A and feature matrix X; the key point map A and feature matrix X are mapped to the key point map corresponding to the beautified face image through the graph attention network and the feature matrix Key point diagram and the feature matrix Constitute the structural features of the face II; Similar to the generative adversarial network, GGAN contains a generator and an encoder. The generator is proposed to adopt a graph attention network structure, in which the output of the multi-layer generative attention module in the graph attention network structure is the feature matrix corresponding to the beautified face. Afterwards, based on Predict the key points of the face after beautification Its mathematical form is: In practical application and are unknown, so the error between the predicted result and the corresponding real result cannot be used to calculate the loss; it is assumed that the key point map and feature matrix of all beautiful faces conform to a certain distribution and have certain differences from non-beautiful faces; in order to map the input face to a more beautiful face, a generative adversarial mechanism is used so that The distribution of key point maps corresponding to beautiful faces; the corresponding discriminant network is a structural discriminant network, based on the adversarial learning mechanism, so that The distance between the distribution and the corresponding distribution of the beautiful face is the smallest; 2-2 Face content migration based on deformable convolution: A residual module based on deformable convolution is used in the middle area of ​​the network; the offset field prediction module in the deformable convolution can automatically learn the information source area of ​​each pixel of the beautified image; in order to improve the rationality of the displacement field, the face image and its coordinate matrix are input at the same time at the input end; and in branch 2, for the key point position, the key point offset before and after passing through the graph attention network GAT is calculated, and a deformation consistency constraint is constructed between the displacement field; In addition, in order to improve the performance of the enhanced network, a direct channel is added to the deformable convolution based on the residual network idea to achieve accurate transfer of facial content information; 2-3 Discriminant network based on multi-distribution constraints: In order to improve the performance of the enhanced network, a multi-distribution constrained discriminant network is adopted, which consists of four sub-networks: 1) Identity discrimination subnetwork: determines whether the synthesized face image retains the identity information of the input face image. It uses the pre-trained SphereFace to extract features and calculates the cosine distance between the feature maps of the input face image and the synthesized face image as the identity discrimination loss. 2) Unconditional discriminant sub-network, using CNN structure, to judge whether the synthesized face image is attractive and whether the synthesized face image is real or fake; L2 distance is used as unconditional adversarial loss; 3) Structure discrimination sub-network: It uses a graph attention network structure with DiffPool pooling to predict whether the synthesized face image is attractive and real, and plans to use L2 loss; 4) The conditional adversarial sub-network predicts whether the synthesized face image matches the input face image, and at the same time predicts whether the input face image is attractive. It is planned to adopt the same structure and L2 loss as the face aesthetic evaluation network. During the training process, the loss function of the above discriminant network is weighted to optimize the discriminant network and the enhancement network.

Images