Face attribute generation method based on cross-modal attention generative adversarial network

Abstract

In view of generating a face image from visual attribute text, the application designs an attribute face generation method based on a cross-modal attention generative adversarial network. The method relates to the application of generating a face image from an attribute text description, and particularly relates to the application of portraiture of a target person according to a language description. The method comprises four parts of data set preprocessing, network model design, network optimal model training and network model performance evaluation. The application fully gives play to the superiority of the generative adversarial network in the aspect of image generation, and designs an end-to-end solution for generating a face from visual attribute text, which has very wide application prospects in the fields of image repair, public security, archaeological research and film and television animation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, specifically to applications in deep learning-based attribute-to-face generation, text-to-image generation, and face generation, and particularly to an application of generating face images based on visual attributes using generative adversarial networks. Background Technology

[0002] In recent years, artificial intelligence technology has made continuous breakthroughs in various industries. Among them, image generation in computer vision is a very popular research topic, and it has already achieved substantial applications in many fields, such as beautification functions in communication terminals, film and television entertainment, computer-aided design, video image face swapping, and image restoration in archaeology. More and more researchers are conducting research on face generation, mainly including two technical routes: face image-to-face image generation and face image generation based on text descriptions. The former belongs to image-to-image conversion within the same modality, while the latter belongs to cross-modal transformation from natural language to image. Attribute text-to-face generation is a more challenging cross-modal transformation task, requiring the resolution of the semantic gap between the two modalities. Currently, there is considerable research on text-to-image generation, such as AttnGAN, DM-GAN, KT-GAN, ControlGAN, and DFGAN, which have achieved good results mainly in the field of text description-to-image generation. These technologies will gradually be applied to various related fields. However, research on attribute text-to-face image generation is relatively limited, and there are no readily available methods for reference. As a subfield of text-to-image generation, attribute text-to-face generation technology has broad application prospects. Research on visual attribute text-to-face image generation has broad application prospects and profound significance.

[0003] Currently, there is very little research focused on face generation from visual attributes, and no mature solutions exist. This invention initiates research on face image generation from visual attributes and is the first to propose an end-to-end solution. Since there are no mature solutions available for comparison, this invention selects the best current method for text-to-image generation for comparison. Through quantitative and qualitative comparisons, the face images generated by the network proposed in this invention achieve the best current performance in both realism and matching degree with the input attribute text. Summary of the Invention

[0004] This invention studies the subdomain of attribute text to face image generation, specifically focusing on text-to-image generation, and provides an attribute-based face generation method based on a cross-modal attention generative adversarial network (CMAGAN). The main structure of this method is a CMAGAN, which innovatively incorporates two modules: Word Feature Transformation (WFT) and Cross-modal Attention Fusion (CMAF). WFT transforms the dimension of the word feature space to match the dimension of the image feature space, converting it from two-dimensional to three-dimensional (2D-to-3D). The transformed word feature vectors better match the network, effectively promoting the generation of higher-quality face images. The CMAF module primarily uses cross-attention encoding to uncover the intrinsic connection between natural language and images, building a semantic bridge between them and improving the matching degree between the generated face and the input attribute text. The principle and framework of the proposed cross-modal attention generative adversarial network are as follows: Figure 1 As shown.

[0005] The present invention achieves the above objectives through the following technical solutions:

[0006] 1. A method for generating faces based on attributes using cross-modal attention generative adversarial networks, characterized by the following steps:

[0007] Step 1: Dataset preprocessing. Extract the corresponding attribute text from the 40-dimensional label attributes of faces in the public face dataset and combine them into visual attribute text separated by commas. Reassemble the face images and the corresponding visual attribute text into a new dataset and randomly split it into training set and test set.

[0008] Step 2: Network Model Design. A cross-modal attention-based generative adversarial network (CMAGAN) model is proposed. In this model, two new modules, Word Feature Transformation (WFT) and Cross-modal Attention Fusion (CMAF), are designed to guide the network to generate more realistic face images based on attribute text.

[0009] Step 3: Network model training. Input the preprocessed training dataset from Step 1 into the CMAGAN network from Step 2 for training. During the training process, adjust and optimize the parameters based on the model's performance.

[0010] Step 4: Network model performance testing. Input the test dataset from Step 1 into the network model trained in Step 3 to generate faces, and evaluate the generated faces quantitatively and visually. The quantitative evaluation mainly includes three indicators: Face Similarity Score (FSS), Face Feature Distance (FFD), and R-precision.

[0011] The following explains the dataset preprocessing steps in step one:

[0012] Currently, there is no readily available dataset for generating faces from visual attribute text. To conduct research on this task, a suitable dataset needs to be constructed for training and testing the network model. This invention selects the publicly available face dataset LFW to train and test the proposed CMAGAN network model. In the LFW dataset, each face image corresponds to a 40-dimensional binary attribute label vector, and 40 corresponding face visualization attributes are defined. Therefore, during dataset preprocessing, the corresponding visualization attribute text needs to be extracted based on the label vector. These visualization attribute texts are then combined using commas to form attribute text and combined with the face image to create a new data pair, thus constructing a new dataset for attribute text-to-face synthesis. The new dataset constructed based on LFW contains 13,143 faces and corresponding visualization attribute texts. It is important to note that in text-to-image generation datasets, each image generally corresponds to multiple text descriptions with complete grammar, while in the visualization attribute text-to-face generation dataset, each image corresponds to a single attribute text composed of multiple attributes without complete grammar. An example of a dataset for generating faces from visual attribute text is provided. Figure 2 As shown.

[0013] The two modules of the Cross-Modal Attention based Generative Adversarial Network (CMAGAN) in step two, namely Word Feature Transformation (WFT) and Cross-Modal Attention Fusion (CMAF), are described as follows:

[0014] CMAGAN pioneered the generation of faces from visualized attributes, providing an end-to-end solution. Quantitative and qualitative evaluations of the network's performance demonstrate its state-of-the-art capabilities. Its main innovations are the Word Feature Vector Transformation (WFT) module and the Cross-Modal Attention Fusion (CMAF) module. The principles and functions of these two innovative modules are described below.

[0015] The WFT module primarily transforms word feature vectors to better guide the network in generating high-quality face images. Its framework principle is as follows: Figure 3 As shown in Equation 1, the algorithm primarily has two functions. First, it transforms variable-length word feature vectors into fixed-length word feature vectors through transpose multiplication to better match network operations. The second function involves linearly transforming the feature vectors through a fully connected layer, and then using a reshape layer to convert the two-dimensional word feature vectors into three-dimensional feature vectors (2D to 3D) that match the image feature space. This transformation is expressed in Equation 2.

[0016] (1)

[0017] (2)

[0018] in For word feature vectors, This represents the feature vector after the size transformation. , Indicates constant coefficients, "" represents the transformation of tensor size, completing the transformation from 2D to 3D.

[0019] The main function of the CMAF module is to cross-encode and align text and image feature vectors to uncover their inherent relationships. This allows for the construction of a semantic bridge between the two modalities, enabling the network model to better understand the input attribute descriptions and generate face images that more closely match the input attribute text. Its principles and framework are as follows: Figure 4 As shown, its calculation expression is shown in formula (3-10).

[0020] (3)

[0021] (4)

[0022] (5)

[0023] (6)

[0024] (7)

[0025] (8)

[0026] (9)

[0027] (10)

[0028] in, , , , , , , , These are the weights of the eigenvector matrix; , , , , , and These are the constant coefficients of the linear transformation.

[0029] The parameters for training the network model in step three are explained below:

[0030] The CMAGAN network requires adjustment of hyperparameters based on the changing trend and characteristics of the loss function during multiple training sessions. It also requires multiple verifications and adjustments through experiments to finally obtain the optimal trained model.

[0031] The following is an explanation of the performance testing of the network model in step four:

[0032] The proposed CMAGAN model generates face images quantitatively and qualitatively. Based on the characteristics of this task, Face Similarity Score (FSS), Face Feature Distance (FFD), and R-precision are introduced to quantitatively evaluate the generated face images. A visual comparison of the generated faces with those generated by other publicly available methods is also provided. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the cross-modal attention generative adversarial network principle proposed in this invention.

[0034] Figure 2 This is a dataset example of the attribute text-based face generation method of the present invention.

[0035] Figure 3 This is a diagram illustrating the principle and architecture of the Word Feature Vector Transformation (WFT) module proposed in this invention.

[0036] Figure 4 This is the principle and architecture diagram of the cross-modal attention fusion (CMAF) module proposed in this invention.

[0037] Figure 5 This is a comparison of the visualization results of the method proposed in this invention with the current best method. Detailed Implementation

[0038] The attribute-based face generation based on cross-modal attention generative adversarial networks mainly consists of four steps: dataset preprocessing, generative network model construction, optimal network model training, and network model performance verification and evaluation.

[0039] In data preprocessing, this invention uses the publicly available face dataset LFW to extract and combine 13,143 faces and their corresponding attribute texts into a new dataset. The process and examples are provided below. Figure 2 As shown, each face in the preprocessed dataset corresponds to a visual attribute text. These attribute texts are characterized by being simply groups of text separated by commas, not forming a complete grammatical sentence. This is the most fundamental difference from text-to-image generation datasets. Finally, the 13,143 processed data pairs were randomly divided into a training set of 10,143 data pairs (face images and their corresponding attribute texts) and a test set of 3,000 pairs.

[0040] The prepared training dataset of 10,143 faces and their corresponding attribute texts was input into the designed CMAGAN network model for training. During training, the parameters, such as the hyperparameters in Equation 11, were continuously optimized and adjusted based on the performance of the loss function. Extensive experiments demonstrated that when the hyperparameters... =1.0, =5.0; the learning rate of the discriminator LR =0.00002 and the generator's learning rate LR The model performs relatively well when the value is 0.00001.

[0041] (11)

[0042] in, For the total loss function, The loss function for the first-level generator. The loss function for the second-level generator. The loss function for the third-level generator. The loss function computed for the Deep Attentional Multimodal Similarity Model. Let KL divergence loss function be used. and This is a hyperparameter.

[0043] Based on the convergence of the loss function, the CMAGAN network model reaches its optimal performance after 1100 epochs of training. At this point, 3000 data pairs from the test dataset are input into the network for face generation testing. The generated faces are then quantitatively and visually compared with the current state-of-the-art method. The intuitive qualitative comparison results are as follows: Figure 5 As shown, the face generated by the method of this invention is of higher quality than other methods, mainly reflected in the realism of the face image and its consistency with the input attribute description. Quantitative and objective comparison results are shown in Table 1. Compared with the four current best algorithms—DM-GAN, AttnGAN, ControlGAN, and DFGAN—CMAGAN proposed in this invention achieves the best performance in all four evaluation metrics: Fréchet Inception Distance (FID), FFD, FSS, and R-precision.

[0044] Table 1 Comparison of evaluation metrics for generated face images on the LFW dataset

[0045]

Claims

1. A method for generating faces based on attributes using cross-modal attention generative adversarial networks, characterized in that... Includes the following steps: Step 1: Dataset preprocessing. Extract the corresponding attribute text from the 40-dimensional binary label attributes of faces in the public face dataset and combine them into visual attribute text separated by commas. Reassemble the face images and the corresponding visual attribute text into a new dataset and randomly split it into training set and test set. Step 2: Network Model Design. A Cross-Modal Attention-based Generative Adversarial Network (CMAGAN) model is proposed. This model incorporates two new modules: Word Feature Transformation (WFT) and Cross-Modal Attention Fusion (CMAF), to guide the network in generating more realistic face images based on attribute text. The WFT module primarily transforms word feature vectors to better guide the network in generating high-quality face images. It has two main functions: first, it converts variable-length word feature vectors into fixed-length word feature vectors through transpose multiplication to better match network operations (calculated as shown in Equation 1); second, after linear transformation of the feature vectors through a fully connected layer, it transforms the two-dimensional word feature vectors into three-dimensional feature vectors (2D to 3D) that match the image feature space through a reshape layer (expressed as Equation 2). (1) (2) in For word feature vectors, This represents the feature vector after the size transformation. , Indicates constant coefficients, "" represents tensor size transformation, completing the 2D to 3D conversion; the main function of the CMAF module is to perform cross-encoding alignment of text feature vectors and image feature vectors to explore their intrinsic relationship, so as to build a semantic bridge between the two modalities of text features and image features, so that the network model can better understand the input attribute description and generate a face image that matches the input attribute text better. Its calculation expression is shown in formula (3-10): (3) (4) (5) (6) (7) (8) (9) (10) in, , , , , , , , These are the weights of the eigenvector matrix; , , , , , and These are the constant coefficients of the linear transformation; Step 3: Network model training. Input the preprocessed training dataset from Step 1 into the CMAGAN network from Step 2 for training. During the training process, adjust and optimize the parameters based on the model's performance. Step 4: Network model performance testing. Input the test dataset from Step 1 into the network model trained in Step 3 to generate faces, and evaluate the generated faces quantitatively and visually. The quantitative evaluation mainly includes three indicators: Face Similarity Score (FSS), Face Feature Distance (FFD), and R-precision.

Images