A global face reconstruction method based on multi-scale feature integration

Abstract

With the continuous development of deep learning, communication technology and mobile terminal, people's demand for self-media, film and television is also increasing. Therefore, it is necessary to propose a global face reconstruction technology based on multi-scale feature integration. The present application is a kind of global face reconstruction technology based on multi-scale feature integration for face image. First, the global information vector representing the posture, expression, identity information and background of the person is extracted according to the global instance vector extraction network. Secondly, according to the comprehensive feature information and source domain characteristics fitted by the fitting network, the global face reconstruction is realized by means of the multi-scale feature integration network proposed in the present application. The present application can be widely applied to the face reconstruction demand in multimedia in different environments.

Description

Technical Field

[0001] This invention relates to a global face reconstruction technology based on multi-scale feature integration, belonging to the fields of computer vision, deep learning, face manipulation, and film and television special effects. Background Technology

[0002] With the development of deep learning, image detection, image evaluation, image generation, and other computer vision fields have made great strides. As one of the most outstanding representatives, multimedia editing and generation have received increasing attention. Due to the sensitivity of the human eye to facial structure and head movements, editing facial images has become an extremely challenging problem. In recent years, with the advent of generative adversarial networks, image generation and editing have undergone qualitative changes, but certain problems still exist. Face reconstruction refers to using information such as facial expressions and head poses from the source domain to drive the actions of the target person. Most current face reconstruction technologies are based on two aspects: (1) graph-based face reconstruction: this part of the technology requires separating the face from the background and then modeling the face; (2) keypoint-based face reconstruction: this type of technology uses facial keypoints to drive changes in expression and pose. The above two sets of technologies are technically complex and have obvious traces after segmentation, which greatly limits the application scenarios of face reconstruction. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention aims to provide a global face reconstruction technology based on multi-scale feature integration. To solve the aforementioned technical problem, this invention adopts the following technical solution:

[0004] A global face reconstruction technique based on multi-scale feature integration includes the following steps:

[0005] (1) This invention proposes a global instance vector extraction network, in which the main body of the network uses a residual network as the backbone network to extract global instance vectors. A global instance vector is a vector containing facial identity information, pose information, expression information, and other relevant details extracted from an unsegmented face image. The global instance vector extraction network is trained using a loss function similar to that of a face recognition network, and finally, this network is used as a pre-trained network to extract global instance information of the face.

[0006] (2) Adaptive Global Instance Vector Fitting Network. The global instance vectors extracted in step (1) contain identity, background, expression, and pose information. Providing this information directly to the target person can cause uncertainty, such as distortion of pose and expression. To improve the robustness of the entire network, this invention proposes an adaptive global instance vector fitting network. After fitting the network, the combined instance vectors contain the expression and pose information of the source domain person and the background and identity information of the target person.

[0007] (3) The source domain image is input into the encoder in the backbone network. The encoder consists of a simple set of multiple convolutional structures. Each convolutional structure consists of a normalization layer, a convolutional layer, a pooling layer, and an activation function. After passing through four sets of convolutional structures, the image is input into an intermediate layer, which consists of two residual structures to increase the network depth. During the deep image extraction process, the intermediate features encoded by the encoder contain a large amount of deep semantic information and shallow color and structural information. To make full use of this information, this invention proposes a multi-scale feature integration generator.

[0008] The multi-scale feature integration network utilizes features from the encoder and features from the generator, and with the help of an adaptive mask converter (detailed in step (4)), it achieves the mutual combination between source domain character features and target character features.

[0009] (4) Adaptive Mask Converter. Inspired by AdaIN, this invention proposes an adaptive mask converter for fusing information from the source and target domains. The structure mainly consists of three branches, one of which is used for AdaIN feature transformation. The specific steps are as follows:

[0010] (a) To facilitate the calculation of the mean and variance, the feature map is re-normalized.

[0011] (b) Integrate the features of the source domain characters with the adaptive weighted global instance vector (obtained from step (2)) according to AdaIN.

[0012] (c) Resize the feature maps to suit the network requirements.

[0013] Another branch is primarily used to generate an adaptive mask, which is used to weight the original feature map and the feature map after feature transformation. The third branch is used to input the original feature map.

[0014] The advantages and positive effects of this invention are as follows:

[0015] (1) This invention focuses on solving the effect of face reproduction, especially the overall coordination and absence of cutting marks before and after face reproduction.

[0016] (2) This invention achieves accurate reproduction of facial posture and expression information.

[0017] (3) The present invention has been tested and verified by a large number of experiments, which has effectively improved the effectiveness of the method. Attached Figure Description

[0018] Figure 1 This is an overall process diagram in a specific embodiment of the present invention;

[0019] Figure 2 This is a structural diagram of the adaptive mask converter in a specific embodiment of the present invention; Detailed Implementation

[0020] This example uses a global face reconstruction technique based on multi-scale feature integration as the research object, and describes the implementation method of the present invention in detail. To make the objectives and technical solutions of the present invention clearer, the specific implementation steps of the present invention are described in detail below with reference to the accompanying drawings.

[0021] See Figure 1 The diagram shows the overall structure of the invention. Details are as follows;

[0022] (1) This invention proposes a global instance vector extraction network, in which the main body of the network uses a residual network as the backbone network to extract global instance vectors. A global instance vector is a vector containing facial identity information, pose information, expression information, and other relevant details extracted from an unsegmented face image. The global instance vector extraction network is trained using a loss function similar to that of a face recognition network, and finally, this network is used as a pre-trained network to extract global instance information of the face.

[0023] (2) Adaptive Global Instance Vector Fitting Network. The global instance vectors extracted in step (1) contain identity, background, expression, and pose information. Providing this information directly to the target person can cause uncertainty, such as distortion of pose and expression. To improve the robustness of the entire network, this invention proposes an adaptive global instance vector fitting network. After fitting the network, the combined instance vectors contain the expression and pose information of the source domain person and the background and identity information of the target person.

[0024] The adaptive global instance vector fitting network consists of a mapping network and a weighting network. Identity information, background information, facial expression information, and pose information are coupled together in the global instance vector. To address this issue, inspired by StyleGAN, this invention proposes a mapping network composed of three fully connected networks, which separates the identity, pose, facial expression, and background information in the global instance vector. The weighting network consists of two fully connected layers and one sigmoid layer, aiming to balance the obtained mask W between 0 and 1. The global instance vector of the character in the source domain is defined as E. s The global instance vector of the target is defined as E. t Then the fitted global instance vector E s，t for:

[0025] E s，t =W×E s +(1-W)×E t

[0026] (3) The source domain image is input into the encoder in the backbone network. The encoder consists of a simple set of multiple convolutional structures. Each convolutional structure consists of a normalization layer, a convolutional layer, a pooling layer, and an activation function. After passing through four sets of convolutional structures, the image is input into an intermediate layer, which consists of two residual structures to increase the network depth. During the deep image extraction process, the intermediate features encoded by the encoder contain a large amount of deep semantic information and shallow color and structural information. To make full use of this information, this invention proposes a multi-scale feature integration generator.

[0027] The multi-scale feature fusion network utilizes features from the encoder and generator, and with the help of an adaptive mask converter (detailed in step (4)), achieves the combination of source domain character features and target character features. The role of the multi-scale feature fusion generator is to combine E... s，t This is integrated with a multi-scale feature map (Fz). Since the source domain provides information on facial expressions and postures, spatial representation is needed more than identity and background information. Therefore, this invention utilizes a multi-scale method to extract global and local information from the global facial image to obtain more information. The input image size is 256×256. After four sets of convolutional structures, the resulting feature map is defined as Fz. i (I s，t ), i = 1, 2, 3, 4. Furthermore, the representation of background and identity information on the global face image is similar to that of the texture image. Therefore, to further improve the quality of the generated image, this invention proposes an adaptive mask converter to integrate the feature map of the source domain person and the adaptive weighted global instance vector at multiple scale levels, the process of which is as follows:

[0028]

[0029]

[0030]

[0031] (4) Adaptive Mask Converter. Inspired by AdaIN, this invention proposes an adaptive mask converter for fusing information from the source and target domains. The structure is mainly divided into three branches, one of which is used for AdaIN feature transformation. This part of the structure mainly consists of AdaIN residual blocks, each of which is designed with six residual layers. Each residual layer consists of a 1×1 convolutional layer and an AdaIN layer. The specific calculation steps of the AdaIN layer are as follows:

[0032] (a) To facilitate the calculation of the mean and variance, the feature map is re-normalized.

[0033]

[0034] (b) Integrate the features of the source domain characters with the adaptive weighted global instance vector (obtained from step (2)) according to AdaIN.

[0035]

[0036] Here, mean is the operation for calculating the mean, std is the operation for calculating the variance, and eps is a minimum value, the purpose of which is to avoid the denominator being 0.

[0037] (c) Resize the feature maps to suit the network requirements.

[0038] Another branch is primarily used to generate an adaptive mask, which is used to weight the original feature map and the feature map after feature transformation. The third branch is used to input the original feature map.

Claims

1. A global face hallucination method based on multi-scale feature integration, characterized in that Face reconstruction is achieved based on global instance vectors, multi-scale feature integration, and adaptive mask transformation based on AdaIN, including the following steps: (1) This invention proposes a global instance vector extraction network. The main part of the network uses a residual network as the backbone network to extract global instance vectors. The global instance vector extraction network is trained by a loss function similar to that of the face recognition network. The source domain image is encoded by a global semantic instance encoding network. The global instance vector containing identity information, expression information and pose information is directly extracted from the unsegmented face image as the global instance information of the face. (2) The present invention proposes an adaptive global instance vector fitting network to eliminate the uncertainty of information, posture distortion and expression distortion caused by directly providing the information extracted in step (1) to the target person, and improve the robustness of the entire network. The present invention proposes an adaptive global instance vector fitting network, which contains identity, background, expression and posture information in the global instance vector extracted in step (1). After the fitting network, the integrated instance vector has the expression and posture information of the source domain person and the background and identity information of the target person. (3) This invention proposes a multi-scale feature integration generator, which makes full use of the large amount of deep semantic information and shallow color and structural information contained in the intermediate features of the encoder during the deep image extraction process. The source domain image is input into the encoder in the backbone network. The encoder consists of multiple sets of convolutional structures. Each set of convolutional structures consists of a normalization layer, a convolutional layer, a pooling layer and an activation function. After passing through four sets of convolutional structures, it is input into the intermediate layer composed of two residual structures to increase the depth of the network. The multi-scale feature integration network uses the features in the encoder and the features in the generator to integrate the intermediate features obtained by the encoder with the features of the generator to obtain joint features. With the help of the adaptive mask converter (step (4)), the combination between the source domain character features and the target character features was realized; (4) This invention proposes an adaptive mask conversion module. The joint features obtained in step (3) are input into the adaptive mask conversion module. One branch is used for AdaIN-inspired style transfer processing, one branch is used to learn the adaptive fusion weight mask, and one branch is used to receive the original feature map and integrate the fused features with the original features through a weighting strategy. The final output is a face image with source domain identity features and target domain expression and pose features. The specific steps of the AdaIN feature conversion branch are as follows: (a) The feature map is re-normalized to facilitate the calculation of the mean and variance; (b) Integrate the features of the source domain characters with the adaptive weighted global instance vector (obtained from step (2)) according to AdaIN; (c) Resize the feature maps to fit the network requirements.

Images