Face fusion method and device based on neural network, equipment and storage medium

Abstract

The application relates to a face fusion method and device based on a neural network, equipment and a storage medium, wherein the method comprises the following steps: acquiring a source face image and a target face image; inputting the source face image and the target face image into a target adversarial neural network, and performing feature coding on the target face image to obtain coded features; performing feature extraction on the source face image through a target identity feature extractor to obtain identity features; injecting the identity features into the coded features through an identity injector to generate target fusion features; performing feature enhancement on the coded features to generate a first target feature, a second target feature and a third target feature; and generating a target face fusion image by performing feature decoding on the first target feature, the second target feature, the third target feature and the target fusion features. The application improves the accuracy of face fusion and is beneficial to improving the generation quality of face fusion.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a face fusion method, apparatus, device and storage medium based on neural networks. Background Technology

[0002] Face fusion technology transfers the identity features of a source face to a target face, making the generated face resemble the source face while preserving other attributes of the target face (such as expression, pose, and lighting). This technology can generate faces that do not currently exist and has wide applications in film, games, and the entertainment industry. Face fusion has also attracted considerable research interest in computer graphics and vision, but significant challenges remain in generating high-fidelity, high-quality faces.

[0003] Existing neural network-based face fusion techniques are 3D-based methods that convert a human image into a 3D model. They then express various facial features (such as pose, expression, texture, lighting, etc.) as parameters of the 3D model, adjusting these parameters to achieve neural network-based face fusion. However, this method requires a precise 3DMM model, but due to a lack of training data for 2D image-to-3D model conversion, obtaining a high-precision 3D model is often difficult. Furthermore, this method requires explicit semantic decoupling (distinguishing between identity features and attribute features). These features exist in two dimensions: one in the coefficients of the 3D model and the other in a high-dimensional feature space. However, semantic features are highly intertwined. For example, facial expressions are semantically attribute features, but some iconic expressions can also express identity, making it difficult to separate facial expressions from identity features. Moreover, due to the ambiguity of semantic feature concepts, accurate decoupling of features is impossible. Forced decoupling leads to contradictions between features, ultimately resulting in flaws in the generated image and functional degradation. Ultimately, the accuracy of face fusion is poor. Summary of the Invention

[0004] The purpose of this application is to provide a face fusion method, apparatus, device, and storage medium based on neural networks, so as to improve the accuracy of face fusion based on neural networks.

[0005] To address the aforementioned technical problems, embodiments of this application provide a face fusion method based on a neural network, comprising:

[0006] Obtain the source face image and the target face image;

[0007] The source face image and the target face image are input into the target adversarial neural network, and the target face image is feature-encoded to obtain the encoded features;

[0008] The source face image is used to extract features using a target identity feature extractor to obtain identity features;

[0009] The identity features are injected into the encoded features through the identity injector to generate the target fusion features;

[0010] The encoded features are enhanced to generate a first target feature, a second target feature, and a third target feature;

[0011] A target face fusion image is generated by performing feature decoding on the first target feature, the second target feature, the third target feature, and the target fusion feature.

[0012] To address the aforementioned technical problems, embodiments of this application provide a face fusion device based on a neural network, comprising:

[0013] The face image acquisition unit is used to acquire source face images and target face images;

[0014] The encoding feature generation unit is used to input the source face image and the target face image into the target adversarial neural network, and to encode the target face image to obtain encoded features;

[0015] The identity feature generation unit is used to extract features from the source face image through the target identity feature extractor to obtain identity features;

[0016] The target fusion feature generation unit is used to inject the identity features into the encoded features through an identity injector to generate target fusion features;

[0017] A feature enhancement processing unit is used to enhance the encoded features to generate a first target feature, a second target feature, and a third target feature;

[0018] The target face fusion image generation unit is used to generate a target face fusion image by performing feature decoding on the first target feature, the second target feature, the third target feature and the target fusion feature.

[0019] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer device, including one or more processors; and a memory for storing one or more programs, so that the one or more processors implement the neural network-based face fusion method described in any one of the above-mentioned methods.

[0020] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the neural network-based face fusion method described above.

[0021] This invention provides a face fusion method, apparatus, device, and storage medium based on a neural network. The method includes: acquiring a source face image and a target face image; inputting the source face image and the target face image into a target adversarial neural network, and encoding features in the target face image to obtain encoded features; extracting features from the source face image using a target identity feature extractor to obtain identity features; injecting the identity features into the encoded features using an identity injector to generate target fusion features; enhancing the encoded features to generate a first target feature, a second target feature, and a third target feature; and decoding the first target feature, the second target feature, the third target feature, and the target fusion features to generate a target face fusion image. This invention decouples the identity features and attribute features of a face image by extracting identity features and injecting them into the encoded features, avoiding confusion between identity features and attribute features, thus improving the accuracy of face fusion and the quality of face fusion generation. Attached Figure Description

[0022] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an implementation of the face fusion method based on neural networks provided in this application embodiment;

[0024] Figure 2 This is a flowchart illustrating an implementation of a sub-process in the neural network-based face fusion method provided in this application embodiment;

[0025] Figure 3 This is another implementation flowchart of a sub-process in the face fusion method based on neural networks provided in the embodiments of this application;

[0026] Figure 4 This is another implementation flowchart of a sub-process in the face fusion method based on neural networks provided in the embodiments of this application;

[0027] Figure 5 This is another implementation flowchart of a sub-process in the face fusion method based on neural networks provided in the embodiments of this application;

[0028] Figure 6 This is another implementation flowchart of a sub-process in the face fusion method based on neural networks provided in the embodiments of this application;

[0029] Figure 7 This is another implementation flowchart of a sub-process in the face fusion method based on neural networks provided in the embodiments of this application;

[0030] Figure 8 This is a schematic diagram of a face fusion device based on a neural network provided in an embodiment of this application;

[0031] Figure 9 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0035] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the face fusion method based on neural networks provided in this application is generally executed by a server, and correspondingly, the face fusion device based on neural networks is generally configured in the server.

[0037] Please see Figure 1 , Figure 1 This paper illustrates a specific implementation of a face fusion method based on neural networks.

[0038] It should be noted that if substantially the same result is obtained, the method of this invention is not based on... Figure 1 Limited to the order of the processes shown, this method includes the following steps:

[0039] S1: Obtain the source face image and the target face image.

[0040] Specifically, the purpose of face fusion in this application embodiment is to implant the identity features of the source portrait image into the target face image, so that the final output image has the identity features of the source face image and the attribute features of the target face image.

[0041] It should be noted that the source and target face images have undergone face alignment. Face alignment refers to aligning the facial landmarks of the two images, such as the left and right eye landmarks, nose landmarks, etc.

[0042] Furthermore, a specific implementation method for face alignment between source face images and target face images is provided: acquiring initial source face images and initial target face images; performing face recognition on the initial source face images and initial target face images respectively to obtain initial source face key points and initial target face key points; based on the initial source face key points and initial target face key points, performing face alignment processing on the initial source face images and initial target face images in a way that the face key points are aligned with each other to obtain face-aligned source face images and target face images.

[0043] Specifically, the initial source face image and the initial target face image refer to the source face image and the target face image that have not undergone face alignment, respectively. In this embodiment, face recognition is performed on the initial source face image and the initial target face image to obtain initial source face key points and initial target face key points. The initial source face key points and the initial target face key points each include at least the left and right eye key points, nose key points, left and right mouth key points, and chin key points. Then, the initial source face key points and the initial target face key points are aligned one by one, that is, identical key points are aligned, thereby performing face alignment processing on the initial source face image and the initial target face image to obtain the face-aligned source face image and target face image.

[0044] S2: Input the source face image and the target face image into the target adversarial neural network, and encode the features of the target face image to obtain the encoded features.

[0045] Specifically, the source and target face images are input into a target adversarial neural network (ANN), and the target face image is feature-encoded by the NAN's encoder to obtain encoded features. The encoder consists of one convolutional layer and three connected downsampling convolutional layers (DnConv). The first convolutional layer in the encoder has 64 channels and a convolution radius of 3. The downsampling convolutional layers include an average pooling layer (vgPool) and a convolutional layer (convolution). AvgPool is an average pooling layer with a stride of 2 used for 1 / 2 downsampling, and the convolutional layer has a convolution radius of 3. The three downsampling convolutional layers (DnConv) have 64, 128, and 256 convolutional channels, respectively.

[0046] Please see Figure 2 , Figure 2 A specific implementation method prior to step S2 is shown below in detail:

[0047] S2A: Acquire multiple training face images.

[0048] S2B: Align the training face image with the face to obtain the aligned training face image.

[0049] S2C: Randomly select aligned training face images as training source face images and training target face images.

[0050] S2D: Randomly degrade the training source face image and the training target face image to obtain degraded training source face image and degraded training target face image.

[0051] S2E: Using stochastic gradient descent, degraded training source face images and degraded training target face images are input into the adversarial neural network for training, resulting in the target adversarial neural network.

[0052] Specifically, before implementing step S2, this embodiment first trains the adversarial neural network to obtain the trained target adversarial neural network. This embodiment uses stochastic gradient descent to train the adversarial neural network; the adversarial neural network consists of two parts: a generator and a discriminator. The generator and discriminator compete and iterate with each other until they reach an equilibrium state and converge, causing the generator to output an image that satisfies the loss function. The specific training process of this embodiment is as follows: multiple training face images are acquired and aligned to obtain aligned training face images. Face images are randomly selected from the aligned training face images as training source face images and training target face images, respectively. Then, the training source face images and training target face images are randomly downgraded to obtain downgraded training source face images and downgraded training target face images. The training source face images and training target face images are input into the adversarial neural network for inference calculation. The loss is calculated using the output of the adversarial neural network and the loss function, and then the gradient is calculated and backpropagated to update the weights of the generator and discriminator. The above process is repeated continuously with selected samples until convergence, thus obtaining the target adversarial neural network.

[0053] Specifically, multiple loss values ​​are generated during the training of adversarial neural networks. Identity loss: ID Loss =1.0-cosine_similarity(S id O id In this embodiment, it is necessary to determine whether two facial images represent the same person; therefore, this embodiment utilizes the ID-Extrator identity feature extractor to extract identity features. id It is the extracted source portrait ID feature vector, O id It is the extracted identity feature vector generated from the graph, denoted by S. id and O id The cosine distance is used as the identity loss to constrain the consistency between the generated image and the identity features of the source image.

[0054] Discriminator Loss GAN Loss :GAN Loss =max(0,1-D(O)); The generated image O is input into the discriminator, and the discriminator loss is calculated according to the above formula. This indicates the realism of the generated image and is used to constrain the generator to produce realistic human figures. Here, D(O) refers to the probability of generating image O in the discriminator. Attribute Loss Attr loss Formula used: Perform calculations, where IDExtr i() indicates that the output features of the i-th convolutional network layer of the ID-Extrator are used as attribute features. The ID-Extrator is trained on face images, and the features it extracts are more representative of the face's uniqueness. The attribute features of a face refer to features unrelated to identity. The attribute features are determined through the following experimental method: several portrait images with the same pose and expression but different IDs are taken, and the convolutional layer in the ID-Extrator that makes the features of each portrait the closest is found; in this embodiment, the output features of the 3rd to 5th convolutional layers are selected to represent the attribute features of the portrait; the L2 distance between the attribute features of the source portrait and the target portrait is taken as the attribute loss; this loss is used to constrain the attribute consistency between the generated image and the target image.

[0055] The formula for calculating the total generator loss is Tota. loss =w0*GAN Loss +w1*ID Loss +w2*Attr loss Where w0, w1, and w2 are empirical parameters, and Total loss For the total loss of the generator, GAN Loss For discriminator loss, ID Loss For identity loss, Attr loss This is attribute loss.

[0056] S3: Extract features from the source face image using the target identity feature extractor to obtain identity features.

[0057] Specifically, the ID-Extrator target identity feature extractor is used to extract identity features with a higher degree of decoupling. Existing face recognition models like Arcface can extract highly recognizable facial features, but these features are highly coupled with identity and attribute features, requiring further decoupling to obtain purer identity features. The ID-Extrator target identity feature extractor consists of an Arcface module and a Linear module. The source face image is processed by the Arcface module to extract features, and then by the Linear module to extract the decoupled identity features. The Linear module is a 512-channel fully connected layer.

[0058] Please see Figure 3 , Figure 3 A specific implementation method prior to step S3 is shown below in detail:

[0059] S3A: Acquire sample face images, where several sample face images exist under the same body identifier.

[0060] S3B: Extract Arcface features from sample face images under the same body identifier to obtain target Arcface features corresponding to multiple identity identifiers.

[0061] S3C: Cluster the target Arcface features under the same identity identifier to obtain multiple clustering results, and calculate the cluster centers in the clustering results to obtain the feature center corresponding to multiple identity identifiers.

[0062] S3D: Based on feature class heart training, an identity feature extractor is obtained to obtain the target identity feature extractor.

[0063] Specifically, multiple sample face images are acquired, including several sample face images under the same body identifier; that is, face images of multiple identity identifiers are acquired. Then, Arcface (additive angular margin) features are extracted from the sample face images under the same body identifier to obtain target Arcface features corresponding to multiple identity identifiers. Next, clustering processing is performed on the target Arcface features under the same identity identifier to obtain multiple clustering results. The cluster centers in the clustering results are calculated to obtain feature centroids corresponding to multiple identity identifiers. Feature centroids are the features corresponding to the cluster centers after Arcface feature clustering. Finally, an identity feature extractor is trained based on the feature centroids to obtain the target identity feature extractor.

[0064] Furthermore, the ID-Extrator feature extractor is trained to output features that are closer to the feature center of the identity to which the sample belongs. This results in extracted features that are more representative of a particular identity, eliminating interference from other elements. Its loss function is expressed as follows:

[0065] Loss id_extractor =cosine(arcface(x) i ),Cid i ); where x i It is a sample in the i-th class of IDs, Cid i Let be the feature center of the i-th class of samples. Using the above loss function and face samples, train ID-Extrator to obtain the identity feature extractor.

[0066] S4: Using an identity injector, identity features are injected into encoded features to generate target fusion features.

[0067] Specifically, identity features are injected into the encoded features through the ID-Embed identity injector to generate the target fusion features. The ID-Embed identity injector consists of three contiguous identity injection blocks, namely, a first identity injection block, a second identity injection block, and a third identity injection block. Each ID-Block consists of two convolutional layers (Conv) and two adaptive normalization layers (AdaIN), which are respectively composed of one convolutional layer (Conv), one normalization layer (AdaIN), one convolutional layer (Conv), and one normalization layer (AdaIN). The two convolutional layers (Conv) each have 256 channels and a convolution radius of 3.

[0068] Please see Figure 4 , Figure 4 A specific implementation of step S4 is shown below:

[0069] S41: The identity features are injected into the coding features through the first identity injection block to obtain the first fused features.

[0070] Please see Figure 5 , Figure 5 A specific implementation of step S41 is shown below:

[0071] S411: In the first identity injection block, the encoded features are convolved to obtain the first convolutional features.

[0072] S412: The first convolutional feature is fused with the identity feature to obtain the initial fused feature.

[0073] S413: Perform convolution processing on the initial fused features to obtain the second convolutional features.

[0074] S414: The second convolutional feature is fused with the identity feature to obtain the first fused feature.

[0075] Specifically, the first identity injection block is composed of a convolutional layer Conv, a normalization layer AdaIN, another convolutional layer Conv, and a normalization layer AdaIN in sequence. Therefore, in the first identity injection block, the encoded features are sequentially convolved to obtain the first convolutional feature. Then, the first convolutional feature is fused with the identity feature to obtain the initial fused feature. Next, the initial fused feature is convolved to obtain the second convolutional feature. Finally, the second convolutional feature is fused with the identity feature to obtain the first fused feature.

[0076] Furthermore, feature fusion is performed in the AdaIN layer using the following formula:

[0077] Where Feat is the encoded feature, Sid is the convolutional feature; μ(Feat) and δ(Feat) are the mean and variance of Feat calculated by channel; δ Sid and μ Sid These are two identity feature coefficients, obtained by transformation using two fully connected layers; the parameters of the fully connected layers are obtained through training. The number of channels in the fully connected layers is 256.

[0078] It should be noted that the same implementation process from steps S411 to S414 is used in both the second identity injection block and the third identity injection block. To avoid repetition, it will not be described again here.

[0079] S42: The identity features are injected into the first fusion feature through the second identity injection block to obtain the second fusion feature.

[0080] S43: By injecting the identity features into the second fusion feature through the third identity injection block, the target fusion feature is obtained.

[0081] Specifically, identity features are injected into the encoded features sequentially through the first identity injection block, the second identity injection block, and the third identity injection block, so that the identity features of the source face image can be integrated into the encoded features of the target image.

[0082] S5: Perform feature enhancement on the encoded features to generate the first target feature, the second target feature, and the third target feature.

[0083] Please see Figure 6 , Figure 6 A specific implementation of step S5 is shown below:

[0084] S51: The encoded features are upsampled and convolved through the first upsampling convolutional layer to generate the first target feature.

[0085] S52: The encoded features are upsampled and convolved through a second upsampling convolutional layer to generate the second target features.

[0086] S53: The encoded features are upsampled and convolved through a third upsampling convolutional layer to generate the third target feature.

[0087] Specifically, in this embodiment, since the target face image may be a low-quality portrait, or the encoded features may be interfered with during the identity feature injection process, image quality enhancement of the encoded features is required to ensure high-quality output portrait. The enhancer consists of three connected upsampling convolutional layers UpConv, outputting three levels of enhanced features E1, E2, and E3, i.e., outputting the first target feature, the second target feature, and the third target feature. The upsampling convolutional layer includes UpScale and convolution layers. UpScale in the upsampling convolutional layer is a 2x upsampling; the three convolutional layers have a radius of 3 and channel numbers of 256, 128, and 64, respectively.

[0088] S6: Generate a target face fusion image by performing feature decoding on the first target feature, the second target feature, the third target feature, and the target fusion feature.

[0089] Please see Figure 7 , Figure 7 A specific implementation of step S6 is shown below:

[0090] S61: Input the first target feature, the second target feature, the third target feature, and the target fusion feature into the decoder of the target adversarial neural network.

[0091] The decoder consists of three interconnected upsampling convolutional layers and one image transformation layer.

[0092] S62: The first target feature, the second target feature, the third target feature, and the target fusion feature are sequentially upsampled and convolved through three interconnected upsampling convolutional layers to obtain the decoded features.

[0093] S63: The convolutional features are processed by the image transformation layer to convert the decoded features into a three-channel image, thus obtaining the target face fusion image.

[0094] Specifically, the first target feature, the second target feature, the third target feature, and the target fusion feature are input into the decoder of the target adversarial neural network, outputting a fused face image O (target face fusion image). The decoder consists of three interconnected upsampling convolutional layers (UpConv) and one ToRGB (image conversion layer). Each input feature is merged with the enhanced target feature after passing through the UpConv layer (using the concat method) before being input into the next UpConv layer. Finally, it is converted to an RGB three-channel image output via ToRGB. The UpConv layer consists of 2x upsampling and one convolutional layer. The convolutional radius in each of the three UpConv layers is 3, with output channels of 256, 128, and 64 respectively. The ToRGB layer contains one convolutional layer with a convolutional radius of 3 and 3 output channels.

[0095] In this embodiment, a source face image and a target face image are acquired; the source face image and the target face image are input into a target adversarial neural network, and the target face image is feature-encoded to obtain encoded features; the source face image is feature-extracted using a target identity feature extractor to obtain identity features; the identity features are injected into the encoded features using an identity injector to generate target fusion features; the encoded features are enhanced to generate a first target feature, a second target feature, and a third target feature; and the first target feature, the second target feature, the third target feature, and the target fusion features are decoded to generate a target face fusion image. This embodiment of the invention extracts identity features and injects them into the encoded features, achieving decoupling between identity features and attribute features of the face image, avoiding confusion between identity features and attribute features, and improving the accuracy of face fusion, thereby improving the generation quality of face fusion.

[0096] Please refer to Figure 8 As a response to the above Figure 1 To implement the method shown, this application provides an embodiment of a face fusion device based on a neural network, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0097] like Figure 8 As shown, the neural network-based face fusion device of this embodiment includes: a face image acquisition unit 71, an encoded feature generation unit 72, an identity feature generation unit 73, a target fusion feature generation unit 74, a feature enhancement processing unit 75, and a target face fusion image generation unit 76, wherein:

[0098] The face image acquisition unit 71 is used to acquire source face images and target face images;

[0099] The encoding feature generation unit 72 is used to input the source face image and the target face image into the target adversarial neural network, and to encode the target face image to obtain the encoded features;

[0100] The identity feature generation unit 73 is used to extract features from the source face image through the target identity feature extractor to obtain identity features;

[0101] The target fusion feature generation unit 74 is used to inject identity features into the encoded features through the identity injector to generate target fusion features;

[0102] The feature enhancement processing unit 75 is used to enhance the encoded features and generate a first target feature, a second target feature, and a third target feature.

[0103] The target face fusion image generation unit 76 is used to generate a target face fusion image by performing feature decoding on the first target feature, the second target feature, the third target feature and the target fusion feature.

[0104] Furthermore, the identity feature generation unit 73 also includes the following before:

[0105] The sample face image acquisition unit is used to acquire sample face images, wherein there are several sample face images under the same body identifier;

[0106] The feature extraction unit is used to extract the Arcface features of sample face images under the same body identifier, and obtain the target Arcface features corresponding to multiple identity identifiers.

[0107] The feature class center calculation unit is used to perform clustering processing on the target Arcface features under the same identity identifier, obtain multiple clustering results, and calculate the cluster center in the clustering results to obtain the feature class center corresponding to multiple identity identifiers;

[0108] The identity feature extractor training unit is used to train the identity feature extractor based on feature class centers to obtain the target identity feature extractor.

[0109] Furthermore, the target fusion feature generation unit 74 includes:

[0110] The first fusion feature generation unit is used to inject identity features into the encoded features through the first identity injection block to obtain the first fusion feature;

[0111] The second fusion feature generation unit is used to inject identity features into the first fusion feature through the second identity injection block to obtain the second fusion feature;

[0112] The second fusion feature injection unit is used to inject identity features into the second fusion feature through the third identity injection block to obtain the target fusion feature.

[0113] Furthermore, the first fusion feature generation unit includes:

[0114] The first convolutional feature generation unit is used to perform convolution processing on the encoded features in the first identity injection block to obtain the first convolutional feature;

[0115] The initial fusion feature fusion unit is used to fuse the first convolutional features with the identity features to obtain the initial fusion features;

[0116] The second convolutional feature generation unit is used to perform convolution processing on the initial fused features to obtain the second convolutional features;

[0117] The second convolutional feature fusion unit is used to fuse the second convolutional features with the identity features to obtain the first fused feature.

[0118] Furthermore, the feature enhancement processing unit 75 includes:

[0119] The first target feature generation unit is used to perform upsampling and convolution processing on the encoded feature through the first upsampling convolutional layer to generate the first target feature;

[0120] The second target feature generation unit is used to perform upsampling and convolution processing on the encoded feature through the second upsampling convolutional layer to generate the second target feature.

[0121] The third target feature generation unit is used to perform upsampling and convolution processing on the encoded features through the third upsampling convolution layer to generate the third target feature.

[0122] Furthermore, the target face fusion image generation unit 76 includes:

[0123] The target feature input unit is used to input the first target feature, the second target feature, the third target feature and the target fusion feature into the decoder of the target adversarial neural network. The decoder includes three interconnected upsampling convolutional layers and one image transformation layer.

[0124] The decoding feature generation unit is used to perform upsampling and convolution processing on the first target feature, the second target feature, the third target feature, and the target fusion feature in sequence through three interconnected upsampling convolutional layers to obtain the decoding features;

[0125] The image conversion unit is used to perform convolution processing on the convolutional features through the image conversion layer, so that the decoded features are converted into a three-channel image to obtain the target face fusion image.

[0126] Furthermore, the encoding feature generation unit 72 also includes the following before:

[0127] A face image acquisition unit is used to acquire multiple training face images;

[0128] The face alignment unit is used to align the training face image to obtain an aligned training face image.

[0129] The face image random acquisition unit is used to randomly select aligned training face images as training source face images and training target face images;

[0130] The degradation processing unit is used to randomly degrade the training source face image and the training target face image to obtain the degraded training source face image and the degraded training target face image.

[0131] The adversarial neural network training unit is used to train the adversarial neural network by inputting degraded training source face images and degraded training target face images into the adversarial neural network using the stochastic gradient descent method, thereby obtaining the target adversarial neural network.

[0132] In this embodiment, a source face image and a target face image are acquired; the source face image and the target face image are input into a target adversarial neural network, and the target face image is feature-encoded to obtain encoded features; the source face image is feature-extracted using a target identity feature extractor to obtain identity features; the identity features are injected into the encoded features using an identity injector to generate target fusion features; the encoded features are enhanced to generate a first target feature, a second target feature, and a third target feature; and the first target feature, the second target feature, the third target feature, and the target fusion features are decoded to generate a target face fusion image. This embodiment of the invention extracts identity features and injects them into the encoded features, achieving decoupling between identity features and attribute features of the face image, avoiding confusion between identity features and attribute features, and improving the accuracy of face fusion, thereby improving the generation quality of face fusion.

[0133] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.

[0134] Computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected via a system bus. It should be noted that only a computer device 8 with these three components (memory 81, processor 82, and network interface 83) is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0135] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0136] The memory 81 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 81 may be an internal storage unit of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 81 may also be an external storage device of the computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 8. Of course, the memory 81 may include both internal storage units and external storage devices of the computer device 8. In this embodiment, the memory 81 is typically used to store the operating system and various application software installed on the computer device 8, such as program code for a face fusion method based on neural networks. In addition, the memory 81 may also be used to temporarily store various types of data that have been output or will be output.

[0137] In some embodiments, processor 82 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 82 is typically used to control the overall operation of the computer device 8. In this embodiment, processor 82 is used to run program code stored in memory 81 or process data, for example, to run the program code of the aforementioned neural network-based face fusion method to implement various embodiments of the neural network-based face fusion method.

[0138] The network interface 83 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 8 and other electronic devices.

[0139] This application also provides another embodiment, namely, providing a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the neural network-based face fusion method described above.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0141] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A face fusion method based on neural networks, characterized in that, include: Obtain the source face image and the target face image; The source face image and the target face image are input into the target adversarial neural network, and the target face image is feature-encoded to obtain the encoded features; The source face image is used to extract features using a target identity feature extractor to obtain identity features; The identity features are injected into the encoded features through the identity injector to generate the target fusion features; The encoded features are enhanced to generate a first target feature, a second target feature, and a third target feature; A target face fusion image is generated by performing feature decoding on the first target feature, the second target feature, the third target feature, and the target fusion feature; The step of enhancing the encoded features to generate a first target feature, a second target feature, and a third target feature includes: The encoded feature is upsampled and convolved by a first upsampling convolutional layer to generate the first target feature. The encoded feature is upsampled and convolved by a second upsampling convolutional layer to generate the second target feature. The encoded feature is upsampled and convolved by a third upsampling convolutional layer to generate the third target feature. The step of generating a target face fusion image by decoding the first target feature, the second target feature, the third target feature, and the target fusion feature includes: The first target feature, the second target feature, the third target feature, and the target fusion feature are input into the decoder of the target adversarial neural network, wherein the decoder includes three interconnected upsampling convolutional layers and one image transformation layer; The first target feature, the second target feature, the third target feature, and the target fusion feature are sequentially upsampled and convolved through the three interconnected upsampling convolutional layers to obtain the decoded features; The convolutional features are processed by an image transformation layer to convert the decoded features into a three-channel image, thereby obtaining the target face fusion image.

2. The face fusion method based on neural networks according to claim 1, characterized in that, Before extracting features from the source face image using a target identity feature extractor to obtain identity features, the method further includes: Acquire sample face images, wherein several sample face images exist under the same body identifier; Arcface features are extracted from the sample face images under the same body identifier to obtain target Arcface features corresponding to multiple identity identifiers; Clustering is performed on the target Arcface features under the same identity identifier to obtain multiple clustering results, and the cluster centers in the clustering results are calculated to obtain the feature class centers corresponding to multiple identity identifiers; The target identity feature extractor is obtained by training an identity feature extractor based on feature class centers.

3. The face fusion method based on neural networks according to claim 1, characterized in that, The identity injector includes a first identity injection block, a second identity injection block, and a third identity injection block; The step of injecting the identity features into the encoded features through an identity injector to generate target fusion features includes: The identity features are injected into the encoded features through the first identity injection block to obtain the first fusion feature; The identity feature is injected into the first fusion feature through the second identity injection block to obtain the second fusion feature; The identity feature is injected into the second fusion feature through the third identity injection block to obtain the target fusion feature.

4. The face fusion method based on neural networks according to claim 3, characterized in that, The step of injecting the identity features into the encoded features through the first identity injection block to obtain the first fused feature includes: In the first identity injection block, the encoded features are convolved to obtain the first convolutional features; The first convolutional feature is fused with the identity feature to obtain the initial fused feature; The initial fused features are convolved to obtain the second convolutional features; The second convolutional feature is fused with the identity feature to obtain the first fused feature.

5. The face fusion method based on neural networks according to any one of claims 1 to 4, characterized in that, Before inputting the source face image and the target face image into the target adversarial neural network and performing feature encoding on the target face image to obtain the encoded features, the method further includes: Acquire multiple training face images; The training face images are aligned to obtain aligned training face images; The aligned training face images are randomly selected as the training source face images and the training target face images; The training source face image and the training target face image are randomly downgraded to obtain downgraded training source face image and downgraded training target face image; The degraded training source face image and the degraded training target face image are input into the adversarial neural network for training using the stochastic gradient descent method to obtain the target adversarial neural network.

6. A face fusion device based on a neural network, characterized in that, include: The face image acquisition unit is used to acquire source face images and target face images; The encoding feature generation unit is used to input the source face image and the target face image into the target adversarial neural network, and to encode the target face image to obtain encoded features; The identity feature generation unit is used to extract features from the source face image through the target identity feature extractor to obtain identity features; The target fusion feature generation unit is used to inject the identity features into the encoded features through an identity injector to generate target fusion features; A feature enhancement processing unit is used to enhance the encoded features to generate a first target feature, a second target feature, and a third target feature; The target face fusion image generation unit is used to generate a target face fusion image by performing feature decoding on the first target feature, the second target feature, the third target feature and the target fusion feature; The feature enhancement processing unit includes: The first target feature generation unit is used to perform upsampling and convolution processing on the encoded feature through a first upsampling convolutional layer to generate the first target feature. The second target feature generation unit is used to perform upsampling and convolution processing on the encoded feature through the second upsampling convolutional layer to generate the second target feature. The third target feature generation unit is used to perform upsampling and convolution processing on the encoded feature through the third upsampling convolutional layer to generate the third target feature. The target face fusion image generation unit includes: A target feature input unit is used to input the first target feature, the second target feature, the third target feature and the target fusion feature into the decoder of the target adversarial neural network, wherein the decoder includes three interconnected upsampling convolutional layers and one image transformation layer; The decoding feature generation unit is used to perform upsampling and convolution processing on the first target feature, the second target feature, the third target feature and the target fusion feature in sequence through the three interconnected upsampling convolutional layers to obtain the decoding features; The image conversion unit is used to perform convolution processing on the convolutional features through the image conversion layer, so that the decoded features are converted into a three-channel image to obtain the target face fusion image.

7. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the neural network-based face fusion method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the neural network-based face fusion method as described in any one of claims 1 to 5.

Images