Facial expression generation method based on generative adversarial network and related equipment

Abstract

The invention provides a generative adversarial network-based facial expression generation method and related equipment, and relates to the technical field of facial expression generation, and the method comprises the steps: generating a difficult sample based on a facial image and a target expression code; determining a first feature distance between the difficult sample and the simple sample; based on a generative adversarial network, shortening the first feature distance to a second feature distance to generate training data; training the generative adversarial network by using the training data to obtain a trained generative adversarial network; and determining a facial expression generation result based on the trained generative adversarial network. Facial expression codes are added into the input of the generator, so that the generator can generate a series of continuously changing expressions, an attention mechanism is provided, so that the model has robustness, and the adaptive mixing module reduces the feature space by reducing the distance between a difficult sample and an easy sample in the feature space, so that the robustness of the model is improved. And the picture generation quality in expression generation research is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of facial expression generation, and more specifically, to a facial expression generation method based on a generative adversarial network, a facial expression generation device based on a generative adversarial network, an electronic device, and a storage medium. Background Art

[0002] Generative adversarial networks (GANs) have made significant improvements in the field of facial expression generation, however, current methods still have some limitations. For example, artifacts and blurring are easily generated around expression-intensive areas, expression alignment is insufficient, and undesirable overlapping artifacts may be generated when dealing with some large expression transitions, which seriously affects the expression generation results.

[0003] Therefore, a new technical solution is urgently needed to solve the above technical problems. Summary of the invention

[0004] A series of simplified concepts are introduced in the Summary of the Invention, which will be further described in detail in the Detailed Description of the Invention. The Summary of the Invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0005] In a first aspect, the present invention proposes a facial expression generation method based on a generative adversarial network, comprising:

[0006] Generate difficult samples based on facial images and target expression encoding;

[0007] Determine the first characteristic distance between the difficult sample and the easy sample;

[0008] Based on a generative adversarial network, the first feature distance is shortened to a second feature distance to generate training data;

[0009] Using the training data, training the generative adversarial network to obtain a trained generative adversarial network;

[0010] Determine the facial expression generation results based on the trained generative adversarial network.

[0011] In some embodiments, the generative adversarial network includes a generator, which is used to output a color mask and an attention mask based on the facial image and the target expression encoding to output a generated image.

[0012] In some embodiments, the generative adversarial network includes a discriminator, which is used to discriminate the authenticity of the generated image.

[0013] In some embodiments, the discriminator is further configured to decode activation values ​​of action units in the generated image.

[0014] In some embodiments, the discriminator includes an adaptive mixing module for embedding and classifying the result of generating image mixing.

[0015] In some embodiments, the method further comprises:

[0016] Compute image adversarial loss, attention loss, conditional expression loss, identity loss, and adaptive hybrid loss.

[0017] In some embodiments, the method further comprises:

[0018] Calculate the global loss based on image adversarial loss, attention loss, conditional expression loss, identity loss, and adaptive hybrid loss;

[0019] Generate facial expression generation results based on the global loss.

[0020] In the second aspect, a facial expression generation device based on a generative adversarial network is also proposed, comprising:

[0021] A first generation unit, for generating a difficult sample based on the facial image and the target expression code;

[0022] A first determining unit, used to determine a first characteristic distance between a difficult sample and an easy sample;

[0023] A second generating unit, configured to shorten the first feature distance to a second feature distance based on a generative adversarial network to generate training data;

[0024] A training unit, used to train the generative adversarial network using the training data to obtain a trained generative adversarial network;

[0025] The second determining unit is used to determine the facial expression generation result based on the trained generative adversarial network.

[0026] In a third aspect, an electronic device is also proposed, including a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the facial expression generation method based on the generative adversarial network as described above when the processor is running.

[0027] In a fourth aspect, a storage medium is also proposed, on which program instructions are stored, and the program instructions are used to execute the facial expression generation method based on the generative adversarial network as described above when running.

[0028] According to the above technical scheme, based on the facial image and the target expression encoding, a difficult sample is generated; the first feature distance between the difficult sample and the simple sample is determined; based on the generative adversarial network, the first feature distance is shortened to the second feature distance to generate training data; the generative adversarial network is trained using the training data to obtain a trained generative adversarial network; based on the trained generative adversarial network, the facial expression generation result is determined. Therefore, the facial expression encoding is added to the input of the generator, so that the generator can generate a series of continuously changing expressions, and an attention mechanism is proposed, which allows the model to pay more attention to the places that need to be changed, while the background and identity attributes remain unchanged, so that the model is robust, and the adaptive hybrid module reduces the feature space by reducing the distance between the difficult sample and the easy sample in the feature space, where the difficult sample can be understood as a sample that is difficult to classify. We generate difficult samples by linearly combining real samples, making the image difficult to classify, and adding noise to the generated picture to increase the robustness and generalization ability of the model, and then reduce the distance between the difficult sample and the easy sample through the feature extractor. It improves the training of GAN by reducing the area of ​​training data in the discriminator image representation space, and can improve the quality of generated pictures in expression generation research.

[0029] The facial expression generation method based on generative adversarial network of the present invention, other advantages, objectives and features of the present invention will be reflected in part through the following description, and in part will be understood by technicians in this field through research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the exemplary embodiments below. The accompanying drawings are only for the purpose of illustrating exemplary embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0033] Figure 1 A schematic flowchart of a facial expression generation method based on a generative adversarial network provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the structure of a generative adversarial network provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of an adaptive hybrid module provided in an embodiment of the present application;

[0036] Figure 4 A schematic block diagram of a facial expression generation device based on a generative adversarial network provided in an embodiment of the present application;

[0037] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0039] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0041] In order to solve the above technical problems, according to the first aspect of the present application, a facial expression generation method based on a generative adversarial network is proposed. Figure 1 The following is a schematic flow chart of a facial expression generation method based on a generative adversarial network provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0042] Step S110, generating a difficult sample based on the facial image and the target expression code.

[0043] Exemplarily, a generative adversarial network is a powerful generative model. A generative adversarial network may include a generator and a discriminator. The generator can generate realistic images, and the discriminator can distinguish the authenticity of the generated images. Through the adversarial training of the generator and the discriminator, the learning of the generator is getting closer and closer to the distribution of real data, and the discriminator is increasingly difficult to distinguish between real and false data. For example, the generator can be used to encode facial images and target expressions, and the facial images and target expression encodings are used as inputs. The facial action coding system (FACS) is used, which uses expression encoding as action units (ACTI-on Units, AU), and the open source OpenFace can be used to extract action units. An adaptive hybrid module can be added to GANimation to stabilize the training of GAN and improve the results of expression generation. Among them, GANimation is a condition GAN based on action units. The adaptive hybrid module can include two parts, which are used for difficult sample generation and feature space contraction respectively.

[0044] For example, a difficult sample is a sample with a large error from the true value label when predicted, that is, a sample that is difficult to classify for the classification network. For example, if the true value one-hot label is [1,0,0], when the predicted probability distribution is [0.3,0.4,0.4], it is far from the true value label, and this sample is a difficult sample. On the other hand, when the predicted probability distribution is [0.98,0.01,0.01], it is far from the true value label, and this sample is a simple sample. For example, let X represent N training images, x i represents the i-th training image, x j Represents a random j-th training image. Mixing a bunch of training images can generate a difficult sample:

[0045] Among them, g(x i ,x j ,λ) is a linear combination function, and λ is a hyperparameter sampled from the Beta distribution Beta(α,α).

[0046] Exemplarily, the generator is applied twice in training, using the input image and the target action unit to generate the image, and then reconstructing the image and rendering it back to the expression of the input image. Instead of directly outputting the generated image, the generator outputs two masks, a color mask C and an attention mask A. The final image can be obtained as follows:

[0047] in, is the input image, To generate an image, A is a single-channel attention mask output by the generator. The function of this attention mask is to focus on the face and ignore the background to prevent the influence of complex background and lighting changes. It represents the facial parts that need to be changed in the image. A is a matrix composed of floating-point numbers between 0 and 1. The size of the matrix is ​​the same as the size of the input image. The closer the value is to 1, the more it represents that the pixels of the input image do not need to be changed. Figure 2 A schematic diagram of the structure of a generative adversarial network provided in an embodiment of the present application.

[0048] Step S120, determining a first characteristic distance between the difficult sample and the simple sample.

[0049] For example, in the feature space formed by all training samples, most difficult samples are more likely to deviate from the center than easy samples, which results in the feature space needing to expand the decision boundary in order to cover these difficult samples. Therefore, if the distance between difficult samples and easy samples can be reduced, the corresponding feature space can be reduced.

[0050] Step S130: based on a generative adversarial network, shorten the first feature distance to a second feature distance to generate training data.

[0051] In some embodiments, the generative adversarial network includes a discriminator, which is used to discriminate the authenticity of the generated image and decode the activation value of the action unit in the generated image. The discriminator includes an adaptive mixing module, which is used to embed and classify the result of mixing the generated image, and reduce the feature distance between images through the loss function to reduce the area occupied by the category in the feature space, thereby improving the training stability of the GAN. Figure 3 The schematic diagram of an adaptive hybrid module provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, in (a), a simple sample x in the category i and x j Through feature extractor Projected into the feature space, In (b), the difficult sample The training sample pair x i and x j The combination is generated and projected into the feature space. In (c), the adaptive hybrid module reduces the region of the category in the feature space by reducing the feature distance between easy samples and difficult samples.

[0052] Step S140: train the generative adversarial network using the training data to obtain a trained generative adversarial network.

[0053] For example, the role of image adversarial loss is to make the distribution of generated images closer to the distribution of training images, making the generated images look more realistic. For example, image adversarial loss can be defined as:

[0054] in, is the input image with initial condition y0, y f is the desired final condition, is the data distribution of the input image, is a random interpolation distribution, λ gp is a random penalty coefficient.

[0055] When training the model, the dataset does not have the true labels of the attention mask A and the RGB color mask C, so the attention mask is easily oversaturated, that is, the value of A is easy to approach 1, which makes This causes the difference between the generated image and the original image to be too small, making the result of the color mask C ineffective, that is, the generator has no effect. To prevent this, the mask is regularized by L2-weight penalty. In addition, in order to achieve a smooth spatial color transformation when combining the pixels of the input image and the color transformation C, A is regularized by total variation. Therefore, the attention loss can be defined as:

[0056]

[0057] in, A i,j Represents the element of the i-th row and j-th column of A. The first term is the full difference loss, which is used to smooth the image, and the second term is the L2 penalty term, λ TV is a penalty coefficient.

[0058] Exemplarily, the conditional expression loss is calculated as follows:

[0059]

[0060] The loss function defined previously allows the generator to perform realistic facial expression conversion. However, without baseline supervision, there is no constraint to ensure that the faces in the input and output images correspond to the same person. Therefore, in order to ensure the consistency of identity before and after generation, we can use cycle consistency loss, which penalizes the original image. The difference between the images it reconstructs forces the generator to maintain the identity consistency of each individual. In order to generate realistic images, it is crucial for the generator to model both low-frequency and high-frequency signals. The discriminator ID based on PatchGan has enforced high-frequency correctness by limiting its attention to the structure in the local image patch. In order to capture low-frequency signals, it is sufficient to use the L1 norm. The identity loss can be defined as:

[0061]

[0062] For example, according to x i In x ij The linear combination ratio in reduces its distance in the feature space. Here, σ is a noise term sampled from a Gaussian distribution, which can be used to prevent overfitting. It is a metric used to evaluate the distance, such as L1 norm, L2 norm, etc. In simple terms, it is to limit the range of the latent space by limiting the range of feature map generation, thereby reducing the distance.

[0063] Step S150, determining a facial expression generation result based on the trained generative adversarial network.

[0064] Exemplarily, to generate a target image, a loss function L can be constructed by linearly combining all previous partial losses:

[0065] Among them, λ A , y , idt and λ ada is a hyperparameter that controls the relative importance of each loss term. Finally, the following minimax problem can be defined: Among them, G * Generate samples from the data distribution. In addition, the discriminator is constrained represents the set of 1-Lipschitz functions.

[0066] Therefore, facial expression encoding is added to the input of the generator, so that the generator can generate a series of continuously changing expressions. An attention mechanism is proposed to allow the model to pay more attention to the places that need to be changed, while the background and identity attributes remain unchanged, making the model robust. The adaptive hybrid module reduces the feature space by reducing the distance between difficult samples and easy samples in the feature space. Among them, difficult samples can be understood as samples that are difficult to classify. We generate difficult samples by linearly combining real samples, making the images difficult to classify. At the same time, noise is added to the generated pictures to increase the robustness and generalization ability of the model, and then the distance between difficult samples and easy samples is reduced through the feature extractor. It improves the training of GAN by reducing the area of ​​training data in the discriminator image representation space, which can improve the quality of generated pictures in expression generation research.

[0067] According to the second aspect of the present application, a facial expression generation device based on a generative adversarial network is also proposed. Figure 4 A schematic block diagram of a facial expression generation device based on a generative adversarial network provided in an embodiment of the present application. Figure 4 As shown, the apparatus 400 may include:

[0068] A first generating unit 410, configured to generate a difficult sample based on the facial image and the target expression code;

[0069] A first determining unit 420, configured to determine a first characteristic distance between a difficult sample and an easy sample;

[0070] A second generating unit 430, configured to shorten the first feature distance to a second feature distance based on a generative adversarial network to generate training data;

[0071] A training unit 440 is used to train the generative adversarial network using the training data to obtain a trained generative adversarial network;

[0072] The second determining unit 450 is used to determine the facial expression generation result based on the trained generative adversarial network.

[0073] According to a third aspect of the present application, an electronic device is also provided. Figure 5 FIG. 5 shows a schematic block diagram of an electronic device 500 according to an embodiment of the present invention. Figure 5 As shown, the electronic device 500 may include a processor 510 and a memory 520. The memory 520 stores computer program instructions, which are used by the processor 510 to execute the facial expression generation method based on the generative adversarial network when executed.

[0074] According to a fourth aspect of the present application, a storage medium is also provided, on which program instructions are stored, and the program instructions are used to execute the facial expression generation method based on the generative adversarial network as described above when running. The storage medium may include, for example, a storage component of a tablet computer, a hard disk of a computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0075] A person of ordinary skill in the art can understand the specific details and beneficial effects of the facial expression generation device, electronic device and storage medium based on a generative adversarial network by reading the above description of the facial expression generation method based on a generative adversarial network, and will not be repeated here for the sake of brevity.

[0076] In the several embodiments provided in the present application, it should be understood that the disclosed devices and / or equipment can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0080] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A facial expression generation method based on a generative adversarial network, characterized in that: include: Generate difficult samples based on facial images and target expression encoding; Determining a first characteristic distance between the difficult sample and the simple sample; Based on the generative adversarial network, shortening the first feature distance to a second feature distance to generate training data; Using the training data, training the generative adversarial network to obtain a trained generative adversarial network; Based on the trained generative adversarial network, a facial expression generation result is determined.

2. The facial expression generation method based on generative adversarial network according to claim 1, characterized in that: The generative adversarial network includes a generator, which is used to output a color mask and an attention mask based on the facial image and the target expression encoding to output a generated image.

3. The facial expression generation method based on generative adversarial network according to claim 2, characterized in that: The generative adversarial network includes a discriminator, which is used to discriminate the authenticity of the generated image.

4. The facial expression generation method based on generative adversarial network according to claim 3, characterized in that: The discriminator is also used to decode the activation value of the action unit in the generated image.

5. The facial expression generation method based on generative adversarial network according to claim 3 or 4, characterized in that: The discriminator includes an adaptive mixing module for embedding and classifying the mixed result of the generated image.

6. The facial expression generation method based on generative adversarial network according to claim 1, characterized in that: The method further comprises: Compute image adversarial loss, attention loss, conditional expression loss, identity loss, and adaptive hybrid loss.

7. The facial expression generation method based on generative adversarial network according to claim 6, characterized in that: The method further comprises: Calculating a global loss based on the image adversarial loss, the attention loss, the conditional expression loss, the identity loss, and the adaptive hybrid loss; Based on the global loss, the facial expression generation result is generated.

8. A facial expression generation device based on a generative adversarial network, characterized in that: include: A first generation unit, for generating a difficult sample based on the facial image and the target expression code; A first determining unit, configured to determine a first characteristic distance between the difficult sample and the simple sample; A second generating unit, configured to shorten the first feature distance to a second feature distance based on the generative adversarial network to generate training data; A training unit, configured to train the generative adversarial network using the training data to obtain a trained generative adversarial network; The second determining unit is used to determine a facial expression generation result based on the trained generative adversarial network.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the facial expression generation method based on the generative adversarial network as claimed in any one of claims 1 to 7 when the processor is running.

10. A storage medium having program instructions stored thereon, wherein the program instructions are used to execute the facial expression generation method based on a generative adversarial network as claimed in any one of claims 1 to 7 when run.

Images