Face personalization paper-cut generation method based on constraint cycle generative adversarial network

Abstract

This application discloses a method for generating personalized paper-cut faces based on constrained recurrent generative adversarial networks (RBANs), comprising: S10: establishing a face dataset including face images and a paper-cut dataset including paper-cut images; S20: using a pre-trained model to obtain a corresponding face parsing dataset based on the face dataset, wherein the face parsing images in the face parsing dataset are distinguished by black and white to distinguish key regions and non-key regions; S30: designing a recurrent adversarial network for fusing key facial region features; S40: training the recurrent generative adversarial network using the face dataset, the paper-cut dataset, and the face parsing dataset, and obtaining a face-generating paper-cut generator model that fuses key facial regions after the recurrent generative adversarial network for fusing key facial region constraints reaches stability; S50: inputting the face images and face parsing images into the face-generating paper-cut generator model to obtain personalized paper-cut faces.

Description

Technical Field

[0001] This application relates to the field of computer vision image transformation, and in particular to a method for generating personalized paper-cut faces based on constrained cyclic generative adversarial networks. Background Technology

[0002] Paper cutting is an art form that uses paper as the material and scissors as the tool for creation. It was among the first batch of national intangible cultural heritage items, and personalized paper cutting of faces is a type of paper cutting. Given the current national emphasis on promoting traditional culture, personalized paper cutting of faces has immense potential. However, non-intelligent methods are not conducive to large-scale creation, limiting the inheritance and promotion of personalized paper cutting of faces. Therefore, intelligent generation of personalized paper cuttings is a new way to inherit and promote this intangible cultural heritage. With the rapid development of deep learning in recent years, it has become possible to generate personalized paper cuttings using image-to-image conversion technology.

[0003] Currently, there are three methods for generating personalized paper-cut faces. First, processing with computer-aided design tools, such as Photoshop. Second, using face paper-cut templates, for example, using ASM (Active Shape Modeling, which identifies and separates components of a target face) to identify and extract facial features, and then using deformation methods to generate the target face paper-cut image. Third, using generative adversarial networks (GANs), where the generator and discriminator learn against each other to obtain an optimized generator that produces a paper-cut that closely resembles the facial features of a real person. Of these methods, using computer-aided design tools and face paper-cut templates results in poor intelligence and is time-consuming and labor-intensive; using GANs removes excessive noise, leading to significant differences between the resulting paper-cut face and the real facial features, failing to meet the needs of personalization. Summary of the Invention

[0004] The purpose of this application is to propose a method for generating personalized paper-cut faces based on constrained recurrent generative adversarial networks, which generates paper-cuts that closely resemble facial features based on the key facial regions after facial image analysis.

[0005] This application is achieved through the following technical measures: a method for generating personalized paper-cut faces based on constrained recurrent generative adversarial networks, comprising: S10: establishing a face dataset including face images and a paper-cut dataset including paper-cut images; S20: using a pre-trained model to obtain a corresponding face parsing dataset based on the face dataset, wherein the face parsing images in the face parsing dataset are distinguished by black and white to distinguish key regions and non-key regions; S30: designing a recurrent adversarial network for fusing key facial region features; S40: training the recurrent generative adversarial network using the face dataset, the paper-cut dataset, and the face parsing dataset to obtain a face-generating paper-cut generator model that fuses key facial regions; S50: inputting the face images and the face parsing images into the face-generating paper-cut generator model to obtain personalized paper-cut faces.

[0006] Preferably, the key areas include hair, eyes, nose, and mouth.

[0007] Preferably, the recurrent adversarial network for fusing facial key region features adopts a two-stage fusion of facial key regions: in the face generation paper-cutting stage, facial key region features are fused to assist in the generation of paper-cutting; in the paper-cutting face generation stage, facial key region features are fused to assist in the generation of face.

[0008] Preferably, in the personalized face paper-cutting generation method based on constrained recurrent generative adversarial network according to claim 3, the design process of the recurrent adversarial network for fusing key facial region features includes: firstly, establishing a generator G_A for generating paper-cutting from a face and a generator G_B for generating a face from paper-cutting, wherein G_A consists of G_A_conv and G_A_tran, where G_A_conv represents the convolutional layer of the face paper-cutting generator and G_A_tran represents the deconvolutional layer of the face paper-cutting generator; G_B consists of G_B_conv and G_B_tran, where G_B_conv represents the convolutional layer of the paper-cutting generator and G_B_tran represents the deconvolutional layer of the paper-cutting generator; and adding a convolutional layer Conv_3 to G_A_conv to obtain the convolutional data of the face image. G_A_conv_data is obtained by adding a Conv_3 convolutional layer to G_B_conv to obtain the convolutional data G_B_conv_data of the paper-cut image. Next, a feature model is used to extract high-level semantic features from the face image to obtain convolutional data E_A_data. G_A_conv_data and G_B_conv_data are then fused with E_A_data to obtain new convolutional data G_A_conv_new_data and G_B_conv_new_data, respectively. Then, the convolutional data G_A_conv_new_data and G_B_conv_new_data are input to G_A_tran and G_B_tran, respectively, to obtain the paper-cut image and the face image. Finally, a recurrent adversarial network for fusing key facial region features is obtained.

[0009] As a preferred option, P-Net is used as the pre-trained model.

[0010] Beneficial Effects: This application extracts high-level semantic features from face parsing images and fuses them with low-level semantic features extracted from face images and paper-cutting images to establish positional consistency constraints on key facial regions. This allows the generator to guide image generation. The results obtained by guiding the high-level semantic features in a two-stage generator show a significant improvement over baseline recurrent generative adversarial networks. Through this design, the invention enriches the dependency relationship between the high-level semantic features after face parsing and the low-level features of the face and paper-cutting, ultimately achieving better generation results. Attached Figure Description

[0011] The accompanying drawings are provided to further illustrate the present application and form part of the specification. They are used together with the embodiments of the present application to explain the application and do not constitute a limitation thereof. In the drawings:

[0012] Figure 1 Flowchart illustrating the method for generating personalized paper-cut faces for this applicant;

[0013] Figure 2 These are schematic diagrams of face images, paper-cut images, and face analysis images;

[0014] Figure 3 This is a fusion model diagram of a human face image, a paper-cut image, and then a human face.

[0015] Figure 4 The paper-cut image generated using the original network model;

[0016] Figure 5 The paper-cut image generated using the method of this application. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0018] A personalized face paper-cutting generation method based on constrained cyclic generative adversarial networks, such as Figure 1 As shown, it includes:

[0019] S10: Create a face dataset that includes face images and a paper-cut dataset that includes paper-cut images.

[0020] S20: A pre-trained model is used to obtain the corresponding face parsing dataset based on the face dataset. In this embodiment, the face parsing images in the face parsing dataset are distinguished by black and white to differentiate key and non-key regions. The pre-trained model decomposes the face image into multiple constituent elements, with key regions including hair, nose, eyes, and mouth. In this embodiment, P-Net, which has a relatively good decomposition effect, is selected as the pre-training model. The face parsing training set is obtained by using the face dataset on the P-Net pre-training model.

[0021] Examples of processed faces, paper cutouts, and face analysis images, such as... Figure 2 As shown.

[0022] S30: Design a recurrent adversarial network for fusing facial key region features. This recurrent adversarial network for fusing facial key region features adopts a two-stage fusing of facial key regions. In the first stage (face generation paper-cutting stage), facial key region features are fused to assist in the generation of paper-cutting; in the second stage (paper-cutting face generation stage), facial key region features are fused to assist in the generation of face.

[0023] Specifically, firstly, a generator G_A is established to generate paper-cut images from faces, and a generator G_B is established to generate faces from paper-cut images. G_A consists of G_A_conv and G_A_tran, where G_A_conv represents the convolutional layer of the face-to-paper-cut image generator and G_A_tran represents the deconvolutional layer of the face-to-paper-cut image generator. G_B consists of G_B_conv and G_B_tran, where G_B_conv represents the convolutional layer of the paper-cut image generator and G_B_tran represents the deconvolutional layer of the paper-cut image generator. A convolutional layer Conv_3 with a kernel of 3 is added to G_A_conv to obtain the convolutional data G_A_conv_data of the face image, and Conv_3 is added to G_B_conv to obtain the convolutional data G_B_conv_data of the paper-cut image.

[0024] Taking G_A as an example, the original G_A_conv output data has a dimension of 1×256×64×64. To perform feature fusion of key facial regions, a convolutional layer Conv_3 with a kernel size of 3 is added, reducing the output dimension from 256 to 128. The added convolutional layer Conv_3 and G_A_conv then form a new convolutional model, with the output data G_A_conv_data having a dimension of 1×128×64×64. G_B uses the same approach as G_A to obtain the output data G_B_conv_data. The original G_A_tran and G_B_tran of G_A and G_B are then deconvolutionally transformed into their respective deconvolutional models.

[0025] Secondly, a feature model is used to extract high-level semantic features from the face image to obtain convolutional data E_A_data; G_A_conv_data and G_B_conv_data are respectively fused with E_A_data to obtain new convolutional data G_A_conv_new_data and G_B_conv_new_data.

[0026] In this embodiment, the feature model extracted after face image parsing is called E_A. This model mainly extracts high-level semantic features from the face parsing image. The network structure of this feature model mainly consists of four convolutional layers. For the first convolutional layer, a convolutional-ReLU activation structure with a kernel of 7 is used. Two convolutional-batch normalization-ReLU activation structures with a kernel of 3 are added in the middle. Finally, a convolutional network with only a kernel of 3 is added to convolve the input dimension of 1×3×256×256 to the output dimension of 1×128×64×64. This output data is E_A_data.

[0027] Next, the convolutional data G_A_conv_new_data and G_B_conv_new_data are input into G_A_tran and G_B_tran respectively to obtain the paper-cut image and the face image; then, a recurrent adversarial network for fusing key facial region features is obtained.

[0028] This network design allows for feature extraction of key facial regions with fewer convolutional layers, reducing the model size and enabling the extracted features to be fused with the two-stage network.

[0029] Specifically, the output E_A_data of the high-level semantic feature extraction model E_A is fused with the convolutional data G_A_conv_data to obtain convolutional data with a dimension of 1×256×64×64. This is convolutional data that fuses facial key region features with facial features, providing guidance for personalized paper-cutting generation. The same approach is used for G_B, fusing the data E_A_data obtained from E_A with the G_B convolutional data G_B_conv_data, allowing facial key regions to guide face generation. The fused convolutional data G_A_conv_new_data and G_B_conv_new_data are then input into G_A_tran and G_B_tran respectively to obtain the final result. The model is as follows. Figure 3 As shown.

[0030] S40: A recurrent generative adversarial network (RGAN) is trained using face datasets, paper-cutting datasets, and face parsing datasets. Once the RGAN with key region constraints reaches stability (i.e., the various losses obtained during training tend to stabilize), a face-to-paper-cutting generator model with fused key facial regions is obtained, and the generator and discriminator model parameters are saved. Since face images, paper-cutting images, and face parsing images belong to different modalities, different networks are used to model them. This ensures that when training each face image, a corresponding face parsing image assists in both stages (face-to-paper-cutting stage and paper-cutting-to-face stage), meaning that during training, each face image is accompanied by a corresponding face parsing image.

[0031] Set the training parameters for the recurrent generative adversarial network based on the fusion of key facial region features: training iterations N = 100, batch size b = 1, Adam optimizer learning rate α = 0.0002, exponential decay rate β = 0.5, and train the model.

[0032] Model loss design includes:

[0033] Step 4.1: In this example, the model loss includes the total adversarial loss L. GAN_new_G Total cyclic consistent loss L cycle. λ c It's a set parameter, G A (x;θ e ) represents the generator for generating paper cutouts from faces, G B (y;θ e ) represents a generator for generating faces from paper-cut designs, θ e This indicates the fusion of key facial region data, D A D represents a face detector. B This refers to a paper-cutting identification device. The overall calculation formula is as follows:

[0034] L(G A (x;θ e ),G B (y;θ e ),D A D B ) = L GAN_new_G +λ c ×L cycle (1)

[0035] L GAN_new_G =L GAN_new_G_A +L GAN_new_G_B (2)

[0036]

[0037] For formula (2), L GAN_new_G_A L represents the adversarial loss of face generation paper-cutting. GAN_new_G_B This represents the adversarial loss in generating a face from paper-cutting. For formula (3), This represents the cycle consistency loss between the original face and the reconstructed face. This indicates the cyclical consistency loss between paper cutting and reconstructed paper cutting.

[0038] Step 4.2: As described in Step 4.1, the adversarial loss for face generation and paper cutting is calculated as follows:

[0039]

[0040] Step 4.3: As described in Step 4.1, the adversarial loss for generating a face from paper-cutting is calculated as follows:

[0041]

[0042] Step 4.4: As described in Step 4.1, cycle-consistent loss. The calculation is as follows, where ||.||1 represents L 1 Norm:

[0043]

[0044] Step 4.5: As described in Step 4.1, cycle-consistent loss. The calculation is as follows:

[0045]

[0046] Step 4.6: Substitute (4) and (5) into (2) to obtain L. GAN_new_G Substituting (6) and (7) into (3) yields L. cycle Finally, substituting into (1) yields the final loss.

[0047] Step 4.7: The final optimization objective is expressed as:

[0048] G * =arg min G max D L(G A (x;θ e ),G B (y;θ e ),D A D B (8)

[0049] G * This refers to the optimized generator obtained after model training.

[0050] S50: Input the face image and the face analysis image into the face generation paper-cut generator model to obtain personalized face paper-cuts.

[0051] The paper-cutting image was generated using the original recurrent generative adversarial network model, and the result is as follows: Figure 4 As shown, the model before improvement suffers from significant loss, resulting in facial shadows and misaligned features in the generated paper-cuts. The proposed method, which uses a two-stage fusion of key facial region features to generate personalized paper-cuts, yields the following results: Figure 5 As shown, the loss is relatively small, and the generated paper-cut not only retains the paper-cutting style but also closely resembles the facial features of the original human face image.

[0052] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for generating personalized paper-cut faces based on constrained cyclic generative adversarial networks, characterized in that, include: S10: Establish a face dataset including face images and a paper-cutting dataset including paper-cutting images; S20: A pre-trained model is used to obtain the corresponding face parsing dataset based on the face dataset. The face parsing images in the face parsing dataset are distinguished by black and white to distinguish key regions and non-key regions. S30: Design a recurrent adversarial network for fusing key facial region features; the recurrent adversarial network for fusing key facial region features adopts a two-stage fusing of key facial regions, fusing key facial region features in the face generation paper-cutting stage to assist in the generation of paper-cutting; In the process of generating a face from paper-cutting, key facial region features are integrated to assist in the face generation. The design process of the recurrent adversarial network for fusing key facial region features includes: First, we establish a generator G_A for generating paper-cut images from faces, and a generator G_B for generating faces from paper-cut images. G_A consists of G_A_conv and G_A_tran, where G_A_conv represents the convolutional layer of the face-to-paper-cut image generator, and G_A_tran represents the deconvolutional layer. G_B consists of G_B_conv and G_B_tran, where G_B_conv represents the convolutional layer of the paper-cut image generator, and G_B_tran represents the deconvolutional layer. We then add a convolutional layer Conv_3 to G_A_conv to obtain the convolutional data G_A_conv_data of the face image, and add a convolutional layer Conv_3 to G_B_conv to obtain the convolutional data G_B_conv_data of the paper-cut image. Secondly, a feature model is used to extract high-level semantic features from face images to obtain convolutional data E_A_data; G_A_conv_data and G_B_conv_data are respectively fused with E_A_data to obtain new convolutional data G_A_conv_new_data and G_B_conv_new_data; Next, the convolutional data G_A_conv_new_data and G_B_conv_new_data are input into G_A_tran and G_B_tran respectively to obtain the paper-cut image and the face image; then a recurrent adversarial network for fusing key facial region features is obtained. S40: The recurrent generative adversarial network is trained using a face dataset, a paper-cutting dataset, and a face parsing dataset to obtain a face-generating paper-cutting generator model that integrates key facial regions; S50: Input the face image and the face analysis image into the face generation paper-cut generator model to obtain a personalized face paper-cut.

2. The method for generating personalized face paper-cuts based on constrained cyclic generative adversarial networks according to claim 1, characterized in that, The key areas include hair, eyes, nose, and mouth.

3. The method for generating personalized face paper-cutting based on constrained cyclic generative adversarial networks according to claim 1, characterized in that, P-Net was used as the pre-trained model.

Images