Method and system for converting facial images into sketches based on generative adversarial networks

Abstract

The present invention belongs to the technical field of image data conversion, and provides a method and system for converting a face image into a sketch based on a generative adversarial network, wherein a face image is acquired; a face sketch image is acquired based on the acquired face image and a preset face sketch image conversion model; wherein the face sketch image conversion model is obtained by generative adversarial network training; a network residual block with a multi-scale structure is adopted in the training process of the face sketch image conversion model, and a self-attention module for parallel processing is added after the network residual block with a multi-scale structure; the present invention adopts a network residual block with a multi-scale structure in the training process of the face sketch image conversion model, and simultaneously adds a self-attention module for parallel processing, adopts a multi-scale method to learn deep and shallow features, and fuses feature maps of different categories, so as to improve the synthesis quality while maintaining the speed.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data conversion, and in particular relates to a method and system for converting a face image into a sketch based on a generative adversarial network. Background Art

[0002] Face sketches have been widely used in many fields, especially in tracking suspects, which can provide convenience. At present, only a few forensic sketch artists can reach professional level, which is difficult to meet the needs of target reconnaissance and solving cases. Therefore, how to efficiently and accurately convert face photos into sketches is an urgent issue.

[0003] The inventors found that the current photo-to-sketch translation is based on a small number of sketch samples that focus on texture details, such as eyes, nose and mouth, and global facial contours; at the same time, the problem can also be viewed as an image-to-image translation problem, such as using facial composition information to synthesize facial photo sketches and adopting a novel composition-assisted generative adversarial network. However, the network structure used in the existing method is relatively complex, the training time is long, and there is still room for improvement in image clarity and predicted image quality. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a method and system for converting face images into sketches based on a generative adversarial network. The present invention uses a lightweight network to synthesize sketch images for samples in a face database, thereby improving the speed of synthesizing images while ensuring the quality of the synthesized images.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0006] In a first aspect, the present invention provides a method for converting a face image into a sketch based on a generative adversarial network, comprising:

[0007] Get face image;

[0008] Obtaining a face sketch image based on the acquired face image and a preset face sketch image conversion model;

[0009] Among them, the face sketch image conversion model is obtained by generative adversarial network training; a multi-scale network residual block is used in the face sketch image conversion model training process, and a parallel processing self-attention module is added after the multi-scale network residual block.

[0010] Furthermore, when training the face sketch image conversion model, the face image database is divided into a training set and a test set; when the face image database is larger than a preset value, a leave-one-out cross-validation method is used to select the training set and the test set; when the face image database is not larger than a preset value, a leave-one-out cross-validation method is used to select the training set and the test set.

[0011] Furthermore, the face images in the face image database are preprocessed to obtain images of the same size.

[0012] Furthermore, the generative adversarial network includes:

[0013] The upsampling part includes a convolution layer with a stride of 1 and a kernel of 7×7, and two convolution layers with a stride of 2 and a kernel of 3×3;

[0014] The first multi-scale network residual block, each residual block is a convolution layer with a step size of 1 and a convolution kernel of 3×3;

[0015] Parallel processing of self-attention parts;

[0016] The second multi-scale network residual block, each residual block is a convolution layer with a step size of 1 and a convolution kernel of 3×3;

[0017] The downsampling part adopts a structure symmetrical to the upsampling part;

[0018] The discriminator includes a downsampling module, a fully connected layer, and a convolutional layer with a stride of 1 and a convolution kernel of 4×4; the downsampling module includes three convolutional layers with a stride of 2 and a convolution kernel of 4×4, and a convolutional layer with a stride of 1 and a convolution kernel of 4×4.

[0019] Furthermore, an adaptive normalization method is adopted in the network residual block of the second multi-scale structure.

[0020] Furthermore, the loss functions include generative adversarial network loss function, identity loss function, cycle consistency loss function and class activation loss function.

[0021] Furthermore, the self-attention module processed in parallel focuses on the regions that distinguish the image from the sketch based on the feature map obtained by the auxiliary classifier.

[0022] In a second aspect, the present invention also provides a system for converting a face image into a sketch based on a generative adversarial network, comprising:

[0023] The data acquisition module is configured to: obtain a face image;

[0024] The conversion module is configured to: obtain a face sketch image according to the acquired face image and a preset face sketch image conversion model;

[0025] Among them, the face sketch image conversion model is obtained by generative adversarial network training; a multi-scale network residual block is used in the face sketch image conversion model training process, and a parallel processing self-attention module is added after the multi-scale network residual block.

[0026] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for converting a facial image into a sketch based on a generative adversarial network as described in the first aspect.

[0027] In a fourth aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for converting a facial image into a sketch based on a generative adversarial network as described in the first aspect are implemented.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention adopts a multi-scale network residual block in the training process of the face sketch image conversion model. The multi-scale structure can obtain deep and shallow information, so as to better distinguish the subsequent face information. At the same time, a self-attention module for parallel processing is added to improve the image generation quality details; a multi-scale method is used to learn deep and shallow features, and feature maps of different categories are merged to improve the synthesis quality while maintaining the speed;

[0030] 2. The present invention uses a lightweight generative adversarial network to synthesize sketch images from samples in a face database, which improves the speed of synthesizing images while ensuring the quality of the synthesized images; the generative adversarial network structure in the lightweight generative adversarial network can meet the problem of synthesized image quality, and the fusion of feature maps at different times fully utilizes shallow and deep information;

[0031] 3. The algorithm proposed in the present invention can effectively provide assistance for image migration, such as criminal tracking, while meeting the accuracy and real-time requirements, and can overcome the problems of few professional sketch artists, large memory usage for calculation, and long training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0033] Figure 1 This is a flow chart of Embodiment 1 of the present invention;

[0034] Figure 2This is a flow chart of the image preprocessing method according to Embodiment 1 of the present invention;

[0035] Figure 3 This is a schematic diagram of the network framework of Example 1 of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0038] Embodiment 1:

[0039] This embodiment provides a method for converting a face image into a sketch based on a generative adversarial network, including:

[0040] Get face image;

[0041] Obtaining a face sketch image based on the acquired face image and a preset face sketch image conversion model;

[0042] Among them, the face sketch image conversion model is obtained by generative adversarial network training; a multi-scale network residual block is used in the face sketch image conversion model training process, and a parallel processing self-attention module is added after the multi-scale network residual block.

[0043] The method of converting a face image into a sketch based on a generative adversarial network in this implementation is an image transfer learning method. In addition to being applied to face photo-sketch synthesis, it can also be applied to fields such as animation character generation. Taking the method of converting a face image into a sketch as an example, the content includes:

[0044] The face photo images in the face image library are converted into sketch images using a face sketch image conversion model. The face images can be understood as face photos, and the face sketch image conversion model can be understood as a lightweight generative adversarial network; wherein, the training process of the lightweight generative adversarial network is as follows: the method for processing the reference photo sketch adds an attention module, which focuses on the area that distinguishes the photo from the sketch based on the feature map obtained by the auxiliary classifier; different residual layers correspond to different numbers of discriminator layers to optimize the network in this embodiment. Among them, the reference photo sketch can be understood as the input image, the sketch can be understood as the output image, and the discriminator can be understood as the part of the generative adversarial network that determines whether the image generation is true; the auxiliary classifier is a classifier that assists in correctly generating the corresponding output image. The specific steps of this embodiment are:

[0045] S1. Collect a face image library, such as a face photo library, and divide the images in the face image library into a training set and a test set, such as Figure 2 As shown; Specifically, the face image library can be collected through a mobile phone, a camera, etc., or existing face images can be collected on the Internet; it can be understood that the face image library can include face photos and sketch images corresponding to the face photos;

[0046] The specific process of step S1 is:

[0047] S1.1. Preprocess the face image to obtain images of the same size and input them into the neural network. When obtaining images of the same size, you can first use the python library to process files of different formats and image types, so that numpyndarray can store the image, and use PIL to crop the image.

[0048] S1.2, when the amount of data in the face image database is greater than the preset value, that is, when the database is large, the method for selecting the training set and the test set is the leave-one-out validation method; when the amount of data in the face image database is not greater than the preset value, that is, when the amount of data is not large, the leave-one-out cross-validation method is used, such as Figure 2 As shown; the preset value can be understood as a constant, which can be defined and changed according to actual conditions; the leave-one-out validation method is to directly split the data set into two mutually exclusive sets, one of which is the training set and the other is the test set; the leave-one-out cross-validation method is to divide the data set into mutually exclusive subsets of the data set size, and then each time use the union of the data set size-1 subsets as the training set, and the remaining subset as the test set, so as to obtain the data set size group training / test set, and perform data set size times of training and testing.

[0049] S2, input the training set of pre-processed face photo images into Figure 3 The lightweight generative adversarial network shown in FIG2 is used for training. During the training process, a multi-scale method is used to learn deep and shallow features, and feature maps of different categories are fused to improve the synthesis quality while maintaining the speed. The specific process of step S2 is as follows:

[0050] S2.1, the downsampling part, passes through the convolution layer with a stride of 1 and a kernel of 7×7, and two convolution layers with a stride of 2 and a kernel of 3×3;

[0051] S2.2, such as Figure 3 As shown in the figure, more abundant image information can be obtained through the residual network. Here, a multi-scale structure is used to perform 4-layer and 6-layer network residual blocks on the input, that is, the network residual blocks of the first multi-scale structure. Each residual block is a convolution layer with a step size of 1 and a convolution kernel of 3×3. The multi-scale structure can obtain deep and shallow information, so as to better distinguish the subsequent face information.

[0052] S2.3, such as Figure 3 As shown in the figure, after a fully connected layer, the generator is trained through multi-scale feature maps, and a parallel self-attention module is added to improve the image generation quality details;

[0053] S2.4, through the multi-scale structure, the input is respectively subjected to 4-layer and 6-layer network residual blocks, that is, the network residual blocks of the second multi-scale structure; wherein each residual block is a convolution layer with a step size of 1 and a convolution kernel of 3×3. The difference is that an adaptive normalization method is adopted, which linearly combines instance normalization and layer normalization, and adjusts the weight ratio of instance normalization and layer normalization during training to achieve the best normalization effect;

[0054] S2.5, the downsampling module adopts a symmetrical structure with the upsampling module, and passes through two convolution layers with a stride of 2 and a convolution kernel of 3×3, and a convolution layer with a stride of 1 and a convolution kernel of 7×7. So far, the structure of the generator has been introduced;

[0055] S2.6: In terms of the discriminator structure, such as Figure 3 As shown in the figure, the image is encoded through a downsampling module, which is specifically 3 convolutional layers with a stride of 2 and a convolution kernel of 4×4, and a convolutional layer with a stride of 1 and a convolution kernel of 4×4. This structure is pruned on the basis of the super-resolution synthesis network, which improves the computational complexity.

[0056] S2.7, after downsampling, the network passes through a fully connected layer and enters the calculation and judgment of the feature map; in terms of loss function, the training loss of the class activation function map and the sketch map is used;

[0057] S2.8. Finally, the discrimination result is obtained through a convolution layer with a stride of 1 and a convolution kernel of 4×4.

[0058] In this embodiment, the overall loss function includes four parts: generative adversarial network loss, identification loss, cycle consistency loss and class activation loss. In order to stabilize the training photos, a least squares generative adversarial network can be used in this embodiment.

[0059] Among them, the loss function of the generated adversarial network is:

[0060]

[0061] in, For the discriminator, the ability to match the distribution of photo images with the distribution of sketch images is modeled; D s is the discriminator of sketch image; G p->s A face photo to sketch generator.

[0062] The identification loss function is:

[0063]

[0064] in, The color contribution applied to the generator, which can convert the photo image into the sketch image accordingly.

[0065] The cycle consistency loss is:

[0066]

[0067] in, G is the cycle consistency loss of the generator, which correctly distinguishes between photo images and sketch images during the conversion process; p->s is the conversion of the generator face photo to sketch; G s->p It is the conversion from the generator sketch to the face photo.

[0068] The class activation loss is:

[0069]

[0070]

[0071] Among them, for the generator, use to judge the difference between the two images and select to bridge the gap between the two domains of the discriminator.

[0072] This embodiment uses a lightweight generative adversarial network to solve the face photo-sketch conversion problem; the face photos in the face photo library are divided into a training set and a test set, the images of the face photos are preprocessed to obtain images of the same size, and are input into the neural network; through the generative adversarial network, the generator uses a self-attention module and a residual module to extract features based on a multi-scale structure, and the discriminator adjusts the number of layers in the corresponding downsampling module, reducing the number of convolutional layers, saving computer resources, and meeting the lightweight requirement; the face photo-sketch conversion algorithm proposed in this embodiment can effectively provide assistance for application fields such as criminal investigation and solving cases, while meeting the accuracy and real-time requirements, and can overcome the problems of few sketch artists, large memory usage for calculations, and long training time.

[0073] Embodiment 2:

[0074] This embodiment provides a system for converting a face image into a sketch based on a generative adversarial network, including:

[0075] The data acquisition module is configured to: obtain a face image;

[0076] The conversion module is configured to: obtain a face sketch image according to the acquired face image and a preset face sketch image conversion model;

[0077] Among them, the face sketch image conversion model is obtained by generative adversarial network training; a multi-scale network residual block is used in the face sketch image conversion model training process, and a parallel processing self-attention module is added after the multi-scale network residual block.

[0078] The working method of the system is the same as the method of converting facial images into sketches based on a generative adversarial network in Example 1, and will not be repeated here.

[0079] Embodiment 3:

[0080] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the method for converting a facial image into a sketch based on a generative adversarial network described in Embodiment 1 are implemented.

[0081] Embodiment 4:

[0082] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for converting a facial image into a sketch based on a generative adversarial network described in Embodiment 1 are implemented.

[0083] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A method for converting a face image into a sketch based on a generative adversarial network, characterized in that: include: Get face image; Obtaining a face sketch image based on the acquired face image and a preset face sketch image conversion model; The face sketch image conversion model is obtained by generative adversarial network training; a multi-scale network residual block is used in the face sketch image conversion model training process, and a parallel processing self-attention module is added after the multi-scale network residual block; The generative adversarial network comprises: The upsampling module includes a convolutional layer with a stride of 1 and a kernel of 7×7, and two convolutional layers with a stride of 2 and a kernel of 3×3; The first multi-scale network residual block, each residual block is a convolution layer with a step size of 1 and a convolution kernel of 3×3; Self-attention module for parallel processing; The second multi-scale network residual block, each residual block is a convolution layer with a step size of 1 and a convolution kernel of 3×3; The downsampling module adopts a structure symmetrical to the upsampling module; The discriminator includes a downsampling module, a fully connected layer, and a convolutional layer with a stride of 1 and a convolution kernel of 4×4. The downsampling module includes three convolutional layers with a stride of 2 and a convolution kernel of 4×4, and a convolutional layer with a stride of 1 and a convolution kernel of 4×4. Among them, an adaptive normalization method is used in the network residual block of the second multi-scale structure; The loss functions of the generative adversarial network include the generative adversarial network loss function, the identification loss function, the cycle consistency loss function, and the class activation loss function; Among them, the loss function of the generated adversarial network is: in, For the discriminator, the ability to match the distribution of photo images with the distribution of sketch images is modeled; D s is the discriminator of sketch image; G p->s A generator for face photos to sketches; The identification loss function is: in, The color contribution applied to the generator converts the photo image into a sketch image accordingly; The cycle consistency loss is: in, G is the cycle consistency loss of the generator, which correctly distinguishes between photo images and sketch images during the conversion process; p->s is the conversion of the generator face photo to sketch; G s->p is the conversion from the generator sketch to the face photo; The class activation loss is: Among them, for the generator, use to judge the difference between the two images and select to bridge the gap between the two domains of the discriminator.

2. The method for converting a face image into a sketch based on a generative adversarial network as claimed in claim 1, characterized in that: When training the face sketch image conversion model, the face image database is divided into a training set and a test set; when the face image database is larger than a preset value, the leave-out validation method is used to select the training set and the test set; when the face image database is not larger than the preset value, the leave-one-out cross-validation method is used to select the training set and the test set.

3. The method for converting a face image into a sketch based on a generative adversarial network as claimed in claim 2, characterized in that: The face images in the face image database are preprocessed to obtain images of the same size.

4. The method for converting a face image into a sketch based on a generative adversarial network as claimed in claim 1, characterized in that: The self-attention module processed in parallel focuses on the regions that distinguish images from sketches based on the feature maps obtained by the auxiliary classifier.

5. A system for converting facial images into sketches based on generative adversarial networks, characterized in that: include: The data acquisition module is configured to: obtain a face image; The conversion module is configured to: obtain a face sketch image according to the acquired face image and a preset face sketch image conversion model; The face sketch image conversion model is obtained by generative adversarial network training; a multi-scale network residual block is used in the face sketch image conversion model training process, and a parallel processing self-attention module is added after the multi-scale network residual block; The generative adversarial network comprises: The upsampling module includes a convolutional layer with a stride of 1 and a kernel of 7×7, and two convolutional layers with a stride of 2 and a kernel of 3×3; The first multi-scale network residual block, each residual block is a convolution layer with a step size of 1 and a convolution kernel of 3×3; Self-attention module for parallel processing; The second multi-scale network residual block, each residual block is a convolution layer with a step size of 1 and a convolution kernel of 3×3; The downsampling module adopts a structure symmetrical to the upsampling module; The discriminator includes a downsampling module, a fully connected layer, and a convolutional layer with a stride of 1 and a convolution kernel of 4×4. The downsampling module includes three convolutional layers with a stride of 2 and a convolution kernel of 4×4, and a convolutional layer with a stride of 1 and a convolution kernel of 4×4. Among them, an adaptive normalization method is used in the network residual block of the second multi-scale structure; The loss functions of the generative adversarial network include the generative adversarial network loss function, the identification loss function, the cycle consistency loss function, and the class activation loss function; Among them, the loss function of the generated adversarial network is: in, For the discriminator, the ability to match the distribution of photo images with the distribution of sketch images is modeled; D s is the discriminator of sketch image; G p->s A generator for face photos to sketches; The identification loss function is: in, The color contribution applied to the generator, which can convert the photo image into a sketch image accordingly; The cycle consistency loss is: in, G is the cycle consistency loss of the generator, which correctly distinguishes between photo images and sketch images during the conversion process; p->s is the conversion of the generator face photo to sketch; G s->p is the conversion from the generator sketch to the face photo; The class activation loss is: Among them, for the generator, use to judge the difference between the two images and select to bridge the gap between the two domains of the discriminator.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for converting a facial image into a sketch based on a generative adversarial network as described in any one of claims 1 to 4 are implemented.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for converting a facial image into a sketch based on a generative adversarial network as described in any one of claims 1 to 4 are implemented.

Images