A portrait sketch image generation method and device based on a GAN, equipment and medium

Abstract

The present application provides a kind of portrait sketch image generation method, device, equipment and medium based on GAN, comprising: obtaining original drawing to be drawn;The original drawing is input to the face-to-sketch generator set based on GAN generation;Wherein, the face-to-sketch generator set includes N image-to-sketch generator, the original drawing is regarded as the input parameter of the first image-to-sketch generator, the output process image and the original drawing are regarded as the input parameter of the next image-to-sketch generator, and the original drawing is sequentially input to the last image-to-sketch generator to output the target sketch image corresponding to the original drawing.The purpose is to solve the problem that the existing face sketch synthesis technical scheme loses the information of face recognition, the quality of sketch is very poor and lacks the process of drawing sketch.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human sketch, and in particular to a human sketch image generation method and device based on GAN, equipment and medium. BACKGROUND

[0002] In daily entertainment, face sketch is one of the most popular portrait styles. For example, tourists often keep the sketch works of artists in tourist attractions or scenic spots as souvenirs. However, it takes a lot of time and effort to let professional artists create realistic face sketches, and the drawing of face sketch is a skill that needs long-term training, so beginners cannot draw sketch images as realistic as artists. At the same time, in law enforcement and criminal justice cases, the face photos of potential criminals may not be available because only low-quality images are captured under partial occlusion, at this time, by constructing the mapping between face photos and sketches, face sketch can be used as an alternative method to find suspects. In recent years, face sketch synthesis technology (FPSS, Face photo-sketch synthesis) has attracted widespread attention, and it plays an important role in many fields. It is a very challenging task in computer vision; so far, many methods have been proposed to facilitate the synthesis of face sketches, which can be roughly divided into three categories: example-based methods, linear regression-based methods and model-based methods.

[0003] The example-based method is mainly used for finding the correlation between the sample photo patch and the test photo patch in the photo-sketch paired dataset, and directly reconstructs the final output sketch by mixing the sample patch corresponding to the test photo patch, although these methods have good performance in the photo-sketch paired dataset, but the generated sketch has obvious defects, the synthesized sketch is too smooth, and the fidelity and recognizability of the corresponding face photo cannot be maintained, which may cause the loss of face recognition information. The linear regression-based method mainly assumes that there is a linear mapping between the face photo and the sketch, and then generates the sketch by modeling the mapping from the photo; these methods are mainly inspired by the local linear embedding (LLE), which improves the early sketch generation task due to its low computational complexity, however, the mapping function may not be accurately formulated, resulting in poor quality of the generated sketch. In recent years, with the rapid development of deep learning, great progress has been made in image translation, and a large number of researchers have begun to use GAN in the photo-to-sketch task; for example, Pix2Pix uses conditional GAN to complete the image style conversion task in a supervised manner, but the paired training data may be difficult to obtain in actual situation; CycleGAN solves the problem of non-existent paired training data by keeping the key attributes between the input and the translated image through the cycle consistency loss, however, the drawing of the sketch image by the painter is a gradual process, and the current methods only focus on how to better learn the direct mapping between different domains, without considering the process of drawing the sketch.

[0004] Therefore, the present application is proposed. SUMMARY

[0005] Therefore, the present application is proposed.

[0006] The present application discloses a GAN-based portrait sketch image generation method, which comprises the following steps:

[0007] Obtaining an original image to be drawn;

[0008] input the original image into a GAN-based face-to-sketch generator set; wherein the face-to-sketch generator set comprises N image-to-sketch generators, the original image is taken as an input parameter of a first image-to-sketch generator, a process image output by the first image-to-sketch generator and the original image are taken as input parameters of a next image-to-sketch generator, and a target sketch image corresponding to the original image is output by a last image-to-sketch generator in sequence.

[0009] Preferably, before inputting the original image into the GAN-based face-to-sketch generator set, the method further comprises:

[0010] obtaining a plurality of training images and a set of hand-drawn sketch images corresponding to each training image, wherein the set of sketch images comprises N sketch images at different drawing stages;

[0011] for each training image and each sketch image, establishing a face-to-sketch generator, a sketch-to-face generator and a discriminator corresponding thereto;

[0012] inputting the sketch image into the sketch-to-face generator to generate a corresponding fake image;

[0013] inputting the training image and an intermediate image into the face-to-sketch generator after splicing to generate a corresponding fake sketch, wherein the intermediate image is a blank image or a fake image generated in a previous drawing stage;

[0014] calling the discriminator set to compare the fake image with the training image and the fake sketch with the sketch image to generate a corresponding comparison score;

[0015] comparing the comparison score with a preset score value, and generating a GAN-based face-to-sketch generator set when the comparison score reaches the preset score value.

[0016] Preferably, the discriminator calculates the adversarial loss of the fake sketch and the sketch image according to a formula L ad1 (G A ,G B )=E b [logD B (b)]+E a [log(1-D B (G A (a)))] and calculates the adversarial loss of the fake image and the training image according to a formula L ad2 (G A ,G B )=E a [logD A (a)]+E b [log(1-D A (GB (b)))] the calculation of an adversarial loss for the fake image and the training image, wherein G A is a face-to-sketch generator, G A (a) is the fake sketch image, G B is a sketch-to-face generator, G B (b) is the fake image, D A is an A-domain face discriminator, D B is a B-domain sketch discriminator, L ad is an adversarial loss.

[0017] Preferably, the discriminator according to the formula L cyc (G A ,G B )=E a [(G B (G A (a))-a||1]+E B [(G A (G B (b))-b||1] calculates the cycle consistency loss for the fake sketch image and the sketch image, and the fake image and the training image, wherein the L cyc is a cycle consistency loss.

[0018] Preferably, the discriminator according to the formula L im (G A ,G B )=E a [(G A (a))-a||1]+E B [(G B (b))-b||1] calculates the identity mapping loss for the fake sketch image and the sketch image, and the fake image and the training image, wherein the L im is an identity mapping loss.

[0019] Preferably, the discriminator according to the formula L A_id (G A1 ,G A2 ,G A3 )=E a [(G A1 (a))-a||1]+E a,b1 [(G A2 (a),b1)-a||1]+E a,b2 [(G B (G A2 (G A1 (a),b1),b2)-a||1] calculates the identity information loss for the fake sketch image and the sketch image, according to the formula LB_id (G B1 G B2 G B3 ) = E a,b1 [(G B1 (b1))-a||1]+E a,b2 [G B2 (b2)-a||1]+E a,b3 [G B3 (b3)-a||1] calculates the identity information loss for the fake image and the training image, wherein the L A_id and the L B_id This is due to the loss of identity information.

[0020] Preferably, the loss function of the discriminator is given by formula L. total =λ1*(L ad1 +L ad2 )+λ2*L cyc +λ3*L im +λ4*(L A_id +L B_id ), where λ1, λ2, λ3, and λ4 represent the relative importance of adversarial loss, cycle consistency loss, identity mapping loss, and identity information loss, respectively.

[0021] The present invention also provides a GAN-based human portrait sketching image generation device, comprising:

[0022] Image acquisition unit, used to acquire the original image to be drawn;

[0023] An image conversion unit is used to input the original image into a face-to-sketch generator group based on GAN; wherein the face-to-sketch generator group includes N image-to-sketch generators, the original image is used as the input parameter of the first image-to-sketch generator, the output process image and the original image are used as the input parameters of the next image-to-sketch generator, and so on until the last image-to-sketch generator outputs the target sketch image corresponding to the original image.

[0024] The present invention also provides a GAN-based portrait sketching image generation device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a GAN-based portrait sketching image generation method as described above.

[0025] The present invention also provides a readable storage medium storing a computer program that can be executed by a processor of the device in which the storage medium is located, to implement a GAN-based portrait sketching image generation method as described in any of the above claims.

[0026] To sum up, the embodiment provides a kind of portrait sketch image generation method, device, equipment and medium based on GAN, obtains original drawing to be drawn, and input the original drawing into the face conversion sketch generator group based on GAN generation, after the generation of multiple image conversion sketch generators, corresponding target sketch image of original drawing will be finally output, to solve the obvious defects of sketch generated in the face sketch synthesis technical scheme in prior art, the sketch synthesized is too smooth, cannot keep the fidelity and identifiable of corresponding face photo, this can cause the information loss of face recognition, the quality of sketch generated is very poor and the problem of lack of sketch drawing process. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 It is the flowchart of the portrait sketch image generation method based on GAN provided by the embodiment of the application.

[0028] Figure 2 It is the 25-step schematic diagram of sketch drawing of the portrait sketch image generation method based on GAN provided by the embodiment of the application.

[0029] Figure 3 It is the processing flowchart of the portrait sketch image generation method based on GAN provided by the embodiment of the application.

[0030] Figure 4 It is the overall structure schematic diagram of the portrait sketch image generation method based on GAN provided by the embodiment of the application.

[0031] Figure 5 It is the first step flowchart of the first stage of the portrait sketch image generation method based on GAN provided by the embodiment of the application.

[0032] Figure 6 It is the generator model structure schematic diagram of the portrait sketch image generation method based on GAN provided by the embodiment of the application.

[0033] Figure 7 It is the discriminator model structure schematic diagram of the portrait sketch image generation method based on GAN provided by the embodiment of the application.

[0034] Figure 8 It is the three basic loss function constitution schematic diagram of the portrait sketch image generation method based on GAN provided by the embodiment of the application.

[0035] Figure 9 It is the CUHK student data set experimental effect schematic diagram under six different illumination conditions of the portrait sketch image generation method based on GAN provided by the embodiment of the application.

[0036] Figure 10Fig. 1 is a schematic diagram of CUFS dataset experimental results of the embodiment of the application under six different light conditions of the portrait sketch image generation method based on GAN.

[0037] Figure 11 Fig. 2 is a schematic diagram of the module of the embodiment of the application of the portrait sketch image generation device based on GAN. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the application. Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the application.

[0039] The specific embodiments of the application will be described in detail below with reference to the drawings.

[0040] Referring to Figure 1 The first embodiment of the application provides a portrait sketch image generation method based on GAN, which can be executed by a portrait sketch image generation device based on GAN (hereinafter referred to as generation device), in particular, by one or more processors in the generation device to implement the following steps:

[0041] S101, obtaining an original image to be drawn;

[0042] S102, inputting the original image to a GAN-based face-to-sketch generator group; wherein the face-to-sketch generator group includes N image-to-sketch generators, taking the original image as an input parameter of the first image-to-sketch generator, taking the process image output by the first image-to-sketch generator and the original image as input parameters of the next image-to-sketch generator, and sequentially taking the original image as an input parameter of the last image-to-sketch generator to output a target sketch image corresponding to the original image.

[0043] Referring to Figures 2 to 5 Specifically, in the embodiment, before the original image is input to the GAN-based face-to-sketch generator group, the method further includes:

[0044] Obtain multiple training images and a set of hand-drawn sketch images corresponding to each training image, wherein the set of sketch images includes N sketch images at different drawing stages;

[0045] For each training image and each sketch image, a corresponding face-to-sketch generator, sketch-to-face generator, and discriminator are established.

[0046] The sketch image is input into the sketch-to-face generator to generate a corresponding fake image;

[0047] The training image and the intermediate image are stitched together and then input into the face to sketch generator to generate a corresponding fake sketch image. The intermediate image is either a blank image or a fake image generated in the previous drawing stage.

[0048] The discriminator group is invoked to compare the fake image and the training image, as well as the fake sketch and the sketch image, to generate corresponding comparison scores;

[0049] The comparison score is compared with a preset score value. When the comparison score reaches the preset score value, a face-to-sketch generator group based on GAN is generated.

[0050] For example, with Figure 2 For example, to better represent details and textures, a painter's painting process involves many steps. To enable our model to create a clearly defined sketch, similar to the painter's, we selected three of the most representative step-by-step images from 25 pictures as our compositing targets. For example... Figure 3 As shown, we divide the face-to-sketch conversion into two stages. In the first stage, we input a face image and real images of the three key steps of sketching to obtain relevant sketch information. In the second stage, we input the original image and synthesize the target sketch image with the help of the sketch information obtained in the first stage.

[0051] In this process, the discriminator and generator are adversarial at each step. The generator aims to produce sketches that resemble real images, while the discriminator judges the similarity between the generated image and the original image. The main difference between this model and traditional GAN ​​models is that the images obtained from the generative network can be reused to train the generative model. The internal structure of the first stage is as follows: Figure 4 As shown on the left, each step involves a pair of generators: a face-to-sketch generator G_AB and a sketch-to-face generator G_BA. These need to be converted at each step. Our goal in the first stage is to obtain the face-to-sketch generator G_AB, which extracts the most information, and then proceed to the second stage. The fake face images obtained from the sketch-to-face generator G_BA in the previous two steps are passed to the next stage along with the original image. Figure 1The first stage trains the face-to-sketch generator G_AB. The internal structure of the second stage is as follows: Figure 4 As shown on the right, a fake sketch is generated using the face-to-sketch generator G_AB, and then transferred to the next step to generate a new face sketch together with the real face. This process generates three different sketches in sequence, and finally the target image is obtained.

[0052] For example, with Figure 5 Taking the first step of the first stage as an example, each step requires two generators and two discriminators. First, the original image 'a' with a blank image is input into the first face-to-sketch generator G1_AB to generate the first fake sketch image 'fake_G1_B'. Then, this fake sketch image and the original image 'b1' are fed into the discriminator D1_B for scoring. Next, using cycle consistency, the fake_G1_B with a blank image is input into the first sketch-to-face generator G1_BA to generate the reconstructed fake face image 'fake_c1_A'. Then, it is compared with the original image... Figure 1 The data is fed into the discriminator D1_A for scoring. Figure 5 The method in the second half is similar to the above. The original sketch image b1 is converted into a fake face image fake_G1_A, and then into a reconstructed fake sketch image fake_c1_B. The generated fake_G1_A is sent to D1_A for discrimination, and the generated fake_c1_B is sent to D1_B for discrimination. Each step of the generated sketch image is compared with the original image, which strengthens the connection between the two images in terms of content and enhances the identity information to a certain extent. The results generated in each stage are passed to the next stage, which strengthens the continuity between different steps. The results of each step are output, which facilitates the adjustment of different details. For example, if the facial features are poorly generated, the weight of the first stage can be increased to enhance the details of the generated facial features.

[0053] The first stage, steps two and three have the same structure as the first stage. The difference lies in the input: in step two, when converting the real face image 'a' into the fake sketch image 'fake_G2_B', the blank input image becomes 'fake_G1_A' from the previous stage; and in step three, when converting the real face image 'a' into the fake sketch image 'fake_G3_B', the blank input image becomes 'fake_G2_A' from the previous stage. The second stage has a similar structure to the first stage, except that in step two, when converting the real face image 'a' into the fake sketch image 'fake_G2_B', the blank input image becomes 'fake_G1_B' from the previous stage; and in step three, when converting the real face image 'a' into the fake sketch image 'fake_G3_B', the blank input image becomes 'fake_G2_B' from the previous stage.

[0054] Please see Figures 6 to 7 The number above represents the number of output channels, as described above. Figure 2For example, a total of 6 generators and 6 discriminators are used, all of which have the same structure. The generator of the GAN can be regarded as a symmetrical encoder-decoder network, which is composed of standard convolution, residual block, inverse convolution, up-sampling and down-sampling modules. In the generator, the last convolution layer with a 1*1 convolution kernel does not use a normalization layer, followed by a tanh nonlinear activation function. PatchGAN is used as the discriminator. In some experiments, it is shown that the GAN network performs poorly in the field of processing high-resolution and high-detail images. To solve this problem, PatchGAN is used. The main difference between the two is the discriminator; the traditional GAN discriminator finally outputs a vector, which only needs to output a true or false representing the evaluation of the entire image, but the PatchGAN discriminator discriminates true or false for each patch, and the output result is an N*N matrix, each element has only two choices 1 or 0, and the average value is calculated as the final discriminator output result. In this embodiment, the size of the output matrix is designed to be 15*15, and the use of 15*15 Patch can alleviate the tiling artifact, and fewer parameters are required. Compared with other PatchGAN models of different sizes, better results in visual quality and efficiency are obtained. It should be noted that in other embodiments, other types of generators can also be used, which are not limited here, but these schemes are within the scope of the present application.

[0055] Please refer to Figure 8 , specifically, in this embodiment, the loss function of each stage of the network learns the network parameters of each stage of sketch generation by minimizing the objective function; the discriminator calculates the adversarial loss of the false sketch and the sketch image according to the formula L ad1 (G A ,G B )=E b [logD B (b)]+E a [log(1-D B (G A (a)))];the adversarial loss of the false image and the training image is calculated according to the formula L ad2 (G A ,G B )=E a [logD A (a)]+E b [log(1-D A (G B (b)))],wherein G A is a face-to-sketch generator, G A (a) is the false sketch, G B is a sketch-to-face generator, and G B(b) is the fake image, D A For A-domain face discriminator, D B For B-domain sketch discriminator, L ad For adversarial loss, our goal is to achieve the transformation between different domains under the adversarial relationship of the four: domain A to domain B and domain B to domain A. The generator tries to minimize this loss, while the discriminator tries to maximize it.

[0056] The discriminator is based on formula L cyc (G A G B ) = E a [G B (G A (a))-a||1]+E B [G A (G B (b))-b||1] calculates the cycle consistency loss for the fake sketch and the sketch image, as well as the fake image and the training image, wherein the L cyc This is the cycle consistency loss. The supervised cycle consistency loss assumes that the generated image can always be transformed back to the original domain. Therefore, for a face image in domain A, our goal is to make A→G... A (A)→G B (G A (A))≈A; Similarly, for the sketch image of domain B, our goal is to make B→G B (B)→G A (G B (B))≈B, the cycle consistency determination uses pixel-level loss to judge the similarity between two images.

[0057] When real samples are provided as input from the target domain, pixel-level consistency is introduced between the input image and the generated image to constrain the generator to approximate the identity mapping. The discriminator is based on formula L. im (G A G B ) = E a [(G A (a))-a||1]+E B [(G B (b))-b||1] calculate the identity mapping loss for the fake sketch and the sketch image, as well as the fake image and the training image, wherein the L im This is the loss due to identity mapping.

[0058] Each fake sketch face must be generated under the identity constraints of the original image, and each time a sketch is converted into a real face, it must also be constrained by the identity information of the original image. The discriminator is based on formula L. A_id(G A1 G A2 G A3 ) = E a [(G A1 (a))-a||1]+E a,b1 [(G A2 (a),b1)-a||1]+E a,b2 [G B (G A2 (G A1 (a),b1),b2)-a||1] calculates the loss of identity information for the fake sketch and the sketch image, according to formula L B_id (G B1 G B2 G B3 ) = E a,b1 [(G B1 (b1))-a||1]+E a,b2 [G B2 (b2)-a||1]+E a,b3 [G B3 (b3)-a||1] calculates the identity information loss for the fake image and the training image, wherein the L A_id and the L B_id This is due to the loss of identity information.

[0059] Specifically, in this embodiment, we express this loss as follows: where λ1, λ2, λ3, and λ4 control the relative importance of four different losses, and the final optimization objective can be represented as a minima problem. The loss function of the discriminator is given by formula L. total =λ1*(L ad1 +L ad2 )+λ2*L cyc +λ3*L im +λ4*(L A_id +L B_id ), where λ1, λ2, λ3, and λ4 represent the relative importance of adversarial loss, cycle consistency loss, identity mapping loss, and identity information loss, respectively. It should be noted that other types of loss functions may be used in other embodiments; no specific limitations are made here, but all such schemes are within the scope of protection of this invention.

[0060] Please see Figures 9 to 10In particular, in the embodiment, the portrait sketch image generation method based on GAN is used on the built platform, and the CUHK student ID and the CUFS foreign face dataset are used for experiments, and excellent effects are shown. The portrait sketch image generation method based on GAN uses the GAN network to generate the sketch image of face drawing, and uses the step-by-step processing mode to first draw the outline of the five organs, then gradually increase more details, and finally draw the portrait sketch. The one-step mapping between different domains of the image is cancelled, and it is converted into two stages. The first stage uses the reconstructed image to assist in generating sketch information, and the second stage uses the sketch information obtained in the first stage to assist in synthesizing the target sketch image. Among them, the first stage can have different structures, even if the auxiliary information obtained by different methods in the stage is different, but the auxiliary information obtained by using the algorithm solving idea can still be put into the second stage. The method can not only be used in face sketch synthesis, but also can provide ideas for other step-by-step cases, such as picture coloring, map drawing, house design and the like.

[0061] Please refer to Figure 11 The second embodiment of the present application provides a portrait sketch image generation device based on GAN, comprising:

[0062] The image acquisition unit 201 is used for acquiring an original image to be drawn.

[0063] The image conversion unit 202 is used for inputting the original image into the face-to-sketch generator group based on GAN; wherein the face-to-sketch generator group comprises N image-to-sketch generators, the original image is used as the input parameter of the first image-to-sketch generator, the process image output by the first image-to-sketch generator and the original image are used as the input parameters of the next image-to-sketch generator, and the process image output by the last image-to-sketch generator is used as the input parameter of the last image-to-sketch generator.

[0064] The third embodiment of the present application provides a portrait sketch image generation device based on GAN, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to realize the portrait sketch image generation method based on GAN according to any one of the above.

[0065] The fourth embodiment of the present application provides a readable storage medium, which stores a computer program, and the computer program can be executed by the processor of the device to realize the portrait sketch image generation method based on GAN according to any one of the above.

[0066] Exemplarily, the computer program in the third and fourth embodiments of the present application can be divided into one or more modules stored in the memory and executed by the processor to complete the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the implementation of the present application. For example, the device in the second embodiment of the present application.

[0067] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The processor is the control center of the present application, and is connected with all parts of the present application through various interfaces and lines.

[0068] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the present application by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, a text conversion function, etc.), etc.; and the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0069] The implemented modules, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the contents included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0070] It should be noted that the above-described device embodiments are only schematic, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be realized as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0071] The above is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solution falling within the idea of the present application belongs to the protection scope of the present application.

Claims

1. A method for generating a portrait sketch image based on a GAN, characterized in that, The method comprises the following steps: obtaining an original image to be drawn; inputting the original image into a GAN-based face-to-sketch generator set; wherein the face-to-sketch generator set comprises N image-to-sketch generators, the original image is taken as an input parameter of a first image-to-sketch generator, a process image output by the first image-to-sketch generator and the original image are taken as input parameters of a next image-to-sketch generator, and a target sketch image corresponding to the original image is output by a last image-to-sketch generator in sequence; before inputting the original image into the GAN-based face-to-sketch generator set, the method further comprises the following steps: obtaining a plurality of training images and a set of hand-drawn sketch images corresponding to each training image, wherein the set of sketch images comprises N sketch images at different drawing stages; for each training image and each sketch image, establishing a face-to-sketch generator, a sketch-to-face generator and a discriminator corresponding thereto; inputting the sketch image into the sketch-to-face generator to generate a corresponding fake image; inputting the training image and an intermediate image into the face-to-sketch generator after splicing to generate a corresponding fake sketch, wherein the intermediate image is a blank image or a fake image generated in a previous drawing stage; calling the discriminator set to compare the fake image with the training image and the fake sketch with the sketch image to generate a corresponding comparison score; comparing the comparison score with a preset score value, and generating a GAN-based face-to-sketch generator set when the comparison score reaches the preset score value; The loss function of the discriminator is formula wherein are the relative importance of the adversarial loss, the relative importance of the cycle consistency loss, the relative importance of the identity mapping loss, and the relative importance of the identity information loss, respectively, is the adversarial loss, is the cycle consistency loss, is the identity mapping loss, and is the identity information loss. 2.The GAN-based portrait sketch image generation method of claim 1, wherein, The discriminator is according to the formula The calculation of the adversarial loss of the false sketch and the sketch image is carried out; according to the formula The calculation of the adversarial loss of the false image and the training image is carried out, wherein, For a face-to-sketch generator, For the false sketch, For a sketch-to-face generator, For the false image, For an A-domain face discriminator, For a B-domain sketch discriminator. 3.The GAN-based portrait sketch image generation method of claim 2, wherein, The discriminator is according to the formula The false sketch image and the sketch image, and the false image and the training image are subjected to cycle consistency loss calculation.

4. The GAN-based portrait sketch image generation method of claim 3, wherein, The discriminator is according to the formula Identity mapping loss is calculated for the fake sketch and the sketch image, and the fake image and the training image.

5. The GAN-based portrait sketch image generation method of claim 4, wherein, The discriminator is according to the formula The calculation of the identity information loss of the fake sketch and the sketch image is according to the formula The calculation of the identity information loss of the fake image and the training image. 6.A GAN-based portrait sketch image generation device, characterized by comprising: The method comprises the following steps: an image acquisition unit is configured to obtain an original image to be drawn; an image conversion unit is configured to input the original image into a GAN-based face-to-sketch generator set; wherein the face-to-sketch generator set comprises N image-to-sketch generators, the original image is taken as an input parameter of a first image-to-sketch generator, a process image output by the first image-to-sketch generator and the original image are taken as input parameters of a next image-to-sketch generator, and a target sketch image corresponding to the original image is output by a last image-to-sketch generator in sequence; before inputting the original image into the GAN-based face-to-sketch generator set, the method further comprises the following steps: obtaining a plurality of training images and a set of hand-drawn sketch images corresponding to each training image, wherein the set of sketch images comprises N sketch images at different drawing stages; for each training image and each sketch image, establishing a face-to-sketch generator, a sketch-to-face generator and a discriminator corresponding thereto; inputting the sketch image into the sketch-to-face generator to generate a corresponding fake image; inputting the training image and an intermediate image into the face-to-sketch generator after splicing to generate a corresponding fake sketch, wherein the intermediate image is a blank image or a fake image generated in a previous drawing stage; calling the discriminator set to compare the fake image with the training image and the fake sketch with the sketch image to generate a corresponding comparison score; comparing the comparison score with a preset score value, and generating a GAN-based face-to-sketch generator set when the comparison score reaches the preset score value; The loss function of the discriminator is formula wherein are the relative importance of the adversarial loss, the relative importance of the cycle consistency loss, the relative importance of the identity mapping loss, and the relative importance of the identity information loss, respectively, is the adversarial loss, is the cycle consistency loss, is the identity mapping loss, and is the identity information loss.

7. A GAN-based portrait sketch image generation device, characterized by, The application discloses a GAN-based portrait sketch image generation method and device, a computer readable storage medium and a computer program product.

8. A readable storage medium, characterized by, The application discloses a GAN-based portrait sketch image generation method and device, a computer readable storage medium and a computer program product.

Images