Image automatic generation method and system based on hand-drawn sketch

Abstract

The application discloses a hand-drawing sketch-based image automatic generation method and system, and relates to the technical field of computer vision image processing. The method comprises the following steps: obtaining a hand-drawing sketch and a color image corresponding to the sketch, and constructing a training set and a test set; constructing an image automatic generation model, and performing adversarial training on the image automatic generation model by using the training set data; and inputting the test set into the trained image automatic generation model to generate a corresponding color image. In the image automatic generation model, the shape discriminator pays attention to the shape of the generated image, and the shape of the generated image is constrained, so that the shape of the generated image is rationalized, and problems such as object deformation and line distortion in the generated image are solved in a targeted manner; and the image global attention module pays attention to the whole of the generated image, so that the overall color, texture and the like of the generated image are adapted to the image shape, and the problems of color and texture distortion are solved, and the quality of the generated image is improved by the combination of the two.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision image processing, in particular to a method and system for automatically generating images based on hand-drawn sketches. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Drawing a sketch is one of the most natural and flexible ways for humans to express and convey information. In recent years, with the increasing popularity of touch screen devices such as smartphones and tablets, drawing a sketch containing only simple lines on a touch screen to express human visual needs has become one of the important ways of human-computer interaction. With the rapid development of deep learning in the field of computer vision and image processing, a series of technical applications in the field of sketch, such as sketch completion, sketch editing, object recognition based on sketch, and object retrieval based on sketch, have developed rapidly. Image generation task is a popular research topic in computer vision in recent years, and sketch-image translation is one of the image generation tasks. Combined with the user interface, the user can draw a sketch in the sketch drawing area or import a sketch from a local file, and then quickly create an image that aligns with the object category, shape, etc. of the sketch in the image generation area. Due to the highly abstract nature of sketches, and the fact that sketches drawn by different users may differ greatly in accuracy and style, even sketches of the same object drawn by different users may differ greatly and express different specific visual needs, the sketch-image translation problem still faces great challenges.

[0004] The inventors have found that existing methods and systems for automatically converting sketches into images have image generation quality problems: the generated image shape differs greatly from the shape of the input sketch, does not align with the input sketch, and even the generated image is not the same object as the input sketch; the color and texture of the generated image are relatively blurred and inaccurate, and there are cases of generating incorrect color and texture, i.e., the color and texture do not match the input sketch, such as inputting a zebra sketch but generating an image with color and texture similar to an elephant. In addition to the image quality problem, existing methods and systems for automatically converting sketches into images have complex operation steps, low efficiency, and some methods are only suitable for generating images of a certain type of object, and have low generalization ability. If the above problems in automatically converting sketches into images can be solved, the quality, accuracy, and generation efficiency of automatically generating images from hand-drawn sketches will be significantly improved, thereby improving the efficiency of drawing artistic paintings or design drawings, reducing the creation threshold, and making the technology of automatically converting sketches into images more widely applicable, which has higher practical significance and practical value. SUMMARY

[0005] In order to solve the problems in the prior art, the application provides a hand-drawn sketch-based image automatic generation method and system, which generates a single-body image similar to a photo based on a hand-drawn sketch containing only line contour information, adopts an edge extraction module and a shape discriminator module to rationalize the shape of the generated image, and introduces image global attention to focus on the shape contour and color and texture of the object, thereby improving the quality of the generated image and the image generation efficiency.

[0006] In a first aspect, the application provides a hand-drawn sketch-based image automatic generation method.

[0007] The hand-drawn sketch-based image automatic generation method comprises the following steps.

[0008] Obtaining a hand-drawn sketch and a color image corresponding to the sketch, and constructing a training set and a test set;

[0009] Constructing an image automatic generation model and performing adversarial training on the image automatic generation model using the training set data;

[0010] Inputting the test set into the trained image automatic generation model to generate a corresponding color image;

[0011] The image automatic generation model comprises an image generation network based on adversarial learning and an image shape optimization network based on adversarial learning; the image generation network is used to convert the hand-drawn sketch into an image containing a target contour and corresponding color; the image shape optimization stage is used to convert an image with unreasonable shape into a real image with reasonable shape, thereby generating an image corresponding to the hand-drawn sketch.

[0012] Further, the image generation network based on adversarial learning comprises a sketch generator, an image generator, a sketch discriminator and an image discriminator. Further, the image generator introduces an image global attention module to focus on the feature learning of the whole image; the image discriminator calculates the loss between the generated image and the input image and reduces the value thereof, aiming to improve the quality of the generated image.

[0013] Further, the image shape optimization network based on adversarial learning comprises an edge extractor and a shape discriminator.

[0014] Further, the edge extractor is used to extract an edge map of the color image; the shape discriminator inputs the edge map extracted by the edge extractor and a reconstructed sketch generated by the sketch generator; the shape of the generated image is constrained by calculating the loss between the reconstructed sketch and the corresponding edge map, aiming to reduce the shape loss to solve the problem of shape deformation of the generated image.

[0015] Further, the adversarial training is specifically: inputting the training set into the constructed generative adversarial network for training, the generator generating a predicted image; the discriminator discriminating the predicted image and the real image, and calculating a total loss function value; when the loss function stops decreasing, the generator and the discriminator reach a certain balance state, the training is stopped, and a trained image automatic generation model is obtained.

[0016] Further, in the whole training process, the loss function includes: an adversarial loss function, a cycle consistency loss function and a shape loss function.

[0017] In a second aspect, the present application provides a hand-drawn sketch-based image automatic generation system.

[0018] The hand-drawn sketch-based image automatic generation system comprises:

[0019] An acquisition module configured to acquire a hand-drawn sketch and a color image corresponding to the sketch, and construct a training set and a test set;

[0020] A training module configured to construct an image automatic generation model, and perform adversarial training on the image automatic generation model using the training set data;

[0021] A generation module configured to input the test set into the trained image automatic generation model to generate a corresponding color image;

[0022] The image automatic generation model comprises an image generation network based on adversarial learning and an image shape optimization network based on adversarial learning; the image generation network is used to convert the hand-drawn sketch into an image containing a target contour and a corresponding color; and the image shape optimization stage is used to convert an image with unreasonable shape into a reasonable and real image, to generate an image corresponding to the hand-drawn sketch.

[0023] In a third aspect, the present application further provides an electronic device comprising one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method of the first aspect.

[0024] In a fourth aspect, the present application further provides a computer readable storage medium for storing computer instructions, which are executed by a processor to complete the method of the first aspect.

[0025] Compared with the prior art, the present application has the following beneficial effects:

[0026] (1) The application generates images by taking the focal frequency loss as an image loss constraint of a non-conditional term, so that the generator pays more attention to generating subtle parts of the image, effectively improving the quality of the generated image.

[0027] (2) The application ingeniously designs an edge extraction module, a shape discriminator module and an image global attention module; the edge extraction module extracts an edge map of the image as input of the shape discriminator module; the shape discriminator module takes the reconstructed sketch and the edge map as input, focuses on the shape of the generated image, and constrains the shape of the generated image to make the shape of the generated image reasonable; the image global attention module focuses on the whole of the generated image, so that the overall color and texture of the generated image adapt to the image shape, avoiding the situation that the shape does not match the color and texture. The shape discriminator specifically solves the problems of object deformation and line distortion in the generated image, and the image global attention module solves the problems of color and texture distortion, both of which further improve the quality of the generated image.

[0028] (3) The application adopts an end-to-end generation strategy, and generates an image corresponding to a hand-drawn sketch by using a trained image generation model. The model adopts an unsupervised learning manner, and only needs a data set belonging to the same category, without a paired training data set, thereby reducing the production cost of the data set. In addition, the model training steps are simple, the generation system is easy to operate, time cost is saved, and image generation efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0029] The drawings accompanying the specification of this application form a part thereof, serve to further provide a further understanding of the application, and the illustrative embodiments of the application and their description serve to explain the application, but do not constitute improper limitations on the application.

[0030] Figure 1 The figure is a schematic diagram of the image automatic generation method based on the hand-drawn sketch of the embodiment of the application.

[0031] Figure 2 The figure is a flowchart of the image automatic generation method based on the hand-drawn sketch of the embodiment of the application.

[0032] Figure 3 The figure is a network structure diagram of the image automatic generation method based on the hand-drawn sketch of the embodiment of the application.

[0033] Figure 4 The figure is a network structure diagram of the image global attention module used in the embodiment of the application.

[0034] Figure 5 The figure is a comparison diagram of qualitative results of the method of the embodiment of the application and the existing method on a shoe data set.

[0035] Figure 6A comparison chart of the qualitative results of the method of the embodiment of the present application and the prior method on the giraffe data set. DETAILED DESCRIPTION

[0036] Embodiment one

[0037] As shown in the drawings, the embodiment provides a hand-drawn sketch-based image automatic generation method, which can generate a single body color image similar to a photo based on a hand-drawn sketch containing only line contour information. Figure 1

[0038] Referring to Figure 2 , the hand-drawn sketch-based image automatic generation method comprises the following steps:

[0039] Step S1: Obtain a hand-drawn sketch and a color image corresponding to the sketch, and construct a training set and a test set;

[0040] Step S2: Construct an image automatic generation model, and use the training set data to adversarially train the image automatic generation model;

[0041] Step S3: Input the test set into the trained image automatic generation model to generate a corresponding color image;

[0042] Wherein, the image automatic generation model comprises: an image generation network based on adversarial learning and an image shape optimization network based on adversarial learning; the image generation network is used to convert the hand-drawn sketch into an image containing a target contour and a corresponding color; the image shape optimization stage is used to convert the image with unreasonable shape into a reasonable shape and real image, and generate an image corresponding to the hand-drawn sketch.

[0043] In step S1, the hand-drawn sketch includes: basic line contour designates the category, size and pose of the object, and has enough information to indicate what the sketch is.

[0044] By collecting hand-drawn sketches and image data sets, the sample data is preprocessed to determine the training set and the test set. The hand-drawn sketches in the data set only contain a number of basic lines, which outline the general contour of the object, and the image part is a high-definition color picture. The resolution of the sample data is 256x256.

[0045] Because the collected data set contains samples that do not meet the requirements, such as incomplete pictures containing only part of the object, the sample data is preprocessed in step 1 to obtain clear sketches and color images to construct the training set and the test set.

[0046] The training set and the test set are different, and the contents of the training set and the test set both contain sketches and color images, and the difference is that the two have no intersection, that is, the contained picture files are different. ​

[0047] In step S2, the constructed image automatic generation model comprises an image generation network based on adversarial learning and an image shape optimization network based on adversarial learning.

[0048] The network structure diagram of the image automatic generation method based on the hand-drawn sketch in this embodiment is shown in Figure 3

[0049] The image generation network based on adversarial learning comprises a generator and a shape discriminator.

[0050] The generator comprises a 7x7 Convolutional-Instance-Norm-ReLU layer with 32 filters and a stride of 1, a 3x3 Convolutional-Instance-Norm-ReLU layer with 64 filters and a stride of 1, a 3x3 Convolutional-Instance-Norm-ReLU layer with 128 filters and a stride of 2, nine residual blocks each comprising two 3x3 convolution layers, a 3x3 Convolutional-Instance-Norm-ReLU layer with 64 filters and a stride of 1, a 3x3 Convolutional-Instance-Norm-ReLU layer with 32 filters and a stride of 2, and a 7x7 Convolutional-Instance-Norm-ReLU layer with 32 filters and a stride of 1.

[0051] In addition, the generator part further comprises an image global attention module, which focuses on the feature learning of the whole image. The network structure diagram of the image global attention module is shown in Figure 4 .

[0052] ​The shape discriminator adopts a 70x70 PatchGAN structure, and the discriminator architecture is C64-C128-C256-C512, including: a 4x4 Convolutional-Instance-Norm-Leaky-ReLU layer with 64 filters and a stride of 2; a 4x4 Convolutional-Instance-Norm-Leaky-ReLU layer with 128 filters and a stride of 2; a 4x4 Convolutional-Instance-Norm-Leaky-ReLU layer with 256 filters and a stride of 2; and a 4x4 Convolutional-Instance-Norm-Leaky-ReLU layer with 512 filters and a stride of 2. The ReLU slope is 0.2, and we apply convolution to generate a 1D output in the last layer.

[0053] The image shape optimization network based on adversarial learning includes an edge extractor and a shape discriminator.

[0054] The edge extractor adopts the structure of TOM and is composed of three modules: a pre-trained VGG, a sketch generator, and a sketch discriminator.

[0055] The edge extractor is used to extract the edge map of a color image.

[0056] The shape discriminator also adopts a 70x70 PatchGAN structure, and its input is the edge map extracted by the edge extractor and the reconstructed sketch generated by the sketch generator. The discriminator mainly constrains the shape of the generated image by calculating the loss of the reconstructed sketch and the corresponding edge map, aiming to reduce the shape loss to solve the problem of image deformation.

[0057] The shape discriminator determines the true or false of the reconstructed sketch and the edge map, wherein the reconstructed sketch is an intermediate result generated in the image generation stage. As the model training proceeds, the shape discriminator calculates the loss between the reconstructed sketch and the edge map, and when the loss no longer decreases, the training ends.

[0058] The loss function used to train the image generation network includes an adversarial loss and a cycle consistency loss.

[0059] The adversarial loss formula is as follows:

[0060]

[0061] The cycle consistency loss formula is as follows:

[0062] L cyc (G, F, s, i) = L FFL(s, F(G(s, i)) + L FFL (i, G(F(i, s))) (2)

[0063] In formula (1) and (2), G is an image generator, F is a sketch generator, D I is an image discriminator; S and I represent a hand-drawn sketch and an image data domain respectively, and i and s represent a sample in S and I respectively.

[0064] In addition, the cycle consistency loss adopts a focal frequency loss calculation formula, L FFL represents a focal frequency formula, and the specific formula is as follows:

[0065]

[0066] In formula (3), (x, y) represents a frequency coordinate, m x n represents an image size, w(x, y) represents a frequency weight at point (x, y), F r (x, y) and F f (x, y) represent a real frequency and a generated frequency at point (x, y) respectively, r represents a real image, and f represents a false image.

[0067] The loss function used in the generated image shape optimization network includes a shape loss.

[0068] The shape loss is shown in formula (4):

[0069]

[0070] In formula (4), D shape represents a shape discriminator, E represents an edge map data domain, and e represents a sample in E, and the remaining symbols have the same meaning as in formula (1).

[0071] When training the image automatic generation model, the entire network model is trained end-to-end without being subdivided into multiple stages: the performance of the image generator is affected by the shape discriminator and the image discriminator, the generator and the discriminator interact and are trained in an adversarial manner, and finally reach a certain balance state, at which time the performance of the generator reaches the best. The specific training method includes:

[0072] Building a generative adversarial network: connecting the image generator, the image discriminator, the sketch generator, the sketch discriminator, and the shape discriminator to form a generative adversarial network;

[0073] Building a training set: the training set is a hand-drawn sketch and a known image corresponding to the sketch; the hand-drawn sketch contains information such as the category, size, and pose of the object specified by the basic line contour; and the known image is a high-definition image;

[0074] The training set is input into the constructed generative adversarial network for training: the generator generates a predicted image; the discriminator discriminates between the predicted image and the real image, and calculates the total loss function value; when the loss function stops decreasing, the generator and the discriminator reach a certain balance state, the training is stopped, and the trained image generator is obtained.

[0075] In step S3, after the whole model is trained, the hand-drawn sketch test data set is input into the generator network to generate a corresponding image, and the effectiveness of the detection model is detected.

[0076] Specific case:

[0077] Fig. and Figure 6 The image generation result comparison chart of the method in the embodiment and the existing method is given.

[0078] Figure 5 The qualitative result comparison chart of the method in the embodiment and the existing method on the shoe data set is given.

[0079] Figure 6 The qualitative result comparison chart of the method in the embodiment and the existing method on the giraffe data set is given. As shown in Figure 5 、 Figure 6 The image generated by the existing method has a big problem in quality: the generated image is deformed, the contour is distorted, and the shape is greatly different from the target object; the color, texture and the like are seriously distorted. The method of the embodiment solves these problems to a great extent, the shape of the generated image is not obviously deformed, the color, texture and the like are relatively realistic, and the generated image is basically consistent with the target image as a whole.

[0080] Embodiment two

[0081] The embodiment provides an image automatic generation system based on hand-drawn sketches.

[0082] The image automatic generation system based on hand-drawn sketches comprises:

[0083] An acquisition module configured to acquire a hand-drawn sketch and a color image corresponding to the sketch, and construct a training set and a test set;

[0084] A training module configured to construct an image automatic generation model, and perform adversarial training on the image automatic generation model using the training set data;

[0085] A generation module configured to input the test set into the trained image automatic generation model to generate a corresponding color image;

[0086] The image automatic generation model comprises an image generation network based on adversarial learning and an image shape optimization network based on adversarial learning; the image generation network is used for converting the hand-drawn sketch into an image containing a target contour and corresponding color; and the image shape optimization stage is used for converting an image with unreasonable shape into a reasonable and real image, thereby generating an image corresponding to the hand-drawn sketch.

[0087] Embodiment three

[0088] The embodiment also provides an electronic device, comprising one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, the one or more computer programs are stored in the memory, and the processor executes the one or more computer programs stored in the memory to enable the electronic device to execute the method in the above embodiment one.

[0089] Embodiment four

[0090] The embodiment also provides a computer readable storage medium for storing computer instructions, which are executed by a processor to complete the method in the above embodiment one.

[0091] The above merely provides preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An automatic image generation method based on hand-drawn sketches, characterized in that, include: Obtain hand-drawn sketches and their corresponding color images, and construct training and testing sets; Construct an image automatic generation model and use the training set data to perform adversarial training on the image automatic generation model; The test set is input into the trained image automatic generation model to generate the corresponding color image; The automatic image generation model includes: an image generation network based on adversarial learning and an image shape optimization network based on adversarial learning; the image generation network is used to convert hand-drawn sketches into images containing target outlines and corresponding colors; the image shape optimization stage is used to convert images with unreasonable shapes into images with reasonable and realistic shapes, generating images corresponding to hand-drawn sketches; The image generation network based on adversarial learning includes: a sketch generator, an image generator, a sketch discriminator, and an image discriminator; The image shape optimization network based on adversarial learning consists of an edge extractor and a shape discriminator. The edge extractor is used to extract the edge map of the color image; the input of the shape discriminator is the edge map extracted by the edge extractor and the reconstructed sketch generated by the sketch generator.

2. The automatic image generation method based on hand-drawn sketches as described in claim 1, characterized in that, The image generator incorporates a global attention module to focus on learning the overall features of the image; the image discriminator calculates the loss between the generated image and the input image and reduces its value, aiming to improve the quality of the generated image.

3. The automatic image generation method based on hand-drawn sketches as described in claim 1, characterized in that, The shape of the generated image is constrained by calculating the loss of the reconstructed sketch and the corresponding edge map, aiming to reduce shape loss and solve the problem of generated image deformation.

4. The automatic image generation method based on hand-drawn sketches as described in claim 1, characterized in that, The adversarial training described herein is as follows: the training set is input into the constructed generative adversarial network for training, the generator generates predicted images; the discriminator distinguishes between the predicted images and the real images and calculates the total loss function value; when the loss function stops decreasing, the generator and the discriminator reach a certain equilibrium state, training stops, and a trained automatic image generation model is obtained.

5. The automatic image generation method based on hand-drawn sketches as described in claim 1, characterized in that, Throughout the training process, the loss functions include: adversarial loss function, cycle consistency loss function, and shape loss function.

6. An automatic image generation system based on hand-drawn sketches, characterized in that, include: The acquisition module is configured to: acquire hand-drawn sketches and their corresponding color images, and construct training and testing sets; The training module is configured to: build an image auto-generation model and perform adversarial training on the image auto-generation model using training set data; The generation module is configured to: input the test set into the trained image automatic generation model and generate the corresponding color image; The automatic image generation model includes: an image generation network based on adversarial learning and an image shape optimization network based on adversarial learning; the image generation network is used to convert hand-drawn sketches into images containing target outlines and corresponding colors; the image shape optimization stage is used to convert images with unreasonable shapes into images with reasonable and realistic shapes, generating images corresponding to hand-drawn sketches; The image generation network based on adversarial learning includes: a sketch generator, an image generator, a sketch discriminator, and an image discriminator; The image shape optimization network based on adversarial learning consists of an edge extractor and a shape discriminator. The edge extractor is used to extract the edge map of the color image; the input of the shape discriminator is the edge map extracted by the edge extractor and the reconstructed sketch generated by the sketch generator.

7. An electronic device, characterized in that, include: One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-5.

Images