Method and device for automatically generating stroke drawing by using sketch

Abstract

The invention discloses a method and a device for automatically generating a stroke drawing by using a sketch. The method comprises the following steps of: firstly, training a real image stroke information prediction module Transform-S module and a stroke global information and sketch information fusion module Transform-M module in a model; inputting the sketch into a pre-trained real image generation model Maskit to obtain a real image, obtaining image features of the real image corresponding to the sketch through an image encoder, obtaining canvas features of the blank canvas through the image encoder, and fusing the two features in a channel dimension to obtain fused image features; meanwhile, the sketch is input into a sketch encoder to obtain corresponding sketch space features; the fused image features are input into an encoder of the trained Transform-S module, and further fused image features are obtained; then, the fused image features are input into a decoder of a Transform-S module, and stroke information features are obtained; the stroke information features and the sketch space features are connected together and input into a trained Transform-M module, the stroke information features fused with sketch space information are obtained, parameters of strokes are obtained through a full connection layer, the strokes are obtained through rendering of the stroke parameters, and the drawing based on the stroke style is generated through superposition of the multiple strokes.

Description

Technical Field

[0001] The present invention relates to the field of image generation methods, and in particular to a method and device for automatically generating stroke paintings using sketches. Background Art

[0002] Painting has always been a wonderful way to express and record people's ideas. It can make the abstract concepts hidden in people's minds concretely expressed as pictures that can be shared with others and express emotions. However, painting has always been regarded as an art with relatively high technical requirements. It is not easy for a person with no painting experience to draw beautiful scenery and accurate expressions. Compared with sketches, ink painting, oil painting and other painting methods require painters to have higher proficiency. When painting these paintings, painters must not only consider the structural distribution of the entire picture, but also need to look at the application of color in each picture. Sketches only need to think about the structural construction of the picture, and even so, the required proficiency is not low. Nowadays, computer assistance has greatly reduced the demand for proficiency in painting. Artificial intelligence technology has been applied to painting, which greatly reduces the time spent on painting and can generate vivid color paintings from sketches.

[0003] Sketches can be converted into paintings through artificial intelligence technologies such as image transfer and image translation. The former mainly uses existing sketch and painting sample pairs to train deep networks, so that the network learns how to change the image from sketch style to painting style while maintaining the image spatial information and semantic information as much as possible; the latter explores the difference between the image domains of sketches and paintings, so that the network learns the path of conversion from sketches to paintings in the latent code representation. However, these methods are all pixel-based conversions of images, and their painting process does not have the general understanding of people's painting process.

[0004] Stroke painting is based on adding a few strokes to the canvas at each step, and finally these strokes are superimposed to form a painting, rather than directly obtaining the final painting from pixels after transformations that do not conform to ordinary people's thinking. The painting process based on strokes is more in line with the painting thinking process of ordinary people, and the painting with a brush feeling is more in line with people's expectations of painting.

[0005] With the development of Transformer neural network, it has been extended from the original application of one-dimensional sequence conversion or generation tasks such as text to two-dimensional data conversion or generation tasks such as images. When applying Transformer in image translation tasks, the loss of style information and spatial content information is controlled between input and output to achieve a high similarity between the output image and the image to be converted in terms of spatial content, and a high similarity with the style image in terms of style. However, most of these methods have a single input, and they focus on the mapping of image domains in terms of style and spatial content. In the process of converting sketches to color images, if only the sketch is used as the control information of spatial content, information such as color will be missing, making the generated image uncontrollable in color. Summary of the invention

[0006] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a method and device for automatically generating stroke drawings using sketches.

[0007] To solve the above technical problems, the present invention adopts the following steps:

[0008] A first aspect of the present invention relates to a method for automatically generating a stroke painting using a sketch, comprising the following steps:

[0009] Step 1: First, train the Transformer-S module, which is a real image stroke information prediction module, and the Transformer-M module, which is a stroke global information and sketch information fusion module, in the model, so that the parameters in the modules are fixed for sketch-to-stroke-based painting generation;

[0010] Step 2: Input the sketch into the pre-trained real image generation model Maskgit to obtain the real image. Pass the image encoder to obtain the image features of the real image corresponding to the sketch. Pass the blank canvas through the image encoder to obtain the canvas features. The features of the two are fused in the channel dimension to obtain the fused image features. At the same time, input the sketch into the sketch encoder to obtain the corresponding sketch space features.

[0011] Step 3: Input the fused image features into the encoder of the trained Transformer-S module to obtain further fused image features; then input the fused image features into the decoder of the Transformer-S module to obtain stroke information features;

[0012] Step 4: Connect the stroke information features and sketch space features and input them into the trained Transformer-M module to obtain the stroke information features that integrate the sketch space information. After passing through the fully connected layer, the stroke parameters are obtained. The strokes are rendered using the stroke parameters, and a painting based on the stroke style is generated by superimposing multiple strokes.

[0013] A second aspect of the present invention relates to an apparatus for automatically generating stroke drawings using sketches, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method of automatically generating stroke drawings using sketches of the present invention.

[0014] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method of the present invention for automatically generating a stroke painting using a sketch.

[0015] The present invention uses the corresponding real image features of the sketch and the spatial structure features of the sketch itself to propose a generation model for sketch-based painting. The model combines the advantages of the convolutional neural network that can extract image features with the self-attention mechanism and cross-attention mechanism of the Transformer network to fuse various information features. For the sketch image, the real image of the sketch is obtained by using the pre-trained real image generation model Maskgit, and the color distribution of the sketch is completed. At the same time, the attention mechanism of the Transformer is introduced. In the real image stroke information prediction module Transformer-S module, further fusion information of the real image features is obtained through the encoder. In the decoder part, the stroke information features are fused by querying the image information features to obtain the stroke information features whose stroke distribution shape is closer to the real image; in the stroke global information and sketch information fusion module Transformer-M module, only the encoder is used to fuse the stroke information features with the sketch space features, so that the generated stroke-based painting has a high consistency in spatial structure with the input sketch. The present invention can generate more artistic stroke-based paintings, which have a high consistency with the input sketch in terms of painting content and spatial structure, and the generated paintings are diverse.

[0016] The advantages of the present invention are: 1. Compared with pixel-based painting generation, the painting generation of the present invention is based on strokes, which is more in line with the painting thinking process of ordinary people in terms of the logic of the painting process, and can generate the painting process, which is more convenient for teaching painting. 2. The painting generation results of the present invention are diverse. The generation of a sketch can generate a variety of painting results that are close to the sketch in spatial structure and similar to the real image corresponding to the sketch in painting content. 3. Compared with other sketch-to-painting image conversion models, the present invention uses Maskgit to provide corresponding supplements for the lack of color information in the sketch, and can generate results with richer details and content closer to the sketch. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1It is a flowchart of the sketch generation based on stroke painting of the present invention.

[0018] Figure 2 It is a structural diagram of the Transformer-S module of the present invention.

[0019] Figure 3 It is a structural diagram of the Transformer-M module of the present invention.

[0020] Figure 4 It is a schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0022] Example 1

[0023] This embodiment relates to a method for automatically generating stroke paintings using sketches, which combines the advantages of CNN feature extraction with the attention mechanism of Transformer. Its overall architecture is as follows Figure 1As shown in the figure, it is mainly divided into four steps: Step 1, use the random stroke dataset to train the real image stroke information prediction module Transformer-S module, and then use the sketch dataset to train the real image stroke information prediction module Tranformer-S and the stroke global information and sketch information fusion module Transformer-M module, and determine the parameters in the two modules; Step 2, use the pre-trained sketch to generate the real image model Maskgit, which provides corresponding supplement for the lack of color information in the sketch, and obtains the real image corresponding to the sketch. At the same time, the sketch encoder and the image encoder extract the sketch and the real image respectively to obtain the corresponding features and perform feature fusion; Step 3, according to the fused image features obtained in step 2, input them into the Transformer-S module, which predicts the module of the corresponding painting strokes through image features, thereby generating stroke information that can constitute the painting with similar overall spatial structure and roughly the same color information distribution. The Transformer in this part uses the encoder part of the Transformer and the decoder part of the Transformer. The encoder is used to fuse the features of the real image, and the decoder is used to fuse the real image features and the empty stroke information, and finally obtain the stroke information that fuses the real image information; Step 4, according to the stroke information obtained in step 3, input it into the Transformer-M module, which fuses the stroke information and the sketch features to obtain the stroke information that constitutes the painting, including the stroke parameters and the generated stroke confidence. In this module, only the encoder part of the Transformer is used, relying on the self-attention mechanism to fuse the stroke information and the sketch features, and finally outputs the stroke parameters and the generated stroke confidence through the fully connected layer. After obtaining the stroke parameters, the strokes are rendered and the strokes with confidence values ​​lower than a certain value are discarded. A stroke-based painting is finally formed by superimposing multiple levels of strokes.

[0024] Step 1: The training of the real image stroke information prediction module Transformer-S and the stroke global information and sketch information fusion module Transformer-M requires two steps. The Transformer-S module needs to be trained separately first, and then the Transformer-S and Transformer-M modules are trained together. The specific training method includes the following steps:

[0025] Step 1.1: Input the sketch into the pre-trained real image generation model Maskgit to obtain the real image X corresponding to the sketch, and then input the obtained real image X into the image encoder E to downsample it to 1 / 4 of the original size to obtain the feature map Z containing the features of the image. xAt the same time, the canvas C with some strokes is input into the image encoder E and downsampled to 1 / 4 of the original size, and the feature map Z containing the features of the canvas with some strokes is obtained. c ; Concatenate the two features in the channel dimension to obtain the fused image feature Z;

[0026] Step 1.2: Input the existing fused image features Z into the encoder module of Transformer-S, use the self-attention mechanism of the encoder part of the module to extract the fused image features, input the obtained fused image features into the decoder of Transformer-S as the key and value in the cross attention, let the stroke information features to be queried be fused with the image information features to obtain the stroke information features, the stroke information features are passed through the fully connected layer to obtain the stroke parameters, and the intermediate stroke S is rendered. t , calculate the loss between the generated strokes and the sample strokes, the stroke loss function is:

[0027] L S =L pixel +L param +L w +L decision (1)

[0028] Where L pixel =|S pixel -S′ pixel | represents the pixel loss between the generated stroke and the sample stroke, and the absolute value of the color difference between the generated stroke and the sample stroke; L param =|S param -S′ param | represents the L1 loss between the parameters of the generated strokes and the parameters of the sample strokes, specifically the absolute value of the parameters; L w =WLoss(S distribution , S′ distribution ) represents the distance between the distribution of eight strokes generated at a time and the distribution of eight strokes of the sample. Specifically, the stroke parameters (x, y, w, h, θ) represent the horizontal coordinate, vertical coordinate, width, height, and rotation angle of the stroke on the canvas, and the stroke distribution is calculated by the following formula:

[0029]

[0030] The resulting stroke distribution is S distribution =N(μ,∑).

[0031] L decision =BCELoss(S c , S′ c ) represents the binary cross entropy loss between the stroke confidence obtained by the model and the stroke confidence obtained by calculating the overlap of sample strokes using the traditional algorithm, specifically Ldecision =-λ·g v ·log(c u )-(1-g v )·log(1-c u ), where g v represents the confidence in the sample stroke, c u Represents the confidence of the generated strokes predicted by the Transformer-S module. S Loss Train the Transformer-S module on the random strokes dataset;

[0032] Step 1.3, then input the stroke information features into the Transformer-M module to fuse the spatial structure information of the sketch, and finally obtain the final stroke information features. The stroke information features are passed through the fully connected layer to obtain the stroke parameters, which are rendered to obtain the stroke S′ and superimposed to obtain the painting X′. In the Transformer-M module, the loss of generating the painting includes not only the stroke loss in step 1.2, but also the loss of the discriminator that calculates the spatial structure gap between the sketch and the generated painting. The loss function is:

[0033]

[0034] Dis is the discriminator, which is actually a convolutional network used to extract the spatial structural features of the image. M is the total number of layers of the convolutional network of the discriminator, and N is the number of features that need to be compared at the t layer. It represents the i-th feature of image X in the t-th layer of the convolutional network; the total loss function for generating a painting is the sum of the stroke loss function and the discriminator loss function:

[0035] L total =L S +L D (4)

[0036] By L total Loss for training Transformer-S and Transformer-M modules on the Sketch dataset.

[0037] Step 2: Input the sketch into the pre-trained real image generation model Maskgit, and use its ability to generate real images to obtain the real image. Pass the image encoder to obtain the image features of the real image corresponding to the sketch. Pass the blank canvas through the image encoder to obtain the canvas features, and then fuse the features of the two in the channel dimension to obtain the fused image features. After the real image passes through the image encoder E, the real image features are obtained. The canvas also passes through the image encoder E to obtain the canvas features Then the two image features are concatenated in the channel dimension to obtain the fused image feature

[0038] Step 3: In order to make the final generated painting have a sense of strokes and be able to show a continuous painting process, the present invention does not focus on pixels, but uses a method of predicting stroke information. The generated image is composed of a stack of strokes rendered with the predicted stroke information. The stroke information includes the parameters required for rendering the strokes and the confidence of the generated strokes.

[0039] In order to predict the independent rendering parameters of each stroke, the parameters of each detection box are independently predicted in target detection. It is necessary to query the fused features of the real image through each stroke information feature, and then obtain the corresponding stroke information feature. This corresponds to the query, key and attention mechanism of the Transformer model. It can not only make the real image features to be queried use the self-attention mechanism to self-fuse the real image features to be queried before fusion with the stroke information features, making the fused image features more clear and effective, but also meet the needs of each stroke information feature to be output. It is directly fused with the fused real image features in the Transformer decoder part to obtain each stroke information feature.

[0040] like Figure 2 As shown in the figure, after obtaining the spliced ​​image feature Z, the fused image feature Z is added with position encoding so that the tensor at each position has different characteristics. Then, the encoder part of the trained Transformer-S transforms the image features through the fully connected layers of the query, key and value transformation dimensions to obtain the Q, K, and V matrices. The self-fused image features are obtained through the self-attention mechanism:

[0041]

[0042] Then the image features are further fused through the fully connected layer again. It should be noted that the dimensional shape will not change before the input features and after the output features. This allows the encoder to perform more self-fusion of fused image features and further extract effective fused image features.

[0043] In the decoder part of Tranformer-S, the query Q matrix of the cross attention mechanism needs to be provided by the queried stroke information, while the key K matrix and value V matrix are provided by the fused image features. First, set 8 blank stroke information features Z S ∈R 8×h , and then add the corresponding position information, first use the self-attention mechanism to obtain the stroke information feature Z after the self-fusion of each stroke information feature S1As the Q matrix input, the fused image features after the previous self-fusion are input into the cross attention mechanism as K and V matrices to obtain the final stroke information features:

[0044]

[0045] Z S2 It is the output of the stroke information prediction module Transformer-S. The original blank stroke information features are integrated with the image features to obtain the stroke information features that represent the image with strokes.

[0046] Step 4: After obtaining the stroke information feature Z S2 Finally, we only need to fuse the spatial features of the sketch into the stroke information features to obtain a painting that has a high degree of similarity with the sketch in both spatial structure and content.

[0047] In this step, Transformer is used as a model to fuse sketch spatial features. The sketch is passed through the image encoder E2 to obtain the sketch spatial features. In this step, there is no need to query the stroke information features again. Instead, the stroke information features obtained by the previous step that integrate the real image information are directly used to integrate the sketch space features and the stroke information features. In other words, the encoder part of the trained Transformer-M is directly used to integrate the sketch space features and the stroke information features without using the decoder part. Figure 3 As shown, Z SK1 After the fully connected layer, it is transformed into a feature matrix Z with the same number of hidden layers as the stroke information feature. SK2 ∈R 8×h , and stroke information feature concatenation Z S2 Get the splicing matrix Z M1 ∈R 8×2h , through the self-attention mechanism, the final stroke information features are obtained:

[0048]

[0049] The stroke information feature obtained has integrated the content information of the real image corresponding to the sketch and the spatial structure information of the sketch itself. M2 Input them into the stroke parameter fully connected layer and the stroke confidence fully connected layer respectively, and obtain the stroke parameters P∈R 8×5 and stroke confidence C.

[0050] The obtained stroke parameters include the horizontal and vertical coordinates, vertical coordinates, length, width, and rotation angle of the stroke. The process of stroke rendering is to use these parameters to display the original stroke style on the canvas to be painted through affine transformation. The color parameter of the stroke is the color of the center position of the stroke in the real image corresponding to the sketch. The stroke confidence determines whether the stroke is rendered. Generally, the stroke with confidence C>0.5 is rendered.

[0051] Example 2

[0052] like Figure 4 This embodiment relates to a device for automatically generating stroke paintings using sketches, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for automatically generating stroke paintings using sketches in Example 1.

[0053] Example 3

[0054] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method of automatically generating a stroke painting using a sketch in embodiment 1 is implemented.

[0055] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for automatically generating a stroke painting using a sketch, characterized in that: The following steps are involved: Step 1: First, train the Transformer-S module, which is a real image stroke information prediction module, and the Transformer-M module, which is a stroke global information and sketch information fusion module, in the model, so that the parameters in the modules are fixed for sketch-to-stroke-based painting generation; Step 2: Input the sketch into the pre-trained real image generation model Maskgit to obtain the real image. Pass the image encoder to obtain the image features of the real image corresponding to the sketch. Pass the blank canvas through the image encoder to obtain the canvas features. The features of the two are fused in the channel dimension to obtain the fused image features. At the same time, input the sketch into the sketch encoder to obtain the corresponding sketch space features. Step 3: Input the fused image features into the encoder of the trained Transformer-S module to obtain further fused image features; then input the fused image features into the decoder of the Transformer-S module to obtain stroke information features; Step 4: Connect the stroke information features and sketch space features and input them into the trained Transformer-M module to obtain the stroke information features that integrate the sketch space information. After passing through the fully connected layer, the stroke parameters are obtained. The strokes are rendered using the stroke parameters, and a painting based on the stroke style is generated by superimposing multiple strokes.

2. The method for automatically generating a stroke painting using a sketch according to claim 1, characterized in that: In the step 1, the training of the real image stroke information prediction module Transformer-S and the stroke global information and sketch information fusion module Transformer-M requires two steps. The Transformer-S module needs to be trained separately first, and then the Transformer-S and Transformer-M modules are trained together. The specific training method includes the following steps: Step 1.1: Input the sketch into the pre-trained real image generation model Maskgit to obtain the real image X corresponding to the sketch, and then input the obtained real image X into the image encoder E to downsample it to 1 / 4 of the original size to obtain the feature map Z containing the features of the image. x At the same time, the canvas C with some strokes is input into the image encoder E and downsampled to 1 / 4 of the original size, and the feature map Z containing the features of the canvas with some strokes is obtained. c ; Concatenate the two features in the channel dimension to obtain the fused image feature Z; Step 1.2: Input the existing fused image features Z into the encoder module of Transformer-S, use the self-attention mechanism of the encoder part of the module to extract the fused image features, input the obtained fused image features into the decoder of Transformer-S as the key and value in the cross attention, let the stroke information features to be queried be fused with the image information features to obtain the stroke information features, the stroke information features are passed through the fully connected layer to obtain the stroke parameters, and the intermediate stroke S is rendered. t , calculate the loss between the generated strokes and the sample strokes, the stroke loss function is: L S =L pixel +L param +L w +L decision (1) Where L pixel =|S pixel -S′ pixel | represents the pixel loss between the generated stroke and the sample stroke, and the absolute value of the color difference between the generated stroke and the sample stroke; L param =|S param -S′ param | represents the L1 loss between the parameters of the generated strokes and the parameters of the sample strokes, specifically the absolute value of the parameters; L w =WLoss(S distribution , S′ distribution ) represents the distance between the distribution of eight strokes generated at a time and the distribution of eight strokes of the sample. Specifically, the stroke parameters (x, y, w, h, θ) represent the horizontal coordinate, vertical coordinate, width, height, and rotation angle of the stroke on the canvas, and the stroke distribution is calculated by the following formula: The resulting stroke distribution is S distribution =N(μ,∑). L decision =BCELoss(S c , S′ c ) represents the binary cross entropy loss between the stroke confidence obtained by the model and the stroke confidence obtained by calculating the overlap of sample strokes using the traditional algorithm, specifically L decision =-λ·g v ·log(c u )-(1-g v )·log(1-c u ), where g v represents the confidence in the sample stroke, c u Represents the confidence of the generated strokes predicted by the Transformer-S module. S Loss Train the Transformer-S module on the random strokes dataset; Step 1.3, then input the stroke information features into the Transformer-M module to fuse the spatial structure information of the sketch, and finally obtain the final stroke information features. The stroke information features are passed through the fully connected layer to obtain the stroke parameters, which are rendered to obtain the stroke S′ and superimposed to obtain the painting X′. In the Transformer-M module, the loss of generating the painting includes not only the stroke loss in step 1.2, but also the loss of the discriminator that calculates the spatial structure gap between the sketch and the generated painting. The loss function is: Dis is the discriminator, which is actually a convolutional network used to extract the spatial structural features of the image. M is the total number of layers of the convolutional network of the discriminator, and N is the number of features that need to be compared at the t layer. It represents the i-th feature of image X in the t-th layer of the convolutional network; the total loss function for generating a painting is the sum of the stroke loss function and the discriminator loss function: L total =L S +L D (4) By L total Loss for training Transformer-S and Transformer-M modules on the Sketch dataset.

3. The method for automatically generating a stroke painting using a sketch according to claim 1, characterized in that: In the step 3, the fused image features are input into the encoder of the trained Transformer-S module to obtain further fused image features, and the fused image features are input into the decoder of the Transformer-S module, and the empty stroke information features are used to query the fused image features to obtain the stroke information features with the fused image information.

4. The method for automatically generating a stroke painting using a sketch according to claim 1, characterized in that: In the step 4, the stroke information features and the sketch space features are connected together and input into the Transformer-M module. The self-attention mechanism of the module is used to obtain the stroke information features that integrate the sketch information. Then, the stroke information features are input into the fully connected layer, and the stroke parameters that integrate the sketch space information and the generated stroke confidence are output.

5. The method for automatically generating a stroke painting using a sketch according to claim 1, characterized in that: In step 4, the output of the Transformer-M module includes the rendering parameters required for generating strokes and the confidence of the generated strokes; the strokes are rendered according to the stroke parameters, and the strokes with confidence levels below a certain threshold are removed. By superimposing multiple strokes, a painting based on the stroke style is finally generated.

6. A device for automatically generating stroke drawings using sketches, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement the method of automatically generating stroke paintings using sketches as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the method for automatically generating a stroke painting using a sketch as described in any one of claims 1-5 is implemented.

Images