A method and apparatus for automatically generating a sketch from a sketch

Abstract

A method and device for automatically generating strokes drawing by sketch, the method comprising: firstly training a real image stroke information prediction module Transformer-S module and a stroke global information and sketch information fusion module Transformer-M module in the model; inputting the sketch into a pre-trained real image generation model Maskgit to obtain a real image, obtaining image features of the real image corresponding to the sketch through an image encoder, inputting a blank canvas into the image encoder to obtain canvas features, fusing the features of the two in the channel dimension to obtain fused image features; simultaneously inputting the sketch into a sketch encoder to obtain corresponding sketch spatial features; inputting the fused image features into an encoder of the trained Transformer-S module to obtain further fused image features; then inputting the fused image features into a decoder of the Transformer-S module to obtain stroke information features; connecting the stroke information features and the sketch spatial features together and inputting them into the trained Transformer-M module to obtain stroke information features fused with sketch spatial information, obtaining stroke parameters through a fully connected layer, rendering the stroke parameters to obtain strokes, and generating a drawing based on a stroke style by superimposing multiple strokes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image generation methods, in particular to a method and device for automatically generating stroke painting from a sketch. BACKGROUND

[0002] Painting has always been a wonderful way to express and record people's ideas. It can make abstract concepts that are difficult to express in words into tangible expressions that can be shared with others, and express emotions. However, painting has always been considered an art that requires high technical skills. It is not easy for someone with no painting experience to paint a beautiful and accurate painting. Compared with sketches, ink paintings, oil paintings and other painting methods require higher proficiency of the painter. When painting these paintings, the painter not only needs to consider the structure and distribution of the entire painting, but also needs to consider the application of color on each part of the painting. Sketches only need to think about the structure of the painting. Even so, the required proficiency is not low. Today, computer-aided painting greatly reduces the demand for proficiency in painting. Artificial intelligence technology is applied to painting, greatly reducing the time spent on painting, and enabling the generation of color vivid paintings from sketches.

[0003] Sketches can be converted into color paintings through image migration and image translation artificial intelligence technologies. The former mainly uses existing sketch and color painting samples to train a deep network, so that the network learns to change the image from the sketch style to the color painting style while maintaining the image spatial information and semantic information as much as possible. The latter explores the differences between the sketch and color painting image domains, allowing the network to learn the path of converting from the sketch to the color painting in the latent code representation. However, these methods are pixel-based image conversion, and their painting process does not have the general understanding of the painting process.

[0004] Stroke painting is based on adding a few strokes to the canvas at each step, and finally these strokes are superimposed to form a painting, rather than directly obtaining the final painting from pixels through a transformation that does not conform to human thinking. The stroke-based painting process is more consistent with the general painting thinking process, and the brush-stroke painting is more consistent with people's expectations of painting.

[0005] With the development of the Transformer neural network, it has been extended from the original application to one-dimensional sequence such as text transformation or generation task to two-dimensional data such as image transformation or generation task. When applying the Transformer in the image translation task, the loss control is taken on the style information and the spatial content information between the input and the output, so that the output picture has a high similarity in spatial content with the image to be transformed and has a high similarity in style with the style image. However, the input of these methods is mostly single, and they focus on the mapping of the image domain in style and spatial content. In the process of sketch conversion to color picture, if only the sketch is used as the control information of the spatial content, the generated image will lack color information and be uncontrollable in color. SUMMARY

[0006] The present application aims to overcome the above-mentioned shortcomings of the prior art and provide a method and device for automatically generating stroke paintings using sketches.

[0007] To solve the above technical problems, the present application adopts the following steps:

[0008] The first aspect of the present application relates to a method for automatically generating stroke paintings using sketches, comprising the following steps:

[0009] Step 1, first train the real image stroke information prediction module Transformer-S module and the stroke global information and sketch information fusion module Transformer-M module in the model, so that the parameters in the module are fixed, and are used for sketch to stroke-based painting generation;

[0010] Step 2, input the sketch into the pre-trained real image generation model Maskgit to obtain a real image, pass the image through an image encoder to obtain the image features of the real image corresponding to the sketch, pass a blank canvas through the image encoder to obtain the canvas features, and fuse the features of the two in the channel dimension to obtain the fused image features; at the same time, input the sketch into a sketch encoder to obtain the corresponding sketch spatial features;

[0011] Step 3, input the fused image features into the encoder of the trained Transformer-S module to obtain further fused image features; then input the fused image features into the decoder of the Transformer-S module to obtain stroke information features;

[0012] Step 4, connect the stroke information features and the sketch spatial features together and input them into the trained Transformer-M module to obtain stroke information features fused with the sketch spatial information, and pass the stroke information features through a fully connected layer to obtain stroke parameters. Use the stroke parameters to render to obtain strokes, and generate stroke style-based paintings through superposition of multiple strokes.

[0013] The second aspect of the present application relates to a device for automatically generating stroke painting from sketch, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the method for automatically generating stroke painting from sketch of the present application.

[0014] The third aspect of the present application relates to a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the method for automatically generating stroke painting from sketch of the present application.

[0015] The present application uses the corresponding real image features of the sketch and the spatial structure features of the sketch itself to propose a generation model for generating stroke-based paintings from sketches. The model combines the advantages of convolutional neural networks for extracting image features and the self-attention mechanism and cross-attention mechanism of the Transformer network for fusing various information features. For sketch images, a pre-trained real image generation model Maskgit is used to obtain the real image of the sketch, which complements the color distribution of the sketch. At the same time, the attention mechanism of the Transformer is introduced, and in the real image stroke information prediction module Transformer-S module, the encoder is used to obtain further fusion information of the real image features, and in the decoder part, the stroke distribution shape is closer to the stroke information feature of the real image by querying and fusing image information features. In the stroke global information and sketch information fusion module Transformer-M module, only the encoder is used to fuse the stroke information feature and the sketch spatial feature, so that the generated stroke-based painting has high consistency with the input sketch in spatial structure. The present application can generate stroke-based paintings with artistic feeling, which have high consistency with the input sketch in painting content and spatial structure, and the generated paintings have diversity.

[0016] The advantages of the present application are: 1. Compared with pixel-based painting generation, the painting generation of the present application is based on strokes, which is more consistent with the general painting thinking process in the logic of the painting process, and the painting generation process is more convenient for teaching. 2. The painting generation result of the present application has diversity, and one sketch generation can generate multiple paintings similar to the real image in spatial structure and corresponding to the sketch in painting content. 3. Compared with other sketch-to-painting image conversion models, the present application uses Maskgit to provide corresponding supplements for the lack of color information of the sketch, which can generate results with more details and closer to the sketch in content. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1is a flowchart of sketch generation based on stroke drawing of the present application.

[0018] Figure 2 is a structure diagram of the Transformer-S module of the present application.

[0019] Figure 3 is a structure diagram of the Transformer-M module of the present application.

[0020] Figure 4 is a schematic diagram of the device of the present application. DETAILED DESCRIPTION

[0021] The technical solutions of the present application are further described below in combination with the drawings.

[0022] Example 1

[0023] This embodiment relates to a method for automatically generating stroke drawing by sketch, which combines the advantages of CNN feature extraction and the attention mechanism of Transformer, and the overall architecture is as follows Figure 1As shown, it is mainly divided into four steps: step 1, the real image stroke information prediction module Transformer-S module is trained using the random stroke data set, and then the real image stroke information prediction module Transformer-S and the stroke global information and sketch information fusion module Transformer-M module are trained using the sketch data set, and the parameters in the two modules are determined; step 2, the pre-trained sketch generates a real image model Maskgit, which provides corresponding supplement for the lack of color information of the sketch, and obtains the corresponding real image of the sketch. At the same time, the sketch encoder and the image encoder respectively extract the sketch and the real image to obtain the corresponding features and perform feature fusion; step 3, according to the fused image features obtained in step 2, input into the Transformer-S module, the module predicts the corresponding drawing stroke through the image features, so as to generate the stroke information which can constitute the drawing in the overall spatial structure similar and the color information distribution approximately same, the Transformer in this part uses the encoder part of the Transformer and the decoder part of the Transformer, the encoder is used to fuse the features of the real image, and the decoder is used to fuse the real image features and the empty stroke information, and finally the stroke information fused with the real image information is obtained; step 4, according to the stroke information obtained in step 3, input into the Transformer-M module, the module fuses the stroke information and the sketch features to obtain the stroke information constituting the drawing, which includes stroke parameters and generates stroke confidence. In this module, only the encoder part of the Transformer is used, relying on the self-attention mechanism to fuse the stroke information and the sketch features, and finally the stroke parameters and the generated stroke confidence are output through the full connection layer. After obtaining the stroke parameters, the strokes are rendered, and the strokes with confidence lower than a certain value are discarded, and finally the stroke-based drawing is formed through multi-level stroke superposition.

[0024] Step 1, the training of the real image stroke information prediction module Transformer-S and the stroke global information and sketch information fusion module Transformer-M needs two steps, the Transformer-S module needs to be trained separately first, and then the Transformer-S and the Transformer-M modules are trained together, and the specific training method includes the following steps:

[0025] Step 1.1, input the sketch into the pre-trained real image generation model Maskgit to obtain the corresponding real image X of the sketch, and then input the obtained real image X into the image encoder E to downsample to 1 / 4 of the original size to obtain the feature map Z containing the features of the image x; meanwhile, the existing canvas C with partial strokes is input into the image encoder E to downsample to 1 / 4 of the original size, and the feature map Z containing the features of the canvas with partial strokes c ; the two features are spliced in the channel dimension to obtain the fused image feature Z

[0026] Step 1.2, input the existing fused image feature Z into the encoder module of the Transformer-S, use the self-attention mechanism of the encoder part to extract the fused image feature, input the obtained fused image feature into the decoder of the Transformer-S as the key and value in the cross-attention, let the stroke information feature query the fused image information feature, obtain the stroke information feature, and the stroke information feature passes through a fully connected layer to obtain the stroke parameter, and render to obtain the intermediate stroke S t , calculate the loss between the generated stroke and the sample stroke, and the stroke loss function is:

[0027] L S =L pixel +L param +L w +L decision (1)

[0028] Wherein L pixel =|S pixel -S′ pixel | represents the pixel loss between the generated stroke and the sample stroke, and the absolute value of the color difference between the generated stroke and the sample stroke; L param =|S param -S′ param | represents the L1 loss between the parameters of the generated stroke and the parameters of the sample stroke, specifically the absolute value of the parameters; L w =WLoss(S distribution , S′ distribution ) is the distance between the distribution composed of the eight strokes generated at one time and the distribution composed of the eight strokes of the sample, specifically the stroke distribution calculated by the following formula:

[0029]

[0030] The obtained stroke distribution is S distribution =N(μ,∑).

[0031] L decision =BCELoss(S c , S′ c ) represents the binary cross-entropy loss between the stroke confidence obtained by the model and the stroke confidence obtained by calculating the coincidence degree of the sample stroke using the traditional algorithm, specifically Ldecision = -λ-g v log(c u ) - (1-g v ) log(l-c u ), where g v represents the confidence in the sample stroke, c u represents the confidence of the stroke predicted by the Transformer-S module. The Transformer-S module is trained in the random stroke dataset by L S loss;

[0032] Step 1.3, then the stroke information features are input into the Transformer-M module to fuse the spatial structure information of the sketch, and finally the final stroke information features are obtained. The stroke information features pass through the full connection layer to obtain the stroke parameters, render the stroke S' and superimpose to obtain the painting X'. In the Transformer-M module, the loss of generating the painting not only includes the stroke loss in step 1.2, but also needs to calculate the discriminator loss of the spatial structure gap between the sketch and the generated painting, and the loss function is:

[0033]

[0034] where Dis is the discriminator, which is essentially a convolutional network used to extract image spatial structure features, M is the total number of layers of the discriminator's convolutional network, N is the number of features to be compared at layer t, is the i-th feature in the t-th layer of the convolutional network representing the image X; the total loss function of the generated painting is the sum of the stroke loss function and the discriminator loss function:

[0035] L total = L S + L D (4)

[0036] The Transformer-S and Transformer-M modules are trained in the sketch dataset by L total loss.

[0037] Step 2, input the sketch into the pre-trained real image generation model Maskgit, use its ability to generate real images to obtain real images, pass the images through the image encoder to obtain the image features of the real images corresponding to the sketch, pass the blank canvas through the image encoder to obtain the canvas features, then fuse the features of the two in the channel dimension to obtain the fused image features. After the real image passes through the image encoder E, the real image features are obtained After the canvas passes through the image encoder E, the canvas features are obtained Then the two image features are spliced in the channel dimension to obtain a fused image feature

[0038] Step 3, in order to make the finally generated drawing have a stroke feeling and be able to show a continuous drawing process, the present application does not focus on pixels, but uses a method of predicting stroke information, and the generated image is composed of stroke stacks rendered by the predicted stroke information, and the stroke information includes parameters required for rendering the stroke and confidence of generating the stroke.

[0039] In order to predict the rendering parameters of each stroke, the parameters of each detection box in target detection are predicted independently, and the corresponding stroke information feature is obtained by querying the real image fused feature through each stroke information feature. This corresponds to the query, key and value attention mechanism of the Transformer model, which not only enables the real image feature to be queried to be self-fused using the self-attention mechanism before being fused with the stroke information feature, making the fused image feature more clear and effective, but also enables each stroke information feature to be directly fused with the fused real image feature in the Transformer decoder part, thereby obtaining each stroke information feature.

[0040] As shown in Figure 2 After obtaining the spliced image feature Z, the fused image feature Z is added with position encoding to make the tensor of each position have different characteristics, and then passes through the encoder part of the trained Transformer-S, and the query, key and value transformation dimension full connection layers are used to transform the image feature respectively to obtain Q, K and V matrices, and the self-fused image feature is obtained through the self-attention mechanism:

[0041]

[0042] Then the image feature is further fused through the full connection layer again. It should be noted that the dimension shape will not be transformed before the input feature and after the output feature, which enables the self-fusion of the fused image feature in the encoder part to be performed more times, thereby further extracting effective fused image features.

[0043] In the decoder part of the Transformer-S, the query Q matrix of the cross-attention mechanism therein needs to be provided by the queried stroke information, and the key K matrix and the value V matrix are provided by the fused fused image feature. First, 8 blank stroke information features Z S ∈R 8×h are set, and then corresponding position information is added, and the self-fused stroke information features Z S1As the Q matrix input, the fused image features after the previous self-fusion are input into the cross attention mechanism as K and V matrices to obtain the final stroke information features:

[0044]

[0045] Z S2 It is the output of the stroke information prediction module Transformer-S. The original blank stroke information features are integrated with the image features to obtain the stroke information features that represent the image with strokes.

[0046] Step 4: After obtaining the stroke information feature Z S2 Finally, we only need to fuse the spatial features of the sketch into the stroke information features to obtain a painting that has a high degree of similarity with the sketch in both spatial structure and content.

[0047] In this step, Transformer is used as the model to fuse sketch spatial features. Sketch spatial features are obtained after the sketch passes through the image encoder E2. In this step, there is no need to query the stroke information features again. Instead, the stroke information features obtained in the previous step that are integrated with the real image information are directly used to fuse the sketch space features and the stroke information features. In other words, the encoder part of the trained Transformer-M is directly used to fuse the sketch space features and the stroke information features without using the decoder part. Figure 3 As shown, Z SK1 After the fully connected layer, it is transformed into a feature matrix Z with the same number of hidden layers as the stroke information feature. SK2 ∈R 8×h , and stroke information feature splicing Z S2 Get the splicing matrix Z M1 ∈R 8×2h , through the self-attention mechanism, the final stroke information features are obtained:

[0048]

[0049] The obtained stroke information features have integrated the content information of the real image corresponding to the sketch and the spatial structure information of the sketch itself. M2 Input them into the stroke parameter fully connected layer and the stroke confidence fully connected layer respectively, and obtain the stroke parameters P∈R 8×5 and stroke confidence C.

[0050] The obtained stroke parameters include the horizontal and vertical coordinates, the vertical axis coordinate, the length, the width, the rotation angle of the stroke, and the process of stroke rendering is to display the original existing brush stroke style on the canvas to be drawn by using the parameters through affine transformation, wherein the color parameter of the stroke is the color of the position of the center of the stroke in the corresponding real image of the sketch. The stroke confidence is used to determine whether the stroke is rendered, and generally, the strokes with a confidence C>0.5 are rendered.

[0051] Embodiment 2

[0052] As Figure 4 This embodiment relates to a device for automatically generating stroke drawing by using a sketch, comprising a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the method for automatically generating stroke drawing by using a sketch in the embodiment 1.

[0053] Embodiment 3

[0054] This embodiment relates to a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the method for automatically generating stroke drawing by using a sketch in the embodiment 1.

[0055] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A method for automatically generating a stroke painting from a sketch, the method comprising: Comprise the following steps: Step 1, first train the model real image stroke information prediction module Transformer-S module and stroke global information and sketch information fusion module Transformer-M module, make the parameters in the module fixed, for sketch to stroke-based painting generation; Step 2, input the sketch into the pre-trained real image generation model Maskgit, get the real image, get the image features of the real image corresponding to the sketch through the image encoder, also get the canvas features through the image encoder, fuse the features of the two in the channel dimension to get the fused image features; At the same time, input the sketch into the sketch encoder to get the corresponding sketch spatial features; Step 3, input the fused image features into the encoder of the trained Transformer-S module to get further fused image features; Then input the fused image features into the decoder of the Transformer-S module to get stroke information features; Step 4, connect the stroke information features and the sketch spatial features together and input them into the trained Transformer-M module to get stroke information features fused with sketch spatial information, and get stroke parameters through the fully connected layer; Use the stroke parameters to render to get strokes, and generate paintings based on the stroke style through the superposition of multiple strokes. 2.The method of claim 1, wherein, In step 1, the training of the real image stroke information prediction module Transformer-S and the stroke global information and sketch information fusion module Transformer-M needs two steps, the Transformer-S module needs to be trained separately first, and then the Transformer-S and Transformer-M modules are trained together, the specific training method comprises the following steps: Step 1.1, input the sketch into the pre-trained real image generation model Maskgit to obtain a real image X corresponding to the sketch, and then input the obtained real image X into an image encoder E to downsample to 1 / 4 of the original size to obtain a feature map Z containing image features x ; meanwhile, input the canvas C with the existing partial strokes into the image encoder E to downsample to 1 / 4 of the original size to obtain a feature map Z containing features of the canvas with the partial strokes c ; and splice the two features in the channel dimension to obtain a fused image feature Z; Step 1.2, input the existing fusion image feature Z to the encoder module of the Transformer-S, use the self-attention mechanism of the encoder part of the module to extract the fusion image feature, input the obtained fusion image feature into the decoder of the Transformer-S as the key and value in the cross-attention, let the stroke information feature fusion image information feature query, obtain the stroke information feature, and the stroke information feature is obtained through a full connection layer to obtain the stroke parameter, and the generated stroke is rendered , the loss between the generated stroke and the sample stroke is calculated, and the stroke loss function is: wherein represents the pixel loss between the generated stroke and the sample stroke, the absolute value between the color difference of the generated stroke and the sample stroke; represents the L1 loss between the parameters of the generated stroke and the parameters of the sample stroke, specifically the absolute value of the parameters; is the distance between the distribution of the eight strokes generated at one time and the distribution of the eight strokes of the sample, specifically the stroke parameters , representing the horizontal coordinate, vertical coordinate, width, height, and rotation angle of the stroke on the canvas, the stroke distribution is calculated by the following formula: The resulting stroke distribution is ; represents the binary cross-entropy loss between the stroke confidence obtained by the model and the stroke confidence obtained by the coincidence degree calculated by the traditional algorithm for the sample stroke, and is specifically wherein represents the confidence in a sample stroke, represents the confidence of the Transformer-S module in predicting a generated stroke; by losses in a random stroke dataset; Step 1.3, then the stroke information features are input into the Transformer-M module to fuse the spatial structure information of the sketch, and finally the final stroke information features are obtained; the stroke information features pass through a fully connected layer to obtain stroke parameters, and the stroke parameters are rendered to obtain strokes and superimposed to obtain a painting In the Transformer-M module, the loss of generating the painting includes not only the stroke loss in step 1.2, but also the loss of the discriminator calculating the spatial structure gap between the sketch and the generated painting, and the loss function is: where Dis is the discriminator, which is essentially a convolutional network to extract the spatial structure features of the image, M is the total number of layers of the convolutional network of the discriminator, N is the number of features to be compared at the tth layer, is the i th feature of the image X in the t th layer of the convolutional network; and the total loss function of the generated drawing is the sum of the stroke loss function and the discriminator loss function: By The Transformer-S and Transformer-M modules are trained on the sketch data set with loss. 3.The method of claim 1, wherein, In step 3, the fused image features are input into the encoder of the trained Transformer-S module to get further fused image features, and the fused image features are input into the decoder of the Transformer-S module to query the fused image features with empty stroke information features to get stroke information features with fused image information. 4.The method of claim 1, wherein, In step 4, the stroke information features and the sketch spatial features are connected together and input into the Transformer-M module, the stroke information features fused with sketch information are obtained using the self-attention mechanism of the module, and then the stroke information features are input into the fully connected layer to output stroke parameters fused with sketch spatial information and stroke generation confidence. 5.The method of automatically generating a stroke drawing with a sketch according to claim 1, wherein, In step 4, the Transformer-M module outputs include rendering parameters required for stroke generation and confidence of stroke generation; According to the stroke parameters, strokes are rendered and generated, and strokes with confidence lower than a certain threshold are removed, and finally paintings based on the stroke style are generated through the superposition of multiple strokes.

6. An apparatus for automatically generating a stroke drawing using a sketch, characterized by, An apparatus comprising a memory having executable code stored therein and one or more processors that, when executing the executable code, implement the method of any of claims 1-5 for automatically generating a stroke painting from a sketch.

7. A computer readable storage medium characterized by A program stored on a computer readable medium, the program, when executed by a processor, implements the method of any of claims 1-5 for automatically generating a stroke painting from a sketch.

Images