Method for coloring embroidery draft picture based on generative adversarial network and computer equipment

Abstract

The application discloses a kind of based on generative adversarial network's pile embroidery sketch map coloring method and computer equipment, the method includes the following steps: collecting animation and pile embroidery color image and pretreatment, then extract corresponding sketch and color prompt information, construct training data set;Pile embroidery sketch coloring model is constructed;Pile embroidery sketch coloring model includes feature extraction module, generator and discriminator of CSA module are added;First, the training data set of animation is used to train pile embroidery sketch coloring model, then migration training is carried out on the training data set of pile embroidery;The pile embroidery sketch to be processed and corresponding color prompt information are input into the trained pile embroidery sketch coloring model, and the target pile embroidery color image is output.The application uses generative adversarial network to complete the coloring of pile embroidery sketch, improves the generalization ability of model by introducing feature extraction module in network, CSA module is introduced simultaneously, improves coloring precision, reduces color overflow problem, and the coloring effect is better.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a heap embroidery sketch coloring method based on a generative adversarial network and a computer device. BACKGROUND

[0002] Huangzhong pile embroidery mainly represents Tibetan Buddhism themes, and shapes images by cutting and piling techniques, and is mainly used for Tangka production. Pile embroidery handicrafts are made by cutting various patterns and shapes designed by various colored cotton, silk and satin, and carefully piling them into a complete picture. The process includes pattern design, cutting, piling and coloring of individual pattern parts, and combines embroidery and relief art together, showing delicate and rough, simple and beautiful, and presenting a moving national characteristic and strong artistic appeal.

[0003] In the design stage of pile embroidery, designers need to draw sketches and then color the sketches. Sketch coloring is a very important step in the creation process, but the coloring process involves a lot of redundant labor and complex procedures, consuming a lot of manpower and being low in efficiency. How to effectively reduce the repetitive work in the pile embroidery sketch coloring process to improve efficiency is a problem that needs to be solved. The traditional method of pile embroidery sketch coloring is to first segment the sketch, and then fill the regions. With the rise of deep learning, neural networks have been widely used in the field of sketch coloring.

[0004] Traditional propagation-based sketch coloring can be traced back to the flood-filling algorithm used in the "bucket" tool of Apple computer in 1976. The algorithm can fill the color for the closed region of the image. There are two conditions for using the algorithm: (1) the region has a closed boundary; (2) the pixels in the region have the same color. The disadvantage of the flood-filling algorithm is that it cannot deal with the phenomena of discontinuous lines, auxiliary lines, shadow lines and textures in the sketch, resulting in color leakage at the broken line, color interruption in small areas of auxiliary lines and textures. After that, many algorithms for filling colors in user-specified regions have been proposed, mainly including color transmission and color diffusion:

[0005] (1) Color transmission, that is, learning a color pattern from a reference image or sample set and applying it to the target image, mainly using statistical techniques;

[0006] (2) Color diffusion, mainly through human-computer interaction, obtaining some basic color strokes from the user as seed points, and diffusing to a certain range. The disadvantage of the traditional algorithm is that it can only fill the entire region with a single color according to the user's prompt, and there are problems such as color leakage at the discontinuous boundary and color interruption in small areas.

[0007] Deep learning-based line art coloring methods can be divided into two categories. The first category does not require any color prompts from the user; the network can directly generate the corresponding color artwork based on the line art provided by the user. The second category colors the line art based on the user's color prompts, which can be text information, reference images, or color prompt lines provided by the user. For example, a two-stage coloring model based on CNN divides the coloring process into two stages. The first stage aggressively applies color to the canvas to increase color diversity, while the second stage addresses issues such as color errors, overflow, and blurring. Refined coloring can be achieved by repeatedly adjusting color lines, but this also requires more human-computer interaction. Introducing local feature networks into the CGAN structure enhances the generalization ability of networks trained on synthetic datasets, improving their performance on realistic line art compared to previous work. The generator uses deeper network structures and several specially designed layers to increase network capacity and receptive field, making the results more realistic. There is also a method that can generate color artwork based on color label information and line art, namely a network structure called SECat, which improves the generator's ability to handle details. However, because it involves understanding different modalities of data, text labels perform poorly in locating the color scope. Style transfer has also been used in image coloring tasks, such as using residual U-Net and the auxiliary classifier GAN to color line art based on style images. Style transfer can color images to some extent, but the effect is not realistic enough. Summary of the Invention

[0008] The purpose of this invention is to provide a method and computer device for coloring line drawings of piled embroidery based on generative adversarial networks, so as to solve the problem of a large amount of repetitive work and low efficiency in coloring line drawings during the design stage of piled embroidery production, thereby improving speed and efficiency.

[0009] The present invention adopts the following technical solution:

[0010] A method for coloring embroidery line art based on generative adversarial networks includes the following steps:

[0011] Collect color images of anime and appliqué embroidery, preprocess the collected color images, extract the corresponding line art and color hints from the preprocessed color images, and construct a training dataset from the color images and the corresponding line art and color hints.

[0012] Construct a coloring model for piled embroidery line art; the coloring model for piled embroidery line art includes a feature extraction module, a generator with CSA module, and a discriminator;

[0013] First, the coloring model of the piled embroidery line drawing was trained using the training dataset of animation, and then transfer training was performed on the training dataset of piled embroidery.

[0014] The embroidery line drawing to be processed and the corresponding color prompt information are input into the trained embroidery line drawing coloring model, and the target embroidery coloring image is output.

[0015] Furthermore, the preprocessing includes dataset augmentation and size cropping, where dataset augmentation includes rotation and mirroring.

[0016] Furthermore, the XDOG algorithm is used for line art extraction, and the value of σ is randomly set during the extraction process to control the thickness of the extracted line art edges.

[0017] Furthermore, the color-coded information is extracted using a method of randomly selecting pixels.

[0018] Furthermore, the color cues are extracted during the training process. It is randomly determined whether a certain pixel needs color sampling. If so, the mask is superimposed on the color image to obtain the color cues.

[0019] Furthermore, the generator for the coloring model of the embroidery line art includes a downsampling module, an upsampling module, and a CSA module, which are used to generate the target embroidery coloring image based on the embroidery line art and the corresponding color cues. The process is as follows:

[0020] The single-channel embroidery line drawing is subjected to three convolution operations using a downsampling module to obtain a feature map. The color cues are convolved with a convolutional layer and then concatenated with the feature map. The downsampling module is then used to further downsample the image, with consecutive CSA modules in between to reduce color overflow using spatial attention and residual mechanisms. On the right are four upsampling modules, each starting with a convolutional submodule followed by Bn ResNeXtBlocks. Finally, a subpixel convolutional layer is used to improve the resolution of the feature map. After the upsampling operation on the right, the image passes through a convolutional layer and an activation layer to finally generate a color image.

[0021] Furthermore, the CSA module's feature map processing flow includes:

[0022] Assuming the feature map output by the final downsampling is x, the feature map y obtained by inputting x into a residual module is concatenated with the feature map extracted from the color image using the feature extraction module, and then convolved and activated using an activation function. After that, it is fed into another convolutional layer and activated using an activation function to obtain z. The product of z and y is added to x to obtain the output of a CSA module.

[0023] Furthermore, the discriminator of the embroidery line drawing coloring model takes a color image and a feature map extracted from the color image using a feature extraction module as input. The discriminator inputs the line drawing and the color image generated by the generator of the embroidery line drawing coloring model or the line drawing and the real image into the discriminator network, and obtains the discrimination result after a series of convolutions.

[0024] Furthermore, before inputting the embroidery line drawing to be processed into the trained embroidery line drawing coloring model, the XDOG algorithm is used to extract the line drawing from the embroidery line drawing to be processed. During the line drawing extraction process, the value of σ is adjusted as needed to obtain line drawings similar to those in the training dataset.

[0025] A computer device includes a processor and a memory connected to the processor; the memory is used to store a computer program for executing the above-described method for coloring line drawings based on generative adversarial networks.

[0026] The processor is used to call and execute computer programs stored in memory.

[0027] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0028] This invention utilizes generative adversarial networks (GANs) to assist in coloring embroidery line drawings. By introducing a feature extraction module into the network, the generalization ability of the model is improved. At the same time, the introduction of a CSA module into the embroidery line drawing coloring model improves coloring accuracy, reduces color overflow problems, and results in better coloring effects.

[0029] Furthermore, the distribution of line art data in the training dataset differs significantly from that in the real line art dataset. This invention first converts the distribution of real line art data to the distribution in the training dataset before coloring the latter, thus achieving coloring of the real embroidery line art data and avoiding problems such as color overflow and coloring errors. To address the issue of the small size of the embroidery line art dataset, the model is first trained using animation data, and then a transfer learning method is used to transfer the model trained on the animation dataset to the embroidery scene. Attached Figure Description

[0030] Figure 1 This is a flowchart of a method for coloring line drawings in embroidery based on generative adversarial networks;

[0031] Figure 2 This is a flowchart of a method for coloring line drawings in embroidery based on generative adversarial networks;

[0032] Figure 3 This is a schematic diagram of the generator for coloring the embroidery line drawing model;

[0033] Figure 4 This is a structural diagram of the CSA module in the model of coloring the embroidery outline;

[0034] Figure 5 This is a schematic diagram of the discriminator for the coloring model of the piled embroidery line drawing. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0036] Manually coloring embroidery line drawings is labor-intensive and time-consuming. This invention can automatically color embroidery line drawings with user prompts, greatly improving work efficiency. At the same time, most existing line drawing coloring technologies suffer from problems such as color overflow (the network cannot accurately identify the boundaries of the line drawing, causing the filled color to spread to the surrounding background area), coloring results that differ from the specified color, and excessive randomness in coloring results. This invention improves the model's generalization ability by introducing a pre-trained VGG16 feature extraction module into the network, and improves coloring accuracy by introducing a CSA module into the coloring model, thus reducing the color overflow problem.

[0037] Previous applications of line art coloring were primarily in illustration or comics. This invention addresses the problem of line art coloring in the process of creating piled embroidery. While previous methods could produce decent color drafts based on given line art and color cues, they failed to accurately capture the data distribution of real piled embroidery line art, often resulting in color overflow and coloring errors. Some attempts involved introducing local feature networks into CGAN networks to enhance the model's generalization ability and improve performance on real line art data. Others involved converting the line art to grayscale before coloring. To address the significant difference between the line art data distribution in the training dataset and the real line art dataset, this invention attempts to first convert the real line art data distribution to the distribution in the training dataset before coloring the latter. Furthermore, a CSA module is introduced into the generative adversarial network to resolve the color overflow issue.

[0038] like Figure 1 As shown, the coloring method for embroidery line art based on generative adversarial networks in this embodiment of the invention includes the following steps:

[0039] S101. Collect color images of animation and pile embroidery, preprocess the collected images, use the XDOG algorithm to extract the corresponding line art based on the preprocessed images, and use the method of randomly selecting pixels to extract the corresponding color cues to construct a training dataset.

[0040] S102. Construct a coloring model for piled embroidery line art; the coloring model for piled embroidery line art includes a pre-trained VGG16 feature extraction module, a generator with added CSA module, and a discriminator.

[0041] For example, the CSA (Concatenation and SpatialAttention) module can be introduced into the generator model of the AlacGan (Anime Line Art Colorization with ConditionalAdversarial Networks) model to improve the AlacGan model and obtain a coloring model for piled embroidery line art.

[0042] S103. The model for coloring embroidery line drawings is trained using the animation dataset, and then transferred to the embroidery dataset for training.

[0043] S104. Input the embroidery line drawing to be processed and the corresponding color prompt information into the trained embroidery line drawing coloring model, color the embroidery line drawing to be colored, and output the target embroidery coloring image.

[0044] The coloring method for embroidery line art based on generative adversarial networks can be divided into three parts, such as... Figure 2 As shown, it includes: the collection of embroidery and animation datasets, the training of the embroidery line drawing color generator and discriminator, and the use of the trained embroidery color model to automatically color the embroidery line drawing and the corresponding color prompts.

[0045] Specifically, after collecting color images of anime and embroidery, the color images are preprocessed. The preprocessing of the color images includes rotating and mirroring the collected color patterns, and cropping all images to a size of 512×512.

[0046] The XDOG algorithm was used to extract line art from the cropped image, generating color images and line art image pairs. To ensure good generalization ability of the model, the value of σ was randomly set to 0.3 / 0.4 / 0.5 during the line art extraction process to control the thickness of the extracted line art edges. The color images and extracted line art images were kept with the same name and used as the training dataset for the model.

[0047] This invention generates color-coded cues in images during network training and applies this information to the training process. To obtain color-coded cues, the color image is first downsampled four times. Then, a mask is used to randomly determine whether a particular pixel needs color sampling. This mask is then overlaid on the color image to obtain the color-coded cues. In other words, it uses masking techniques from image processing to generate color-coded cues. Specifically, some pixels are randomly selected as 1 on the mask, while other pixels are set to 0. Applying this mask to the color image yields the color-coded cues.

[0048] The model training consists of two stages. The first stage is training on the anime dataset. Because the coloring of anime line art and the coloring of embroidery line art are highly similar, in order to address the problem that the embroidery line art dataset is too small, the model is first trained on the anime dataset for 250,000 epochs. The learning rate is kept at 0.0001 for the first 120,000 epochs and then at 0.00001 until the model converges. When using the embroidery dataset for transfer learning, the learning rate is kept at 0.00001 and the model is trained for another 50,000 epochs.

[0049] When using the embroidery line art coloring model to color based on the embroidery line art and corresponding color cues, in order to address the issue of significant differences in the data distribution between the line art extracted using the XDOG algorithm during training and the embroidery line art input by the user in actual use, a line art conversion module is used to convert the input line art before inputting it into the embroidery line art coloring model. Specifically, the XDOG algorithm is used to extract the line art from the input line art. During the line art extraction process, the value of σ is adjusted as needed to obtain a synthetic line art similar to the training dataset. Then, the synthetic line art and the corresponding color cues are input into the embroidery line art coloring network for coloring. Finally, the model generates a 3-channel color image.

[0050] In this embodiment, the generator model for the coloring model of the embroidery line drawing mainly includes a downsampling module, an upsampling module, and a CSA (Concatenation and Spatial Attention) module, and its structure is as follows: Figure 3 As shown. The generator model is designed based on the U-Net architecture. The process by which this generator model generates a color image based on the embroidery line drawing and the corresponding color cues is as follows:

[0051] First, a single-channel stacked embroidery drawing of size 512×512 is subjected to three convolution operations. Except for the first layer, where the kernel size is 3, stride is 1, and padding is 1, the kernel size of the remaining two layers is 4, stride is 2, and padding is 1. The color cues are convolved using a convolutional layer and then connected to the feature map obtained from the previous stacked embroidery drawing convolution. Downsampling continues, and in between, a series of CSA modules are used to minimize color overflow by utilizing spatial attention and residual mechanisms. A special feature in the network structure is the four sub-modules on the right side of U-net. Each sub-module is started with a convolutional sub-module, followed by Bn ResNeXtBlocks. The value of Bn is set to {20, 10, 10, 5}. The purpose of using this module instead of ResNet is to increase the network capacity. Finally, subpixel convolutional layers are used to improve the resolution of the feature map. After the upsampling operation on the right, it goes through a 3×3 convolutional layer and a Tanh activation layer to finally generate a 512×512 color image.

[0052] The embroidery line drawing coloring model introduces the CSA module based on AlacGan, such as... Figure 4 As shown, the CSA module's feature map processing flow includes:

[0053] Assuming the feature map output by the previous convolutional layer is x, the feature map y obtained by inputting x into a residual module is concatenated with the feature map extracted by the VGG16 feature extraction module, and then convolved once. After activation by the ReLU activation function, it is fed into another convolutional layer and activated by the Sigmoid function to obtain z. The product of z and y is added to x to obtain the output of a CSA module.

[0054] The discriminator model for line art coloring is as follows: Figure 5 As shown, its input includes a color image and feature maps extracted using the VGG16 feature extraction module. The discriminator model also uses the ResNeXtBlock module, but with fewer modules compared to the generator. The discriminator model inputs the generated image (either the line drawing or the real image) from the generator model of the embroidery line drawing coloring model into the discriminator network, and obtains the discrimination result after a series of convolutions. The loss function of the discriminator in the embroidery line drawing coloring module consists of a discrimination loss and a penalty loss.

[0055]

[0056] This invention also provides a computer device, including a processor and a memory connected to the processor; the memory is used to store a computer program, which is used to execute the coloring method for embroidery line drawings based on generative adversarial networks provided in any of the above embodiments.

[0057] The processor is used to call and execute computer programs stored in memory.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] In summary, this invention provides a method and computer device for coloring embroidery line art based on generative adversarial networks (GANs). A pre-constructed embroidery line art coloring model is used, comprising a pre-trained VGG16 feature extraction module, a generator, and a discriminator. The embroidery line art to be processed and its corresponding color cues are input into the trained coloring model, which outputs the target colored image. This invention utilizes generative adversarial networks to assist in the coloring of embroidery line art. By introducing a pre-trained VGG16 feature extraction module into the network, the generalization ability of the model is improved. Simultaneously, the introduction of a CSA module into the coloring model improves coloring accuracy and reduces color overflow issues.

[0060] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A method for coloring line drawings in embroidery based on generative adversarial networks, characterized in that, Includes the following steps: Collect color images of anime and appliqué embroidery, preprocess the collected color images, extract the corresponding line art and color hints from the preprocessed color images, and construct a training dataset from the color images and the corresponding line art and color hints. Construct a coloring model for piled embroidery line art; the coloring model for piled embroidery line art includes a feature extraction module, a generator with CSA module, and a discriminator; First, the coloring model of the piled embroidery line drawing was trained using the training dataset of animation, and then transfer training was performed on the training dataset of piled embroidery. The embroidery line drawing to be processed and the corresponding color prompt information are input into the trained embroidery line drawing coloring model, and the target embroidery coloring image is output. The generator for the coloring model of the embroidery line art includes a downsampling module, an upsampling module, and a CSA module, which are used to generate the target embroidery coloring image based on the embroidery line art and the corresponding color cues. The process is as follows: The single-channel embroidery line drawing is subjected to three convolution operations using a downsampling module to obtain a feature map. The color cues are convolved with a convolutional layer and then concatenated with the feature map. The downsampling module is then used to further downsample the image, with consecutive CSA modules in between to reduce color overflow using spatial attention and residual mechanisms. Following this are four upsampling modules, each starting with a convolutional submodule followed by Bn ResNeXtBlocks. Finally, a subpixel convolutional layer is used to improve the resolution of the feature map. After the upsampling operation on the right, the image passes through a convolutional layer and an activation layer to finally generate a color image. The CSA module's feature map processing flow includes: Assuming the feature map output by the final downsampling is x, the feature map y obtained by inputting x into a residual module is concatenated with the feature map extracted from the color image using the feature extraction module, and then convolved and activated using an activation function. After that, it is fed into another convolutional layer and activated using an activation function to obtain z. The product of z and y is added to x to obtain the output of a CSA module.

2. The method for coloring embroidery line drawings based on generative adversarial networks according to claim 1, characterized in that, Preprocessing includes dataset augmentation and size cropping, where dataset augmentation includes rotation and mirroring.

3. The method for coloring embroidery line drawings based on generative adversarial networks according to claim 1, characterized in that, The line art extraction uses the XDOG algorithm, and random settings are used during the extraction process. σ The value controls the thickness of the edges of the extracted line art.

4. The method for coloring embroidery line drawings based on generative adversarial networks according to claim 1, characterized in that, Color-coded information is extracted using a method that randomly selects pixels.

5. The method for coloring embroidery line drawings based on generative adversarial networks according to claim 4, characterized in that, The color cues are extracted during the training process. It is randomly determined whether a certain pixel needs color sampling. If so, a mask is superimposed on the color image to obtain the color cues.

6. The method for coloring embroidery line drawings based on generative adversarial networks according to claim 1, characterized in that, The discriminator of the embroidery line drawing color model takes a color image and a feature map extracted from the line drawing using a feature extraction module as input. The discriminator inputs the line drawing and the color image generated by the generator of the embroidery line drawing color model or the line drawing and the real image into the discriminator network, and obtains the discrimination result after a series of convolutions.

7. The method for coloring embroidery line drawings based on generative adversarial networks according to claim 1, characterized in that, Before inputting the embroidery line drawing to be processed into the trained embroidery line drawing coloring model, the XDOG algorithm is used to extract the line drawing from the embroidery line drawing, and adjustments are made as needed during the line drawing extraction process. σ The value is used to obtain line drawings similar to those in the training dataset.

8. A computer device, comprising a processor and a memory connected to the processor; the memory is used to store a computer program, the computer program being used to execute the coloring method for embroidery line drawings based on generative adversarial networks as described in any one of claims 1 to 7; The processor is used to call and execute computer programs stored in memory.

Images