A method and system for coloring a comic line drawing based on a color tag

Abstract

The application discloses a kind of based on color label's cartoon line drawing figure coloring method and system, it is related to image processing field, the method includes: obtaining target line drawing figure and corresponding text label;The target line drawing figure and corresponding text label are input line drawing figure coloring model, obtain the color image and color illustration of the target line drawing figure;Wherein, the determination method of the line drawing figure coloring model includes: using training dataset to train MulTag2Pix network, and the trained generator in the trained MulTag2Pix network is determined as the line drawing figure coloring model;The MulTag2Pix network includes generator and discriminator connected in turn;The generator includes: feature extraction module, style transfer module and refinement coloring module.The application can realize accurate coloring on multi-person line drawing art.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a coloring method and system for comic line art based on color tags. Background Technology

[0002] In comic book art creation, coloring is crucial because it enhances storytelling, conveys emotion, and increases visual appeal. Coloring line art requires consideration of appropriate color combinations, intricate textures and shading details, and often demands specialized expertise. However, manual coloring can fail to achieve the desired effect due to technical or creative limitations, or a lack of complete understanding of the artist's intent, leading to deviations. Furthermore, it incurs additional time and financial costs. Therefore, research into high-quality automated line art coloring is both necessary and highly attractive.

[0003] Existing automated line art coloring methods can be divided into two categories: non-user-guided and user-guided. Non-user-guided methods typically generate color images by perceiving the internal line art details and structural information of the image. However, they cannot meet users' customization needs and lack interactivity. User-guided coloring methods involve three main types of user prompts: doodles, reference images, and language.

[0004] Compared to doodles and reference images, language is the most direct tool for humans to accurately express their thoughts. It can convey expectations and requirements for the coloring result through simple textual descriptions or instructions. Therefore, the use of language (text labels) in line art coloring is a key area of ​​research.

[0005] Currently, various studies have been conducted on language-based coloring. Some literature uses text sentences describing the colors of specific objects as user prompts to colorize grayscale images; others use text sentences with color descriptions as user prompts to modify the colors of specific parts of an image; still others propose using text labels to color artistic line art images. This method has some effect on coloring single-character line art images, but it performs poorly when coloring artistic line art images of multiple characters. It not only struggles to correctly match text labels but also suffers from color bleeding and artifacts. In these existing methods, when the user-provided requirements are incomplete or insufficiently detailed, unreasonable coloring is often applied to parts of the character where no custom colors are defined, leading to color mismatches in the final generated color image.

[0006] It is evident that while some research exists on language-based image coloring, research on language-based line art coloring, especially multi-figure line art coloring, remains lacking. The relationship between text labels and corresponding pixels is highly non-linear, making it difficult to learn this association. The designed model needs to be able to capture this complex relationship and generate appropriate colors and textures based on the text labels. Therefore, the problem of text-label-based multi-figure line art coloring remains extremely challenging. Summary of the Invention

[0007] Based on this, embodiments of the present invention provide a coloring method and system for comic line art based on color tags, so as to achieve accurate coloring in multi-character line art.

[0008] To achieve the above objectives, embodiments of the present invention provide the following solutions:

[0009] A coloring method for comic line art based on color tags, comprising:

[0010] Obtain the target line art and its corresponding text labels; the target line art includes at least two anime characters; the text labels are color labels used to color the target line art.

[0011] The target line drawing and its corresponding text labels are input into the line drawing coloring model to obtain a color image and a color illustration of the target line drawing;

[0012] The method for determining the line drawing coloring model includes:

[0013] The MulTag2Pix network was trained using the training dataset, and the trained generator in the trained MulTag2Pix network was determined as the line drawing coloring model.

[0014] The training dataset includes: training line drawings and corresponding text labels, real color images, and real color illustrations; the MulTag2Pix network includes a generator and a discriminator connected in sequence; the generator includes: a feature extraction module, a style transfer module, and a thinning and coloring module;

[0015] The feature extraction module is used to extract features from the input line drawing to obtain feature extraction information; the feature extraction module, style transfer module, and fine coloring module include: edge segmentation information of different anime characters and feature information of the internal region of a single anime character;

[0016] The style transfer module is used to extract spatial and semantic features from the input text labels, and perform style transfer based on the extracted features, the semantic features, and the spatial features to obtain a stylized feature map.

[0017] The refinement and coloring module is used to decode based on the feature extraction information, the semantic features, and the stylized feature map to obtain a color image and a color illustration corresponding to the input line drawing.

[0018] Optionally, the feature extraction module includes: a first feature extraction unit, a second feature extraction unit, and a first feature fusion unit; the first feature extraction unit includes: a U-Net network; the second feature extraction unit includes: a ResNeXt-50 network;

[0019] The first feature extraction unit is used to extract features from the input line drawing to obtain a first feature; the first feature includes: edge segmentation information of different anime characters;

[0020] The second feature extraction unit is used to determine the skeleton prediction map based on the input line drawing, and to extract features from the skeleton prediction map to obtain the second feature; the second feature includes: feature information of the internal region of a single anime character;

[0021] The first feature fusion unit is used to fuse the first feature and the second feature to obtain the feature extraction information.

[0022] Optionally, the style transfer module includes: a CVT encoder, an AdaIN block, and a second feature fusion unit;

[0023] The encoder is used to extract spatial and semantic features from the input text labels;

[0024] The AdaIN block is used to perform style transfer based on the feature extraction information and the semantic features to obtain a third feature;

[0025] The second feature fusion unit is used to fuse the third feature and the spatial feature to obtain the stylized feature map.

[0026] Optionally, the CVT encoder includes: a first encoding unit and a second encoding unit;

[0027] The first encoding unit comprises: a fully connected layer and three convolutional layers connected in sequence; the first encoding unit is used to extract spatial features from the input text labels;

[0028] The second encoding unit includes four fully connected layers connected in sequence; the second encoding unit is used to extract semantic features from the input text labels.

[0029] Optionally, the AdaIN block includes three identical and sequentially connected sub-modules; each sub-module includes a sequentially connected convolutional layer, an adaptive instance normalization layer, and an activation function.

[0030] Optionally, the refined coloring module includes: a generation decoder and a guide decoder;

[0031] The generator decoder is used to decode based on the feature extraction information, the semantic features, and the stylized feature map to obtain a color image corresponding to the input line drawing;

[0032] The guided decoder is used to decode based on the stylized feature map to obtain a color illustration corresponding to the input line drawing.

[0033] Optionally, the generator decoder includes: four decoding layers and a convolutional layer connected in sequence; the decoding layer includes: a ConvNeXt block and a PixelShuffle layer connected in sequence.

[0034] Optionally, the guided decoder includes four convolutional layers connected in sequence.

[0035] The present invention also provides a coloring system for comic line art based on color tags, comprising:

[0036] The line art acquisition module is used to acquire a target line art image and its corresponding text labels; the target line art image includes at least two anime characters; the text labels are color labels used to color the target line art image.

[0037] The line art coloring model determination module is used to determine the line art coloring model;

[0038] The line art coloring module is used to input the target line art and the corresponding text label into the line art coloring model to obtain a color image and a color illustration of the target line art;

[0039] The line drawing coloring model determination module specifically includes:

[0040] The model training unit is used to train the MulTag2Pix network using the training dataset, and to determine the trained generator in the trained MulTag2Pix network as the line drawing coloring model.

[0041] The training dataset includes: training line drawings and corresponding text labels, real color images, and real color illustrations; the MulTag2Pix network includes a generator and a discriminator connected in sequence; the generator includes: a feature extraction module, a style transfer module, and a thinning and coloring module;

[0042] The feature extraction module is used to extract features from the input line drawing to obtain feature extraction information; the feature extraction information includes: edge segmentation information of different anime characters and feature information of the internal region of a single anime character;

[0043] The style transfer module is used to extract spatial and semantic features from the input text labels, and perform style transfer based on the extracted features, the semantic features, and the spatial features to obtain a stylized feature map.

[0044] The refinement and coloring module is used to decode based on the feature extraction information, the semantic features, and the stylized feature map to obtain a color image and a color illustration corresponding to the input line drawing.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] This invention implements coloring of multi-character comic line art based on color tags and MulTag2Pix. It integrates edge segmentation information of different anime characters, feature information of the internal region of a single anime character, spatial features of text tags, and semantic features of text tags to achieve coloring, thereby improving the accuracy of coloring in multi-character line art. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating a coloring method for comic line art based on color tags, provided in an embodiment of the present invention;

[0049] Figure 2 This is an overall flowchart of generating a colored image from a line drawing, provided in an embodiment of the present invention.

[0050] Figure 3 The diagram illustrates the specific implementation process of the coloring method for comic line art based on color tags provided in the embodiments of the present invention in a practical application.

[0051] Figure 4 This is a structural diagram of a CVT encoder provided in an embodiment of the present invention;

[0052] Figure 5 This is a structural diagram of the decoding layer provided in an embodiment of the present invention;

[0053] Figure 6 This is a structural diagram of the SECat module provided in an embodiment of the present invention;

[0054] Figure 7 This is a structural diagram of a coloring system for comic line art based on color tags, provided in an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0056] Most existing research on image colorization is based on GAN frameworks and can be divided into two categories based on user guidance: non-user-guided and user-guided colorization. Non-user-guided automatic colorization methods automatically learn the rules and patterns of comic book coloring to color artistic line art. However, these methods often fail to meet the personalized needs of users. With the increasing demand for customized coloring, user-guided colorization methods have been extensively explored. These methods can be divided into three categories: doodle-based image colorization, reference-based image colorization, and text-based image colorization.

[0057] (1) Image coloring based on graffiti.

[0058] A doodle-like approach (colored dots and strokes) is used as a user guide. Users are required to color the blank areas between lines by drawing short lines or dots with color. PaintsChainer and Style2Paints are two mature and evolving open-source solutions for this method. However, coloring a single image often involves multiple color gradients and brightness contrasts, requiring illustrators to have expertise in tonal variations, saturation contrasts, and grayscale contrasts. Therefore, using doodles as a user guide for each coloring detail is not feasible for non-professionals.

[0059] (2) Reference-based image coloring.

[0060] This method uses reference images with a similar style as input. For uncolored line art, this method does not require direct color guidance from the user. However, it requires pre-prepared reference images with the desired style, and the quality of the generated color images is limited by the selection and quality of the reference images. Moreover, when there are many instances of images to be colored, visual correspondences are difficult to determine.

[0061] Graffiti requires specialized knowledge and involves many details, making it very time-consuming. While using reference images as user guides does not require specialized knowledge, it can be difficult to find images that closely approximate the desired style and structure while achieving the desired coloring effect. Therefore, the method in this embodiment is based on image coloring using text labels.

[0062] (3) Text-based image coloring.

[0063] Because language excels at conveying user intent, it is frequently used as a user guide. For example, descriptive text is used to manipulate the style of content images, and then the visual content or emotional effect within the text (e.g., "water" or "calm") is learned for further style transfer. Some works use sentences describing object colors to colorize grayscale images. For instance, sentences describing instance-specific colors are used to colorize line art images of scenes, improving coloring accuracy. However, this model struggles with part-level information (such as "the character's eyes") and produces many mixed pseudo-colors and uncolored pixels when coloring anime characters. For example, using text labels as user guidance for line art coloring, proposing a two-step training method and the SECat module, can produce seemingly reasonable results in single-character line art coloring, but performs poorly in multi-character line art coloring.

[0064] None of these studies addressed the coloring of multi-figure line art. Furthermore, they failed to consider that incomplete or inaccurate user coloring requests could lead to illogical color combinations that do not conform to user aesthetic standards.

[0065] The purpose of this invention is to provide a coloring method and system for comic line art based on color tags, achieving accurate coloring of multi-character line art based on color tags and MulTag2Pix. This invention proposes using text tags as auxiliary style cues to determine the overall style of multi-character line art, resulting in more natural and user-pleasing color images.

[0066] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] Example 1

[0068] See Figure 1 The coloring method for comic line art based on color tags in this embodiment includes:

[0069] Step 101: Obtain the target line art and the corresponding text labels; the target line art includes at least two anime characters; the text labels are color labels used to color the target line art.

[0070] Step 102: Input the target line drawing and its corresponding text labels into the line drawing coloring model to obtain a color image and a color illustration of the target line drawing. Both the color image and the color illustration can be referred to as a colorized image.

[0071] The method for determining the line drawing coloring model includes:

[0072] The MulTag2Pix network was trained using the training dataset, and the trained generator in the trained MulTag2Pix network was determined as the line drawing coloring model.

[0073] The training dataset includes: training line drawings and corresponding text labels, real color images and real color illustrations; the MulTag2Pix network includes a generator and a discriminator connected in sequence; the generator includes: a feature extraction module, a style transfer module and a thinning and coloring module.

[0074] The feature extraction module is used to extract features from the input line drawing to obtain feature extraction information. The feature extraction information includes edge segmentation information for different anime characters and feature information of the internal regions of a single anime character. The style transfer module is used to extract spatial and semantic features from the input text labels, and perform style transfer based on the feature extraction information, the semantic features, and the spatial features to obtain a stylized feature map. The thinning and coloring module is used to decode based on the feature extraction information, the semantic features, and the stylized feature map to obtain a color image and a color illustration corresponding to the input line drawing.

[0075] This embodiment uses simple text labels instead of long text descriptions to generate high-quality color images, thus significantly reducing workload. The overall process of generating a colorized image from a line drawing is as follows: Figure 2 As shown, Figure 2 The image on the left is a line drawing, the image on the right is the generated colored image, and the content in the middle box represents the text label.

[0076] In one example, the feature extraction module includes: a first feature extraction unit, a second feature extraction unit, and a first feature fusion unit; the first feature extraction unit includes: a U-Net network; the second feature extraction unit includes: a ResNeXt-50 network.

[0077] The first feature extraction unit is used to extract features from the input line drawing to obtain a first feature; the first feature includes edge segmentation information of different anime characters.

[0078] The second feature extraction unit is used to determine the skeleton prediction map based on the input line drawing, and to extract features from the skeleton prediction map to obtain the second feature; the second feature includes: feature information of the internal region of a single anime character.

[0079] The first feature fusion unit is used to fuse the first feature and the second feature to obtain the feature extraction information.

[0080] In one example, the style transfer module includes a CVT (Color Variable Tag) encoder, an AdaIN block, and a second feature fusion unit. The encoder extracts spatial and semantic features from the input text tags. The AdaIN block performs style transfer based on the extracted features and the semantic features to obtain a third feature. The second feature fusion unit fuses the third feature and the spatial feature to obtain the stylized feature map.

[0081] In this example, the CVT encoder includes a first encoding unit and a second encoding unit. The first encoding unit includes a fully connected layer (FC) and three convolutional layers connected in sequence; the first encoding unit is used to extract spatial features from the input text labels. The second encoding unit includes four fully connected layers connected in sequence; the second encoding unit is used to extract semantic features from the input text labels.

[0082] The AdaIN block comprises three identical and sequentially connected sub-modules; each sub-module comprises a sequentially connected convolutional layer, an adaptive instance normalization layer, and an activation function.

[0083] In one example, the refinement coloring module includes a generator decoder and a guide decoder. The generator decoder is used to decode based on the feature extraction information, the semantic features, and the stylized feature map to obtain a color image corresponding to the input line drawing. The guide decoder is used to decode based on the stylized feature map to obtain a color illustration corresponding to the input line drawing.

[0084] In this example, the generating decoder includes four sequentially connected decoding layers and one convolutional layer; the decoding layers include a ConvNeXt block and a PixelShuffle layer connected in sequence. The guiding decoder includes four sequentially connected convolutional layers.

[0085] The following is a more specific embodiment, combined with Figure 3 The specific implementation process of the above-mentioned coloring method for comic line art based on color tags in practical applications is described in detail, and the effectiveness of the method in this embodiment is verified.

[0086] Step 1: Build the MulTag2Pix network based on a generator and a discriminator.

[0087] The generator is based on the U-Net network and is designed to generate high-resolution images. For example... Figure 3As shown, the generator first fuses the feature maps (32×32×256) from the convolutional layers with features extracted from the skeleton prediction map. Then, multiple decoder blocks are run to generate high-quality color insets. Specifically, each decoder block takes the feature map from the previous decoder block and convolutional layers of the same spatial dimensions from the U-Net network as input. Each decoder block is based on a pixel shuffle operation, an upsampling method used to reduce checkerboard artifacts.

[0088] (1) Construct a generator, which includes a feature extraction module, a style transfer module, and a fine coloring module.

[0089] In the feature extraction stage, the input is a monotonous line drawing (512×512×1) containing two or more people. After feature extraction by five convolutional layers (Conv+ReLU), a 32*32*256 tensor and a skeleton prediction map are obtained. This is then processed by the first two layers of a pre-trained SE-ResneXt (e.g., ResNeXt-50) tensor, resulting in another 32*32*256 tensor. These two tensors are then fused. The former provides edge segmentation information for different figures, while the latter provides feature information for the internal regions of a single figure. The fusion of these two feature information segments different figures and provides information about the internal regions of each figure.

[0090] Based on the rich feature extraction information for each individual during the feature extraction stage, a CVT encoder is used in the style transfer stage. (See [link to documentation]). Figure 4 The input text labels are embedded into a latent space. It has two outputs: the first layer (first coding unit) outputs the spatial features of the text labels (32*32*64) after passing through a fully connected layer (FC) and three convolutional layers (convolutional layers are better at handling overhead and extracting spatial features than fully connected layers). The second layer (second coding unit) outputs the latent style of the image (1*1*64) after passing through four FC layers, which adjusts the image colors. The first, the spatial features of the text labels (32*32*64), will be merged with the output tensor (32*32*256) of the style transfer module before the thinning and colorization stage, serving as input for the thinning stage and providing rich spatial feature information for the color labels in the next stage. The second is the output of the latent style, which is... Figure 3 and Figure 2It can be seen that, on the one hand, as input to the AdaIN Block, it adjusts and controls the overall color and style of the image during the style transfer stage, performing the initial colorization. On the other hand, as input to the detail stage, it adjusts the weights of each decoding layer (ConvNeXt block + Pixelshuffle) through the SECat module in the decoder, making the network pay more attention to the shading areas, helping the network to achieve more accurate colorization, and effectively suppressing the situation where the overall color matching of the image does not meet the user's requirements.

[0091] The style transfer stage initially colored the overall image. The refinement and colorization stage aims to further refine and accurately colorize the image. This stage has two branches: a generative decoder (used to generate the final color image) and a guided decoder (connected to the first decoding layer in the main decoder and generating color insets). The guided decoder provides a new loss path for intermediate feature maps, thus improving quality and helping to mitigate the vanishing gradient problem.

[0092] The following is a detailed introduction to each part.

[0093] 1. Feature extraction module of the generator

[0094] Traditional coloring models often focus on single-figure line art and perform poorly on multi-figure line art coloring tasks. These models are mainly limited by the following factors: insufficient spatial information, insufficient boundary recognition ability, generalization problems, and insufficient capture of key features.

[0095] To address these issues, a multi-figure line art coloring method combining skeleton prediction maps is proposed. First, existing segmentation methods, such as the U-net-based convolutional neural network architecture used for biomedical image segmentation, are employed to predict the skeleton map of the line art. This skeleton map is then used to enhance the model's spatial localization and boundary recognition capabilities, thereby achieving high-quality multi-figure line art coloring. Specifically, the skeleton prediction map plays two main roles: First, it provides spatial information and relative positions of multiple figures. By capturing the outlines and key points of figures, the skeleton prediction map accurately obtains their relative positions and boundaries. This spatial information helps the model better understand the relationships between different figures, enabling it to distinguish and identify different characters. This is crucial for multi-figure line art coloring tasks, as the model needs to associate text labels with corresponding figures. Second, it provides boundary and feature information during the coloring process. The region segmentation information in the skeleton prediction map helps the model better understand the boundaries and features of different regions in the image. Incorporating this information into the model effectively guides it to follow the structure of the object during color filling, avoiding problems such as color bleeding and artifacts. Furthermore, the key points in the skeleton prediction map help the model more accurately identify character features such as eyes and hair, thus achieving more accurate color filling.

[0096] More specifically, in the feature extraction stage, a pre-trained network, SE-ResneXt, is first used to extract high-level information from the skeleton image. Then, feature fusion is performed with intermediate features from the line art image to effectively capture spatial, boundary, and feature information in subsequent style transfer and refinement coloring stages, ultimately achieving high-quality multi-figure line art coloring.

[0097] In line art, manually assigning specific colors to each object individually can be extremely time-consuming. Furthermore, when users provide incomplete or imprecise requirements, the coloring results can be severely affected. To address this issue, a style transfer module is provided. For multi-figure line art, a style transfer-based approach is implemented. Specifically, learned user-guided text labels are used as auxiliary style cues to determine the overall style of the multi-figure line art image, generating a more natural color image that better matches the user's preferences.

[0098] 2. Generator's Style Transfer Module

[0099] (1) CVT encoder

[0100] A CVT encoder is used as the text label decoder. Its ability to extract features from text labels and capture rich color information and spatial distribution makes it suitable for solving the above problems.

[0101] In the CVT encoder structure, the functionality of the first layer is retained, with an output of (32*32*64). This output is fused with the feature map processed by the AdaIN block and used as input to the thinning and colorization stage to provide spatial features. The second layer's functionality is modified (output is 64*1*1) to learn latent style features present in the text labels. The learned features serve two purposes in the network structure: first, they transfer the style predicted by the text labels to the image to be colored, helping the network better grasp the overall style of the image; second, the SEcat module is used to adjust the intermediate feature maps, injecting style information into the weight rebalancing, which allows the network to focus more on locally colored regions.

[0102] (2) AdaIN block

[0103] To generate high-quality color images even when users provide incomplete information about their needs, the AdaIN Block was designed, incorporating the idea of ​​style transfer. It consists of three identical sub-modules, each containing a convolutional layer, an adaptive instance normalization (AdaIN) layer, and an activation function (ReLU). Specifically, the first sub-module includes Conv(1×1), AdaIN, and ReLU; the second sub-module includes Conv(3×3), AdaIN, and ReLU; and the third sub-module includes Conv(1×1), AdaIN, and ReLU, as shown below. Figure 3 The middle section is shown. This three-layer structure can learn different feature representations from low to high levels. This helps to find a balance between detailed and global information, enabling the model to learn how to naturally apply color distributions to different regions, resulting in a more natural final color image.

[0104] After feature extraction, a convolutional layer further extracts features. The output of the feature extraction part is 32*32*512, which is then fed into the AdaIN layer for style transfer. In the AdaIN block, the output (1*64) of the second layer in the CVT encoder is used as semantic features to guide the overall style of the image. The specific formula is as follows:

[0105]

[0106] Where f x Let represent the feature representation of the input artistic line drawing image x, and t represent the target semantic features. x and σ x They represent f respectively x The mean and standard deviation, μ t and σ t Let f represent the mean and standard deviation of t, respectively. AdaIN takes the input feature f as an example. xThe mean and standard deviation of the target semantic feature t are replaced with the mean and standard deviation of the target semantic feature t, thereby achieving style transfer.

[0107] Finally, the ReLU activation function introduces non-linear features, enabling the network to learn more complex feature representations. After processing by the AdaIN block, which consists of three sub-modules (Conv+AdaIN+ReLU), a stylized feature map with target semantic features is obtained, providing richer style information for the refinement and coloring stage.

[0108] 3. Generator's Refinement Shading Module

[0109] The refinement and colorization stage can be considered a decoder, consisting of five decoding layers. The first four decoding layers consist of ConvNeXt blocks and PixelShuffle, with the last being a convolutional layer. ConvNeXt is a convolutional network structure used to extract image features and generate feature representations, capturing multi-scale information and complex structures in images to help generate high-quality color images. The structure of the decoding layers is as follows: Figure 5 As shown, Figure 5 The structure of the SECat module shown is as follows: Figure 6 As shown.

[0110] Figure 5 The decoder layer (ConvNeXt block + SECat) structure shown employs the SECat module to recalibrate features to prevent the loss of color feature information in deeper layers. Subsequently, the output of this module undergoes PixelShuffle, upsampling the extracted low-resolution feature maps to high resolution, thus avoiding artifacts that may occur in traditional deconvolution operations. This contributes to generating clearer and more natural color images. To address the vanishing gradient problem, a guided decoder is added after the first decoder block of the network, which helps alleviate the problem to some extent. This guided decoder calculates the L1 loss between the generated coarse color image and the true color image, guiding the network to generate a color image with clearer contours and more saturated colors.

[0111] 4. Discriminator

[0112] The discriminator can use any existing discriminator structure. Any discriminator that can effectively distinguish between generated images and real images, keep the generated images consistent with the text labels, and predict the number of people in the input image is acceptable.

[0113] To better meet these task requirements, a fully connected layer can be added after the global average pooling (GAP) of the discriminator. This fully connected layer maps the features of the GAP layer to a new feature space, enabling the model to generate labels corresponding to the number of people in the input image. Through the fully connected layer, the model can perform weighted summation of various features, thereby capturing the relationships between them, improving the ability to predict the number of people in the input image, and maintaining consistency between the generated image and the input text labels.

[0114] Secondly, a sigmoid activation function is added after the fully connected layer. Mapping the output of the fully connected layer to a range between 0 and 1 allows it to be viewed as a probability value representing the number of people in the input image. This enables the model to more effectively predict the number of people in the input image and helps the network better distinguish between different people.

[0115] 5. Loss Function Design

[0116] This embodiment employs a two-step training method (i.e., segmentation training and coloring training). Originally, the coloring training step was designed for coloring a single character, and the loss function did not consider the impact of the number of characters on the network's learning. Therefore, in order for the network to distinguish different characters in line art and perform detailed coloring on the corresponding characters, this embodiment uses a label classification loss function L... cls An additional instruction was added to determine the number of characters. Simply put, segmentation training and colorization training differ only in one aspect: the colorization training discriminator adds prediction of input labels (color labels CVT) and CIT (labels for clothing without color), and calculates the L1 loss for the labels in the loss function to guide the network to accurately predict the labels. Segmentation training, on the other hand, predicts whether the final generated image is real or fake, without including information about the output color labels. Essentially, it adds a label classification loss Lcls to the segmentation loss function, called the colorization loss function (used for colorization training).

[0117] In this experiment, the first 10 iterations used segmentation training, meaning that the loss for label classification was not calculated, and the network was allowed to learn how to segment the input image first. After 10 iterations, colorization training was used, which is when the actual colorization training began, as the loss for label classification was calculated, and the output color labels began to affect the parameters of the network model.

[0118] Step 1: Segmentation Training. This step is crucial for providing semantic information to the artistic line art, as it enables the network to learn how to extract basic features such as edges and shadows. When the network learns to segment line art, the same loss function is used in Step 1 of Tag2pix segmentation training. The total loss for the segmentation step is as follows:

[0119] L D =-L adv ,

[0120] L G =L adv +λ rec L rec

[0121] Where L adv It is a reconstruction loss, L rec It is an adversarial loss, λ rec These are the corresponding weighting factors.

[0122] Step 2: Colorization Training. The total loss for this step is as follows:

[0123] L D =-L adv +λ cls L cls ,

[0124] L G =L adv +λ cls L cls +λ rec L rec

[0125]

[0126] Where x is the input art line drawing, y is the actual color illustration, and c i It is the CIT value, c t It's a label for the number of characters, c v It is the CVT (Color Text Label) value, G f It is a synthesized color image from the generator, D cls (c v ,c i ,c t |y) is each label corresponding to the binary classification of y, L cls It is the label classification loss, λ cls This represents the weighting ratios for the three categories: CIT (Clothing and Accessories), CVT (Color Text Labels), and Number of People. D L represents the loss of the discriminator. G This represents the generator's loss. E is used to calculate the expected loss for different random variables.

[0127] This indicates that for random variables y and c v c i c t The expectation, where -logD cls It is a function of variables, and the desired operation is performed on this function.

[0128] It helps measure model performance, guides the optimization process, and ensures that the model can correctly perform the label classification task.

[0129] The effectiveness of this embodiment was verified by experiments.

[0130] 1. Evaluation Indicators

[0131] (1) User survey

[0132] User feedback on image results is crucial because it provides intuitive and practical feedback on the quality of generated images and their performance in meeting user expectations. When attempting to solve problems involving visual aesthetics and creativity, computer vision algorithms and quantitative evaluation metrics may not fully capture all the details and aspects of generated image quality. Therefore, the user research evaluation criteria proposed in Tag2Pix are used to reflect the quality of generated color images. Scores are categorized into four areas: segmentation, naturalness, accuracy, and overall quality. This embodiment recruited 20 participants for this study and randomly selected 30 instances from the test set to form the research sample.

[0133] (2) Quantitative assessment

[0134] In addition to user research as an evaluation criterion, the Fréchet Inception Distance (FID) is used as an evaluation metric to comprehensively measure the overall quality of the generated images. FID measures the similarity between two sets of images from a statistical perspective of computer vision features, serving as a metric for calculating the distance between the feature vectors of the real and generated images. These visual features are extracted and computed using the Inception-v3 image classification model. The best-case FID score is 0.0, indicating that the two sets of images are completely similar. Lower scores mean higher similarity or closer statistical properties between the images. FID scores are commonly used to evaluate the quality of images generated by Generative Adversarial Networks (GANs), with lower scores associated with higher-quality images.

[0135] 2. Comparison with existing methods

[0136] Two segmentation fusion mechanisms applicable to shading models are proposed for comparison to improve color bleeding and artifacts during the shading process. The first, called "Direct Concatenation," connects the line drawing with its corresponding skeleton diagram as input to the generator, providing more accurate region segmentation for the shading network. The second, "Dual-branch," introduces an additional segmentation branch in the decoding stage, connecting the positional information of the segmentation and shading branches, adding more region segmentation information to the shading branch and reducing bleeding and artifacts. The method in this embodiment is compared with Tag2Pix and the two explicit segmentation fusion mechanisms mentioned above, respectively, applied to Tag2Pix models.

[0137] In the multi-character coloring task, the network proposed in this embodiment outperforms the baseline in both user research and the FID metric of the multi-character coloring task, as shown in Tables 1 and 2.

[0138] Table 1. FID Indicators for Coloring Line Artwork in Multi-Function Anime

[0139] Method FID(↓) Tag2Pix 20.89 DirectConcatenation 24.14 Dual-branch 22.56 MulTag2Pix 16.72

[0140] Table 2. Results of a user study on coloring line art for anime characters.

[0141]

[0142] In the single-character coloring task, Tables 3 and 4 show that the method of this embodiment achieved the best results in both user studies and the FID metric for single-character line art coloring. This highlights the strong generalization ability of the method proposed in this embodiment, performing excellently in both tasks. It was observed that the FID score for single-character line art was slightly higher than that for multi-character line art, while the FID score for single-character line art in the user study was slightly better than that for multi-character line art using the same method. This can be attributed to the relative simplicity of the single-character line art task. Since the complex structures of multiple characters do not need to be considered, accurate color matching in key color areas (such as hair, eyes, and clothing) is easier to achieve. Therefore, although the FID score for single-character line art may be slightly higher than that of multi-character line art, it achieved a higher score in the user study due to its better performance in key areas.

[0143] Table 3. FID Index of Single Anime Line Art Coloring

[0144] Method FID(↓) Tag2Pix 23.46 DirectConcatenation 27.87 Dual-branch 26.21 MulTag2Pix 21.11

[0145] Table 4. User Research Results on Individual Anime Line Art Coloring

[0146]

[0147] 3. Ablation test

[0148] In addressing the challenges of multi-character coloring tasks, this embodiment focuses on two key aspects. First, to address color inconsistencies caused by incomplete or inaccurate user coloring requirements, style transfer technology is introduced during the automatic coloring process. This ensures overall style consistency and the reasonableness of the final generated image. Second, to resolve color bleeding issues during the coloring process and improve the coloring effect in overlapping areas of multiple characters, a skeleton map is introduced as auxiliary input during the feature extraction stage. This provides more accurate character structure and region segmentation information, avoiding color inconsistencies within a given semantic region.

[0149] To further verify the effectiveness of the method proposed in this embodiment, ablation experiments were conducted by removing two key components from the model: the proposed AdaIN block and the use of a skeleton graph as additional guidance. As shown in Table 5, the ablation experiment results show that removing either the AdaIN block or the skeleton graph component leads to a decrease in the model's FID index. This demonstrates the effectiveness and importance of the AdaIN block and skeleton graph guidance in achieving high-quality colorization results. The presence of these two components in the model makes the color images more accurate and visually appealing, while their absence leads to poor performance, highlighting their crucial role in the overall method proposed in this embodiment.

[0150] Table 5 Comparison results of ablation experiments on FID index

[0151] Method FID(↓) Our 16.72 Ourw / oskeletonmap 20.37 Ourw / oAdaINBlock 17.80

[0152] In this embodiment, the proposed MulTag2Pix network is designed based on the U-Net network and CVT encoder, combined with style transfer, and can complete multi-character coloring tasks with high quality. This method can generate high-quality color images based on user-input text tags. Even when user requirements are not precise, it can ensure that the styles of multiple cartoon characters in the line art are reasonable and consistent with the overall style of the image. Experiments comparing the method of this embodiment with existing methods, trained and evaluated on a large-scale illustration dataset, show that the method of this embodiment not only achieves accurate coloring in multi-character line art but also outperforms existing methods in single-character line art coloring. The main contributions of the method of this embodiment are as follows:

[0153] (1) A dataset of anime characters consisting of three parts was constructed: colored illustrations (multiple characters and single characters), monochromatic line art, and text tags. Then, a network, MulTag2Pix, was designed to efficiently color line art, including multiple characters and single characters, according to the user's personalized coloring needs.

[0154] (2) Use skeleton prediction maps to provide rich character structure information and region segmentation information for the coloring network, reducing shadow artifacts and bleeding.

[0155] (3) A new module, AdaIN block, combining style transfer and artistic line drawing coloring is proposed. Even if the user's requirements are not detailed and comprehensive, this method can ensure that the style of each character is reasonable.

[0156] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a coloring system for comic line art based on color tags is provided below.

[0157] See Figure 7 The system includes:

[0158] The line art acquisition module 201 is used to acquire a target line art and its corresponding text labels; the target line art includes at least two anime characters; the text labels are color labels used to color the target line art.

[0159] The line art coloring model determination module 202 is used to determine the line art coloring model.

[0160] The line drawing coloring module 203 is used to input the target line drawing and the corresponding text label into the line drawing coloring model to obtain a color image and a color illustration of the target line drawing.

[0161] The line drawing coloring model determination module 204 specifically includes:

[0162] The model training unit is used to train the MulTag2Pix network using the training dataset, and to determine the trained generator in the trained MulTag2Pix network as the line drawing coloring model.

[0163] The training dataset includes: training line drawings and corresponding text labels, real color images and real color illustrations; the MulTag2Pix network includes a generator and a discriminator connected in sequence; the generator includes: a feature extraction module, a style transfer module and a thinning and coloring module.

[0164] The feature extraction module is used to extract features from the input line drawing to obtain feature extraction information; the feature extraction information includes: edge segmentation information of different anime characters and feature information of the internal region of a single anime character.

[0165] The style transfer module is used to extract spatial and semantic features from the input text labels, and perform style transfer based on the extracted features, the semantic features, and the spatial features to obtain a stylized feature map.

[0166] The refinement and coloring module is used to decode based on the feature extraction information, the semantic features, and the stylized feature map to obtain a color image and a color illustration corresponding to the input line drawing.

[0167] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0168] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A coloring method for comic line art based on color tags, characterized in that, include: Obtain the target line art and its corresponding text tags; the target line art includes at least two anime characters; The text labels are color labels used to color the target line drawing; The target line drawing and its corresponding text labels are input into the line drawing coloring model to obtain a color image and a color illustration of the target line drawing; The method for determining the line drawing coloring model includes: The MulTag2Pix network was trained using the training dataset, and the trained generator in the trained MulTag2Pix network was determined as the line drawing coloring model. The training dataset includes: training line drawings and corresponding text labels, real color images, and real color illustrations; the MulTag2Pix network includes a generator and a discriminator connected in sequence; the generator includes: a feature extraction module, a style transfer module, and a thinning and coloring module; The feature extraction module is used to extract features from the input line drawing to obtain feature extraction information; the feature extraction information includes: edge segmentation information of different anime characters and feature information of the internal region of a single anime character; The style transfer module is used to extract spatial and semantic features from the input text labels, and perform style transfer based on the extracted features, the semantic features, and the spatial features to obtain a stylized feature map. The refinement and coloring module is used to decode based on the feature extraction information, the semantic features, and the stylized feature map to obtain a color image and a color illustration corresponding to the input line drawing.

2. The coloring method for comic line art based on color tags according to claim 1, characterized in that, The feature extraction module includes: a first feature extraction unit, a second feature extraction unit, and a first feature fusion unit; the first feature extraction unit includes: a U-Net network; the second feature extraction unit includes: a ResNeXt-50 network; The first feature extraction unit is used to extract features from the input line drawing to obtain a first feature; the first feature includes: edge segmentation information of different anime characters; The second feature extraction unit is used to determine the skeleton prediction map based on the input line drawing, and to extract features from the skeleton prediction map to obtain the second feature; the second feature includes: feature information of the internal region of a single anime character; The first feature fusion unit is used to fuse the first feature and the second feature to obtain the feature extraction information.

3. The coloring method for comic line art based on color tags according to claim 1, characterized in that, The style transfer module includes: a CVT encoder, an AdaIN block, and a second feature fusion unit; The encoder is used to extract spatial and semantic features from the input text labels; The AdaIN block is used to perform style transfer based on the feature extraction information and the semantic features to obtain a third feature; The second feature fusion unit is used to fuse the third feature and the spatial feature to obtain the stylized feature map.

4. The coloring method for comic line art based on color tags according to claim 3, characterized in that, The CVT encoder includes: a first encoding unit and a second encoding unit; The first encoding unit comprises: a fully connected layer and three convolutional layers connected in sequence; the first encoding unit is used to extract spatial features from the input text labels; The second encoding unit includes four fully connected layers connected in sequence; the second encoding unit is used to extract semantic features from the input text labels.

5. The coloring method for comic line art based on color tags according to claim 3, characterized in that, The AdaIN block comprises three identical and sequentially connected sub-modules; each sub-module comprises a sequentially connected convolutional layer, an adaptive instance normalization layer, and an activation function.

6. The coloring method for comic line art based on color tags according to claim 1, characterized in that, The refined coloring module includes: a generation decoder and a guide decoder; The generator decoder is used to decode based on the feature extraction information, the semantic features, and the stylized feature map to obtain a color image corresponding to the input line drawing; The guided decoder is used to decode based on the stylized feature map to obtain a color illustration corresponding to the input line drawing.

7. The coloring method for comic line art based on color tags according to claim 6, characterized in that, The generator decoder includes four decoding layers and one convolutional layer connected in sequence; the decoding layers include a ConvNeXt block and a PixelShuffle layer connected in sequence.

8. The coloring method for comic line art based on color tags according to claim 6, characterized in that, The guided decoder comprises four convolutional layers connected in sequence.

9. A coloring system for comic line art based on color tags, characterized in that, include: The line art acquisition module is used to acquire target line art and corresponding text tags; the target line art includes at least two anime characters; The text labels are color labels used to color the target line drawing; The line art coloring model determination module is used to determine the line art coloring model; The line art coloring module is used to input the target line art and the corresponding text label into the line art coloring model to obtain a color image and a color illustration of the target line art; The line drawing coloring model determination module specifically includes: The model training unit is used to train the MulTag2Pix network using the training dataset, and to determine the trained generator in the trained MulTag2Pix network as the line drawing coloring model. The training dataset includes: training line drawings and corresponding text labels, real color images, and real color illustrations; the MulTag2Pix network includes a generator and a discriminator connected in sequence; the generator includes: a feature extraction module, a style transfer module, and a thinning and coloring module; The feature extraction module is used to extract features from the input line drawing to obtain feature extraction information; the feature extraction information includes: edge segmentation information of different anime characters and feature information of the internal region of a single anime character; The style transfer module is used to extract spatial and semantic features from the input text labels, and perform style transfer based on the extracted features, the semantic features, and the spatial features to obtain a stylized feature map. The refinement and coloring module is used to decode based on the feature extraction information, the semantic features, and the stylized feature map to obtain a color image and a color illustration corresponding to the input line drawing.

Images