A cartoon line drawing coloring method based on wide attention and graph convolution optimization

Abstract

The present invention relates to a cartoon line drawing coloring method based on wide attention and graph convolution optimization. (1) Use the pre-trained VGG16 model to extract the features of the cartoon image and perform cosine similarity comparison, and select the top three images with similarity to form a one-to-one matching original image-reference image pair; (2) Construct a cartoon line drawing-coloring image feature matching model: First, the U-Net network extracts the multi-scale deep features of the cartoon line drawing and the coloring image respectively; secondly, the line drawing and coloring image features of each scale in the upsampling stage are input into the wide attention feature matching network for feature matching, and then the graph convolution module is used for overall optimization. (3) The matched features and line drawing images are input into the generation model to obtain the final coloring result. This method uses an end-to-end network model, which not only reduces the complexity of the project, but also effectively solves the matching problem of color features and semantic features between the original image and the reference image, and generates exquisite and high-resolution colored cartoon images.

Description

Technical Field

[0001] The present invention relates to a cartoon line drawing coloring method based on wide attention and graph convolution optimization, and belongs to the field of computer vision and image translation. Background Art

[0002] With the rise of digital painting, more and more artists prefer to use digital devices to create cartoon illustrations. Coloring is very important in the process of cartoon illustration drawing, but it usually takes a lot of labor and time. Therefore, how to use machine learning or deep learning methods to assist coloring has become a major research hotspot in recent years. Using intelligent algorithms to color cartoon illustrations is a research with great research significance and application value, whether for improving production efficiency or providing coloring references for artists.

[0003] At present, traditional cartoon line drawing coloring technology mainly relies on simple flood fill algorithms or mathematical algorithms such as energy functions for coloring. However, such methods cannot accurately color non-enclosed areas, and artists' hand-drawn line drawings often do not constitute closed areas, so traditional coloring methods often result in large areas of overflow or lack of color; and traditional coloring methods can only provide simple filling of a certain chroma, and cannot add appropriate shadows and highlight gradients according to the picture, making it difficult to meet the painting needs of artists.

[0004] In recent years, with the development of artificial intelligence and deep learning algorithms, a series of research results based on deep learning for cartoon line drawing coloring have also emerged in the field of line drawing coloring. It is mainly divided into two categories: automatic coloring algorithms and semi-automatic coloring algorithms based on user guidance. The former is colored by learning the texture gradient information of the grayscale image, but because the cartoon illustration area is dense and it is difficult to annotate the semantic information, the results obtained by the unsupervised automatic coloring algorithm often have serious color diffusion or color gap problems. The latter can be colored with the help of sparse color hint points or hint lines provided by the user, and can also be colored by learning the color semantic information of the reference image. Compared with the automatic coloring algorithm, the latter has a more flexible coloring effect and is more in line with the needs of artists.

[0005] PanZhang et al. proposed a general framework for image translation based on reference images. This method does not encode the style as a whole latent style vector, but rather aligns the features of images from different domains in a common intermediate domain, and then generates cross-domain images based on semantic correspondence. This avoids the problem of style convergence in image conversion and can flexibly generate images with corresponding relationships based on reference images.

[0006] Junsoo Lee et al. proposed a line drawing coloring method that is universal for natural images and cartoon images. This method obtains a self-reference image of the original image through image processing methods such as cropping and distortion, calculates the pixel-dense correspondence between the self-reference image and the original image, and fuses the features of the two to obtain the final coloring result.

[0007] CLPChen et al. proposed the concept of wide learning system to solve the problem of high computational cost and large amount of calculation in deep learning. The essence of wide network is random vector functional link neural network (RVFLNN). Unlike convolutional neural network, this network does not change the kernel of feature extractor through back propagation, but uses pseudo-inverse to calculate the weight of each feature node and enhancement node, so the training is fast and the computational complexity is low.

[0008] The graph convolutional network proposed by Thomas Kipf et al. uses the information of neighboring nodes as a supplement to the information of the current node, solving the problem that the translation variability of traditional convolutional neural networks is not applicable when processing non-matrix structured data. The essence of graph convolution is to extract the spatial features of topological graphs, standardize and augment the feature node matrix, and map the feature nodes to a high-dimensional subspace through nonlinear mapping, so as to enhance the network's expression ability and reduce the spatial loss of feature positions.

[0009] Yuxi Jin et al. proposed a grayscale image colorization method based on width learning. This method obtains the chromaticity value of each pixel through local width learning system training, and then refines the chromaticity map through the global width learning system to obtain the colorization result. This method has high training efficiency and can calculate new weights without retraining to generate the colorization result expected by the user. Summary of the invention

[0010] The purpose of the present invention is to address the problem that semantic and color information cannot be well matched in the current reference-based cartoon line drawing coloring. In order to solve the above technical problems, the technical solution adopted is: to provide a cartoon line drawing coloring method based on wide attention and graph convolution optimization.

[0011] The purpose of the present invention is achieved by the following measures:

[0012] A cartoon line drawing coloring method based on wide attention and graph convolution optimization. The specific steps are as follows:

[0013] Step 1), preprocessing of cartoon image dataset: To facilitate training, the cartoon colored images are sorted by cosine similarity to obtain a dataset in which the original image and the reference image match each other;

[0014] Step 2), construct a feature matching network: use U-Net to extract the features of the line drawing and the color reference image respectively, perform width attention feature matching at each scale in the upsampling stage, and use the GCN module to optimize the matching results after matching at each scale;

[0015] Step 3), build a coloring generation model: fuse the matched features of each scale with the line drawing image to generate the final coloring result.

[0016] Furthermore, the specific steps of step 1) are as follows:

[0017] Firstly, the image data is resized, and the xDoG and sketchKeras algorithms are used to extract the line drawings of the cartoon coloring pictures. Then, the deep features of all the cartoon coloring pictures are extracted through the VGG16 network, and the cosine similarity is used to detect the top three images with the highest similarity to each image, which are used as the reference images of the image, thus obtaining a data sequence in which the original image and the reference image match each other.

[0018] Furthermore, the specific steps of step 2) are as follows:

[0019] The preprocessed line drawings and colored reference images are respectively input into the residual U-Net network to extract multi-scale deep features. In the upsampling stage, the features of each scale are densely matched based on width attention, and the graph convolution module is used for optimization after each scale feature matching. Among them, the residual U-Net introduces jump connections in the encoder-decoder structure to splice the features together in various dimensions to form a thicker feature map; the residual connection retains the detailed information of different feature layers in the encoding part, reducing the feature processing time; the width learning system is an efficient learning system. Given the input data, more effective features can be extracted by enhancing the feature nodes; the graph convolution optimization module regards pixels as nodes, and the lines connecting the spatial coordinates between pixels as edges; the adjacent features are used as information supplements for the current node, avoiding the loss of local position information and enabling better feature matching;

[0020] 2-1) U-Net feature extraction network

[0021] The characteristic of U-Net network is that it integrates the features of different scales of downsampling and upsampling to learn the position information and semantic information. When training the network, the line drawing and the colored reference image are first input into the network respectively. The downsampling stage undergoes feature extraction at four resolution levels. The features of each scale pass through the convolution layer, normalization layer and two stacked residual blocks in turn. The bottleneck layer passes through the Squeeze-and-Excitation channel attention module after the two convolution layers, which can solve the loss problem caused by the different importance of different channels in the convolution pooling process. The upsampling stage integrates the features of the corresponding scales and outputs the features of four resolution levels. The output features are used to match the line drawing domain with the reference image domain.

[0022] 2-2) Width Attention Module

[0023] The network will perform the corresponding matching from the lowest resolution and use the matching results as the initial guidance for subsequent higher resolution matching; given the input feature X, the input layer By feature node Z i and enhanced node H j Composition, expressed as The mapping feature of the input data X and the transformation function φ, the i-th mapping feature is calculated as:

[0024] Φ(XW ei +β ei , i=1,2...n) (1)

[0025] Where φ is the transformation function of the i-th group of mapping features, and the weighting matrix W ei and the bias term β ei are parameters randomly initialized to the corresponding dimensions; these randomly generated mapping features are combined together and represented as Z n =[Z 1 , Z 2 , ..., Z n ]; the width attention will use Z composed of these mapped features n The enhancement layer is expanded by a nonlinear activation function; the corresponding formula is as follows:

[0026]

[0027] Where ξ is a nonlinear activation function, similar to the mapping feature, and is also a randomly generated parameter; the final width attention formula is expressed as:

[0028]

[0029] Weight coefficient It is obtained through pseudo-inverse calculation in the wide learning system; since the wide attention retains the random weight mechanism of RVFL and the calculation of weight coefficients does not require a time-consuming iterative process, it quickly maps the input sample to a more appropriate space to assist feature matching;

[0030] 2-3) Graph Convolution Optimization Module

[0031] The feature matching stage introduces graph convolution on the basis of the PatchMatch matching algorithm for optimization; the PatchMatch algorithm is based on the idea of ​​random sampling and provides a mechanism for rapid propagation in the entire image area to improve the search and matching efficiency according to the similarity of image regions; that is, the matching of the current block is improved by the position of the nearest neighbor (NNF) propagation result; specifically, as shown in the following formula, where N′ t It is the result of nearest neighbor propagation;

[0032] N′ t =propagation(N t , f x , f y ) (4)

[0033] However, the PatchMatch algorithm is prone to fall into local optimality and is overly dependent on the spatial smoothness assumption. The introduction of graph convolution extracts the relationship between targets, combines the neighbor features with the features of the current node, and improves the matching results by optimizing the loss of pixel-level position information. The graph convolution formula is expressed as follows:

[0034] F=σ(AXW) (5)

[0035] Where F is the feature after graph convolution, A is the adjacency matrix, X is the input feature node, W is the parameter that the network needs to learn, and σ is the nonlinear activation function.

[0036] Furthermore, the specific steps of step 3) are as follows:

[0037] The line drawing and matched features are input into the spatial adaptive deregularization layer, and the self-attention matrix M of the line drawing input is calculated. ij =x i ·x j The structural semantic information of the generated image is optimized and gradually upsampled to obtain the final result.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] Users do not need to do the tedious coloring prompt point marking. They only need to upload a line drawing and a reference image. The model can efficiently transfer the color and semantic information of the reference image based on the information of the reference image, and generate a coloring image with clear color restoration and high resolution. Not only can ordinary users who are not familiar with painting and coloring use this algorithm to obtain cartoon coloring images that meet their expectations, but it also greatly reduces the painting costs of artists, allowing them to create paintings more conveniently. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a framework diagram of the cartoon line drawing coloring method based on width attention and graph convolution optimization.

[0041] Figure 2 This is a diagram of the dataset preprocessing framework.

[0042] Figure 3 Network model structure diagram based on wide attention and graph convolution optimization.

[0043] Figure 4 It is a wide attention module structure.

[0044] Figure 5 are the network model training parameters.

[0045] Figure 6 It is the network loss curve.

[0046] Figure 7 It is the output result of the network training phase. DETAILED DESCRIPTION

[0047] The preferred embodiments of the present invention are described as follows in conjunction with the accompanying drawings:

[0048] See the overall process Figure 1 , a cartoon line drawing coloring method based on width feature graph convolution optimization, the example uses the kaggle cartoon coloring dataset and the selected Danbooru coloring dataset, the line drawing and the reference image are input into the feature matching network based on width attention and graph convolution optimization, and then the matching result is input into the coloring generation network to obtain the final coloring result.

[0049] (1) Cartoon dataset preprocessing

[0050] The cartoon dataset used is from the Kaggle open source dataset and a total of 28,000 images selected from the Danbooru data;

[0051] First, the collected cartoon data is uniformly processed into 512x512 size, and all cartoon colored images are divided into two sets using the Random function. The corresponding line drawings are extracted using the XDoG operator and the sketchKeras line drawing extraction algorithm respectively. Secondly, all cartoon colored images are input into the VGG16 network to extract features. The obtained feature set is sorted by cosine similarity to obtain the data sequence of the original image and the reference image matching each other. See the data set preprocessing process for details. Figure 2 , where the first data in each row of the original image-reference image matching sequence is the original image, and the last three data are the first three images with the highest cosine similarity compared with the original image.

[0052] (2) Constructing a feature matching network

[0053] The feature matching network framework adopts the U-Net structure. Figure 3 As shown in the figure, in the downsampling stage of the network, the features of the line drawing and the colored reference image are extracted respectively; in the upsampling stage of the network, the features of the line drawing and the reference image at each scale are input into the middle wide attention feature matching module for feature matching. See the wide attention module structure for details. Figure 4 , where the input layer is represented as Contains n groups of feature nodes Z i and m groups of enhanced nodes H j The feature node is the mapping feature generated from the input data X, and the enhancement node is the improved feature node. After feature matching, the graph convolution module is connected to optimize the matching results.

[0054] (3) Constructing a color generation network

[0055] The matched features are fused with the line drawing image, and the features of the reference image are finely transferred to the generated image through the self-attention space adaptive deregularization layer, and the final colorization result is generated by step-by-step upsampling. Figure 4 , the model adopts Figure 5 It takes about 7 days to train with the parameters shown in the figure. Figure 6 , the output results of the training phase can be found in Figure 7 .

[0056] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A cartoon line drawing coloring method based on wide attention and graph convolution optimization, characterized in that: The specific steps are as follows: Step 1), preprocessing of cartoon image dataset: To facilitate training, the cartoon colored images are sorted by cosine similarity to obtain a dataset in which the original image and the reference image match each other; Step 2), construct a feature matching network: use U-Net to extract the features of the line drawing and the color reference image respectively, perform width attention feature matching at each scale in the upsampling stage, and use the GCN module to optimize the matching results after matching at each scale; Step 3), constructing a coloring generation model: fusing the matched features of each scale with the line drawing image to generate the final coloring result; The specific steps of step 2) are as follows: The preprocessed line drawings and colored reference images are respectively input into the residual U-Net network to extract multi-scale deep features. In the upsampling stage, the features of each scale are densely matched based on width attention, and the graph convolution module is used for optimization after each scale feature matching. Among them, the residual U-Net introduces jump connections in the encoder-decoder structure to splice the features together in various dimensions to form a thicker feature map; the residual connection retains the detailed information of different feature layers in the encoding part, reducing the feature processing time; the width learning system is an efficient learning system. Given the input data, more effective features can be extracted by enhancing the feature nodes; the graph convolution optimization module regards pixels as nodes, and the lines connecting the spatial coordinates between pixels as edges; the adjacent features are used as information supplements for the current node, avoiding the loss of local position information and enabling better feature matching; 2-1) U-Net feature extraction network The characteristic of U-Net network is that it integrates the features of different scales of downsampling and upsampling to learn the position information and semantic information. When training the network, the line drawing and the colored reference image are first input into the network respectively. The downsampling stage undergoes feature extraction at four resolution levels. The features of each scale pass through the convolution layer, normalization layer and two stacked residual blocks in turn. The bottleneck layer passes through the Squeeze-and-Excitation channel attention module after the two convolution layers, which can solve the loss problem caused by the different importance of different channels in the convolution pooling process. The upsampling stage integrates the features of the corresponding scales and outputs the features of four resolution levels. The output features are used to match the line drawing domain with the reference image domain. 2-2) Width Attention Module The network will perform the corresponding matching from the lowest resolution and use the matching results as the initial guidance for subsequent higher resolution matching; given the input feature X, the input layer By feature node Z i and enhanced node H j Composition, expressed as The mapping feature of the input data X and the transformation function φ, the i-th mapping feature is calculated as: Φ(XW ei +b ei ,i=1,2...n) (1) Where φ is the transformation function of the i-th group of mapping features, and the weighting matrix W ei and the bias term β ei are parameters randomly initialized to the corresponding dimensions; these randomly generated mapping features are combined together and represented as Z n =[Z1,Z2,…,Z n ]; the width attention will use Z composed of these mapped features n The enhancement layer is expanded by a nonlinear activation function; the corresponding formula is as follows: Where ξ is a nonlinear activation function, similar to the mapping feature, and is also a randomly generated parameter; the final width attention formula is expressed as: Weight coefficient It is obtained through pseudo-inverse calculation in the wide learning system; since the wide attention retains the random weight mechanism of RVFL and the calculation of weight coefficients does not require a time-consuming iterative process, it quickly maps the input sample to a more appropriate space to assist feature matching; 2-3) Graph Convolution Optimization Module The feature matching stage introduces graph convolution for optimization based on the PatchMatch matching algorithm. The PatchMatch algorithm is based on the idea of ​​random sampling and provides a mechanism to quickly propagate across the entire image area to improve the search and matching efficiency according to the similarity of the image area. That is, the matching of the current block is improved by the position of the nearest neighbor NNF propagation result. The specific formula is as follows, where N′ t It is the result of nearest neighbor propagation; N′ t =propagation(N t ,f x ,f y ) (4) However, the PatchMatch algorithm is prone to fall into local optimality and is overly dependent on the spatial smoothness assumption. The introduction of graph convolution extracts the relationship between targets, combines the neighbor features with the features of the current node, and improves the matching results by optimizing the loss of pixel-level position information. The graph convolution formula is expressed as follows: F=σ(AXW) (5) Where F is the feature after graph convolution, A is the adjacency matrix, X is the input feature node, W is the parameter that the network needs to learn, and σ is the nonlinear activation function.

2. The cartoon line drawing coloring method based on wide attention and graph convolution optimization according to claim 1, characterized in that: The specific steps of step 1) are as follows: Firstly, the image data is resized, and the xDoG and sketchKeras algorithms are used to extract the line drawings of the cartoon coloring pictures. Then, the deep features of all the cartoon coloring pictures are extracted through the VGG16 network, and the cosine similarity is used to detect the top three images with the highest similarity to each image, which are used as the reference images of the image, thus obtaining a data sequence in which the original image and the reference image match each other.

3. The cartoon line drawing coloring method based on wide attention and graph convolution optimization according to claim 1, characterized in that: The specific steps of step 3) are as follows: The line drawing and matched features are input into the spatial adaptive deregularization layer, and the self-attention matrix M of the line drawing input is calculated. ij =x i ·x j The structural semantic information of the generated image is optimized and gradually upsampled to obtain the final result.

Images