Chinese drawing draft coloring method based on time sequence modeling and cross-modal fusion

Abstract

The invention belongs to the technical field of artificial intelligence generation, and particularly relates to a Chinese painting line draft coloring method based on time sequence modeling and cross-modal fusion, which comprises the following steps of: acquiring a multi-modal Chinese painting data set, and taking a line draft image, text description and a reference image in the image data set as original input; after preprocessing, dividing into a training set and a test set; the line draft image, the text description and the reference image serve as input, a Chinese line draft coloring network model is trained, and a training model is obtained; the Chinese line drawing draft coloring network structure comprises a text time sequence modeling module and a multi-modal feature fusion module, the text time sequence modeling module extracts semantic features through a text encoder and introduces position coding and bidirectional LSTM to construct a time sequence relation between words, the multi-modal feature fusion module fuses text and image features, and the text and image features are integrated to form a multi-modal feature fusion model. Detail optimization and feature enhancement are carried out; through the multi-modal data set, time sequence modeling and cross-modal fusion, automatic high-quality coloring of the Chinese drawing draft is achieved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence generation technology, specifically a method for coloring Chinese painting line drawings based on temporal modeling and cross-modal fusion. Background Technology

[0002] Traditional Chinese landscape painting has a unique artistic style, and its use of color and brushwork techniques contain profound cultural connotations.

[0003] In recent years, artificial intelligence, especially deep generative models, has made significant breakthroughs in areas such as image creation and style transfer, providing new technological opportunities for the digital preservation and innovative re-creation of traditional Chinese landscape paintings. From single-modal cues to multi-modal fusion, and from early generative adversarial networks (GANs) to the current mainstream diffusion models, these technological evolutions have laid a solid foundation for the intelligent generation and reproduction of Chinese paintings, revitalizing this traditional art form.

[0004] However, existing image coloring techniques have the following limitations when dealing with Chinese paintings with complex structures and strong artistic styles: (1) The semantic analysis of Chinese painting line drawings is difficult, which can easily lead to large-scale color mismatch and confusion in the coloring results guided by the reference image; (2) There are obvious differences between the reference image and the line drawing in terms of structural configuration and semantic distribution, and existing methods often lack a deep understanding of the large amount of relative positional relationships and object layout information contained in the text, making it difficult to achieve accurate semantic-driven coloring; (3) Current multimodal feature fusion mechanisms generally lack the ability to model local fine-grained information, which limits the fine-grained alignment effect between text and image at the spatial and semantic levels, thus affecting the overall coloring quality and artistic consistency.

[0005] Therefore, it is necessary to design a method for coloring detailed line drawings in Chinese painting to improve upon the above-mentioned problems. Summary of the Invention

[0006] To address the problems of existing technologies, this invention provides a method for coloring Chinese painting line drawings based on temporal modeling and cross-modal fusion, comprising the following steps:

[0007] Step 1: Construct the CLPainter multimodal dataset, including line art images I sketch and its corresponding text description I text and reference image I img And divide the training set and test set;

[0008] Step 2: Transfer the line art image I sketch and its corresponding text description I text and reference image I img As input, train a coloring model for Chinese painting line drawings;

[0009] The Chinese painting line drawing coloring model includes a text temporal modeling module and a multimodal feature fusion module; among them, the text temporal modeling module describes the text I... text Enhance semantic understanding of spatial descriptions through bidirectional LSTM and positional coding;

[0010] The multimodal feature fusion module performs a multimodal feature fusion on the reference image I. img and text description I text By fusing text and image features through a multi-head attention mechanism and multi-level dilated convolution, fine-grained feature alignment is achieved through detail optimization and feature enhancement.

[0011] Step 3: After completing the model training, input the test set into the trained Chinese painting line drawing coloring model to test the model's generation performance;

[0012] Step 4: Input the Chinese painting line drawing image to be colored into the trained Chinese painting line drawing coloring model to generate a color image with the style of traditional landscape painting.

[0013] Furthermore, in step 1, the CLPainter multimodal dataset is constructed, including: text descriptions are extracted using a two-stage process to obtain semantic and spatial information of objects in the image; simultaneously, line drawing images I are extracted from Chinese paintings. sketch It retains some grayscale details and enhances the reproduction effect of the line drawing style.

[0014] Furthermore, in step 1, the text description employs a two-stage extraction method:

[0015] Phase 1: Recognizing Chinese Painting Reference Images Using a Large Language Model img The main visual elements and their positional information in the image are represented as follows:

[0016] O=(c k ,b k )|k=1,2,...,K=Det(I img (1)

[0017] Det(·) represents the head detection operation, outputting K detected objects, c k Indicates the object category, b k This indicates the position information of objects in the image;

[0018] The second stage involves inputting the results from the first stage along with the Chinese painting line drawing image into a multimodal large language model to generate a detailed text description containing spatial and interactive relationships between objects. This process is represented as follows:

[0019]

[0020] Φ represents the construction of a virtual scene graph based on the detection results, ε represents the multimodal large language model, and ε represents the virtual scene graph construction based on the detection results. img This indicates image feature extraction.

[0021] Furthermore, in step 1, the method for extracting the line art image includes:

[0022] The colored landscape painting was processed to grayscale, and color inversion and blurring operations were used to enhance the contrast.

[0023] The pixel values ​​of the original grayscale image and the improved grayscale image are calculated using a pixel value calculation function to enhance the details and contrast in the dark areas and generate a grayscale line drawing that conforms to the style of line drawing.

[0024] Furthermore, the text temporal modeling module described in step 2 describes the text description I. text The timing relationship is established through bidirectional LSTM and position coding, specifically including:

[0025] For the input text description I text First, a Chinese text encoder is used to segment the text and extract semantic features F. text Furthermore, a positional encoding based on a sine function is added to adapt to input sequences of different lengths. The process is as follows:

[0026] E = F text +{p i ∈P|0≤i <T} (3)

[0027] Where T is the length of the input sequence. This is a sinusoidal position encoding matrix;

[0028] Next, a bidirectional LSTM network combined with positional encoding is used to construct a dynamic memory structure to capture long-range word dependencies. The process is represented as follows:

[0029]

[0030] Z t H represents the hidden output state of the bidirectional LSTM at time t. t M represents the hidden state of the forward LSTM at time t. t Let x represent the hidden state of the inverse LSTM at time t. t Let C be the input vector at time t. t This represents the memory unit at time t. This represents a vector concatenation operation;

[0031] Finally, by dynamically adjusting the feature fusion weights using a Sigmoid-based gating mechanism, finer-grained information transmission is achieved. The process is described as follows:

[0032] Etext =(1-G(Z) t ))⊙E+G(Z t )⊙Z t (5)

[0033] Where G represents the gating weight extraction

[0034] Furthermore, in step 2, the multimodal feature fusion module performs a multimodal feature fusion on the reference image I. img and text description I text The processing includes:

[0035] The multimodal feature fusion module includes a multi-head attention mechanism, a multi-level dilated convolutional pyramid, and a spatial gating network;

[0036] First, using text features F text As a key / value pair, and used to extract the reference image I by an image encoder. img Image feature information F img Image features F img As a query, cross-modal feature alignment is performed to dynamically capture textual semantic information related to the current image region globally, establishing a fine-grained mapping relationship between textual descriptions and local image regions, thus obtaining preliminary fusion features. This process can be represented as follows:

[0037]

[0038] It is a learnable parameter matrix, where h is the number of heads, and W is the number of heads. O It is the output projection matrix, and ∪ represents the result of splicing the heads in the last dimension;

[0039] Next, the preliminarily fused features are input into the constructed multi-level dilated convolutional pyramid to extract local and global features and perform feature refinement. The multi-level dilated convolutional pyramid extracts local detail features and global color distribution features by using convolutional kernels with different dilation rates.

[0040] Finally, a spatial attention map is learned through a spatial gating network to dynamically adjust the feature response and optimize the color distribution. This process can be represented as follows:

[0041] F ms =∑ r＝1,3,5 Γ r (F m (7)

[0042] 2D feature map, This represents the feature portion after removing the spatial representation, Υ represents the feature extraction operation, and λ represents the activation operation.

[0043] Furthermore, in the process of training the coloring model for Chinese painting line drawings, a joint loss function is used to balance color accuracy and structural fidelity;

[0044] The joint loss function includes the loss functions of L1 loss and structural similarity loss, where:

[0045] L1 loss: used to optimize pixel-level color reproduction. By measuring the pixel differences between the generated image and the real image, it restores the color details of the real image as much as possible while maintaining the overall style.

[0046]

[0047] in, with I t (i,j) represent the pixel values ​​of the generated image and the real image at position (i,j), respectively, and H and W are the height and width of the image;

[0048] Structural similarity loss: Used to maintain the structural consistency of the line art, measuring the consistency of the generated image with the real image in local structure, thereby enabling the model to retain the original structural features of the line art, avoid color overflow or edge blurring, and improve the overall perceptual quality.

[0049]

[0050] in, and This represents the local mean of the generated image and the real image. Let their covariance be... and Local standard deviation used for predicting and ground truth images.

[0051] The beneficial effects of this invention are:

[0052] (1) This invention constructs a detailed multimodal dataset of Chinese landscape paintings, adopts a two-stage large language model progressive annotation strategy, the text description includes object category, spatial location and color features, the average length of the text description is 60 characters, and grayscale line drawings are used instead of ordinary sketches, combining the features of grayscale and line drawings to make coloring more accurate.

[0053] (2) Set up a text feature time sequence construction module, introduce a position encoding model based on sine function to enhance the understanding of spatial relationships, and use bidirectional LSTM to capture the contextual semantic dependence of the text, so as to better understand the description of the spatial distribution of the line drawing in the text description and achieve more accurate coloring.

[0054] (3) A multimodal feature fusion module based on dynamic attention mechanism is set up to achieve fine-grained semantic alignment between text and image through cross-modal attention, and to enhance color feature expression by combining multi-level dilated convolutional pyramids. Finally, spatial gating network is used to achieve precise control of the coloring process, thereby improving the color accuracy and visual naturalness of automatic coloring of landscape paintings. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the Chinese painting line drawing coloring method based on temporal modeling and cross-modal fusion of the present invention.

[0056] Figure 2 This is a diagram of the text description extraction framework in this invention;

[0057] Figure 3 This is the overall framework diagram of the text temporal modeling model of this invention;

[0058] Figure 4 This is a framework diagram of the multimodal feature fusion module of the present invention;

[0059] Figure 5 This is a comparative analysis diagram of the method of the present invention and the image coloring method based on the Transformer structure;

[0060] Figure 6 This is a schematic diagram comparing the method of this invention with the method based on GAN structure;

[0061] Figure 7 This is a schematic diagram comparing the method of this invention with the diffusion model method;

[0062] Figure 8 This is an ablation result diagram comparing the coloring results of Chinese line drawings using the multimodal input of the model of this invention;

[0063] Figure 9 This is an ablation result image comparing the coloring results of Chinese line drawings using the text encoder of this invention;

[0064] Figure 10 This is an ablation result image comparing the coloring results of Chinese line drawings using the text temporal modeling and feature fusion module of this invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] refer to Figure 1 This invention provides a method for coloring Chinese painting line drawings based on temporal modeling and cross-modal fusion, including the following steps: Step 1: Construct the CLPainter multimodal dataset, including line drawing images I sketch and its corresponding text description I text and reference image I img And divide the training set and test set;

[0067] Step 2: Transfer the line art image I sketch and its corresponding text description I text and reference image I img As input, train a coloring model for Chinese painting line drawings;

[0068] The Chinese painting line drawing coloring model includes a text temporal modeling module and a multimodal feature fusion module; among them...

[0069] The text time series modeling module describes the text as I. text Enhance semantic understanding of spatial descriptions through bidirectional LSTM and positional coding;

[0070] The multimodal feature fusion module performs a multimodal feature fusion on the reference image I. img and text description I text By fusing text and image features through a multi-head attention mechanism and multi-level dilated convolution, fine-grained feature alignment is achieved through detail optimization and feature enhancement.

[0071] Step 3: After completing the model training, input the test set into the trained Chinese painting line drawing coloring model to test the model's generation performance;

[0072] Step 4: Input the Chinese painting line drawing image to be colored into the trained Chinese painting line drawing coloring model to generate a color image with the style of traditional landscape painting.

[0073] It should be noted that this invention constructs a multimodal dataset: combining grayscale line drawings, text descriptions, and reference images to enhance the model's learning ability and provide accurate semantic and style constraints for coloring;

[0074] The text temporal modeling module is set up to improve the model's semantic understanding of spatial relationships in the text through bidirectional LSTM and positional encoding.

[0075] A multimodal feature fusion module is set up: using multi-head attention mechanism and multi-level dilated convolution to achieve fine-grained feature alignment and improve the accuracy and naturalness of coloring;

[0076] Furthermore, in step 1, the CLPainter multimodal dataset is constructed, including: text descriptions are extracted using a two-stage process to obtain semantic and spatial information of objects in the image; simultaneously, line drawing images I are extracted from Chinese paintings.sketch Preserve some grayscale details and enhance the reproduction effect of the line drawing style;

[0077] The CLPainter includes 10,000 Chinese paintings of different colors from different dynasties, with a one-to-one correspondence. This method sets up 8,000 training sets and 2,000 test sets.

[0078] Furthermore, in step 1, a multimodal large language model is used, employing a two-stage text feature extraction method to extract text descriptions from the corresponding images, specifically including:

[0079] Phase 1: Recognizing Chinese Painting Reference Images Using a Large Language Model img The main visual elements and their positional information in the image are represented as follows:

[0080] O=(c k ,b k )|k=1,2,...,K=Det(I img (1)

[0081] Det(·) represents the head detection operation, outputting K detected objects, c k Indicates the object category, b k This indicates the position information of objects in the image;

[0082] The second stage involves inputting the results from the first stage along with the Chinese painting line drawing image into a multimodal large language model to generate a detailed text description containing spatial and interactive relationships between objects. This process is represented as follows:

[0083]

[0084] Φ represents the construction of a virtual scene graph based on the detection results, ε represents the multimodal large language model, and ε represents the virtual scene graph construction based on the detection results. img This represents image feature extraction. The overall framework of the text description extraction module is as follows: Figure 2 As shown:

[0085] It should be noted that, in order to comprehensively obtain textual descriptions that accurately depict the attributes and spatial locations of objects in the image, this paper adopts a two-stage text information extraction strategy. In the first stage, this method performs coarse-grained semantic parsing on the input image, focusing on identifying the main object categories contained in the image and their approximate spatial distribution within the image. This stage employs the DeepSeek-R1-LMM, a large language model with powerful visual understanding capabilities, to extract the target entities and their corresponding spatial location information, thereby providing a basic semantic framework for subsequent fine-grained modeling.

[0086] In the second stage, this method further introduces the multimodal large language model Qwen 3.0, and inputs the object information, location information, and original image obtained in the first stage into the model. In this stage, the model can not only fuse multimodal information to generate more contextually logical descriptive text, but also perform more refined modeling of the relative spatial relationships, hierarchical structure, and potential semantic or functional interactions between objects. The final generated text description has stronger structural and semantic integrity, providing high-quality semantic priors and spatial constraints for downstream text-driven image colorization tasks.

[0087] Furthermore, in step 1, the method for extracting the line art image includes:

[0088] The colored landscape painting was processed to grayscale, and color inversion and blurring operations were used to enhance the contrast.

[0089] The pixel values ​​of the original grayscale image and the improved grayscale image are calculated using a pixel value calculation function to enhance the details and contrast in the dark areas and generate a grayscale line drawing that conforms to the style of line drawing.

[0090] Furthermore, the text temporal modeling module described in step 2 describes the text description I. text The timing relationship is established through bidirectional LSTM and position coding, specifically including:

[0091] For the input text description I text First, a Chinese text encoder is used to segment the text and extract semantic features F. text Furthermore, a positional encoding based on a sine function is added to adapt to input sequences of different lengths. The process is as follows:

[0092] E = F text +{p i ∈P|0≤i <T} (3)

[0093] Where T is the length of the input sequence. This is a sinusoidal position encoding matrix;

[0094] Next, a bidirectional LSTM network combined with positional encoding is used to construct a dynamic memory structure to capture long-range word dependencies. The process is represented as follows:

[0095]

[0096] Z t H represents the hidden output state of the bidirectional LSTM at time t. t M represents the hidden state of the forward LSTM at time t. t Let x represent the hidden state of the inverse LSTM at time t. t Let C be the input vector at time t. tThis represents the memory unit at time t. This represents a vector concatenation operation;

[0097] Finally, by dynamically adjusting the feature fusion weights using a Sigmoid-based gating mechanism, finer-grained information transmission is achieved. The process is described as follows:

[0098] E text =(1-G(Z) t ))⊙E+G(Z t )⊙Z t (5)

[0099] Where G represents the gating weight extraction. The overall framework diagram of the text time series modeling module PBiLSTM is as follows: Figure 3 As shown;

[0100] The PBiLSTM module employs a strategy combining positional encoding and a bidirectional long short-term memory (BiLSTM) network to construct text feature representations with temporal semantics and spatial awareness. First, by introducing learnable positional encoding, the extracted text features are explicitly embedded with positional information, enabling the model to perceive the relative and absolute positions of words within a sentence when processing sequential data, thus facilitating spatial relationship modeling. Subsequently, the bidirectional LSTM structure is combined to simultaneously capture contextual semantic information from both forward and backward directions, enhancing the model's ability to understand inter-lexical dependencies and semantic integrity within the text.

[0101] Building upon the above, PBiLSTM further introduces a gated fusion mechanism based on the Sigmoid activation function to dynamically adjust the fusion weights between the original input features and the LSTM output. This mechanism effectively mitigates semantic shifts and information loss that may occur during deep modeling while preserving key information transmission. Simultaneously, this strategy enhances the model's adaptability to different textual semantic patterns, improving overall expressive flexibility and convergence stability during training.

[0102] Furthermore, in step 2, the multimodal feature fusion module performs a multimodal feature fusion on the reference image I. img and text description I text The processing includes:

[0103] The multimodal feature fusion module includes a multi-head attention mechanism, a multi-level dilated convolutional pyramid, and a spatial gating network;

[0104] First, using text features F text As a key / value pair, and used to extract the reference image I by an image encoder. img Image feature information F img Image features F imgAs a query, cross-modal feature alignment is performed to dynamically capture textual semantic information related to the current image region globally, establishing a fine-grained mapping relationship between textual descriptions and local image regions, thus obtaining preliminary fusion features. This process can be represented as follows:

[0105]

[0106] It is a learnable parameter matrix, where h is the number of heads, and W is the number of heads. O It is the output projection matrix, and ∪ represents the result of splicing the heads in the last dimension;

[0107] Next, the preliminarily fused features are input into the constructed multi-level dilated convolutional pyramid to extract local and global features and perform feature refinement. The multi-level dilated convolutional pyramid extracts local detail features and global color distribution features by using convolutional kernels with different dilation rates.

[0108] Finally, a spatial attention map is learned through a spatial gating network to dynamically adjust the feature response and optimize the color distribution. This process can be represented as follows:

[0109] F ms =Σ r＝1,3,5 Γ r (F m (7)

[0110] 2D feature map, This represents the feature portion after removing spatial representation, Υ represents the feature extraction operation, and λ represents the activation operation. The framework diagram of the Multimodal Feature Fusion Module (MFFEM) is shown below. Figure 4 As shown;

[0111] It should be noted that the MFFEM module employs a multi-head attention mechanism to fuse text and image features. Text and image data are respectively encoded by their respective encoders to extract embedding vectors. Then, text features are used as keys and values, and image features as queries, achieving cross-modal feature association and fusion under the multi-head attention mechanism. The fusion result undergoes multi-scale processing via a convolutional pyramid containing kernel sizes of 1, 3, and 5 to extract global and local features. Finally, a spatial gating mechanism is introduced, dynamically adjusting the feature response by learning the spatial attention map to enhance the representation of key regions and improve fusion accuracy and model robustness.

[0112] Furthermore, in the process of training the coloring model for Chinese painting line drawings, a joint loss function is used to balance color accuracy and structural fidelity;

[0113] The joint loss function includes the loss functions of L1 loss and structural similarity loss, where:

[0114] L1 loss: used to optimize pixel-level color reproduction. By measuring the pixel differences between the generated image and the real image, it restores the color details of the real image as much as possible while maintaining the overall style.

[0115]

[0116] in, with I t (i,j) represent the pixel values ​​of the generated image and the real image at position (i,j), respectively, and H and W are the height and width of the image;

[0117] Structural similarity loss: Used to maintain the structural consistency of the line art, measuring the consistency of the generated image with the real image in local structure, thereby enabling the model to retain the original structural features of the line art, avoid color overflow or edge blurring, and improve the overall perceptual quality.

[0118]

[0119] in, and This represents the local mean of the generated image and the real image. For covariance, and Local standard deviation used for predicting and ground truth images.

[0120] This invention presents a coloring method for traditional Chinese painting line drawings based on temporal modeling and cross-modal fusion, namely M3FPainter (Mutual Multi-Modal Fusion Guided Diffusion Model for Traditional Chinese Painting Colorization). This method can effectively color Chinese painting line drawings and generate color images with artistic beauty and stylistic consistency.

[0121] Performance Testing: First, qualitative comparison: Compared with models such as SANet (stochastic attention for deep convolutional neural networks), SaMam (style-aware state-space model for arbitrary image style transfer), Controlnet (adding conditional control to the text-based image diffusion model), and UCAST (a unified arbitrary style transfer framework based on adaptive contrastive learning), the M3FPainter model has significant advantages in preserving the details of Chinese painting line drawings, understanding the semantics of line drawings, and ensuring the coloring effect of line drawings;

[0122] To more specifically verify the generative performance of the M3FPainter method, it was compared with current mainstream image generation methods based on Transformer, Generative Adversarial Networks (GANs), and diffusion models. All methods were trained on the same dataset to ensure the fairness and comparability of the evaluation results.

[0123] A visual comparison of the coloring effects of Transformer-based image coloring methods on Chinese painting line drawings is presented, and the results are as follows: Figure 5 As shown, from Figure 5 It can be seen that the SANet method fails to fully model the color information in the reference image, resulting in a dull and monochromatic generated image with obvious artifacts, affecting the coloring quality. While StyTr2 retains some semantic information, it fails to utilize all color tones and ignores the traditional Chinese painting technique of using varying ink density to represent distance and depth, resulting in a flat and lacking-depth image. The SaMam method produces images with edge shadow artifacts and overly monochromatic colors, lacking aesthetic appeal. In contrast, the method of this invention can perform differentiated coloring on the semantic regions of the line drawing, fully integrating the color information in the reference image to generate a natural, harmonious, and richly colored image effect.

[0124] A visual comparison of the coloring effects of GAN-based image coloring methods on Chinese painting line drawings is presented, and the results are as follows: Figure 6 As shown, from Figure 6 It can be seen that UCAST is easily affected by the color distribution of the reference image, resulting in a heavy color and blurred structure in the generated result; AdAattN (Re-exploration of attention mechanisms in arbitrary neural style transfer) preserves the structure well, but has weak color transfer ability and a monotonous tone; SCFT (Coloring of reference sketch images based on enhanced self-reference and dense semantic correspondence) has rich colors, but the layout is messy and lacks coordination. In contrast, the method of this invention can accurately extract and transfer the colors of the reference image while maintaining the integrity of the line drawing structure, achieving dual consistency of structure and color, and generating a more artistic and visually appealing result.

[0125] A visual comparison of the coloring effects of diffusion-based image coloring methods on Chinese painting line drawings is presented, with the results as follows: Figure 7 As shown, from Figure 7It can be seen that IP-Adapter (a text-compatible image cue adapter based on the text-generated image diffusion model) has high color fidelity but weak structure preservation ability, resulting in messy and disorganized images; ControlNet has high structural fidelity but monotonous color expression and lacks a sense of hierarchy; AttenST (a training-free attention-driven style transfer framework based on a pre-trained diffusion model) achieves a certain balance between color and structure, but suffers from problems such as blurred edges and missing details, and is easily interfered with in the processing of Chinese painting elements such as seals and inscriptions. In contrast, the method of this invention can not only accurately identify and independently process cultural elements such as seals and inscriptions in Chinese paintings, but also achieve a natural, harmonious, layered, and culturally aesthetic image coloring effect while preserving the line drawing structure.

[0126] Second, quantitative comparison: To comprehensively evaluate the quality and visual effect of the generated results, two main evaluation metrics were adopted: image quality metrics and aesthetically related metrics. By introducing several representative quantitative evaluation methods, the aim was to systematically analyze the model output from both objective fidelity and subjective aesthetic dimensions. Specifically, for image quality evaluation, several mainstream metrics widely used in image reconstruction, super-resolution, and other tasks were selected, including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Learned Perceptual Patch Similarity (LPIPS), and Frachert Initial Distance (FID). These metrics quantify the similarity between the generated image and the target image from multiple perspectives, such as pixel-level error, structural consistency, perceptual similarity, and distributional differences.

[0127] On the other hand, to further explore the visual aesthetics of the generated results, this paper also introduces aesthetic-related indicators. Considering that aesthetic evaluation is usually highly subjective, a quantifiable auxiliary indicator—color level variance—is selected to initially measure the balance and richness of the image's color distribution. This indicator, by statistically analyzing the variance of the histograms of each color channel of the image, can reflect, to some extent, the overall tonal harmony and visual appeal of the image.

[0128] Table 1

[0129] method <![CDATA[PSNR (↑) ]]> <![CDATA[SSIM (↑) ]]> <![CDATA[LPIPS (↓) ]]> <![CDATA[FID (↑) ]]> <![CDATA[ΔE00 (↑) ]]> SANet 13.630 0.543 0.415 141.611 17.825 StyTr2 16.646 0.737 0.382 148.544 12.379 StyleFormer 13.251 0.571 0.390 171.583 18.084 Sketch2Photo 14.937 0.694 0.408 168.876 16.175 SaMam 13.053 0.521 0.382 147.190 19.264 This method 22.098 0.890 0.326 71.705 6.543

[0130] A comparison of the Transformer-based image coloring method with the coloring effect on Chinese painting line drawings was conducted, and the results are shown in Table 1. Table 1 shows that the method of this invention significantly outperforms existing technologies in all indicators, demonstrating outstanding overall performance. This result fully verifies the effectiveness of this method in color feature learning and line drawing image coloring tasks.

[0131] In terms of image reconstruction quality, our method achieves a PSNR improvement of approximately 32% compared to the second-ranked StyTr2, indicating less distortion and higher fidelity in the generated image. The SSIM score is also improved by approximately 20%, further demonstrating our superior performance in preserving image structure and detail.

[0132] In terms of perceptual similarity, this method achieves a 15% improvement in the LPIPS metric compared to the SaMam method, demonstrating image quality that better aligns with human visual perception. In the FID metric, which evaluates the diversity and realism of generated images, this method achieves a 50% improvement compared to SANet, significantly enhancing the naturalness and diversity of the generated results. Regarding color accuracy, the ΔE00 metric is used for evaluation, showing a 47% improvement over StyTr2, indicating its strong capabilities in color understanding and reproduction, enabling more accurate reproduction of target image tones and meeting the requirements for high-quality image colorization.

[0133] Table 2

[0134] method <![CDATA[PSNR (↑) ]]> <![CDATA[SSIM (↑) ]]> <![CDATA[LPIPS (↓) ]]> <![CDATA[FID (↑) ]]> <![CDATA[ΔE00 (↑) ]]> UCAST 12.004 0.505 0.476 235.146 22.428 SCFT 11.457 0.533 0.401 230.452 22.440 AdaAttN 17.759 0.822 0.344 223.213 11.159 This method 22.098 0.890 0.326 71.705 6.543

[0135] A comparison of the GAN-based image coloring methods for coloring Chinese painting line drawings was conducted, and the results are shown in Table 2. Table 2 shows that the method of this invention achieved the best results across all evaluation metrics, demonstrating comprehensive performance advantages. Specifically, in terms of image sharpness, generation quality, and color accuracy, this method improved by 24%, 68%, and 41% respectively in PSNR, FID, and ΔE00. Furthermore, in terms of structure preservation and perceptual consistency, the SSIM and LPIPS metrics also improved by 8% and 5% respectively. These results fully demonstrate that this method has significant advantages in image generation capability and color expression accuracy, possessing good application potential and versatility.

[0136] Table 3

[0137]

[0138]

[0139] A comparison of the coloring effects of the diffusion model-based image coloring method on Chinese painting line drawings is shown in Table 3. Table 3 shows that, in terms of the key indicator ΔE00 for evaluating color difference, the method of this invention improves by 31% compared to the best-performing IP-Adapter, indicating a significant advantage in color reproduction and color distribution rationality. The FID indicator shows a 36% improvement, indicating that the generated image is closer to the real image in terms of visual quality and diversity, possessing a stronger sense of naturalness and expressiveness. The SSIM and PSNR indicators show improvements of 18% and 25% respectively, further verifying the superior performance of this method in image structure preservation and distortion control. Furthermore, the LPIPS indicator also shows a 10% improvement, indicating that the generated image is more in line with human visual preferences in terms of perceptual quality. In summary, the method of this invention achieves comprehensive improvements in color accuracy, structural integrity, and visual perception consistency.

[0140] In summary, this method outperforms existing comparative methods across all key evaluation metrics, achieving a comprehensive lead in line art image coloring quality. It is particularly outstanding in structural fidelity and color perception consistency, providing a new technical approach and reference paradigm for line art coloring research.

[0141] To verify the effectiveness of each component of the model, three types of ablation experiments were set up, focusing on evaluating the impact of multimodal input, text encoder, text temporal modeling, and feature fusion modules on line art coloring and generation quality. The results were then analyzed qualitatively and quantitatively through visualization generation. Figure 8 , Figure 9 , Figure 10 These are qualitative results regarding the impact of the model's multimodal input, text encoder, text temporal modeling, and feature fusion modules on the coloring generation of Chinese line drawings. From... Figure 8 It's quite evident that with pure text input, the model, unable to obtain clear color cues, produces results with chaotic colors that deviate significantly from expectations. While the overall coloring of pure image input is more reasonable, local abnormal color spots still exist. For example, the trees in the lower left corner are incorrectly assigned multiple colors, which not only ruins the visual aesthetics but also violates the logic of painting. Figure 9 As can be seen, when using a general text encoder, the model struggles to accurately understand the semantic description of "tree," misinterpreting the trees in the lower right corner as a single entity, resulting in the entire area being filled with a single shade of green, which violates logical coloring rules. In contrast, the model in this method accurately captures the structured description of "tree" in the text, assigning different colors to the trunk and branches during the coloring process, resulting in a more natural and reasonable final effect, demonstrating stronger semantic understanding and detail control. Figure 10The model that did not incorporate text temporal modeling produced results with unnatural colors, abnormal brightness, and local overexposure, indicating that this module helps improve the model's understanding of the semantics of the line art and achieve more accurate coloring effects. In contrast, the model that did not employ a multimodal feature fusion strategy suffered a significant drop in quality due to simple addition and fusion, manifesting as black artifacts in blank areas, blurred details, chaotic coloring, and color interference in text areas, severely impacting the overall visual effect.

[0142] Table 4

[0143] method <![CDATA[PSNR (↑) ]]> <![CDATA[SSIM (↑) ]]> <![CDATA[LPIPS (↓) ]]> <![CDATA[FID (↑) ]]> <![CDATA[ΔE00 (↑) ]]> Remove text time series modeling module 18.651 0.822 0.343 100.483 9.232 Remove feature fusion module 18.947 0.842 0.333 87.285 8.744 Enter text only 14.079 0.697 0.367 115.760 16.979 Input image only 19.079 0.836 0.341 98.727 8.704 General text encoder 20.627 0.869 0.314 79.192 7.249 This method 22.098 0.890 0.326 71.705 6.543

[0144] To further quantify the effectiveness of the M3FPainter module, a series of image quality metrics were used for evaluation, as shown in Table 4. Experimental results show significant improvements across all test samples, fully validating the effectiveness of each part of the design. Specifically, when the model uses only text as guiding information, performance drops sharply. ΔE00 decreases by 62%, indicating that text has significant limitations in color expression and struggles to provide sufficient color guidance. Simultaneously, PSNR decreases by 56%, SSIM by 29%, LPIPS by 10%, and FID by 38%, demonstrating that without support from other modal information, the model struggles to understand semantics and generate high-quality images, highlighting the importance of multimodal fusion. When using only images as input, although the performance drop is relatively smaller, certain problems still exist, especially with ΔE00 decreasing by 27.5%, mainly attributed to uneven color distribution in the image feature space and the lack of textual semantic guidance.

[0145] When comparing the results with a text encoder that is not optimized for Chinese, it was found that although LPIPS performance improved slightly (by about 0.01), other metrics generally showed a slight decrease (fluctuation of no more than 10%). This indicates that an encoder designed for Chinese semantics helps to understand text descriptions more accurately and performs better in color filling and transition, thus generating a well-structured and visually natural coloring result.

[0146] Furthermore, after removing the text temporal modeling module, the ΔE00 metric decreased by 30%, and PSNR and FID decreased by 18% and 28%, respectively, indicating that this module played a crucial role in improving color accuracy and structural consistency. After removing the feature fusion module, FID and ΔE00 decreased by 18% and 26%, respectively, and PSNR decreased by 16%, further demonstrating the core value of this module in cross-modal feature integration. It effectively fuses image and text information, improving the overall generation quality and performance of the model.

Claims

1. A Chinese painting sketch coloring method based on timing modeling and cross-modal fusion, characterized by, The method comprises the following steps: Step 1: Constructing the CLPainter multi-modal dataset, including line drawing images I sketch and their corresponding textual descriptions I text and reference images I img and performing training set and test set division; Step 2: The line drawing image I sketch and its corresponding textual description I text and the reference image I img is input to train the Chinese painting line drawing coloring model; The Chinese painting line drawing coloring model comprises a text time sequence modeling module and a multi-modal feature fusion module; wherein the text time sequence modeling module, for text description I text Through bidirectional LSTM and position encoding, the semantic understanding of the spatial description is improved. A multi-modal feature fusion module, for a reference image I img and a text description I text The text and image features are fused through a multi-head attention mechanism and a multi-level cavity convolution, details are optimized and features are enhanced, and fine-grained feature alignment is realized. Step 3: After completing the model training, input the test set into the trained Chinese painting sketch coloring model to test the generation performance of the model; Step 4: input the Chinese painting sketch image to be colored into the trained Chinese painting sketch coloring model to generate a color image with a traditional landscape painting style.

2. The method of claim 1, wherein, In step 1, the CLPainter multimodal dataset is constructed, including: the text description adopts a two-stage extraction to obtain the semantic information and spatial position information of the objects in the picture; at the same time, the line drawing image I is extracted from the Chinese painting sketch , the remaining part of the gray details is retained, and the white drawing style restoration effect is enhanced.

3. The method of claim 2, wherein, In step 1, the text description adopts a two-stage extraction method: Phase 1: Recognizing Chinese Painting Reference Images Using a Large Language Model img The main visual elements and their positional information in the image are represented as follows: O = (c k ,b k )|k = 1,2,...,K = Det(I img ) (1) Det(·) represents a detection head operation, outputting K detection objects, c k represents the object class, b k represents the position information of the object in the picture; The second stage: input the first stage result and the Chinese painting sketch image into a multi-modal large language model to generate a detailed text description containing object spatial relationship and interaction relationship, and the process is represented as: denotes the virtual scene graph construction based on the detection result, Φ denotes the multi-modal large language model, and ε img denotes the image feature extraction.

4. The method of claim 2, wherein, In step 1, the extraction method of the sketch image includes: The color landscape painting is subjected to gray scale processing, and inverse color and blur operations are adopted to enhance the contrast; The pixel values of the original gray scale image and the improved gray scale image are calculated by a pixel value calculation function to enhance the dark details and contrast, and a gray scale sketch conforming to the white drawing style is generated.

5. The method of claim 1, wherein, The text time sequence modeling module described in step 2 models the text description I text The time sequence relationship is established through bidirectional LSTM and position encoding, specifically including: For the input text description I text , first, a text encoder for Chinese is adopted to segment it and extract semantic features F text , and position encoding based on the sine function is added to adapt input sequences of different lengths, the process of which is represented as: E = F text + {p i ∈ P | 0 < i < T} (3) where T is the input sequence length, is the sine position encoding matrix; Then, a bidirectional LSTM network combined with position encoding is used to construct a dynamic memory structure to capture long-range dependencies of words, and the process is represented as: Z t = Γ(x t , H t-1 , C t-1 ) ⊕ χ(x t , M t+1 , C t+1 ) (4) Z t denotes the output hidden state of the bidirectional LSTM at the t-th time, H t denotes the hidden state of the forward LSTM at the t-th time, M t denotes the hidden state of the backward LSTM at the t-th time, x t is the input vector at the t-th time, C t denotes the memory cell at the t-th time, and denotes the vector concatenation operation; Finally, the feature fusion weight is dynamically adjusted by a Sigmoid-based gating mechanism to realize more fine-grained information transmission, and the process is represented as: E text = (1 - G(Z t )) 0 E + G(Z t ) 0 Z t (5) Where G represents the gating weight extraction.

6. The method of claim 1, wherein, In step 2, the multi-modal feature fusion module processes the reference image I img and the text description I text , specifically including: The multi-modal feature fusion module includes a multi-head attention mechanism, a multi-level dilated convolution pyramid and a spatial gating network; First, the text feature F text is taken as the key / value, and the image feature information F img of the reference image I img is extracted by the image encoder, and the image feature F img is taken as the query, cross-modal feature alignment is performed, text semantic information related to the current image region is dynamically captured in the global range, a fine-grained mapping relationship between the text description and the image local region is established, and a preliminary fusion feature is obtained, and the process can be represented as: is a learnable parameter matrix, h is the number of heads, W O is the projection matrix of the output, ∪ denotes concatenating the results of each head in the last dimension; Then, the preliminarily fused features are input into the constructed multi-level dilated convolution pyramid to extract local and global features for feature refinement processing, and the multi-level dilated convolution pyramid extracts local detail features and global color distribution features by using convolution kernels with different dilation rates; Finally, a spatial attention map is learned by the spatial gating network to dynamically adjust the feature response and optimize the color distribution, and the process can be represented as: F ms =∑ r＝1,3,5 Γ r (F m ) (7) r represents different void fractions, Γ r represent convolution operations at different void fractions, represents converting the feature to a 2D feature map, represents the feature part after removing the space, Y represents the feature extraction operation, and λ represents the activation operation.

7. The method of claim 1, wherein, In the process of training the Chinese painting sketch coloring model, a joint loss function is adopted to balance the color accuracy and structure fidelity; The joint loss function includes an L1 loss and a structural similarity loss function, wherein: L1 loss: used to optimize pixel-level color restoration by measuring the pixel difference between the generated image and the real image, while maintaining the overall style and restoring the color details of the real image as much as possible; wherein with I t (i,j) denote the pixel values of the generated image and the real image at position (i,j) respectively, H and W are the height and width of the image; Structural similarity loss: used to maintain the consistency of the sketch structure, measure the consistency of the generated image and the real image in the local structure, so that the model retains the original structure characteristics of the sketch, avoids color overflow or edge blur, and improves the overall perceptual quality: wherein, and denotes the local mean of the generated image and the real image, is the covariance, and the local standard deviation of the predicted image and the real image.

Images