Multi-scale color embedded Chinese painting style migration method

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a Chinese painting style migration method based on multi-scale color embedding, which comprises the following steps: acquiring a content image data set and a style image data set, and constructing a Chinese painting style migration network with multi-scale color embedding, including a content-aware position encoder, a hierarchical encoder and a multi-mode decoder; inputting the content image into a content perception position encoder, and extracting through linear mapping and a Transform encoder to obtain a global semantic feature sequence of the content image; inputting the style image into a hierarchical encoder, and generating image embedding fusing content and style through a progressive decoder in combination with the global semantic feature sequence; embedding and inputting the image with the fused content and style into a multi-mode decoder, and dynamically optimizing color embedding representation; according to the method, efficient fusion of the content image and the polychrome wood print style is realized through a deep learning technology, artistic expressive force and semantic consistency are achieved, and the quality and stability of polychrome wood print style migration are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a Chinese painting style transfer method based on multi-scale color embedding, which is particularly suitable for the transfer task of color woodcut style, and realizes efficient fusion of content images and color woodcut style through deep learning technology, and has artistic expressiveness and semantic consistency. BACKGROUND

[0002] Woodcut is an important part of traditional Chinese painting art. It carries rich regional culture and folk images, and embodies the unique pursuit of Eastern aesthetics in terms of virtual and real, color and structure. However, due to the changes in modern lifestyle and the influence of inheritance gap, woodcut is facing the challenges of cultural heritage protection and style inheritance.

[0003] In recent years, with the vigorous development of high-tech such as artificial intelligence, the research on style transfer and image generation of artistic works has gradually achieved outstanding results in the field of cultural heritage protection. These methods provide a basic framework for artistic style transfer and promote the development of image stylization technology.

[0004] However, although the existing image style transfer methods have achieved good results in various artistic styles, there are the following limitations in color woodcut transfer:

[0005] First, the color semantic modeling capability is limited, which is difficult to match the high-contrast and hierarchical color characteristics of color woodcut, resulting in color distortion of the generated image;

[0006] Second, the style consistency is insufficient, and problems such as color overflow and texture disorder are prone to occur;

[0007] Third, the global semantic feature extraction is insufficient, resulting in rigid fusion of content structure and style features;

[0008] Therefore, it is necessary to design a Chinese painting style transfer method to improve the above problems. SUMMARY

[0009] In order to solve the problems in the prior art, the present application provides a Chinese painting style transfer method based on multi-scale color embedding, comprising the following steps:

[0010] S1: Obtain content image dataset and style image dataset, and divide training set and test set; wherein the content image dataset is CelebA face dataset, and the style image dataset is color woodcut dataset;

[0011] S2: Construct a Chinese painting style transfer network based on multi-scale color embedding, including: a content-aware position encoder, a hierarchical encoder, and a multi-modal decoder;

[0012] S3: input the content image into the content-aware position encoder, extract the global semantic feature sequence of the content image through linear mapping and transformer encoder;

[0013] S4: input the style image into the hierarchical encoder, combine the global semantic feature sequence, and generate the image embedding fused with content and style through the progressive decoder;

[0014] S5: input the image embedding fused with content and style into the multi-modal decoder, dynamically optimize the color embedding representation, obtain the semantic-aligned color feature, and output the color-aligned style transfer image;

[0015] S6: adopt multi-dimensional color loss, content loss and style loss, train the network by training the total loss.

[0016] S7: input the content image to be converted into the trained network model to obtain the woodcut style image.

[0017] Further, in step S1, the acquisition of the style image dataset includes: using the keywords "black and white woodcut" and "black and white woodcut portrait woodcut" to crawl the woodcut dataset.

[0018] Further, in step S3, the processing process of the content-aware position encoder includes:

[0019] The content image is divided into fixed-size image blocks, an embedding sequence is generated through linear projection, spatial structure information is retained, a multi-layer Transformer encoder is used to capture global semantic dependency, each layer contains multi-head self-attention and feedforward network, and residual connection and layer normalization are used to stabilize training;

[0020] Wherein, the position encoding of the image block is calculated by spatial distance to realize the robustness support for different resolution inputs, and finally output the content feature sequence, which specifically includes:

[0021] First, given a content image and a woodcut style image with a height of H, a width of W, and a channel number of 3, and m is a constant 8, the height and width dimensions of the image are segmented;

[0022] Then, an embedding sequence feature is generated through linear projection, wherein By dynamically adjusting the block number, the robustness support for different resolution inputs is realized;

[0023] In order to obtain the structure information, when using the Transformer-based model, the position encoding is integrated into the input sequence, and the attention score between the i-th image block and the j-th image block is expressed as:

[0024]

[0025] where W q and W k are parameter matrices computed from the query and key, P i represents the i-th one-dimensional positional encoding.

[0026] In a two-dimensional scenario, the positional relationship of image patches at pixel (x i, y i ) and (x j, y j ) is calculated by spatial distance:

[0027]

[0028] where, The positional relationship of two image patches only depends on the spatial distance in this way, and in this way, different image scales will not affect the spatial relationship between two image patches:

[0029] P L = F pos (AvgPool n×n (ε c )) (3)

[0030] where AvgPool n×n is the average pooling function, F pos represents a 1x1 convolution operation of the learnable positional encoding function, P L follows the learnable positional encoding sequence ε c , and n is set to 18.

[0031] The CAPE of the image patch (i.e., P CA(x,y) ) is represented as:

[0032]

[0033] where a kl is the interpolation weight, and s represents the number of adjacent image patches.

[0034] Finally, P CA at the corresponding pixel position is added to its embedding feature ε c to represent the final feature embedding of a certain image patch.

[0035] Finally, the content sequence Z c = {S c1 + P cA1 , S c2 + P cA2 ,..., S cL + PcAL}, which is fed into a Transformer encoder, input sequence Z c is encoded into query (Q), key (K) and value (V), and the expression is:

[0036] Q=Z c W q ,K=Z c W k ,V=Z c W v (5)

[0037] wherein,

[0038] F MSA (Q,K,V)=Concat(Att1(Q,K,V),…,Att N (Q,K,V))W0 (6)

[0039] wherein, is a learnable parameter, N is the number of attention heads, represents the dimension of each attention head, and the encoded content sequence is obtained by applying a residual connection;

[0040] wherein, the calculation of the i-th attention head is defined as:

[0041]

[0042] Y c =F FFN (Y c ′)+Y c ′ (8)

[0043] wherein F FFN (Y c′ )=max(0,Y c′ W1+b1)W2+b2 a layer normalization operation is applied after each module, and finally a multi-layer Transformer encoder is used to extract global semantic features, outputting a content feature sequence

[0044] Further, in step S4, the processing process of the hierarchical encoder includes:

[0045] input style image into the hierarchical encoder, a backbone network is used to extract high-level style information of the style image, enhancing the context semantics and texture correlation;

[0046] The encoder skeleton is stacked by multiple down-sampling layers and Style modules, and the multi-scale feature extraction capability is used to learn hierarchical image representation.

[0047] Further, in step S5, the processing procedure of the multi-modal decoder includes:

[0048] stacked by a set of convolution modules, each module receives visual features and color queries as input; the multi-modal decoder is based on an improved Transformer decoder design, dynamically optimizes color embedding representation through a multi-scale color query mechanism, ensures color and semantic boundary alignment, and specifically includes:

[0049] First, the correlation between semantic representation and color embedding is established through the cross-attention layer, represented as:

[0050]

[0051] wherein, l represents the network layer index, is the color embedding vector of the lth layer; is the query matrix, are image features processed by the transformed functions f K (·) and f V (·), respectively; H l ,H w represent the spatial resolution of the image features, f Q, ,f K ,f V are linear mapping functions;

[0052] After the above cross-attention operation, the color embedding is further converted using the standard Transformer layer, and the process is defined as:

[0053] Z l ″=MSA(LN(Z l ′))+Z l ′,

[0054] Z l ″′=MLP(LN(Z l ″))+Z l ″,

[0055] Z l =LN(Z l ″′), (10)

[0056] wherein, MSA() represents multi-head self-attention, MLP() represents multi-layer perceptron, and LN() is layer normalization operation. Meanwhile, in the proposed CDB module, the cross-attention operation is performed before the self-attention operation; in order to balance the calculation complexity and representation ability, three different scale image features are selected, and the process of the multi-modal decoder is formalized, which can be defined as:

[0057] E c= δ (Z0, F1, F2, F3) (11)

[0058] where F1, F2, F3 and are visual features on three different scales;

[0059] The multi-scale features are used in the multi-modal decoder to model the relationship between the color query and the visual embedding; the color query is associated with semantic regions step by step, and a dynamic color embedding is output; the K-dimensional style semantic-color feature is mapped to the RGB space through a 1x1 convolution layer to generate a stylized result, denoted as:

[0060]

[0061] At this time, the convolution kernel parameters The K-channel dynamic is mapped to the RGB three channels to generate the final stylized result

[0062] Further, in step S6, the definition of the loss function includes: multi-dimensional color loss, content loss and style loss, and the total loss function is the weighted sum of the multi-dimensional color loss, the content loss and the style loss;

[0063] Where the multi-dimensional color loss includes color concentration constraint and color perception consistency constraint, which is used to enhance the color contrast and style consistency of the generated image, and the multi-dimensional color loss L MDC is denoted as:

[0064]

[0065] where μ c and σ c are defined as:

[0066]

[0067] The content loss defines the content perception loss as:

[0068]

[0069] where φ i represents the feature extracted from the i-th layer of the pre-trained VGG-19, and N is the number of layers.

[0070] The style loss defines the style perception loss L s as:

[0071]

[0072] where μ(·) and σ(·) represent the mean and variance of the extracted features, respectively.

[0073] The total loss is trained by the weight, and the total loss function defined by the present method is denoted as:

[0074] L sum = λ1L MDC + λ2L c + λ3L s (17)

[0075] wherein λ1, λ2, λ3 are weight parameters, and the weight parameters are respectively set to 1, 0.6, 0.9.

[0076] The beneficial effects of the present application are:

[0077] The present application is suitable for the migration task of color woodcut style, realizes efficient fusion of content image and color woodcut style through deep learning technology, has artistic expressiveness and semantic consistency, improves the quality and stability of color woodcut style migration, extracts multi-scale style features through a layered encoder, enhances the relevance of color and texture, so that the color of the generated image will not be distorted, the multi-modal decoder dynamically optimizes color embedding, reduces color overflow phenomenon, a multi-dimensional color loss function improves the color contrast and style consistency of the generated image, wherein the ConvNeXt module in the layered encoder pre-training network extracts style color features of different scales in the style image, enhances the context semantics and texture relevance, then is associated with the progressive decoder through a skip connection, guides and optimizes the color representation of the multi-modal decoder, introduces a multi-scale color query mechanism through the multi-modal decoder, learns the hierarchical color query mechanism in the multi-scale style features to optimize the color embedding representation, ensures the alignment of color and semantic boundary, enhances the color semantic consistency of the generated image, and reduces the color overflow phenomenon, a multi-dimensional color loss function enhances the color features of the generated image, promotes the color distribution of the generated image to conform to the artistic characteristics of color woodcut, such as clear color and high contrast, HieWoodNet learns hierarchical style representation to enable the model to effectively improve the ability of the model to extract rich color style features, reduces the problems such as color semantic distortion, low color contrast and chaotic texture distribution in the current color woodcut style migration task, and improves the color woodcut style migration effect. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 is the flow chart of the color woodcut style migration network method of the present application.

[0079] Figure 2 is the overall framework diagram of HieWoodNet of the present application.

[0080] Figure 3 (a) is the multi-scale feature extraction flow chart of Style encoder of the present application, and (b) is the attention mechanism interaction flow chart of ChromaNova module of the present application.

[0081] Figure 4is the schematic diagram of the data set of the application.

[0082] Figure 5 is the visualization result graph of the application and other comparative methods.

[0083] Figure 6 is the visualization result graph of the ablation experiment of each module of the application.

[0084] Figure 7 is the visualization comparison graph of the method of the application on different data sets. DETAILED DESCRIPTION

[0085] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0086] Reference Figure 1 The application provides a multi-scale color embedding Chinese painting style transfer method, comprising the following steps:

[0087] S1: Obtain a content image data set and a style image data set, and divide the training set and the test set; wherein the content image data set is a CelebA (CelebFaces Attributes Dataset, celebrity attribute face data set) face data set, and the style image data set is a color matching woodblock print data set;

[0088] S2: Construct a multi-scale color embedding Chinese painting style transfer network (HieWoodNet, HierStyle Multi-scale Semantic for Chinese Woodblock Painting Style Transfer Network), comprising a content-aware positional encoder (Content-Aware Positional Encoder), a hierarchical encoder (HierStyle), and a multi-modal decoder (ChromaNova);

[0089] S3: Input the content image into the content-aware positional encoder, and extract the global semantic feature sequence of the content image through linear mapping and transformer (a deep learning model architecture based on self-attention mechanism) encoder;

[0090] S4: Input the style image into the hierarchical encoder, combine the global semantic feature sequence, and generate a content and style fused image embedding through a progressive decoder;

[0091] S5: embedding the fusion content and style image into a multi-modal decoder (ChromaNova), dynamically optimizing the color embedding representation, obtaining semantic-aligned color features, and outputting a color-aligned style transfer image;

[0092] S6: training the network by training the total loss with multi-dimensional color loss, content loss, and style loss.

[0093] S7: inputting the content image to be converted into the trained network model to obtain a woodcut print style image.

[0094] Further, with reference to Figure 4 In step S1, the acquisition of the style image dataset includes: using the keywords "black and white woodcut print" and "black and white woodcut portrait woodcut print" to crawl the woodcut print dataset. Among them, the works of famous woodcut print artists such as Lu Xun and Wang Qinglong are collected as the CelebA face dataset. In the experiment, the dataset of woodcut print is selected as the style image dataset, and the same number of CelebA face dataset is selected as the content image dataset, with 2000 training sets and 100 test sets. In addition, in order to verify the generalization of the experiment, the animation dataset, the natural landscape dataset and the landscape painting dataset are tested respectively.

[0095] Further, with reference to Figure 2 In step S3, the processing process of the content-aware position encoder includes:

[0096] The content image is divided into fixed-size image blocks (Patch), an embedding sequence is generated by linear projection, the spatial structure information is retained, a multi-layer Transformer encoder is used to capture global semantic dependencies, each layer contains multi-head self-attention and feedforward network, and residual connection and layer normalization are used to stabilize training;

[0097] Among them, the position encoding of the image block is calculated by spatial distance, which realizes the robustness support for different resolution inputs, and finally outputs the content feature sequence, which specifically includes:

[0098] First, given a content image and a woodcut print style image with a height of H, a width of W, and a channel number of 3, the height and width dimensions of the image are segmented according to m (the default value is 8);

[0099] Then, an embedding sequence feature is generated by linear projection, where By dynamically adjusting the block number, the robustness support for different resolution inputs is realized;

[0100] To obtain structural information, positional encodings are incorporated into input sequences in the Transformer-based model, the attention score between the ith image patch and the jth image patch is denoted as:

[0101]

[0102] where W q and W k are parameter matrices computed from the query and key, P i represents the ith one-dimensional positional encoding.

[0103] In a two-dimensional scene, the positional relative relationship between the image patch at pixel (x i, y i ) and the image patch at (x j, y j ) is calculated by the spatial distance:

[0104]

[0105] where, The positional relative relationship between the two image patches only depends on the spatial distance, in this way, different image scales will not affect the spatial relationship between the two image patches:

[0106] P L = F pos (AvgPool n×n (ε c )) (3)

[0107] where AvgPool n×n is the average pooling function, F pos represents a 1x1 convolution operation of the learnable positional encoding function, P L follows the learnable positional encoding sequence ε c , and n is set to 18 in this experiment;

[0108] The CAPE of the image patch (i.e., P CA(x,y) ) is denoted as:

[0109]

[0110] where a kl is the interpolation weight, and s represents the number of adjacent image patches.

[0111] Finally, P CA at the corresponding pixel position is added to its embedded feature ε c to represent the final feature embedding of an image patch.

[0112] Finally, the content sequence Z c = {S c1+P cA1 ,S c2 +P cA2 ,...,S cL +P cAL}, feed into the Transformer encoder, input sequence Z c are encoded into query (Q), key (K) and value (V), the expression is:

[0113] Q=Z c W q ,K=Z c W k ,V=Z c W v (5)

[0114] wherein,

[0115] wherein, are learnable parameters, N is the number of attention heads, denotes the dimension of each attention head, and the encoded content sequence is obtained by applying a residual connection;

[0116] wherein, the calculation of the i-th attention head is defined as:

[0117]

[0118]

[0119] wherein F FFN (Y c′ )=max(0,Y c′ W1+b1)W2+b2 a layer normalization operation is applied after each module, and finally a multi-layer Transformer encoder is used to extract global semantic features, outputting a content feature sequence

[0120] In order to meet the input requirements of the progressive decoder, the content features and style features are spliced, which needs to ensure that the spatial size and the number of channels are aligned with the style feature map. Specifically, the content sequence feature Y c output by the Transformer is reshaped into a spatial feature map, and then two downsampling operations are performed to make it consistent with the size of the style feature map, obtaining a content feature map. This feature map retains the spatial position information and global semantic information of the content image;

[0121] Further, with reference to Figure 2 and Figure 3 (a), the processing process of the hierarchical encoder in step S4 includes:

[0122] input style image Into the hierarchical encoder, the backbone network is used to extract high-level style information of the style image, and the context semantics and texture correlation are enhanced;

[0123] The encoder skeleton is composed of multiple down-sampling layers and Style modules, and the multi-scale feature extraction capability is used to learn hierarchical image representation.

[0124] Among them, the backbone network generates four intermediate feature maps F s1 ,F s2, F s3, F s4 , whose resolutions are The first three feature maps are transmitted to the progressive decoder through the skip connection. The channel number of F c is adjusted by using 1×1 convolution, so that it matches the channel number of the highest level style feature map Fs4. Finally, the style feature map F s4 and the content feature map F c are spliced along the channel dimension, and are used as the main input of the progressive decoder.

[0125] In the decoding stage, the decoder is composed of four stages, which gradually improve the image resolution. Each stage contains an up-sampling layer and a shortcut layer. Specifically, unlike the previous methods using deconvolution or interpolation, the present application uses pixel reorganization as the up-sampling layer, which avoids the chessboard artifact through sub-pixel rearrangement and improves the reconstruction quality. This layer will rearrange the low-resolution feature map of shape into a high-resolution feature map of shape (h, w, c). The shortcut layer uses convolution to integrate the features of the corresponding stage of the style encoder through the shortcut connection.

[0126] The complete image feature pyramid is captured through the process of progressive step-by-step up-sampling. These multi-scale features are further used as the input of the multi-modal decoder to guide the optimization of the color query. The final output of the progressive decoder is the image embedding , whose spatial resolution is the same as the input image. This architecture provides high-quality color feature representation with semantic accuracy and multi-scale style distribution for subsequent color decoding through multi-stage style feature interaction and global content semantic information fusion.

[0127] Further, with reference to Figure 2 , Figure 3 (a) and Figure 3 (b), the processing process of the multi-modal decoder in step S5 includes:

[0128] It is stacked by a group of convolution modules, each module receives visual features and color queries as input; the multi-modal decoder (ChromaNova decoding module) is based on an improved Transformer decoder design, and its structure is as followsFigure 2 The present application proposes a learnable color embedding memory for storing color representation sequences: where K denotes the number of embeddings, and C is the channel dimension. These color embeddings are initialized to zero at the beginning of training and are used as color queries in the first multi-modal decoder.

[0129] The color embedding representation is dynamically optimized through a multi-scale color query mechanism to ensure that the color and semantic boundaries are aligned, which specifically includes:

[0130] First, the correlation between the semantic representation and the color embedding is established through a cross-attention layer, which is represented as:

[0131]

[0132] where l represents the network layer index, is the color embedding vector of the l-th layer; is the query matrix, are image features processed by the transformed functions f K (·) and f V (·), respectively; H l ,H w denote the spatial resolution of the image features, f Q, ,f K ,f V is a linear mapping function;

[0133] After the above cross-attention operation, the color embedding is further converted using a standard Transformer layer, and the process is defined as:

[0134] Z l ″=MSA(LN(Z l ′))+Z l ′,

[0135] Z l ″′=MLP(LN(Z l ″))+Z l ″,

[0136] Z l =LN(Z l ″′), (10)

[0137] where MSA() denotes multi-head self-attention, MLP() denotes multi-layer perceptron, and LN() denotes layer normalization operation; meanwhile, in the proposed multi-modal decoder (CDB, ChromaNova Decoder Block), the cross-attention operation is performed before the self-attention operation; in order to balance the computational complexity and the representation ability, three different scales of image features are selected, specifically, the intermediate visual features with downsampling rates of 1 / 16, 1 / 8 and 1 / 4 generated by the progressive decoder are used in the multi-modal decoder. Then the modules are grouped, and each group contains 3 convolution modules (CNB, ConNeXt Block), and in each group, the multi-scale features are sequentially input into each CNB. The model will repeat the execution of the group M times in a loop, and the multi-modal decoder is composed of 3M CNBs.

[0138] The process of the multi-modal decoder is formulated, and the multi-modal decoder can be defined as:

[0139] E c = δ (Z0, F1, F2, F3) (11)

[0140] where F1, F2, F3 and are visual features at three different scales;

[0141] The multi-scale features are used in the multi-modal decoder to model the relationship between the color query and the visual embedding; the color query is gradually associated with the semantic region, and the dynamic color embedding is output; the multi-scale feature modeling ensures that the color and semantic boundaries are aligned, so that the color embedding is more sensitive to semantic information, and thus can more accurately identify the semantic boundary and ensure color semantic consistency.

[0142] where the feature fusion aims to combine the content-aware positional encoder and the output of ChromaNova to generate a woodcut style image. This module receives two main inputs: the image embedding from the hierarchical encoder and the color embedding from the multi-modal decoder, where C denotes the embedding dimension, and K represents the number of color queries. First, the semantic association between the image embedding and the color query is established through matrix multiplication. The dot product operation is used to aggregate the two embeddings to generate an association weight matrix Then, the weight matrix is adjusted in dimension to restore the spatial structure. At this time, the feature vector of each spatial position (i, j) represents the association strength between the position and the K color queries.

[0143] The K-dimensional style semantic-color feature is mapped to the RGB (Red Green Blue, an additive color mixing model based on three primary colors of light) space through a 1x1 convolutional layer to generate a stylized result, denoted as:

[0144]

[0145] At this time, the convolution kernel parameters The K-channel dynamic mapping is performed to the RGB three channels to generate a final stylized result

[0146] Further, in step S6, the definition of the loss function includes: multi-dimensional color loss, content loss and style loss, and the total loss function is the weighted sum of the multi-dimensional color loss, the content loss and the style loss;

[0147] The multi-dimensional color loss includes a color concentration constraint and a color perception consistency constraint, which are used to enhance the color contrast and style consistency of the generated image. The multi-dimensional color loss is used because the traditional pixel-level loss is prone to generate images with dark colors, while the woodcut print prefers high-contrast and color-rich results.

[0148] The multi-dimensional color loss L MDC is denoted as:

[0149]

[0150] Wherein, μ c and σ c are defined as:

[0151]

[0152] Wherein, c∈{a,b} represents the chroma channel in the CIELAB color space (established by the International Commission on Illumination CIE, a device-independent color space): a represents the color distribution from green to red, and b represents the color distribution from blue to yellow. L MDC The color style expression is optimized from two dimensions of color concentration and perception consistency. Specifically, based on the color style characteristics of the woodcut print, the LAB (CIELAB) color model is selected for color gamut analysis. Among them, the lightness component (L) dominates the image contrast control and controls the color contrast; the a / b channel represents the chroma information and controls the hue-saturation distribution to capture the typical hue characteristics of the woodcut print. The first term is used to constrain the consistency of the generated image and the style image in color concentration and dispersion, avoiding excessive dispersion of color distribution. The a / b channels of the generated image and the style image are calculated respectively, and the first moment (mean value) is The second moment (variance) is

[0153] The second item is used to capture the deep semantic association of color distribution, avoiding the harsh transition between different color gamuts. The color features are extracted using a pre-trained model to color the ChromaNova multi-modal decoder, and the L channel (gray scale) and random noise of the generated image and the style image are input to force the network to focus on the chroma information. The output high-level feature map The loss function enhances the style consistency constraint by calculating the color semantic correlation, making the style color transition more natural, which helps to migrate complex color textures. The present application uses two different perceptual loss terms to measure the content difference between the output image I cs and the input content image I c , and the style difference between I cs and the input style reference image Is. Moreover, the feature map extracted by the pre-trained VGG-19 (Visual Geometry Group-19, a variant of a classic convolutional neural network model proposed by the University of Oxford) model is used to construct the content loss and style loss.

[0154] The content loss defines the content perceptual loss as:

[0155]

[0156] Where φ i represents the feature extracted from the i-th layer of the pre-trained VGG-19, and N is the number of layers.

[0157] The style loss defines the style perceptual loss L s as:

[0158]

[0159] Where μ(·) and σ(·) represent the mean and variance of the extracted features, respectively.

[0160] The total loss function defined by the present method is represented as:

[0161] L sum = λ1L MDC + λ2L c + λ3L s (17)

[0162] Where λ1, λ2, λ3 are weight parameters, and the weight parameters are set to 1, 0.6, and 0.9, respectively.

[0163] Further, in step S7, after the to-be-converted content image is processed by the trained network, a woodcut style image with high contrast color, clear texture and unique artistic style is output.

[0164] The to-be-converted content image is input into the trained generator, at this time, the generated image has joint optimization color style expression in multiple dimensions such as color matrix matching and perceptual consistency, and comprehensively considers color contrast and distribution difference, so that the generated style is more in line with the color characteristics of woodcut prints. The problems of difficult semantic feature extraction, difficulty in maintaining style consistency such as color and texture, and serious dependence on artificial annotation are solved.

[0165] Effect test:

[0166] First, comparison with the most advanced model:

[0167] In order to verify the superiority of the method proposed in the application in the task of style transfer of color woodcut, the model HieWoodNet is compared with the mainstream style transfer model CAST (Contrastive Arbitrary Style Transfer), StyTr2 (Adaptive Attention Normalization), SANet (Style Transformer 2), and AdaAttN (Style-Attentional Network) method, and the visual results are as shown in Figure 5

[0168] Figure 5 The visual generation results are shown. It can be found that the image generated by the SANet model has the problem of disordered texture. Although the color semantic alignment of the image generated by the AdaAttN model is improved, there is a problem of unreasonable color distribution. The color transition of the image generated by the StyTr2 model is relatively natural, but there is a problem of inconsistent color semantics. Although the color tone deviation of the image generated by the CAST model is small, there is a problem of single color tone. The HieWoodNet model has obvious advantages in the task of style transfer of color woodcut, and realizes more ideal style transfer effect of color woodcut. The results are shown in Table 1.

[0169] Table 1 Comparison of the most advanced methods

[0170] Method SSIM↑ PSNR↑ FID↓ LPIPS↓ CAST 0.849 25.416 121.416 0.271 StyTr2 0.856 26.133 120.467 0.250 AdaAttN 0.825 23.092 125.902 0.268 SANet 0.817 22.783 124.433 0.284 HieWoodNet 0.873 27.101 118.032 0.241

[0171] From Table 1, it can be seen that the method HieWoodNet model of the application is significantly better than the mainstream style transfer method in the task of style transfer of color woodcut. Among them, the style encoder enhances the color representation and style expressiveness. The progressive decoder realizes the artistic balance between content preservation and style color. The multi-modal decoder significantly optimizes the color transition and semantic consistency. The multi-dimensional color loss function improves the color contrast of the generated image. ​

[0172] Secondly, ablation experiments were conducted on the core components of the WoodNet, including ChromaNova multi-modal decoder, multi-scale style feature extraction module hierarchical encoder, and multi-dimensional color loss function MDCLoss. In addition, ablation experiments on the hierarchical encoder were designed to verify the effectiveness of each module.

[0173] Firstly, quantitative experiments were conducted to remove different modules to evaluate the influence of each module on the style transfer results, and the results are shown in Table 2.

[0174] 1) In the comparison with w / oHierStyle, the model decreased by 4.9% in SSIM (Structural Similarity Index), indicating that the multi-scale style feature extraction model significantly enhanced the model's adaptability to different color set woodcut style semantics.

[0175] Table 2 Comparison of ablation experiments of each component

[0176] Variants SSIM↑ PSNR↑ FID↓ LPIPS↓ w / oHierStyle 0.830 24.416 183.611 0.273 w / oChromaNova 0.849 21.182 162.402 0.291 w / oMDCLoss 0.865 23.184 159.930 0.287 HieWoodNet 0.873 27.101 118.032 0.241

[0177] Table 3 Comparison of hierarchical encoder

[0178] Encoder SSIM PSNR FID LPIPS Time(ms) ResNet-50 0.849 24.782 131.561 0.268 15.982 VGG-19 0.814 22.306 128.074 0.271 29.313 Transformer 0.882 26.891 122.953 0.253 48.745 Ours 0.873 27.101 118.032 0.241 11.068

[0179] 2) In the comparison with w / oChromaNova, the model's PSNR (Peak Signal-to-Noise Ratio) decreased by 21.8%, and the SSIM decreased by 2.8%, indicating that the multi-modal decoder was crucial for improving color semantic consistency and optimizing style color matching. The visualization results are shown in Figure 6

[0180] In the comparison with w / oMDCLoss (multi-dimensional color loss function), the model's PSNR decreased by 14.5%, indicating that the color loss constraint played a key role in controlling global chroma distribution and enhancing color contrast.

[0181] Figure 6 The results after removing different modules are shown. The model of w / oHierStyle generated images with missing style detail information. The model of w / oChromaNova generated images with monotonous colors and lack of multi-level color transitions of style images. The model of w / oMDCLoss generated images with low color contrast. When the model structure is complete, the generated images have strong color semantic consistency, high color contrast, and reasonable color distribution.

[0182] ​In addition, to verify the effectiveness of the hierarchical encoder, the present application replaces it with a ResNet-50 (Residual Network-50), a VGG-19, and a Transformer encoder, respectively, for experiments. The experimental results are shown in Table 3. The experimental results show that the ResNet-50 and the VGG-19 have poor FID (Frechet Inception Distance) effects due to the limitation of feature expression capability. The VGG-19 has a large number of parameters and is prone to overfitting during training due to the possibility of over-abstracting textures in deep networks, resulting in low SSIM and PSNR. The FID index of the images generated by the ConvNeXt (New Generation Convolutional Network) and the Transformer encoder is low, and the generated image distribution is closer to the real woodcut. However, according to related research, the Transformer usually only performs style attention operation on a single scale of image feature maps, and cannot fully capture low-level semantic clues, which may cause style color overflow problems when processing complex context scenarios. Moreover, the method using the ConvNeXt as the baseline network extracts style features the fastest, shortening by more than 30% compared to other models. Therefore, the present application uses the ConvNeXt pre-trained model to efficiently extract hierarchical style features and improve the ability of the model to transfer the style of the color-matching woodcut.

[0183] Third, to verify the generalization of the model, the present application performs cross-dataset testing on an animation dataset, a natural landscape dataset, and a Chinese landscape painting dataset. The experimental results are shown in Table 4, and the visual results are shown in Figure 7

[0184] 1) The indicators of the animation dataset are close to those of the CelebA, with a decrease of 1.24% in LPIPS (Learned Perceptual Image Patch Similarity), indicating that the animation dataset is more excellent in visual perception. As shown in Figure 7 , the comprehensive effect of style transfer of the animation dataset is better.

[0185] Table 4 is a comparison of different datasets

[0186]

[0187] 2) The natural landscape dataset has an effect close to that of the CelebA, with an increase of 3.90% in PSNR, indicating that the generated image has high detail authenticity and is close to the real image. The Lc (Content Perceptual Loss) increases by 12.12%, indicating that the generated image is slightly insufficient in content difference from the real image. As shown in Figure 7 ​As shown, the generated image is basically restored in structure, has less style loss, and has good style transfer effect.

[0188] 3) The indicators of the Chinese landscape painting dataset are similar to those of CelebA, in which the FID is reduced by 0.47%, indicating that the distribution of the generated image is better. While the Lc is increased by 12.48%, indicating that the content loss of the generated image is more, and the high-level semantics is distorted. For example, Figure 7 As shown, the Chinese landscape painting dataset can also complete the transfer well.

Claims

1. A method for style transfer in Chinese painting using multi-scale color embedding, characterized in that, Includes the following steps: S1: Obtain the content image dataset and style image dataset, and divide them into training set and test set; The content image dataset is the CelebA face dataset, and the style image dataset is the polychrome woodblock print dataset. S2: Construct a Chinese painting style transfer network with multi-scale color embedding, including: content-aware position encoder, hierarchical encoder and multimodal decoder; S3: Input the content image into the content-aware location encoder, and extract the global semantic feature sequence of the content image through linear mapping and transformer encoder; S4: Input the style image into the hierarchical encoder, combine it with the global semantic feature sequence, and generate an image embedding that integrates content and style through a progressive decoder; S5: Embed the image with fused content and style into the input multimodal decoder, dynamically optimize the color embedding representation, obtain semantically aligned color features, and output a color-aligned style transfer image; S6: Employ multidimensional color loss, content loss, and style loss, and train the total loss through weights to train the network; S7: Input the image to be converted into the trained network model to obtain a color woodblock print style image.

2. The method for transferring Chinese painting style through multi-scale color embedding according to claim 1, characterized in that, In step S1, obtaining the style image dataset includes crawling the woodblock print dataset using the keywords "black and white woodblock prints" and "black and white woodcut portrait woodblock prints".

3. The method for transferring Chinese painting style through multi-scale color embedding according to claim 1, characterized in that, In step S3, the processing of the content-aware position encoder includes: The content image is segmented into fixed-size image blocks, and an embedding sequence is generated by linear projection to preserve spatial structure information. A multi-layer Transformer encoder is used to capture global semantic dependencies. Each layer contains a multi-head self-attention and feedforward network, and training is stabilized by residual connections and layer normalization. The image patch location encoding achieves robust support for inputs at different resolutions through spatial distance calculation, and the final output is a content feature sequence, specifically including: First, given a content image A color woodblock print style image Its height is H, its width is W, its number of channels is 3, and m is a constant 8. The image is segmented in the height and width dimensions. Then, embedded sequence features are generated through linear projection. in By dynamically adjusting the number of blocks, robust support for inputs at different resolutions is achieved; To obtain structural information, when using a Transformer-based model, positional encoding is incorporated into the input sequence. The attention score between the i-th image patch and the j-th image patch is represented as: Among them, W q and W k For querying and calculating the parameter matrix of the key, P i Represents the i-th one-dimensional position code; In a two-dimensional scene, an image patch (x) at a pixel i, y i ) and (x j, y j The relative positions of image patches at location ) are calculated using spatial distance: in, When d = 512, the relative positions of the two image patches depend only on their spatial distance. In this way, different image scales will not affect the spatial relationship between the two image patches. P L =F pos (AvgPool n×n (e c )) (3) Among them, AvgPool n×n F is the average pooling function. pos P represents a 1x1 convolution operation for a learnable positional encoding function. L Following sequence ε c Learnable positional encoding, n is set to 18; CAPE (i.e., P) of an image patch CA(x,y) ) is represented as: Among them, a kl The interpolation weights are s, where s represents the number of adjacent image patches; Finally, P at the corresponding pixel position CA Its embedding features ε c The sum represents the final feature embedding of a certain image patch; The final content sequence Z is obtained. c ={S c1 +P cA1 ,S c2 +P cA2 ,...,S cL +P cAL }, fed into the Transformer encoder, input sequence Z c Encoded as query (Q), key (K), and value (V), the expression is: Q=Z c W q ,K=Z c W k ,V=Z c W v (5) in, F MSA (Q,K,V)=Concat(Att1(Q,K,V),...,Att N (Q,K,V))W0 (6) in, Here, N is the number of attention heads, which are learnable parameters. The dimension of each attention head is represented, and the encoded content sequence is obtained by applying residual connections; The calculation of the i-th attention head is defined as follows: Y c′ =F MSA (Q,K,V)+Q AND c =F FFN (AND c ')+Y c′ (8) Where F FFN (Y c′ ) = max(0,Y c′ Each module (W1+b1) and W2+b2) is followed by a layer normalization operation. Finally, a multi-layer Transformer encoder is used to extract global semantic features and output a content feature sequence.

4. The method for transferring Chinese painting style through multi-scale color embedding according to claim 1, characterized in that, In step S4, the processing procedure of the layer encoder includes: Input style image The image is then fed into a hierarchical encoder, where a backbone network is used to extract high-level style information from the style image, enhancing contextual semantics and texture correlation. The encoder skeleton is composed of multiple downsampling layers and Style modules stacked together, and hierarchical image representations are learned by leveraging multi-scale feature extraction capabilities.

5. The method for transferring Chinese painting style through multi-scale color embedding according to claim 1, characterized in that, In step S5, the processing procedure of the multimodal decoder includes: It consists of a stack of convolutional modules, each receiving visual features and a color query as input; the multimodal decoder is based on an improved Transformer decoder design, dynamically optimizing the color embedding representation through a multi-scale color query mechanism to ensure alignment between color and semantic boundaries, specifically including: First, a correlation between semantic representation and color embedding is established through a cross-attention layer, represented as follows: Where l represents the network layer index. Let l be the color embedding vector of the l-th layer; For querying the matrix, These are the transformation functions f K (·) and f V (·) Processed image features; H l H w The spatial resolution of image features, f Q, ,f K ,f V It is a linear mapping function; After the above cross-attention operation, the color embedding is further transformed using a standard Transformer layer. The process is defined as follows: Z l ″=MSA(LN(Z l ′))+Z l ′, WITH l ″′=MLP(LN(Z l ″))+Z l ″, WITH l =LN(Z l ″′), (10) Here, MSA() represents multi-head self-attention, MLP() represents multilayer perceptron, and LN() is the layer normalization operation. In the proposed CDB module, the cross-attention operation is performed before the self-attention operation. To balance computational complexity and representational power, image features at three different scales are selected, and the multimodal decoder process is formalized. The multimodal decoder can be defined as follows: E c =δ(Z0,F1,F2,F3) (11) Among them, F1ΔF2F3 and F3 are visual features at three different scales; In a multimodal decoder, multi-scale features are used to model the relationship between color queries and visual embeddings; color queries are progressively associated with semantic regions, outputting dynamic color embeddings; K-dimensional stylistic semantic-color features are mapped to the RGB space through a 1×1 convolutional layer to generate stylized results, represented as follows: At this point, the convolution kernel parameters The K channel is dynamically mapped to the RGB three channels to generate the final stylized result.

6. The method for multi-scale color embedding in Chinese painting style transfer according to claim 1, characterized in that, In step S6, the loss function is defined as follows: multidimensional color loss, content loss, and style loss. The total loss function is the weighted sum of multidimensional color loss, content loss, and style loss. The multidimensional color loss includes color concentration constraints and color perception consistency constraints, used to enhance the color contrast and style consistency of the generated image. The multidimensional color loss L... MDC Represented as: Where, μ c With σ c The definition of is: Content loss is defined as follows: Where, φ i This represents the feature extracted from the i-th layer of the pre-trained VGG-19, where N is the layer number. Style loss will reduce style perception loss L s Represented as: Where μ(·) and σ(·) represent the mean and variance of the extracted features, respectively. The total loss function defined in this method is expressed as follows: (The total loss is trained using weights.) L sum =λ1L MDC +λ2L c +λ3L s (17) Wherein, λ1, λ2, and λ3 are weight parameters, which are set to 1, 0.6, and 0.9, respectively.

Images