Landscape painting style migration method based on diffusion model and feature statistics

Abstract

The invention discloses a landscape painting style migration method based on a diffusion model and feature statistics. According to the style mapping module based on the bidirectional state space model and the attention mechanism, the unique style of the landscape painting can be accurately and efficiently learned and stored in the prompt space. The invention further provides a feature statistic adjustment method based on AdaIN and WCT, unique strokes of landscape paintings can be effectively transferred, and color information of the landscape paintings can be faithfully transferred. By combining the pre-training diffusion model and the CLIP prompt space, the stroke and color of the landscape painting are fully learned, the content structure is well reserved, the white space technology of the landscape painting is reflected, and the highly-vivid landscape painting with artistic conception can be generated.

Description

Technical Field

[0001] The present invention belongs to the field of image style transfer, and in particular relates to a landscape painting style transfer method based on a diffusion model and feature statistics. Background Art

[0002] As a type of ink painting, landscape painting mainly depicts natural mountains and rivers, and occupies an important position in art history and culture. However, creating an exquisite landscape painting in reality is a highly professional, time-consuming and laborious task. With the vigorous development of artificial intelligence, it has become possible to quickly transform a real landscape photo into a lifelike landscape painting.

[0003] Existing methods for landscape painting style transfer mainly revolve around generative adversarial networks (GANs). ChipGAN proposes three core constraints based on CycleGan for the three basic techniques of ink painting: blanks, brushstrokes, and ink tones and diffusion. Paint-CUT combines shuffle attention with residual blocks, introduces perceptual loss and edge loss, and achieves high-quality landscape painting generation. However, due to the instability of GAN training and mode collapse problems, these methods are difficult to generate realistic landscape paintings. In contrast, diffusion models are stable in training and can fit complex image distributions. The massive data information learned by the pre-trained diffusion model can be used to complete artistic style transfer. For example, InST maps images to text embeddings, regards style as a learnable text description, and proposes a style transfer method based on text inversion.

[0004] Although these methods can generate high-quality landscape paintings in some cases, they cannot accurately learn the unique style of landscape paintings and cannot faithfully transfer their color information. Therefore, the present invention proposes a landscape painting style transfer method based on diffusion model and feature statistics, which can effectively transfer the unique brushstrokes and color information of landscape paintings and generate highly realistic landscape paintings. Summary of the invention

[0005] The present invention aims at the shortcomings of several existing landscape painting style transfer technologies, focuses on faithfully transferring the color information of landscape paintings, and provides a landscape painting style transfer method based on a diffusion model and feature statistics.

[0006] The main steps of the present invention are as follows:

[0007] Step 1: Prepare a dataset containing a certain number of landscape painting style datasets and a dataset of real landscape content that needs style transfer;

[0008] Step 2: Design a style transfer model based on the pre-trained diffusion model and CLIP prompt space;

[0009] Step 3: training the style transfer model;

[0010] Step 4: Take a set of images from the landscape painting style dataset, encode them into the latent space style representation, calculate the style feature statistics and save them;

[0011] Step 5: Encode the real landscape content image into the latent space, add initial noise after modulation using the style feature statistics, and transform the real landscape image into a highly realistic landscape painting under the guidance of the prompt space and content preservation architecture.

[0012] Furthermore, the step 1 also includes: preprocessing all images, standardizing the images, and then performing data normalization to adjust all pixel values ​​to between [-1, 1].

[0013] Furthermore, the step 1 also includes: preparing model weights required for the pre-trained diffusion model, including Stable Diffusion weights of version 1.5, CLIP weights of version clip-vit-large-patch14, and ControlNet_Canny weights of version 1.1.

[0014] Furthermore, the style transfer model is divided into a training phase and an inference phase. The training phase includes the model body, the cue space to be trained and the style mapping module; the inference phase includes the model body and the trained cue space.

[0015] The model body is a pre-trained diffusion model, including a set of encoders and decoders, and the main network adopts a U-net network. The prompt space is a step-perceived and layer-perceived prompt space, which is mapped from a learnable vector through a style mapping module.

[0016] Furthermore, the style mapping module includes a bidirectional state space model layer, a cross attention layer, a feedforward layer and a normalization layer. The learnable vector is first normalized by the normalization layer, then enters the bidirectional state space model layer, then performs cross attention calculation, and finally outputs through the feedforward layer.

[0017] Furthermore, the inference stage also includes a style modulation module and a content preservation architecture. The style modulation module is used to obtain corresponding style feature statistics; the content preservation architecture includes edge detection, and the main network is a U-net network.

[0018] Further, the step 3 comprises:

[0019] Step 3.1: Encode the preprocessed landscape painting style image into the latent space style representation, add initial noise, and obtain a noisy version of the style image.

[0020] Step 3.2: Map the learnable vectors in the CLIP model to the target cue space through the style mapping module, and use it as a condition to guide the pre-trained diffusion model during training.

[0021] Step 3.3: Calculate the LDM loss, back-propagate and update the weights, and repeat the preset number of iterations.

[0022] Furthermore, the step 3 also includes:

[0023] Step 3.4: Once the LDM loss converges, the target cue space stores the style information of landscape paintings and serves as a condition to guide the generation of landscape paintings during the inference phase.

[0024] Further, the step 5 comprises:

[0025] Step 5.1: Encode the real landscape content image into the latent space, use the style feature statistics to make corresponding adjustments, and add initial noise;

[0026] Step 5.2: Sampling denoising, computing content guidance of content-preserving architecture;

[0027] Step 5.3: Under the guidance of the hint space and content-preserving architecture, the latent space representation of the stylized result is obtained, and the image is reconstructed through the decoder to obtain the final landscape painting.

[0028] Furthermore, the step 5.1 is specifically as follows:

[0029] Step 5.1.1: Obtain the Canny contour map of the real landscape content image through the edge detection function.

[0030] Step 5.1.2: Encode the processed real landscape content image into the latent space to obtain the content latent space representation.

[0031] Step 5.1.3: Use the style feature statistics to make corresponding adjustments, add initial noise, and obtain a noisy version of the stylized content image.

[0032] The beneficial effects of the present invention are:

[0033] The present invention shows excellent generation effect in experimental comparison with various advanced methods. By introducing a style mapping module that combines bidirectional state space and attention mechanism, the prompt space fully learns the style information of landscape painting. Combined with the style feature statistics adjustment method, the model can faithfully transfer the color information and unique brushstrokes of landscape painting, generating highly realistic landscape paintings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is an architecture diagram of the style transfer model training phase and reasoning phase in an embodiment of the present application;

[0035] Figure 2 This is a workflow diagram of the style transfer model training phase and reasoning phase in an embodiment of the present application;

[0036] Figure 3 The figure is a comparison chart of the experimental results of the embodiment of the present application and the prior art. DETAILED DESCRIPTION

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] like Figure 1 and Figure 2 As shown, the embodiment of the present application provides a landscape painting style migration method based on a diffusion model and feature statistics, specifically:

[0039] Step 1: Prepare a dataset containing a certain number of landscape painting styles and a dataset of real landscape content that needs style transfer (in .jpg or .png format). Prepare the model weights required for the large-scale pre-trained diffusion model, including Stable Diffusion weights of version 1.5, CLIP weights of version clip-vit-large-patch14, and ControlNet_Canny weights of version 1.1.

[0040] These images are read through the Pillow library and converted into the standard RPG format, and then the size is standardized to 512 × 512 pixels through the resize function. The pixel value of the image is divided by 127.5 and subtracted from 1 for data normalization, and all pixel values ​​are adjusted to between [-1, 1].

[0041] Step 2: In order to accurately learn the style information of landscape paintings and faithfully transfer their colors and brushstrokes, the embodiment of the present application designs a model based on the pre-trained diffusion model and the CLIP prompt space - the style transfer model. The model is divided into a training phase and an inference phase. The training phase includes the model body, the prompt space to be trained, and the style mapping module; the inference phase includes the model body, the trained prompt space, the style modulation module, and the content preservation architecture.

[0042] The main body of the model is the pre-trained diffusion model Stable Diffusion, which includes a set of VAE encoders and decoder , the main network is U-net network. The prompt space is the step perception and layer perception prompt space P* , which is mapped by the learnable vector P through the style mapping module BiSSM-CA. The style modulation module uses two feature statistics adjustment methods, AdaIN and WCT, to obtain the corresponding style feature statistics SFstyle. The content preservation architecture ControlNet includes a Canny edge detection, and the main network is a U-net network. The training content of the training phase includes the style mapping module BiSSM-CA and the prompt space P * The former is responsible for mapping the initialized learnable vector P to the latter, learning and storing the landscape painting style information; the inference stage uses the style modulation method combined with the content-preserving architecture to guide the pre-trained diffusion model to generate highly realistic landscape paintings.

[0043] like Figure 1 As shown in the embodiment of the present application, the style mapping module BiSSM-CA is composed of a bidirectional state space model (i.e., Bidirectional SSMs) layer, a cross-attention layer, a feedforward layer, and three normalization layers. The style mapping module is mainly responsible for converting the learnable vector Mapping to space , learn landscape painting style information.

[0044] Learnable vectors First, it is normalized by the normalization layer and then enters the bidirectional state space model layer:

[0045]

[0046] in, Representation layer normalization operation, Represents a linear projection layer. The bidirectional state space model is different from the traditional SSMs. The mapping results obtained from the forward and backward processing are , respectively and To express.

[0047] right and After one-dimensional convolution and SiLU activation, we get and , after SSM operation and after SiLU activation and Perform gated calculations:

[0048]

[0049]

[0050] in represents the convolutional layer, represents the activation function SiLU, represents the state-space model calculation, Represents a gating operation.

[0051] Finally and The two results are added together to get the final output of the bidirectional state space model layer:

[0052]

[0053] The above normalization layer and bidirectional state space model layer process can be generally expressed as:

[0054]

[0055] Then the results and Perform CrossAttention calculation:

[0056]

[0057]

[0058] in Represent the query matrix, key matrix, and value matrix respectively. Represents a learnable weight matrix, and softmax is used as an activation function to normalize the weights into a probability distribution. Finally, it passes through the feedforward layer.

[0059] The entire BiSSM-CA process can be expressed as:

[0060]

[0061] Step 3: Train the style transfer model.

[0062] First, the preprocessed landscape painting style image VAE encoder via Stable Diffusion Encoding to latent space style representation , add initial noise according to time step t , get the noisy version of the style image Then, the learnable vector in the CLIP model is Mapping to the target step-aware and layer-aware cue space through the style mapping module , for step-aware and layer-aware cue spaces , which can be expressed as:

[0063]

[0064] in represents the prompt injected into the jth layer of U-Net at the i-th stage of the denoising process.

[0065] The resulting step-aware and layer-aware cue space It consists of thirty unit vectors, which are divided into ten groups of three as conditions to guide the U-net network of the incoming diffusion model. The three unit vectors in each group guide the front, middle and back three layers of the U-net network respectively.

[0066] The noisy version of the style image Enter the style transfer model and train it with the following loss:

[0067]

[0068] in represents the neural network trained to predict the added noise. Calculate the loss , back-propagation updates the weights, repeating the preset number of iterations (the default is 100,000). Once convergence, The style information of landscape paintings will be stored and used as a condition to guide the generation of landscape paintings during the inference phase.

[0069] Step 4: Calculate style feature statistics. Take a small set of images from the landscape painting style dataset , through the VAE encoder of Stable Diffusion Encoding to latent space style representation , based on two different methods, AdaIN and WCT, to calculate this set of tensors The feature statistics are saved.

[0070] 1. Based on AdaIN: The core idea of ​​AdaIN is to match the mean and variance of content features with the mean and variance of style features. The tensor obtained by calculation is The channel mean and standard deviation of , as SFstyle; at inference time, the latent space representation of the content is also calculated The channel mean and standard deviation are modulated according to AdaIN:

[0071]

[0072]

[0073]

[0074] in, ()and ) represent the calculation of the mean along the channel and standard deviation .

[0075] 2. Based on WCT: WCT matches the covariance matrix of content features with the covariance matrix of style features through whitening and coloring operations. Merge the batch, height, and width dimensions of , arrange all elements of each channel linearly, and calculate the mean of each channel along the new dimension , then subtract the mean Get the centralization feature :

[0076]

[0077]

[0078] Compute the covariance matrix and singular value decomposition of the centered features, passing the mean and singular value decomposition as SFstyle:

[0079]

[0080]

[0081] in, They represent left singular matrix, diagonal matrix, and right singular matrix respectively.

[0082] At inference time, the latent space representation of the content Perform the same statistical calculation operation to obtain , perform whitening, then perform coloring according to SFstyle, and finally reconstruct the features:

[0083]

[0084]

[0085]

[0086] in, After whitening , Indicates the colored .

[0087] Step 5: Perform inference.

[0088] The real landscape content image The image edge information is obtained through the Canny edge detection function in OpenCV, and the edge detection result can be controlled by the low threshold and high threshold.

[0089] Then the processed real landscape content image VAE encoder via Stable Diffusion Encode into latent space to get content latent space representation , use the style feature statistics SFstyle obtained in step 4 to make corresponding adjustments and add initial noise , get the stylized content image with noise version .

[0090] The DDIM algorithm is used for sampling. During the denoising process, the content guidance of the content-preserving architecture ControlNet is calculated based on the current latent variables and combined with the image edge information, specifically the outputs of the first and middle layers of the U-net network in ControlNet. In the U-net network of the pre-trained diffusion model, the skip layer connection not only connects the outputs of the first and middle layers of the current network, but also connects the output results of ControlNet.

[0091] In step-aware and layer-aware cue space Under the guidance of the content-preserving architecture ControlNet, the latent space representation of the stylized results is obtained , through the VAE decoder Reconstruct the image and get the final landscape painting.

[0092] Specific experimental data and parameters:

[0093] Specific results such as Figure 3 The first column shows the input real landscape image, the second and third columns are the results of the present application embodiment adjusted according to AdaIN and WCT, and the remaining columns are the results of the existing latest method. Specific experimental details include:

[0094] Step 1: Use the pre-trained Stable Diffusion 1.5 and clip-vit-large-patch14 CLIP model, and use the style training set in ChipPhi published by ChipGan as the training dataset, including 1388 landscape paintings. During training, all images were resized to 512×512, the training batch size Batch=1, the learning rate was set to 0.000001, and a total of 100,000 times were trained on a single NVIDIA RTX 3090 GPU, and the training took about 10 hours.

[0095] Table 1 Quantitative comparison results

[0096]

[0097] Step 2: During inference, the Landscape Pictures dataset obtained from Kaggle is used. The dataset contains 4319 real landscape photos, from which 700 are randomly selected as the input content images, i.e., the test set. CLIPScore and FID are selected as evaluation indicators. The former is used to measure the similarity between the text prompt and the artistic style image, and the latter is used to evaluate the style fidelity between the stylized image and the corresponding style image. When using CLIP Score, the text is fixed to "A Chinese Landscape Painting", and the style dataset (train B in the ChipPhi dataset) is used as the ground truth. As shown in Table 1, the higher the Clip score, the better, while the lower the FID score, the better. Comparison method 1 is LSAST, comparison method 2 is ArtBank, comparison method 3 is InST, and comparison method 4 is ChipGan. The embodiment of the present application obtained the results of AdaIN (27.75, 82.67) and WCT (27.66, 84.3) on CLIP Score and FID respectively, and the highest CLIP Score was obtained by being closest to the benchmark truth value, which means that the stylized image generated by this method is closer to a landscape painting. The above results show that the CLPFusion proposed in this application can generate highly realistic landscape paintings.

[0098] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the embodiments here. All obvious changes or modifications derived from the technical solution of the present invention are still within the protection scope of the present invention.

Claims

1. A landscape painting style transfer method based on a diffusion model and feature statistics, characterized in that: The method comprises the following steps: Step 1: Prepare a dataset containing a certain number of landscape painting style datasets and a dataset of real landscape content that needs style transfer; Step 2: Design a style transfer model based on the pre-trained diffusion model and CLIP prompt space; Step 3: training the style transfer model; Step 4: Take a set of images from the landscape painting style dataset, encode them into the latent space style representation, calculate the style feature statistics and save them; Step 5: Encode the real landscape content image into the latent space, add initial noise after modulation using the style feature statistics, and transform the real landscape image into a highly realistic landscape painting under the guidance of the prompt space and content preservation architecture; The style transfer model is divided into a training phase and an inference phase: The training phase includes a model body, a prompt space to be trained, and a style mapping module; The inference stage includes the model body and the trained prompt space; The style mapping module includes a bidirectional state space model layer, a cross attention layer, a feedforward layer and a normalization layer; The learnable vector is first normalized by the normalization layer, then enters the bidirectional state-space model layer, followed by cross-attention calculation, and finally output through the feedforward layer.

2. The landscape painting style transfer method based on diffusion model and feature statistics according to claim 1, characterized in that: The step 1 also includes: preprocessing all images, standardizing the images, and then performing data normalization to adjust all pixel values ​​to between [-1, 1].

3. The method for landscape painting style transfer based on diffusion model and feature statistics according to claim 1 or 2, characterized in that: The step 1 also includes: preparing model weights required for the pre-trained diffusion model, including StableDiffusion weights of version 1.5, CLIP weights of version clip-vit-large-patch14, and ControlNet_Canny weights of version 1.

1.

4. The landscape painting style transfer method based on diffusion model and feature statistics according to claim 1, characterized in that: The model body is a pre-trained diffusion model, including a set of encoders and decoders, and the main network adopts a U-net network; The cue space is a step-aware and layer-aware cue space, which is mapped from a learnable vector through a style mapping module.

5. The method for landscape painting style transfer based on diffusion model and feature statistics according to claim 4, characterized in that: The inference stage also includes a style modulation module and a content preservation architecture; The style modulation module is used to obtain corresponding style feature statistics; The content preservation architecture includes edge detection, and the main network is a U-net network.

6. The landscape painting style transfer method based on diffusion model and feature statistics according to claim 1, characterized in that: The step 3 comprises: Step 3.1: Encode the preprocessed landscape painting style image into the latent space style representation, add initial noise, and obtain a noisy version of the style image; Step 3.2: Map the learnable vectors in the CLIP model to the target cue space through the style mapping module, and use it as a condition to guide the pre-trained diffusion model during training; Step 3.3: Calculate the LDM loss, back-propagate and update the weights, and repeat the preset number of iterations.

7. The method for landscape painting style transfer based on diffusion model and feature statistics according to claim 6, characterized in that: The step 3 also includes: Step 3.4: Once the LDM loss converges, the target cue space stores the style information of landscape paintings and serves as a condition to guide the generation of landscape paintings during the inference phase.

8. The method for landscape painting style transfer based on diffusion model and feature statistics according to claim 1, characterized in that: The step 5 comprises: Step 5.1: Encode the real landscape content image into the latent space, use the style feature statistics to make corresponding adjustments, and add initial noise; Step 5.2: Sampling denoising, computing content guidance of content-preserving architecture; Step 5.3: Under the guidance of the hint space and content-preserving architecture, the latent space representation of the stylized result is obtained, and the image is reconstructed through the decoder to obtain the final landscape painting.

9. The method for landscape painting style transfer based on diffusion model and feature statistics according to claim 8, characterized in that: The step 5.1 is specifically as follows: Step 5.1.1: Obtain the Canny contour map of the real landscape content image through the edge detection function; Step 5.1.2: Encode the processed real landscape content image into the latent space to obtain the content latent space representation; Step 5.1.3: Use the style feature statistics to make corresponding adjustments, add initial noise, and obtain a noisy version of the stylized content image.

Images