Pattern generation method based on diffusion model fine tuning

Abstract

The pattern generation method based on diffusion model fine tuning comprises the following steps: constructing a data set; the generation model comprises an edge detection model ControlNet and a stable diffusion model, the stable diffusion model comprises a text encoder CLIP, a diffusion model U-Net and an image decoder VAE, the CLIP text encoder converts an input natural language into word vector features and inputs the word vector features into a U-Net network in a submerged space, and the diffusion model U-Net and the image decoder VAE are connected with the edge detection model ControlNet and the diffusion model U-Net; the U-Net network is controlled to carry out iterative denoising on pure noise in a low dimension to generate a compressed image, and finally the compressed image is restored to an original pixel image through a decoder VAE, so that text content control image generation is realized; carrying out fine tuning training on the generative model by adopting a LoRA parameter efficient fine tuning technology; and intelligently generating a batik style pattern. The method has the advantages that the texture precision of the generated image can be improved, and the personalized style image can be generated.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and specifically to a pattern generation method based on fine-tuning of a diffusion model. Background Art

[0002] In recent years, generative artificial intelligence (AIGC) has made breakthrough progress in the field of image generation. As early as 2014, Goodfellow et al. first proposed the Generative Adversarial Network (GAN). The GAN network reaches the Nash equilibrium state through the interaction between the generator and the discriminator, generating target images that are as close as possible to the distribution characteristics of the sample data, providing a research basis for image style transfer. With the further development of the image generation model, Ho et al. proposed the Denoising Diffusion Probabilistic Model (DDPM) in 2020. By adding noise to the image forward and denoising it backward, the learning rules of the sample are learned, and new images are synthesized from random inputs, which can effectively solve the problem of unstable convergence of the GAN network and is widely used in the field of text-to-image generation. In the prior art, in order to improve the flexibility of the diffusion model and solve the problems of dependence on large-scale data sets and high demand for hardware resources of the diffusion model, various training methods for fine-tuning large models have emerged. Chen Yanbo et al. used the Low-Rank Adaption (LoRA) method to control the generation style of the stable diffusion model and were able to generate innovative Song Dynasty shengsehua patterns; Ji Lefu et al. proposed a Dai brocade pattern generation model based on the DreamBooth fine-tuning method, realizing the generation of Dai brocade style feature patterns by the text-to-image large model; Zhuo et al. used the curriculum learning method to fine-tune the stable diffusion model, generating high-fidelity Yao ethnic clothing images while ensuring text semantic consistency. The above research enables the diffusion model to obtain the simulation ability in specific style fields by using a small number of training samples. However, most of these generated images focus on the similarity and fidelity with the original training set images, rarely study the texture features of the generated images, and the texture accuracy of the generated images is low. Summary of the Invention

[0003] The present invention overcomes the above disadvantages and proposes a pattern generation method based on fine-tuning of a diffusion model that can improve the texture accuracy of generated images and generate personalized style images.

[0004] The pattern generation method based on fine-tuning of a diffusion model according to the present invention, wherein: the specific steps of the method include:

[0005] Step 1: Dataset construction: Obtain batik style images and form a dataset through preprocessing;

[0006] After the batik-style image is taken in kind, image preprocessing is performed. The specific steps include:

[0007] a) First, filter out low-quality images, and then perform sub-image segmentation on large-sized long images to obtain sample images;

[0008] b) Unify the sizes of the obtained sample images and divide them into a training set and a test set;

[0009] c) Use an image label backpropagation tool to divide the input sample image into multiple fixed-size image blocks, convert them into vector embeddings, and generate prompt words related to the content of the image blocks;

[0010] d) Use a label cleaning tool to clean the prompt word labels above a certain confidence level and delete redundant labels irrelevant to the image content;

[0011] e) Add descriptive words for the ice crack texture features of the batik image and prompt words for the image layout information, and add a model trigger word to each image label to refine and semantically supplement the text description of the label;

[0012] Step 2: Build a smart generation model for batik-style patterns: This generation model includes an edge detection model ControlNet and a stable diffusion model;

[0013] The stable diffusion model includes a text encoder CLIP, a noise prediction network U-Net, and an image decoder VAE. The CLIP text encoder converts the input natural language into word vector features, and then inputs them into the U-Net network inside the latent space to control the U-Net network to iteratively denoise pure noise in the low dimension to generate a compressed image. Finally, through the decoder VAE, the compressed image is restored to the original pixel image to realize text content control of image generation. Among them:

[0014] a) In the forward diffusion process of the diffusion model, noise is gradually added to the batik image, and the original image data is gradually converted into pure noise; assuming that the initial data distribution is x0~q(x), the forward noise addition process is divided into multiple discrete time steps T. Through T rounds of continuously adding noise to the original batik image until the original batik image x0 is transformed into a pure Gaussian noise image x t , the Gaussian distribution q formula for the noise addition process from time t-1 to time step t is expressed as:

[0015]

[0016] Among them, X t-1 represents the image state at time step t-1; β tThe noise parameter at time step t, a value between 0 and 1, is used to control the intensity of the added noise; I is the identity matrix; N is the Gaussian distribution. As the time step T increases, the resulting distribution approaches the standard normal distribution. This process is a parameterized Markov chain, and the formula is expressed as:

[0017]

[0018] b) The diffusion model performs step-by-step denoising on the Gaussian noise x randomly sampled from the standard normal distribution during the reverse inverse diffusion process. The noisy image is input into the denoising network to gradually remove the noise, and the original batik image data distribution is restored through the enhanced activation module. Finally, the real batik image x0 consistent with the initial data distribution is obtained. The probability distribution p of the denoising process from time step t to time step t - 1 t is expressed by the formula: θ The formula is expressed as:

[0019]

[0020] where μ θ (x t , t) is the mean of the Gaussian noise predicted by the denoising network, is the variance of the predicted Gaussian noise;

[0021] Step 3: Model training: Use the dataset to train the intelligent generation model for batik-style patterns. Adopt the model fine-tuning technology LoRA based on low-rank adaptation. Add a new weight matrix to the pre-trained matrix of the stable diffusion model. Utilize the low-rank matrix decomposition technology to decompose the weight matrix into the product of a dimensionality reduction matrix A and a dimensionality increase matrix B. By freezing the training weights of the original diffusion model, only the newly added matrices A and B are trained, reducing the number of parameters in the fine-tuning process and significantly reducing computational resources. Among them: Use the Adam optimizer for learning optimization. Set the number of training epochs Epoch of the LoRA model to 20 times, save the model every 2 epochs, set the batch size during training to 8, and set the learning rate to 0.0001;

[0022] Step 4: Intelligent generation of batik-style patterns: Use the edge detection model ControlNet to extract the contour of the input image, and then input the extracted contour information into the trained stable diffusion model to achieve the accurate and controllable generation of high-fidelity batik-style patterns.

[0023] For the above pattern generation method based on diffusion model fine-tuning, where: in step 1, the size of the sample image is uniformly adjusted to 1024×1024 pixels, and the resolution is set to 300 DPI.

[0024] The above pattern generation method based on fine-tuning of the diffusion model, wherein: in step 1, the prompt word tags with a cleaning confidence of more than 40% are selected.

[0025] The above pattern generation method based on fine-tuning of the diffusion model, wherein: in step 2, the stable diffusion model is the SDXL version, which supports generating images at a high pixel level of 1024×1024.

[0026] The above pattern generation method based on fine-tuning of the diffusion model, wherein: in step 2, the output of the edge detection model ControlNet in the "0 convolutional layer" is all zero, and the diffusers_xl_canny_full model is used to control the edge contour of the generated image, the control model weight is 0.8, the sampler is Eluer a, and the number of iteration steps is 35.

[0027] Compared with the prior art, the present invention has obvious beneficial effects. As can be seen from the above solutions, the present invention inserts low-rank trainable parameters into the bypass branch of the stable diffusion model to enable it to obtain the ability to generate realistic batik patterns. In view of the particularity of the batik pattern texture, precise text prompt words are added to describe the batik texture features, improving the understanding and learning of the diffusion model for detail features, and combining the ControlNet network to extract edge features from different input images, so as to realize the personalized generation of batik-style innovative patterns under any content.

[0028] The beneficial effects of the present invention are further illustrated below through specific embodiments. Brief Description of the Drawings

[0029] Figure 1 It is a flow chart of the present invention. Detailed Embodiments

[0030] The specific embodiments, features and effects of a pattern generation method based on fine-tuning of the diffusion model proposed according to the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0031] As Figure 1 shown, a batik pattern generation method based on fine-tuning of the diffusion model, wherein: the specific steps of the method include:

[0032] Step 1: Dataset construction: Obtain batik-style images and corresponding text description words to form a training dataset;

[0033] After the batik-style images are taken in kind, image preprocessing is performed. The specific steps include:

[0034] a) First, low-quality images are screened and removed, and then large-size long images are segmented into sub-images, and a total of 200 batik sample images are obtained;

[0035] b) Uniformly adjust the size of all samples to 1024×1024 pixels, set the resolution to 300 DPI, and select 20 of them as test samples for subsequent model performance testing.

[0036] The text description words generate corresponding text descriptions by extracting semantic features from the images; the specific steps for extracting the text description words are as follows:

[0037] a) Use the image label reverse inference tool Waifu Diffusion 1.4Tagger to divide the input image into multiple image blocks of a fixed size, convert them into vector embeddings, and generate prompt words related to the content of the image blocks at the same time;

[0038] b) Use the label cleaning tool BD TagManager to clean the prompt word labels with a confidence level above 40% and delete the redundant labels irrelevant to the image content.

[0039] c) Add the prompt words "irregular fissures, crack texture" for the batik ice crack texture feature and the prompt word "asymmetrical" for the image layout information, and add the model trigger word "Waxprinting" to each image label to refine and semantically supplement the text description. Taking the batik pattern in the shape of bamboo leaves as an example, the final words added to the prompt word file after cleaning and supplementing are Wax printing, blue background, blue theme, bamboo, leaf, irregular fissures, crack texture, asymmetrical.

[0040] Step 2: Build an intelligent generation model for batik-style patterns: This generation model includes an edge detection model ControlNet and a stable diffusion model; first, use the edge detection model in ControlNet to extract the contour of the input image, and then input the extracted contour information into the stable diffusion model to achieve the precise and controllable generation of high-fidelity batik-style patterns.

[0041] The ControlNet network part: The neural network block is divided into two parts: "locked" and "trainable". The "locked" part retains the performance of the original diffusion model because both the input and output of ControlNet are zero. The "trainable" part is trained to output by applying control conditions. The two are connected through a zero convolution layer, and precise control of the images generated by the diffusion model is achieved by adding additional input conditions.

[0042] The Stable Diffusion model part: text encoder CLIP, noise prediction network U-Net, and image decoder VAE. First, the CLIP text encoder converts the input natural language into word vector features that can be recognized by the machine, and then inputs them into the U-Net network inside the latent space to control the U-Net network to iteratively denoise pure noise in the low dimension to generate a compressed image. Finally, the decoder of VAE restores the compressed image to the original pixel image to achieve text content control of image generation, such as Figure 1 Shown is the network architecture diagram of the Stable Diffusion model. The specific steps for the Stable Diffusion model to generate batik-style patterns are as follows:

[0043] a) During the forward diffusion process, noise is gradually added to the batik image, and the original data is gradually transformed into pure noise; assuming the initial data distribution is x0 ∼ q(x), the forward noise addition process is divided into multiple discrete time steps T. Through T rounds of continuously adding noise to the original batik image until the original batik image x0 is transformed into a pure Gaussian noise image xt. The Gaussian distribution q formula for the noise addition process from time t - 1 to time step t is expressed as:

[0044]

[0045] where, X t-1 represents the image state at time step t - 1; β t represents the noise parameter at time step t, a value between 0 and 1, used to control the intensity of the added noise; I is the identity matrix; N is the Gaussian distribution. As the time step T increases, the resulting distribution approaches the standard normal distribution. This process is a parameterized Markov chain, and the formula is expressed as:

[0046]

[0047] b) During the reverse inverse diffusion process, Gaussian noise xt randomly sampled from the standard normal distribution is gradually denoised. The noisy image is input into the denoising network to gradually remove the noise, and the original batik image data distribution is restored through the enhanced activation module. Finally, a real batik image x0 consistent with the initial data distribution is obtained. The denoising process formula from time step t to time step t - 1 is expressed as:

[0048]

[0049] where, μ θ (x t ,t) is the mean of the Gaussian noise predicted by the denoising network, is the variance of the predicted Gaussian noise.

[0050] Step 3: Model training: Use the dataset to train the batik LoRA stylization model;

[0051] The LoRA model adds a new weight matrix to the pre-training matrix of the original diffusion model. Using low-rank matrix factorization technology, the matrix is decomposed into the product of a dimensionality reduction matrix A and a dimensionality increase matrix B. By freezing the training weights of the original model and only training the newly added low-rank matrices A and B, the number of parameters in the fine-tuning process is reduced, significantly reducing computational resources. The specific steps for training the LoRA stylization model are as follows:

[0052] a) Use the Adam optimizer for learning optimization. Set the number of training epochs (Epoch) of the LoRA model to 20 times, save the model every 2 epochs, set the training batch size (Train batch Size) during training to 8, and set the learning rate (Learning rate) to 0.0001.

[0053] b) Conduct XYZ script comparison tests in the stable diffusion model to test the training effect of the LoRA model. Select NUM000016 as the optimal training model and call this model to fine-tune the stable diffusion model for batik image generation.

[0054] Step 4: Obtaining the optimal generation parameters of the diffusion model: Use text-to-image and image-to-image experiments to obtain the optimal parameters for batik image generation. Use the edge detection model ControlNet to extract the contour of the input image, and then input the extracted contour information into the trained stable diffusion model to achieve precise controllable generation of high-fidelity batik-style patterns.

[0055] The specific steps for obtaining the optimal generation parameters are as follows:

[0056] a) The stable diffusion model selects 5 sampling methods: Euler, Euler a, LMS, PLMS, and Heun, and the number of iterative steps is set to 15, 25, 35, and 45 respectively. The text prompt word correlation coefficient (CFG Scale) is uniformly set to 7.0.

[0057] b) The stable diffusion model selects two ControlNet models: diffusers_xl_canny_full and sargezt_xl_softedge. Taking 3 different contour edge patterns as input images, set the ControlNet weight range to 0.4 - 1, the prompt word guidance coefficient to 7.0, and the sampling method and the number of iterative steps according to the best sampling method Eluer a obtained from the previous experiment, and set the number of iterative steps to 35.

[0058] c) Compare the generation effects and image quality of the batik patterns in the two experiments, and determine that the generation parameter settings of the StableDiffusion model used are as follows: the sampler selects Euler a, the number of iteration steps is 35, ControlNet selects the Canny model, and the control model weight is 0.8.

[0059] Effect Analysis

[0060] To verify the effectiveness of the present invention, comparative test experiments are carried out on two image generation models, SD1.5 and the cyclic adversarial network CycleGAN. Compared with SD1.5 and CycleGAN, the method provided by the present application has obtained better scores in terms of FID and SSIM metrics. Table 1 shows the comparison results of the present invention with the other two models.

[0061] Table 1 Comparison Results of the Present Invention with the Other Two Models

[0062]

[0063]

[0064] As can be seen from Table 1, compared with SD1.5 and CycleGAN, the present invention has a lower FID metric score, indicating that the generated batik images have better effects and are more similar to real images in terms of image feature distribution; while the SSIM metric score is higher, 21.81% higher than the optimal result in the comparative method, indicating that the generated images are more excellent in terms of structure and texture details and have competitive performance in batik pattern generation.

[0065] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A pattern generation method based on fine-tuning of diffusion models, characterized in that: The specific steps of this method include: Step 1: Dataset construction: Obtain batik-style images and form a dataset through preprocessing; After the batik-style images are taken in kind, image preprocessing is carried out. The specific steps include: a) First, screen and eliminate low-quality images, and then perform sub-image segmentation on large-size long images to obtain sample images; b) Unify the sizes of the obtained sample images and divide them into a training set and a test set; c) Use an image label backpropagation tool to divide the input sample images into multiple image blocks of a fixed size, convert them into vector embeddings, and generate prompt words related to the semantic content of the image blocks; d) Use a label cleaning tool to clean the prompt word labels above a certain confidence level and delete redundant labels irrelevant to the image content; e) Add the crack texture features of batik images, add model trigger words to each image label, and refine and semantically supplement the text description of the labels; Step 2: Build an intelligent generation model for batik-style patterns: This generation model includes an edge detection model ControlNet and a stable diffusion model; The edge detection model ControlNet includes two parts of neural network blocks, locked and trainable. The locked part retains the performance of the original diffusion model because both the input and output of ControlNet are zero. The trainable part is trained to output by applying control conditions. The two are connected through a zero convolution layer, and precise control of the images generated by the diffusion model is achieved by adding additional input conditions; The stable diffusion model includes a text encoder CLIP, a noise prediction network U-Net, and an image decoder VAE. The text encoder CLIP converts the input natural language into word vector features, and then inputs them into the U-Net network inside the latent space to control the U-Net network to iteratively denoise pure noise in the low dimension to generate a compressed image. Finally, through the decoder VAE, the compressed image is restored to the original pixel image to realize text content control of image generation, where: a) In the forward diffusion process of the Stable Diffusion model, noise is gradually added to the batik image, and the original image data is gradually transformed into pure noise; assuming that the initial data distribution is x0 ~ q(x), the forward noise addition process is divided into multiple discrete time steps T. Through T rounds of continuously adding noise to the original batik image until the original batik image x0 is transformed into a pure Gaussian noise image x t , the Gaussian distribution q formula of the noise addition process from time t - 1 to time step t is expressed as: Among them, X t-1 represents the image state at time step t - 1; β t represents the noise parameter at time step t, a value between 0 and 1, used to control the intensity of the added noise; I is the identity matrix; N is the Gaussian distribution. As the time step T increases, the resulting distribution approaches the standard normal distribution. This process is a parameterized Markov chain, and the formula is expressed as: b) The stable diffusion model performs step-by-step denoising on the Gaussian noise x randomly sampled from the standard normal distribution during the reverse inverse diffusion process, inputs the noisy image into the denoising network to gradually remove the noise, restores the original batik image data distribution through the enhanced activation module, and finally obtains the real batik image x0 consistent with the initial data distribution. The probability distribution p of the denoising process from time step t to time step t-1 t is expressed by the formula as follows: θ is expressed as: Among them, μ θ (x t , t) is the mean of the Gaussian noise predicted by the denoising network, is the variance of the predicted Gaussian noise; Step 3: Model training: Use the dataset to train the intelligent generation model for batik-style patterns. Adopt the LoRA (Low-Rank Adaptation) technology for efficient fine-tuning of model parameters. Add a new weight matrix to the pre-training matrix of the stable diffusion model. Use the low-rank matrix decomposition technology to decompose the weight matrix into the product of a dimensionality reduction matrix A and a dimensionality increase matrix B. By freezing the training weights of the original diffusion model, only the newly added matrices A and B are trained, reducing the number of parameters in the fine-tuning process and significantly reducing computing resources; where: Use the Adam optimizer for learning optimization, set the number of training epochs Epoch of the LoRA model to 20 times, save the model once every 2 cycles, set the batch size during training to 8, and set the learning rate to 0.0001; Step 4: Intelligent generation of batik-style patterns: Use the edge detection model ControlNet to extract the contours of the input image, and then input the extracted contour information into the trained StableDiffusion model to achieve the precise and controllable generation of high-fidelity batik-style patterns.

2. Use the pattern generation method based on fine-tuning of the diffusion model according to claim 1, characterized in that: In Step 1, the size of the sample images is uniformly adjusted to 1024×1024 pixels, and the resolution is set to 300 DPI.

3. Use the pattern generation method based on fine-tuning of the diffusion model according to claim 1, characterized in that: In Step 1, the prompt word tags with a cleaning confidence of more than 40% are removed.

4. Use the pattern generation method based on fine-tuning of the diffusion model according to claim 1, characterized in that: In Step 2, the StableDiffusion model is the SDXL version, which supports generating images at a high pixel level of 1024×1024.

5. Use the pattern generation method based on fine-tuning of the diffusion model according to claim 1, characterized in that: In Step 2, the output of the edge detection model ControlNet in the 0 convolutional layer is all zero. The generative model diffus ers_xl_canny_full is used to control the generation of the image edge contours. The control model weight is 0.8, the sampler is Eluer a, and the number of iteration steps is 35.

Images