Method and system for generating artistic work figure painting based on diffusion model, and storage medium

Abstract

The invention discloses a diffusion model-based artistic work figure picture generation method and system and a storage medium. The generation method comprises the following steps of: generating a text potential vector; randomly generating a potential space representation, and inputting the potential space representation and the text potential vector into a main body of a pre-trained artistic character diffusion model to obtain model output; the model output is converted to a pixel space by adopting the VAE of a pre-trained artistic character diffusion model, and a character generation image is obtained; the method comprises the following steps: processing a figure to generate an image by adopting PPMatting, and segmenting a foreground and a background by predicting transparency of each pixel to obtain a background mask; inputting the character generation image and the background mask into a VAE of a pre-trained artistic pen flower bird diffusion model to obtain a first potential space representation; and inputting the first potential space representation and the text potential vector into a main body of a pre-trained artistic work, flower and bird diffusion model, inputting the model output of the main body into the VAE of the artistic work, flower and bird diffusion model, and decoding to obtain the artistic work figure picture.

Description

Technical Field

[0001] The invention relates to image generation technology, and in particular to a method, system and storage medium for generating meticulous figure paintings based on a diffusion model. Background Art

[0002] With the continuous evolution of technologies such as Generative Adversarial Networks (GAN) and Diffusion Models, image generation tasks have made great progress, promoting the development of creative design, virtual reality and other fields. The images generated by these models can achieve highly realistic effects, but when dealing with complex and sophisticated user needs, the generated results are often difficult to accurately control in accordance with human expectations and deviate from the target, especially in the presentation of specific styles or situations.

[0003] With the rise of cross-modal technology, text-based image generation technology has emerged, aiming to guide the generation of visual content through natural language. The core of this technology is to use advanced methods in deep learning, especially cross-modal contrastive learning models such as CLIP, to embed text and images into a unified semantic space, thereby achieving accurate matching of text and images. Compared with previous generation technologies, cross-modal methods give humans greater control, turning image generation from an automated process into a highly customized creation tool, thereby significantly enhancing the expressiveness and conformity of character-generated images.

[0004] Although text-based image generation technology has made significant progress in both theory and practice, there are still several technical challenges in practical applications. One of the most prominent bottlenecks is the adaptability of the model when facing the generation of specific stylized images, especially when generating traditional art styles such as Chinese fine brushwork figure paintings, the model's expressiveness is still insufficient. The lack of data sets and the limitations of complex semantic understanding directly affect the generation accuracy and detail control of the model, restricting its wide application in niche areas. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the method, system and storage medium for generating meticulous figure paintings based on a diffusion model provided by the present invention solve the problem of poor precision when the existing model generates meticulous figure paintings.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] In a first aspect, a method for generating a meticulous figure painting based on a diffusion model is provided, comprising the steps of:

[0008] S1, read the text information input by the user and convert it into a text potential vector using the CLIP model;

[0009] S2, randomly generate a latent space representation, and input it and the text latent vector into the main body of the pre-trained fine brushwork figure diffusion model to obtain the model output after reverse denoising;

[0010] S3, using the variational autoencoder VAE of the pre-trained fine brushwork figure diffusion model to convert the model output to the pixel space, and obtain the character generated image corresponding to the text latent vector;

[0011] S4, using PPMatting to process the character generation image, segmenting the foreground and background by predicting the transparency of each pixel, and obtaining the background mask;

[0012] S5, inputting the character generated image and the background mask into the variational autoencoder VAE of the pre-trained fine-brush flower-and-bird diffusion model to obtain the first latent space representation;

[0013] S6. Input the first latent space representation and the text latent vector into the main body of the pre-trained meticulous flower-and-bird diffusion model, and input the model output of the main body into the variational autoencoder VAE of the meticulous flower-and-bird diffusion model to decode and obtain the meticulous figure painting.

[0014] Furthermore, the expression of the meticulous figure painting is decoded as follows:

[0015]

[0016] Among them, I new For meticulous figure painting; M bg is the background mask; is the model output of the main body of the pre-trained meticulous figure diffusion model; ε is the mapping of the variational autoencoder VAE of the meticulous flower and bird diffusion model from the latent space to the pixel space; σ is the activation function; W is the original pre-trained weight matrix of the deep learning layer of the meticulous flower and bird diffusion model; B (D′) and A (D′) All of them are low-rank matrices of the meticulous flower and bird diffusion model using the meticulous flower and bird painting dataset in weight update; D is the meticulous figure painting training set for training the meticulous figure diffusion model; D′ is the meticulous flower and bird painting dataset for training the meticulous flower and bird diffusion model; s is an adjustable scaling factor; b L is the offset vector of the Lth deep learning layer of the main body of the fine-brush flower-and-bird diffusion model; T is the transpose.

[0017] Furthermore, the meticulous figure painting training set includes a number of meticulous figure paintings and text information describing each meticulous figure painting; the meticulous flower and bird painting data set includes a number of meticulous flower and bird paintings and text information describing each meticulous flower and bird painting.

[0018] Furthermore, the expressions of the model outputs of the main bodies of the meticulous figure diffusion model and the meticulous flower and bird diffusion model are the same, specifically:

[0019] I L =σ((W+s·BA T )X L +b L )

[0020] Among them, I L is the model output of the subject; σ is the activation function; W is the original pre-trained weight matrix of the deep learning layer of the corresponding diffusion model; s is an adjustable scaling factor; B and A are both low-rank matrices of the subject in the weight update of the diffusion model; X L is the input feature of the Lth deep learning layer of the main body of the diffusion model; b L is the offset vector of the Lth layer of the subject.

[0021] Furthermore, the pre-training method of the meticulous figure diffusion model and the meticulous flower and bird diffusion model is the same;

[0022] The pre-training method of the meticulous figure diffusion model includes:

[0023] A1. Input the meticulous figure painting training set into the variational autoencoder VAE to extract the latent space representation of the meticulous figure paintings in the meticulous figure painting training set;

[0024] A2. The low-rank adaptation model Lora is used to introduce a low-rank matrix to adjust the weights of the deep learning layer based on the self-attention mechanism of the main body of the diffusion model;

[0025] A3. After the weight adjustment is completed, the main body of the diffusion model is used to inject noise into the latent space representation step by time to obtain a latent space representation close to pure noise. Then, the noise is removed from the latent space representation step by time to obtain a pre-trained fine brushwork figure diffusion model.

[0026] Furthermore, the expression of the latent space representation is:

[0027]

[0028] Where z0 is the latent space representation obtained in step A1; x is the meticulous flower and bird painting in the meticulous flower and bird painting dataset or the meticulous figure painting in the meticulous figure painting training set; μ φ (x) and Σ φ (x) are the mean and variance of x respectively; ∈ is the normal distribution The noise sampled in

[0029] In step A3, the main body of the diffusion model after adding noise at the tth time step is expressed as:

[0030]

[0031] Among them, z t is the latent space representation after adding noise to the t-th time step; β t is the noise adjustment parameter at the tth time step.

[0032] Furthermore, the loss function of the diffusion model is expressed as:

[0033]

[0034] Among them, Loss is the loss function; y is the text information describing x; is the square of the Euclidean norm; ∈θ is the prediction of the diffusion model on the noise at the time step; W is the original pre-trained weight matrix of the deep learning layer of the diffusion model; B and A are both low-rank matrices in the weight update of the diffusion model; τ θ A is a conditional encoder combined with an address adaptation module; T is the transpose of the low-rank matrix A.

[0035] In a second aspect, a system for generating meticulous figure paintings based on a diffusion model is provided, which comprises:

[0036] The text latent vector generation module is used to read the text information input by the user and convert it into a text latent vector using the CLIP model;

[0037] The character generated image generation module is used to randomly generate a latent space representation, input it and the text latent vector into the main body of the pre-trained fine brushwork character diffusion model, and obtain the model output after reverse denoising; the variational autoencoder VAE of the pre-trained fine brushwork character diffusion model is used to convert the model output to the pixel space to obtain the character generated image corresponding to the text latent vector;

[0038] The background mask generation module is used to process the character generation image using PPMatting, segment the foreground and background by predicting the transparency of each pixel, and obtain the background mask;

[0039] The meticulous figure painting generation module is used to input the character generation image and the background mask into the variational autoencoder VAE of the pre-trained meticulous flower and bird diffusion model to obtain the first latent space representation; input the first latent space representation and the text latent vector into the main body of the pre-trained meticulous flower and bird diffusion model, and input the model output of the main body into the variational autoencoder VAE of the meticulous flower and bird diffusion model to decode and obtain the meticulous figure painting.

[0040] In a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, a method for generating a meticulous figure painting based on a diffusion model is implemented.

[0041] The beneficial effects of the present invention are as follows: when generating a meticulous figure painting, the present scheme first generates a figure generation image based on the text information input by the user in combination with a pre-trained meticulous figure diffusion model, and then combines PPMatting to obtain a background mask, keeping the figure part in the image unchanged, and using the pre-trained meticulous flower and bird diffusion model to regenerate the background part, thereby being able to more accurately map complex text to the image and improving the degree of image-text matching.

[0042] When pre-training the diffusion model, by introducing low-rank matrices, an effective large-model training and adjustment method is provided, allowing the model to quickly adapt to new tasks and fields; the fine-brush figure painting is divided into characters and backgrounds, and low-rank matrices are trained separately and given text to guide the generation, which enhances the generative model (pre-trained diffusion model)'s understanding of complex texts; and allows independent adjustment and optimization of the character and background parts of the image.

[0043] This scheme trains diffusion models for figures and flower-and-bird backgrounds in meticulous figure painting respectively. By highlighting the figures in meticulous figure painting and introducing meticulous flower-and-bird data, the diffusion models are divided into figures and background parts as data sets to train the diffusion models in conjunction with the low-rank adaptation model, thereby achieving local generation of figures and backgrounds in meticulous figure painting, making up for the scarcity of meticulous figure painting data sets and the lack of expressiveness of the low-rank adaptation module trained only with meticulous figure paintings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of the method for generating meticulous figure paintings based on the diffusion model.

[0045] Figure 2 Flowchart for generating character images for the pre-trained Gongbi character diffusion model.

[0046] Figure 3 Flowchart for generating fine-brush figure paintings for the pre-trained fine-brush flower-and-bird diffusion model.

[0047] Figure 4 This is a schematic diagram of the meticulous figure painting in the meticulous figure painting training set.

[0048] Figure 5 This is a schematic diagram of the meticulous flower and bird paintings in the meticulous flower and bird painting dataset.

[0049] Figure 6 This is a comparison chart of the effects of the mainstream generation model and this method in the task of generating fine brushwork figure paintings. DETAILED DESCRIPTION

[0050] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0051] refer to Figure 1 , Figure 1 A flowchart of a method for generating a fine-brush figure painting based on a diffusion model is shown; Figure 1 As shown, the method S includes steps S1 to S6.

[0052] In step S1, the text information input by the user is read and converted into a text latent vector using the CLIP model; the CLIP model is a relatively mature model in the prior art, and how the CLIP model obtains the text latent vector will not be described in detail here.

[0053] In step S2, a latent space representation is randomly generated and input into the main body of the pre-trained fine brushwork figure diffusion model together with the text latent vector to obtain the model output after reverse denoising.

[0054] During implementation, this solution preferably uses the same expression for the model output of the main body of the meticulous figure diffusion model and the meticulous flower and bird diffusion model, specifically:

[0055] I L =σ((W+s·BA T )X L +b L )

[0056] Among them, I L is the model output of the subject; σ is the activation function; W is the original pre-trained weight matrix of the deep learning layer of the diffusion model; s is an adjustable scaling factor; B and A are both low-rank matrices of the subject in the diffusion model in weight update; X L is the input feature of the Lth deep learning layer of the main body of the diffusion model; b L is the offset vector of the Lth layer of the subject.

[0057] In step S3, a variational autoencoder VAE of a pre-trained fine-brush figure diffusion model is used to convert the model output into a pixel space to obtain a generated character image corresponding to the text latent vector;

[0058] The processing of steps S1 to S3 can be referred to Figure 2 , Figure 2The image in the middle left of the figure is a randomly generated latent space representation, and the bottom row is the operation of generating a text latent vector based on the text information input by the user; U-NET refers to the main part of the diffusion model, and the input is the randomly generated latent space representation and text latent vector; the model output of U-NET is input into the variational autoencoder VAE, and the rightmost character image is the generated character image.

[0059] In step S4, PPMatting is used to process the character generation image, and the foreground and background are segmented by predicting the transparency of each pixel to obtain a background mask.

[0060] In step S5, the character generated image and the background mask are input into the variational autoencoder VAE of the pre-trained fine-brush flower and bird diffusion model to obtain a first latent space representation;

[0061] In step S6, the first latent space representation and the text latent vector are input into the main body of the pre-trained meticulous flower-and-bird diffusion model, and the model output of the main body is input into the variational autoencoder VAE of the meticulous flower-and-bird diffusion model to be decoded to obtain the meticulous figure painting.

[0062] The processing of steps S5 to S6 can refer to Figure 3 , Figure 3 The upper middle part on the left shows the input of the generated image and the background mask into the variational autoencoder VAE. The bottom row is the operation of generating a text latent vector based on the text information input by the user. Then, the first latent space representation of the variational autoencoder VAE and the text latent vector are input into U-NET, and the model output of U-NET is input into the variational autoencoder VAE to obtain the meticulous figure painting on the far right, which is the meticulous figure painting that the user hopes to output from the diffusion model.

[0063] During implementation, this solution preferably decodes the expression of fine brushwork figure painting as follows:

[0064]

[0065] Among them, I new For meticulous figure painting; M bg is the background mask; is the model output of the main body of the pre-trained meticulous figure diffusion model; ε is the mapping of the variational autoencoder VAE of the meticulous flower and bird diffusion model from the latent space to the pixel space; σ is the activation function; W is the original pre-trained weight matrix of the deep learning layer of the meticulous flower and bird diffusion model; B (D′) and A (D′) All of them are low-rank matrices in the weight update of the meticulous flower and bird diffusion model using the meticulous flower and bird painting dataset; D is the meticulous figure painting training set for training the meticulous figure diffusion model; D′ is the meticulous flower and bird painting dataset for training the meticulous flower and bird diffusion model; bL is the offset vector of the Lth deep learning layer of the main body of the fine-brush flower-and-bird diffusion model; T is the transpose.

[0066] The meticulous figure painting training set of this scheme includes several meticulous figure paintings and text information describing each meticulous figure painting; the meticulous flower and bird painting data set includes several meticulous flower and bird paintings and text information describing each meticulous flower and bird painting; in order to facilitate a clearer understanding of meticulous figure paintings and meticulous flower and bird paintings, Figure 4 and Figure 5 Several illustrations of meticulous figure painting and meticulous flower and bird painting are given respectively.

[0067] In one embodiment of the present invention, the pre-training method of the meticulous figure diffusion model and the meticulous flower and bird diffusion model is the same, except that the data sets used to train the models are different. The pre-training method of the meticulous figure diffusion model includes:

[0068] A1. Input the meticulous figure painting training set into the diffusion model, and use the variational autoencoder VAE of the diffusion model to extract the latent space representation of the meticulous figure paintings in the meticulous figure painting training set:

[0069]

[0070] Where z0 is the latent space representation obtained in step A1; x is the meticulous flower and bird painting in the meticulous flower and bird painting dataset or the meticulous figure painting in the meticulous figure painting training set; μ φ (x) and Σ φ (x) are the mean and variance of fine-brush figure paintings respectively; x is the fine-brush figure paintings in the fine-brush figure painting training set; ∈ is the normal distribution The noise sampled in .

[0071] A2. The low-rank adaptation model Lora is used to introduce a low-rank matrix to adjust the weights of the deep learning layer based on the self-attention mechanism of the main body of the diffusion model;

[0072] A3. After the weight adjustment is completed, the main body of the diffusion model is used to inject noise into the latent space representation step by time to obtain a latent space representation close to pure noise. Then, the noise is removed from the latent space representation step by time to obtain a pre-trained fine brushwork figure diffusion model.

[0073] In step A3, the main body of the diffusion model after adding noise at the tth time step is expressed as:

[0074]

[0075] Among them, z t is the latent space representation after adding noise to the t-th time step; β tis the noise adjustment parameter at the tth time step.

[0076] During implementation, the loss function of the optimal diffusion model of this scheme is expressed as:

[0077]

[0078] Among them, Loss is the loss function; y is the text information describing x; is the square of the Euclidean norm; ∈θ is the prediction of the diffusion model on the noise at the time step; W is the original pre-trained weight matrix of the deep learning layer of the diffusion model; B and A are both low-rank matrices in the weight update of the diffusion model; τ θ A is a conditional encoder combined with an address adaptation module; T is the transpose of the low-rank matrix A.

[0079] The present invention also provides a system for generating meticulous figure paintings based on a diffusion model, which includes:

[0080] The text latent vector generation module is used to read the text information input by the user and convert it into a text latent vector using the CLIP model;

[0081] The character generated image generation module is used to randomly generate a latent space representation, input it and the text latent vector into the main body of the pre-trained fine brushwork character diffusion model, and obtain the model output after reverse denoising; the variational autoencoder VAE of the pre-trained fine brushwork character diffusion model is used to convert the model output to the pixel space to obtain the character generated image corresponding to the text latent vector;

[0082] The background mask generation module is used to process the character generation image using PPMatting, segment the foreground and background by predicting the transparency of each pixel, and obtain the background mask;

[0083] The meticulous figure painting generation module is used to input the character generation image and the background mask into the variational autoencoder VAE of the pre-trained meticulous flower and bird diffusion model to obtain the first latent space representation; input the first latent space representation and the text latent vector into the main body of the pre-trained meticulous flower and bird diffusion model, and input the model output of the main body into the variational autoencoder VAE of the meticulous flower and bird diffusion model to decode and obtain the meticulous figure painting.

[0084] The present solution also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a method for generating a meticulous figure painting based on a diffusion model is implemented.

[0085] In order to verify the effectiveness of this method, this method is compared with other advanced models in the text-to-image generation task of fine brushwork figure painting:

[0086] This scheme conducted a comparative experiment on the text-to-image generation task with other advanced models (such as DALL-E3 and StableDiffusion). The specific generated text description was: "Chinese meticulous figure painting, girl, black hair, wearing a head flower, Chinese clothing, closed eyes, flower background, Chinese background". The generated text description was input into the method of this scheme (ours), DALL-E3 model and StableDiffusion model respectively to generate meticulous figure painting. At the same time, the generated text description was sent to multiple designers with professional painting skills. Based on the understanding of the text, meticulous figure painting was drawn, that is, Figure 6 Hand-draw in.

[0087] The hand-drawn meticulous figure paintings and the meticulous figure paintings generated by our method (ours), DALL-E3 model and StableDiffusion model are respectively referenced Figure 6 ,pass Figure 6 By comparison:

[0088] This method shows higher accuracy in facial features and clothing details, and shows better coordination in style consistency and the integration of characters and backgrounds, which is closer to hand-drawn meticulous figure paintings. In comparison, other models are slightly insufficient in generating results when dealing with the complex details and brushstrokes unique to meticulous paintings.

[0089] It can be seen that this method has a stronger ability in processing long text descriptions, can more accurately map complex text to images, and improves the degree of image-text matching.

Claims

1. A method for generating meticulous figure paintings based on a diffusion model, characterized in that: Includes steps: S1, read the text information input by the user and convert it into a text potential vector using the CLIP model; S2, randomly generate a latent space representation, and input it and the text latent vector into the main body of the pre-trained fine brushwork figure diffusion model to obtain the model output after reverse denoising; S3, using the variational autoencoder VAE of the pre-trained fine brushwork figure diffusion model to convert the model output to the pixel space, and obtain the character generated image corresponding to the text latent vector; S4, using PPMatting to process the character generation image, segmenting the foreground and background by predicting the transparency of each pixel, and obtaining the background mask; S5, inputting the character generated image and the background mask into the variational autoencoder VAE of the pre-trained fine-brush flower-and-bird diffusion model to obtain the first latent space representation; S6. Input the first latent space representation and the text latent vector into the main body of the pre-trained meticulous flower-and-bird diffusion model, and input the model output of the main body into the variational autoencoder VAE of the meticulous flower-and-bird diffusion model to decode and obtain the meticulous figure painting.

2. The method for generating meticulous figure paintings based on a diffusion model according to claim 1, characterized in that: The decoding expression of fine brushwork figure painting is: Among them, I new For meticulous figure painting; M bg is the background mask; is the model output of the main body of the pre-trained meticulous figure diffusion model; ε is the mapping of the variational autoencoder VAE of the meticulous flower and bird diffusion model from the latent space to the pixel space; σ is the activation function; W is the original pre-trained weight matrix of the deep learning layer of the meticulous flower and bird diffusion model; B(D′) and A(D′) are both low-rank matrices of the meticulous flower and bird diffusion model using the meticulous flower and bird painting dataset in weight update; D is the meticulous figure painting training set for training the meticulous figure diffusion model; D′ is the meticulous flower and bird painting dataset for training the meticulous flower and bird diffusion model; s is an adjustable scaling factor; b L is the offset vector of the Lth deep learning layer of the main body of the fine-brush flower and bird diffusion model; T is the transpose.

3. The method for generating meticulous figure paintings based on a diffusion model according to claim 2, characterized in that: The meticulous figure painting training set includes a number of meticulous figure paintings and text information describing each meticulous figure painting; the meticulous flower and bird painting data set includes a number of meticulous flower and bird paintings and text information describing each meticulous flower and bird painting.

4. The method for generating meticulous figure paintings based on a diffusion model according to claim 2, characterized in that: The expressions of the main model outputs of the meticulous figure diffusion model and the meticulous flower and bird diffusion model are the same, specifically: I L =σ((W+s·BA T )X L +b L ) Among them, I L is the model output of the subject; σ is the activation function; W is the original pre-trained weight matrix of the deep learning layer of the diffusion model; B and A are both low-rank matrices of the subject in the weight update of the diffusion model; X L is the input feature of the Lth deep learning layer of the main body of the diffusion model; b L is the offset vector of the Lth layer of the subject.

5. The method for generating meticulous figure paintings based on a diffusion model according to any one of claims 1 to 4, characterized in that: The pre-training method of the meticulous figure diffusion model and the meticulous flower and bird diffusion model is the same; The pre-training method of the meticulous figure diffusion model includes: A1. Input the meticulous figure painting training set into the diffusion model, and use the variational autoencoder (VAE) of the diffusion model to encode the meticulous figure paintings in the meticulous figure painting training set to obtain the latent space representation; A2. The low-rank adaptation model Lora is used to introduce a low-rank matrix to adjust the weights of the deep learning layer based on the self-attention mechanism of the main body of the diffusion model; A3. After the weight adjustment is completed, the main body of the diffusion model is used to inject noise into the latent space representation step by time to obtain a latent space representation close to pure noise. Then, the noise is removed from the latent space representation step by time to obtain a pre-trained fine brushwork figure diffusion model.

6. The method for generating meticulous figure paintings based on a diffusion model according to claim 5, characterized in that: The expression of latent space representation is: Where z0 is the latent space representation obtained in step A1; x is the meticulous flower and bird painting in the meticulous flower and bird painting dataset or the meticulous figure painting in the meticulous figure painting training set; μ φ (x) and Σ φ (x) are the mean and variance of x respectively; ∈ is the normal distribution The noise sampled in In step A3, the main body of the diffusion model after adding noise at the tth time step is expressed as: Among them, z t is the latent space representation after adding noise to the t-th time step; β t is the noise adjustment parameter at the tth time step.

7. The method for generating meticulous figure paintings based on a diffusion model according to claim 6, characterized in that: The loss function of the diffusion model is expressed as: Among them, Loss is the loss function; y is the text information describing x; is the square of the Euclidean norm; ∈θ is the prediction of the diffusion model on the noise at the time step; W is the original pre-trained weight matrix of the deep learning layer of the diffusion model; B and A are both low-rank matrices in the weight update of the diffusion model; τ θ A is a conditional encoder combined with a low-rank adaptation module; T is the transpose of the low-rank matrix A.

8. A system for generating meticulous figure paintings based on a diffusion model, characterized in that: include: The text latent vector generation module is used to read the text information input by the user and convert it into a text latent vector using the CLIP model; The character generated image generation module is used to randomly generate a latent space representation, input it and the text latent vector into the main body of the pre-trained fine brushwork character diffusion model, and obtain the model output after reverse denoising; the variational autoencoder VAE of the pre-trained fine brushwork character diffusion model is used to convert the model output to the pixel space to obtain the character generated image corresponding to the text latent vector; The background mask generation module is used to process the character generation image using PPMatting, segment the foreground and background by predicting the transparency of each pixel, and obtain the background mask; The meticulous figure painting generation module is used to input the character generation image and the background mask into the variational autoencoder VAE of the pre-trained meticulous flower and bird diffusion model to obtain the first latent space representation; input the first latent space representation and the text latent vector into the main body of the pre-trained meticulous flower and bird diffusion model, and input the model output of the main body into the variational autoencoder VAE of the meticulous flower and bird diffusion model to decode and obtain the meticulous figure painting.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a meticulous figure painting based on a diffusion model as described in any one of claims 1 to 7 is implemented.

Images