Controllable generation method for text on image based on large diffusion model

Abstract

The invention discloses a method for controllably generating a text on an image based on a large diffusion model. According to the method, an image required by a user can be customized and generated. Specifically, firstly, a Transform is used for carrying out semantic extraction on cue words input by a user, then a font layout image meeting the requirements of the user is generated, the font layout image and the cue words of the user are input into a control network to generate a control vector, and the purpose of controllable generation is achieved by guiding a gradual noise reduction process. And the effect of controllable generation is enhanced through character perception loss. Besides, the method provides an effective parameter fine tuning strategy to avoid over-fitting or disastrous forgetting of the generation model, so that coherent text generation is realized under the condition that only updating is less than 3% of total parameters, and the convergence process of the model is remarkably accelerated. And finally, for the target region generation part, in order to reduce the influence caused by the background part, a background reconstruction mechanism is used to reserve the non-target region part in the original image, and the loss of background information is reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision image generation, and in particular relates to a controllable generation method of text on an image based on a diffusion macro model. Background Art

[0002] Recently, research on large-scale diffusion model-based text-to-image generation has achieved remarkable success. However, due to the ambiguity of natural language, the effect of using only text as a condition to control image generation is still not ideal when conveying subtle visual details. Even using highly detailed text descriptions cannot achieve precise control, so enhancing the controllability of text-to-image generation has always been an important research direction.

[0003] At present, the controllability of text-to-image research is mainly achieved through customized learning, adding additional control conditions, etc. Customized learning aims to controllable generation by capturing and utilizing concepts as generation conditions. These conditions are not suitable for description through text, but are extracted from example images. However, since there are only a few example images, the generated images will have the problem of insufficient diversity. The method of adding additional control conditions generally achieves controllable generation of text-to-image by enhancing text descriptions or adding image conditions. The method of enhancing text descriptions may cause text misalignment when involving multiple topics or complex text descriptions, and it is difficult for text to represent structural information. In addition, the text encoders in existing methods usually lack character-level perception when dealing with token length. The method of adding image conditions as additional control conditions may have the problem of conflict between text conditions and image conditions, and there may also be problems with incomplete semantic matching between text and image.

[0004] Therefore, how to ensure the customized generation effect of the diffusion model while enabling it to learn the corresponding semantic information and achieve a more powerful and controllable text-to-image capability is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the shortcomings of the prior art, the present invention proposes a controllable method for generating text on an image based on a diffusion macro model. The method focuses on generating customized text images that meet the personalized needs of users and are consistent with the control conditions provided by users.

[0006] The technical solution of the present invention to solve the technical problem is to design a controllable method for generating text on an image based on a diffusion model, characterized in that the method comprises the following steps:

[0007] Step 1: Get training data

[0008] The open source dataset MARIO-10M is used, in which one piece of data is a picture containing a keyword, a corresponding text description containing the keyword, and the actual layout information of the corresponding keyword on the picture;

[0009] Step 2: Build an image generation network

[0010] The image generation network includes a pre-trained SD network model, an OCR detection module, a Transformer encoding-decoding module, and a control network;

[0011] The SD network model includes a VAE encoder, a U-Net network, a VAE decoder, a text encoder, and a picture encoder; the U-Net network is a diffusion model, which consists of an SD encoder, an SD middle layer, and an SD decoder;

[0012] The control network includes the first zero-convolution layer, the encoding module, and the zero-convolution module, wherein the encoding module is composed of a structure of a replicated SD encoder and an SD intermediate layer; the number of layers of the zero-convolution module is equal to the sum of the number of layers of the SD intermediate layer and the SD decoder;

[0013] Step 3: Training of Transformer Encoding-Decoding Module

[0014] The Transfomer encoding-decoding module is trained using the text descriptions in the training set and the corresponding real layout of keywords;

[0015] For the text description A, the Transformer encoding-decoding module is used to obtain the keyword text layout. The specific process is as follows:

[0016] First, tokenize the text description, that is:

[0017] Q(A)=(q0,q1,……,q L-1 )

[0018] Where Q(A) refers to the tokenized representation of the text description and L is its length;

[0019] Operate on Q(A) to get the corresponding embedding representation Emb(Q(A)):

[0020] Emb(Q(A))=CLIP(Q(A))+Position(Q(A))+Key(Q(A))+Width(Q(A))

[0021] Where CLIP(Q(A)) represents the sequence encoding obtained by the CLIP model, Position(Q(A)) represents the learnable position embedding, Key(Q(A)) represents the keyword embedding, and Width(Q(A)) represents the width of the keyword with the embedding layer;

[0022] The text description A is processed into the corresponding keyword text layout F(A) by the Transformer encoding-decoding module:

[0023] D(A)=ΦD(ΦE(Emb(Q(A))))=(f0,f1,...,f K-1 )

[0024] Among them, ΦD represents the Transfomer decoder, ΦE represents the Transfomer encoder;

[0025] Compare the obtained keyword text layout F(A) with the keyword real layout F(A) GT Assume L1 loss:

[0026] loss layout =|F(A) GT -F(A)|

[0027] The above loss is back-propagated, and the parameters in the Transformer encoding-decoding module are iteratively updated. The training of the Transformer encoding-decoding module is completed after the set number of iterations.

[0028] Step 4: Fine-tuning training of the U-Net network of the SD network model

[0029] The SD network model adopts the Stable Diffusion v1.5 pre-trained model, in which the VAE encoder, U-Net network, VAE decoder, text encoder, and image encoder all have pre-trained network parameter values, and the control network is the pre-trained GlyphControl control network;

[0030] Freeze the network parameters of the VAE encoder, VAE decoder, text encoder, image encoder and control network, and use the training set data to fine-tune the network parameters of the cross-attention module of the U-Net network; for a piece of data in the training set, the image x and the corresponding text description A are first encoded into a latent space representation X through the VAE encoder, and then the latent space representation X is gradually denoised: the sampling time step T~Uniform(0,T max ), and sample a noise ∈ that satisfies the Gaussian distribution to gradually destroy the original representation, and obtain the noisy representation Z t :

[0031]

[0032] where a t is the diffusion coefficient at time step t, ∈ is Gaussian noise;

[0033] At the same time, the image x is processed by the OCR detection module to obtain the corresponding target generation area, and then the target generation area is encoded to obtain the encoded target generation area mask image I m ;

[0034] For the keywords in the text description A, the corresponding keyword real layout F(A) is used GT , and render it to get the keyword text layout image g m ;

[0035] Layout the keyword text image m , target generation area mask image I m And characterize Z t Splice and get the combined noise Z' t :

[0036] Z' t =con(Z t ,g m ,I m )

[0037] For keyword text layout image g m , after being processed by the image encoder, it is encoded into the keyword text layout image embedding vector e g :

[0038] e g =Φe g (g m )

[0039] where ΦE g Represents a picture encoder;

[0040] On the other hand, the text description A is processed by the text encoder to obtain the text embedding vector e t :

[0041] e t =ΦE t (Q(A))

[0042] where ΦE t Represents a text encoder;

[0043] Embed keyword text layout image into vector e g , text embedding vector e t , merge noise Z' t As the input of the control network, the control vector C is obtained:

[0044] C=Cont(Z' t ;e t ;e g )

[0045] Where Cont represents the pre-trained control network;

[0046] The noise Z' will be merged t , text embedding vector e t The control vector C is input into the U-Net network, and the final image latent vector is obtained through step-by-step iterative denoising. The U-Net network is optimized by the mean square error of the noise added during the denoising process and the noise predicted during the denoising process in each iterative step, which is specifically expressed as follows:

[0047]

[0048] ∈ represents the noise added in each iterative step during the noise addition process, ∈ θ (Z' t ,T,Q,C) represents the noise predicted at the corresponding iteration step in the denoising process;

[0049] Layout image g by the above keyword text m After the mask operation, the character-level segmentation mask M is obtained. In each iteration, the corresponding image latent vector is sampled by another pre-trained U-Net network to obtain the character-level segmentation mask M'. From this, the character perception loss can be obtained as:

[0050]

[0051] in represents the cross entropy loss;

[0052] Using the hyperparameter λ to balance the U-Net network loss and the character perception loss, the final loss is as follows:

[0053] loss=loss U-Net +λ×loss char

[0054] During the training process, back propagation is used to update the parameters in the U-Net network. After two rounds of training on the training set data, the training is stopped to obtain a trained U-Net network model.

[0055] Step 5: Generate image

[0056] The given target image I is recorded as the background image template, and the target keyword text layout image G is recorded as the foreground image. The foreground image is obtained by processing the target text description P through the Transformer encoding-decoding module trained in step 3 and then rendering;

[0057] Generate a mask image I of the target generation area according to the anchor box drawn by the user mask , combined with the mask image I of the target generation area maskGradually add noise to the background image template and combine it with the mask image I of the target generation area mask Gradually add noise to the foreground image to obtain the background noise map and foreground noise map Combining the two can get the combined noise image Z * ; The method of stepwise noise processing is as described in step 4;

[0058] Input the target keyword text layout image G into the image encoder in step 4 to obtain the conditional control latent vector v, and input the target text description P into the text encoder in step 4 to obtain the text conditional vector w;

[0059] The noise image Z * , conditional control latent vector v, text condition vector w as the input of the control network in step 4, and obtain the control vector; the noise image Z * , the text condition vector w and the control vector are input into the U-Net network, and the U-Net network outputs the generated image latent vector of the first round; the generated image latent vector is used as the input of the control network and the U-Net network in the second round;

[0060] At round t, the corresponding control vector It needs to be obtained based on the latent vector of the generated image in the t-1th round; specifically, the latent vector of the generated image in the t-1th round The conditional control latent vector v and the text condition vector w are input into the control network, and the control vector at round t can be output. The formula is as follows:

[0061]

[0062] Where ΦCON(;·;·) represents the control network;

[0063] Next, the latent vector of the image generated in round t-1 is The text condition vector w is input into the U-Net network trained in step 4, and then combined with the control vector Add them together to get the hidden vector of the image generated in the tth round The formula is as follows:

[0064]

[0065] Where UNet(·;·) represents the U-Net network that has completed the training above;

[0066] In the iterative process, the generated background image is replaced by the reconstructed background image, which can be expressed as:

[0067]

[0068] Among them, in the formula, represents the background image after reconstruction, is the generated image latent vector of the tth iteration, I mask is the target generation area mask image, is the noise image of the background area, ⊙ represents the dot product operation, T represents the maximum number of iterations, and λ is the threshold parameter that controls the time step;

[0069] The process of obtaining the latent vector of the generated image is repeated until the number of iterations reaches T, and the final latent vector of the generated image is obtained, and then the final generated image can be obtained by VAE decoding; the generated image is an image in which the target keyword is generated in the target generation area of ​​the target image I.

[0070] Compared with the prior art, the present invention has the following beneficial effects: the controllable generation method of text on images based on the diffusion large model designed by the present invention can customize the images required by users. Specifically, firstly, the prompt words input by the user are semantically extracted by using Transformer, and then a font layout image that meets the user's needs is generated. The font layout image and the user prompt words are input into the control network to generate a control vector, and the purpose of controllable generation is achieved by guiding the step-by-step noise reduction process, and the effect of controllable generation is enhanced by character perception loss. In addition, the method of the present invention proposes an effective parameter fine-tuning strategy to avoid overfitting or catastrophic forgetting of the generative model. Specifically, only W of the cross-attention layer is fine-tuned during the learning phase. K and W V The two parts of parameters achieve coherent text generation with less than 3% of the total parameters updated, significantly accelerating the convergence process of the model. Finally, for the target area generation part, in order to reduce the impact of the background part, the background reconstruction mechanism is used to retain the non-target area of ​​the original image to improve the background deviation between the generated image and the original image and reduce the loss of background information. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A schematic diagram of the process and principle of fine-tuning training of a U-Net network of an SD network model in an embodiment of a controllable generation method of text on an image based on a diffusion model of the present invention.

[0072] Figure 2 The present invention is a schematic diagram of the image generation process and principle of an embodiment of a method for controllable generation of text on an image based on a diffusion large model.

[0073] Figure 3It is a structural diagram of a U-Net network and a control network of an embodiment of a method for controllable generation of text on an image based on a diffusion large model of the present invention. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0075] The present invention provides a controllable generation method of text on an image based on a diffusion model (hereinafter referred to as method, see Figure 1-3 ), the method comprises the following steps:

[0076] Step 1: Get training data

[0077] The open source dataset MARIO-10M is selected. The dataset MARIO-10M contains more than 10 million image-text pairs from various data sources such as natural images, posters, book covers, etc. The text is a description of the image, that is, one of the data is a picture containing keywords, the corresponding text description containing keywords, and the actual layout information of the corresponding keywords on the picture. The picture contains the keyword V, and the corresponding text description consists of a fixed sentence plus keywords. For example, a Harry Potter poster with the words "Harry Potter" printed on it has a corresponding text description: A movie poster of "Harry Potter", V = Harry Potter.

[0078] Step 2: Build an image generation network

[0079] The image generation network includes a pre-trained SD (Stable Diffusion v1.5 pre-trained model) network model, an OCR detection module, a Transfomer encoding-decoding module, and a control network.

[0080] The SD network model includes a VAE encoder, a U-Net network, a VAE decoder, a text encoder, and a picture encoder. The U-Net network is a diffusion model, which consists of an SD encoder, an SD middle layer, and an SD decoder;

[0081] The control network includes the first zero-convolution layer, the encoding module, and the zero-convolution module, wherein the encoding module is composed of a structure of a replicated SD encoder and an SD intermediate layer; the number of layers of the zero-convolution module is equal to the sum of the number of layers of the SD intermediate layer and the SD decoder. The control condition is first input into the first zero-convolution layer of the control network, and the result is added to the noise feature input into the U-Net network, and then processed by each layer of the encoding module in turn. The output of each layer of the encoding module is input into the zero-convolution module from the bottom layer to the top layer in a top-to-bottom order; the output of each layer of the zero-convolution module is input into the SD intermediate layer and the SD decoder in a top-to-bottom order to guide the output of the SD decoder.

[0082] Step 3: Training of Transformer Encoding-Decoding Module

[0083] The Transfomer encoding-decoding module is trained using the text descriptions in the training set and the corresponding real layout of keywords.

[0084] For text description A, for example: A cake with colorful letters spelling "happy brithday" arranged in two lines and centered on the cake. On the one hand, the Transformer encoding-decoding module is used to process and obtain the keyword text layout. The keyword is the part "happy brithday" enclosed in double quotes. The specific process is:

[0085] First, tokenize the text description, that is:

[0086] Q(A)=(q0,q1,……,q L-1 )

[0087] Here, Q(A) refers to the tokenized representation of the text description and L is its length.

[0088] Operate on Q(A) to get the corresponding embedding representation Emb(Q(A)):

[0089] Emb(Q(A))=CLIP(Q(A))+Position(Q(A))+Key(Q(A))+Width(Q(A))

[0090] Where CLIP(Q(A)) represents the sequence encoding obtained by the CLIP model, Position(Q(A)) represents the learnable position embedding, Key(Q(A)) represents the keyword embedding, and Width(Q(A)) represents the width of the keyword with the embedding layer.

[0091] The text description A is processed into the corresponding keyword text layout F(A) by the Transformer encoding-decoding module:

[0092] D(A)=ΦD(ΦE(Emb(Q(A))))=(f0,f1,...,f K-1 )

[0093] Among them, ΦD represents the Transfomer decoder and ΦE represents the Transfomer encoder.

[0094] Compare the obtained keyword text layout F(A) with the keyword real layout F(A) GT Assume L1 loss:

[0095] loss layout =|F(A) GT -F(A)|

[0096] The above loss is back-propagated, and the parameters in the Transformer encoding-decoding module are iteratively updated. The training of the Transformer encoding-decoding module is completed after the set number of iterations.

[0097] Step 4: Fine-tuning training of the U-Net network of the SD network model

[0098] The SD network model adopts the Stable Diffusion v1.5 pre-trained model, in which the VAE encoder, U-Net network, VAE decoder, text encoder, and image encoder all have pre-trained network parameter values, and the control network is the pre-trained GlyphControl (for details, see: Yang Y, Gui D, YuanY, et al. Glyphcontrol: Glyphconditional control for visual text generation [J]. Advances in Neural Information Processing Systems, 2024, 36.) control network.

[0099] Freeze the network parameters of the VAE encoder, VAE decoder, text encoder, image encoder and control network, and use the training set data to fine-tune the network parameters of the cross-attention module of the U-Net network. The fine-tuning refers to using the pre-trained network parameters as the initial parameter values ​​for the training process. The optimal value has a small change compared to the initial value. This description is not used to limit the specific technical features. An image x and a corresponding text description A in a piece of data in the training set. First, the image x, for example, a picture with "happy birthday" written in two rows in different colors in the middle of a cake, is encoded into a latent space representation X through a VAE encoder, and then the latent space representation X is gradually denoised: the sampling time step T~Uniform(0,T max ), and sample a noise ∈ that satisfies the Gaussian distribution to gradually destroy the original representation, and obtain the noisy representation Z t :

[0100]

[0101] where a t is the diffusion coefficient at time step t, and ∈ is Gaussian noise.

[0102] At the same time, the image x is processed by the OCR detection module to obtain the corresponding target generation area, and then the target generation area is encoded to obtain the encoded target generation area mask image (i.e., binary image) I m .

[0103] For the keywords in the text description A, the corresponding keyword real layout F(A) is used GT , and render it to get the keyword text layout image g m The rendering is implemented using the Pillow library in Python. Specifically, the text layout generation step is to generate boxes to specify the position of each word in the keyword, use Pillow to put the corresponding words into the corresponding boxes, and render the final text layout image.

[0104] Layout the keyword text image m , target generation area mask image I m And characterize Z t Splice and get the combined noise Z' t :

[0105] Z' t =con(Z t ,g m ,I m )

[0106] For keyword text layout image g m, after being processed by the image encoder, it is encoded into the keyword text layout image embedding vector e g :

[0107] e g =ΦE g (g m )

[0108] where ΦE g Represents a picture encoder.

[0109] On the other hand, the text description A is processed by the text encoder to obtain the text embedding vector e t :

[0110] e t =ΦE t (Q(A))

[0111] where ΦE t Represents a text encoder.

[0112] Embed keyword text layout image into vector e g , text embedding vector e t , merge noise Z' t As the input of the control network, the control vector C is obtained:

[0113] C=Cont(Z' t ;e t ;e g )

[0114] Where Cont represents the pre-trained control network.

[0115] The noise Z' will be merged t , text embedding vector e t The control vector C is input into the U-Net network, and the final image latent vector is obtained through step-by-step iterative denoising. The U-Net network is optimized by the mean square error of the noise added during the denoising process and the noise predicted during the denoising process in each iterative step, which is specifically expressed as follows:

[0116]

[0117] ∈ represents the noise added in each iterative step during the noise addition process, ∈ θ (Z' t ,T,Q,C) represents the noise predicted at the corresponding iterative step in the denoising process.

[0118] Layout image g by the above keyword text mThe character-level segmentation mask M is obtained through the mask operation. In each iteration, the corresponding image latent vector is sampled by another pre-trained U-Net network (i.e., frozen parameters) to obtain the character-level segmentation mask M'. From this, the character perception loss can be obtained as:

[0119]

[0120] in represents the cross entropy loss.

[0121] Furthermore, the hyperparameter λ is used to balance the U-Net network loss and the character perception loss, and the final loss is as follows:

[0122] loss=loss U-Net +λ×loss char

[0123] During the training process, back propagation is used to update the parameters in the U-Net network. After two rounds of training on the training set data, the training is stopped to obtain a trained U-Net network model.

[0124] Analyzing the weights of the backbone network U-Net, the weights of the cross-attention layer are closely related to the text input sequence. In other words, when updating the mapping from a given text to an image distribution, since the text feature is the only input to the key-value projection matrix, only the weights in each cross-attention block i are updated. So when fine-tuning the model, freeze all parameters except these two and keep These two key-value projection matrices are learnable, which can not only achieve the effect of controllable generation, but also effectively improve model performance and accelerate convergence.

[0125] Step 5: Image generation

[0126] The network trained in step 4 can generate images with controllable text layout in a fixed area of ​​a given image.

[0127] The given target image I is recorded as the background image template, and the target keyword text layout image G is recorded as the foreground image, which is obtained by processing the target text description P through the Transformer encoding-decoding module trained in step three and then rendering it.

[0128] Generate a mask image I of the target generation area according to the anchor box drawn by the user mask , combined with the mask image I of the target generation area mask The background image template is gradually added with noise (that is, the feature obtained after adding noise is then point-multiplied (1-I mask), indicating that the target generation area is marked without adding noise), and combined with the mask image I of the target generation area mask Gradually add noise to the foreground image to obtain the background noise map and foreground noise map Combining the two can get the combined noise image Z * The stepwise noise adding process is as described in step 4.

[0129] Input the target keyword text layout image G into the image encoder in step 4 to obtain the conditional control latent vector v, and input the target text description P into the text encoder in step 4 to obtain the text conditional vector w.

[0130] The noise image Z * , conditional control latent vector v, and text conditional vector w are used as the input of the control network in step 4 to obtain the control vector. * , the text condition vector w and the control vector are input into the U-Net network, and the U-Net network outputs the generated image latent vector of the first round; the generated image latent vector is used as the input of the control network and the U-Net network in the second round;

[0131] At round t, the corresponding control vector It needs to be obtained based on the latent vector of the generated image in the t-1th round. Specifically, the latent vector of the generated image in the t-1th round The conditional control latent vector v and the text condition vector w are input into the control network, and the control vector at round t can be output. The formula is as follows:

[0132]

[0133] Wherein ΦCON(;·;·) represents the control network.

[0134] Next, the latent vector of the image generated in round t-1 is The text condition vector w is input into the U-Net network trained in step 4, and then combined with the control vector Add them together to get the hidden vector of the image generated in the tth round The formula is as follows:

[0135]

[0136] Wherein UNet(·;·) represents the U-Net network that has completed the training mentioned above.

[0137] Since the generation process of the background image is full of randomness, there may be a large gap between the generated background image and the original background image object, so a background reconstruction mechanism is needed to control the consistency between the generated image and the background image. Specifically, the generated background image is replaced with the reconstructed background image during the iteration process, and the formula is expressed as:

[0138]

[0139] Among them, in the formula, represents the background image after reconstruction, is the generated image latent vector of the tth iteration, I mask is the target generation area mask image, is the noise image of the background area, ⊙ represents the dot product operation, T represents the maximum number of iterations, and λ is the threshold parameter that controls the time step.

[0140] The process of obtaining the latent vector of the generated image is repeated until the number of iterations reaches T, and the final latent vector of the generated image is obtained, and then the final generated image is obtained by VAE decoding. The generated image is an image in which the target keyword is generated in the target generation area of ​​the target image I.

[0141] This method can realize controllable generation of the generated font in terms of position, number of lines, width, etc., and perfectly blend it into the target generation area of ​​the background image.

[0142] In order to verify the fine-tuning effect of the model, the model parameters in this method are fine-tuned in different proportions on the Laion-5B dataset: full parameter fine-tuning (fine-tuning all U-Net layers), attention layer parameter fine-tuning (fine-tuning all U-Net attention layers), cross-attention layer partial parameter fine-tuning (fine-tuning only the cross-attention layers of U-Net), and the cross-attention layer partial parameter fine-tuning (fine-tuning only the cross-attention layers of U-Net). ). This method uses OCR tools to calculate the character recognition accuracy corresponding to the generated image, and adopts the widely used FID to compare the distribution of the generated image with the distribution of the real image. The distribution difference between the generated image and the real image is measured by the two indicators of OCRAccuracy and FID. The specific experimental results are shown in the table below (the upward arrow indicates that the larger the better, and the downward arrow indicates that the smaller the better):

[0143] Table 1 Test results of different methods on the Laion-5B dataset

[0144]

[0145]

[0146] As can be seen from Table 1, SD (see literature: Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 10684-10695.) can hardly generate correct characters. Even if the traditional fine-tuning method is used to fine-tune SD, it is difficult to significantly improve the accuracy, which is only about 15%. However, this method has greatly improved the accuracy compared with the baseline model by designing different fine-tuning strategies for experiments. This shows that this method can capture the precise spatial structure of the characters and produce the correct font layout. In addition, the FID value is also improved to a certain extent compared with the fine-tuned model, which shows that the image generated by this method is closer to the real image. Observing the three fine-tuning strategies, it can be seen that only fine-tuning The method only needs to update a very small number of parameters, but it can achieve the accuracy of the generated image closest to the accuracy of the real image while maintaining a considerable accuracy.

[0147] In addition, to verify the effectiveness of this method, it is tested on the MARIO-10M dataset, which contains 5,000 image-text pairs. This method and existing text-to-image method models (including SD, SDXL (see reference: Podell D, English Z, Lacey K, et al. Sdxl: Improving latent diffusion models for high-resolution image synthesis [J]. arXiv preprint arXiv: 2307.01952, 2023.), ControlNet (see reference: Zhang L, Rao A, Agrawala M. Adding conditional control to text-to-image diffusion models [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023: 3836-3847.), DeepFloyd (see reference: DeepFloyd. Github link: https: / / github.com / deep-floyd / if, 2023)) are used for image generation, and CLIPScore, FID, and OCR indicators are used to evaluate the image generation quality. CLIP-Score is based on the pre-trained CLIP model. It evaluates the matching degree between the generated image and its corresponding text description by scoring the generated image. It can also be used to measure the similarity between two images. The higher the score, the higher the matching degree or similarity. Accuracy and precision are indicators of whether the keywords appear in the generated image, and are important criteria for measuring controllable generation.

[0148] Table 2 Comparison results of image generation quality of different methods

[0149]

[0150] As shown in Table 2, this method is slightly inferior to DeepFloyd in the FID indicator, which indicates that the distribution of images generated by this method is more different from that of real images, and the ability to generate diversity is retained; this method is ahead of other model methods in CLIPScore; this method achieves the best performance in both OCR-related indicators, and it can be found that this method has greater advantages than methods without explicit text-related guidance, highlighting the importance of explicit guidance.

[0151] SD is an image generation method based on a diffusion model, which generates realistic images from noise through a step-by-step inverse diffusion process. SD is known for its efficient computing performance and open ecosystem, supporting text-to-image, image editing, and multimodal tasks. Compared with traditional generation methods, SD has achieved a good balance between generation quality and flexibility. SDXL (Stable DiffusionXL) is an extended version of the Stable Diffusion model. Compared with the basic version, SDXL has been optimized in architecture and training, introducing a larger model scale, richer training data, and a multi-stage extended decoding strategy, making it better in generating details, texture performance, and light and shadow processing. DeepFloyd uses a pre-trained language model to encode text, and trains three different diffusion models to achieve image generation. Specifically, the first diffusion model is used to generate images, and the latter two diffusion models further achieve image super-resolution. In addition, DeepFloyd also introduces an additional attention pooling layer to extract an embedding from the text embedding, which is used to add the time embedding in U-Net to play the role of a global text embedding. ControlNet is used for image generation and editing tasks. It integrates additional conditional information (such as edge maps, depth maps, posture, etc.) into the generation process through a conditional control mechanism, thereby achieving precise guidance of the generated image. This mechanism can not only improve the diversity of generated images, but also significantly enhance the ability to control image details and structures; in terms of editing, ControlNet supports operations such as shape adjustment and texture modification, providing a flexible and high-precision solution for image generation and editing.

[0152] The present invention realizes the generation of text to image by a two-stage controllable generation method. In the first stage, noise is gradually added to the background image and the foreground image respectively according to the mask corresponding to the binary image and the font layout image of the target generation area, and then a merged noise image is obtained and used as the starting noise of the second stage. In the second stage, the image generation process is controlled by conditional control information (including font layout generated based on factors such as position, number of lines, width, etc., text description input by the user, etc.) to achieve the purpose of controllable generation. In addition, in order to ensure the consistency between the generated image and the background image, a background reconstruction mechanism is introduced to replace the generated background with the reconstructed background to achieve the purpose of seamless fusion, so that the generated image is more in line with the requirements.

[0153] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A controllable method for generating text on an image based on a diffusion model, characterized in that: The method comprises the following steps: Step 1: Get training data The open source dataset MARIO-10M is used, in which one piece of data is a picture containing a keyword, a corresponding text description containing the keyword, and the actual layout information of the corresponding keyword on the picture; Step 2: Build an image generation network The image generation network includes a pre-trained SD network model, an OCR detection module, a Transformer encoding-decoding module, and a control network; The SD network model includes a VAE encoder, a U-Net network, a VAE decoder, a text encoder, and a picture encoder; the U-Net network is a diffusion model, which consists of an SD encoder, an SD middle layer, and an SD decoder; The control network includes the first zero-convolution layer, the encoding module, and the zero-convolution module, wherein the encoding module is composed of a structure of a replicated SD encoder and an SD intermediate layer; the number of layers of the zero-convolution module is equal to the sum of the number of layers of the SD intermediate layer and the SD decoder; Step 3: Training of Transformer Encoding-Decoding Module The Transfomer encoding-decoding module is trained using the text descriptions in the training set and the corresponding real layout of keywords; For the text description A, the Transformer encoding-decoding module is used to obtain the keyword text layout. The specific process is as follows: First, tokenize the text description, that is: Q(A)=(q0,q1,……,q L-1 ) Where Q(A) refers to the tokenized representation of the text description and L is its length; Operate on Q(A) to get the corresponding embedding representation Emb(Q(A)): Emb(Q(A))=CLIP(Q(A))+Position(Q(A))+Key(Q(A))+Width(Q(A)) Where CLIP(Q(A)) represents the sequence encoding obtained by the CLIP model, Position(Q(A)) represents the learnable position embedding, Key(Q(A)) represents the keyword embedding, and Width(Q(A)) represents the width of the keyword with the embedding layer; The text description A is processed into the corresponding keyword text layout F(A) by the Transformer encoding-decoding module: F(A)=φD(ΦE(Emb(Q(A))))=(f0,f1,...,f K-1 ) Among them, ΦD represents the Transfomer decoder, ΦE represents the Transfomer encoder; Compare the obtained keyword text layout F(A) with the keyword real layout F(A) GT Assume L1 loss: loss layout =|F(A) GT -F(A)| The above loss is back-propagated, and the parameters in the Transformer encoding-decoding module are iteratively updated. The training of the Transformer encoding-decoding module is completed after the set number of iterations. Step 4: Fine-tuning training of the U-Net network of the SD network model The SD network model adopts the Stable Diffusion v1.5 pre-trained model, in which the VAE encoder, U-Net network, VAE decoder, text encoder, and image encoder all have pre-trained network parameter values, and the control network is the pre-trained GlyphControl control network; Freeze the network parameters of the VAE encoder, VAE decoder, text encoder, image encoder and control network, and use the training set data to fine-tune the network parameters of the cross-attention module of the U-Net network; for a piece of data in the training set, the image x and the corresponding text description A are first encoded into a latent space representation X through the VAE encoder, and then the latent space representation X is gradually denoised: the sampling time step T~Uniform(0,T max ), and sample a noise ∈ that satisfies the Gaussian distribution to gradually destroy the original representation, and obtain the noisy representation Z t : where a t is the diffusion coefficient at time step t, ∈ is Gaussian noise; At the same time, the image x is processed by the OCR detection module to obtain the corresponding target generation area, and then the target generation area is encoded to obtain the encoded target generation area mask image I m ; For the keywords in the text description A, the corresponding keyword real layout F(A) is used GT , and render it to get the keyword text layout image g m ; Layout the keyword text image m , target generation area mask image I m And characterize Z t Splice and get the combined noise Z' t : Z' t =con(Z t ,g m ,I m ) For keyword text layout image g m , after being processed by the image encoder, it is encoded into the keyword text layout image embedding vector e g : e g =ΦE g (g m ) where ΦE g represents a picture encoder; On the other hand, the text description A is processed by the text encoder to obtain the text embedding vector e t : e t =ΦE t (Q(A)) where ΦE t Represents a text encoder; Embed keyword text layout image into vector e g , text embedding vector e t , merge noise Z' t As the input of the control network, the control vector C is obtained: C=Cont(Z' t ;And t ;And g ) Where Cont represents the pre-trained control network; The noise Z' will be merged t , text embedding vector e t The control vector C is input into the U-Net network, and the final image latent vector is obtained through step-by-step iterative denoising. The U-Net network is optimized by the mean square error of the noise added during the denoising process and the noise predicted during the denoising process in each iterative step, which is specifically expressed as follows: ∈ represents the noise added in each iterative step during the noise addition process, ∈ θ (Z' t ,T,Q,C) represents the noise predicted at the corresponding iteration step in the denoising process; Layout image g by the above keyword text m After the mask operation, the character-level segmentation mask M is obtained. In each iteration, the corresponding image latent vector is sampled by another pre-trained U-Net network to obtain the character-level segmentation mask M'. From this, the character perception loss can be obtained as: in represents the cross entropy loss; Using the hyperparameter λ to balance the U-Net network loss and the character perception loss, the final loss is as follows: loss=loss U-Net +λ×loss char During the training process, back propagation is used to update the parameters in the U-Net network. After two rounds of training on the training set data, the training is stopped to obtain a trained U-Net network model. Step 5: Generate image The given target image I is recorded as the background image template, and the target keyword text layout image G is recorded as the foreground image. The foreground image is obtained by processing the target text description P through the Transformer encoding-decoding module trained in step 3 and then rendering; Generate a mask image I of the target generation area according to the anchor box drawn by the user mask , combined with the mask image I of the target generation area mask Gradually add noise to the background image template and combine it with the mask image I of the target generation area mask Gradually add noise to the foreground image to obtain the background noise map and foreground noise map Combining the two can get the combined noise image Z * ; The method of stepwise noise processing is as described in step 4; Input the target keyword text layout image G into the image encoder in step 4 to obtain the conditional control latent vector v, and input the target text description P into the text encoder in step 4 to obtain the text conditional vector w; The noise image Z * , conditional control latent vector v, text condition vector w as the input of the control network in step 4 to obtain the control vector; the noise image Z * , the text condition vector w and the control vector are input into the U-Net network, and the U-Net network outputs the generated image latent vector of the first round; the generated image latent vector is used as the input of the control network and the U-Net network in the second round; At round t, the corresponding control vector It needs to be obtained based on the latent vector of the generated image in the t-1th round; specifically, the latent vector of the generated image in the t-1th round The conditional control latent vector v and the text condition vector w are input into the control network, and the control vector at round t can be output. The formula is as follows: Where ΦCON(;·;·) represents the control network; Next, the latent vector of the image generated in round t-1 is The text condition vector w is input into the U-Net network trained in step 4, and then combined with the control vector Add them together to get the hidden vector of the image generated in the tth round The formula is as follows: Where UNet(·;·) represents the U-Net network that has completed the training above; In the iterative process, the generated background image is replaced by the reconstructed background image, which can be expressed as: Among them, in the formula, represents the background image after reconstruction, is the generated image latent vector of the tth iteration, I mask is the target generation area mask image, is the noise image of the background area, ⊙ represents the dot product operation, T represents the maximum number of iterations, and λ is the threshold parameter that controls the time step; The process of obtaining the latent vector of the generated image is repeated until the number of iterations reaches T, and the final latent vector of the generated image is obtained, and then the final generated image can be obtained by VAE decoding; the generated image is an image in which the target keyword is generated in the target generation area of ​​the target image I.

Images