Diffusion model-based training-free text-image generation method, device and medium

Abstract

The present invention discloses a training-free text-image generation method, device and medium based on a diffusion model, the method comprising using a grammar interpreter to automatically extract concepts from input text; using a pre-trained natural language model BERT to predict neighbor objects of each subject object; constructing positive sample, negative sample and unconditional sample concepts based on neighbor objects; estimating the target magnetic field direction of the subject object in the feature space; applying the target magnetic field direction to the subject object, and using the diffusion model to achieve text-image generation. The present invention adopts a training-free method to make the output of the image generation model based on the diffusion model more in line with the text requirements, and the modulation process of the feature vector is completed outside the diffusion denoising link, so the increased computational cost can be ignored, and a better text and image alignment effect can be achieved, and the image-text matching degree and accuracy of the diffusion model generation result are improved.

Description

Technical Field

[0001] The present invention belongs to the field of diffusion models and image generation, and in particular relates to a diffusion model-based training-free text-image generation method, device and medium. Background Art

[0002] Text-to-image synthesis via diffusion models has attracted great attention from both the research and industry communities. Given natural language cues, large-scale visual-language diffusion models can generate impressively faithful images that meet user expectations. Among them, the Stable Diffusion model and its variants usually utilize pre-trained text encoders based on contrastive language-image pre-training (CLIP) for text conditioning. The CLIP text encoder uses a causal masking mechanism to generate a unidirectional context, which means that a labeled word can only consider the words to its left. However, it turns out that this leads to attribute binding problems, affecting the quality of diffusion-based text-to-image generation.

[0003] The input text composed of multiple concepts has attribute binding problems, which means that the generated subject object is bound to the wrong attribute, or a subject object in the generated result is not generated correctly. Existing work explains this phenomenon as the mixing and entanglement of contexts. Due to the characteristics of causal masking technology, the attribute token of the latter concept is mixed with the subject object on its left, resulting in generation failure.

[0004] The present invention points out that pre-trained large-scale models have prior preferences in semantic feature space because these models are trained on massive data with implicit human biases. In addition, the present invention links this phenomenon to semantic "magnetic field", in which the binding relationship between target attributes and subject objects depends on the attractiveness therein. If the attributes in other concepts or implicit prior attributes have greater attractiveness, then the desired target attribute cannot be bound to the current subject object. Therefore, the present invention proposes a new type of training-free method to improve the attribute binding problem of text-image generation. Summary of the invention

[0005] Purpose of the invention: The present invention provides a training-free text-image generation method based on a diffusion model to solve the attribute binding problem of text-image generation and to improve the image-text matching and accuracy of the diffusion model generation results.

[0006] Technical solution: The present invention comprises the following steps:

[0007] (1) Use a grammar interpreter to automatically extract concepts from input text;

[0008] (2) Use the pre-trained natural language model BERT to predict the neighbor objects of each topic object;

[0009] (3) Construct the concepts of positive samples, negative samples, and unconditional samples based on neighbor objects;

[0010] (4) estimating the target magnetic field direction of the subject in the feature space;

[0011] (5) Apply the target magnetic field direction to the subject object and use the diffusion model to achieve text-image generation.

[0012] Furthermore, the implementation process of step (1) is as follows:

[0013] Use the grammar interpreter in natural language processing technology to extract input text There are N concepts in the , each concept consists of a subject object and its corresponding target attribute, and the extracted concept set is represented as {A1&S1,…,A N &S N}, where A N represents the target attribute of the Nth concept, S N Represents the subject object of the Nth concept, A N &S N Indicates A N and S N There is a dependency relationship; the subject object only has a dependency relationship with the target attribute of the corresponding index.

[0014] Furthermore, the implementation process of step (2) is as follows:

[0015] The i-th concept A i &S i For, where i = 1...N, construct the text "A i &S i and [MASK]” and input it into the BERT model to predict the noun of the mask prompt “[MASK]” and obtain the subject object S i K i Neighbor object B k , where k = 1…K i In addition, a threshold α is introduced to filter the words output by the BERT model to ensure that the predicted neighbor objects are nouns with similar features to the current subject object.

[0016] Furthermore, the implementation process of step (3) is as follows:

[0017] Construct different concepts using different attributes; for the i-th concept A i &S i , current subject object Si With K i Neighbor object B k , where i = 1…N, k = 1…K i , the positive sample concept is defined as a concept with the same target attribute as the subject object, then the positive sample concept set is Indicates that the attribute in the concept is the current subject object S i The corresponding attributes of ; the negative sample concept is defined as a concept with different target attributes from the subject object, then the negative sample concept set is Indicates that the attribute in the concept is not the current subject object S i The corresponding attributes of the unconditional sample concept are defined as concepts without attributes, so the unconditional concept set is Indicates that the object in the concept does not have any attributes.

[0018] Furthermore, the implementation process of step (4) is as follows:

[0019] Use the text encoder of the CLIP model to convert text into text feature space; introduce operators The operation used to extract sub-embeddings in the feature vector of a text sequence is specifically expressed as Where v represents the input text The text sequence feature vector in the text feature space, p represents the natural language pattern of a specific word in the input text, then v (p) Represents the text feature vector corresponding to the word in the text sequence feature vector in the text feature space; the positive sample concept set Negative sample concept set Unconditional concept set The set of feature vectors in the feature space is expressed as The direction based on comparison is:

[0020]

[0021] The directions of the unconditional constraints are:

[0022]

[0023] Furthermore, the process of applying the target magnetic field direction to the subject in step (5) is as follows:

[0024] The required direction is estimated in a weighted manner. For the vectors in the feature space based on the concepts constructed based on the neighbor objects, the direction based on the comparison and the direction of the unconditional constraint are calculated. The magnetic field direction is estimated by weighted addition to control the correction intensity. The required modulation direction is:

[0025]

[0026] Among them, β i is the control applied to the subject object S i The hyperparameter of the target magnetic field direction strength on the , i=1,...N is the input text The concept in Represents the subject object S i The corresponding text feature vector in the text sequence feature vector v, After the modulation is completed, the text feature vector is represented; finally, the input text The text feature vectors corresponding to all the subject objects in are modulated according to the above process to obtain the enhanced text sequence feature vector, which is expressed as v * .

[0027] Furthermore, the text-image generation process using the diffusion model in step (5) is as follows:

[0028] When generating images, a diffusion model ∈ based on text conditions is used. The denoising process follows the denoising diffusion implicit model DDIM. The diffusion model will obey the Gaussian normal distribution of noise z T As the initial noisy image, the denoising process includes T time steps, so the denoising process at time step t is expressed as z t-1 =∈(z t ,t,v), where z t represents the noisy image at the current time step, z t-1 represents the image after removing the noise of one time step, v represents the text sequence feature vector of the input diffusion model; the original text sequence feature vector is replaced by the modulated feature vector v * , perform the denoising process for T time steps:

[0029] z t-1 =∈(z t ,t,v * ),t=T,...,1

[0030] Finally, the denoised image z0 is input into the decoder to obtain the final image.

[0031] A device according to the present invention includes a memory and a processor, wherein:

[0032] A memory for storing computer programs that can be run on the processor;

[0033] The processor is used to execute the steps of the above-mentioned diffusion model-based training-free text image generation method when running the computer program.

[0034] The present invention provides a storage medium having a computer program stored thereon, and when the computer program is executed by at least one processor, the steps of the above-mentioned diffusion model-based training-free text image generation method are implemented.

[0035] Beneficial effects: Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: the present invention adopts a training-free method, and the pre-trained models used are all in a frozen state, and no additional training is required; the modulation process of the feature vector is completed outside the diffusion denoising link, and the diffusion model is used as a black box for image generation, and the increased computational cost can be ignored; the present invention adopts a training-free method, and uses the prior knowledge space of the multimodal model to modulate the feature vector, thereby solving the attribute binding problem of text-image generation, improving the image-text matching and accuracy of the diffusion model generation results, making the output of the image generation model based on the diffusion model more in line with the text requirements, and being able to achieve better text and image alignment effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flowchart of the text-image generation method based on the diffusion model without training;

[0037] Figure 2 A schematic diagram of the present invention for modulating a vector in a text feature space;

[0038] Figure 3 Schematic diagrams of the results of text-based image generation based on different methods; (a) is a schematic diagram of the results of text-based image generation based on the stable diffusion model; (b) is a schematic diagram of the results of text-based image generation based on Structure Diffusion; (c) is a schematic diagram of the results of text-based image generation based on Attend-and-Excite; (d) is a schematic diagram of the results of text-based image generation based on the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0040] like Figure 1 As shown, the present invention proposes a text-image generation method based on a diffusion model without training, comprising the following steps:

[0041] Step 1: Use a grammar interpreter to automatically extract concepts from the input text.

[0042] Assume the input text is Use the grammar interpreter in natural language processing technology to extract text There are N concepts in the text, each of which consists of a subject object (noun) and its corresponding target attribute (adjective). The extracted concept set is represented as {A1&S1,…,A N &S N}, where A N represents the target attribute of the Nth concept, S N Represents the subject object of the Nth concept, A N &S N Indicates A N and S N There is a dependency relationship. The subject object only has a dependency relationship with the target attribute of the corresponding index, for example, attribute A1 and S N There are no dependencies. Enter text For “a brown bear and a green apple”, the set of concepts automatically extracted by the grammar interpreter is {(A1) “brown” & (S1) “bear”, (A2) “green” & (S2) “apple”}.

[0043] Step 2: Predict the neighbor objects for each subject object.

[0044] The pre-trained natural language model BERT (Bidirectional Encoder Representations from Transformers, BERT) is used to automatically predict the neighboring objects of each topic object. i &S i For example, where i = 1...N, construct the text "A i &S i and [MASK]” and input it into the BERT model to predict the noun of the mask prompt “[MASK]”, thereby obtaining the subject object S i K i Neighbor object B k , where k = 1…K i In addition, a threshold α is introduced to filter the words output by the BERT model to ensure that the predicted neighbor objects are nouns with similar features to the current subject object. "A brown bear and a green apple", the i=2nd concept, namely "green" & "apple", constructs the text "green apple and [MASK]" with a mask prompt, inputs the BERT model, uses the threshold α to filter the output, and obtains K2=3 neighbor objects {"peach", "pear", "lemon"}. The above operation is completed for all the subject objects in the concept set.

[0045] Step 3: Construct the concepts of positive samples, negative samples, and unconditional samples based on neighbor objects.

[0046] Take the i-th concept A i &S i For example, the current subject object S i With K i Neighbor object B k , where i = 1…N, k = 1…K i The positive sample concept is defined as a concept that has the same target attribute as the subject object, so the positive sample concept set is expressed as Negative sample concepts are defined as concepts with different target attributes from the subject object, so the set of negative sample concepts is expressed as Unconditional sample concepts are defined as concepts without attributes, so the unconditional concept set is expressed as

[0047] For input text "A brown bear and a green apple", the current concept is "green apple", where the subject object is "apple", the target attribute is "green", and the neighbor objects are {"peach", "pear", "lemon"}. Then there are positive sample sets {"green peach", "green pear", "green lemon"}; negative sample sets {"brown peach", "brown pear", "brown lemon"}; unconditional sample sets {"peach", "pear", "lemon"}.

[0048] Step 4: Estimate the target magnetic field direction of the subject in the feature space.

[0049] Use the text encoder of CLIP (Contrastive Language-Image Pre-training), a large text-image multimodal model based on contrastive learning, to convert all concepts constructed in step 3 into the text feature space. Assume that the input text After encoding with CLIP text encoder Here we introduce the operator Extract the text feature vector from the text sequence feature vector, expressed as Where v represents the input text The text sequence feature vector in the feature space, p n Represents text If a specific word in Represents the text feature vector corresponding to the word in the text feature space. For the positive sample concept set After encoding, Concept set for negative samples After encoding, Non-conditional concept set After encoding, it is

[0050] Then the target magnetic field direction based on the comparison is expressed as:

[0051]

[0052] The unconditional target magnetic field direction is expressed as:

[0053]

[0054] Step 5: Apply the target magnetic field direction to the subject object and use the diffusion model to achieve text-image generation.

[0055] The target magnetic field direction based on comparison and the target magnetic field direction without conditional constraints obtained in step 4 are added together, and a weighted method is used for the addition, and the formula is expressed as follows:

[0056]

[0057] Among them, β i is the control applied to the subject object S i The hyperparameter of the target magnetic field direction strength on , Represents the subject object S i The corresponding text feature vector in the text sequence feature vector v, Indicates the text feature vector after modulation is completed. Finally, the input text The text feature vectors corresponding to all the subject objects in are modulated according to the above process to obtain the enhanced text sequence feature vector, which is expressed as v * .

[0058] When generating images, a diffusion model ∈ based on text conditions is used, and the denoising process follows the denoising diffusion implicit model DDIM (Denoising Diffusion Implicit Model). The diffusion model will obey the Gaussian normal distribution of noise z T As the initial noisy image, the denoising process includes T time steps, so the denoising process at time step t is expressed as z t-1 =∈(z t ,t,v), where z t represents the noisy image at the current time step, z t-1 represents the image after removing the noise of one time step, and v represents the text sequence feature vector of the input diffusion model. After adopting the enhancement method of the present invention, the original text sequence feature vector is replaced by the modulated feature vector v * , perform the denoising process for T time steps:

[0059] z t-1 =∈(z t ,t,v * ),t=T,...,1

[0060] Finally, the denoised image z0 is input into the decoder to obtain the final image.

[0061] The present invention also provides a device, including a memory and a processor, wherein: the memory is used to store a computer program that can be run on the processor; the processor is used to execute the steps of the above-mentioned diffusion model-based training-free text image generation method when running the computer program.

[0062] The present invention also provides a storage medium having a computer program stored thereon, and when the computer program is executed by at least one processor, the steps of the above-mentioned method for generating text images based on a diffusion model without training are implemented.

[0063] The method proposed in the present invention uses Python natural language processing libraries Stanza and NLTK as grammar interpreters, uses pre-trained BLIP models and CLIP text encoders, and latent diffusion model Stable Diffusion as diffusion models for image generation. Assuming that the input text is "a green bear and a brown apple", the grammar interpreter is first used to extract all concepts in the input text, and the output result is {green & bear, brown & apple}, where "green" is the corresponding target attribute of the subject object "bear", and "brown" is the target attribute of the subject object "apple". In this embodiment, a relatively simple input text case is selected, so only two concepts are included.

[0064] Next, it is necessary to obtain the "neighbor" objects of different subject objects. Taking "apple" (denoted as S2) as an example, the designed mask participation text prompt is "a brown apple and [MASK]", where "[MASK]" represents the mask prompt. The text prompt is input into the pre-trained natural language model BLIP, and the model is required to predict the words in the mask area. The model outputs "peach", "pear"... words, and sorts them from high to low according to confidence. In order to ensure that the predicted "neighbor" object and the subject object are in adjacent positions in the prior knowledge space, the present invention introduces a threshold value α to filter the words output by the BLIP model. It was found in the experiment that when the threshold value α is set to 0.01, it has a relatively stable performance. Therefore, when the confidence of the word output by the model is lower than 0.01, the word is not selected as a "neighbor" object.

[0065] Next, we need to construct different concepts based on the predicted “neighbor” objects. Take “peach” (denoted as B1) as an example. Since the corresponding attribute of “apple” is “brown” (denoted as A2), the constructed positive sample concept is “brown peach” (A j&B1,j=2), and the corresponding target attribute of “apple” is “green” (denoted as A1), which is constructed as a negative sample concept, namely “green peach” (A j &B1,j≠2), the final unconditional concept is "peach". Based on all the "neighbor" objects of "apple", the above positive sample concepts, negative sample concepts and unconditional concepts are constructed, that is, the complete concept set of "apple" is obtained. For other subject objects, that is, "bear", the above steps are repeated to construct its concept set based on different "neighbor" objects.

[0066] Since the concepts constructed based on the “neighborhood” objects are still in the natural language space, the next step is to convert them into the text feature space. Mapped to the corresponding vector set, expressed as Similarly, the negative sample concept set Mapped to a corresponding vector set Unconditional concept set Mapped to a corresponding vector set

[0067] Next, calculate the direction of the "magnetic field" of the subject object "apple" in the feature space pointing to the corresponding target attribute "brown". The direction based on the comparison is:

[0068]

[0069] The directions of the unconditional constraints are:

[0070]

[0071] Finally, the original eigenvector and the calculated "magnetic field" direction are weighted summed:

[0072]

[0073] Where β2 represents the modulation intensity parameter for the word "apple". Similarly, when modulating the word "bear", the modulation intensity parameter set is β1. Since the first subject in the context has fewer mixed features, in actual use, an increasing modulation intensity from 0.3 to 0.5 is used for different subject objects. Finally, after modulating both the text feature vectors of "apple" and "bear", the final text sequence feature vector v is obtained. * .

[0074] Figure 2 A schematic diagram of the before and after modulation of the "apple" object in the feature space is given. Figure 2As shown in (a), before modulation, due to the prior preference of prior knowledge, "apple" is more easily attracted by the attribute of "green"; Figure 2 As shown in (b), after modulation, the feature vector of “apple” is given additional directional guidance, so that “apple” is bound to the target attribute “brown”.

[0075] Finally, using the diffusion model ∈ based on the text condition, the denoising process follows DDIM and contains T time steps. The denoising process at time step t is expressed as z t-1 =∈(z t ,t,v), where z t represents the current noisy image, z t-1 represents the denoised image, and v represents the text conditional feature vector. Using the feature vector modulation step of the present invention, the feature vector v is replaced by the modulated feature vector v * , and finally obtain the corresponding output image, which has the subject objects and corresponding attributes that are more in line with the required text conditions; where the step size T of DDIM is set to 50.

[0076] In order to verify the effectiveness of the present invention, comparative tests were conducted on the original Stable Diffusion and three other enhancement algorithms. In these three network models, in addition to the algorithm proposed by the present invention, two more popular algorithms Structure Diffusion (Training-free structured diffusion guidance for compositional text-to-image synthesis) and Attend-and-Excite (Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models) were also used. All methods generate images under the same random number seed setting and the same input text ("a green banana and a brown sheep"). Figure 3 Medium (a) to Figure 3 The generated results shown in (d) are Stable Diffusion, Structure Diffusion, Attend-and-Excite and the generated results of the algorithm proposed by the present invention. The present invention can make the output of the image generation model based on the diffusion model more in line with the text requirements, achieve better text and image alignment effect, and improve the image-text matching and accuracy of the diffusion model generation results with less calculation and time cost.

[0077] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A training-free text-image generation method based on a diffusion model, characterized in that: The following steps are involved: (1) Use a grammar interpreter to automatically extract concepts from input text; (2) Use the pre-trained natural language model BERT to predict the neighbor objects of each topic object; (3) Construct the concepts of positive samples, negative samples, and unconditional samples based on neighbor objects; (4) estimating the target magnetic field direction of the subject in the feature space; (5) Applying a target magnetic field direction to the subject object and using a diffusion model to achieve text-image generation; The implementation process of step (4) is as follows: Use the text encoder of the CLIP model to convert text into text feature space; introduce operators The operation used to extract sub-embeddings in the feature vector of a text sequence is specifically expressed as Where v represents the input text The text sequence feature vector in the text feature space, p represents the natural language pattern of a specific word in the input text, then v (p) Represents the text feature vector corresponding to the word in the text sequence feature vector in the text feature space; the positive sample concept set Negative sample concept set Unconditional concept set The set of feature vectors in the feature space is expressed as The direction based on comparison is: The directions of the unconditional constraints are: The process of applying the target magnetic field direction to the subject in step (5) is as follows: The required direction is estimated in a weighted manner. For the vectors in the feature space based on the concepts constructed based on the neighbor objects, the direction based on the comparison and the direction of the unconditional constraint are calculated. The magnetic field direction is estimated by weighted addition to control the correction intensity. The required modulation direction is: Among them, β i is the control exerted on the subject object S i The hyperparameter of the target magnetic field direction strength on the , i=1,...N is the input text The concept in Represents the subject object S i The corresponding text feature vector in the text sequence feature vector v, After the modulation is completed, the text feature vector is represented; finally, the input text The text feature vectors corresponding to all the subject objects in are modulated according to the above process to obtain the enhanced text sequence feature vector, which is expressed as v * ; The process of implementing text-image generation using the diffusion model in step (5) is as follows: When generating images, a diffusion model ∈ based on text conditions is used. The denoising process follows the denoising diffusion implicit model DDIM. The diffusion model will obey the Gaussian normal distribution of noise z T As the initial noisy image, the denoising process includes T time steps, so the denoising process at time step t is expressed as z t-1 =∈(z t ,t,v), where z t represents the noisy image at the current time step, z t-1 represents the image after removing the noise of one time step, v represents the text sequence feature vector of the input diffusion model; the original text sequence feature vector is replaced by the modulated feature vector v * , perform the denoising process for T time steps: z t-1 =∈(z t ,t,v * ),t=T,...,1 Finally, the denoised image z0 is input into the decoder to obtain the final image.

2. The text-image generation method based on diffusion model without training according to claim 1 is characterized in that: The implementation process of step (1) is as follows: Use the grammar interpreter in natural language processing technology to extract input text There are N concepts in the , each concept consists of a subject object and its corresponding target attribute, and the extracted concept set is represented as {A1&S1,…,A N &S N }, where A N represents the target attribute of the Nth concept, S N Represents the subject object of the Nth concept, A N &S N Indicates A N and S N There is a dependency relationship; the subject object only has a dependency relationship with the target attribute of the corresponding index.

3. The text-image generation method based on diffusion model without training according to claim 1, characterized in that: The implementation process of step (2) is as follows: The i-th concept A i &S i For, where i = 1...N, construct the text "A i &S i and [MASK]" and input it into the BERT model to predict the noun of the mask prompt "[MASK]" and obtain the subject object S i K i Neighbor object B k , where k = 1…K i ; In addition, a threshold α is introduced to filter the words output by the BERT model to ensure that the predicted neighbor objects are nouns with similar features to the current topic object.

4. The text-image generation method based on diffusion model without training according to claim 1, characterized in that: The implementation process of step (3) is as follows: Construct different concepts using different attributes; for the i-th concept A i &S i , current subject object S i With K i Neighbor object B k , where i = 1…N, k = 1…K i , the positive sample concept is defined as a concept with the same target attribute as the subject object, then the positive sample concept set is Indicates that the attribute in the concept is the current subject object S i The corresponding attributes of ; the negative sample concept is defined as a concept with different target attributes from the subject object, then the negative sample concept set is Indicates that the attribute in the concept is not the current subject object S i The corresponding attributes of the unconditional sample concept are defined as concepts without attributes, so the unconditional concept set is Indicates that the object in the concept does not have any attributes.

5. The text-image generation method based on diffusion model without training according to claim 1, characterized in that: The T is set to 50.

6. A device, characterized in that: comprising a memory and a processor, wherein: A memory for storing computer programs that can be run on the processor; A processor is used to execute the steps of the text-image generation method based on diffusion model without training as described in any one of claims 1 to 5 when running the computer program.

7. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by at least one processor, the steps of the diffusion model-based training-free text-image generation method according to any one of claims 1 to 5 are implemented.

Images