A method of text graph based on consistency model

Abstract

The invention discloses a text graph method based on a consistency model, and the steps include: using the original data set to train a Tokenizer and a variational autoencoder; using the Tokenizer and the variational autoencoder to process the original image and text pairs, and converting the image and text pairs into image feature vectors and text feature vector pairs; constructing a consistency model for text conditional generation and training it; constructing an inference model by using a Tokenizer, randomly sampled Gaussian noise, a consistency model, and a decoder of a variational autoencoder, inputting the text to be generated into the Tokenizer, and obtaining the generated image from the decoder of the variational autoencoder. The text graph method uses a Tokenizer to embed text information, uses a variational autoencoder to compress a data set, and realizes one-step sampling. In the training process, only a single conditional sampler needs to be trained, which saves training time.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning, and in particular to a method of using a consistency model to generate a vinculum graph. Background Art

[0002] The denoising diffusion probability model has been a research hotspot in recent years. It has the advantages of high-quality generated images, rich details, and flexible conditions. However, sampling requires about a thousand steps of reasoning, which makes it difficult to implement in the industry. In recent years, many plug-and-play sampling techniques have been proposed, which greatly reduces the number of reasoning times for the denoising diffusion probability model. However, the denoising diffusion probability model still requires at least about ten steps of reasoning to obtain a high-quality generated image.

[0003] As a variant of the denoising diffusion probability model, the consistency model directly learns the trajectory of the ordinary differential equation of the denoising diffusion probability model and achieves results comparable to those of the denoising diffusion probability model with only one inference sampling. Therefore, the consistency model can be applied to the Wensheng graph technology. Summary of the invention

[0004] The purpose of the invention is to provide a text-generated image method based on a consistency model, which can quickly generate high-quality images according to text in one step.

[0005] Technical solution: The method of the text graph based on the consistency model described in the present invention comprises the following steps:

[0006] Step 1: Use the original dataset containing corresponding image and text pairs to train Tokenizer and variational autoencoder. Tokenizer is used to convert discrete text conditions into continuous text feature vectors, and variational autoencoder is used to compress the dimension of image data to accelerate training.

[0007] Step 2: Use the trained Tokenizer and variational autoencoder to process the original image and text pairs, convert the image and text pairs into image feature vectors and text feature vector pairs, and use the image feature vectors and text feature vector pairs as training data sets for training consistency models;

[0008] Step 3: construct a consistency model generated by text conditions, and train the consistency model using the training data set;

[0009] Step 4: construct an inference model by using Tokenizer, randomly sampled Gaussian noise, consistency model and decoder of variational autoencoder. In the inference model, the output result of Tokenizer and randomly sampled Gaussian noise are input into the consistency model together, the output result of the consistency model is input into the decoder of variational autoencoder, the text to be generated is input into Tokenizer, and the generated image is obtained from the decoder of variational autoencoder.

[0010] Furthermore, in step 1, when training the Tokenizer:

[0011] First, establish the first reconstruction task for training Tokenizer;

[0012] Then, an 8-layer Transformer encoder is used to form the encoder E text , using 8 layers of Transformer encoder to form decoder D text ;

[0013] Then set the loss function L for the first reconstruction task reconstruction , the specific formula is:

[0014] L reconstruction =(D text (E text (x text ))-x text ) 2

[0015] In the formula, x text is the vector represented by the text data;

[0016] Finally, the encoder E text Then connect the decoder D text , and use the image and text pairs in the original dataset to input the encoder E text and decoder D text Perform codec training to complete the first reconstruction task and implement Tokenizer training.

[0017] Furthermore, when the image and text are input to the encoder E text and decoder D text When performing codec training:

[0018] First, count the number of occurrences of all texts in the original data set;

[0019] Then use the BPE algorithm to segment each text and generate a unified dictionary, setting the upper limit of the dictionary size to N;

[0020] Finally, the vector in the dictionary is input into the encoder Etext In the output, the Token corresponding to the text is input into the decoder D text middle.

[0021] Furthermore, the encoder E text It consists of a linear layer and 8 Transformer layers. Each Transformer layer has 8 attention heads. The input feature dimension of the linear layer is 1, and the output feature dimension is 768. The feature dimension of the tokens of all Transformer layers is 768, and the output dimension of the fully connected layers in the Transformer layer is 768.

[0022] Furthermore, the decoder D text It consists of 8 Transformer layers and one linear layer. Each Transformer layer has 8 attention heads. The input and output feature dimensions of each Transformer layer are 768. The output dimension of the fully connected layer in the Transformer layer is 768. The output feature dimension of the last linear layer of the decoder is 1.

[0023] Furthermore, in step 1, when training the variational autoencoder:

[0024] First, establish a second reconstruction task for training the variational autoencoder;

[0025] Then construct the encoder E of the variational autoencoder img and decoder D img ;

[0026] Then set the loss function L for the second reconstruction task vae for:

[0027] L vae =∑ω*KL(N(0,I)||N(μ(E img (x img )),σ(E img (x img ))))+∑(D img (E img (x img ))-x img ) 2

[0028] Where KL(·||·) represents the KL divergence, I represents the identity matrix, and x img represents the original image data, σ(·) represents the variance, μ(·) represents the mean, N(μ(E img (x img )),σ(E img (ximg )) indicates that the mean is μ(E img (x img ), and the covariance is σ(E img (x img ) is a Gaussian distribution, ω is a positive coefficient;

[0029] Finally, the image in the original dataset is used to input the encoder E img and decoder D img Encoder-decoder training is performed to complete the second reconstruction task and realize the training of the variational autoencoder.

[0030] Furthermore, the encoder E img It includes 4 downsampling units; each downsampling unit includes a sequentially connected residual block and a downsampling layer; the decoder D img It includes 4 upsampling units; each upsampling unit includes a sequentially connected residual block and an upsampling layer; the residual block includes a sequentially connected normalization layer, a SiLU activation function, a first convolution layer, a normalization layer, a SiLU activation function, a second convolution layer and a residual connection; the downsampling layer includes a sequentially connected normalization layer, a SiLU activation function, a third convolution layer, a normalization layer, a SiLU activation function and a fourth convolution layer; the upsampling layer includes a sequentially connected nearest neighbor interpolation operation, a normalization layer, a SiLU activation function, a fifth convolution layer, a normalization layer, a SiLU activation function and a sixth convolution layer.

[0031] Furthermore, in step 2, the specific steps of converting the image and text pair into an image feature vector and a text feature vector pair are:

[0032] First, we obtain each image and text pair in the original data set in turn. The image and text pair are represented as (x img 、x text );

[0033] Then the text data x of each image and text pair text Encoder E after Tokenizer text Get the image feature vector E text (x text ), the image data x of each image and text pair img Encoder E after variational autoencoder img Get the text feature vector as E img (x img );

[0034] Finally, the corresponding image feature vector and text feature vector form an image feature vector and text feature vector pair as {E img (x img ), E text (xtext )}.

[0035] Furthermore, in step 3, the specific steps of using the training data set to train the consistency model generated by text conditions are:

[0036] First, set the noise intensity increasing sequence sigma and the total number of steps n_timesteps, and then set the maximum noise intensity to 80;

[0037] Then set the consistency model F and the loss function L of the consistency model F. The loss function L is specifically:

[0038] L condition =E t，x，∈ (d(F infer (E img (x img )+sigma[t]*∈,t,E text (x text ))-F(E img (x img )+sigma[t+1]*∈,t+1,E text (x text ))))

[0039] L=L condition

[0040]

[0041] t~U(0,n_timesteps-1),∈~N(0,I)

[0042] x=(E img (x img ), E text (x text ))~P(x)

[0043] Where ∈ is randomly sampled Gaussian noise, n_timesteps is the total number of steps, P(x) is the distribution of the results after encoding text and image pairs, and F infer Indicates that no gradient is generated and F infer =F, and F satisfies the following formula:

[0044] F(E img (x img )+sigma[0]*∈,0,E text (x text ))=E img (x img ).

[0045] Furthermore, in step 4, the formula of the inference model is:

[0046] x output =D img (F(∈*80,n_timesteps,E text (x text )))

[0047] In the formula, x output is the generated image, ∈ is Gaussian noise with mean 0 and variance I, 80 is the maximum noise intensity, D img is the decoder, n_timesteps is the total number of steps of increasing noise intensity, E text (x text ) is the output result obtained by inputting the text condition into the Tokenizer.

[0048] Compared with the prior art, the present invention has the following advantages: (1) Tokenizer is used to embed text information, and variational autoencoder is used to compress image data to achieve one-step sampling, and the sampling result is theoretically the same as the result of the first-order ODE trajectory of the diffusion model; (2) during the model training process, there is no need to train a conditional sampler and an unconditional sampler at the same time like the denoising diffusion probability model, and only a single conditional sampler needs to be trained, which saves training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the method of the present invention;

[0050] Figure 2 It is a schematic diagram of the reasoning model structure of the present invention;

[0051] Figure 3 Schematic diagram of the structure of the variational autoencoder of the present invention;

[0052] Figure 4 Schematic diagram of the consistency model network structure of the present invention.

[0053] Figure 5 It is a structural diagram of the Tokenizer of the present invention DETAILED DESCRIPTION

[0054] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0055] like Figure 1 As shown, the Wensheng graph method based on the consistency model disclosed in the present invention includes the following steps:

[0056] Step 1: Use the original dataset containing the corresponding image and text pairs to train the Tokenizer and variational autoencoder. The Tokenizer is used to convert discrete text conditions into continuous text feature vectors. The variational autoencoder is used to compress the dimension of image data to accelerate training. That is, compressing an image into a feature vector with a smaller dimension makes the network calculate faster, and the original image can be reconstructed through this feature vector, so it has a compression effect.

[0057] Step 2: Use the trained Tokenizer and variational autoencoder to process the original image and text pairs, convert the image and text pairs into image feature vectors and text feature vector pairs, and use the image feature vectors and text feature vector pairs as training data sets for training consistency models;

[0058] Step 3: construct a consistency model generated by text conditions, and train the consistency model using the training data set;

[0059] Step 4: Construct an inference model by using Tokenizer, random sampled Gaussian noise, consistency model and the decoder of variational autoencoder. In the inference model, the output of Tokenizer and random sampled Gaussian noise are input into the consistency model together, and the output of the consistency model is input into the decoder of variational autoencoder, as shown in Figure 2 As shown, the text to be generated is input into the Tokenizer, and the generated image is obtained from the decoder of the variational self-encoder.

[0060] Furthermore, in step 1, when training the Tokenizer:

[0061] First, establish the first reconstruction task for training Tokenizer;

[0062] Then, an 8-layer Transformer encoder is used to form the encoder E text , using 8 layers of Transformer encoder to form decoder D text ,like Figure 5 As shown;

[0063] Then set the loss function L for the first reconstruction task reconstruction , the specific formula is:

[0064] L reconstruction =(D text (E text (x text ))-x text ) 2

[0065] In the formula, x textis the vector represented by the text data;

[0066] Finally, the encoder E text Then connect the decoder D text , and use the image and text pairs in the original dataset to input the encoder E text and decoder D text Perform codec training to complete the first reconstruction task and implement Tokenizer training.

[0067] Furthermore, when the image and text are input to the encoder E text and decoder D text When performing codec training:

[0068] First, count the number of occurrences of all texts in the original data set;

[0069] Then use the BPE algorithm to segment each text and generate a unified dictionary. Preferably, N is set to 4096;

[0070] Finally, the vector in the dictionary is input into the encoder E text In the output, the Token corresponding to the text is input into the decoder D text middle.

[0071] Furthermore, the encoder E text It consists of a linear layer and 8 Transformer layers. Each Transformer layer has 8 attention heads. The input feature dimension of the linear layer is 1, and the output feature dimension is 768. The feature dimension of the tokens of all Transformer layers is 768, and the output dimension of the fully connected layers in the Transformer layer is 768.

[0072] Furthermore, the decoder D text It consists of 8 Transformer layers and one linear layer. Each Transformer layer has 8 attention heads. The input and output feature dimensions of each Transformer layer are 768. The output dimension of the fully connected layer in the Transformer layer is 768. The output feature dimension of the last linear layer of the decoder is 1.

[0073] Furthermore, in step 1, when training the variational autoencoder:

[0074] First, establish a second reconstruction task for training the variational autoencoder;

[0075] Then construct the encoder E of the variational autoencoder img and decoder D img ;

[0076] Then set the loss function L for the second reconstruction task vae for:

[0077] L vae =∑ω*KL(N(0,I)||N(μ(E img (x img )),σ(E img (x img ))))+∑(D img (E img (x img ))-x img ) 2

[0078] Where KL(·||·) represents the KL divergence, I represents the identity matrix, and x img represents the original image data, σ(·) represents the variance, μ(·) represents the mean, N(μ(E img (x img )),σ(E img (x img )) indicates that the mean is μ(E img (x img ), and the covariance is σ(E img (x img ) is a Gaussian distribution, ω is a positive coefficient;

[0079] Finally, the image and text in the original dataset are used to input the encoder E img and decoder D img Encoder-decoder training is performed to complete the second reconstruction task and realize the training of the variational autoencoder.

[0080] Further, such as Figure 3 As shown, encoder E img It includes 4 downsampling units, each of which includes a residual block and a downsampling layer connected in sequence; the decoder D img It includes 4 upsampling units, each of which includes a sequentially connected residual block and an upsampling layer; the residual block includes a sequentially connected normalization layer, a SiLU activation function, a first convolution layer, a normalization layer, a SiLU activation function, a second convolution layer and a residual connection; the downsampling layer includes a sequentially connected normalization layer, a SiLU activation function, a third convolution layer, a normalization layer, a SiLU activation function and a fourth convolution layer; the upsampling layer includes a sequentially connected nearest neighbor interpolation operation, a normalization layer, a SiLU activation function, a fifth convolution layer, a normalization layer, a SiLU activation function and a sixth convolution layer.

[0081] Furthermore, the convolution kernel of the first convolution layer is 3×3, the stride is 1, and the padding is 1; the convolution kernel of the second convolution layer is 3×3, the stride is 1, and the padding is 1; the convolution kernel of the third convolution layer is 3×3, the stride is 2, and the padding is 1; the convolution kernel of the fourth convolution layer is 3×3, the stride is 1, and the padding is 1; the convolution kernel of the fifth convolution layer is 3×3, the stride is 1, and the padding is 1; the convolution kernel of the sixth convolution layer is 3×3, the stride is 1, and the padding is 1.

[0082] Furthermore, in step 2, the specific steps of converting the image and text pair into an image feature vector and a text feature vector pair are:

[0083] First, we obtain each image and text pair in the original data set in turn. The image and text pair are represented as (x img 、x text );

[0084] Then the text data x of each image and text pair text Encoder E after Tokenizer text Get the image feature vector E text (x text ), the image data x of each image and text pair img Encoder E after variational autoencoder img Get the text feature vector as E img (x img );

[0085] Finally, the corresponding image feature vector and text feature vector form an image feature vector and text feature vector pair as {E img (x img ), E text (x text )}.

[0086] Furthermore, in step 3, when building the consistency model for text conditional generation:

[0087] First, set the sequence sigma of increasing noise intensity and the total number of steps n_timesteps, and then set the maximum noise intensity to 80. Preferably, n_timesteps is set to 1281, and sigma is set as follows:

[0088]

[0089] Then set the consistency model F and the loss function L of the consistency model F. The loss function L is specifically:

[0090] L condition =E t，x，∈ (d(F infer (E img(x img )+sigma[t]*∈,t,E text (x text ))-F(E img (x img )+sigma[t+1]*∈,t+1,E text (x text ))))

[0091] L=L condition

[0092]

[0093] t~U(0,n_timesteps-1),∈~N(0,I)

[0094] x=(E iimg (x img ), E text (x text ))~P(x)

[0095] Where ∈ is randomly sampled Gaussian noise, n_timesteps is the total number of steps, P(x) is the distribution of the results after encoding text and image pairs, and F infer Indicates that no gradient is generated and F infer =F, and F satisfies the following formula:

[0096] F(E img (x img )+sigma[0]*∈,0,E text (x text ))=E img (x img ).

[0097] Furthermore, the network structure of the consistency model F is as follows Figure 4 As shown, E img (x img )+sigma[t]*∈ as input, and E text (x text ) as the text conditional feature, and t as the time conditional feature; first, the input is converted into a 2×2 patch and then converted into tokens through a fully connected layer with an input feature number of 4 and an output feature number of 768; then E text (x text) is passed through a fully connected layer with an input and output dimension of 768, and then an average pooling operation is performed, so that for different texts, the pooled result becomes a token; for the time condition feature, the same Position Embedding of Transformer is used to convert it into a 768-dimensional token; then the tokens of the two conditions are concatenated to obtain another 2 tokens, and then a fully connected layer with an input dimension of 2 and an output dimension of 1 is passed to convert the other 2 tokens into 1; finally, the converted 1 token and E img (x img )+sigma[t]*∈ are concatenated and input into 12 Transformer layers, then pass through Layer Norm and a fully connected layer with an input dimension of 768 and an output dimension of 4, and then take all tokens except the last one, and then de-patch to get the output of the consistency model F.

[0098] Furthermore, in step 4, the formula of the inference model is:

[0099] x output =D img (F(∈*80,n_timesteps,E text (x text )))

[0100] In the formula, x output is the generated image, ∈ is Gaussian noise with mean 0 and variance I, 80 is the maximum noise intensity, D img is the decoder, n_timesteps is the total number of steps of increasing noise intensity, E text (x text ) is the output result obtained by inputting the text condition into the Tokenizer.

[0101] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method of using a consistency model for a text graph, characterized in that: The steps include: Step 1: Use the original dataset containing the corresponding image and text pairs to train the Tokenizer and the variational autoencoder. The Tokenizer is used to convert discrete text conditions into continuous text feature vectors, and the variational autoencoder is used to compress the dimension of the image data to obtain the image feature vector. Step 2: Use the trained Tokenizer and variational autoencoder to process the image and text pairs in the original data set, convert the image and text pairs into image feature vectors and text feature vector pairs, and use the image feature vectors and text feature vector pairs as training data sets for training consistency models; Step 3: construct a consistency model generated by text conditions, and train the consistency model using the training data set; Step 4: construct an inference model by using Tokenizer, random sampled Gaussian noise, consistency model and the decoder of variational autoencoder. In the inference model, the output of Tokenizer and the random sampled Gaussian noise are input into the consistency model together, the output of the consistency model is input into the decoder of variational autoencoder, the text to be generated is input into Tokenizer, and the generated image is obtained from the decoder of variational autoencoder. In step 1, when training the variational autoencoder: First, establish a second reconstruction task for training the variational autoencoder; Then construct the encoder E of the variational autoencoder img and decoder D img ; Then set the loss function L for the second reconstruction task vae for: L vae =∑ω*KL(N(0,I)||N(μ(E img (x img )),σ(E img (x img ))))+∑(D img (E img (x img ))-x img ) 2 Where KL(·||·) represents the KL divergence, I represents the identity matrix, and x img represents the original image data, σ(·) represents the variance, μ(·) represents the mean, N(μ(E img (x img )),σ(E img (x img )) indicates that the mean is μ(E img (x img ), and the covariance is σ(E img (x img ) is a Gaussian distribution, ω is a positive coefficient; Finally, the images in the original dataset are used to input the encoder E img and decoder D img Perform codec training to complete the second reconstruction task and train the variational autoencoder; In step 4, the formula of the inference model is: x output =D img (F(∈*80,n_timesteps,E text (x text ))) In the formula, x output is the generated image, ∈ is Gaussian noise with mean 0 and variance I, 80 is the maximum noise intensity, D img is the decoder, n_timesteps is the total number of steps of increasing noise intensity, E text (x text ) is the output result obtained by inputting the text condition into the Tokenizer.

2. The method of using a consistency model based on a text graph according to claim 1, characterized in that: In step 1, when training the Tokenizer: First, establish a first reconstruction task for training Tokenizer; Then, the encoder E is composed of 8 layers of Transformer layers. text , using 8 layers of Transformer layers to form a decoder D text ; Then set the loss function L for the first reconstruction task reconstruction , the specific formula is: L reconstruction =(D text (E texy (x text ))-x text ) 2 In the formula, x text is the vector represented by the text data; Finally, the encoder E text Then connect the decoder D text , and use the image and text pairs in the original dataset to input the encoder E text and decoder D text Perform codec training to complete the first reconstruction task and implement Tokenizer training.

3. The method of using a consistency model based on a text graph according to claim 2, characterized in that: Input the encoder E with image and text pairs text and decoder D text When performing codec training: First, count the number of occurrences of all texts in the original data set; Then use the BPE algorithm to segment each text and generate a unified dictionary, setting the upper limit of the number of dictionary words to N; Finally, the vector in the dictionary is input into the encoder E text In the output, the Token corresponding to the text is input into the decoder D text middle.

4. The method of using a consistency model based on a text graph according to claim 2, characterized in that: Encoder E text It consists of a linear layer and 8 Transformer layers. Each Transformer layer has 8 attention heads. The input feature dimension of the linear layer is 1, and the output feature dimension is 768. The feature dimension of the tokens of all Transformer layers is 768, and the output dimension of the fully connected layers in the Transformer layer is 768.

5. The method of using a consistency model based on a text graph according to claim 2, characterized in that: Decoder D text It consists of 8 Transformer layers and one linear layer. Each Transformer layer has 8 attention heads. The input and output feature dimensions of each Transformer layer are 768. The output dimension of the fully connected layer in the Transformer layer is 768. The output feature dimension of the last linear layer of the decoder is 1.

6. The method of using a consistency model based on a text graph according to claim 1, characterized in that: Encoder E img It includes 4 downsampling units; each downsampling unit includes a sequentially connected residual block and a downsampling layer; the decoder D img It includes 4 upsampling units; each upsampling unit includes a sequentially connected residual block and an upsampling layer; the residual block includes a sequentially connected normalization layer, a SiLU activation function, a first convolution layer, a normalization layer, a SiLU activation function, a second convolution layer, and a residual connection; the downsampling layer includes a sequentially connected normalization layer, a SiLU activation function, a third convolution layer, a normalization layer, a SiLU activation function, and a fourth convolution layer; The upsampling layer includes a nearest neighbor interpolation operation, a normalization layer, a SiLU activation function, a fifth convolutional layer, a normalization layer, a SiLU activation function, and a sixth convolutional layer, which are connected in sequence.

7. The method of using a consistency model based on a text graph according to claim 1, characterized in that: In step 2, the specific steps of converting the image and text pair into an image feature vector and a text feature vector pair are: First, we obtain each image and text pair in the original data set in turn. The image and text pair are represented as (x img 、x text ); Then the image data x of each image and text pair img Encoder E after Tokenizer text Get the image feature vector E img (x img ), the text data x of each image and text pair text Encoder E after variational autoencoder img Get the text feature vector as E text (x text ); Finally, the corresponding image feature vector and text feature vector form an image feature vector and text feature vector pair as {E img (x img )、E text (x text )}.

8. The method of using a consistency model based on a text graph according to claim 1, characterized in that: In step 3, when building the consistency model for text conditional generation: First, set the sequence sigma of increasing noise intensity and the total number of steps n_timesteps, then set the maximum noise intensity to 80, n_timesteps to 1281, and sigma as follows: Then set the consistency model F and the loss function L of the consistency model F. The loss function L is specifically: L condition =E t,x,∈ (d(F infer (E img (x img )+sigma[t]*∈,t,E text (x text ))-F(E img (x img )+sigma[t+1]*∈,t+1,E text (x text )))) L=L condition t~U(0,n_timesteps-1),∈~N(0,I) x=(E img (x img ),E text (x text ))~P(x) Where ∈ is randomly sampled Gaussian noise, n_timesteps is the total number of steps, P(x) is the distribution of the results after encoding text and image pairs, and F infer Indicates that no gradient is generated and F infer =F, and F satisfies the following formula: F(E img (x img )+sigma[0]*∈,0,E text (x text ))=E img (x omg )。

Images