Figure graph model training method and device and text graph method and device

Abstract

The invention discloses a text graph model training method, a text graph method and a text graph device. The text graph model training method comprises the steps of obtaining a potential image and a corresponding prompt word, performing noise reduction processing on the potential image to obtain a first noise reduction feature graph, and performing image restoration processing on the first noise reduction feature graph to obtain a first to-be-scored image; performing quality scoring on the first to-be-scored image to obtain a first quality score, determining a first similarity according to the cue word and the first to-be-scored image, and training the text graph model according to the first quality score and the first similarity. In the process of training the text graph model, the first quality score obtained by carrying out quality scoring on the first to-be-scored image is introduced, and the text graph model can be guided to generate image content with a good quality level; by using the text graph model trained by the text graph model training method, images which can better meet aesthetic requirements of human readers can be generated. The method is widely applied to the technical field of text graphics.

Description

Technical Field

[0001] The present invention relates to the technical field of grammar, and in particular to a grammar model training method, a grammar method and a grammar device. Background Art

[0002] Text-generated image technology is a technology that generates images based on text. The content of the generated image is the same as the content described in the text. Text-generated image technology can be applied to fields such as manuscript writing and art design, and can improve the efficiency of generating materials, especially image materials.

[0003] Although current text-based image technology can quickly and efficiently generate images based on text, the visual effects of the generated images are often inconsistent with the readers' aesthetic style and / or the image quality is low. For example, when a text describing contemporary life is provided, current text-based image technology often generates images describing the future society and other content that does not exist in reality, making it difficult to meet the needs of readers. Summary of the invention

[0004] In view of the technical problems that the pictures generated by the current culture graph technology are of low quality and difficult to meet the needs of readers, the purpose of the present invention is to provide a culture graph model training method, a culture graph method and a device.

[0005] In one aspect, an embodiment of the present invention includes a method for training a text graph model, the method comprising the following steps:

[0006] Obtaining potential images and corresponding prompt words;

[0007] Performing noise reduction processing on the latent image to obtain a first noise reduction feature map;

[0008] Performing image restoration processing on the first denoising feature map to obtain a first image to be scored;

[0009] Performing a quality score on the first image to be scored to obtain a first quality score;

[0010] Determining a first similarity according to the prompt word and the first image to be rated;

[0011] The Vincent graph model is trained according to the first quality score and the first similarity to obtain a trained Vincent graph model.

[0012] Furthermore, the obtaining of the potential image and the corresponding prompt word includes:

[0013] Generate a random vector in the latent space;

[0014] Using the random vector as the latent image;

[0015] The prompt word is generated according to the random vector.

[0016] Furthermore, performing image restoration processing on the first denoising feature map to obtain a first image to be scored includes:

[0017] Establish a variable differential encoder;

[0018] The first denoising feature map is input into the variational differential encoder for processing to obtain the first image to be scored.

[0019] Furthermore, the establishment of a variable differential encoder includes:

[0020] Establish multiple convolutional layers and multiple upsampling layers;

[0021] Fully connecting each of the convolutional layers and each of the upsampling layers alternately to obtain the variational differential encoder;

[0022] The step of inputting the first noise reduction feature map into the variational differential encoder for processing to obtain the first image to be scored comprises:

[0023] Performing convolution processing on the first denoising feature map through each of the convolution layers to obtain mean data and variance data corresponding to the first denoising feature map;

[0024] Constructing latent variables according to the mean data and the variance data;

[0025] The latent variable is upsampled through each of the upsampling layers to obtain the first image to be scored.

[0026] Furthermore, the performing denoising on the latent image to obtain a first denoising feature map includes:

[0027] Performing noise prediction processing on the latent image to obtain first predicted noise information;

[0028] The first predicted noise information is subtracted from the potential image to obtain the first denoising feature map.

[0029] Furthermore, the step of performing a quality score on the first image to be scored to obtain a first quality score includes:

[0030] Building deep learning models;

[0031] Training the deep learning model as a quality scoring model;

[0032] The first image to be rated is input into the quality rating model for processing to obtain the first quality score.

[0033] Furthermore, determining a first similarity based on the prompt word and the first image to be rated includes:

[0034] Establishing a similarity calculation module; the similarity calculation module includes a text encoder and an image encoder;

[0035] Inputting the prompt word into the text encoder for processing to obtain a first text embedding vector;

[0036] Inputting the first image to be rated into the image encoder for processing to obtain a first image embedding vector;

[0037] A similarity calculation is performed based on the first text embedding vector and the first image embedding vector to obtain the first similarity.

[0038] Further, performing a weighted operation on the first quality score and the first similarity to determine a first weighted value includes:

[0039] Set the preset parameter p;

[0040] The first quality score and the first similarity are weighted and summed according to the preset parameters to determine the first weighted value; wherein the calculation formula of the first weighted value is:

[0041]

[0042] Among them, Z is the first weighted value, A is the first quality score, and B is the first similarity.

[0043] Furthermore, the training of the Vincent graph model according to the first quality score and the first similarity to obtain the trained Vincent graph model includes:

[0044] Perform at least one round of training process; each round of the training process obtains a corresponding first denoising feature map, a first quality score and a first similarity;

[0045] For any round of the training process, performing a weighted operation on the first quality score and the first similarity corresponding to the training process to determine a first weighted value corresponding to the training process;

[0046] Determining a cumulative loss function value according to the first denoising feature map and the first weighted value of each round of the training process;

[0047] Parameters of the Wensheng graph model are updated according to the cumulative loss function value to obtain a trained Wensheng graph model.

[0048] Further, determining the cumulative loss function value according to the first denoising feature map and the first weighted value of each round of the training process includes:

[0049] For any round of the training process, determining a weighted probability according to the corresponding first weighted value and the first denoising feature map obtained in the previous round of the training process;

[0050] The cumulative loss function value is obtained according to the weighted probabilities corresponding to all the training processes.

[0051] Further, for any round of the training process, determining the weighted probability according to the corresponding first weighted value and the first noise reduction feature map obtained in the previous round of the training process includes:

[0052] Normalizing the first denoising feature map obtained in the previous round of the training process according to the probability distribution of the Wensheng graph model, and taking a logarithmic value;

[0053] The product of the logarithmic value and the first weighted value corresponding to the current round of the training process is calculated to obtain the weighted probability corresponding to the current round of the training process.

[0054] Furthermore, obtaining the cumulative loss function value according to the weighted probabilities corresponding to all the training processes includes:

[0055] The mathematical expectation of the weighted probabilities corresponding to all the training processes is calculated to obtain the cumulative loss function value.

[0056] Furthermore, the updating of parameters of the Wensheng graph model according to the cumulative loss function value to obtain a trained Wensheng graph model includes:

[0057] A gradient back propagation algorithm is executed according to the accumulated loss function value, and the parameters of the Wensheng graph model are back propagated and updated to obtain a trained Wensheng graph model.

[0058] In another aspect, an embodiment of the present invention includes a method for a Vincent diagram, the method comprising:

[0059] Acquire a Wensheng graph model; the Wensheng graph model is trained by the Wensheng graph model training method in the embodiment;

[0060] Get the text to be processed;

[0061] Inputting the text to be processed into the text graph model for processing to obtain a generated feature graph;

[0062] Perform image restoration processing on the generated feature map to obtain a generated image.

[0063] On the other hand, an embodiment of the present invention provides a device for training a text graph model, the device comprising:

[0064] A training data acquisition module, used to acquire potential images and corresponding prompt words;

[0065] A denoising module, configured to perform denoising on the latent image to obtain a first denoising feature map;

[0066] A restoration module, used for performing image restoration processing on the first denoising feature map to obtain a first image to be scored;

[0067] A scoring module, used for performing a quality score on the first image to be scored to obtain a first quality score;

[0068] A similarity module, used for determining a first similarity according to the prompt word and the first image to be rated;

[0069] A training module is used to train the document graph model according to the first quality score and the first similarity to obtain a trained document graph model.

[0070] Furthermore, the obtaining of the potential image and the corresponding prompt word includes:

[0071] Generate a random vector in the latent space;

[0072] Using the random vector as the latent image;

[0073] The prompt word is generated according to the random vector.

[0074] Furthermore, performing image restoration processing on the first denoising feature map to obtain a first image to be scored includes:

[0075] Establish a variable differential encoder;

[0076] The first denoising feature map is input into the variational differential encoder for processing to obtain the first image to be scored.

[0077] Furthermore, the establishment of a variable differential encoder includes:

[0078] Establish multiple convolutional layers and multiple upsampling layers;

[0079] Fully connecting each of the convolutional layers and each of the upsampling layers alternately to obtain the variational differential encoder;

[0080] The step of inputting the first noise reduction feature map into the variational differential encoder for processing to obtain the first image to be scored comprises:

[0081] Performing convolution processing on the first denoising feature map through each of the convolution layers to obtain mean data and variance data corresponding to the first denoising feature map;

[0082] Constructing latent variables according to the mean data and the variance data;

[0083] The latent variable is upsampled through each of the upsampling layers to obtain the first image to be scored.

[0084] Furthermore, the performing denoising on the latent image to obtain a first denoising feature map includes:

[0085] Performing noise prediction processing on the latent image to obtain first predicted noise information;

[0086] The first predicted noise information is subtracted from the potential image to obtain the first denoising feature map.

[0087] Furthermore, the step of performing a quality score on the first image to be scored to obtain a first quality score includes:

[0088] Building deep learning models;

[0089] Training the deep learning model as a quality scoring model;

[0090] The first image to be rated is input into the quality rating model for processing to obtain the first quality score.

[0091] Furthermore, determining a first similarity based on the prompt word and the first image to be rated includes:

[0092] Establishing a similarity calculation module; the similarity calculation module includes a text encoder and an image encoder;

[0093] Inputting the prompt word into the text encoder for processing to obtain a first text embedding vector;

[0094] Inputting the first image to be rated into the image encoder for processing to obtain a first image embedding vector;

[0095] A similarity calculation is performed based on the first text embedding vector and the first image embedding vector to obtain the first similarity.

[0096] Further, performing a weighted operation on the first quality score and the first similarity to determine a first weighted value includes:

[0097] Set the preset parameter p;

[0098] The first quality score and the first similarity are weighted and summed according to the preset parameters to determine the first weighted value; wherein the calculation formula of the first weighted value is:

[0099]

[0100] Among them, Z is the first weighted value, A is the first quality score, and B is the first similarity.

[0101] Furthermore, the training of the Vincent graph model according to the first quality score and the first similarity to obtain the trained Vincent graph model includes:

[0102] Perform at least one round of training process; each round of the training process obtains a corresponding first denoising feature map, a first quality score and a first similarity;

[0103] For any round of the training process, performing a weighted operation on the first quality score and the first similarity corresponding to the training process to determine a first weighted value corresponding to the training process;

[0104] Determining a cumulative loss function value according to the first denoising feature map and the first weighted value of each round of the training process;

[0105] Parameters of the Wensheng graph model are updated according to the cumulative loss function value to obtain a trained Wensheng graph model.

[0106] Further, determining the cumulative loss function value according to the first denoising feature map and the first weighted value of each round of the training process includes:

[0107] For any round of the training process, determining a weighted probability according to the corresponding first weighted value and the first denoising feature map obtained in the previous round of the training process;

[0108] The cumulative loss function value is obtained according to the weighted probabilities corresponding to all the training processes.

[0109] Further, for any round of the training process, determining the weighted probability according to the corresponding first weighted value and the first noise reduction feature map obtained in the previous round of the training process includes:

[0110] Normalizing the first denoising feature map obtained in the previous round of the training process according to the probability distribution of the Wensheng graph model, and taking a logarithmic value;

[0111] The product of the logarithmic value and the first weighted value corresponding to the current round of the training process is calculated to obtain the weighted probability corresponding to the current round of the training process.

[0112] Furthermore, obtaining the cumulative loss function value according to the weighted probabilities corresponding to all the training processes includes:

[0113] The mathematical expectation of the weighted probabilities corresponding to all the training processes is calculated to obtain the cumulative loss function value.

[0114] Furthermore, the updating of parameters of the Wensheng graph model according to the cumulative loss function value to obtain a trained Wensheng graph model includes:

[0115] A gradient back propagation algorithm is executed according to the accumulated loss function value, and the parameters of the Wensheng graph model are back propagated and updated to obtain a trained Wensheng graph model.

[0116] In another aspect, an embodiment of the present invention provides a Wensheng diagram device, the Wensheng diagram device comprising:

[0117] A Wensheng graph module, used to obtain a Wensheng graph model; the Wensheng graph model is trained by a Wensheng graph model training method;

[0118] A module for obtaining text to be processed, used for obtaining text to be processed;

[0119] A processing module, used for inputting the text to be processed into the text graph model for processing to obtain a generated feature graph;

[0120] The restoration module is used to perform image restoration processing on the generated feature map to obtain a generated image.

[0121] The beneficial effects of the present invention are as follows: the Vincent graph model training method in the embodiment can guide the Vincent graph model to generate image content with a good quality level by introducing a first quality score obtained by quality scoring the first image to be scored in the process of training the Vincent graph model; the Vincent graph method in the embodiment can generate images that better meet the needs of human readers using the Vincent graph model trained by the Vincent graph model training method. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] Figure 1 Schematic diagram of the steps of implementing the Chinese raw graph model training method;

[0123] Figure 2 It is a schematic diagram of the Chinese raw graph model in the embodiment;

[0124] Figure 3 It is a schematic diagram of the structure of the Unet network in the embodiment;

[0125] Figure 4 Schematic diagram of the principle of any round of training process performed on the Wensheng graph model in the embodiment;

[0126] Figure 5 Schematic diagram of the steps of the method for training a text graph model when only one round of training process is performed in an embodiment;

[0127] Figure 6 A schematic diagram of the working principle of the Unet network in the embodiment;

[0128] Figure 7 Schematic diagram of the structure of the CrossAttnDownBlock2D unit in the embodiment;

[0129] Figure 8 It is a structural diagram of the ResnetBlock2D unit in the embodiment;

[0130] Fig. 9 Schematic diagram of the structure of the Transformer2DModel unit in the embodiment;

[0131] Fig.10 It is a schematic diagram of the structure of the MidBlock2DCrossAttn unit in the embodiment;

[0132] Fig.11 Schematic diagram of the structure of the CrossAttnUpBlock2D unit in the embodiment;

[0133] Fig.12 Schematic diagram of the structure of the variable differential encoder VAE in the embodiment;

[0134] Fig.13 Schematic diagram of the principle of a decoder in a variational differential encoder VAE in an embodiment;

[0135] Fig.14 Schematic diagram of the principle of the quality scoring model in the embodiment;

[0136] Fig.15 A schematic diagram of the steps of the method for generating text with semantic image information in an embodiment;

[0137] Fig.16 A schematic diagram showing a comparison between the effect of using the Chinese language graph model in the embodiment and using a common Chinese language graph model;

[0138] Fig.17 Another schematic diagram comparing the effects of using the Chinese language graph model in the embodiment and using a common Chinese language graph model;

[0139] Fig.18 Schematic diagram of the structure of the Chinese raw image model training device in the embodiment;

[0140] Fig.19 It is a structural schematic diagram of a Chinese image generating device in an embodiment;

[0141] Fig. 20 is a schematic diagram of the structure of a computer device in an embodiment;

[0142] Fig.21 Schematic diagram of the working principle of the computer-readable storage medium in the embodiment. DETAILED DESCRIPTION

[0143] Terminology explanation:

[0144] Unet: UNet (also known as U-Net) is a convolutional neural network (CNN) architecture for image segmentation. The structure of UNet consists of two parts: an encoder and a decoder, which is figuratively U-shaped, hence the name. The encoder in Unet consists of multiple convolutional blocks, each of which includes a convolutional layer (usually a 3x3 convolution kernel), batch normalization, and an activation function (usually ReLU). These convolutional blocks help the network learn the low-level features of the image. After each convolutional block, a pooling layer is usually added, such as maximum pooling or average pooling, to reduce the size of the feature map while retaining important information. The decoder in Unet consists of multiple deconvolutional blocks, each of which contains a deconvolutional layer (also known as a transposed convolution), batch normalization, and an activation function. These blocks help learn the high-level semantic information of the image. After each deconvolutional block, an upsampling layer can be added. Layer) to increase the size of the feature map, corresponding to the pooling layer in the encoder; each layer in the decoder is connected to the corresponding layer in the encoder (Concatenation), through skip connections, which helps to combine the semantic information of the bottom and high layers together;

[0145] CLIP: CLIP stands for Contrastive Language-Image Pre-training, which is a pre-training method or model based on contrastive text-image pairs. CLIP is a multimodal model based on contrastive learning. The training data of CLIP is text-image pairs: an image and its corresponding text description. It is hoped that through contrastive learning, the model can learn the matching relationship between text-image pairs. CLIP includes an image encoder and a text encoder, where the image encoder can map the input image to a shared vector space, and the text encoder can map the input text to the same vector space. For the image input to the image encoder and the text input to the text encoder, CLIP can output their similarity, which means the degree of consistency between the content of the image and the semantic content expressed by the text. CLIP can be used in combination with Unet. For example, the text features extracted by the text encoder in CLIP are embedded in the UNet of the diffusion model through cross attention. Specifically, the text features are used as the key and value of attention, and the features of UNet are used as queries. In other words, CLIP can serve as a bridge between text and pictures.

[0146] VIT: The full name is Vision Transformer, which is a model that applies the Transformer concept in natural language processing to image classification.

[0147] ResNet, the full name of which is Residual Network, is a deep convolutional neural network. ResNet is easy to optimize and can improve accuracy by increasing the depth. Its internal residual block uses skip connections (shortcut connections) to alleviate the gradient vanishing problem caused by increasing the depth in deep neural networks. The design of ResNet solves the problem of optimizing training of ordinary networks when the depth increases. By introducing residual learning, the network can better fit the classification function, thereby improving classification accuracy.

[0148] BERT: The full name is Bidirectional Encoder Representations from Transformers, which is a language representation model. The model structure of BERT is a multi-layer bidirectional Transformer encoder. BERT can be used for vocabulary representation tasks and generates word vectors by learning context-related representations of text.

[0149] In this embodiment, the task to be performed is to generate an image with corresponding semantic content based on a piece of text to be processed, and the style of the image meets human needs.

[0150] In order to perform the above task, in this embodiment, a trained text-graph model is used, and the text to be processed is input into the text-graph model for processing. This text-graph model can extract the semantics of the text to be processed and generate an image, and the content of the image is the content described by the text to be processed.

[0151] In this embodiment, a text graph model training method may be executed to obtain a text graph model having a function of generating images with semantic information.

[0152] In this embodiment, refer to Figure 1 ,The Wensheng graph model training method includes the following steps:

[0153] P1. Establish a Wensheng graph model;

[0154] P2. Perform at least one round of training process;

[0155] P3. Determine the cumulative loss function value according to the first denoising feature map and the first weighted value of each round of training process;

[0156] P4. Update the parameters of the Wensheng graph model according to the cumulative loss function value.

[0157] In step P1, the main structure of the text graph model established is the Unet network. Based on the Unet network, modules such as text encoding module, potential image generation module, image restoration module, similarity calculation module and quality scoring module can be set to obtain Figure 2 The text graph model shown. Among them, the Unet network, text encoding module, potential image generation module, image restoration module, similarity calculation module and quality scoring module can be hardware modules or software modules with corresponding functions. For example, the Unet network is a software program that runs the Unet structure, the text encoding module is a software program with text encoding function, and the potential image generation module is a software program with potential image generation function.

[0158] When executing step P1 to establish the Unet network, multiple cross-attention downsampling blocks, one downsampling block, one cross-attention intermediate block, one upsampling block, and multiple cross-attention upsampling blocks can be established. Figure 3, the cross-attention downsampling block is DownBlock2D unit, the cross-attention middle block is MidBlock2DCrossAttn unit, the upsampling block is UpBlock2D unit, and the cross-attention upsampling block is CrossAttnUpBlock2D unit. These units are fully connected in turn to obtain the Unet network.

[0159] In this embodiment, the structure of the Unet network established in step P1 is as follows: Figure 3 See Figure 3 , each CrossAttnDownBlock2D unit and each CrossAttnUpBlock2D unit have their own corresponding dimension sizes. For example, Figure 3 The CrossAttnDownBlock2D units on the left side have dimensions of 32×32×640, 16×16×1280, and 8×8×1280 from top to bottom, respectively, while the CrossAttnUpBlock2D units on the right side have the same dimensions as the CrossAttnDownBlock2D units in the same row, i.e., the CrossAttnUpBlock2D units on the right side have dimensions of 32×32×640, 16×16×1280, and 8×8×1280 from bottom to top, respectively.

[0160] Reference Figure 3 , the Unet network also includes an input convolution layer Conv_in and an output convolution layer Conv_out.

[0161] After executing step P1 to establish the Unet network as the main part of the Wensheng graph model, execute step P2 to perform at least one round of training process on the Wensheng graph model. In this embodiment, the process of any round of training process performed on the Wensheng graph model is as follows: Figure 4 See Figure 1 and Figure 4 , when executing step P2, the following steps are performed during the i-th round of training:

[0162] P201. Get latent image latent-image i and the corresponding prompt word prompt i ;

[0163] P202. Change the prompt word to i Input Unet network as the control condition of denoising process, and transform the latent image into i Input the Unet network for denoising and obtain the first denoised feature map denoised-feature i ;

[0164] P203. Denoised-feature of the first denoised feature map i Perform image restoration processing to obtain the first image to be scored i ;

[0165] P204. For the first image to be scored i Perform quality scoring and obtain the first quality score score i ;

[0166] P205. According to the prompt word prompt i with the first tobescore-image i , determine the first similarity i ;

[0167] P206. For the first quality score i Similarity with the first i Perform weighted operation to determine the first weighted value i .

[0168] Specifically, only one round of training process can be performed on the Wensheng graph model, or multiple rounds of training process can be performed on the Wensheng graph model. The principle of each round of training process is the same. For example, if the Wensheng graph model has been established and only one round of training process is performed, steps P1 and P3 can be omitted. The specific process of steps P1-P4 is as follows: Figure 5 As shown, including:

[0169] P201. Get latent image latent-image i and the corresponding prompt word prompt i ;

[0170] P202. Change the prompt word to i Input Unet network as the control condition of denoising process, and transform the latent image into i Input the Unet network for denoising and obtain the first denoised feature map denoised-feature i ;

[0171] P203. Denoised-feature of the first denoised feature map i Perform image restoration processing to obtain the first image to be scored i ;

[0172] P204. For the first image to be scored i Perform quality scoring and obtain the first quality score score i ;

[0173] P205. According to the prompt word prompt i with the first tobescored-image i , determine the first similarity i ;

[0174] P206. For the first quality score i Similarity with the first i Perform weighted operation to determine the first weighted value i ;

[0175] P3. According to the first denoised feature map denoised-feature i and the first weighted value i , determine the loss function value loss i ;

[0176] P4. According to the loss function value loss i Update the parameters of the Wensheng graph model to achieve the first quality score score i Similarity with the first i The Wensheng graph model is trained to obtain a trained Wensheng graph model.

[0177] It should be noted that the above step P202 is equivalent to performing noise reduction processing on the latent image to obtain the first noise reduction feature map. The above steps P206, P3 and P4 are equivalent to training the Vincent graph model according to the first quality score and the first similarity to obtain the trained Vincent graph model.

[0178] In steps P201-P206 and steps P3-P4, by introducing the first quality score obtained by quality scoring the first image to be scored during the training of the Vincent graph model, the Vincent graph model can be guided to generate image content with a good quality level.

[0179] In this embodiment, when executing step P201, that is, the step of obtaining a potential image and a corresponding prompt word, the following steps may be specifically performed:

[0180] P20101. Generate a random vector in the latent space;

[0181] P20102. Using random vectors as latent images;

[0182] P20103. Generate prompt words based on random vectors.

[0183] In step P20101, Figure 2 In the latent image generation module, a random vector (tensor) is generated in the latent space. The size of the random vector can be 64×64×4. Specifically, in step P20101, a random vector that obeys a Gaussian distribution can be generated. For example, the random vector contains 64×64×4 variables, each of which obeys a Gaussian distribution, that is, each variable takes a value in a Gaussian distribution determined by a corresponding mean and variance, thereby forming a random vector. In step P20102, the random vector generated in step P20101 is used as the latent image latent-image i .

[0184] Since the latent image i It is obtained by randomly generating random vectors, so different rounds of training generally result in different potential images.

[0185] In step P20103, the latent image i Perform semantic recognition and obtain the prompt word prompt i .

[0186] In step P201, the latent image latent-image i is an image expressed in a latent space. The latent space is defined as an abstract multidimensional space that encodes a meaningful internal representation of an external observation event; the latent space is a low-dimensional space, and the latent image latent-image expressed in the latent space i It can be input into the Unet network for processing.

[0187] A batch of prompt words can be prepared manually to form a prompt word library. These prompt words describe some content with realistic meanings, such as describing the subject (which can be a real person, a real animal or a virtual image, etc.), the environment in which the subject is located, the objects held by the subject, the actions taken by the subject, and the objects to which the subject's actions are directed. When executing step P201, a prompt word can be read from the prompt word library. Different prompt words can be read in different training processes. The prompt word read out in step P201 of the i-th round of training process is prompt. i .

[0188] Reference Figure 4 , you can use the text encoding module to encode the prompt word as prompt i Encode the prompt word as prompti Converted to a vector of size 1×77×768.

[0189] Reference Figure 4 In step P202, the prompt word prompt i and latent-image i All are input into the Unet network. i As the control condition for the Unet network to perform noise prediction processing, the Unet network performs noise prediction on the latent image latent-image i Perform denoising to obtain the first denoised feature map i .

[0190] In this embodiment, when executing step P202, that is, the prompt word prompt i Input Unet network as the control condition of denoising process, and transform the latent image into i Input the Unet network for denoising and obtain the first denoised feature map denoised-feature i In this step, you can perform the following steps:

[0191] P20201. Latent-image i Input the Unet network for noise prediction processing to obtain the first predicted noise information forecast-noise i ;

[0192] P20202. From latent-image i Subtract the first forecast noise information forecast-noise i , get the first denoised feature map denoised-feature i .

[0193] In step P20201, the Unet network performs latent-image i The process of noise prediction includes the following steps:

[0194] P2020101. Receive prompt word through multiple cross-attention downsampling blocks (CrossAttnDownBlock2D units) i , for the latent image latent-image i Perform feature integration to obtain the first low-dimensional feature information;

[0195] P2020102. Process the first low-dimensional feature information through a downsampling block (DownBlock2D unit) to obtain second low-dimensional feature information;

[0196] P2020103. Process the second low-dimensional feature information through the cross-attention middle block (MidBlock2DCrossAttn unit) to obtain the third low-dimensional feature information;

[0197] P2020104. Process the third low-dimensional feature information through an upsampling block (UpBlock2D unit) to obtain the first high-dimensional feature information;

[0198] P2020105. Receive prompt word prompt through multiple cross-attention upsampling blocks (CrossAttnUpBlock2D units) i And the processing results of the cross-attention downsampling block (CrossAttnDownBlock2D unit) of the same dimension size, according to the prompt word prompt i The first high-dimensional feature information is integrated with features, and features are added according to the processing results of the cross-attention downsampling block (CrossAttnUpBlock2D unit) to obtain the second high-dimensional feature information;

[0199] P2020106. Determine the second highest dimensional feature information and latent image i The difference between them is used to obtain the first prediction noise information forecast-noise i .

[0200] In step P2020101, refer to Figure 6 , latent-image i After being input into the image segmentation module (Unet network), the latent image latent-image of size 64×64×4 i First, it is processed by an input convolution layer Conv_in in the Unet network to obtain features with a dimension of 64×64×320, and then it passes through 3 layers of cross-attention downsampling blocks (CrossAttnDownBlock2D units) and 1 layer of downsampling blocks (DownBlock2D units) to obtain the first low-dimensional feature information of the deep layer. The dimension of the first low-dimensional feature information is 8×8×1280.

[0201] Specifically, in this embodiment, the structure of the cross-attention downsampling block (CrossAttnDownBlock2D unit) used is as follows: Figure 7 As shown, refer to Figure 7, a cross-attention downsampling block (CrossAttnDownBlock2D unit) consists of two 2D residual blocks, two 2D transformer models, and a 2D downsampling layer.

[0202] Among them, the 2D residual block is ResnetBlock2D, ResnetBlock2D can introduce residual connections, so that the gradient can be directly transferred through the residual connection, thereby effectively alleviating the gradient disappearance problem; the 2D transformer model is Transformer2DModel, Transformer2DModel can implement the self-attention mechanism, so as to deeply understand the context information of the text; the 2D downsampling layer is Downsample2D, Downsample2D can reduce the dimension of the data, thereby reducing the computational burden of the model and accelerating the reasoning process.

[0203] The structure of the 2D residual block (ResnetBlock2D) is as follows Figure 8 As shown, refer to Figure 8 , a 2D residual block (ResnetBlock2D) consists of two 3×3 grouped convolutions; the structure of the 2D transformer model (Transformer2DModel) is as follows Fig. 9 As shown, refer to Fig. 9 A 2D transformer model (Transformer2DModel) consists of 1 grouped convolution, 2 transformer blocks, and 2 fully connected layers, and in the transformer block, the prompt word is used through cross attention. i The 2D downsampling layer (Dowmsample2D) is a convolution layer with a convolution kernel of 3×3 and a stride of 2.

[0204] In this embodiment, Figure 6 The downsampling block (DownBlock2D unit) in consists of two convolutional layers with a kernel size of 3×3, where the convolution kernel of the second convolutional layer is 3×3 and the stride is 2.

[0205] Since the transformer block is set in the CrossAttnDownBlock2D unit, the CrossAttnDownBlock2D unit can incorporate the prompt word in the feature extraction process through cross attention. i , so that the first low-dimensional feature information obtained by executing step P2020101 contains the prompt word prompt ifeature information.

[0206] Reference Figure 6 After executing step P2020101 to obtain the first low-dimensional feature information, step P2020102 is then executed. In step P2020102, the first low-dimensional feature information is further extracted through a downsampling block (DownBlock2D unit).

[0207] In this embodiment, Figure 6 The downsampling block (DownBlock2D) in Fig. 1 consists of two convolutional layers with a convolution kernel size of 3×3, where the convolution kernel of the second convolutional layer is 3×3 and the stride is 2.

[0208] In step P2020102, the downsampling block (DownBlock2D unit) downsamples the first low-dimensional feature information to obtain second low-dimensional feature information. Figure 6 , the dimension of the second low-dimensional feature information is 8×8×1280.

[0209] Reference Figure 6 After executing step P2020102 to obtain the second low-dimensional feature information, step P2020103 is then executed. In step P2020103, the cross-attention intermediate block (MidBlock2DCrossAttn unit) is used to further extract features. Among them, MidBlock2DCrossAttn is as follows Fig.10 As shown, a MidBlock2DCrossAttn consists of 2 ResnetBlock2D and 1 Transformer2DModel.

[0210] In step P2020103, the cross-attention intermediate block (MidBlock2DCrossAttn unit) outputs the third low-dimensional feature information.

[0211] In step P2020104, refer to Figure 6 , the third low-dimensional feature information extracted by the cross-attention middle block (MidBlock2DCrossAttn unit) is input into the upsampling block (UpBlock2D unit) for feature extraction. Among them, UpBlock2D consists of a 3×3 convolution layer and a deconvolution layer with a stride of 2.

[0212] In step P2020104, the upsampling block (UpBlock2D unit) upsamples the third low-dimensional feature information to obtain the first high-dimensional feature information.

[0213] In step P2020105, refer to Figure 6, the first high-dimensional feature information extracted by the upsampling block (UpBlock2D unit) is input into the 3-layer cross-attention upsampling block (CrossAttnUpBlock2D unit) for feature extraction.

[0214] In this embodiment, the structure of the cross-attention upsampling block (CrossAttnUpBlock2D unit) is as follows Fig.11 See Fig.11 , since the cross-attention up-sampling block (CrossAttnUpBlock2D unit) is the same as the cross-attention down-sampling block (CrossAttnDownBlock2D unit), it also has Figure 7 The 2D transformer model (Transformer2DModel) shown in Figure 1 has a transformer block set so that the CrossAttnUpBlock2D unit can incorporate the prompt word in the feature extraction process through cross attention. i Moreover, refer to Figure 6 When performing feature extraction, each cross-attention upsampling block (CrossAttnUpBlock2D unit) obtains the processing results of the cross-attention downsampling block (CrossAttnDownBlock2D unit) of the same dimension (same row), adds it to the extracted features and outputs it. After being processed by multiple cross-attention upsampling blocks (CrossAttnUpBlock2D units), the scale of the feature map is improved. During this period, the deep features and shallow features of the same dimension are added together to fuse features of different scales. The final second-highest dimensional feature information can be processed by an output convolution layer Conv_out and become the latent image latent-image. i The second highest dimensional feature information obtained is the latent-image information obtained by Unet. i Therefore, in step P2020106, by calculating the high-dimensional feature information and the latent image latent-image i The difference between them can be used to obtain the first prediction noise information forecast-noise i .

[0215] In step P20201, i.e., steps P2020101-P2020106, the noise prediction function of the Unet network is used to predict the latent image i Perform noise prediction and obtain the first predicted noise information forecast-noise iThen, step P2020102 is performed to calculate the latent image i With the first forecast noise information forecast-noise i Subtract pixel by pixel to obtain the first denoised feature map denoised-feature i .

[0216] In this embodiment, by executing steps P20201-P20203, the prompt word prompt can be i and latent-image i Input the Unet network for denoising, so that the first denoised feature map obtained by the Unet network for denoising is denoised-feature i Incorporated into the prompt word prompt i The features of , thereby guiding the Unet network to generate prompt words i The image of the feature, that is, the first denoised feature map generated by the Unet network i With the prompt word orompt i semantic features.

[0217] In this embodiment, refer to Figure 3 and Figure 4 , the first denoised feature map generated by the Unet network i The dimension of the latent image received by the Unet network is i The dimensions of . For example, the latent image latent-image i The dimension is 64×64×4, then the first denoised feature map denoised-feature i The dimensions are also 64×64×4.

[0218] Execute step P202 to obtain the first denoised feature map denoised-feature i It is the data in the latent space, and the first denoised feature map can be denoised-feature i After the data is converted into RGB format [a color standard in the industry, a variety of colors are obtained by changing the three color channels of red, green, and blue and superimposing them on each other. RGB represents the colors of the three channels of red, green, and blue. This standard covers almost all colors that can be perceived by human vision and is one of the most widely used color systems], the subsequent steps such as steps P204-P206 are processed.

[0219] After executing step P202, the first denoised feature map denoised-feature is obtained. i After that, step P203 is executed to perform image restoration processing, so as to convert the first denoised-feature map of the data in the latent space generated by the Unet network into i Convert data into RGB space, that is, the first image to be scored i .

[0220] In this embodiment, when executing step P203, that is, the first denoised feature map denoised-feature i Perform image restoration processing to obtain the first image to be scored i In this step, you can perform the following steps:

[0221] P20301. Establish variable differential encoder;

[0222] P20302. The first denoised feature map is denoised-feature i Input the variable differential encoder for processing to obtain the first image to be scored i .

[0223] In step P20301, the variational differential encoder used can also be called a variational autoencoder (VAE).

[0224] The variational differential encoder VAE used in step P20301 is improved on the basis of the basic autoencoder (AE). The original purpose of the autoencoder was to learn the main rules of the data, perform data compression or feature extraction, and is a type of unsupervised learning algorithm. The field of using machine learning or deep learning methods to allow the algorithm to solve the data representation results is called representation learning. The autoencoder is a type of algorithm in representation learning, but various subsequent improved algorithms have enabled the autoencoder to have generation capabilities, such as the image generation capability of the variational differential encoder VAE.

[0225] Reference Fig.12The variational differential encoder VAE used in step P20301 consists of two parts: encoder and decoder. The encoder can map the input data to the probability distribution in the latent space, while the decoder can reconstruct the input data from the hidden variables in the latent space. The variational differential encoder VAE learns the distribution of the latent space by maximizing the marginal likelihood of the observed data and achieves effective training through reparameterization techniques.

[0226] In this embodiment, the decoder Decoder in the variational differential encoder VAE used in step P20301 is as follows: Fig.13 See Fig.13 , the decoder includes multiple 2-layer 3×3 convolutional layers, one 1-layer 3×3 convolutional layer and multiple upsampling layers, and these convolutional layers and upsampling layers are alternately fully connected. Specifically, refer to Fig.13 , the first 2-layer 3×3 convolutional layer is fully connected to the first upsampling layer, the first upsampling layer is fully connected to the second 2-layer 3×3 convolutional layer, the second 2-layer 3×3 convolutional layer is fully connected to the second upsampling layer... Alternate in this way until the last upsampling layer is fully connected to the 1-layer 3×3 convolutional layer, and the decoder Decoder in the variational differential encoder VAE is obtained.

[0227] In step P20302, the decoder Decoder in the variational differential encoder VAE is used to denoise the latent space data after denoising, that is, the first denoised feature map denoised-feature i Perform up-dimensional decoding to obtain the result of RGB three channels, that is, the first image to be scored i .

[0228] Specifically, refer to Fig.13 , the first denoised-feature map of the data in the latent space generated by the Unet network i The size is 64×64×4, the first denoised feature map is denoised-feature iAfter being input into the decoder Decoder in the variational differential encoder VAE, the first 2-layer 3×3 convolutional layer performs feature extraction to obtain features of size 64×64×128, and the first upsampling layer upsamples the features of size 64×64×128 to obtain features of size 128×128×128; then, the second 2-layer 3×3 convolutional layer performs feature extraction on the features of size 128×128×128 to obtain features of size 256×256×256, and the second upsampling layer upsamples the features of size 256×256×256 to obtain features of size Then, the last 2 layers of 3×3 convolutional layers extract the features output by the second upsampling layer to obtain features of size 256×256×512, and the last upsampling layer upsamples the features of size 256×256×512 to obtain features of size 512×512×512; finally, a 1 layer of 3×3 convolutional layer extracts the features of size 512×512×512 to obtain an output result of size 512×512×3, that is, the first to be scored image represented by the RGB three channels. i .

[0229] like Fig.13 As shown, in step P20302, the features output by each convolution layer after convolution processing are the first denoising feature map tobescored-image i The corresponding mean data and variance data constitute the hidden variable; the hidden variable is upsampled through each upsampling layer to obtain the first tobescored-image i .

[0230] pass Fig.13 The step-by-step upsampling process shown in the figure can eventually obtain an RGB image of size 512×512×3, which is the first image to be scored. i , as the output image of the variational differential encoder VAE.

[0231] Since the learning goal of the Wensheng graph model is to generate images with high-quality scores, the images generated by the trained Wensheng graph model will meet the quality standards of the quality model. Therefore, after executing step P203, the first to-be-scored-image is obtained. i Then, step P204 is executed to process the first image to be scored i Perform quality rating.

[0232] When executing step P204, it can be done manually. Specifically, a person can read the first to-be-scored-image i , according to the first image to be scored i The perception of the content, according to certain aesthetic standards and / or quality standards, for the first to be scored image tobescored-image i The first quality score is obtained by quantifying the degree of conformity to human aesthetics. i .

[0233] In this embodiment, the first to-be-scored-image i The quality standard for quality scoring can be the first image to be scored-image i Whether the content in the text conforms to the laws of nature, the first image to be scored i Whether the content in the video conforms to the principles of art (such as perspective principle and color combination principle, etc.), the first image to be scored i Content and prompt word prompt i The degree of fit between the semantic description of the content and the first image to be scored i Whether the message conveyed by the content is positive, etc.

[0234] In this embodiment, in step P204, the first image to be scored is i When performing quality scoring, it can be controlled within a certain scoring range so that the first quality score obtained is i Specifically, the scoring range can be set to 0-10 points.

[0235] In this embodiment, in step P204, the first image to be scored is i When performing quality scoring, a trained deep learning model may also be used. Specifically, when executing step P204, that is, performing quality scoring on the first image to be scored and obtaining a first quality score, the following steps may be performed:

[0236] P20401.Build a deep learning model;

[0237] P20402. Train the deep learning model as a quality scoring model;

[0238] P20403. The first image to be scored tobescored-image i Input the quality scoring model for processing to obtain the first quality score score i .

[0239] The deep learning model established in step P20401 can be VIT, ResNet or GCN (Graph Convolutional Network), etc.

[0240] In step P20402, a batch of sample images and sample scores can be obtained to form a training set, wherein the sample score is the score obtained by scoring the sample images according to aesthetic standards and / or image quality standards. The deep learning model is trained using the training set, and the trained deep learning model becomes a quality scoring model. Image quality scoring aims to learn a model that automatically classifies a given image as high quality or low quality (the higher the score, the higher the quality, i.e., the image is better for the user's visual perception). Its key task is to find a suitable image representation method that can establish a relationship between image content and image quality.

[0241] In step P20403, the first to-be-scored-image i Input into the trained deep learning model, obtain the output of the deep learning model, and thus obtain the first quality score score i .

[0242] By using the trained deep learning model instead of manually scoring the first tobescored-image i Perform quality scoring and obtain the first quality score with a stable scoring standard i ,It can reduce the problems of inconsistent scoring standards caused by subjective feelings or decreased energy, and can improve the efficiency of quality scoring, which is conducive to the multi-round training process of the Wensheng graph model.

[0243] For example, refer to Fig.14 When executing steps P20401-P20403, the Fig.14 The deep learning model shown in FIG. 1 includes a full convolutional layer, a softmax classifier, a first layout-aware graph convolution module (Layout-aware GCN 1) and a second layout-aware graph convolution module (Layout-aware GCN 2). Such a deep learning model can be used as a quality scoring model. In step P20403, the first image to be scored is iAfter the quality scoring model is input, the quality scoring model first performs a quality scoring on the first image to be scored. i Perform scaling to preserve aspect ratio information and use the full convolutional layer to extract the first image to be scored-image i The feature graph of is regarded as the node feature, and an undirected fully connected graph considering the quality attribute is constructed. The first layout-aware graph convolution module is used to transfer information through graph convolution in the coordinate space, and the score corresponding to the first layout-aware graph convolution module is output through the softmax classifier; after the first layout-aware graph convolution module maps the node to the latent space, the second layout-aware graph convolution module is used to implement more efficient graph convolution on the aggregated nodes, and the score corresponding to the second layout-aware graph convolution module is output through the softmax classifier; finally, the score corresponding to the first layout-aware graph convolution module and the score corresponding to the second layout-aware graph convolution module are fused through the full convolution layer to obtain the first quality score score i .

[0244] In step P205, for the prompt word prompt i The semantic description of the content, and the first image to be scored tobescored-image i The depicted scene contents are compared to obtain the first similarity i .

[0245] When executing step P205, that is, according to the prompt word prompt i with the first tobescored-image i , determine the first similarity i This step can be done manually, or you can perform the following steps:

[0246] P20501. Establish similarity calculation module;

[0247] P20502. Change the prompt word to i Input the text encoder in the similarity calculation module for processing to obtain the first text embedding vector text-embedding i ;

[0248] P20503. The first image to be scored is tobescored-image i Input the image encoder in the similarity calculation module for processing to obtain the first image embedding vector image-embedding i ;

[0249] P20504. According to the first text embedding vector text-embedding i image-embedding with the first image embedding vector i Perform similarity calculation to obtain the first similarity simillarity i .

[0250] The similarity calculation module established in step P20501 is as follows Figure 4 As shown, it can be a CLIP network, including a text encoder and an image encoder. Among them, the text encoder in the CLIP network uses the Transformer-based BERT model, while the image encoder uses the VIT structure.

[0251] In step P20502, the prompt word prompt i The text encoder in the CLIP network is processed. The text encoder, BERT model, converts the prompt word into i Convert into the corresponding word vector as the first text embedding vector text-embedding i .

[0252] In step P20503, the first image to be scored output by the variational differential encoder VAE is i The image encoder in the CLIP network is processed. The image encoder, i.e., the VIT model, converts the first image to be scored into i Mapping to the first text embedding vector text-embedding i The space where it is located is converted into the corresponding vector as the first image embedding vector image-embedding i .

[0253] In step P20504, the first text embedding vector text-embedding may be calculated. i image-embedding with the first image embedding vector i The distance between them is taken as the first text embedding vector text-embedding i image-embedding with the first image embedding vector i The similarity between them, that is, the first similarity i Specifically, the first text embedding vector text-embedding can be calculated i image-embedding with the first image embedding vector iThe cosine distance between them is used to obtain the first similarity i .

[0254] Reference Figure 4 , in step P204, the first quality score score is obtained i And execute step P205 to obtain the first similarity i Then, execute step P206 to calculate the first quality score score i Similarity with the first i Perform weighted operation to determine the first weighted value i Specifically, the first quality score score can be set i With the first similarity slmilarityi i The corresponding weights are used to calculate the first quality score. i Similarity with the first i The weighted sum of , thereby obtaining the first weighted value value i .

[0255] In this embodiment, when executing step P206, the calculation formula of the first weighted value is:

[0256]

[0257] Among them, Z is the first weighted value i , A is the first quality score score i , B is the first similarity i , p is the preset parameter. That is, when calculating the first weighted value, the first quality score score i The weight is The first similarity slmilarityi i The weight of is 1-p.

[0258] The range of the preset parameter p is 0-1. The size of the preset parameter p can be set according to the requirements of the trained text graph model. For example, if you want the trained text graph model to generate an image based on the text to be processed, pay more attention to the content similarity between the image generated by the text graph model and the text to be processed, then the preset parameter p can be set to a smaller value; if you pay more attention to the aesthetics of the image generated by the text graph model, then the preset parameter p can be set to a larger value.

[0259] By using the formula in step P206, the first quality score score is adjusted according to the preset parameter p. i Similarity with the firsti The weight size is used to calculate the first weighted value i By adjusting the preset parameter p, the trained text graph model can focus more on the content similarity of the generated image or the aesthetics of the generated image, thereby improving the controllability of the training process of the text graph model.

[0260] Steps P201-P206 are to execute the i-th round of training process to obtain the corresponding first weighted value value i After executing multiple rounds of training processes, each round of training process will obtain the corresponding first denoising feature map and the first weighted value, and step P3 is executed based on these data.

[0261] When executing step P3, that is, determining the cumulative loss function value according to the first denoising feature map and the first weighted value of each round of training process, the following steps may be specifically performed:

[0262] P301. For any round of training, a weighted probability is determined according to the corresponding first weighted value and the first noise reduction feature map obtained in the previous round of training;

[0263] P302. Obtain the cumulative loss function value based on the weighted probabilities corresponding to each of the entire training processes.

[0264] Taking the i-th round of training process as an example, step P301 is explained.

[0265] In step P301, for the i-th round of training process, obtain the first weighted value value corresponding to the i-th round of training process i , and the first denoised feature map denoised-feature obtained in the previous round, i.e. the i-1th round of training process i-1 , thereby calculating the weighted probability corresponding to the i-th round of training process.

[0266] Specifically, for the i-th round of training, when executing step P301, the first denoised feature map obtained in the previous round, i.e., the i-1th round of training, is converted into i-1 Represented as X i-1 , X i-1 According to the probability distribution p of the Unet network θ After normalization, it is expressed as p θ (X i-1 |X i ,C), and the logarithmic value is expressed as logp θ (X i-1 |X i ,C), calculate the logarithmic value logp θ (X i-1 |Xi ,C) The first weighted value corresponding to this round, i.e. the i-th round of training process i [expressed as r(X i ,C)] to obtain the weighted probability logp corresponding to this round, i.e. the i-th round of training process θ (X i-1 |X i ,C)r(X i ,C).

[0267] If the i-th round of training is the first round of training, then there is no i-1-th round of training. i-1 Get a random value.

[0268] When T rounds of training have been performed, step P301 is performed for each round of training, that is, i=1, 2, ... T are traversed to calculate the weighted probability logp corresponding to each round of training. θ (X i-1 |X i ,C)r(X i , C). Then execute step P302.

[0269] In step P302, the cumulative loss function value is obtained according to the weighted probabilities corresponding to all training processes. Specifically, the cumulative loss function value can be calculated according to the following formula:

[0270]

[0271] in, is the cumulative loss function value, represents the mathematical expectation operation, Represents the gradient operation.

[0272] In step P302, the cumulative loss function value is calculated by a new method. This method is different from the traditional L1 distance or perceptual loss. In the calculation process of this method, the weighted probability corresponding to each round of training process is maximized, thereby enhancing the first quality score. i Similarity with the first i The influence of is conducive to completing the training with fewer training rounds, improving the training efficiency, and training a text-based graph model that can generate images with similar content or beautiful appearance.

[0273] After executing steps P1-P3, T rounds of training are completed and the corresponding cumulative loss function value is obtained. After that, execute step P4, according to the cumulative loss function value Update the parameters of the Unet network, including:

[0274] According to the cumulative loss function value Execute the gradient back-propagation algorithm to back-propagate and update the parameters of the Unet network to train the Wensheng graph model.

[0275] In this embodiment, the Wensheng graph method can be performed using the Wensheng graph model trained through steps P1-P4. Fig.15 ,The text generation method with semantic image information includes the following steps:

[0276] S1. Obtaining the Wensheng graph model;

[0277] S2. Get the text to be processed;

[0278] S3. Input the text to be processed into the text graph model for processing to obtain a generated feature graph;

[0279] S4. Perform image restoration processing on the generated feature map to obtain a generated image.

[0280] In step S1, the Wensheng graph model trained in steps P1-P4 is obtained.

[0281] In step S2, the content of the text to be processed may be "a rabbit warrior holding a sword" or "a surreal panda sitting at a bar drinking beer and celebrating together", etc. Before executing step S3, a text encoding module may be used to encode the text to be processed, thereby converting the text to be processed into a vector that can be processed by the text graph model.

[0282] In step S3, the text to be processed obtained in step S2 is input into the Unet network for processing, and a generated feature map output by the Unet network is obtained.

[0283] In step S4, the decoder Decoder in the variational differential encoder VAE can be used to convert the generated feature map output by the Unet network into an image of the RGB channels to obtain a generated image.

[0284] In this embodiment, for the text to be processed with the content "a rabbit warrior, holding a sword in his hand", an image is generated according to the text to be processed with the common text-generated graph model as follows: Fig.16 As shown in part (a) of FIG. 1 , the image generated by the text-generated graph model in this embodiment according to the text to be processed is as follows: Fig.16 As shown in part (b) of Fig.16As can be seen from part (a) of the figure, the sword is in a suspended state, which does not match the content of "holding a sword in hand" in the text to be processed. In addition, the rabbit has few ornaments on its body, which cannot well reflect the image of "rabbit warrior". It is difficult for people to summarize the description of "rabbit warrior holding a sword in hand" when viewing the picture. Therefore, the image generated by the ordinary text image model is not aesthetically pleasing. Fig.16 As can be seen from part (b) of the figure, the sword is held by the rabbit, which is consistent with the content of "holding a sword in hand" in the text to be processed. In addition, the rabbit is wearing armor, wrist guards and other equipment, which can well reflect the image of "rabbit warrior". Therefore, the image generated by the text-generated graph model in this embodiment has a high aesthetic level.

[0285] In this embodiment, for the text to be processed with the content of "Super realistic pandas sitting in front of a bar drinking beer and toasting together", the image generated by the ordinary text graph model according to the text to be processed with the content of "Super realistic pandas sitting in front of a bar drinking beer and toasting together" is as follows: Fig.17 As shown in part (a) of FIG. 1 , the image generated by executing steps S1-S4 using the Unet network in this embodiment according to the text to be processed is as follows: Fig.17 As shown in part (b) of Fig.17 As can be seen from part (a) of the figure, the beer is placed on the table, which does not match the content of "toasting" in the text to be processed. In addition, there is only one glass of beer and the panda's expression is calm, which cannot well reflect the image of "toasting together to celebrate". It is difficult for people to summarize the description of "pandas toasting together to celebrate" when reading the picture. Therefore, the image generated by the ordinary text image model is not aesthetically pleasing. Fig.17 As can be seen from part (b) of the figure, the two pandas each hold a glass of beer, which is consistent with the content of "raising a toast" in the text to be processed, and the pandas' excited expressions can well reflect the image of "raising a toast to celebrate together". Therefore, the image generated by the cultural graph model in this embodiment has a high aesthetic level.

[0286] This embodiment provides a text graph model training device, referring to Fig.18 , the Wensheng graph model training device includes:

[0287] A training data acquisition module, used to acquire potential images and corresponding prompt words;

[0288] A denoising module, used for performing denoising on the latent image to obtain a first denoising feature map;

[0289] A restoration module, used for performing image restoration processing on the first denoising feature map to obtain a first image to be scored;

[0290] A scoring module, used for scoring the quality of the first image to be scored to obtain a first quality score;

[0291] A similarity module, used for determining a first similarity according to the prompt word and the first image to be rated;

[0292] The training module is used to train the Wensheng graph model according to the first quality score and the first similarity to obtain the trained Wensheng graph model.

[0293] By running the text graph model training device, the same technical effect as the text graph model training method in the embodiment can be achieved.

[0294] This embodiment provides a Wensheng map device, referring to Fig.19 , the Wenshengtu device includes:

[0295] A Wensheng graph module, used to obtain a Wensheng graph model; the Wensheng graph model is trained by a Wensheng graph model training method;

[0296] A module for obtaining text to be processed, used for obtaining text to be processed;

[0297] A processing module, used for inputting the text to be processed into the text graph model for processing to obtain a generated feature graph;

[0298] The restoration module is used to perform image restoration processing on the generated feature map to obtain a generated image.

[0299] By running the Wensheng diagram device, the same technical effect as the Wensheng diagram method in the embodiment can be achieved.

[0300] In this embodiment, the computer is programmed to execute Figure 1 and storing such a computer program in a Fig. 20 The memory of the computer device shown, or Fig.21 The computer program stored in the memory or computer-readable storage medium may be Fig. 20 The processor in Fig.21 The processor in the processor reads it out and obtains the execution Figure 1 The processor executes the instructions of each step in the process, and can execute the Wensheng graph model training method or the Wensheng graph method.

[0301] because Fig. 20 The computer device and Fig.21 The computer-readable storage media shown can enable the processor to execute the text-image model training method or the text generation method with semantic image information, so Fig. 20 The computer device and Fig.21The computer-readable storage media shown can achieve the same technical effects as the Wensheng graph model training method or Wensheng graph method in the embodiment.

[0302] It should be noted that, unless otherwise specified, the singular forms of "a" and "the" used in this disclosure are also intended to include plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by those skilled in the art. The terms used in the present embodiment specification are only for describing specific embodiments, not for limiting the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0303] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0304] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.

[0305] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.

[0306] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0307] The above are only preferred embodiments of the present invention. The present invention is not limited to the above embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical solutions and / or implementation methods may have various modifications and changes.

Claims

1. A method for training a text graph model, characterized in that: The method for training the culture graph model comprises: Obtaining potential images and corresponding prompt words; Performing noise reduction processing on the latent image to obtain a first noise reduction feature map; Performing image restoration processing on the first denoising feature map to obtain a first image to be scored; Performing a quality score on the first image to be scored to obtain a first quality score; Determining a first similarity according to the prompt word and the first image to be rated; The Vincent graph model is trained according to the first quality score and the first similarity to obtain a trained Vincent graph model.

2. The method for training a text graph model according to claim 1, characterized in that: The step of obtaining the potential image and the corresponding prompt word includes: Generate a random vector in the latent space; Using the random vector as the latent image; The prompt word is generated according to the random vector.

3. The method for training a text graph model according to claim 1, characterized in that: The performing image restoration processing on the first denoising feature map to obtain a first image to be scored includes: Establish a variable differential encoder; The first denoising feature map is input into the variational differential encoder for processing to obtain the first image to be scored.

4. The method for training a text graph model according to claim 3, characterized in that: The step of establishing a variable differential encoder comprises: Establish multiple convolutional layers and multiple upsampling layers; Fully connecting each of the convolutional layers and each of the upsampling layers alternately to obtain the variational differential encoder; The step of inputting the first noise reduction feature map into the variational differential encoder for processing to obtain the first image to be scored comprises: Performing convolution processing on the first denoising feature map through each of the convolution layers to obtain mean data and variance data corresponding to the first denoising feature map; Constructing latent variables according to the mean data and the variance data; The latent variable is upsampled through each of the upsampling layers to obtain the first image to be scored.

5. The method for training a text graph model according to claim 1, characterized in that: The step of performing noise reduction processing on the latent image to obtain a first noise reduction feature map includes: Performing noise prediction processing on the latent image to obtain first predicted noise information; The first predicted noise information is subtracted from the potential image to obtain the first denoising feature map.

6. The method for training a text graph model according to claim 1, characterized in that: The step of performing a quality score on the first image to be scored to obtain a first quality score includes: Building deep learning models; Training the deep learning model as a quality scoring model; The first image to be rated is input into the quality rating model for processing to obtain the first quality score.

7. The method for training a text graph model according to claim 1, characterized in that: The determining a first similarity according to the prompt word and the first image to be rated includes: Establishing a similarity calculation module; the similarity calculation module includes a text encoder and an image encoder; Inputting the prompt word into the text encoder for processing to obtain a first text embedding vector; Inputting the first image to be rated into the image encoder for processing to obtain a first image embedding vector; A similarity calculation is performed based on the first text embedding vector and the first image embedding vector to obtain the first similarity.

8. The method for training a text graph model according to claim 1, characterized in that: The performing a weighted operation on the first quality score and the first similarity to determine a first weighted value includes: Set the preset parameter p; The first quality score and the first similarity are weighted and summed according to the preset parameters to determine the first weighted value; wherein the calculation formula of the first weighted value is: Among them, Z is the first weighted value, A is the first quality score, and B is the first similarity.

9. The method for training a text graph model according to any one of claims 1 to 8, characterized in that: The step of training the Vincent graph model according to the first quality score and the first similarity to obtain the trained Vincent graph model includes: Perform at least one round of training process; each round of the training process obtains a corresponding first denoising feature map, a first quality score and a first similarity; For any round of the training process, performing a weighted operation on the first quality score and the first similarity corresponding to the training process to determine a first weighted value corresponding to the training process; According to the first denoising feature map and the first weighted value of each round of the training process, a cumulative loss function value is determined; and according to the cumulative loss function value, parameters of the Vincent graph model are updated to obtain a trained Vincent graph model.

10. The method for training a text graph model according to claim 9, characterized in that: The step of determining a cumulative loss function value according to the first denoising feature map and the first weighted value of each round of the training process includes: For any round of the training process, determining a weighted probability according to the corresponding first weighted value and the first denoising feature map obtained in the previous round of the training process; The cumulative loss function value is obtained according to the weighted probabilities corresponding to all the training processes.

11. The method for training a text graph model according to claim 10, characterized in that: For any round of the training process, determining the weighted probability according to the corresponding first weighted value and the first noise reduction feature map obtained in the previous round of the training process includes: Normalizing the first denoising feature map obtained in the previous round of the training process according to the probability distribution of the Wensheng graph model, and taking a logarithmic value; The product of the logarithmic value and the first weighted value corresponding to the current round of the training process is calculated to obtain the weighted probability corresponding to the current round of the training process.

12. The method for training a text graph model according to claim 10, characterized in that: The step of obtaining the cumulative loss function value according to the weighted probabilities corresponding to all the training processes comprises: The mathematical expectation of the weighted probabilities corresponding to all the training processes is calculated to obtain the cumulative loss function value.

13. The method for training a text graph model according to any one of claim 10, characterized in that: The updating of parameters of the Wensheng graph model according to the cumulative loss function value to obtain a trained Wensheng graph model includes: A gradient back propagation algorithm is executed according to the accumulated loss function value, and the parameters of the Wensheng graph model are back propagated and updated to obtain a trained Wensheng graph model.

14. A Vincent diagram method, characterized in that: The Wensheng diagram method comprises: Acquire a Wensheng graph model; the Wensheng graph model is trained by the Wensheng graph model training method according to any one of claims 1 to 13; Get the text to be processed; Inputting the text to be processed into the text graph model for processing to obtain a generated feature graph; Perform image restoration processing on the generated feature map to obtain a generated image.

15. A device for training a text graph model, characterized in that: The Wensheng graph model training device comprises: A training data acquisition module, used to acquire potential images and corresponding prompt words; A denoising module, configured to perform denoising on the latent image to obtain a first denoising feature map; A restoration module, used for performing image restoration processing on the first denoising feature map to obtain a first image to be scored; A scoring module, used for performing a quality score on the first image to be scored to obtain a first quality score; A similarity module, used for determining a first similarity according to the prompt word and the first image to be rated; A training module is used to train the document graph model according to the first quality score and the first similarity to obtain a trained document graph model.

16. A Wensheng map device, characterized in that: The Wensheng map device comprises: A Wensheng graph module, used to obtain a Wensheng graph model; the Wensheng graph model is trained by the Wensheng graph model training method according to any one of claims 1 to 13; A module for obtaining text to be processed, used for obtaining text to be processed; A processing module, used for inputting the text to be processed into the text graph model for processing to obtain a generated feature graph; The restoration module is used to perform image restoration processing on the generated feature map to obtain a generated image.

Images