A method, device, storage medium and terminal for generating text to image

Abstract

The present invention discloses a method, device, storage medium and terminal for generating text to image. The method includes: obtaining a text description, tokenizing the text description and generating a text shape sequence; generating at least one first image based on the text shape sequence, a pre-trained image generation model and a vector quantization autoencoder; inputting each first image into a pre-trained scoring model to obtain a probability value for each first image; based on the probability value of each first image, screening first images with a probability value greater than a preset threshold to generate at least one second image; and increasing the resolution of the second image based on a pre-trained resolution enhancement model to generate a target image. Therefore, by adopting the embodiments of the present application, it is possible to ensure that the content of the generated image is consistent with the semantics of the descriptive text, greatly reducing the error between the two, and effectively improving the resolution of the generated image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, storage medium and terminal for generating text into an image. Background Art

[0002] Generating images from text is a comprehensive, cross-disciplinary task that combines computer vision and natural language processing. It's a key branch of the image generation field and has broad applications, such as matching text with illustrations and assisting with design. In recent years, with the rise of deep learning technology, which has achieved breakthroughs in image recognition, classification, and image deblurring, researchers are increasingly eager to achieve lower discrepancies between image content and text semantics in text-generated images, achieving higher resolution.

[0003] In the existing technology, AttnGAN (Attention Generative Adversarial Network) is used to generate images from text. The images obtained using this network have a large error between the image content and the text semantics. In order to reduce the error, researchers have proposed a multi-step method that first generates an object frame based on the text description, then generates the object contour, and finally obtains the image. Due to the accumulation of errors in each step of the multi-step method, the generated image effect is still quite different from the real image and has a low resolution. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, storage medium, and terminal for generating text from images. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is provided below. This summary is not intended to be a comprehensive review, identify key or important components, or delineate the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simplified form, serving as a prelude to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a method for generating text to an image, the method comprising:

[0006] Get the text description, tokenize it and generate a text shape sequence;

[0007] generating at least one first image based on a sequence of text glyphs, a pre-trained image generation model, and a vector quantized autoencoder;

[0008] Input each first image into a pre-trained scoring model to obtain a probability value for each first image;

[0009] Based on the probability value of each first image, first images having a probability value greater than a preset threshold are screened to generate at least one second image;

[0010] The resolution of the second image is increased based on the pre-trained resolution enhancement model to generate a target image; wherein,

[0011] The pre-trained image generation model is generated by concatenating the image shape sequence trained by the model and the text shape sequence trained by the model. The pre-trained scoring model and the pre-trained resolution enhancement model are generated by training on the pre-trained image generation model.

[0012] Optionally, generating at least one first image according to the text glyph sequence, a pre-trained image generation model, and a vector quantized autoencoder includes:

[0013] Input the text glyph sequence into a pre-trained image generation model and output a probability distribution;

[0014] Get the image label corresponding to the maximum probability value from the probability distribution;

[0015] After splicing the image tag corresponding to the maximum probability value to the text shape sequence, a spliced ​​text shape sequence is generated;

[0016] Count the number of image tags;

[0017] When the number of image tags is less than a preset first number, continue inputting the spliced ​​text shape sequence into the pre-trained image generation model until the number of image tags equals the preset first number, thereby generating an image tag set;

[0018] Inputting the image label set into the decoder of a pre-trained vector quantized autoencoder to generate a first image;

[0019] Counting the number of first images, and when the number of first images is less than a preset second number, continuing to input the text shape sequence into a pre-trained image generation model until at least one first image is generated when the number of first images is equal to the preset second number.

[0020] Optionally, increasing the resolution of the second image based on a pre-trained resolution enhancement model to generate a target image includes:

[0021] dividing the second image into a preset number of block sets;

[0022] Inputting each block in the block set into a pre-trained resolution enhancement model, and outputting a plurality of blocks of the second image after the resolution is enhanced;

[0023] Concatenate multiple blocks of the second image to generate the target image.

[0024] Optionally, generate a pre-trained image generation model by following these steps:

[0025] Collect model training images;

[0026] Input the model training image into the pre-trained vector quantization autoencoder for processing to obtain the image shape sequence of the model training;

[0027] Get the text description corresponding to the model training image;

[0028] Tokenize the text description corresponding to the model training image to generate the text shape sequence for model training;

[0029] The image glyph sequence trained by the model is concatenated with the text glyph sequence trained by the model, and a sequence of text-image pairs is generated after inserting preset characters into the concatenated sequence.

[0030] Continue to execute the step of collecting model training images until the number of sequences corresponding to the text-image pairs is greater than or equal to the preset number of sequences, and then generate model training samples;

[0031] The model is trained using model training samples to generate a pre-trained image generation model.

[0032] Optionally, perform model training using model training samples to generate a pre-trained image generation model, including:

[0033] Create an image generation model;

[0034] Input the model training samples one by one into the image generation model for training, and output the loss value of the model; wherein, self-supervised learning is used for training;

[0035] When the loss value reaches the minimum, the pre-trained image generation model is generated.

[0036] Optionally, when the loss value reaches a minimum, generate a pre-trained image generation model, including:

[0037] When the loss value does not reach the minimum, the image generation model is adjusted based on the loss value, and the step of inputting the model training samples one by one into the image generation model for training is executed.

[0038] Optionally, the method further includes:

[0039] When generating a pre-trained image generation model, obtain training samples for the scoring model and the resolution improvement model;

[0040] The pre-trained image generation model is trained according to the training samples of the scoring model to generate a pre-trained scoring model;

[0041] The pre-trained image generation model is trained according to the training samples of the resolution enhancement model to generate a pre-trained resolution enhancement model.

[0042] In a second aspect, an embodiment of the present application provides a device for generating text to an image, the device comprising:

[0043] A text shape sequence generation module is used to obtain a text description, tokenize the text description, and generate a text shape sequence;

[0044] A first image generation module, configured to generate at least one first image based on a text glyph sequence, a pre-trained image generation model, and a vector quantized autoencoder;

[0045] A probability value output module, configured to input each first image into a pre-trained scoring model to obtain a probability value for each first image;

[0046] A second image generating module is configured to screen first images having a probability value greater than a preset threshold based on the probability value of each first image, and generate at least one second image;

[0047] The target image generation module is used to increase the resolution of the second image based on the pre-trained resolution enhancement model to generate a target image; wherein,

[0048] The pre-trained image generation model is generated by concatenating the image shape sequence trained by the model and the text shape sequence trained by the model. The pre-trained scoring model and the pre-trained resolution enhancement model are generated by training on the pre-trained image generation model.

[0049] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions suitable for being loaded by a processor and executing the above-mentioned method steps.

[0050] In a fourth aspect, an embodiment of the present application provides a terminal, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0051] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0052] In an embodiment of the present application, a text-to-image generation device first obtains a text description, tokenizes the text description, and generates a text glyph sequence. Then, based on the text glyph sequence, a pre-trained image generation model, and a vector quantization autoencoder, at least one first image is generated. Each first image is then input into a pre-trained scoring model to obtain a probability value for each first image. Next, based on the probability value of each first image, first images with a probability value greater than a preset threshold are screened to generate at least one second image. Finally, the resolution of the second image is increased based on a pre-trained resolution enhancement model to generate a target image. The pre-trained image generation model is generated by splicing the image glyph sequence trained by the model with the text glyph sequence trained by the model. The pre-trained scoring model and the pre-trained resolution enhancement model are trained and generated on the pre-trained image generation model. Since the present application trains the model after splicing the image glyph sequence with the text glyph sequence, it can ensure that the generated image content is consistent with the semantics of the description text, greatly reducing the error between the two. In addition, by training the scoring model and the resolution enhancement model on the pre-trained image generation model, the resolution of the generated image can be effectively improved.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0055] Figure 1 This is a flowchart of a text-to-image generation method provided in an embodiment of the present application;

[0056] Figure 2 This is an example diagram of an actual image before and after pixel enhancement provided by an embodiment of the present application;

[0057] Figure 3 This is a schematic diagram of image segmentation in an application scenario provided by an embodiment of the present application;

[0058] Figure 4 This is a flow chart of a method for training an image generation model provided in an embodiment of the present application;

[0059] Figure 5 This is a flow chart of another method for training an image generation model provided in an embodiment of the present application;

[0060] Figure 6 This is a structural diagram of a text-to-image generation device provided in an embodiment of the present application;

[0061] Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.

[0063] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0064] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0065] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, in the description of the present invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0066] The present application provides a text-to-image generation method, device, storage medium, and terminal to solve the problems existing in the above-mentioned related technical problems. In the technical solution provided by the present application, since the present application trains the model after splicing the image glyph sequence and the text glyph sequence, it can ensure that the generated image content is consistent with the semantics of the descriptive text, greatly reducing the error between the two, and since the scoring model and the resolution improvement model are trained on the pre-trained image generation model, the resolution of the generated image can be effectively improved. The following is a detailed description using an exemplary embodiment.

[0067] The following will be combined with the Figure 1 -Attached Figure 5, the text-to-image generation method provided in the embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a text-to-image generation device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application. Among them, the text-to-image generation device in the embodiment of the present application can be a user terminal, including but not limited to: a personal computer, a tablet computer, a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing device connected to a wireless modem, etc. In different networks, the user terminal can be called different names, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), 5G network or terminal equipment in future evolution network, etc.

[0068] See Figure 1 , provides a flow chart of a method for generating text to image in an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:

[0069] S101, obtaining a text description, and tokenizing the text description to generate a text shape sequence;

[0070] A text description is a sentence consisting of several characters that can express the meaning, such as "a tiger playing football" or "the head of a lovely cat". Tokenization is the process of converting a sequence of characters into a sequence of tokens.

[0071] Generally, when tokenizing text descriptions, the SentencePiec algorithm or other text encoding algorithms, such as BPE, can be used.

[0072] In a possible implementation, a text description is first obtained, and then the text description is tokenized using the SentencePiec algorithm to generate a text shape sequence corresponding to the text description, that is, a token sequence of the text.

[0073] Specifically, the text description can be a text description received from a user, or it can be a text description automatically acquired based on the actual scenario. In one embodiment, the user enters the text description into the client, and the client receives the input text description. In another embodiment, when an illustration is required for an identified sentence in an article, the sentence identifier in the article is first identified, and then the sentence corresponding to the screenshot identifier is determined as the text description.

[0074] For example, the received or intercepted sentence is "a tiger playing football". At this time, "a tiger playing football" can be tokenized according to the SentencePiec algorithm to obtain a text shape sequence, which can also be called a tag sequence or a symbol sequence.

[0075] S102, generating at least one first image according to the text glyph sequence, a pre-trained image generation model, and a vector quantization autoencoder;

[0076] The pre-trained image generation model is generated by concatenating a model-trained image grapheme sequence with a model-trained text grapheme sequence. The pre-trained image generation model is a mathematical model that can generate corresponding images based on text descriptions. The vector quantized autoencoder is the component that performs encoding and decoding.

[0077] Typically, a vector quantized autoencoder consists of an encoder, a labeling dictionary, and a decoder.

[0078] In an embodiment of the present application, when generating a first image, the text shape symbol sequence is first input into a pre-trained image generation model, and a probability distribution is output. Then, the image tag corresponding to the maximum probability value is obtained from the probability distribution. After the image tag corresponding to the maximum probability value is spliced ​​into the text shape symbol sequence, a spliced ​​text shape symbol sequence is generated. Secondly, the number of image tags is counted. When the number of image tags is less than a preset first number, the spliced ​​text shape symbol sequence is continuously input into the pre-trained image generation model until the number of image tags is equal to the preset first number, an image tag set is generated, and the image tag set is input into the decoder of the pre-trained vector quantization autoencoder to generate the first image. Finally, the number of first images is counted, and when the number of first images is less than the preset second number, the step of inputting the text shape symbol sequence into the pre-trained image generation model is continued until at least one first image is generated when the number of first images is equal to the preset second number.

[0079] In one possible implementation, when generating the first image, the text glyphs are first input into a pre-trained image generation model to obtain a probability distribution over a token dictionary. The image token with the maximum probability value from the probability distribution is then concatenated after the initial text glyph sequence and fed into the trained network until 1024 image tokens are generated. Finally, the 1024 image tokens are input into the decoder of the vector quantized autoencoder to restore the image. The token dictionary refers to the set of all image tokens.

[0080] Furthermore, if it is desired to generate multiple images of a text glyph sequence, the above embodiment in a possible implementation may be used to generate multiple images in a loop multiple times. This process may be accelerated by batch processing.

[0081] It should be noted that the image marker with the maximum probability value obtained from the probability distribution is spliced ​​after the initial text shape sequence to ensure that the order of the 1024 image markers will not be disordered.

[0082] S103, inputting each first image into a pre-trained scoring model to obtain a probability value of each first image;

[0083] Among them, the scoring model is fine-tuned on the pre-trained image generation model. The fine-tuning process can be understood as training a scoring model again on the pre-trained image generation model. The training process of the scoring model is the regular training process of the existing model and will not be repeated here.

[0084] In an embodiment of the present application, each image generated in step S102 can be converted into text through the scoring model, and the probability value of each text can be calculated, and then the score is given according to the probability. The higher the probability of generating the corresponding text from the image, the higher the score of the entire image. Several images with higher scores can be selected according to the order of the scores.

[0085] In one possible implementation, when the first image obtained according to step S102 is one, the operation can be performed according to the actual scenario. For example, there is no need to input the scoring model for scoring, and step S105 can be directly executed. Alternatively, the scoring model can be input to determine the probability value of the image.

[0086] In another possible first manner, when there are multiple first images obtained according to step S102, the multiple images can only be input into the scoring model to perform scoring operations and subsequent operations.

[0087] S104, screening first images having a probability value greater than a preset threshold based on the probability value of each first image, and generating at least one second image;

[0088] In one possible implementation, according to step S103, probability values ​​of multiple first images can be obtained, and the probability values ​​of the multiple first images are arranged in descending order according to the probability values. Finally, images corresponding to probability values ​​greater than a preset threshold probability value are selected from the multiple probability values ​​after descending order, and one or more second images can be obtained.

[0089] S105, increasing the resolution of the second image based on a pre-trained resolution enhancement model to generate a target image;

[0090] Among them, the pre-trained resolution enhancement model is trained on the pre-trained image generation model.

[0091] In an embodiment of the present application, when generating a target image, the second image is first divided into a preset number of block sets, and then each block in the block set is input into a pre-trained resolution enhancement model, and multiple blocks of the second image with enhanced resolution are output. Finally, the multiple blocks of the second image are spliced ​​to generate the target image.

[0092] In a possible implementation, the second image obtained in step S104 has a low resolution, for example Figure 2 The image on the left of the arrow, at this time, you need to call the pre-trained resolution enhancement model, and then put the second image (i.e. Figure 2 (Image with arrow on the left) Follow Figure 3 The image is divided into 9 blocks in the same way as in

[15] . Each block is treated as a thumbnail and put into the pre-trained resolution enhancement model for resolution enhancement. Finally, the 9 blocks processed by the model are combined to obtain a higher-definition image. For example, Figure 2 As shown in the image to the right of the arrow.

[0093] It should be noted that there is some overlap between these blocks. The overlapping parts are based on the order of the blocks. During the generation process, if a previous block has been generated at that position, it is directly used as a fixed input to predict the next ungenerated shape. This method is called "sliding window" and maintains the continuity of each block at the edge.

[0094] In an embodiment of the present application, a text-to-image generation device first obtains a text description, tokenizes the text description, and generates a text shape sequence. Then, based on the text shape sequence, a pre-trained image generation model, and a vector quantization autoencoder, at least one first image is generated. Each first image is then input into a pre-trained scoring model to obtain a probability value for each first image. Next, based on the probability value of each first image, first images with a probability value greater than a preset threshold are screened to generate at least one second image. Finally, the resolution of the second image is increased based on a pre-trained resolution enhancement model to generate a target image. The pre-trained image generation model is generated by splicing the image shape sequence trained by the model with the text shape sequence trained by the model. The pre-trained scoring model and the pre-trained resolution enhancement model are generated by training on the pre-trained image generation model. Since the present application trains the model after splicing the image shape sequence with the text shape sequence, it can ensure that the generated image content is consistent with the semantics of the description text, greatly reducing the error between the two. Moreover, since the scoring model and the resolution enhancement model are trained on the pre-trained image generation model, the resolution of the generated image can be effectively improved.

[0095] See Figure 4, which is a flow chart of a training method for an image generation model provided in the embodiment of the present application. Figure 4 As shown in Figure 2, the training method of the image generation model includes the following steps:

[0096] S201, collecting model training images;

[0097] Usually, before training the image generation model, it is necessary to train a vector quantization autoencoder to obtain a pre-trained vector quantization autoencoder.

[0098] Specifically, the vector quantization autoencoder adopts a straight-through method during training, which can make the originally non-differentiable quantization operation differentiable, so that the entire model can be trained through backpropagation of a large number of images.

[0099] Furthermore, a pre-trained vector quantized autoencoder can convert model training images into image-to-image sequences.

[0100] In one possible implementation, when performing model training, it is first necessary to collect a model training image from the training samples.

[0101] S202, inputting the model training image into a pre-trained vector quantization autoencoder for processing to obtain an image shape sequence for model training;

[0102] S203, obtaining a text description corresponding to the model training image;

[0103] The text description corresponding to the model training image may be a text description marked in advance for the image, or a text description input by the user for the image.

[0104] In a possible implementation, a text description input by a client is received, or a text description carried on a query model training image is received.

[0105] S204, tokenizing the text description corresponding to the model training image to generate a text shape sequence for model training;

[0106] In a possible implementation, the text description in step S203 may be tokenized using the SentencePiece method to obtain a text shape sequence for model training.

[0107] S205, splicing the image shape symbol sequence trained by the model with the text shape symbol sequence trained by the model, and inserting preset characters into the spliced ​​sequence to generate a sequence of text-image pairs;

[0108] Typically, a text-image pair sequence can be obtained at this time. The model training process requires a large number of text-image pair sequences, so it is necessary to continue to perform steps S201 to S205 to obtain a set of text-image pair sequences.

[0109] S206, continue to execute the step of collecting model training images until the number of sequences corresponding to the text image pairs is greater than or equal to the preset number of sequences, and then generate model training samples;

[0110] The preset sequence is a number set in advance by the user, and model training samples can be generated after steps S201 to S205 are executed multiple times.

[0111] S207: Perform model training using model training samples to generate a pre-trained image generation model.

[0112] In one possible implementation, after obtaining the model training samples, an image generation model is first created, and then the model training samples are input into the image generation model one by one for training, and the loss value of the model is output; wherein, self-supervised learning is used during training, and finally, when the loss value reaches the minimum, a pre-trained image generation model is generated.

[0113] Alternatively, when the loss value has not reached the minimum, the image generation model is adjusted based on the loss value, and the step of inputting the model training samples into the image generation model one by one for training is performed until the loss value reaches the minimum to generate a pre-trained image generation model.

[0114] Specifically, when the image generation model is created, the model is based on the Transformer-based generative pre-training (GPT) model, that is, the generative pre-training (GPT) model inherits the functional characteristics of the Transformer.

[0115] Furthermore, when generating a pre-trained image generation model, first obtain the training samples of the scoring model and the training samples of the resolution enhancement model, then train the pre-trained image generation model according to the training samples of the scoring model to generate a pre-trained scoring model, and finally train the pre-trained image generation model according to the training samples of the resolution enhancement model to generate a pre-trained resolution enhancement model.

[0116] For example Figure 5 As shown, Figure 5This is a schematic diagram of image generation model training. First, the vector quantization autoencoder (Discrete Auto Encoder) is trained, which can be referred to as the image tagger. Then, the text is input into the text tagger and tokenized using the sentence pieces algorithm to obtain the text shape sequence trained by the model. Next, the image is input into the image tagger and processed using the pre-trained vector quantization autoencoder (Discrete Auto Encoder) to obtain the image shape sequence trained by the model. Then, the text shape sequence trained by the model and the image shape sequence trained by the model are spliced ​​to obtain a spliced ​​image-text sequence. All texts and images in the training sample are processed as above to obtain the training sample. Finally, the text-image sequence in the training sample is used to train the model, and finally the pre-trained image generation model is obtained.

[0117] In an embodiment of the present application, a text-to-image generation device first obtains a text description, tokenizes the text description, and generates a text shape sequence. Then, based on the text shape sequence, a pre-trained image generation model, and a vector quantization autoencoder, at least one first image is generated. Each first image is then input into a pre-trained scoring model to obtain a probability value for each first image. Next, based on the probability value of each first image, first images with a probability value greater than a preset threshold are screened to generate at least one second image. Finally, the resolution of the second image is increased based on a pre-trained resolution enhancement model to generate a target image. The pre-trained image generation model is generated by splicing the image shape sequence trained by the model with the text shape sequence trained by the model. The pre-trained scoring model and the pre-trained resolution enhancement model are generated by training on the pre-trained image generation model. Since the present application trains the model after splicing the image shape sequence with the text shape sequence, it can ensure that the generated image content is consistent with the semantics of the description text, greatly reducing the error between the two. Moreover, since the scoring model and the resolution enhancement model are trained on the pre-trained image generation model, the resolution of the generated image can be effectively improved.

[0118] The following are embodiments of the apparatus of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the apparatus embodiments of the present invention, please refer to the method embodiments of the present invention.

[0119] See Figure 6 , which shows a schematic diagram of the structure of a text-to-image generation device provided by an exemplary embodiment of the present invention. The text-to-image generation device can be implemented as all or part of a terminal through software, hardware, or a combination of both. The device 1 includes a text glyph sequence generation module 10, a first image generation module 20, a probability value output module 30, a second image generation module 40, and a target image generation module 50.

[0120] A text shape sequence generation module 10 is used to obtain a text description and generate a text shape sequence after tokenizing the text description;

[0121] A first image generation module 20 is configured to generate at least one first image based on a text glyph sequence, a pre-trained image generation model, and a vector quantized autoencoder;

[0122] a probability value output module 30, configured to input each first image into a pre-trained scoring model to obtain a probability value for each first image;

[0123] A second image generating module 40 is configured to screen first images having a probability value greater than a preset threshold based on the probability value of each first image, and generate at least one second image;

[0124] The target image generation module 50 is used to increase the resolution of the second image based on the pre-trained resolution enhancement model to generate a target image; wherein,

[0125] The pre-trained image generation model is generated by concatenating the image shape sequence trained by the model and the text shape sequence trained by the model. The pre-trained scoring model and the pre-trained resolution enhancement model are generated by training on the pre-trained image generation model.

[0126] It should be noted that the text-to-image generation device provided in the above embodiment, when executing the text-to-image generation method, only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the text-to-image generation device provided in the above embodiment and the text-to-image generation method embodiment are based on the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0127] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0128] In an embodiment of the present application, a text-to-image generation device first obtains a text description, tokenizes the text description, and generates a text shape sequence. Then, based on the text shape sequence, a pre-trained image generation model, and a vector quantization autoencoder, at least one first image is generated. Each first image is then input into a pre-trained scoring model to obtain a probability value for each first image. Next, based on the probability value of each first image, first images with a probability value greater than a preset threshold are screened to generate at least one second image. Finally, the resolution of the second image is increased based on a pre-trained resolution enhancement model to generate a target image. The pre-trained image generation model is generated by splicing the image shape sequence trained by the model with the text shape sequence trained by the model. The pre-trained scoring model and the pre-trained resolution enhancement model are generated by training on the pre-trained image generation model. Since the present application trains the model after splicing the image shape sequence with the text shape sequence, it can ensure that the generated image content is consistent with the semantics of the description text, greatly reducing the error between the two. Moreover, since the scoring model and the resolution enhancement model are trained on the pre-trained image generation model, the resolution of the generated image can be effectively improved.

[0129] The present invention also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implements the text-to-image generation method provided by each of the above method embodiments.

[0130] The present invention also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the text-to-image generation method of each of the above method embodiments.

[0131] See Figure 7 , provides a schematic diagram of the structure of a terminal according to an embodiment of the present application. Figure 7 As shown, the terminal 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0132] The communication bus 1002 is used to implement the connection and communication between these components.

[0133] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0134] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0135] The processor 1001 may include one or more processing cores. The processor 1001 utilizes various interfaces and circuits to connect various components within the electronic device 1000. It executes instructions, programs, code sets, or instruction sets stored in the memory 1005, and accesses data stored in the memory 1005 to perform various functions and process data within the electronic device 1000. Optionally, the processor 1001 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 1001 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the display; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 1001 but implemented as a separate chip.

[0136] Among them, the memory 1005 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Figure 7 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a text-to-image generation application.

[0137] exist Figure 7In the terminal 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain user input data; and the processor 1001 can be used to call the text-to-image generation application stored in the memory 1005 and specifically perform the following operations:

[0138] Get the text description, tokenize it and generate a text shape sequence;

[0139] generating at least one first image based on a sequence of text glyphs, a pre-trained image generation model, and a vector quantized autoencoder;

[0140] Input each first image into a pre-trained scoring model to obtain a probability value for each first image;

[0141] Based on the probability value of each first image, first images having a probability value greater than a preset threshold are screened to generate at least one second image;

[0142] The resolution of the second image is increased based on the pre-trained resolution enhancement model to generate a target image; wherein,

[0143] The pre-trained image generation model is generated by concatenating the image shape sequence trained by the model and the text shape sequence trained by the model. The pre-trained scoring model and the pre-trained resolution enhancement model are generated by training on the pre-trained image generation model.

[0144] In one embodiment, when the processor 1001 generates at least one first image according to the text glyph sequence, the pre-trained image generation model, and the vector quantization autoencoder, the processor 1001 specifically performs the following operations:

[0145] Input the text glyph sequence into a pre-trained image generation model and output a probability distribution;

[0146] Get the image label corresponding to the maximum probability value from the probability distribution;

[0147] After splicing the image tag corresponding to the maximum probability value to the text shape sequence, a spliced ​​text shape sequence is generated;

[0148] Count the number of image tags;

[0149] When the number of image tags is less than a preset first number, continue inputting the spliced ​​text shape sequence into the pre-trained image generation model until the number of image tags equals the preset first number, thereby generating an image tag set;

[0150] Inputting the image label set into the decoder of a pre-trained vector quantized autoencoder to generate a first image;

[0151] Counting the number of first images, and when the number of first images is less than a preset second number, continuing to input the text shape sequence into a pre-trained image generation model until at least one first image is generated when the number of first images is equal to the preset second number.

[0152] In one embodiment, when the processor 1001 increases the resolution of the second image based on the pre-trained resolution enhancement model to generate the target image, the processor 1001 specifically performs the following operations:

[0153] dividing the second image into a preset number of block sets;

[0154] Inputting each block in the block set into a pre-trained resolution enhancement model, and outputting a plurality of blocks of the second image after the resolution is enhanced;

[0155] The multiple blocks of the second image are concatenated to generate a target image.

[0156] In one embodiment, when generating a pre-trained image generation model, the processor 1001 specifically performs the following operations:

[0157] Collect model training images;

[0158] Input the model training image into the pre-trained vector quantization autoencoder for processing to obtain the image shape sequence of the model training;

[0159] Get the text description corresponding to the model training image;

[0160] Tokenize the text description corresponding to the model training image to generate the text shape sequence for model training;

[0161] The image glyph sequence trained by the model is concatenated with the text glyph sequence trained by the model, and a sequence of text-image pairs is generated after inserting preset characters into the concatenated sequence.

[0162] Continue to execute the step of collecting model training images until the number of sequences corresponding to the text-image pairs is greater than or equal to the preset number of sequences, and then generate model training samples;

[0163] The model is trained using model training samples to generate a pre-trained image generation model.

[0164] In one embodiment, when the processor 1001 performs model training using model training samples to generate a pre-trained image generation model, it specifically performs the following operations:

[0165] Create an image generation model;

[0166] Input the model training samples one by one into the image generation model for training, and output the loss value of the model; wherein, self-supervised learning is used for training;

[0167] When the loss value reaches the minimum, the pre-trained image generation model is generated.

[0168] In one embodiment, when the processor 1001 generates a pre-trained image generation model when the loss value reaches a minimum, the processor 1001 specifically performs the following operations:

[0169] When the loss value does not reach the minimum, the image generation model is adjusted based on the loss value, and the step of inputting the model training samples one by one into the image generation model for training is executed.

[0170] In an embodiment of the present application, a text-to-image generation device first obtains a text description, tokenizes the text description, and generates a text shape sequence. Then, based on the text shape sequence, a pre-trained image generation model, and a vector quantization autoencoder, at least one first image is generated. Each first image is then input into a pre-trained scoring model to obtain a probability value for each first image. Next, based on the probability value of each first image, first images with a probability value greater than a preset threshold are screened to generate at least one second image. Finally, the resolution of the second image is increased based on a pre-trained resolution enhancement model to generate a target image. The pre-trained image generation model is generated by splicing the image shape sequence trained by the model with the text shape sequence trained by the model. The pre-trained scoring model and the pre-trained resolution enhancement model are generated by training on the pre-trained image generation model. Since the present application trains the model after splicing the image shape sequence with the text shape sequence, it can ensure that the generated image content is consistent with the semantics of the description text, greatly reducing the error between the two. Moreover, since the scoring model and the resolution enhancement model are trained on the pre-trained image generation model, the resolution of the generated image can be effectively improved.

[0171] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program for converting text to image can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0172] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for generating text to image, characterized in that: The method comprises: Obtaining a text description, and tokenizing the text description to generate a text shape sequence; At least one first image is generated according to the text glyph sequence, the pre-trained image generation model and the vector quantization autoencoder; wherein, The method of generating at least one first image based on the text glyph sequence, a pre-trained image generation model, and a vector quantization autoencoder includes: inputting the text glyph sequence into the pre-trained image generation model to output a probability distribution; obtaining an image glyph corresponding to a maximum probability value from the probability distribution; splicing the image glyph corresponding to the maximum probability value to the text glyph sequence to generate a spliced ​​glyph sequence; counting the number of the image glyphs; when the number of the image glyphs is less than a preset first number, continuing to input the spliced ​​glyph sequence into the pre-trained image generation model until the number of the image glyphs equals the preset first number, thereby generating an image glyph set; and inputting the image glyph set into a decoder of a pre-trained vector quantization autoencoder to generate the first image. Inputting each of the first images into a pre-trained scoring model to obtain a probability value for each first image; the scoring model converts each generated image into text, calculates a probability value for each text, performs a score based on the probability value, and selects a plurality of images with high scores in order of their scores; Based on the probability value of each first image, the first images having a probability value greater than a preset threshold are screened to generate at least one second image; increasing the resolution of the second image based on a pre-trained resolution enhancement model to generate a target image; The pre-trained image generation model is generated by splicing an image shape sequence trained by the model with a text shape sequence trained by the model.

2. The method according to claim 1, characterized in that The method further comprises: Count the number of the first images, and when the number of the first images is less than a preset second number, continue to perform the step of inputting the text glyph sequence into a pre-trained image generation model until at least one first image is generated when the number of the first images is equal to the preset second number.

3. The method according to claim 1, characterized in that The step of increasing the resolution of the second image based on the pre-trained resolution enhancement model to generate a target image includes: dividing the second image into a preset number of block sets; Inputting each block in the block set into a pre-trained resolution enhancement model, and outputting a plurality of blocks of the second image after the resolution is enhanced; The plurality of blocks of the second image are stitched together to generate a target image.

4. The method according to claim 1, wherein Follow these steps to generate a pre-trained image generation model, including: Collect model training images; Inputting the model training image into a pre-trained vector quantization autoencoder for processing to obtain an image shape sequence for model training; Obtaining a text description corresponding to the model training image; Tokenizing the text description corresponding to the model training image to generate a text shape sequence for model training; Splicing the image glyph sequence trained by the model with the text glyph sequence trained by the model, and inserting preset characters into the spliced ​​sequences to generate a sequence of text-image pairs; Continuing to execute the step of collecting model training images until the number of sequences corresponding to the sequence of text-image pairs is greater than or equal to a preset number of sequences, generating model training samples; Model training is performed using the model training samples to generate a pre-trained image generation model.

5. The method according to claim 4, characterized in that The performing model training by using the model training samples to generate a pre-trained image generation model includes: Create an image generation model; Inputting the model training samples one by one into the image generation model for training, and outputting the loss value of the model; wherein the training is performed using self-supervised learning; When the loss value reaches the minimum, a pre-trained image generation model is generated.

6. The method according to claim 5, characterized in that When the loss value reaches the minimum, generating a pre-trained image generation model includes: When the loss value does not reach the minimum, the image generation model is adjusted based on the loss value, and the step of inputting the model training samples into the image generation model one by one for training is executed.

7. The method according to claim 5 or 6, characterized in that The method further comprises: When generating a pre-trained image generation model, obtain training samples for the scoring model and the resolution improvement model; Training the pre-trained image generation model according to the training samples of the scoring model to generate a pre-trained scoring model; The pre-trained image generation model is trained according to the training samples of the resolution enhancement model to generate a pre-trained resolution enhancement model.

8. A device for generating text to image using the method according to any one of claims 1 to 7, characterized in that: The device comprises: A text shape sequence generation module is used to obtain a text description and generate a text shape sequence after tokenizing the text description; A first image generation module, configured to generate at least one first image based on the text glyph sequence, a pre-trained image generation model, and a vector quantized autoencoder; a probability value output module, configured to input each of the first images into a pre-trained scoring model to obtain a probability value for each of the first images; A second image generating module is configured to screen the first images having a probability value greater than a preset threshold based on the probability value of each first image, and generate at least one second image; A target image generation module is used to increase the resolution of the second image based on a pre-trained resolution enhancement model to generate a target image; wherein, The pre-trained image generation model is generated by training after splicing the model-trained image shape sequence and the model-trained text shape sequence, and the pre-trained scoring model and the pre-trained resolution enhancement model are generated by training on the pre-trained image generation model.

9. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 7.

10. A terminal, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method according to any one of claims 1 to 7.

Images