Text-to-image generation method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of artificial intelligence, and relates to a text-to-image generation method, which comprises the following steps: acquiring a randomly sampled noise vector and an original text of an image; inputting the noise vector and the original text of the image into a pre-training diffusion model to generate a rough image and a fine image; calculating the difference between the rough image and the fine image, and taking the difference as quality guidance; calculating the similarity of the text and the image and taking the similarity as semantic guidance; performing learning prompt on the input text according to quality guidance and semantic guidance; connecting an input text with the learning prompt, and sending the input text and the learning prompt to a text encoder to obtain final text embedding; and sending the final text embedded and randomly sampled noise vector to a pre-trained diffusion model to generate a final image. The invention further provides a text-to-image generation device, computer equipment and a storage medium. According to the invention, the accuracy of text-to-image synthesis is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text-to-image generation method, apparatus, computer device, and storage medium. Background Art

[0002] In recent years, text-to-image synthesis based on diffusion models has shown remarkable performance in generating high-quality images. Although it performs well for simple texts, diffusion models face the following challenges when faced with complex texts containing multiple objects or spatial relationships.

[0003] First, it is difficult to capture complex relationships: Diffusion models have difficulty capturing complex relationships between objects in text, resulting in inaccurate or incomplete image synthesis.

[0004] Second, the text encoder capacity is limited: The performance of the diffusion model is limited by the capacity of the text encoder and may not be able to effectively capture the nuances of complex text.

[0005] Third, it is difficult to handle long texts: Diffusion models may have difficulty processing long texts, resulting in incomplete or inaccurate image synthesis.

[0006] In order to obtain the desired image, some researchers try to decompose the generation process into multiple steps or combine language parsers to improve accuracy; another method is to design text prompts by splicing special phrases or carefully narrating the text to improve text-to-image matching. However, these methods require a lot of manual adjustment of the text description, that is, narrating the text or adding some words. Summary of the invention

[0007] The purpose of the embodiments of the present application is to propose a text-to-image generation method, apparatus, computer device and storage medium to solve the technical problems in the prior art that it is difficult to capture complex relationships, the capacity of the text encoder is limited, and it is difficult to process long texts.

[0008] In order to solve the above technical problems, the present application embodiment provides a text-to-image generation method, which adopts the following technical solution:

[0009] A method for generating text to image comprises the following steps:

[0010] Get the randomly sampled noise vector and the original text of the image;

[0011] Inputting the noise vector and the original text of the image into a pre-trained diffusion model to generate a rough image and a detailed image;

[0012] Calculating the difference between the coarse image and the fine image, and using the difference as a quality guide;

[0013] Compute text and image similarity and use it as semantic guidance;

[0014] Providing learning prompts to the input text according to the quality guidance and semantic guidance;

[0015] Concatenate the input text with the learning hint and send it to the text encoder to get the final text embedding;

[0016] The final text embedding and the randomly sampled noise vector are sent to the pre-trained diffusion model to generate the final image.

[0017] Furthermore, the training steps of the pre-trained diffusion model include:

[0018] Clean, denoise and normalize each training sample;

[0019] Extract a series of noise data pairs from the processed training samples;

[0020] The noise data is trained using a linear regression algorithm to obtain a pre-trained diffusion model.

[0021] Furthermore, the step of inputting the noise vector and the original text of the image into the pre-trained diffusion model to generate a rough image and a detailed image specifically includes:

[0022] Inputting the noise vector and the original text of the image into a pre-trained diffusion model;

[0023] The pre-trained diffusion model performs a forward diffusion process on the original text of the image according to the noise vector to generate a rough image;

[0024] The pre-trained diffusion model performs a reverse generation process on the original text of the image according to the noise vector to generate a detailed image.

[0025] Furthermore, the step of calculating the difference between the rough image and the fine image and using the difference as a quality guide specifically includes:

[0026] Calculating pixel-level differences, structural similarity indexes, or mean square errors between the coarse image and the detailed image;

[0027] The pixel-level difference, structural similarity index or mean square error is set as a quality guide.

[0028] Furthermore, the step of calculating the similarity between text and image and using it as semantic guidance specifically includes:

[0029] Using deep neural networks to encode text and images into vector representations;

[0030] Calculating the cosine similarity and the Euclidean distance between the vectors, and marking the cosine similarity and the Euclidean distance as the similarity between the text and the image;

[0031] The similarity between the text and the image is used as a semantic guide.

[0032] Furthermore, after the step of providing learning prompts for the input text according to the quality guidance and semantic guidance, the method further includes:

[0033] The learning hints are trained using the Adam optimizer with a specific learning rate.

[0034] Furthermore, after the step of sending the final text embedding and the randomly sampled noise vector to the pre-trained diffusion model to generate the final image, the step further includes:

[0035] The words in the input text whose similarity with the final image is less than a set threshold are obtained, the words are shielded, and the input text is subjected to mask processing.

[0036] In order to solve the above technical problems, the embodiment of the present application further provides a text-to-image generation device, which adopts the following technical solution:

[0037] A text-to-image generation device, comprising:

[0038] An acquisition module is used to obtain the randomly sampled noise vector and the original text of the image;

[0039] An input module, used for inputting the noise vector and the original text of the image into the pre-trained diffusion model to generate a rough image and a detailed image;

[0040] A first calculation module, used for calculating the difference between the rough image and the fine image, and using the difference as a quality guide;

[0041] The second calculation module is used to calculate the text-image similarity and serve as a semantic guide;

[0042] A learning prompt module, used for providing learning prompts to the input text according to the quality guidance and semantic guidance;

[0043] A concatenation module, configured to concatenate the input text with the learning prompt and send the concatenated text to a text encoder to obtain a final text embedding;

[0044] A generation module is used to send the final text embedding and the randomly sampled noise vector to a pre-trained diffusion model to generate a final image.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0046] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned text-to-image generation method when executing the computer-readable instructions.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0048] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the above-mentioned text-to-image generation method.

[0049] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the present application effectively learns prompts to improve the match between input text and generated images by utilizing the quality guidance and semantic guidance of the pre-trained diffusion model, thereby effectively improving the accuracy of text-to-image synthesis, especially the accuracy of generating high-quality images based on complex text. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0052] Figure 2 A flowchart of an embodiment of a text-to-image generation method according to the present application;

[0053] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S202;

[0054] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S203;

[0055] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S204;

[0056] Figure 6 is a structural schematic diagram of an embodiment of a text-to-image generation device according to the present application;

[0057] Figure 7 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0059] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0061] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0062] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0063] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Perts Group Audio Layer III), MP4 (Moving Picture Perts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0064] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0065] It should be noted that the text-to-image generation method provided in the embodiment of the present application is generally executed by a server, and accordingly, the text-to-image generation device is generally arranged in a terminal device.

[0066] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0067] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for generating text to an image according to the present application. The method for generating text to an image comprises the following steps:

[0068] Step S201, obtaining a randomly sampled noise vector and the original text of the image.

[0069] In this embodiment, the text-to-image generation method is executed on an electronic device (eg Figure 1 The server shown in the figure) can be connected by wired connection or wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MA text-to-image generation connection, Zigbee connection, UWB (Ultra Wi-Fi Band) connection, and other wireless connection methods currently known or to be developed in the future.

[0070] Specifically, the noise vector is random noise that conforms to the normal distribution, and the original text of the image is also randomly acquired.

[0071] Step S202: input the noise vector and the original text of the image into a pre-trained diffusion model to generate a rough image and a detailed image.

[0072] In this embodiment, the training step of the diffusion model includes:

[0073] First, each training sample is cleaned, denoised and normalized.

[0074] Then, a series of noise data pairs are extracted from the processed training samples.

[0075] Finally, a linear regression algorithm (or logistic regression, decision tree, etc.) is used to train the noise data to obtain a pre-trained diffusion model.

[0076] The training samples can be divided into a training set and a test set. The diffusion model is trained by the training set and tested by the test set.

[0077] This step generates a rough image and a detailed image, so as to facilitate the subsequent calculation of the difference between the two and form a quality guide.

[0078] like Figure 3 As shown, step S202 inputs the noise vector and the original text of the image into the pre-trained diffusion model to generate a rough image and a detailed image, specifically including:

[0079] Step S2021: input the noise vector and the original text of the image into a pre-trained diffusion model.

[0080] Step S2022: The pre-trained diffusion model performs a forward diffusion process on the original text of the image according to the noise vector. The forward diffusion process gradually adds the noise vector to the image to generate a rough image and obtain pure noise.

[0081] Step S2023: The pre-trained diffusion model performs a reverse generation process on the original text of the image according to the noise vector. The reverse generation process is to gradually denoise from pure noise to generate a detailed image.

[0082] Step S203: Calculate the difference between the rough image and the detailed image, and use the difference as a quality guide.

[0083] The quality guidance generated by this embodiment helps with content matching and artifact reduction, enabling more realistic and accurate text-to-image synthesis.

[0084] Specifically, Figure 4 As shown, step S203 specifically includes:

[0085] Step S2031: Calculate the pixel-level difference, structural similarity index or mean square error between the coarse image and the fine image.

[0086] In this embodiment, the pixel-level difference is to calculate the difference between each pixel of the two pictures, that is, to compare the RGB values ​​or grayscale values ​​between the two pictures.

[0087] The structural similarity index takes into account both pixel differences and perceptual differences, and the specific implementation steps are: first, the two images are divided into blocks of the same size; then, for each small block, its brightness (calculated using the mean and variance of the brightness), contrast (calculated using the mean and variance of the contrast) and structural similarity (calculated using the covariance and variance between the two images) are calculated; finally, the similarity of each small block is weighted averaged to obtain the structural similarity of the entire image.

[0088] The mean square error is calculated by calculating the difference between each pixel (RGB value or grayscale value) of the two images and summing the squares of these differences.

[0089] Step S2032: setting the pixel level difference, structural similarity index or mean square error as a quality guide.

[0090] Step S204, calculating the text-image similarity and using it as a semantic guide.

[0091] In this embodiment, semantic guidance can prompt the narration of the current text, enhancing the learning of less focused content, thereby achieving more accurate text-image synthesis.

[0092] Specifically, Figure 5 As shown, step S204 specifically includes:

[0093] Step S2041: Encode text and images into vector representations using a deep neural network.

[0094] Step S2042: After calculating the cosine similarity and the Euclidean distance between the vectors, the cosine similarity and the Euclidean distance are marked as the similarity between the text and the image.

[0095] Step S2043: Use the similarity between the text and the image as a semantic guide.

[0096] Step S205: providing learning prompts for the input text according to the quality guidance and semantic guidance.

[0097] In this embodiment, using quality guidance and semantic guidance, hints can be effectively learned to improve the match between input text and generated images, wherein quality guidance helps content matching and artifact reduction, leading to more realistic and accurate text-to-image synthesis, and semantic guidance can prompt the narration of the current text, enhancing the learning of less focused content and solving the image synthesis problem under complex text conditions.

[0098] This embodiment uses quality guidance and semantic guidance to learn prompts for input text and adjust the text description without manual work, so that prompts can be effectively learned to improve the match between the input text and the generated image.

[0099] Step S206, concatenating the input text with the learning prompt and sending it to a text encoder to obtain a final text embedding.

[0100] In this embodiment, the text encoder divides the input text into a series of tokens, adds markers to the tokens (for example, noun phrases, markers that mark the beginning and end of sentences, or specific grammatical structures), and then connects the learning hints to the markers.

[0101] Specifically, in this embodiment, the input text is connected with the learning prompt obtained in step S205 and then input into the text encoder to obtain the final text embedding for the next step.

[0102] Step S207, sending the final text embedding and the randomly sampled noise vector to the pre-trained diffusion model to generate a final image.

[0103] This embodiment can refine the generated image according to the input text and noise, which helps to improve the accuracy of text-to-image synthesis.

[0104] This application effectively improves the accuracy of text-to-image synthesis, especially the accuracy of generating high-quality images based on complex text, by leveraging the quality guidance and semantic guidance of a pre-trained diffusion model to effectively learn hints to improve the match between input text and generated images.

[0105] In some optional implementations of this embodiment, after step S205 and before step S206, the electronic device may further perform the following steps:

[0106] The learning hint uses a specific learning rate of 10 -2 The Adam optimizer is used for training.

[0107] In this process, the Adam optimizer optimizes the learning prompts by minimizing the loss function, aiming to improve the match between the input text and the generated image and improve the accuracy of text-to-image synthesis.

[0108] In some optional implementations of this embodiment, after step S207, the electronic device may further perform the following steps:

[0109] Get the words in the input text whose similarity with the final image is less than the set threshold.

[0110] The words are shielded and the input text is subjected to mask processing.

[0111] In this embodiment, the method for calculating the similarity between the input text and the final image is referred to step S204.

[0112] In the present application, words in the input text whose similarity with the final image is less than a set threshold are shielded or masked to enhance prompt learning, thereby effectively focusing on previously neglected content.

[0113] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0114] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0115] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0116] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0117] Further references Figure 6 , as a response to the above Figure 2 The present application provides an embodiment of a text-to-image generation device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0118] like Figure 6 As shown, the text-to-image generation device 600 described in this embodiment includes: an acquisition module 601, an input module 602, a first calculation module 603, a second calculation module 604, a learning prompt module 605, a connection module 606, and a generation module 607. Among them:

[0119] The acquisition module 601 is used to acquire the randomly sampled noise vector and the original text of the image.

[0120] The input module 602 is used to input the noise vector and the original text of the image into the pre-trained diffusion model to generate a rough image and a detailed image.

[0121] The first calculation module 603 is used to calculate the difference between the rough image and the fine image, and use the difference as a quality guide.

[0122] The second calculation module 604 is used to calculate the similarity between text and image and serve as a semantic guide.

[0123] The learning prompt module 605 is used to provide learning prompts for the input text according to the quality guidance and semantic guidance.

[0124] The connection module 606 is used to connect the input text with the learning prompt and send it to the text encoder to obtain the final text embedding.

[0125] The generation module 607 is used to send the final text embedding and the randomly sampled noise vector to the pre-trained diffusion model to generate a final image.

[0126] This application effectively improves the accuracy of text-to-image synthesis, especially the accuracy of generating high-quality images based on complex text, by leveraging the quality guidance and semantic guidance of a pre-trained diffusion model to effectively learn hints to improve the match between input text and generated images.

[0127] To solve the above technical problems, the present application also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0128] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 7 with components 71-73, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0129] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0130] The memory 71 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D text to image generation memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 71 can also be an external storage device of the computer device 7, such as a plug-in hard disk equipped on the computer device 7, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 71 can also include both the internal storage unit of the computer device 7 and its external storage device. In this embodiment, the memory 71 is generally used to store the operating system and various application software installed on the computer device 7, such as computer-readable instructions of the text to image generation method, etc. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or are to be output.

[0131] The processor 72 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 72 is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to run computer-readable instructions stored in the memory 71 or process data, such as computer-readable instructions for running the text-to-image generation method.

[0132] The network interface 73 may include a wireless network interface or a wired network interface. The network interface 73 is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0133] This application effectively improves the accuracy of text-to-image synthesis, especially the accuracy of generating high-quality images based on complex text, by leveraging the quality guidance and semantic guidance of a pre-trained diffusion model to effectively learn hints to improve the match between input text and generated images.

[0134] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the text-to-image generation method as described above.

[0135] This application effectively improves the accuracy of text-to-image synthesis, especially the accuracy of generating high-quality images based on complex text, by leveraging the quality guidance and semantic guidance of a pre-trained diffusion model to effectively learn hints to improve the match between input text and generated images.

[0136] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0138] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A text-to-image generation method, characterized in that: The steps include: Get the randomly sampled noise vector and the original text of the image; Inputting the noise vector and the original text of the image into a pre-trained diffusion model to generate a rough image and a detailed image; Calculating the difference between the coarse image and the fine image, and using the difference as a quality guide; Compute text and image similarity and use it as semantic guidance; Providing learning prompts to the input text according to the quality guidance and semantic guidance; Concatenate the input text with the learning prompt and send it to the text encoder to get the final text embedding; The final text embedding and the randomly sampled noise vector are sent to the pre-trained diffusion model to generate the final image.

2. The text-to-image generation method according to claim 1, characterized in that: The training steps of the pre-trained diffusion model include: Clean, denoise and normalize each training sample; Extract a series of noise data pairs from the processed training samples; The noise data is trained using a linear regression algorithm to obtain a pre-trained diffusion model.

3. The text-to-image generation method according to claim 1, characterized in that: The step of inputting the noise vector and the original text of the image into the pre-trained diffusion model to generate a rough image and a detailed image specifically includes: Inputting the noise vector and the original text of the image into a pre-trained diffusion model; The pre-trained diffusion model performs a forward diffusion process on the original text of the image according to the noise vector to generate a rough image; The pre-trained diffusion model performs a reverse generation process on the original text of the image according to the noise vector to generate a detailed image.

4. The text-to-image generation method according to claim 1, characterized in that: The step of calculating the difference between the rough image and the fine image and using the difference as a quality guide specifically includes: Calculating pixel-level differences, structural similarity indexes, or mean square errors between the coarse image and the detailed image; The pixel-level difference, structural similarity index or mean square error is set as a quality guide.

5. The text-to-image generation method according to claim 1, characterized in that: The step of calculating the similarity between text and image and using it as semantic guidance specifically includes: Using deep neural networks to encode text and images into vector representations; Calculating the cosine similarity and the Euclidean distance between the vectors, and marking the cosine similarity and the Euclidean distance as the similarity between the text and the image; The similarity between the text and the image is used as a semantic guide.

6. The text-to-image generation method according to claim 1, characterized in that: After the step of providing learning prompts to the input text according to the quality guidance and semantic guidance, the method further includes: The learning hints are trained using the Adam optimizer with a specific learning rate.

7. The text-to-image generation method according to claim 1, characterized in that: After the step of sending the final text embedding and the randomly sampled noise vector to the pre-trained diffusion model to generate the final image, the step further includes: Get words in the input text whose similarity with the final image is less than a set threshold; The words are shielded and the input text is subjected to mask processing.

8. A text-to-image generation device, characterized in that: include: An acquisition module is used to obtain the randomly sampled noise vector and the original text of the image; An input module, used for inputting the noise vector and the original text of the image into the pre-trained diffusion model to generate a rough image and a detailed image; A first calculation module, used for calculating the difference between the rough image and the fine image, and using the difference as a quality guide; The second calculation module is used to calculate the text-image similarity and serve as a semantic guide; A learning prompt module, used for providing learning prompts to the input text according to the quality guidance and semantic guidance; A concatenation module, configured to concatenate the input text with the learning prompt and send the concatenated text to a text encoder to obtain a final text embedding; A generation module is used to send the final text embedding and the randomly sampled noise vector to a pre-trained diffusion model to generate a final image.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the text-to-image generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the text-to-image generation method according to any one of claims 1 to 7 are implemented.

Images