Text-to-image generation model optimization method and device, equipment and storage medium

Abstract

Embodiments of the present disclosure provide a text-to-image generation model optimization method, device, equipment and storage medium. Applied to the technical field of deep learning, the method comprises: obtaining a reference image group corresponding to a theme, wherein each reference image in the reference image group carries a source text prompt containing a theme identifier and a target text prompt, and the target text prompt is an expanded description of the context of the theme in the reference image in the source text prompt; inputting the reference image group into a pre-trained text-to-image generation model, optimizing the pre-trained text-to-image generation model, and obtaining an optimized text-to-image generation model, wherein the optimized text-to-image generation model is used to generate a target image according to the description information of the context of the theme. In this way, the technical problem that the image generation model cannot realize theme re-contextualization can be solved.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning technology, and in particular to a method, apparatus, device, and storage medium for optimizing a text-to-image generation model. Background Technology

[0002] Text-to-image synthesis aims to generate clear, photorealistic images with a high semantic relevance to the corresponding text descriptions. This task is particularly challenging because textual descriptions are often ambiguous. While current models can generate images based on textual descriptions, the inherent ambiguity and uncertainty of semantics mean that the generated images can vary significantly across different contexts, failing to achieve topic recontextualization.

[0003] There is currently no effective solution to the technical problem that image generation models cannot achieve topic recontextualization. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and storage medium for optimizing a text-to-image generation model.

[0005] According to a first aspect of this disclosure, a method for optimizing a text-to-image generation model is provided. The method includes: acquiring a set of reference images corresponding to a topic, wherein each reference image in the set carries a source text prompt containing a topic identifier and a target text prompt, the target text prompt being an extended description of the context of the topic in the reference image from the source text prompt; inputting the set of reference images into a pre-trained text-to-image generation model, optimizing the pre-trained text-to-image generation model to obtain an optimized text-to-image generation model, wherein the optimized text-to-image generation model is used to generate a target image based on the descriptive information of the topic's context.

[0006] In addition to the aspects described above and any possible implementation, a further implementation is provided, in which a set of reference images is input into a pre-trained text-to-image generation model, and the pre-trained text-to-image generation model is optimized to obtain an optimized text-to-image generation model, including: inputting the set of reference images into the pre-trained text-to-image generation model; and optimizing the parameters of the pre-trained text-to-image generation model through backpropagation and gradient descent until the loss function converges.

[0007] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the loss function is determined according to the following formula:

[0008]

[0009] in, For hyperparameters It is a positive real number. This indicates the embedding of the source text suggestion. Embedding of target text prompts, For the number of embeddings, and For the parameter matrix, For the key vector in the cross-attention layer, This is the value vector in the cross-attention layer.

[0010] In addition to the aspects described above and any possible implementation, a further implementation is provided in which the pre-trained text-to-image generation model is trained through the following steps: initializing a generative adversarial network including a generator and a discriminator; training the generator and discriminator alternately based on a training dataset until the adversarial loss function converges, wherein the training dataset includes real images and descriptive information corresponding to the real images, the descriptive information being used to describe the images to be generated; and using the generator as a pre-trained text-to-image generation model.

[0011] In addition to the aspects and any possible implementations described above, a further implementation is provided that alternately trains the generator and discriminator based on the training dataset until the adversarial loss function converges, including:

[0012] The generator is fixed, and the discriminator is trained, including: generating fake images in the generator using random noise and descriptive information; inputting the fake images and real images into the discriminator and calculating the discriminator loss function; using the discriminator loss function to perform backpropagation to obtain the gradient and updating the parameters in the discriminator using the gradient descent method; repeating the above steps until the discriminator loss function is minimized.

[0013] as well as,

[0014] The discriminator is fixed, and the generator is trained, including: generating fake images in the generator using random noise and descriptive information; inputting the fake images and real images into the discriminator and calculating the generator loss function; using the generator loss function to perform backpropagation to obtain the gradient and updating the parameters in the generator using the gradient descent method; repeating the above steps until the generator loss function is minimized.

[0015] In addition to the aspects described above and any possible implementations, a further implementation is provided whereby the generator includes a text encoding layer, an image encoding layer, a fusion layer, and a deconvolutional neural network. The text encoding layer converts the text in the input descriptive information into text vectors; the image encoding layer converts the image in the input descriptive information into image vectors; the fusion layer fuses the text vectors and image vectors in a cross-attention module and then inputs them into an adapter layer to obtain fused features; and the deconvolutional neural network generates an image based on the input fused features. The adapter layer is a special neural network layer used for information exchange and feature extraction between different network modules. Its function is similar to an intermediary interface, converting different feature data into a form suitable for a specific module to better fuse and utilize these features.

[0016] In addition to the aspects described above and any possible implementation, a further implementation is provided in which the generator further includes a context encoding layer, which is used to encode the context information in the input description information to obtain a context vector, and a fusion layer is used to fuse the text vector, image vector and context vector in the cross-attention module and then input them into the Adapter layer to obtain fused features.

[0017] According to a second aspect of this disclosure, a text-to-image generative model optimization apparatus is provided. The apparatus includes:

[0018] The acquisition module is used to acquire a group of reference images corresponding to the topic. Each reference image in the group of reference images carries a source text hint containing a topic identifier and a target text hint. The target text hint is an extended description of the context of the topic in the reference image in the source text hint.

[0019] The optimization module is used to input the reference image set into the pre-trained text-to-image generation model and optimize the pre-trained text-to-image generation model to obtain the optimized text-to-image generation model. The optimized text-to-image generation model is used to generate the target image based on the descriptive information of the topic's context.

[0020] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0021] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods as described in the first and / or second aspects of this disclosure.

[0022] This disclosure optimizes a pre-trained text-to-image generation model by using a set of reference images containing source text prompts, topic identifiers, and target text prompts. This enables the optimized model to generate images based on the descriptive information of the topic's context. It can also synthesize images of the topic corresponding to the reference image set under various scenes, poses, views, and lighting conditions that do not appear in the reference images, thereby achieving topic recontextualization.

[0023] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0024] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0025] Figure 1 A flowchart illustrating a text-to-image generation model optimization method according to an embodiment of the present disclosure is shown;

[0026] Figure 2 A block diagram of a text-to-image generation model optimization apparatus according to an embodiment of the present disclosure is shown;

[0027] Figure 3 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0029] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0030] In this disclosure, a pre-trained text-to-image generation model is optimized by using a set of reference images containing source text prompts, topic identifiers, and target text prompts. This enables the optimized model to generate images based on the descriptive information of the topic's context, thereby achieving topic recontextualization.

[0031] Figure 1 A flowchart of a text-to-image generation model optimization method 100 according to an embodiment of the present disclosure is shown. Figure 1 As shown, method 100 includes:

[0032] Step S110: Obtain a reference image group corresponding to the topic, wherein each reference image in the reference image group carries a source text hint containing a topic identifier and a target text hint, and the target text hint is an extended description of the context of the topic in the reference image in the source text hint;

[0033] Step S120: Input the reference image group into the pre-trained text-to-image generation model, optimize the pre-trained text-to-image generation model to obtain the optimized text-to-image generation model, wherein the optimized text-to-image generation model is used to generate target images based on the descriptive information of the context in which the topic is located.

[0034] The reference images in each reference image group share the same theme, which can represent any object in the scene from which the image is to be generated, including but not limited to a specific person, a specific animal, or a specific item.

[0035] Source prompts are the raw text used to describe the subject image, guiding and constraining the image generation process. Target prompts, on the other hand, are texts that guide the model to generate specific details about the subject image, such as features like color, shape, expression, action, scene, and style. Typically, source prompts are short and contain core image information, such as "a sheepdog" or "a pair of sunglasses." Target prompts are expanded descriptions of the source prompts, guiding the model to generate specific details of the image, such as "a black and white sheepdog" or "a pair of sunglasses on a dining table."

[0036] The context of the topic describes the visual features in the expected generated topic image, acting as a target text cue. This context can include the action and expression of the object corresponding to the topic, its color and shape, the environment and scene in which the object is located, or the image style. For example, if the topic is "sheepdog," the descriptive information of the topic's context could be "sheepdog running on the grassland," "smiling sheepdog," or "sheepdog oil painting," etc. The optimized model can then generate corresponding target images based on these topic contexts.

[0037] The "descriptive information of the topic's context" used as input to the text-to-image generation model includes, but is not limited to, multimodal information such as text, image, audio, and video information, as long as it can describe the visual features in the expected generated topic image.

[0038] Pre-trained text-to-image models may not be able to generate accurate and realistic images due to limitations in data sources or data annotation granularity. By using a set of reference images to optimize for a given topic, the optimized model can generate accurate and realistic images of that topic.

[0039] For example, a pre-trained text-to-image generation model can output images containing canines based on semantics such as "dog" or "dog" in the input description, but it cannot distinguish between different dog breeds very well. When the input description contains "sheepdog," the dog in the generated image may not actually be a sheepdog. Therefore, the model needs to be optimized using a set of reference images. A set of reference images corresponding to the theme "sheepdog" includes multiple reference images, each representing the theme "sheepdog," with different visual features such as the sheepdog's actions and the surrounding scene. By using this set of reference images for model optimization, the model can expand the range of themes it can generate, meaning that it can accurately synthesize images with the theme "sheepdog" when the input description contains "sheepdog."

[0040] Optionally, the source text hints and target text hints may also include a topic class name. The topic identifier is a unique identifier associated with the topic, while the topic class name indicates the category to which the topic belongs. For example, if the topic is "sheepdog" and the topic class name is "dog", it is convenient to associate the prior knowledge of the topic class name in the model with the topic.

[0041] In some embodiments, step S120 involves inputting a set of reference images into a pre-trained text-to-image generation model, optimizing the pre-trained text-to-image generation model to obtain an optimized text-to-image generation model, including:

[0042] The reference image set is input into the pre-trained text-to-image generation model;

[0043] The parameters of the pre-trained text-to-image generation model are optimized using backpropagation and gradient descent until the loss function converges.

[0044] The gradient is obtained through backpropagation using the loss function, and the model parameters are adjusted using gradient descent. Optionally, the termination condition for model optimization can be the convergence of the loss function, or it can be set by setting a preset number of parameter adjustments as the termination condition.

[0045] According to embodiments of this disclosure, the gradient is obtained through backpropagation using a loss function, and the parameters of the model are adjusted using gradient descent, so that the model gradually learns to better capture the target text prompts, thereby improving the accuracy of the text-to-image generation model and enhancing the quality and visual relevance of the generated images.

[0046] In some embodiments, the loss function is determined according to the following formula:

[0047]

[0048] in, For hyperparameters It is a positive real number. Represents the L2 norm, This indicates the embedding of the source text suggestion. This indicates the embedding of the target text prompt. For the number of embeddings, and For the parameter matrix, For the key vector in the cross-attention layer, This is the value vector in the cross-attention layer.

[0049] For each source tag Source embedding Identify the target embedding corresponding to the same label and label it as In each cross-attention layer of the model, the key vector of the target cue is computed. value vector .

[0050] The first term in the loss function represents the distance between the source and target text prompts in terms of keys. This term adjusts the source embedding to be closer to the keys corresponding to the target embedding. A closer key to the target embedding helps the model better locate objects in the image. The second term represents the distance between the source and target text prompts in terms of values. This term adjusts the source embedding to be closer to the values ​​corresponding to the target embedding, helping the model better identify and classify objects in the image. The third and fourth terms in the loss function are similar to the first and second terms, but they are specific to the target text prompt. These two terms ensure consistency between the source and target text prompts. By comparing each embedding in the source and target text prompts, the loss function adjusts the model to better capture the relationship and similarity between the two.

[0051] According to embodiments of this disclosure, a loss function is used to narrow the distance between source and target text prompts, making their generation behaviors similar while maintaining proximity to the original projection matrix, thereby improving model performance. Specifically, the similarity between source and target text prompt embeddings is measured by calculating the distance difference between them. Furthermore, the weighting of keys and values ​​is controlled using a key-value parameter matrix, allowing for more accurate calculation of the distance difference. The hyperparameter λ is used to balance the importance of keys and values. The overall purpose of the loss function is to minimize the distance difference between embeddings, thereby improving model accuracy.

[0052] In some embodiments, the pre-trained text-to-image generation model is trained through the following steps:

[0053] Initialize the generative adversarial network, which includes a generator and a discriminator;

[0054] The generator and discriminator are trained alternately on the training dataset until the adversarial loss function converges. The training dataset includes real images and corresponding descriptive information, which describes the images to be generated.

[0055] The generator is used as a pre-trained text-to-image generation model.

[0056] The adversarial loss function is an important component of generative adversarial networks (GANs). It measures the adversarial relationship between the generator and the discriminator and improves the generator's performance by optimizing its value.

[0057] According to embodiments of this disclosure, in order to generate more realistic images, embodiments of this disclosure introduce adversarial training into the text-to-image generation model. By introducing an adversarial loss function, the generated images can be made closer to real images, thereby improving the quality of the generated images.

[0058] In some embodiments, the generator and discriminator are trained alternately on a training dataset until the adversarial loss function converges, including:

[0059] The generator is fixed, and the discriminator is trained, including: generating fake images in the generator using random noise and descriptive information; inputting the fake images and real images into the discriminator and calculating the discriminator loss function; using the discriminator loss function to perform backpropagation to obtain the gradient and updating the parameters in the discriminator using the gradient descent method; repeating the above steps until the discriminator loss function is minimized.

[0060] as well as,

[0061] The discriminator is fixed, and the generator is trained, including: generating fake images in the generator using random noise and descriptive information; inputting the fake images and real images into the discriminator and calculating the generator loss function; using the generator loss function to perform backpropagation to obtain the gradient and updating the parameters in the generator using the gradient descent method; repeating the above steps until the generator loss function is minimized.

[0062] The goal of training the generator is to maximize the probability that the discriminator will misclassify the generated fake images as real images, while the goal of training the discriminator is to distinguish the fake images generated by the generator from real images and minimize the misclassification rate.

[0063] The adversarial loss function can be the sum of the generator loss function and the discriminator loss function. The generator and discriminator are trained alternately until the adversarial loss function converges.

[0064] According to embodiments of this disclosure, the discriminator and generator are trained alternately using a discriminator loss function and a generator loss function, and an adversarial loss function is determined after the alternating training. The adversarial loss function measures the adversarial relationship between the generator and the discriminator, and the generator's performance is improved by optimizing its value, thereby obtaining a generator with optimal performance.

[0065] In some embodiments, the generator includes a text encoding layer, an image encoding layer, a fusion layer, and a deconvolutional neural network. The text encoding layer is used to convert the text in the input description information into text vectors. The image encoding layer is used to convert the image in the input description information into image vectors. The fusion layer is used to fuse the text vectors and image vectors in a cross-attention module and then input them into the Adapter layer to obtain fused features. The deconvolutional neural network is used to generate an image based on the input fused features.

[0066] The Adapter layer is a special type of neural network layer used for information exchange and feature extraction between different network modules. It acts as an intermediary interface, transforming different feature data into a form suitable for a specific module, enabling better fusion and utilization of these features. The Adapter layer can adaptively adjust according to specific tasks and data characteristics, enhancing the generalization ability and performance of the entire network. When fusing multimodal information, the Adapter layer helps the network better learn and combine features from text and images to generate more accurate and complete output.

[0067] According to embodiments of this disclosure, text and images are two different modalities. In order to generate more realistic images, text information and image information are fused together. That is, the features encoded by images and text are fused through a cross-attention module, and then an Adapter layer formed by a non-linear layer is introduced to further refine the fused features.

[0068] In some embodiments, the generator further includes a context encoding layer for encoding context information in the input description information to obtain a context vector, and a fusion layer for fusing the text vector, image vector and context vector in the cross-attention module and then inputting them into the Adapter layer to obtain fused features.

[0069] According to embodiments of this disclosure, the problem that traditional text-to-image generation models typically only consider a single sentence and ignore the relationships between sentences is solved by combining multiple sentences and using contextual information to generate more accurate images.

[0070] The method 100 of this disclosure embodiment will be described below using specific implementation examples:

[0071] Humans can naturally image scenes based on textual or audio descriptions. However, for computers, artificial intelligence faces a modal gap in visual perception due to intramodal differences between visual cues and real images, and cross-modal differences between non-visual cues and real images. Effectively retrieving and fusing heterogeneous information from data of different modalities remains a major challenge in image synthesis and editing. Multimodal image synthesis and editing tasks aim to simulate human imagination and creativity in the real world, providing profound insights into how neural networks associate multimodal information with image attributes. Image synthesis and editing aim to create realistic images or edit realistic images with natural textures. While current models can generate stunning photographs, they cannot recreate the appearance of subjects in a given reference set, nor can they synthesize images of the same subject under different conditions. By utilizing semantic priors embedded in the model and specific prior preservation losses, the method proposed in this disclosure is able to synthesize subjects under various scene, pose, view, and lighting conditions not present in reference images. The method proposed in this disclosure can be applied to a variety of tasks, including subject recontextualization, text-guided image synthesis, appearance modification, and artistic rendering, among others.

[0072] Every source or form of information can be called a modality. For example, people have touch, hearing, vision, and smell; information media include speech, video, text, and data recorded by various sensors such as radar, infrared, and accelerometers. Each of these data forms can be called a modality (or pattern). In image synthesis and editing, guiding modalities can be categorized into visual guiding, text guiding, audio guiding, and other modalities. Text-to-image synthesis tasks aim to generate clear, photorealistic images with high semantic relevance to the corresponding text guiding. This task is very challenging because text descriptions are often ambiguous and can result in a large number of images with correct semantics. Furthermore, images and text have heterogeneous features, making it difficult to learn accurate and reliable mappings across the two modalities. Therefore, learning accurate embeddings of text descriptions plays a crucial role in text-guided image synthesis and editing. Text-guided synthesis and editing of images first requires learning useful codes from text representations. Traditional text encoders include Word2Vec and Bag-of-Words. With the popularization of deep neural networks, some scholars have proposed using character-level convolutional recurrent neural networks (char-CNN-RNN). With the development of pre-trained models in the field of natural language processing, some studies have also explored the use of large-scale pre-trained language models (such as BERT) for text encoding. Furthermore, contrastive language image pre-training (such as CLIP) achieves state-of-the-art (SOTA) results by learning the alignment of images and corresponding captions from a large number of image and text pairs. Specifically, CLIP jointly optimizes the text encoder and image encoder to maximize the cosine similarity between positive pairs and minimize the cosine similarity between negative pairs, thereby generating informative text embeddings.

[0073] This disclosure proposes a method that, when reconstructing the same subject, guided by a reference image, and in different environments, and with the guidance of prompts, outperforms most mainstream models, such as Dall-E2 and Imagen. The method proposed in this disclosure mainly consists of two steps:

[0074] Using a set of reference images with specific text cue annotations, fine-tune a low-resolution text-to-image model. These images contain a source text cue followed by the class name of the topic (e.g., "A [V] dog") and a similar, more specific target text cue (e.g., "A black [V] dog"). The goal is to alter the visual association of the source text cue to make it similar to the target text cue.

[0075] 2) This embodiment of the disclosure uses an improved text-image diffusion model to generate images. The main improvements made to the diffusion model in this embodiment are as follows:

[0076] Introducing contextual information. Traditional text-image diffusion models typically consider only individual sentences, ignoring the relationships between them. To address this issue, embodiments of this disclosure attempt to combine multiple sentences, leveraging contextual information to generate more accurate images.

[0077] Multimodal information is fused. Text and images are two different modalities. In order to generate more realistic images, this embodiment of the disclosure fuses text information and image information. Specifically, the features encoded by images and text are fused through a cross-attention module, and then an adapter formed by a non-linear layer is introduced to further refine the fused features.

[0078] Adversarial training is introduced. To generate more realistic images, embodiments of this disclosure introduce adversarial training into the text-image diffusion model. By introducing an adversarial loss function, the generated images can be made closer to real images, thereby improving the quality of the generated images.

[0079] To further enhance the editing function of the target text prompt, this disclosure presents a novel loss function designed to narrow down the source text prompt and the target text prompt.

[0080] set up , These are the embeddings for the source text hint and the target text hint, respectively. For each source tag... Source embedding We identify the target embeddings corresponding to the same label and label them as Note that embeddings from additional tags in the destination cue are discarded. However, their influence exists in other target tags through the text encoder architecture. In each cross-attention layer of the diffusion model, we compute the key and value of the target cue:

[0081]

[0082] The loss function proposed in this embodiment is:

[0083]

[0084] Where λ∈R+ are hyperparameters. This loss function encourages the source cue generation to behave similarly to the target cue generation, while maintaining closeness to the original projection matrix.

[0085] This embodiment of the disclosure uses the improved text-to-image diffusion model (i.e., text-to-image generation model) described above to obtain the final edited image, realizing a fine-tuning method based on a pre-trained text-to-image generation model, and transferring the text-to-image generation model to image editing tasks.

[0086] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0087] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0088] Figure 2 A block diagram of a text-to-image generation model optimization apparatus 200 according to an embodiment of the present disclosure is shown. Figure 2 As shown, the device 200 includes:

[0089] The acquisition module 210 is used to acquire a group of reference images corresponding to the topic. Each reference image in the group of reference images carries a source text hint containing a topic identifier and a target text hint. The target text hint is an extended description of the context of the topic in the reference image in the source text hint.

[0090] The optimization module 220 is used to input the reference image group into the pre-trained text-to-image generation model, optimize the pre-trained text-to-image generation model, and obtain the optimized text-to-image generation model. The optimized text-to-image generation model is used to generate the target image based on the descriptive information of the context in which the topic is located.

[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0092] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0093] Figure 3A block diagram of an exemplary electronic device 300 capable of implementing embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0094] Electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in ROM 302 or a computer program loaded into RAM 303 from storage unit 308. RAM 303 can also store various programs and data required for the operation of electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. I / O interface 305 is also connected to bus 304.

[0095] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0096] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0097] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0099] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0101] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0102] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0103] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0104] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for optimizing a text-to-image generation model, characterized in that, include: Obtain a set of reference images corresponding to a topic, wherein each reference image in the set carries a source text hint and a target text hint containing a topic identifier, and the target text hint is an extended description of the context in which the topic in the reference image is located in the source text hint; The reference image set is input into a pre-trained text-to-image generation model; the parameters of the pre-trained text-to-image generation model are optimized using backpropagation and gradient descent until the loss function converges, resulting in an optimized text-to-image generation model. The optimized text-to-image generation model is used to generate target images based on the descriptive information of the context in which the topic is located; The loss function is determined according to the following formula: in, For hyperparameters It is a positive real number. This indicates the embedding of the source text suggestion. Embedding of target text prompts, For the number of embeddings, and For parameter matrices, For the key vector in the cross-attention layer, This is the value vector in the cross-attention layer.

2. The method according to claim 1, characterized in that, The pre-trained text-to-image generation model is trained through the following steps: Initialize the generative adversarial network, which includes a generator and a discriminator; The generator and the discriminator are trained alternately based on the training dataset until the adversarial loss function converges. The training dataset includes real images and descriptive information corresponding to the real images. The descriptive information is used to describe the images to be generated. The generator is used as the pre-trained text-to-image generation model.

3. The method according to claim 2, characterized in that, The step of alternately training the generator and the discriminator based on the training dataset until the adversarial loss function converges includes: The generator is fixed, and the discriminator is trained, including: generating fake images in the generator using random noise and descriptive information; inputting the fake images and real images into the discriminator and calculating the discriminator loss function; using the discriminator loss function to perform backpropagation to obtain the gradient and updating the parameters in the discriminator using the gradient descent method; repeating the above steps until the discriminator loss function is minimized. as well as, The discriminator is fixed, and the generator is trained, including: generating fake images in the generator using random noise and descriptive information; inputting the fake images and real images into the discriminator and calculating the generator loss function; using the generator loss function to perform backpropagation to obtain the gradient and updating the parameters in the generator using the gradient descent method; repeating the above steps until the generator loss function is minimized.

4. The method according to claim 2, characterized in that, The generator includes a text encoding layer, an image encoding layer, a fusion layer, and a deconvolutional neural network. The text encoding layer is used to convert the text in the input description information into text vectors. The image encoding layer is used to convert the image in the input description information into image vectors. The fusion layer is used to fuse the text vector and the image vector in a cross-attention module and then input them into an adapter layer to obtain fused features. The deconvolutional neural network is used to generate an image based on the input fused features.

5. The method according to claim 4, characterized in that, The generator further includes a context encoding layer, which encodes the context information in the input description information to obtain a context vector. The fusion layer is also used to fuse the text vector, the image vector, and the context vector in the cross-attention module and then input them into the Adapter layer to obtain fused features.

6. A text-to-image generation model optimization device, characterized in that, include: The acquisition module is used to acquire a group of reference images corresponding to a topic, wherein each reference image in the group of reference images carries a source text hint containing a topic identifier and a target text hint, wherein the target text hint is an extended description of the context in which the topic in the reference image is located in the source text hint; An optimization module is used to input the reference image set into a pre-trained text-to-image generation model; and to optimize the parameters of the pre-trained text-to-image generation model using backpropagation and gradient descent until the loss function converges, thereby obtaining an optimized text-to-image generation model. The optimized text-to-image generation model is used to generate target images based on the descriptive information of the context in which the topic is located; The loss function is determined according to the following formula: in, For hyperparameters It is a positive real number. This indicates the embedding of the source text suggestion. Embedding of target text prompts, For the number of embeddings, and For parameter matrices, For the key vector in the cross-attention layer, This is the value vector in the cross-attention layer.

7. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 5.

Images