Image generation model training method, image generation method, apparatus, device and storage medium

Abstract

The present application relates to the technical field of artificial intelligence. Disclosed are an image generation model training method, an image generation method, an apparatus, a device and a storage medium. The image generation model training method comprises: acquiring at least one training sample, wherein each training sample comprises complex descriptive text and simple descriptive text which correspond to an original image; by means of a text encoding module and a neural network module, extracting a shallow representation and a deep representation which correspond to the simple descriptive text; on the basis of the shallow representation and the deep representation, determining a comprehensive text representation corresponding to the simple descriptive text; by means of the text encoding module, extracting a reference text representation corresponding to the complex descriptive text; and on the basis of the comprehensive text representation and the reference text representation corresponding to the complex descriptive text, adjusting parameters of an image generation model to obtain a trained image generation model.

Description

Image generation model training method, image generation method, device, equipment and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 11, 2023, with application number 202311007976.7, and invention name “Training method, device, equipment and storage medium for image generation model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a training method, apparatus, device, and storage medium for an image generation model. Background Art

[0003] With the continuous development of text-to-image technology, in text-to-image models such as diffusion models, it is possible to convert the description text input by the user into a predicted image corresponding to the description text.

[0004] In related technologies, a model's image generation capabilities are trained using triplet samples (original image, predicted image, and description text). The trained model is then able to generate predicted images based on the input description text. To improve model training, constructing the description text in the triplet sample typically requires obtaining complex and detailed description text for the original image, which in turn requires complex description text.

[0005] Technical content

[0006] The embodiments of the present application provide a training method for an image generation model, an image generation method, an apparatus, a device, and a storage medium, which can improve the accuracy of the generated predicted image when the description text is simple description text. The technical solution includes the following aspects.

[0007] According to one aspect of an embodiment of the present application, a method for training an image generation model is provided. The image generation model includes a neural network module, a pretrained text encoding module, and a pretrained diffusion module. The technical solution includes:

[0008] Obtain at least one training sample, each training sample including a complex description text and a simple description text corresponding to the original image;

[0009] Extracting the shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module; wherein the text encoding module is used to extract the shallow representation corresponding to the simple description text, and the neural network module is used to extract the deep representation corresponding to the simple description text; determining a comprehensive text representation corresponding to the simple description text based on the shallow representation and the deep representation, the comprehensive text representation is used to reflect the shallow representation and the deep representation, and the comprehensive text representation is used to generate a predicted image corresponding to the comprehensive text representation in combination with the original image through the diffusion module;

[0010] Inputting the complex description text into the text encoding module to extract a reference text representation corresponding to the complex description text;

[0011] The parameters of the image generation model are adjusted according to the comprehensive text representation and the reference text representation corresponding to the complex description text to obtain a trained image generation model.

[0012] According to one aspect of an embodiment of the present application, a method for generating an image based on an image generation model is provided. The image generation model includes a neural network module, a text encoding module, and a diffusion module. The technical solution includes:

[0013] Get the original image and simple description text;

[0014] Extracting the shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module; wherein the text encoding module is used to extract the shallow representation corresponding to the simple description text, and the neural network module is used to extract the deep representation corresponding to the simple description text; determining the comprehensive text representation corresponding to the simple description text based on the shallow representation and the deep representation, and the comprehensive text representation is used to reflect the shallow representation and the deep representation;

[0015] The diffusion module generates a predicted image corresponding to the comprehensive text representation according to the original image and the comprehensive text representation.

[0016] According to one aspect of an embodiment of the present application, a training device for an image generation model is provided. The image generation model includes a neural network module, a pre-trained text encoding module, and a pre-trained diffusion module. The technical solution includes:

[0017] A sample acquisition module is used to acquire at least one training sample, each training sample including a complex description text and a simple description text corresponding to the original image;

[0018] a representation extraction module configured to extract, through the text encoding module and the neural network module, a shallow representation and a deep representation corresponding to the simple description text; wherein the text encoding module is configured to extract the shallow representation corresponding to the simple description text, and the neural network module is configured to extract the deep representation corresponding to the simple description text; and determine, based on the shallow representation and the deep representation, a comprehensive text representation corresponding to the simple description text, wherein the comprehensive text representation is configured to reflect the shallow representation and the deep representation, and the comprehensive text representation is configured to be combined with the original image to generate a predicted image corresponding to the comprehensive text representation through the diffusion module;

[0019] The representation extraction module is further configured to input the complex description text into the text encoding module to extract a reference text representation corresponding to the complex description text;

[0020] A parameter adjustment module is used to adjust the parameters of the image generation model according to the comprehensive text representation and the reference text representation corresponding to the complex description text to obtain a trained image generation model.

[0021] According to one aspect of an embodiment of the present application, there is provided an image generation device based on an image generation model, wherein the image generation model includes a neural network module, a text encoding module, and a diffusion module;

[0022] Acquisition module, used to obtain original images and simple description text;

[0023] a representation extraction module, configured to extract, through the text encoding module and the neural network module, a shallow representation and a deep representation corresponding to the simple description text; wherein the text encoding module is configured to extract the shallow representation corresponding to the simple description text, and the neural network module is configured to extract the deep representation corresponding to the simple description text; and determine, based on the shallow representation and the deep representation, a comprehensive text representation corresponding to the simple description text, wherein the comprehensive text representation is configured to reflect the shallow representation and the deep representation;

[0024] An image generation module is configured to generate a predicted image corresponding to the comprehensive text representation based on the original image and the comprehensive text representation through the diffusion module.

[0025] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned image generation method.

[0026] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned training method of the image generation model, or to implement the above-mentioned image generation method based on the image generation model.

[0027] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes a computer program, and the computer program is loaded and executed by a processor to implement the above-mentioned training method of the image generation model, or implement the above-mentioned image generation method based on the image generation model.

[0028] BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a schematic diagram of an implementation environment for solutions provided by some embodiments of the present application;

[0030] FIG2 is a schematic diagram of a method for training and using an image generation model provided in some embodiments of the present application;

[0031] FIG3 is a flowchart of a method for training an image generation model provided in some embodiments of the present application;

[0032] FIG4 is a flowchart of a method for training an image generation model provided in some other embodiments of the present application;

[0033] FIG5 is a flowchart of a method for training an image generation model provided in some other embodiments of the present application;

[0034] FIG6 is a schematic diagram of a training method for an image generation model provided in some embodiments of the present application;

[0035] FIG7 is a flowchart of a method for training an image generation model provided in some further embodiments of the present application;

[0036] FIG8 is a schematic diagram of a method for determining a simple description text provided in some embodiments of the present application;

[0037] FIG9 is a schematic diagram of a training method for an image generation model provided in some other embodiments of the present application;

[0038] FIG10 is a flowchart of an image generation method based on an image generation model provided by some embodiments of the present application;

[0039] FIG11 is a schematic diagram of an image generation model provided by some embodiments of the present application;

[0040] FIG12 is a block diagram of a training apparatus for an image generation model provided in some embodiments of the present application;

[0041] FIG13 is a block diagram of an image generation device based on an image generation model provided by some embodiments of the present application;

[0042] FIG14 is a structural block diagram of a computer device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0044] The solutions provided in the embodiments of this application involve artificial intelligence computer vision technology, deep learning, and other technologies. In these embodiments, an image generation model is first adjusted using simple and complex descriptive text corresponding to the original image as training samples. The adjusted image generation model is then used to generate a predicted image based on the simple descriptive text. This is illustrated in the following examples.

[0045] Before introducing the technical solutions of this application, some of the terms involved in this application are explained. The following related explanations can be combined with the technical solutions of the embodiments of this application as optional solutions, and they all fall within the scope of protection of the embodiments of this application. The embodiments of this application include at least part of the following contents.

[0046] Pre-training model (PTM): also known as base model or large model, refers to a deep neural network (DNN) with a large number of parameters. It is trained on massive unlabeled data, and the function approximation ability of the large-scale parameter DNN is used to enable PTM to extract common features from the data. After fine-tuning (fine tune), efficient parameter fine-tuning (including prompt tuning, prefix tuning, adapter, LoRA and other methods), etc., it is suitable for downstream tasks. Therefore, the pre-training model can achieve ideal results in a few-shot or zero-shot scenario. PTM can be divided into language models, visual models (swin-transformer, ViT, V-MOE), speech models, multimodal models, etc. according to the data modality processed, where the multimodal model refers to a model that establishes two or more data modality feature representations. The pre-training model is an important tool for outputting artificial intelligence to generate content, and can also be used as a general interface for connecting multiple specific task models. The pre-trained model in the embodiment of the present application can be considered a pre-training model.

[0047] Text-to-Graph Model: A generative model based on a diffusion process. Given a descriptive text as input, the model applies a series of operations to a random noise image x, and generates a predicted image Y related to the text under cross-attention with the target text. Diffusion models are a type of generative model that generates images from noise samples through a gradual diffusion process.

[0048] Stable Diffusion Models (SDMs) are latent space-based diffusion models, belonging to the category of text-based graph models. They generate images by iteratively denoising and sampling an initialized noisy image. The SDMs in the embodiments of this application include a pretrained text encoding module and a pretrained diffusion module. Furthermore, the image generation model in the embodiments of this application builds on the SDMs with the addition of a neural network module.

[0049] Prompt: Enter the descriptive text for the stable diffusion model.

[0050] Shallow neural networks: These are neural networks with relatively few hidden layers, for example, only one or two. In a neural network, all layers other than the input and output layers are hidden layers. For example, in a convolutional neural network, the hidden layers may include: convolutional layers, activation layers, pooling layers, and fully connected layers.

[0051] Deep neural network: A neural network with a large number of hidden layers, for example, a neural network with three or more hidden layers.

[0052] Shallow representation: also known as shallow features, refers to features extracted using shallow neural networks. Since there are fewer hidden layers, the features extracted by shallow neural networks (for example, the text encoding module in the embodiment of the present application) contain more fine-grained information.

[0053] Deep representation: also known as deep features, refers to features extracted using deep neural networks. Deep neural networks can capture coarser-grained and more abstract information, namely semantic information.

[0054] Reference text representation: a text representation used to evaluate the accuracy of other text representations. For example, in an embodiment of the present application, the reference text representation may be a text representation corresponding to a complex descriptive text, for example, a text representation of a complex descriptive text extracted by a text encoding module. Since the text encoding module is pre-trained based on complex descriptive text as part of the training sample, the text encoding module's extraction result for the text representation of the complex descriptive text is relatively accurate. Therefore, the text representation of the complex descriptive text extracted by the text encoding module can be used as a reference text representation corresponding to the complex descriptive text. In addition, the reference text representation can also be a text representation corresponding to a simple descriptive text. For example, the text representation of a simple descriptive text extracted by a pre-trained language model (for example, a large language model) can be used as a reference text representation corresponding to the simple descriptive text. Since the large language model has excellent semantic understanding capabilities, the text representation of a simple descriptive text extracted by a pre-trained large language model can be used as a reference text representation corresponding to the simple descriptive text.

[0055] Complex descriptions, also known as complex prompts, are the descriptions fed into the diffusion model. Compared to simple descriptions, complex descriptions contain more keywords, enabling the diffusion model to generate high-quality images. For example, complex descriptions can include more than a predetermined number of keywords or be longer than a predetermined threshold.

[0056] Simple description text, also known as simple prompt words, contains fewer keywords than complex description text. When a user inputs simple description text into the diffusion model, the resulting image quality may be poor due to the diffusion model's limited semantic understanding and knowledge reasoning capabilities. For example, a simple description text can contain no more than a predetermined number of keywords, or a description text whose length does not exceed a predetermined threshold.

[0057] Please refer to Figure 1, which shows a schematic diagram of a solution implementation environment provided by some embodiments of the present application. The solution implementation environment may include a model training device 10 and a model use device 20.

[0058] The model training device 10 can be an electronic device such as a mobile phone, a desktop computer, a tablet computer, a laptop computer, an in-vehicle terminal, a server, an intelligent robot, a smart TV, a multimedia player, or other electronic devices with strong computing capabilities, which is not limited in this application. The model training device 10 is used to train the image generation model 30.

[0059] In an embodiment of the present application, the image generation model 30 is a machine learning model. In some embodiments, the model training device 10 can train the image generation model 30 in a machine learning manner so that it has better performance. In some embodiments, the image generation model 30 includes a neural network module, a pre-trained text encoding module and a pre-trained diffusion module. The training process of the image generation model 30 is as follows (this is only a brief description, the specific training process is described in the following embodiment, which is not repeated here): obtain at least one training sample, each training sample includes a complex description text and a simple description text corresponding to the original image; extract the shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module; determine the comprehensive text representation corresponding to the simple description text based on the shallow representation and the deep representation, and the comprehensive text representation is used to generate a predicted image corresponding to the comprehensive text representation in combination with the original image through the diffusion module; input the complex description text into the text encoding module to extract the reference text representation corresponding to the complex description text; adjust the parameters of the image generation model 30 based on the comprehensive text representation and the reference text representation corresponding to the complex description text to obtain the trained image generation model 30. In some embodiments, the text encoding module is used in conjunction with the neural network module to extract a comprehensive text representation corresponding to the descriptive text. In other embodiments, the diffusion module is used to generate a predicted image based on the text representation of the descriptive text and the original image. The specific internal processing flow of the diffusion model is explained in the following embodiments and is not further described here. In some embodiments, both the text encoding module and the diffusion module are machine learning models.

[0060] In some embodiments, the model using device 20 can be an electronic device such as a mobile phone, a desktop computer, a tablet computer, a laptop computer, a vehicle-mounted terminal, a server, an intelligent robot, a smart TV, a multimedia playback device, or other electronic devices with strong computing power, and this application does not limit this. For example, the trained image generation model 30 can be used to generate a predicted image based on a simple description text. In some embodiments, the image generation process of the image generation model 30 is as follows (this is only a brief description, and the specific use process is described in the following embodiments, which will not be repeated at this time): obtain the original image and the simple description text; extract the shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module; determine the comprehensive text representation corresponding to the simple description text based on the shallow representation and the deep representation; the comprehensive text representation is used to reflect the shallow representation and the deep representation; the comprehensive text representation corresponding to the simple description text is input into the diffusion module to generate a simple description text.

[0061] The model training device 10 and the model using device 20 can be two independent devices or the same device.

[0062] In the method provided in the embodiments of the present application, the execution entity of each step may be a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. When the electronic device is a server, the server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The computer device may be the model training device 10 in Figure 1 or the model utilization device 20.

[0063] Please refer to Figure 2, which shows a schematic diagram of the training and use methods of the image generation model provided by some embodiments of the present application.

[0064] As shown in FIG. 2 , the method for training and using an image generation model includes a training process 210 and a use process 220 .

[0065] Exemplarily, the specific training flow of the training process 210 is as follows: obtaining at least one training sample, each training sample including a complex description text and a simple description text corresponding to the original image; extracting the shallow representation and the deep representation corresponding to the simple description text through a text encoding module and a neural network module; determining the comprehensive text representation corresponding to the simple description text based on the shallow representation and the deep representation; inputting the complex description text into the text encoding module to extract the reference text representation corresponding to the complex description text; adjusting the parameters of the image generation model based on the comprehensive text representation and the reference text representation corresponding to the complex description text to obtain a trained image generation model.

[0066] In some embodiments, a first loss function value can be determined based on the difference between the comprehensive text representation and the reference text representation corresponding to the complex description text; based on the first loss function value, the parameters of the image generation model are adjusted to obtain the trained image generation model.

[0067] In some embodiments, the simple description text can be input into a pre-trained language model to extract a reference text representation corresponding to the simple description text; a second loss function value is determined based on the difference between the deep representation corresponding to the simple description text and the reference text representation corresponding to the simple description text; a comprehensive loss function value is obtained based on the first loss function value and the second loss function value; and the parameters of the neural network module in the image generation model are adjusted based on the comprehensive loss function value to obtain a trained neural network module.

[0068] In some embodiments, the adjusting the parameters of the image generation model includes: adjusting the parameters of the neural network module, and the parameters of the text encoding module and the diffusion module in the image generation model remain unchanged.

[0069] The trained image generation model includes the pre-trained diffusion model and the trained neural network module.

[0070] Exemplarily, the specific flow of using process 220 is as follows: obtaining an original image and a simple description text; extracting a shallow representation and a deep representation corresponding to the simple description text through a text encoding module and a neural network module; determining a comprehensive text representation corresponding to the simple description text based on the shallow representation and the deep representation; inputting the comprehensive text representation corresponding to the simple description text into a diffusion module, which generates a predicted image corresponding to the simple description text based on the original image and the comprehensive text representation. The original image here can be considered a noise image, or other related or unrelated image.

[0071] The image generation method in the related art is explained below.

[0072] In the related art, users need to manually write complex prompt words (complex description text) containing many keywords as input for the stable diffusion model to generate relatively high-quality images. When users input short narrative prompt words (simple description text), the stable diffusion model's limited semantic understanding and knowledge reasoning capabilities result in poor image quality, failing to meet user needs. In the related art, users need to write lengthy, complex prompt words as input for the stable diffusion model to generate high-quality images. However, writing complex prompt words is not user-friendly for non-experienced users and requires certain professional knowledge and a high threshold, resulting in a poor user experience. Furthermore, when users input short narrative prompt words, the stable diffusion model's limited semantic understanding and knowledge reasoning capabilities result in poor image quality, failing to meet user needs. In general, writing complex prompt words as input for the stable diffusion model can generate high-quality images, but complex prompt words are difficult to write and require a high user threshold. Conversely, when concise prompt words are input, the stable diffusion model generates poor-quality images.

[0073] The technical solution provided by the embodiment of the present application is based on the excellent semantic understanding and knowledge reasoning ability of the large language model (pre-trained language model). An additional neural network layer (neural network module) is inserted into the stable diffusion model as a semantic adapter. By distilling the knowledge of the large language model, the semantic representations (text representations) of simple prompt words and complex prompt words are aligned, and the semantic understanding and knowledge reasoning ability of the stable diffusion model for short prompt words are improved. The text encoder of the stable diffusion model can construct a high-quality text semantic representation to generate images, thereby improving the effect of simple prompt words generating images. In addition, when fine-tuning the stable diffusion model, the pre-trained model parameters are frozen and only the newly inserted additional neural network layer is trained, which reduces the amount of model parameters that need to be trained and achieves efficient parameter fine-tuning. This not only reduces the memory usage in the fine-tuning stage and reduces the requirements of hardware resources, but also speeds up the training speed and shortens the training time. In general, by utilizing the excellent semantic understanding and knowledge reasoning ability of the large language model, an additional neural network layer for semantic adaptation is inserted into the stable diffusion model, the semantic representations of simple prompt words and complex prompt words are aligned, and the effect of short prompt words generating images is improved. The technical solution provided in this application embodiment bridges the semantic gap between simple and complex prompt words through knowledge distillation from a large language model, improving the image generation performance of a stable diffusion model using simple prompt words. This can be applied to text-based image generation tasks, such as generating avatars and cover images.

[0074] Please refer to Figure 3, which shows a flowchart of the training method of the image generation model provided in some embodiments of the present application. The execution entity of each step of the method can be the model training device introduced above. In the following method embodiments, for ease of description, only the execution entity of each step is introduced as a "computer device". The method can include at least one of the following steps (310-340).

[0075] Step 310: Obtain at least one training sample, each training sample including a complex description text and a simple description text corresponding to the original image.

[0076] During the model training process, the original image is considered to be an image corresponding to the complex description text, that is, the content represented in the original image is consistent with the complex description text. The simple description text is considered to be the text that wants to generate the original image based on the image generation model. Description text corresponding to the original image: used to describe the content of the original image. In the embodiment of the present application, the description text corresponding to the original image can be the real text input by the user, or it can be the text extracted from the original image by the model. The embodiment of the present application does not limit the method of obtaining the description text. Of course, the embodiment of the present application does not limit the number of words, display type, display style, etc. of the description text. The description text can represent the overall scene characteristics of the original image, or it can represent the characteristics of the main objects in the original image, and this application does not limit this. In some embodiments, the description text corresponding to the original image is divided into simple description text and complex description text.

[0077] In the embodiments of the present application, there is no limitation on the sources of complex description texts and simple description texts. For example, the original image and the complex description text corresponding to the original image are crawled from a graphic database website. For example, based on the original image, the simple description text corresponding to the original image is obtained. For example, the simple description text corresponding to the original image is obtained by manual description. For another example, a simple graphic text model is used to obtain the simple description text corresponding to the original image based on the original image, wherein the graphic text model is a machine learning model, the input is the original image, and the output is the simple description text corresponding to the original image.

[0078] In some embodiments, the text contents corresponding to the simple description text and the complex description text are different. In some embodiments, the text length of the simple description text is less than a first threshold, while the text length of the complex description text is greater than a second threshold, wherein the first threshold is less than or equal to the second threshold, and this application does not limit the specific numerical value of the first threshold or the second threshold. In some embodiments, the matching score between the complex description text and the original image is greater than the matching score between the simple description text and the original image. In some embodiments, the first image generated by the text graph model based on the complex description text and the second image generated by the text graph model based on the simple description text have different corresponding resolutions, and the resolution of the first image is greater than the resolution of the second image. In some embodiments, the text content included in the complex description text completely includes the text content included in the simple description text. In some embodiments, the text content included in the complex description text does not completely include the text content included in the simple description text. In some embodiments, for the same original image, the complex description text is "A little rabbit is running across the grassland under a starry night sky. The Milky Way shines brightly overhead, casting a soft glow. The rabbit's fur sparkles under the illumination of countless stars as it skips across the field, its small body moving gracefully through the tall grass. In the distance, meteors streak across the sky, leaving a trail of light behind them. The rabbit pauses for a moment, looks up at the wonders of the sky in awe, and then continues to frolic in the quiet wilderness", while the simple description text is "A white rabbit sitting on the grass under the starry sky".

[0079] Step 320: Extract shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module, wherein the text encoding module is used to extract the shallow representation corresponding to the simple description text, and the neural network module is used to extract the deep representation corresponding to the simple description text; determine the comprehensive text representation corresponding to the simple description text based on the shallow representation and the deep representation; the comprehensive text representation is used to reflect the shallow representation and the deep representation, and is used to generate a predicted image corresponding to the comprehensive text representation in combination with the original image through the diffusion module.

[0080] The image generation model in the embodiments of the present application includes a neural network module, a pre-trained text encoding module, and a pre-trained diffusion module. Both the text encoding module and the diffusion module are pre-trained, and the embodiments of the present application do not limit the specific pre-training process of the text encoding module and the diffusion module. For example, a noise map is generated based on a random noise seed, encoded, and the encoded features are repeatedly denoised through the forward process of the diffusion module to obtain a latent space representation. The text encoding module then generates a text representation based on the descriptive text. Through the reverse process of the diffusion model, the latent space representation is repeatedly denoised based on the text representation to obtain denoised features, which are then decoded to obtain a predicted image. Based on the difference between the original image used as a training sample and the generated predicted image, the parameters of the text encoding module and the diffusion module are adjusted to obtain a pre-trained text encoding module and a pre-trained diffusion module. The embodiments of the present application do not limit the specific architecture of the text encoding module and the diffusion module; both are machine learning modules. The text encoding module takes text as input and outputs a text representation; the diffusion module takes the original image and the text representation as input and outputs a predicted image.

[0081] In the embodiments of the present application, both the neural network module and the text encoding module are modules for extracting text representations. The embodiments of the present application do not limit the connection method between the neural network module and the text encoding module. For example, the text encoding module and the neural network module are connected in series, or the text encoding module and the neural network module are connected in parallel. In some embodiments, the text encoding module is used to extract shallow text representations, while the neural network module is used to extract deep text representations.

[0082] In some embodiments, the text encoding module and the neural network module are connected in parallel. Exemplarily, the number of convolutional layers included in the text encoding module is smaller than the number of convolutional layers included in the neural network module, or the number of pooling layers included in the text encoding module is smaller than the number of pooling layers included in the neural network module. In some embodiments, since the number of convolutional layers included in the text encoding module is smaller than the number of convolutional layers included in the neural network module, or the number of pooling layers included in the text encoding module is smaller than the number of pooling layers included in the neural network module, the text encoding module is used to extract shallow representations of the text, while the neural network module is used to extract deep text representations of the text.

[0083] In other embodiments, the text representation module and the neural network module are connected in series, and the output of the text representation module serves as the input of the neural network module. The output of the text representation module can be considered as a shallow representation, while the output of the neural network module based on the shallow representation is considered as a deep representation.

[0084] In some embodiments, a comprehensive text representation corresponding to the simple description text is extracted through a text encoding module and a neural network module. Exemplarily, when the text encoding module and the neural network module are connected in parallel, the shallow representation output by the text encoding module for the input text and the deep representation output by the neural network module for the input text are comprehensively considered to obtain the comprehensive text representation. Exemplarily, when the text encoding module and the neural network module are connected in series, the shallow representation output by the text encoding module for the input text and the deep representation output by the neural network module for the shallow representation are comprehensively considered to obtain the comprehensive text representation.

[0085] The embodiments of the present application do not limit the method for determining the comprehensive text representation. For example, the shallow representation and the deep representation are dimensionally aligned and then directly added to obtain the comprehensive text representation. For example, the shallow representation and the deep representation are dimensionally aligned and then weighted summed to obtain the comprehensive text representation. For example, the shallow representation and the deep representation are multiplied to obtain the comprehensive text representation.

[0086] Step 330: Input the complex description text into a text encoding module to extract a reference text representation corresponding to the complex description text.

[0087] In some embodiments, a complex description text is input into a text encoding module, and a reference text representation corresponding to the complex description text is extracted. Because the text encoding module is pre-trained based on the complex description text as part of the training sample, the text representation extracted by the text encoding module for the complex description text is relatively accurate. In other words, the text representation extracted by the text encoding module for the complex description text can be considered a reference text representation.

[0088] Step 340 , adjusting the parameters of the image generation model according to the comprehensive text representation and the reference text representation corresponding to the complex description text, to obtain a trained image generation model.

[0089] In the embodiment of the present application, there is no limitation on the method for adjusting the parameters of the image generation model through the comprehensive text representation and the reference text representation corresponding to the complex descriptive text. For example, the loss function value is determined by the comprehensive text representation and the reference text representation corresponding to the complex descriptive text, and the parameters of the image generation model are adjusted according to the loss function value to obtain the trained image generation model. In the embodiment of the present application, an additional neural network module is added on the basis of the pre-trained text encoding module, so that after the training is completed, the comprehensive text representation extracted for the simple descriptive text based on the text encoding module and the neural network can be comparable to the reference text representation of the complex descriptive text by the text encoding module, thereby improving the image generation accuracy.

[0090] The embodiments of the present application do not limit the adjustment method for adjusting the parameters of the image generation model. Exemplarily, all parameters in the image generation model are adjusted based on the comprehensive text representation and the reference text representation corresponding to the complex description text to obtain a trained image generation model. Exemplarily, some parameters in the image generation model are adjusted based on the comprehensive text representation and the reference text representation corresponding to the complex description text to obtain a trained image generation model. For example, based on the comprehensive text representation and the reference text representation corresponding to the complex description text, the parameters of the additional neural network module in the image generation model are adjusted without changing the parameters of other pre-trained modules to obtain a trained image generation model. In this way, the parameter adjustment cost can be reduced and the model training efficiency can be improved. For another example, based on the comprehensive text representation and the reference text representation corresponding to the complex description text, the parameters of the neural network module and the text encoding module in the image generation model are adjusted without changing the parameters of the diffusion module to obtain a trained image generation model. In this way, the consistency of the comprehensive text representation corresponding to the simple description text and the reference text representation corresponding to the complex description text can be ensured.

[0091] The technical solution provided by the embodiment of the present application introduces a neural network module on the basis of a pre-trained text encoding module, and adjusts the parameters of the image generation model through the comprehensive text representation corresponding to the simple description text and the reference text representation corresponding to the complex description text, so that the comprehensive text representation corresponding to the simple description text in the adjusted model can be aligned with the reference text representation corresponding to the complex description text, thereby achieving that when the user input is a simple description text, the comprehensive text representation obtained after passing through the text encoding module and the neural network module can have a semantically rich text representation as the reference text representation corresponding to the complex description text, thereby improving the semantic understanding and knowledge reasoning capabilities of the image generation model, thereby improving the image accuracy of the subsequently generated predicted image.

[0092] Please refer to Figure 4, which shows a flowchart of the training method of the image generation model provided in some other embodiments of the present application. The execution entity of each step of the method can be the model training device introduced above. In the following method embodiments, for the sake of ease of description, only the execution entity of each step is introduced as a "computer device". The method can include at least one of the following steps (410-470).

[0093] Step 410: Obtain at least one training sample, each training sample including a complex description text and a simple description text corresponding to the original image.

[0094] In some embodiments, a simple prompt word (simple description text) is recorded as p s , complex prompt words (complex description text) are recorded as pc , the text encoder (text encoding module) is denoted as function f enc (), the pre-trained language model is denoted as f LLM (), the newly inserted adapter module (neural network module) is denoted as f ad (). The simple prompt word is represented by the text encoder of the stable diffusion model and then sent to the newly inserted adapter module.

[0095] Step 420: extracting the shallow representation corresponding to the simple description text through the text encoding module.

[0096] In some embodiments, the input of the text encoding module is a simple description text, and the output is a shallow representation corresponding to the simple description text. The embodiment of the present application does not limit the size of the shallow representation, and the shallow representation can be considered as a feature vector, vector matrix, etc. output by the text encoding module.

[0097] In some embodiments, the shallow representation corresponding to the simple description text is represented as f enc (p s ).

[0098] Step 430: Obtain a deep representation corresponding to the simple description text based on the shallow representation through a neural network module.

[0099] In some embodiments, the input of the neural network module is a shallow representation, and the output is a deep representation corresponding to the simple description text. The embodiment of the present application does not limit the size of the deep representation, and the deep representation can be considered as a feature vector, vector matrix, etc. output by the neural network module.

[0100] In some embodiments, the deep representation corresponding to the simple description text is represented as f ad (f enc (p s )).

[0101] Step 440 : Perform weighted summation on the shallow representation and the deep representation to obtain a comprehensive text representation.

[0102] In some embodiments, the comprehensive text representation v corresponding to the simple description text LLM =β·f ad (f enc (p s ))+(1-β)·f enc (p s ).

[0103] The embodiment of the present application does not limit the specific numerical value of the weight value β.

[0104] Step 450 : extracting a reference text representation corresponding to the complex description text through a text encoding module.

[0105] In some embodiments, the input of the text encoding module is a complex description text, and the output is a reference text representation corresponding to the complex description text. The embodiments of the present application do not limit the size of the reference text representation. The reference text representation can be considered as a feature vector, vector matrix, etc. output by the text encoding module.

[0106] In some embodiments, the reference text representation corresponding to the complex description text is represented as f enc (p c ).

[0107] Step 460 : Determine a first loss function value based on the difference between the comprehensive text representation and the reference text representation corresponding to the complex description text.

[0108] In some embodiments, there is no limitation on the manner in which the first loss function value is determined based on the difference between the comprehensive text representation and the reference text representation corresponding to the complex descriptive text. In some embodiments, the loss function includes but is not limited to a cross entropy loss function, a mean square error loss function, a Huber loss function, and the like.

[0109] In some embodiments, the loss function is a KL divergence function, also known as a relative entropy function. For example, the first loss function value Loss cp =KL[v LLM ,f enc (p c )].

[0110] In some embodiments, step 460 also includes: extracting a reference text representation corresponding to the simple description text through a pre-trained language model; and determining a second loss function value based on the difference between the deep representation corresponding to the simple description text and the reference text representation corresponding to the simple description text.

[0111] In some embodiments, the input of the pre-trained language model is a simple descriptive text, and the output is a reference text representation corresponding to the simple descriptive text. The embodiments of the present application do not limit the size of the reference text representation, and the reference text representation can be considered as a feature vector, vector matrix, etc. output by the pre-trained language model.

[0112] The embodiments of the present application do not limit the specific architecture of the pre-trained language model and the pre-training method. For example, the pre-trained language model is a large language model. In some embodiments, the large language model here can adopt the open source LLaMA model or BLOOM model.

[0113] In some embodiments, the reference text representation f corresponding to the simple description text LLM (p s ).

[0114] In some embodiments, a second loss function value is determined based on a difference between the comprehensive text representation and a reference text representation corresponding to the simple description text. In some embodiments, there is no limitation on the manner in which the second loss function value is determined based on the difference between the deep representation corresponding to the simple description text and the reference text representation corresponding to the simple description text. In some embodiments, the loss function includes but is not limited to a cross entropy loss function, a mean square error loss function, a Huber loss function, and the like.

[0115] In some embodiments, the loss function is a KL divergence function, also known as a relative entropy function. For example, the second loss function value Loss LLM =KL[f ad (f enc (p s )),f LLM (p s )].

[0116] Step 470: Adjust the parameters of the image generation model according to the first loss function value to obtain a trained image generation model.

[0117] In the embodiments of the present application, the method for adjusting the parameters of the image generation model based on the first loss function value is not limited. For example, the parameters of the image generation model are adjusted with the goal of minimizing the first loss function value to obtain a trained image generation model. For example, the parameter adjustment includes forward gradient updating or reverse gradient updating, which is also not limited in this application.

[0118] The embodiment of the present application adjusts the parameters of the image generation model through the first loss function value, which can align the text representation of complex descriptive text and the text representation of simple descriptive text, thereby improving the accuracy of the predicted image generated by the image generation image based on the text representation.

[0119] In some embodiments, step 470 also includes step 471 .

[0120] Step 471: Adjust the parameters of the image generation model according to the first loss function value and the second loss function value to obtain a trained image generation model.

[0121] In some embodiments, there is no limitation on the manner of adjusting parameters of image generation based on the first loss function value and the second loss function value.

[0122] In some embodiments, step 471 also includes: performing weighted summation of the first loss function value and the second loss function value to obtain a comprehensive loss function value; and adjusting the parameters of the image generation model according to the comprehensive loss function value to obtain a trained image generation model.

[0123] In some embodiments, the comprehensive loss function Loss=λLoss LLM +(1-λ)Loss cp The embodiment of the present application does not limit the specific numerical value of the weight value λ.

[0124] Of course, when calculating the comprehensive loss function value, in addition to using the weighted summation method, other methods can also be used. For example, the first loss function value and the second loss function value are directly added to obtain the comprehensive loss function value. For example, the first loss function value and the second loss function value are multiplied to obtain the comprehensive loss function value.

[0125] In the embodiments of the present application, the method for adjusting the parameters of the image generation model based on the comprehensive loss function value is not limited. For example, the parameters of the image generation model are adjusted with the goal of minimizing the comprehensive loss function value to obtain a trained image generation model. For example, the parameter adjustment includes forward gradient updating or reverse gradient updating, which is also not limited in this application.

[0126] In other embodiments, when adjusting the parameters of the image generation model, the parameters of the neural network module are adjusted, and the parameters of the text encoding module and the diffusion module in the image generation model remain unchanged.

[0127] In the embodiment of the present application, the purpose of introducing the second loss function value is to enable the additional neural network module to be aligned with the pre-trained language model, so that the deep representation of the simple descriptive text obtained by the neural network module can have the same rich semantics as the reference text features output by the large language model, thereby improving the neural network module's ability to understand the text and realizing knowledge distillation of the large language model.

[0128] Of course, in the embodiment of the present application, when adjusting the image generation model, the parameters of the neural network module are adjusted, and the parameters of the text encoding module and the diffusion module in the image generation model remain unchanged. That is, in the fine-tuning stage, the pre-trained model parameters of the stable diffusion model are frozen, and only the newly inserted additional neural network module for semantic adaptation is trained to achieve efficient parameter fine-tuning.

[0129] Please refer to Figure 5, which shows a flowchart of the training method of the image generation model provided by some other embodiments of the present application. The execution entity of each step of the method can be the model training device introduced above. In the following method embodiments, for the sake of ease of description, only the execution entity of each step is introduced as a "computer device". The method can include at least one of the following steps (510-550).

[0130] Step 510: Obtain at least one training sample, each training sample including a complex description text and a simple description text corresponding to the original image.

[0131] Step 510 also includes extracting a reference text representation corresponding to the simple description text using a pre-trained language model. A text encoding module extracts a shallow representation corresponding to the simple description text. A neural network module uses the shallow representation to obtain a deep representation corresponding to the simple description text. The shallow and deep representations are weighted and summed to obtain a comprehensive text representation. The text encoding module also extracts a reference text representation corresponding to the complex description text.

[0132] Step 520 : Determine a second loss function value based on the difference between the deep representation corresponding to the simple description text and the reference text representation corresponding to the simple description text.

[0133] Step 530 : Determine a first loss function value based on the difference between the comprehensive text representation corresponding to the simple description text and the reference text representation corresponding to the complex description text.

[0134] Step 540: Perform a weighted summation on the first loss function value and the second loss function value to obtain a comprehensive loss function value.

[0135] Step 550: Adjust the parameters of the image generation model according to the comprehensive loss function value to obtain a trained image generation model.

[0136] Please refer to Figure 6, which shows a schematic diagram of the training method for the image generation model provided in some embodiments of the present application. As shown in 600 of Figure 6, in order to improve the semantic understanding and knowledge reasoning capabilities of the stable diffusion model (image generation model), an additional neural network module (i.e., an adapter) for semantic adaptation is inserted after the text encoder (text encoding module) of the stable diffusion model. The adapter includes at least one fully connected layer and at least one nonlinear activation function layer. In some embodiments, the neural network module includes two fully connected layers and one nonlinear activation function layer. The specific adjustment process is as follows: After the simple prompt word passes through the text encoder of the stable diffusion model to obtain a shallow representation, it is sent to the newly inserted adapter module to obtain a deep representation. The shallow representation and the deep representation are weighted to obtain a comprehensive text representation corresponding to the simple prompt word. After the simple prompt word passes through the large language model (pre-trained language model), a reference text representation corresponding to the simple prompt word is obtained. After the complex prompt word passes through the text encoder, a reference text representation corresponding to the complex prompt word is obtained. The first loss function value is determined based on the KL divergence between the reference text representation corresponding to the complex prompt word and the comprehensive text representation corresponding to the simple prompt word. The second loss function value is determined based on the KL divergence between the deep representation corresponding to the simple prompt word and the reference text representation corresponding to the simple prompt word. The first and second loss function values ​​are weighted summed to obtain a comprehensive loss function value, which is used to adjust the parameters of the adapter module (neural network module) in the image generation model.

[0137] The technical solution provided by the embodiment of the present application utilizes the excellent semantic understanding ability of the large language model, inserts an additional neural network layer for semantic adaptation into the stable diffusion model, bridges the gap in semantic representation between simple prompt words and complex prompt words, and improves the semantic understanding and knowledge reasoning ability of the stable diffusion model for short prompt words, thereby improving the effect of generating images with concise prompt words. In addition, when fine-tuning the stable diffusion model, only the newly inserted additional neural network layer is trained, achieving efficient parameter fine-tuning. This not only reduces the video memory usage in the fine-tuning stage and reduces the requirements for hardware resources, but also speeds up the training speed and shortens the training time. In general, by utilizing the outstanding semantic understanding and knowledge reasoning ability of the large language model, an additional neural network layer is inserted into the stable diffusion model as a semantic adapter, the semantic representation of concise prompt words and complex prompt words is aligned, and the effect of generating images with short prompt words is improved.

[0138] Please refer to Figure 7, which shows a flowchart of a training method for an image generation model provided in some embodiments of the present application. The execution entity of each step of the method can be the model training device described above. In the following method embodiments, for ease of description, only the execution entity of each step is described as a "computer device". The method can include at least one of the following steps (710-760).

[0139] Step 710: Obtain at least one image-text pair, each image-text pair including an original image and a complex description text corresponding to the original image.

[0140] In some embodiments, step 710 further includes: screening at least one image-text pair according to the length of the complex description text corresponding to the original image in each image-text pair to obtain at least one screened image-text pair, and the screened at least one image-text pair is used to construct a training sample.

[0141] In some embodiments, the complex description texts whose length is less than the third threshold are eliminated, while the complex description texts whose length is greater than the third threshold are retained. The embodiment of the present application does not limit the specific numerical value of the third threshold. After removing the instruction text of the control parameters contained in the prompt words, due to the different lengths of these prompt words, too short prompt words are not suitable as complex prompt words. Therefore, the training sample data whose prompt word length is less than a certain fixed threshold are filtered out. The prompt words in the retained training data are used as complex prompt words, and each training data is a two-tuple, (complex prompt word, original image).

[0142] Step 720: For each image-text pair, generate a simple description text corresponding to the original image of the image-text pair.

[0143] In some embodiments, a simple description text corresponding to the original image is directly generated through the image-to-text model.

[0144] In some embodiments, step 720 includes at least one of steps 721 - 722 .

[0145] Step 721: For each image-text pair, generate at least one candidate simple text, and calculate the matching score between each candidate simple text and the original image in the image-text pair through the text-image matching model. The matching score is used to represent the degree of matching between each candidate simple text and the original image.

[0146] The embodiments of the present application do not limit the specific architecture of the text-image matching model, which is a machine learning model. In some embodiments, the input of the text-image matching model is a candidate simple text and an original image, and the output is a semantic matching score between the candidate simple text and the original image, that is, a matching score. In some embodiments, the input of the text-image matching model is an original image and n candidate simple texts, and the output is a matching score corresponding to each of the n candidate simple texts, that is, n matching scores.

[0147] Step 722: Determine the simple description text corresponding to the original image from at least one candidate simple text according to the matching score corresponding to each candidate simple text.

[0148] In some embodiments, one or more candidate simple texts with the highest matching scores are selected from the matching scores corresponding to at least one candidate simple text as the simple description text corresponding to the original image.

[0149] In some embodiments, after determining the simple description text corresponding to the original image, it can be further determined whether the matching score of the simple description text meets predetermined conditions. If it is determined that the matching score corresponding to the simple description text does not meet the conditions, the training sample constructed by the simple description text is removed from the training sample.

[0150] In some embodiments, when screening simple descriptive text for an original image, it is also necessary to consider that the matching score between the complex descriptive text and the original image should be lower than the matching score between the simple descriptive text and the original image. Therefore, the matching score of the simple descriptive text screened out should be higher than the matching score between the complex descriptive text and the original image. In other words, if the matching score corresponding to the simple text determined to be simple descriptive text is not higher than the matching score between the complex descriptive text and the original image, the training sample constructed by the simple text is discarded.

[0151] In some embodiments, the open-source BLIP (Bootstrapping Language-Image Pre-training, a multimodal model for unified understanding and generation) model is called to generate a brief descriptive text for each image. In some embodiments, the open-source CLIP (Contrastive Language-Image Pre-Training) model is called to calculate the semantic matching score (matching score) of the image (original image) on simple and complex prompt words. Since complex prompt words not only contain text related to the image content, but also contain text unrelated to the image content, such as text describing the image resolution and image style, the semantic matching score of simple prompt words is generally higher than that of complex prompt words. If the semantic matching score of simple prompt words is too low, it indicates that the simple prompt words generated by the BLIP model do not match the image well enough, and such training data needs to be filtered out. After multiple data cleaning and filtering, high-quality training data can be obtained. Each data item is a triple (including a simple prompt word, a complex prompt word, and the original image). Of course, when training the image generation model, only simple prompt words and complex prompt words are required.

[0152] In some embodiments, as shown in 800 of Figure 8, multiple simple texts are generated for the original image using the image-to-text model - BLIP model, and the matching score between each simple text and the original image is calculated using the text-image matching model - CLIP model, and the simple text with the highest score that is not lower than the matching score corresponding to the complex description text is selected as the simple description text of the original image.

[0153] Step 730: Obtain at least one training sample according to the complex description text and the simple description text corresponding to at least one original image.

[0154] Step 740: Extract the comprehensive text representation corresponding to the simple description text through the text encoding module and the neural network module; wherein the text encoding module is used to extract the shallow representation corresponding to the simple description text, the neural network module is used to extract the deep representation corresponding to the simple description text, the comprehensive text representation is used to reflect the shallow representation and the deep representation, and the comprehensive text representation is used to combine with the original image to generate a predicted image corresponding to the comprehensive text representation through the diffusion module.

[0155] Step 750 : extracting the reference text representation corresponding to the complex description text through the text encoding module.

[0156] Step 760 , adjusting the parameters of the image generation model according to the comprehensive text representation and the reference text representation corresponding to the complex description text, to obtain a trained image generation model.

[0157] In the embodiments of this application, when constructing a training sample set, the matching score is used to determine the simple descriptive text that matches the original image, thereby improving the matching degree between the simple descriptive text and the original image and enhancing the accuracy of the training samples. Furthermore, the training data is filtered at least twice: once to filter out short, complex prompt words, and again to filter out simple descriptive text with insufficient matching scores. Both of these are intended to improve the accuracy of the training samples and thus enhance the effectiveness of model training.

[0158] Please refer to Figure 9, which shows a schematic diagram of a method for training an image generation model provided in some embodiments of the present application. The execution entity of each step of the method can be the model training device described above. In the following method embodiments, for ease of description, only the execution entity of each step is described as a "computer device."

[0159] As shown in 900 of Figure 9 , the computer device first crawls raw data from a website, specifically, the original image and the complex prompt words corresponding to the original image. This is based on publicly available online image generation websites such as midjourney and Stable Diffusion Online. These open-source image generation websites offer reliable, user-written prompt words and high-quality generated images. These prompt words are complex and meticulously crafted by experienced users, and the generated images are semantically correct, making them suitable for use as raw data. The computer device crawls raw data from these publicly available online image generation websites. Each piece of data contains a user-written prompt word and a high-quality image. To ensure the quality of the training data, the captured raw data needs to be cleaned. User-written prompt words contain some parameter-controlled instruction text. For example, in the data crawled from midjourney, the "--version" or "--v" parameters are used to control the model version; these parameter-controlling instruction text needs to be cleaned. The computer device then filters the training data based on the length of the prompt words, uses the BLIP model to generate simple prompt words from the original images, and uses the CLIP model to filter out semantically mismatched training data. The filtered data is then used to construct a training dataset (training sample set). Next, after constructing the training sample set, additional neural network modules and a large language model are introduced to efficiently fine-tune the parameters of the stable diffusion model, and the trained model is used to generate predicted images based on simple descriptive text.

[0160] Please refer to Figure 10, which shows a flowchart of an image generation method based on an image generation model provided in some embodiments of the present application. The execution entity of each step of this method can be the model-using device described above. In the following method embodiments, for ease of description, only the execution entity of each step is described as a "computer device." This method can include at least one of the following steps (1010-1030).

[0161] Step 1010: Obtain the original image and simple description text.

[0162] The technical solution provided in the embodiment of the present application includes at least two application scenarios. First, a predicted image is generated entirely based on a simple description text. In this case, the original image during the use of the model can be considered as a noise image, which is generated based on a random seed. Second, a predicted image is generated based on an original image and a simple description text. In this case, the image generation model predicts or modifies the original image based on the simple description text on the basis of the original image to obtain a predicted image. In this case, the original image during the use of the model can be considered as an image to be modified. Of course, in the second case, if the original image obtained is the image to be modified, a noise image can also be superimposed on the original image to obtain an input image input to the diffusion module. Exemplarily, the size of the noise image is the same as that of the original image, and the sum of the pixel values ​​of the corresponding pixel points in the original image and the noise image is determined as the pixel value of the corresponding pixel point in the input image.

[0163] In some embodiments, during the use of the model, simple description text is considered to be the text entered by the user. That is, regardless of whether the user enters complex description text or simple description text, the image generation method provided by this application can be applied, and the accuracy of the predicted image obtained is relatively high.

[0164] Step 1020: Extract the comprehensive text representation corresponding to the simple description text through the text encoding module and the neural network module; wherein, the text encoding module is used to extract the shallow representation corresponding to the simple description text, the neural network module is used to extract the deep representation corresponding to the simple description text, and the comprehensive text representation is used to reflect the shallow representation and the deep representation.

[0165] In some embodiments, step 1020 includes at least one of steps 1021 - 1023 .

[0166] Step 1021: extract the shallow representation corresponding to the simple description text through the text encoding module.

[0167] Step 1022: Obtain a deep representation corresponding to the simple description text based on the shallow representation through a neural network module.

[0168] Step 1023 , performing weighted summation on the shallow representation and the deep representation to obtain a comprehensive text representation.

[0169] For steps 1020 to 1023 in the embodiment of the present application, please refer to the explanation in the embodiment of the above-mentioned model training side and will not be repeated here.

[0170] Step 1030 : Generate a predicted image corresponding to the comprehensive text representation based on the original image and the comprehensive text representation through a diffusion module.

[0171] In some embodiments, the forward process of the diffusion module is also called the diffusion process, which is used to add noise to the input data one by one until the input data approaches pure noise. Exemplarily, the diffusion process as a whole can be a parameterized Markov chain. In some embodiments, the noisy original image is encoded by a first encoder to obtain an initial feature vector of the noisy original image; the initial feature vector is denoised T times by the forward process of the diffusion module to generate a latent space representation corresponding to the noisy original image, where T is a positive integer. In some embodiments, the forward process of the diffusion module denoises the initial feature vector T times to generate a latent space representation corresponding to the random noise image, and the backward process of the diffusion module denoises the latent space representation T times according to the text representation to obtain a denoised latent space representation. The backward process of the diffusion module is used to remove noise from the input data one by one according to the constraints, thereby generating a predicted image. Exemplarily, the backward process of the diffusion module as a whole can also be a parameterized Markov chain. In some embodiments, the latent space representation and the text representation serve as input to the backward process of the diffusion module. The backward process of the diffusion module sequentially denoises and constrains the latent space features based on the text representation, ensuring that the predicted image meets the constraints of the text representation. In some embodiments, the text representation input to the diffusion module can be considered a comprehensive text representation that simply describes the text.

[0172] In some embodiments, as shown in FIG11 , FIG11 shows a schematic diagram of the structure of the image generation model 1100. The input image (noise image or the original image superimposed with the noise image) is encoded by the encoder to obtain the initial feature vector Z of the input image. The text encoding module generates a shallow representation corresponding to the simple description text based on the simple description text, and the neural network module generates a deep representation corresponding to the simple description text based on the shallow representation. The shallow representation and the deep representation are weighted and summed to obtain a comprehensive text representation. The comprehensive text representation is used as the input data of the denoising network. The initial feature vector is denoised T times by the forward process of the diffusion module to generate the latent space representation Z corresponding to the input image. T Latent space representation Z T The text representation is used as the input data of the downsampling network of the denoising network. According to the output data of the downsampling network, the input data of the upsampling network is obtained. The upsampling network obtains the output feature Z after denoising based on the text representation and the input data of the upsampling network. T-1′ After T-1 times of denoising network, the denoised latent space representation Z′ is obtained, and the denoised latent space representation Z′ is decoded by the decoder to generate the predicted image Y.

[0173] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0174] Please refer to Figure 12, which shows a block diagram of a training device for an image generation model provided in some embodiments of the present application. The image generation model includes a neural network module, a pre-trained text encoding module, and a pre-trained diffusion module. As shown in Figure 12, the device 1200 may include a sample acquisition module 1210, a representation extraction module 1220, and a parameter adjustment module 1230.

[0175] The sample acquisition module 1210 is configured to acquire at least one training sample, where each training sample includes a complex description text and a simple description text corresponding to the original image.

[0176] The representation extraction module 1220 is used to extract the shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module; wherein the text encoding module is used to extract the shallow representation corresponding to the simple description text, and the neural network module is used to extract the deep representation corresponding to the simple description text, and the comprehensive text representation corresponding to the simple description text is determined based on the shallow representation and the deep representation, and the comprehensive text representation is used to generate a predicted image corresponding to the comprehensive text representation in combination with the original image through the diffusion module.

[0177] The representation extraction module 1220 is further configured to input the complex description text into the text encoding module to extract a reference text representation corresponding to the complex description text.

[0178] The parameter adjustment module 1230 is used to adjust the parameters of the image generation model according to the comprehensive text representation and the reference text representation corresponding to the complex description text to obtain a trained image generation model.

[0179] Please refer to Figure 13, which shows a block diagram of an image generation device based on an image generation model provided in some embodiments of the present application. The image generation model includes a neural network module, a text encoding module, and a diffusion module. As shown in Figure 13, the device 1300 may include: an acquisition module 1310, a representation extraction module 1320, and an image generation module 1330.

[0180] The acquisition module 1310 is used to acquire the original image and the simple description text.

[0181] The representation extraction module 1320 is used to extract the shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module; wherein the text encoding module is used to extract the shallow representation corresponding to the simple description text, and the neural network module is used to extract the deep representation corresponding to the simple description text, and the comprehensive text representation corresponding to the simple description text is determined based on the shallow representation and the deep representation, and the comprehensive text representation is used to reflect the shallow representation and the deep representation.

[0182] The image generation module 1330 is configured to generate a predicted image corresponding to the comprehensive text representation based on the original image and the comprehensive text representation through the diffusion module.

[0183] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0184] Please refer to Figure 14, which shows a block diagram of a computer device 1400 provided in some embodiments of the present application. The computer device 1400 can be any electronic device capable of data calculation, processing, and storage. The computer device 1400 can be used to implement the aforementioned image generation model training method or the aforementioned image generation method based on the image generation model.

[0185] Typically, the computer device 1400 includes a processor 1401 and a memory 1402 .

[0186] The processor 1401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1401 may also include an AI processor for processing computing operations related to machine learning.

[0187] Memory 1402 may include one or more computer-readable storage media, which may be non-transitory. Memory 1402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 1402 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned image generation model training method, or implement the above-mentioned image generation model-based image generation method.

[0188] Those skilled in the art will appreciate that the structure shown in FIG. 14 does not limit the computer device 1400 , and may include more or fewer components than shown, or combine certain components, or adopt a different component arrangement.

[0189] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by the processor, the training method of the above-mentioned image generation model is implemented, or the image generation method based on the above-mentioned image generation model is implemented. In some embodiments, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random Access Memory), SSD (Solid State Drives) or optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0190] In an exemplary embodiment, a computer program product is also provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned method for training an image generation model or implement the aforementioned method for generating an image based on an image generation model.

[0191] It should be noted that the collection and processing of the relevant data (including original images, simple descriptive text or complex descriptive text) in this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0192] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0193] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for an image generation model, executed by a computer device, wherein the image generation model comprises a neural network module, a pre-trained text encoding module, and a pre-trained diffusion module, and the method comprises: Obtain at least one training sample, each training sample including a complex description text and a simple description text corresponding to an original image; Extracting the shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module; wherein the text encoding module is used to extract the shallow representation corresponding to the simple description text, and the neural network module is used to extract the deep representation corresponding to the simple description text; Determine, according to the shallow representation and the deep representation, a comprehensive text representation corresponding to the simple description text, wherein the comprehensive text representation is used to generate a predicted image corresponding to the comprehensive text representation by combining with the original image through the diffusion module; Inputting the complex description text into the text encoding module to extract the reference text representation corresponding to the complex description text; According to the comprehensive text representation and the reference text representation corresponding to the complex description text, the parameters of the image generation model are adjusted to obtain a trained image generation model.

2. The method according to claim 1, wherein: The extracting of shallow representation and deep representation corresponding to the simple description text by the text encoding module and the neural network module includes: Inputting the simple description text into the text encoding module to extract the shallow representation corresponding to the simple description text; Inputting the shallow representation into the neural network module to obtain a deep representation corresponding to the simple description text according to the shallow representation; Determining a comprehensive text representation corresponding to the simple description text according to the shallow representation and the deep representation includes: The shallow representation and the deep representation are dimensionally aligned and weightedly summed to obtain the comprehensive text representation.

3. The method according to claim 1 or 2, wherein: The step of adjusting the parameters of the image generation model according to the comprehensive text representation and the reference text representation corresponding to the complex description text to obtain a trained image generation model includes: Determining a first loss function value according to a difference between the comprehensive text representation and a reference text representation corresponding to the complex description text; According to the first loss function value, the parameters of the image generation model are adjusted to obtain the trained image generation model.

4. The method according to claim 1 or 2, further comprising: Inputting the simple description text into a pre-trained language model to extract a reference text representation corresponding to the simple description text; Determining a second loss function value according to a difference between the deep representation corresponding to the simple description text and the reference text representation corresponding to the simple description text; According to the comprehensive text representation and the reference text representation corresponding to the complex description text, the parameters of the image generation model are adjusted to obtain the trained image generation model, including: Determining a first loss function value according to a difference between the comprehensive text representation and the first reference text representation; According to the first loss function value and the second loss function value, the parameters of the image generation model are adjusted to obtain the trained image generation model.

5. The method according to claim 4, wherein: The step of adjusting the parameters of the image generation model according to the first loss function value and the second loss function value to obtain the trained image generation model comprises: Performing a weighted summation on the first loss function value and the second loss function value to obtain a comprehensive loss function value; According to the comprehensive loss function value, the parameters of the image generation model are adjusted to obtain the trained image generation model.

6. The method according to any one of claims 1 to 5, wherein: The adjusting the parameters of the image generation model includes: The parameters of the neural network module are adjusted, and the parameters of the text encoding module and the diffusion module in the image generation model remain unchanged.

7. The method according to any one of claims 1 to 6, wherein: The obtaining of at least one training sample comprises: Acquire at least one image-text pair, each image-text pair including an original image and a complex description text corresponding to the original image; For each image-text pair, generate a simple description text corresponding to the original image of the image-text pair; At least one training sample is obtained according to each acquired image-text pair and the simple description text generated for the image-text pair.

8. The method according to claim 7, wherein: The generating of the simple description text corresponding to the original image includes: For each image-text pair, at least one candidate simple text is generated, and a matching score between each candidate simple text and the original image is calculated respectively through a text-image matching model, wherein the matching score is used to represent the matching degree between each candidate simple text and the original image; According to the matching scores respectively corresponding to the at least one candidate simple text, the simple description text corresponding to the original image is determined from the at least one candidate simple text.

9. The method according to claim 8, further comprising: For each image-text pair, if the matching score of the simple description text determined from the at least one candidate simple text does not meet a preset condition, the image-text pair and the simple description text corresponding to the image-text pair are filtered out from the training sample.

10. The method according to any one of claims 7 to 9, wherein: The acquiring of at least one image-text pair comprises: Obtain candidate image-text pairs, and screen the candidate image-text pairs according to the length of the complex description text corresponding to the original image in each image-text pair in the candidate image-text pairs to obtain the at least one image-text pair.

11. An image generation method based on an image generation model, wherein the image generation model comprises a neural network module, a text encoding module and a diffusion module, and the method comprises: Get the original image and simple description text; Extracting the shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module; wherein the text encoding module is used to extract the shallow representation corresponding to the simple description text, and the neural network module is used to extract the deep representation corresponding to the simple description text; Determining a comprehensive text representation corresponding to the simple description text according to the shallow representation and the deep representation; the comprehensive text representation is used to reflect the shallow representation and the deep representation; The comprehensive text representation corresponding to the simple description text is input into the diffusion module, so as to generate a predicted image corresponding to the simple description text through the diffusion module.

12. The method according to claim 11, wherein: The extracting of shallow representation and deep representation corresponding to the simple description text by the text encoding module and the neural network module includes: Inputting the simple description text into the text encoding module to extract the shallow representation corresponding to the simple description text; Inputting the shallow representation into the neural network module to extract the deep representation corresponding to the simple description text according to the shallow representation; Determining a comprehensive text representation corresponding to the simple description text according to the shallow representation and the deep representation includes: The shallow representation and the deep representation are weightedly summed to obtain the comprehensive text representation.

13. A training device for an image generation model, the image generation model comprising a neural network module, a pre-trained text encoding module and a pre-trained diffusion module, the device comprising: A sample acquisition module, used to acquire at least one training sample, each training sample including a complex description text and a simple description text corresponding to the original image; A representation extraction module, used to extract the shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module; wherein the text encoding module is used to extract the shallow representation corresponding to the simple description text, and the neural network module is used to extract the deep representation corresponding to the simple description text; according to the shallow representation and the deep representation, a comprehensive text representation corresponding to the simple description text is determined, the comprehensive text representation is used to reflect the shallow representation and the deep representation, and the comprehensive text representation is used to generate a predicted image corresponding to the comprehensive text representation in combination with the original image through the diffusion module; The representation extraction module is further used to input the complex description text into the text encoding module to extract the reference text representation corresponding to the complex description text; The parameter adjustment module is used to adjust the parameters of the image generation model according to the comprehensive text representation and the reference text representation corresponding to the complex description text to obtain a trained image generation model.

14. An image generation device based on an image generation model, wherein the image generation model comprises a neural network module, a text encoding module and a diffusion module, and the device comprises: Acquisition module, used to obtain original images and simple description text; A representation extraction module, used to extract the shallow representation and deep representation corresponding to the simple description text through the text encoding module and the neural network module; wherein the text encoding module is used to extract the shallow representation corresponding to the simple description text, and the neural network module is used to extract the deep representation corresponding to the simple description text; according to the shallow representation and the deep representation, a comprehensive text representation corresponding to the simple description text is determined, and the comprehensive text representation is used to reflect the shallow representation and the deep representation; An image generation module is used to generate a predicted image corresponding to the comprehensive text representation according to the original image and the comprehensive text representation through the diffusion module.

15. A computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the training method of the image generation model as described in any one of claims 1 to 10, or to implement the image generation method based on the image generation model as described in any one of claims 11 to 12.

16. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program is loaded and executed by a processor to implement the training method of the image generation model as described in any one of claims 1 to 10, or to implement the image generation method based on the image generation model as described in any one of claims 11 to 12.

17. A computer program product, comprising computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, and when the computer instructions are executed, the training method of the image generation model as described in any one of claims 1 to 10 is implemented, or the image generation method based on the image generation model as described in any one of claims 11 to 12 is implemented.

Images