Image generation method, object image generation method, image generation model training method, and cloud training platform

Abstract

Embodiments of the present description provide an image generation method, an object image generation method, an image generation model training method, and a cloud training platform. The image generation model training method is applied to the field of deep learning, and comprises: acquiring a description text of target content; using a target image generation model to perform inference denoising on a random noise feature on the basis of a description text, and generate a plurality of target images corresponding to the description text of target content, wherein the target image generation model is obtained by training an initial image generation model on the basis of a plurality of prediction images and a plurality of sample original images of sample content, the plurality of prediction images are generated by performing inference denoising on a plurality of sample noise features on the basis of sample description texts of the plurality of sample original images, and the plurality of sample noise features are obtained by performing noise injection and feature encoding on the plurality of sample original images. The model learns to generate a plurality of content-related images, thereby improving generalization ability and training flexibility, reducing training complexity, and improving image generation efficiency and accuracy.

Description

Image generation, object image generation, image generation model training method and cloud training platform

[0001] The present disclosure claims priority to Chinese Patent Application No. 202411136703.7, filed on August 16, 2024, with the Chinese Patent Office, entitled "Image generation, object image generation, image generation model training method and cloud training platform", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] Embodiments of the present specification relate to the technical field of deep learning, in particular to an image generation, object image generation, image generation model training method and cloud training platform. BACKGROUND

[0003] With the development of deep learning technology, image generation models based on "text-to-image" technology have been widely used in many visual fields.

[0004] Currently, image generation models represented by attention models (Transformer models), diffusion models (Diffusion models) and generative adversarial networks (GAN models) generate images corresponding to the description content under the description of the text. Such image generation models often use the training paradigm of "large-scale pre-training + domain fine-tuning (e.g., control network, IP adapter, dream house, etc.)" to better adapt to image generation applications in different domains.

[0005] Image generation models without fine-tuning can only generate one image at a time. When applied to multiple image generation fields (e.g., multiple person image generation, image style transfer, picture book generation, expression package group generation, and object multi-view generation, etc.), the efficiency of image generation is insufficient, the content correlation between multiple images generated multiple times is insufficient, resulting in content fragmentation between multiple images, which requires domain fine-tuning for multiple image generation fields. The model has insufficient generalization ability, and the complexity of model training is high and the flexibility is insufficient. Therefore, there is an urgent need for an image generation method that can generate images with high content correlation. SUMMARY

[0006] Therefore, the embodiment of the present specification provides an image generation method. One or more embodiments of the present specification simultaneously relate to an object image generation method, a training method of an image generation model, another training method of an image generation model, a cloud training platform, an image generation device, an object image generation device, a training device of an image generation model, another training device of an image generation model, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0007] According to a first aspect of the embodiment of the present specification, an image generation method is provided, comprising:

[0008] obtaining a description text of target content;

[0009] using a target image generation model, based on the description text, performing inference denoising on random noise features to generate a plurality of target images corresponding to the description text of the target content, wherein the target image generation model is trained based on an initial image generation model, a plurality of prediction images, and a plurality of sample original images of sample content, the plurality of prediction images are generated by performing inference denoising on a plurality of sample noise features based on sample description texts of the plurality of sample original images, and the plurality of sample noise features are obtained by adding noise and feature encoding to the plurality of sample original images.

[0010] According to a second aspect of the embodiment of the present specification, an object image generation method is provided, applied to a server, comprising:

[0011] receiving a task request sent by a user terminal, wherein the task request carries prompt information of an object image generation task;

[0012] based on the prompt information, determining a description text of a target object;

[0013] using a target image generation model, based on the description text, performing inference denoising on random noise features to generate a plurality of target images of the target object corresponding to the description text, wherein the target image generation model is trained based on an initial image generation model, a plurality of prediction images, and a plurality of sample original images of sample content, the plurality of prediction images are generated by performing inference denoising on a plurality of sample noise features based on sample description texts of the plurality of sample original images, and the plurality of sample noise features are obtained by adding noise and feature encoding to the plurality of sample original images.

[0014] feeding back the plurality of target images to the user terminal.

[0015] According to a third aspect of the embodiment of the present specification, a training method of an image generation model is provided, comprising:

[0016] obtain a plurality of sample noise images by respectively adding noise to the plurality of sample original images;

[0017] obtain a plurality of sample noise images by respectively adding noise to the plurality of sample original images;

[0018] obtain a plurality of sample noise features by respectively performing feature encoding on the plurality of sample noise images;

[0019] obtain a plurality of predicted images by respectively performing inference and denoising on the plurality of sample noise features based on the plurality of sample description texts by using the initial image generation model;

[0020] train the initial image generation model based on the plurality of sample original images and the plurality of predicted images to obtain a target image generation model.

[0021] According to a fourth aspect of the embodiments of the present specification, another training method of an image generation model is provided, applied to a cloud training platform, comprising:

[0022] receive a model training request sent by a terminal device;

[0023] obtain a plurality of sample noise images by respectively adding noise to the plurality of sample original images;

[0024] obtain a plurality of sample noise images by respectively adding noise to the plurality of sample original images;

[0025] obtain a plurality of sample noise features by respectively performing feature encoding on the plurality of sample noise images;

[0026] obtain a plurality of predicted images by respectively performing inference and denoising on the plurality of sample noise features based on the plurality of sample description texts by using the initial image generation model;

[0027] train the initial image generation model based on the plurality of sample original images and the plurality of predicted images to obtain a target image generation model.

[0028] feed back the target image generation model to the terminal device.

[0029] According to a fifth aspect of the embodiments of the present specification, a cloud training platform is provided, comprising a request interface and a response unit;

[0030] The request interface is configured to receive a model training request sent by a terminal device, and obtain a plurality of sample original images of sample content and sample description texts of the plurality of sample original images in response to the model training request.

[0031] The response unit is configured to add noise to each of the plurality of sample original images to obtain a plurality of sample noise images, encode features of the plurality of sample noise images to obtain a plurality of sample noise features, infer and denoise the plurality of sample noise features based on the plurality of sample description texts and the initial image generation model to generate a plurality of predicted images, and train the initial image generation model based on the plurality of sample original images and the plurality of predicted images to obtain a target image generation model, and feed the target image generation model to the terminal device.

[0032] According to a sixth aspect of an embodiment of the present specification, a computing device is provided, comprising:

[0033] a memory and a processor; the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the above method.

[0034] According to a seventh aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the steps of the above method.

[0035] According to an eighth aspect of an embodiment of the present specification, a computer program product is provided, comprising computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the steps of the above method.

[0036] In one embodiment of the present specification, after the plurality of sample original images are respectively added with noise, the plurality of sample noise features are obtained by respectively encoding features of the plurality of sample noise images, which are taken as a whole input feature, and the plurality of sample noise features are inferred and denoised based on the sample description texts of the plurality of sample original images to generate a plurality of predicted images. The initial image generation model perceives the sample content in the plurality of sample original images at the same time, and the target image generation model is trained based on the plurality of predicted images and the plurality of sample original images, so that the image generation model can learn to establish content correlation between generated images in the inference and denoising process, avoiding content fragmentation between the plurality of output images, improving the model generalization ability, and the target image generation model with high generalization ability can be well adapted to image generation applications in more fields, reducing the complexity of model training, improving the flexibility of model training, and thus realizing the generation of images with high content correlation, improving the efficiency and accuracy of image generation. BRIEF DESCRIPTION OF DRAWINGS

[0037] FIG. 1 is a flowchart of an image generation method according to an embodiment of the present specification;

[0038]

[0038] FIG. 2 is a structural schematic diagram of an image generation model according to an embodiment of the present specification;

[0039] FIG. 3 is a structural diagram of an inference denoising layer in an image generation model according to an embodiment of the present specification;

[0040] FIG. 4 is a flowchart of an object image generation method according to an embodiment of the present specification;

[0041] FIG. 5 is an effect diagram of an object image generation method applied to portrait generation according to an embodiment of the present specification;

[0042] FIG. 6 is an effect diagram of an object image generation method applied to line drawing coloring according to an embodiment of the present specification;

[0043] FIG. 7 is an effect diagram of an object image generation method applied to sketch to photo conversion according to an embodiment of the present specification;

[0044] FIG. 8 is an effect diagram of an object image generation method applied to picture book generation according to an embodiment of the present specification;

[0045] FIG. 9 is an effect diagram of an object image generation method applied to expression package set generation according to an embodiment of the present specification;

[0046] FIG. 10 is an effect diagram of an object image generation method applied to multi-view view generation of an object according to an embodiment of the present specification;

[0047] FIG. 11 is a flowchart of a training method of an image generation model according to an embodiment of the present specification;

[0048] FIG. 12 is a flowchart of another training method of an image generation model according to an embodiment of the present specification;

[0049] FIG. 13 is a structural diagram of a cloud training platform according to an embodiment of the present specification;

[0050] FIG. 14 is a structural diagram of an image generation device according to an embodiment of the present specification;

[0051] FIG. 15 is a structural diagram of an object image generation device according to an embodiment of the present specification;

[0052] FIG. 16 is a structural diagram of a training device of an image generation model according to an embodiment of the present specification;

[0053] FIG. 17 is a structural diagram of another training device of an image generation model according to an embodiment of the present specification;

[0054] FIG. 18 is a structural block diagram of a computing device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0055] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. However, the present description can be practiced without the specific details, and it is understood that the present description is not limited to the specifics of the description. Accordingly, it is understood that the present description is not limited to the specifics of the description.

[0056] The terminology used in this description of one or more embodiments shall include any and all variations of one or more words that are described herein, including the use of a singular number, plural number, a term that can include one or more embodiments, or a term that can include one or more embodiments of the described use. The use of "and / or" in the description of one or more embodiments shall include any and all via ble combinations of one or more associated listed items. The use of "at least one of" in the description of one or more embodiments shall include any and all via ble combinations of one or more associated listed items.

[0057] It is to be understood that the terms first, second, etc. can be employed in this description of one or more embodiments simply for the purpose of differentiating between one or more information and to not limit the scope of the present description. These terms should be understood as interchangeable unless otherwise expressly identified. For example, a first item can be termed a second item, and, similarly, a second item can be termed a first item without departing from the scope of the present description. The word "if' in context with the word "comprising" can be interpreted as meaning "when" or "upon" or "in response to determining" taking into account the context in which the word is used.

[0058] Further, it is to be understood that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the description of one or more embodiments are information and data that are authorized by the user or sufficiently authorized by all parties, and the collection, use, and processing of the relevant data need to comply with relevant laws, regulations, and standards of the country and region, and provide corresponding operation portals for the user to choose to authorize or refuse.

[0059] In one or more embodiments of the present specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, billions, tens of billions, hundreds of billions, or even tens of billions of model parameters. Large models can also be referred to as foundation models. Through large-scale unlabeled corpus pre-training, a pre-trained model with hundreds of millions of parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLM) and multi-modal pre-training models.

[0060] In practical applications, large models only need a small amount of samples to fine-tune the pre-trained model and can be applied to different tasks. Large models can be widely used in natural language processing (NLP) and computer vision fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), image generation, and natural language processing tasks such as text-based sentiment classification, text summarization generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0061] First, the technical terms related to one or more embodiments of the present specification are explained.

[0062] ControlNets: A technique for guiding generative AI models that allows users to guide the image generation process by inputting additional control signals such as sketches, depth maps, poses, etc., achieving more fine-grained control and more specific results.

[0063] IP-Adapters: A fine-tuning technique that allows the model to adapt to specific styles or topics by introducing a small number of additional parameters without changing the underlying model weights, adding personalized or specialized styles to the generative model. This allows the model to quickly learn new styles without forgetting previously learned knowledge.

[0064] DreamBooths: A fine-tuning method mainly used for text-to-image generation models. It works by showing the model a small dataset of images containing a specific concept and their associated textual descriptions, allowing the model to learn to associate this concept with a specific style or object. This method is particularly useful for customized training, such as teaching the model the style of a specific brand logo or the portrait of a specific person.

[0065] Transformer Model: A deep neural network architecture commonly used in machine learning tasks.

[0066] Diffusion Model: A generative model that, after fitting the data distribution, can sample new data from the distribution. It is commonly used in image / speech / video generation scenarios.

[0067] Diffusion Transformer (DiT): A combination of the Transformer architecture and the diffusion model concept, designed to generate high-quality data. It leverages the feature extraction capabilities of the Transformer and the generation capabilities of the diffusion model, making it particularly suitable for longer data generation tasks.

[0068] GAN Model: A generative model composed of two parts: the Generator and the Discriminator. These two networks are trained through mutual competition, and the Generator eventually generates high-quality samples that are almost indistinguishable from real data.

[0069] In this specification, an image generation method is provided, and the specification also relates to an object image generation method, a training method of an image generation model, a cloud training platform, an image generation device, an object image generation device, a training device of an image generation model, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0070] Referring to FIG. 1, FIG. 1 shows a flowchart of an image generation method according to an embodiment of the present specification, which includes the following specific steps:

[0071] Step 102: Obtain the description text of the target content.

[0072] Exemplarily, the embodiments of the present specification apply to an application, a website or an applet with an image generation function. On the application, the website or the applet, an image generation task is performed. For example, a website deploying a large language model can perform image generation tasks such as multi-person image generation, image style transfer, a set of picture book generation, an emoticon package set generation and object multi-view generation. For another example, an image processing application can perform image generation tasks such as multi-person image generation and image style transfer. For another example, an image drawing application can perform an image generation task of picture book generation. For another example, a software development application can perform an image generation task of emoticon package set generation. For another example, a design application software can perform an image generation task of object multi-view generation, which is not limited herein. The above-mentioned application can deploy a target image generation model locally, or remotely call a target image generation model deployed on the cloud side through an application programming interface (API), which is not limited herein.

[0073] The target content is a core element that the generated target image needs to represent. It includes but is not limited to at least one of the following: a target object, a target theme, a target style, a target scene and a target color tone. For example, the target object is a person, an animal or a room. For another example, the target theme is “summer style” or “spring atmosphere”. For another example, the target style is “animation style” or “vintage photography”. For another example, the target scene is a city, a night scene or a natural scenery. For another example, the target color tone is “warm color” or “cool color”.

[0074] The description text of the target content is a natural language text used to describe the target content. The description text describes the visual details of the target content and provides a text modal guidance condition for the target image generation model to generate the target image. For example, a user wants to generate 20 photos of a person in different poses, different lighting and different expressions. The target description text for generating 20 photos includes: (pose 1: standing, hands in pockets; lighting 1: dusk, soft sunset; expression 1: relaxed smile), (pose 2: sitting, crossed legs; lighting 2: indoor, bright light; expression 2: looking at a book with concentration), (pose 3: walking, arms swinging gently; lighting 3: morning, sunlight through mist; expression 3: looking forward with anticipation) …… For another example, a user wants to generate 50 video frames of an animal in a period of time. The target description text for generating 50 video frames includes: video frame 1: “The snow leopard walks carefully in the snow, searching for traces of food; it stops, ears up, and listens alertly.” Video frame 2: “The snow leopard discovers a string of footprints on the ground, lowers its head to sniff, and shakes its tail gently.” Video frame 3: “It suddenly stops and gazes at the shrubs in front, as if it has heard something moving.”

[0075] For example, on a certain image drawing application, the target image generation model deployed on the cloud side is remotely called through the application programming interface. The user needs to automatically generate a set of expression package on the image drawing application, and the target content is a cartoon character. The user inputs five description texts of the cartoon character as follows: Description text_1: "The cartoon character wears a red hat, smiles, eyes are narrowed into a line, right hand clenches a fist beside the chin, as if saying 'hello! '." Description text_2: "The cartoon character looks surprised at the left, mouth open into a circle, eyebrows raised, hands in front of the chest, showing a shocked look." Description text_3: "The cartoon character is crying, two rows of tears slide from the corners of the eyes, the corners of the mouth droop, the left hand wipes the eyes, and the right hand is on the knee, showing great sadness." Description text_4: "The cartoon character smiles, closes one eye and makes a funny face, points to the audience with the right hand, and puts the left hand on the hip, wearing blue jeans and a yellow T-shirt." Description text_5: "The cartoon character frowns with anger, eyes wide open, hands clenched into fists, body leaning forward, as if preparing to attack."

[0076] The description text of the target content is obtained, which provides a text condition for subsequent generation of multiple target images corresponding to the description text of the target content.

[0077] Step 104: using the target image generation model, based on the description text, the random noise feature is inferred and denoised to generate multiple target images corresponding to the description text of the target content, wherein the target image generation model is trained based on the initial image generation model based on multiple prediction images and multiple sample original images of sample content, the multiple prediction images are generated by inferring and denoising multiple sample noise features based on multiple sample original images, and the multiple sample noise features are obtained by adding noise and feature encoding to the multiple sample original images.

[0078] Illustratively, the image generation model is a deep learning model for generating corresponding images based on description texts. The image generation model is guided to generate images step by step by taking description texts as text conditions. The image generation model includes but is not limited to attention model (Transformer model), diffusion model (Diffusion model), diffusion Transformer model and generative adversarial model (GAN model).

[0079] The target image generation model is a trained image generation model. The target image generation model is trained based on the initial image generation model, the plurality of predicted images, and the plurality of sample original images of the sample content. The target image generation model has the inference denoising capability of the plurality of images. The target image generation model can generate a plurality of images step by step under the guidance of the description text as the text condition. The target image generation model can generate a plurality of images at a time, and is suitable for generating a plurality of images in the image generation field. The plurality of images generated by the target image generation model have high content correlation. Optionally, the target image generation model can include only the inference denoising layer for inference denoising, or can further include at least one of the noise adding layer for feature noise adding, the feature encoding layer for feature encoding, and the feature splicing layer for feature splicing.

[0080] The random noise feature is used as a seed for inference denoising to generate an image under the constraint of the text condition of the description file. The random noise feature includes, but is not limited to, a random Gaussian noise feature, a random exponential noise feature, a random Rayleigh noise feature, a random gamma noise feature, a random Poisson noise feature, and a random uniform noise feature.

[0081] The target image is a visual image of target content corresponding to the description text, and is generated by the target image generation model under the guidance of the inference denoising of the description text as the text condition. The plurality of target images correspond to the description text used for generating the target image, and the plurality of target images corresponding to the description text have high content correlation.

[0082] The sample content is a core element that needs to be expressed in the sample original image, and includes, but is not limited to, at least one of the following: a sample object, a sample theme, a sample style, a sample scene, and a sample color tone.

[0083] The sample original image is a visual image containing sample content. The plurality of sample original images contain sample content, and form a sample image group. The plurality of sample original images have content correlation.

[0084] The predicted image is a visual image of the sample content corresponding to the sample description text generated by the initial image generation model. The predicted image is generated by the initial image generation model using the sample description text as text conditions to guide inference and denoising. Multiple predicted images are applied to the sample description text used to generate their own images. The predicted image is a visual image that reconstructs the original sample image. For example, if the original sample image is a photograph of a person in a portrait collection, showing a specific pose, lighting, and expression, the predicted image is a visual image reconstructed from that photograph. Another example is a video frame from an animal video, where the predicted image is a visual image reconstructed from that video frame. Yet another example is an illustration with the theme "Springtime," where the predicted image is a visual image reconstructed from that illustration. Yet another example is an anime-style character rendering, where the predicted image is a visual image reconstructed from that character rendering. Finally, an original sample image is a tourist photo from a tourist destination, where the predicted image is a visual image reconstructed from that tourist photo. For example, the original sample image is an advertisement image with a "warm" color tone, and the predicted image is a visual image that reconstructs that advertisement image.

[0085] The initial image generation model is an untrained model. While it possesses general inference denoising capabilities and can use sample description text as textual conditions to guide progressive inference denoising and image generation, it currently only generates one image at a time. In applications involving multiple image generation, the content correlation between the generated images is insufficient, and content fragmentation exists between the images. Optionally, the initial image generation model may include only an inference denoising layer for inference denoising, or it may include at least one of the following: a noise-adding layer for feature addition, a feature encoding layer for feature encoding, and a feature concatenation layer for feature concatenation.

[0086] The sample description text is natural language text used to describe the sample content in the original sample image. The sample description text describes the visual details of the sample content in the original sample image. Any pair of original sample images and their sample description texts constitutes a set of image-text sample pairs with content correlation. Multiple image-text sample pairs constitute a grouped list of image-text sample pairs: [(original sample image_1, sample description text_1), (original sample image_2, sample description text_2)……(original sample image_k, sample description text_k)].

[0087] The sample noise feature is a visual coding feature of the sample noise image, which is a high-dimensional and abstract vector representation, for example, a feature sequence in the case of 1xN dimension. The sample noise feature is usually completed by a convolutional neural network (CNN) or other types of feature extraction units, the purpose is to convert the sample noise image into a quantized coding form that can be understood and operated by a computer, and is the direct basis for subsequent inference denoising to generate a predicted image. Continuing the example in step 104, a pre-trained feature extraction network such as a convolutional neural network (CNN) is used to encode the sample noise image and extract a high-dimensional feature vector representing elements such as women, the sea, the beach, and noise characteristics. This vector is the sample noise feature.

[0088] The target image generation model is used to infer and denoise the random noise feature based on the description text to generate a plurality of target images corresponding to the description text of the target content. One optional way is to use the target image generation model to infer and denoise the random noise feature based on the description text under the attention mechanism to generate a plurality of target images corresponding to the description text of the target content. Another optional way is to use the target image generation model to infer and denoise the random noise feature based on the description text by conditional random field (CRF) modeling to generate a plurality of target images corresponding to the description text of the target content. Still another optional way is to use the target image generation model to infer and denoise the random noise feature based on the description text by variational auto-encoder (VAEs) to generate a plurality of target images corresponding to the description text of the target content, which is not limited here.

[0089] Optionally, the plurality of sample noise features are obtained by adding noise and feature coding to the plurality of sample original images, including:

[0090] The plurality of sample noise features are obtained by adding noise, feature coding, and feature splicing to the plurality of sample original images, specifically:

[0091] The plurality of sample noise features are spliced to obtain spliced noise features.

[0092] Correspondingly, the plurality of predicted images are generated by inferring and denoising the plurality of sample noise features based on the sample description text of the plurality of sample original images, specifically: the plurality of predicted images are generated by inferring and denoising the spliced noise features based on the sample description text of the plurality of sample original images.

[0093] For example, a general diffusion Transformer model is obtained as a target image generation model. The description texts of five cartoon characters are input into the target image generation model, and based on the five description texts, the inference denoising of random Gaussian noise features is performed under the attention mechanism to generate the five description texts corresponding to the expression packs of the cartoon characters, thereby forming an expression pack group of the cartoon characters. The expression pack group is rendered in the front end of the image drawing application program, and in the case of user confirmation, it is downloaded to the local and published in the instant messaging software of the application expression pack group.

[0094] In the embodiments of the present specification, after the plurality of sample original images are respectively added with noise, the plurality of sample noise features are respectively encoded to obtain a plurality of sample noise features as a whole input feature. Based on the sample description texts of the plurality of sample original images, the plurality of sample noise features are inferred and denoised to generate a plurality of predicted images. The initial image generation model simultaneously perceives the sample content in the plurality of sample original images. Based on the plurality of predicted images and the plurality of sample original images, the target image generation model is trained to enable the image generation model to learn to establish content correlation between generated images in the inference denoising process, thereby avoiding content fragmentation between the plurality of output images, improving the model generalization capability. Moreover, the target image generation model with high generalization capability can be well adapted to more image generation applications in various fields, thereby reducing the complexity of model training, improving the flexibility of model training, and further realizing the generation of images with high content correlation, improving the efficiency and accuracy of image generation.

[0095] In some image generation tasks, it is necessary to allow a small part of the plurality of images to be generated as a condition (complete visibility) in the process of simultaneously generating a group of pictures, and to generate another part of the plurality of images. In this way, the function of "image generating image" is realized. For example, image style transfer task, picture book generation task, expression pack group generation task and object multi-view generation task.

[0096] In an optional embodiment of the present specification, the plurality of sample noise features are obtained by adding noise to part of the image content of the first sample original image and retaining other sample original images, and the first sample original image is at least one of the plurality of sample original images.

[0097] Before step 104, the following specific steps are further included: obtaining an original image of the target content;

[0098] Correspondingly, step 104 includes the following specific steps: using the target image generation model, based on the description text and the original image, the inference denoising of the random noise features is performed to generate a plurality of target images corresponding to the description text of the target content.

[0099] Exemplarily, in the training stage, a plurality of sample noise features are obtained by adding noise to part of the image content of a first sample original image and retaining other sample original images, and the remaining model structure, generation method, input and output are completely consistent. That is, in the process of simultaneously generating a group of images, part of the image content in the group of images is allowed to be used as a condition (complete visibility) to generate another part of the image content, so as to realize the function of generating a target image guided by the original image of the target content.

[0100] The original image of the target content is a guide image for guiding the target image generation model to generate a target image. The original image provides visual details of the same target content for the generation of the target image, and provides a visual modal guide condition for the target image generation model to generate a target image. For example, the target image generation model generates a side view and a top view of an object, and the original image is a front view of the object. For another example, the target image generation model generates a photo of a person in a certain light and a certain pose, and the original image is a photo of the person in another light and another pose.

[0101] Using the target image generation model, based on the description text and the original image, the random noise feature is inferred and denoised to generate a plurality of target images corresponding to the description text of the target content. Using the target image generation model, based on the description text and the original image, the random noise feature is inferred and denoised to generate a plurality of target images corresponding to the description text of the target content. One optional way is to use the target image generation model, based on the description text, to infer and denoise the random noise feature under the attention mechanism to generate a plurality of target images corresponding to the description text of the target content. Another optional way is to use the target image generation model, based on the description text, to infer and denoise the random noise feature under the conditional random field (CRF) modeling to generate a plurality of target images corresponding to the description text of the target content. Still another optional way is to use the target image generation model, based on the description text, to infer and denoise the random noise feature under the variational auto-encoder (VAEs) to generate a plurality of target images corresponding to the description text of the target content, which is not limited here.

[0102] For example, an example expression package of the cartoon character is obtained, and the description text and the example expression package of the 5 cartoon characters are input into the target image generation model. Based on the 5 description texts and the example expression packages, the random Gaussian noise feature is inferred and denoised under the attention mechanism to generate 5 expression packages corresponding to the description texts of the cartoon character, which constitute an expression package group of the cartoon character.

[0103] In the embodiments of the present specification, the original image is supported as a guiding condition of the visual mode to guide the generation of the target image, and part of the image is supported as a condition to generate another part of the image. On the basis of the "text-to-image" function with high content correlation, the "image-to-image" function is combined to widen the application scenarios of image generation, improve the model performance and the flexibility of model training, and further improve the content correlation of the generated image, and further improve the image generation accuracy.

[0104] In an optional embodiment of the present specification, the target image generation model comprises an inference denoising layer.

[0105] Correspondingly, step 104 comprises the following specific steps: inputting the plurality of description texts and the random noise features into the inference denoising layer, and performing inference denoising on the random noise features based on the plurality of description texts to generate a plurality of target images corresponding to the description text of the target content.

[0106] Illustratively, the inference denoising layer is a model function layer for inference denoising in the initial image generation model, including but not limited to at least one of the following: attention model (Transformer model), diffusion model (Diffusion model), diffusion Transformer model and generative adversarial model (GAN model).

[0107] Optionally, FIG. 2 shows a structural schematic diagram of an image generation model according to an embodiment of the present specification, as shown in FIG. 2:

[0108] The image generation model comprises three noise adding layers: noise adding layer_1, noise adding layer_2 and noise adding layer_3. The three noise adding layers are connected to the corresponding three feature encoding layers: feature encoding layer_1, feature encoding layer_2 and feature encoding layer_3. The three noise adding layers are connected to the feature splicing layer, and the feature splicing layer is connected to the diffusion attention model as the inference denoising layer. Meanwhile, the text embedding layer of the predetermined label y and time step t is also connected to the diffusion attention model for introducing the sample description text. The diffusion attention model is sequentially connected to the regularization layer and the linear reshaping layer. In the linear reshaping layer, the output sequence is re-divided into a plurality of image feature sequences, which are respectively input into the three denoising layers: denoising layer_1, denoising layer_2 and denoising layer_3 to complete the denoising. The linear reshaping layer is simultaneously connected to a mean square error calculation layer.

[0109] Optionally, FIG. 3 shows a structural schematic diagram of an inference denoising layer in an image generation model according to an embodiment of the present specification, as shown in FIG. 3:

[0110] The diffusion attention model as the inference denoising layer comprises two inputs of image input and text input, wherein the image input is the spliced noise feature, and the text input is a plurality of sample description texts.

[0111] The concatenated noise feature input regularization layer, meanwhile, the multiple sample description texts pass through the multi-layer perceptron, and the output of the regularization layer is input into the sequence length changing layer, and then input into the cross-attention layer. Meanwhile, the multiple sample description texts pass through the multi-layer perceptron, and the output of the cross-attention layer is input into the sequence length changing layer. Meanwhile, the concatenated noise feature and the output of the sequence length changing layer are fused to obtain the cross-attention weighted feature.

[0112] The cross-attention weighted feature input regularization layer, meanwhile, the multiple sample description texts pass through the multi-layer perceptron, and the output of the regularization layer is input into the sequence length changing layer, and then input into the self-attention layer. Meanwhile, the multiple sample description texts pass through the multi-layer perceptron, and the output of the self-attention layer is input into the sequence length changing layer. Meanwhile, the self-attention feature and the output of the sequence length changing layer are fused to obtain the attention weighted feature.

[0113] The attention feature input regularization layer, meanwhile, the multiple sample description texts pass through the multi-layer perceptron, and the output of the regularization layer is input into the sequence length changing layer, and then input into the feedforward layer. Meanwhile, the multiple sample description texts pass through the multi-layer perceptron, and the output of the feedforward layer is input into the sequence length changing layer. Meanwhile, the feedforward fusion feature and the output of the sequence length changing layer are fused to obtain the output feature.

[0114] The regularization layer can adopt batch normalization (Batch Normalization), layer normalization (Layer Normalization) or other forms of normalization technology. The sequence length changing layer adapts to the input requirements of the attention layer and the feedforward layer, and ensures that the feature sequence has correct dimensions and shapes. The feedforward layer usually contains one or more fully connected layers for further extracting and converting features.

[0115] In the embodiments of the present specification, the architecture of the unified structure inference denoising layer is adopted, which reduces the complexity of data transmission and parameter adjustment between models, and provides flexibility for continuous optimization and adaptation of new tasks of the model.

[0116] The target image generation model obtained by training can complete multiple object image generation tasks after large-scale training on a large number of group images / video data, such as picture books, emoticons, photo albums, font packs, split-screen video generation, and photo editing, light / shape / expression editing, etc. In addition, the model also has a certain task generalization ability, and can realize the function not seen during training. Through the interactive form, the corresponding object image generation task is performed based on the prompt information provided by the object image generation task, and specific reference is made to FIG. 4, which shows a flowchart of an object image generation method according to an embodiment of the present specification. The method is applied to a server and includes the following specific steps:

[0117] Step 402: receiving a task request sent by a user terminal, wherein the task request carries prompt information of an object image generation task.

[0118] Exemplarily, the embodiments of the present specification are applied to a server of an application program, a website or a mini program with an object generation function. On the application program, the website or the mini program, an object image generation task is performed. For example, a website deploying a large language model can perform multiple human image generation, multiple animal image generation, multiple cartoon character generation, object multi-view generation and the like object image generation tasks, for another example, an image processing application program can perform multiple human image generation, multiple animal image generation, multiple cartoon character generation and the like object image generation tasks, for another example, an image processing application program can perform multiple human image generation, image style transfer and the like object image generation tasks, for another example, an image drawing application program can perform object image generation tasks of picture book generation, for another example, a software development application program can perform object image generation tasks of expression package set generation, for another example, a design application software can perform object image generation tasks of object multi-view generation, and the like, which are not limited herein. The above-mentioned application program can deploy a target image generation model locally, or remotely call a target image generation model deployed on a cloud side through an application programming interface, which is not limited herein.

[0119] The task request is a request sent by a user to a server, requiring to perform a specific object image generation task. The task request usually contains specific requirements such as image type, style, content and the like that the user wants to generate, and possible sample or conditional images.

[0120] The object image generation task is an image generation task of generating an image set of a target object according to the prompt information specified by the user. The object image generation task can be a task of generating a picture book, an expression package, a photo album, a font package, a shot video, and a retouching, light / pose / expression editing and the like.

[0121] The prompt information of the object image generation task is a specific instruction or description provided by the user when initiating the object image generation task. The prompt information guides the target image generation model to understand the user's requirements, such as the content, style, emotional color, etc. of the image. For example, "generate a set of portrait photos (single person)", "generate a set of pictures with the same character identification (multiple people)", "generate a set of picture book stories", "generate a set of creative stickers", "generate different perspectives of the same theme", "help me generate 4 pictures of a black cat eating, with time sequence, and consistent theme identification and details", "help me generate a picture of a living room three-view, with spatial perception ability, and consistent object layout", "help me generate a picture of a game scene from the main view, side view, and top view, with spatial perception ability, and consistent object layout", "help me generate a sketch of a forest and then convert it into a real photo", "help me generate a teddy bear and its passport photo in a suit", "generate different poses of the same character in Figure 1 given Figure 2 as a condition", "generate a line drawing of Figure 1 given Figure 2 as a condition", "generate a colorized result of a line drawing of Figure 1 given Figure 2 as a condition", "generate different action / costume / background images of the same IP in Figure 1 given Figure 2 as a condition", "generate a side view / front view of Figure 1 given the rear view of Figure 2 as a condition", "remove the text and generate Figure 1 given Figure 2 as a condition".

[0122] For example, a certain video production application, the user terminal initiates to generate a video of a cat, the server calls the target image generation model through the application programming interface, and receives the task request sent by the user terminal on the model platform where the target image generation model is deployed, wherein the task request carries the prompt information of the object image generation task "help me generate 4 pictures of a black cat eating, with time sequence, and consistent theme identification and details".

[0123] Receiving the task request sent by the user terminal, wherein the task request carries the prompt information of the object image generation task. This provides information support for subsequent determination of the description text of the target object.

[0124] Step 404: determining the description text of the target object based on the prompt information.

[0125] Exemplarily, the target object is a core element that the generated target image needs to represent, including but not limited to at least one of the following: a person, an animal, a plant, a commodity, an article, and a building, etc.

[0126] The description text of the target object is natural language text used to describe the target object, and the description text describes the visual details of the target object. The description text is the basis for the image generation model to generate images, and the model will generate target images according to the description.

[0127] Based on the prompt information, a plurality of description texts of the target object are determined. An optional manner is to query a pre-constructed description text database based on the prompt information to obtain a plurality of description texts of the target object. An optional manner is to generate a plurality of description texts of the target object based on the prompt information, which is not limited herein.

[0128] It should be noted that different object image generation tasks are performed by determining different description texts. For example, two scene images, "Xiaoming is eating" and "Xiaoming is kicking a ball", are respective description texts, and the object image generation task is a generation task of maintaining the user identifier. If the description is "A style cat" and "A style football field", the object image generation task is a generation task of maintaining the image style. Based on the prompt information, a plurality of description texts of the target object are determined, and a large number of different object image generation tasks can be freely implemented.

[0129] For example, based on the prompt information "Help me generate 4 pictures of a black cat eating food, with time sequence, and the theme identifier and details must be consistent", four description texts of the target object are generated: description text_1: "A black cat is sitting in front of its food bowl, staring at the food in the bowl with a focused gaze, ready to start eating. The outline of the cat is clear in the picture, and the background is a warm room with soft light." Description text_2: "The black cat has started eating, and its mouth is gently touching the food in the bowl, and the tail is gently swinging, showing enjoyment of the food. The amount of food in the bowl is slightly less than the first picture, showing that it is being eaten." Description text_3: "The eating process is approaching the end, and there is only a small amount of food left in the black cat's bowl. It licks its mouth, as if cleaning the last food residue. The cat looks very satisfied, and the body is relaxed, and the details of the room in the background remain unchanged." Description text_4: "The cat finishes the last bite of food and licks the bottom of the bowl, which is completely empty. The cat looks very comfortable and stretches a lazy pose. The background environment remains consistent with the previous images to ensure series coherence and theme consistency."

[0130] Based on the prompt information, a plurality of description texts of the target object are determined, which provides text condition support for subsequent reasoning denoising.

[0131] Step 406: using a target image generation model, based on the description text, reasoning and denoising random noise features to generate multiple target images of the target object corresponding to the description text, wherein the target image generation model is trained based on the initial image generation model based on multiple prediction images and multiple sample original images of sample content, the multiple prediction images are generated by reasoning and denoising multiple sample noise features based on multiple sample original images and sample description texts, and the multiple sample noise features are obtained by adding noise and feature encoding to the multiple sample original images.

[0132] Illustratively, the target image is a visual image of the target object corresponding to the description text generated by the target image generation model, and the target image is generated by the target image generation model based on the description text as the text condition. The multiple target images respectively correspond to the description text used to generate themselves, and the multiple target images corresponding to the description text have high object relevance.

[0133] For example, according to the training method of the image generation model described above, a general diffusion Transformer model is trained to obtain a target image generation model. Four description texts are input into the target image generation model, and based on each description text, random Gaussian noise features are reasoned and denoised to generate four video frames of a cat corresponding to the four description texts, and a video of a black cat eating is obtained.

[0134] Using the target image generation model, based on the description text, reasoning and denoising random noise features to generate multiple target images of the target object corresponding to the description text, realizing the generation of object images with high content correlation, improving the efficiency and accuracy of object image generation.

[0135] Step 408: feeding back the multiple target images to the user terminal.

[0136] For example, the server feeds back the video of the black cat eating to the user terminal through an application programming interface, and renders and displays it on the client side of a certain video production application.

[0137] In the embodiments of the present specification, the user terminal and the server are interacted to realize the generation of object images with high content correlation, improve the efficiency and accuracy of object image generation, and improve the user experience.

[0138] Generally, if multiple target images are to be generated, multiple description texts are to be provided in advance, which may increase the difficulty of creation and reduce the content correlation between the generated target images, thereby reducing the efficiency of object image generation. Therefore, a text generation model can be used to generate multiple description texts of the target object based on the prompt information, thereby reducing the difficulty of creation, ensuring the content correlation between the generated target images, and improving the efficiency of object image generation.

[0139] In an optional embodiment of the present specification, step 404 comprises the following specific steps: generating a plurality of description texts of the target object based on the prompt information by using a text generation model.

[0140] Illustratively, the text generation model is a deep learning model with the function of generating corresponding natural language texts under the guidance of the prompt information. The text generation model converts abstract user requests into specific description texts that can be used for image generation, thereby helping the image generation model better understand and create the required content.

[0141] For example, using a pre-trained generative Transformer model (Generative Pre-trained Transformer, GPT), based on the prompt information "Help me generate 4 pictures of a black cat eating, with time sequence, and theme identification and details must be consistent", four description texts of a cat are generated: Description text_1: "A black cat is sitting in front of its food bowl, eyes focused on the food in the bowl, ready to start eating. The outline of the cat is clear in the picture, and the background is a warm room with soft light." Description text_2: "The black cat has started eating, its mouth lightly touching the food in the bowl, and the tail is gently swaying, showing enjoyment of the food. The amount of food in the bowl is slightly less than the first picture, showing that it is being eaten." Description text_3: "The eating process is approaching the end, and there is only a small amount of food left in the black cat's bowl. It licks its mouth, as if cleaning up the last food residue. The cat looks very satisfied, and the body is relaxed. The details of the room in the background remain unchanged." Description text_4: "The cat has finished eating the last bite of food and licked the bottom of the bowl. The cat looks very comfortable and stretches a lazy pose. The background environment remains consistent with the previous images to ensure series coherence and theme consistency."

[0142] In the embodiments of the present specification, the text generation model is used to generate a plurality of description texts of the target object based on the prompt information, which reduces the difficulty of creation, ensures the content correlation between the generated target images, and improves the efficiency of object image generation.

[0143] The following gives some examples of prompt information and generated target image effects for object image generation tasks:

[0144] The prompt information is "generate a set of portrait photos (single person)", and after the processing of steps 404-408, the target image rendered on the user terminal is shown in FIG. 5, which is an effect schematic diagram of an object image generation method applied to portrait generation according to an embodiment of the present specification:

[0145] Generate a photo album of a girl in four different poses, although the poses are inconsistent, but the target object is the same girl.

[0146] The prompt information is "help me generate a sketch of a forest and then convert it into a real photo". After the processing of steps 404-408, the target image rendered on the user terminal is shown in FIG. 6, which is an effect diagram of an object image generation method applied to sketch coloring according to an embodiment of the present specification:

[0147] In the generated sketch and the colored real photo, the target object is always the same forest, and the lines are consistent.

[0148] The prompt information is "generate the coloring result of the sketch in FIG. 1 under the condition of FIG. 2". After the processing of steps 404-408, the target image rendered on the user terminal is shown in FIG. 7, which is an effect diagram of an object image generation method applied to sketch-to-photo conversion according to an embodiment of the present specification:

[0149] In the colored picture, the target object is always the same scene, and the lines are consistent.

[0150] The prompt information is "generate a set of creative sticker packs". After the processing of steps 404-408, the target image rendered on the user terminal is shown in FIG. 8, which is an effect diagram of an object image generation method applied to picture book generation according to an embodiment of the present specification:

[0151] Generate a picture book of two children playing in the snow, including four pictures, and the four pictures have story coherence and character consistency.

[0152] The prompt information is "generate a set of creative sticker packs". The target image rendered on the user terminal is shown in FIG. 9, which is an effect diagram of an object image generation method applied to sticker pack generation according to an embodiment of the present specification:

[0153] Generate four sticker packs of a cartoon character, forming a sticker pack, although the actions and scenes are inconsistent, but the target object is the cartoon character.

[0154] The prompt information is "generate different perspectives of the same theme". The target image rendered on the user terminal is shown in FIG. 10, which is an effect diagram of an object image generation method applied to multi-perspective view generation of an object according to an embodiment of the present specification:

[0155] Generate four perspective views of a desktop ornament, and the target object in the four perspective views is always the same desktop ornament.

[0156] Referring to FIG. 11, FIG. 11 shows a flowchart of a method for training an image generation model according to an embodiment of the present specification, including the following specific steps:

[0157] Step 1102: Obtain a plurality of sample original images of sample content and sample description texts of the plurality of sample original images.

[0158] Exemplarily, the embodiments of the present specification are applied to an application or platform with a model training function, and a large number of processing units for training a deep learning model are deployed on the application or platform, for example, GPU (Graphic Processing Unit), TPU (Tensor Processing Unit), NPU (Neural Processing Unit), VPU (Vector Processing Unit), etc.

[0159] The sample content is a core element that needs to be expressed in the sample original image, including but not limited to at least one of the following: a sample object, a sample theme, a sample style, a sample scene, and a sample color tone.

[0160] The sample original image is a visual image containing sample content, and the plurality of sample original images all contain sample content, constituting a sample image group, and the plurality of sample original images have content correlation.

[0161] The sample description text is a natural language text for describing the sample content in the sample original image, and the sample description text describes the visual details of the sample content in the sample original image. Any pair of a sample original image and a sample description text of the sample original image constitutes a set of image-text sample pairs with content correlation, and a plurality of image-text sample pairs constitute a set of image-text sample pairs: [(sample original image_1, sample description text_1), (sample original image_2, sample description text_2) … (sample original image_k, sample description text_k)].

[0162] The plurality of sample original images of sample content and the sample description texts of the sample original images can be obtained from a public data set, for example, obtained from an open source sample database, obtained from a historical database, for example, obtained from an advertisement database, obtained from an advertisement image and advertisement text, or generated by a generation model, for example, generated by a pre-trained generative adversarial model, without limitation.

[0163] For example, a image generation model for inputting description text to generate a portrait of a person is needed to be trained, in response to the model training request, 1000 portrait sets of 100 people are obtained from the historical database provided by the terminal device, each portrait set includes 100 photos of different poses, different illuminations and different expressions as sample original images. A pre-trained text generation model for image description is used to generate a sample description text for each picture. 10000 image-text sample pairs with content correlation are obtained, and the 10000 image-text sample pairs constitute a set of image-text sample pair list: [(sample original image_1, sample description text_1), (sample original image_2, sample description text_2) … (sample original image_10000, sample description text_10000)].

[0164] Obtaining a plurality of sample original images of sample content and sample description texts of the sample original images provides sample data for subsequent training of the image generation model.

[0165] Step 1104: Adding noise to each of the plurality of sample original images to obtain a plurality of sample noise images.

[0166] Exemplarily, the sample noise image is a visual image in which noise is added to the sample original image. Since the plurality of sample original images all contain sample content, they constitute a sample image group, and therefore the plurality of sample noise images also have content correlation. The sample noise image can be fully added with noise or partially added with noise, which is not limited herein. The same intensity of noise can be added to the plurality of sample noise images, or different intensities of noise can be added to the plurality of sample noise images, which is not limited herein. The added noise includes, but is not limited to, Gaussian noise, exponential noise, Rayleigh noise, gamma noise, Poisson noise and uniform noise. For example, a sample original image contains sample content of a woman walking on the beach. Gaussian noise is added to the sample original image to obtain a sample noise image.

[0167] For example, Gaussian noise is added to each of the 10000 sample original images to obtain 10000 sample noise images.

[0168] Each sample original image is added with noise to obtain a plurality of sample noise images, which provides noise image support for subsequent inference and denoising based on the sample description text using the model to generate a predicted image.

[0169] Step 1106: Feature encoding is performed on each of the plurality of sample noise images to obtain a plurality of sample noise features.

[0170] Exemplarily, the sample noise feature is a visual coding feature of the sample noise image, which is a high-dimensional and abstract vector representation, for example, a feature sequence in the case of a dimension of 1N. The sample noise feature is usually completed by a convolutional neural network (CNN) or other types of feature extraction units, and the purpose is to convert the sample noise image into a quantized coding form that can be understood and operated by a computer, which is the direct basis for subsequent inference denoising to generate a predicted image. Continuing the example in step 104, a pre-trained feature extraction network such as a convolutional neural network (CNN) is used to encode the sample noise image and extract a high-dimensional feature vector representing elements such as women, the sea, and the beach, as well as noise characteristics. This vector is the sample noise feature.

[0171] Each sample noise image is respectively encoded to obtain a plurality of sample noise features. An optional way is to respectively perform block feature encoding on each sample noise image to obtain a plurality of sample noise features, which can capture local features and possibly enhance the model's understanding of image details. Another optional way is to respectively perform multi-scale feature encoding on each sample noise image to obtain a plurality of sample noise features, which can capture multi-level information in the image. Still another optional way is to respectively perform attention feature encoding on each sample noise image to obtain a plurality of sample noise features, which can make the model focus on the key areas of the image, that is, help the model learn more global and coherent feature representations, which are not limited here.

[0172] For example, the feature extraction layer of the convolutional neural network is used to respectively perform block feature encoding on each group of 100 sample noise images to obtain 100 groups, each group having 100 sample noise features.

[0173] After respectively adding noise to each sample original image, the plurality of sample noise images are respectively encoded to obtain input features for subsequent inference denoising by the model.

[0174] Step 1108: Using the initial image generation model, based on the plurality of sample description texts, the plurality of sample noise features are inferred and denoised to generate a plurality of predicted images.

[0175] Exemplarily, the image generation model is a deep learning model for generating corresponding images based on description texts. The image generation model is guided to generate images step by step through inference denoising with description texts as text conditions. The image generation model includes but is not limited to at least one of the following: an attention model (Transformer model), a diffusion model (Diffusion model), a diffusion Transformer model, and a generative adversarial model (GAN model).

[0176] The initial image generation model is an untrained image generation model that has general inference denoising capability and can generate images step by step guided by sample description text as a text condition. However, the initial image generation model can only generate one image at a time, and when applied to generate multiple images, the content correlation between multiple images generated multiple times is insufficient, and there is a problem of content fragmentation between multiple images. Optionally, the initial image generation model can only include an inference denoising layer for inference denoising, or can further include at least one of a noise adding layer for feature noise, a feature encoding layer for feature encoding, and a feature splicing layer for feature splicing.

[0177] The predicted image is a visual image of the sample content corresponding to the sample description text generated by the initial image generation model. The predicted image is generated by the initial image generation model guided by the sample description text as a text condition. Multiple predicted images correspond to the sample description text used to generate themselves, respectively. The predicted image is a visual image restored from the sample original image. For example, the sample original image is a photo of a specific pose, specific lighting, and specific expression in a portrait of a person, and the predicted image is a visual image restored from the photo. For another example, the sample original image is a video frame at a specific moment in a video of an animal, and the predicted image is a visual image restored from the video frame. For another example, the sample original image is an illustration with a sample theme of “spring in full bloom”, and the predicted image is a visual image restored from the illustration. For another example, the sample original image is a character effect drawing in “animation style”, and the predicted image is a visual image restored from the character effect drawing. For another example, the sample original image is a tourist photo of a certain tourist attraction, and the predicted image is a visual image restored from the tourist photo. For another example, the sample original image is an advertisement image with a sample color tone of “warm color”, and the predicted image is a visual image restored from the advertisement image.

[0178] The initial image generation model is used to infer and denoise the multiple sample noise features based on the multiple sample description texts to generate multiple predicted images. In an optional manner, the initial image generation model is used to infer and denoise the multiple sample noise features under an attention mechanism based on the multiple sample description texts to generate multiple target images corresponding to the multiple sample description texts of the target content. In another optional manner, the initial image generation model is used to infer and denoise the multiple sample noise features based on the multiple sample description texts to generate multiple target images corresponding to the multiple sample description texts of the target content through Conditional Random Fields (CRF) modeling. In another optional manner, the initial image generation model is used to infer and denoise the multiple sample noise features based on the multiple sample description texts to generate multiple target images corresponding to the multiple sample description texts of the target content through Variational Auto-Encoders (VAEs) modeling. In this case, no limitation is made.

[0179] For example, a general diffusion Transformer model is obtained from an open source model library as the initial image generation model. A set of 100 sample description texts and 100 sample noise features are input into the diffusion Transformer model, and the diffusion Transformer model is used to infer and denoise the 100 sample noise features under an attention mechanism based on the 100 sample description texts to generate 100 photos of different poses, different illuminations, and different expressions, thereby restoring the photo album of the person.

[0180] The multiple sample noise features are simultaneously used as input features, and the initial image generation model is used to infer and denoise the multiple sample noise features based on the multiple sample noise features of the multiple sample original images to generate multiple predicted images. The initial image generation model simultaneously perceives the sample content in the multiple sample original images, so that the image generation model can learn to establish the content correlation between the generated images in the inference and denoising process.

[0181] Step 1110: The initial image generation model is trained based on the multiple sample original images and the multiple predicted images to obtain a target image generation model.

[0182] Exemplarily, the target image generation model is a trained image generation model, and the target image generation model is trained based on the plurality of predicted images and the plurality of sample original images of the sample content. The target image generation model has inference denoising capability for a plurality of images, can generate a plurality of images step by step guided by the description text as a text condition, can generate a plurality of images at a time, is suitable for a plurality of image generation fields, and has high content correlation between the generated plurality of images. Optionally, the target image generation model can only include an inference denoising layer for inference denoising, or can further include at least one of a noise adding layer for feature noise adding, a feature encoding layer for feature encoding, and a feature splicing layer for feature splicing.

[0183] The initial image generation model is trained based on the plurality of sample original images and the plurality of predicted images to obtain the target image generation model. An optional manner is that a loss value is calculated based on the plurality of sample original images and the plurality of predicted images, and model parameters of the initial image generation model are adjusted based on the loss value. When a preset training end condition is reached, the target image generation model is obtained. The loss value can be a sum of squares of errors (i.e., mean square error loss, MSE) between the sample original images and the predicted images, or can be any one of mean absolute error (MAE), cross entropy loss, and adversarial loss. The preset training end condition can be any one of a maximum number of iterations, a performance stability index on a validation set, a learning rate threshold, and a loss value threshold, which are not limited herein. The model parameters of the initial image generation model are adjusted based on the loss value. An optional manner is that the model parameters of the initial image generation model are adjusted by using a gradient update method based on the loss value.

[0184] For example, a loss value is calculated based on a sum of squares of errors between the sample original images and the predicted images for each group of 100 photos, and the model parameters of the general diffusion Transformer model are adjusted by using a gradient update method based on the loss value. After 100 groups (each group is used as training data for current iteration) of iterative training are completed, a trained diffusion Transformer model is obtained as the target image generation model, which is used to perform a portrait generation task of generating a portrait of a person based on input description text.

[0185] In the embodiments of the present specification, after adding noise to the plurality of sample original images respectively, the plurality of sample noise images are respectively feature encoded to obtain a plurality of sample noise features, the plurality of sample noise features are simultaneously used as an input feature, an initial image generation model is used to infer and denoise based on the sample description text of the plurality of sample original images to generate a plurality of predicted images, the initial image generation model simultaneously perceives the sample content in the plurality of sample original images, and a target image generation model is trained based thereon, so that the image generation model can learn to establish content correlation between generated images in the inference and denoising process, avoiding content fragmentation between the plurality of output images, improving the model generalization capability, and the target image generation model with high generalization capability can be well adapted to image generation applications in more fields, reducing the complexity of model training and improving the flexibility of model training.

[0186] Due to the limitation of the model structure, the plurality of sample noise features can be spliced into a whole feature, which is input into the initial image generation model to generate a plurality of predicted images by inference and denoising based on the guidance of the plurality of sample description texts, so that the initial image generation model can simultaneously perceive the sample content in the plurality of sample original images and learn to establish content correlation between generated images.

[0187] In an optional embodiment of the present specification, before step 1108, the following specific steps are further included: feature splicing is performed on the plurality of sample noise features to obtain spliced noise features;

[0188] Correspondingly, step 1108 includes the following specific steps: using the initial image generation model to infer and denoise the spliced noise features based on the plurality of sample description texts to generate a plurality of predicted images.

[0189] The spliced noise features are visual coding features spliced from the plurality of sample noise features, which are also a kind of high-dimensional and abstract vector representation. For example, the M sample noise features are all feature sequences with a dimension of 1xN, and the spliced noise features are a long feature sequence with a dimension of 1x(M*N). The spliced noise features retain the content correlation between the sample noise images, so that the information of all images can be considered in the inference and denoising process in subsequent step 108, thereby maintaining continuity and consistency when generating predicted images. The model learns to establish content correlation between generated images in the above training, avoiding content fragmentation between the plurality of output images.

[0190] For example, the feature splicing is performed on each set of 100 sample noise features to obtain 100 sets of spliced noise features. The 100 sample description texts in a set and the spliced noise features of the set are input into the diffusion Transformer model, and the diffusion Transformer model is used to perform inference denoising on the spliced noise features under the attention mechanism based on the 100 sample description texts to generate 100 photos of different postures, different illuminations, and different expressions, and restore the photo album of the person.

[0191] In the embodiments of the present specification, the feature splicing is performed on the plurality of sample noise features to obtain spliced noise features as a whole input feature, and the initial image generation model is used to perform inference denoising based on the sample description texts of the sample original images to generate a plurality of predicted images, so that the initial image generation model can perceive the sample content in the plurality of sample original images at the same time, learn to establish the content correlation between the generated images, better avoid the content fragmentation between the plurality of output images, and further improve the model generalization ability.

[0192] The sample noise feature sequences corresponding to the plurality of sample noise images are spliced into a long sequence to obtain a spliced noise feature sequence, which is input into the initial image generation model. In this way, the model can naturally perceive the sample content in all images at the same time. The model will learn to establish content correlation in the generation process. The long sequence can be encoded by using the block feature coding mode. In combination with the feature element splicing, the model can generate a group of images, which not only correspond to the respective description texts, but also maintain consistency in content with each other, thereby enhancing the diversity and continuity of the generated images.

[0193] In an optional embodiment of the present specification, any sample noise feature is a feature sequence, and the spliced noise feature is a feature sequence.

[0194] Correspondingly, the feature encoding is performed on the plurality of sample noise images to obtain a plurality of sample noise features, including the following specific steps: performing image blocking on the plurality of sample noise images; and performing feature encoding on the image blocks of the plurality of sample noise images to obtain a plurality of sample noise feature sequences, wherein any sample noise feature sequence includes feature elements of the image blocks of the sample noise image.

[0195] Correspondingly, the feature splicing is performed on the plurality of sample noise features to obtain spliced noise features, including the following specific steps: splicing the feature elements of the plurality of image blocks in the plurality of sample noise feature sequences to obtain a spliced noise feature sequence.

[0196] Exemplarily, the sample noise feature sequence is a visual coding feature extracted from the sample noise image, is a high-dimensional and abstract vector representation, and includes feature elements of image blocks of the sample noise image. Each feature element of an image block is a visual coding feature of the image block of the sample noise image, and serves as a feature element (token) in image generation. The feature element of the image block represents color, texture, shape and other visual information of the image block.

[0197] The spliced noise feature sequence is a single long sequence formed by combining multiple sample noise feature sequences, and includes feature elements of image blocks of all sample noise images, and represents visual information of image blocks of multiple sample noise images.

[0198] The multiple sample noise images are respectively subjected to image blocking. In an optional manner, the multiple sample noise images are respectively subjected to image blocking according to a preset image block size. In another optional manner, the multiple sample noise images are respectively subjected to image blocking according to multiple image block sizes, which is not limited herein.

[0199] By means of block coding and feature element splicing, the sample noise image is divided into feature element sequences, which are input into the initial image generation model. The output of the initial image generation model is a feature element sequence with the same length as the input. Finally, the output sequence is restored into a visual image with the same size as the original sample image.

[0200] For example, a 1024x1024 pixel sample noise image is subjected to image blocking to form 16x16 pixel image blocks. The image blocks of the multiple sample noise images are subjected to feature coding, and thus 100 groups of sample noise feature sequences are obtained. Each group includes 100 sample noise feature sequences. The feature elements of the image blocks in each group of 100 sample noise feature sequences are spliced to obtain 100 groups of spliced noise feature sequences.

[0201] In the embodiments of the present specification, the initial image generation model perceives sample content in the image blocks of the multiple sample original images through block coding and splicing. On this basis, the target image generation model is trained to enable the image generation model to learn to establish content correlation between images in the image block granularity in the inference denoising process, thereby avoiding content fragmentation between the output images and further improving the model generalization capability.

[0202] Steps 1104-1108 can be implemented by different models or by a unified initial image generation model. In this case, in an optional embodiment of the present specification, the initial image generation model includes multiple noise adding layers, multiple feature coding layers, a feature splicing layer and an inference denoising layer.

[0203] In an optional embodiment of the present specification, the initial image generation model comprises a plurality of noise adding layers, a plurality of feature encoding layers, a feature splicing layer, and an inference denoising layer.

[0204] Correspondingly, step 1104 comprises the following specific steps: inputting the plurality of sample original images into the plurality of noise adding layers respectively, and splicing the plurality of sample noise features to obtain spliced noise features.

[0205] Correspondingly, step 1106 comprises the following specific steps: inputting the plurality of sample noise images into the plurality of feature encoding layers respectively, and encoding the plurality of sample noise images respectively to obtain the plurality of sample noise features.

[0206] Correspondingly, splicing the plurality of sample noise features to obtain spliced noise features comprises the following specific steps: inputting the plurality of sample noise features into the feature splicing layer, and splicing the plurality of sample noise features to obtain spliced noise features.

[0207] Correspondingly, step 1108 comprises the following specific steps: inputting the plurality of sample description texts and the spliced noise features into the inference denoising layer, and denoising the spliced noise features based on the plurality of sample description texts to generate a plurality of predicted images.

[0208] The noise adding layer is a model function layer in the initial image generation model for adding noise, and the noise adding layer can add at least one of the following, including but not limited to: Gaussian noise, exponential noise, Rayleigh noise, Gamma noise, Poisson noise, and uniform noise, on the input image. The feature encoding layer is a model function layer in the initial image generation model for image feature quantization and encoding, and includes at least one of the following, including but not limited to: a convolutional neural network (CNN), a recurrent neural network (RNN), a VGG network, and a ResNet network. The feature splicing layer is a model function layer in the initial image generation model for splicing a plurality of features, and includes at least one of the following, including but not limited to: a Concate layer, a weighted sum layer, and a gating mechanism layer (for example, a Gate Recurrent Unit (GRU) or a Long-Short Term Memory (LSTM)).

[0209] As shown in FIGS. 2 and 3, details are not described herein.

[0210] The specific manners of the steps in the embodiments of the present specification are described in the contents of steps 1104-1108, and details are not described herein.

[0211] In the embodiments of the present specification, each model function layer can be adjusted in the training process, or part of the model function layer can be frozen, and only the parameters of the remaining model function layer are adjusted, which is not limited herein.

[0212] In the embodiments of the present specification, the unified structure image generation model combines the architecture of the noise adding layer, the feature encoding layer, the feature splicing layer, and the inference denoising layer, simplifies the training process, reduces the complexity of data transmission and parameter adjustment between models, improves the training efficiency of the model, reduces the risk of overfitting, and enables the model to more effectively utilize computing resources. Moreover, the architecture allows easy addition or modification of individual components, such as changing the type of noise adding layer, adjusting the depth of the feature encoding layer, or introducing new inference denoising layer algorithms, providing flexibility for continuous optimization and adaptation to new tasks of the model.

[0213] In step 1104, if the same intensity of noise is added to the plurality of sample original images, the image generation model can learn to recognize image patterns at a specific noise level, but its performance will decrease when facing different noise levels or noise in the real world, forming unnecessary overfitting. Therefore, by adding different intensities of noise to obtain a plurality of sample noise images, the model generalization ability of the target image generation model obtained by training can be improved.

[0214] In an optional embodiment of the present specification, step 1104 comprises the following specific steps: adding different intensities of noise to the plurality of sample original images respectively to obtain a plurality of sample noise images.

[0215] For example, different intensities of noise are added to the plurality of sample original images, so that the noise added to the plurality of sample original images forms random changes, thereby achieving the goal of data enhancement in the training process. Adding different intensities of noise represents slight to significant fluctuations in pixel values, aiming to simulate multiple kinds of interference that the image may suffer in the real world, such as changes in illumination, sensor noise, compression artifacts, etc.

[0216] The noise includes but is not limited to at least one of the following: Gaussian noise, exponential noise, Rayleigh noise, gamma noise, Poisson noise, and uniform noise.

[0217] For example, different intensities of Gaussian noise are added to 10,000 sample original images to obtain 10,000 sample noise images.

[0218] In the embodiments of the present specification, by adding different intensities of noise, data enhancement of the sample noise image is achieved, so that the target image generation model obtained by training performs more stably when facing different noise levels or noise in the real world, avoiding unnecessary overfitting, and further improving the generalization ability of the model.

[0219] In some multiple image generation tasks, it is necessary to allow a part of multiple images to be generated as a condition (complete visibility) in the process of simultaneously generating a set of pictures, to generate another part of multiple images. In this way, the function of "picture generating picture" is realized. For example, image style transfer task, picture book generation task, expression package group generation task and object multi-view generation task.

[0220] In an optional embodiment of the present specification, step 1104 comprises the following specific steps: adding noise to part of the image content of the first sample original image, and retaining other sample original images to obtain a plurality of sample noise images, wherein the first sample original image is at least one of the plurality of sample original images.

[0221] Illustratively, the part of the image content of the first sample original image is a selected sub-region in the first sample original image. This part of the image can be of any size and shape. Optionally, the part of the image content contains sample content to improve the performance of the model.

[0222] It should be noted that in the embodiments of the present specification, under the premise of reusing steps 1102, 1106-1110, a part of the plurality of sample noise images is spliced with an image without added noise as input.

[0223] For example, Gaussian noise is added to part of the image content of 20 sample original images in 100 groups, and other sample original images are retained to obtain sample noise images.

[0224] In the embodiments of the present specification, the training of the image generation model is completed by adding noise to part of the image, which supports the generation of another part of multiple images as a condition for part of the image. On the basis of the "text generating picture" function with high content correlation, the "picture generating picture" function is combined to broaden the application scenario of image generation and improve the flexibility of model training and model performance.

[0225] In step 1108, the sample description text is used as a text condition to guide the step-by-step inference to generate a predicted image. The inference denoising process can use an attention mechanism to help the model focus on the visual information related to the sample content in the sample description text, so as to generate a visual image of the sample content corresponding to the sample description text more accurately according to the highly correlated visual information.

[0226] In an optional embodiment of the present specification, step 1108 comprises the following specific steps: using an initial image generation model, performing attention calculation on a plurality of sample noise features based on a plurality of sample description texts to obtain attention weighted features, and generating a plurality of predicted images based on the attention weighted features.

[0227] Exemplarily, attention calculation is a mechanism used in deep learning models, allowing the model to assign different weights to different inputs when processing the inputs, so as to more effectively process and understand the input information. In an image generation model, attention calculation enables the model to focus on the visual information in the sample description text that is highly relevant to the sample content, to guide the generation process of the predicted image. The attention calculation includes at least one of self-attention calculation and cross-attention calculation. The formula of the attention calculation is shown in Formula 1:

[0228] wherein Q is a query feature, K is a key feature, and V is a value feature. In the self-attention calculation, the query feature Q and the key feature K come from the same input, and in the cross-attention calculation, the query feature Q and the key feature K come from different inputs.

[0229] The attention weighted feature is a visual encoding feature obtained by weighting the spliced noise feature according to the attention weight determined based on the plurality of sample description texts after attention calculation. The attention weighted feature focuses on the visual information in the sample description text that is highly relevant to the sample content, and weakens the irrelevant or less relevant visual information, guiding the generation of a visual image of the sample content corresponding to the sample description text more accurately. For example, a sample description text is “a yellow little duck swims in the pond”. The attention calculation will analyze that “a yellow little duck” and “a pond” in the sample description text are highly relevant visual information, and then determine the feature elements in the spliced noise feature that match these descriptions, and assign higher attention weights to these feature elements, while reducing the weights of feature elements such as background trees and sky that are not directly related. When guiding the model to generate an image based on the attention weighted feature, a scene focusing on a yellow little duck swimming in a pond is generated preferentially, and other visual information is not paid too much attention to, the trees can be red or green, and the sky can be sunny or cloudy.

[0230] Using the initial image generation model, based on the plurality of sample description texts, performing attention calculation on the plurality of sample noise features to obtain an attention weighted feature, and based on the attention weighted feature, generating a plurality of predicted images. An optional way is to use the initial image generation model, based on the plurality of sample description texts, performing self-attention calculation on the plurality of sample noise features to obtain an attention weighted feature, and based on the attention weighted feature, generating a plurality of predicted images. Another optional way is to use the initial image generation model, based on the plurality of sample description texts, performing cross-attention calculation on the plurality of sample noise features to obtain an attention weighted feature, and based on the attention weighted feature, generating a plurality of predicted images. A further optional way is to use the initial image generation model, based on the plurality of sample description texts, performing self-attention calculation and cross-attention calculation on the plurality of sample noise features to obtain an attention weighted feature, and based on the attention weighted feature, generating a plurality of predicted images, which is not limited herein.

[0231] For example, a set of 100 sample description texts and 100 sample noise features are input into the diffusion Transformer model. Based on the 100 sample description texts, the diffusion Transformer model performs cross-attention calculation on the 100 sample noise features to obtain attention-weighted features. Based on the attention-weighted features, 100 photos of different poses, different illuminations, and different expressions are generated to restore the photo album of the person.

[0232] In the embodiments of the present specification, the attention mechanism is adopted to guide the generation of visual images corresponding to the sample description texts more accurately, so that the image generation model can learn to establish high content correlation between the generated images and the description texts with high accuracy during the inference denoising process, thereby improving the training effect and model performance.

[0233] The sample description texts can be introduced into the image generation model for inference denoising in the form of cross-attention calculation, and cross-attention calculation is respectively performed on the respective corresponding feature elements in the respective spliced noise features. Then, self-attention calculation is performed on the whole.

[0234] In an optional embodiment of the present specification, based on a plurality of sample description texts, an initial image generation model is used to perform attention calculation on spliced noise features to obtain attention-weighted features, and based on the attention-weighted features, a plurality of predicted images are generated. The specific steps include: based on a plurality of sample description texts, an initial image generation model is used to perform cross-attention calculation on a plurality of sample noise features respectively to obtain cross-attention-weighted features, self-attention calculation is performed on the cross-attention-weighted features to obtain attention-weighted features, and based on the attention-weighted features, a plurality of predicted images are generated.

[0235] By way of example, cross-attention calculation is a kind of attention mechanism, and the query feature Q and the key feature K come from different inputs. In the image generation model, the query feature usually comes from the sample description text, while the key feature and the value feature come from the spliced noise feature. By calculating the feature similarity between the query feature and the key feature, the model can determine the relevance of the text description and the image feature, so as to assign weights to different feature elements in the generation process.

[0236] The cross-attention-weighted feature is a visual coding feature obtained by weighting the spliced noise feature according to the attention weight determined based on a plurality of sample description texts after cross-attention calculation. The cross-attention-weighted feature helps the model focus on the image elements mentioned in the text description, ensuring that the generated image is highly matched with the sample description text. For example, the sample description text mentions “a yellow little duck swimming in the pond”, so the model will assign higher weights to the image features related to “yellow little duck” and “pond”.

[0237] Self-attention calculates a kind of attention mechanism, and the query feature Q and the key feature K come from the same input.In the image generation model, the query feature, the key feature and the value feature all come from the self-attention feature.By calculating the feature similarity between the feature elements of the self-attention feature, the model can determine the relationship between the visual information in the image, better understand the global structure of the image, and thus allocate weights to different feature elements in the generation process, which is crucial for overall understanding of sample content and generating coherent predicted images.

[0238] For example, input a set of 100 sample description texts and 100 sample noise features into the diffusion Transformer model, use the diffusion Transformer model to perform cross-attention calculation based on the 100 sample description texts and the 100 sample noise features respectively, obtain cross-attention weighted features, perform self-attention calculation on the cross-attention weighted features to obtain attention weighted features, and generate 100 photos with different poses, different illuminations and different expressions based on the attention weighted features to restore the photo album of the person.

[0239] In the embodiments of the present specification, the cross-attention mechanism is used to guide the generation of the visual image corresponding to the sample description text more accurately, so that the image generation model can learn to establish high content correlation between the generated image and the description text with high accuracy in the inference denoising process, and the self-attention mechanism is used to better understand the global structure of the image, guide the model to understand the sample content as a whole and generate coherent predicted images, and improve the training effect and model performance.

[0240] Referring to FIG. 12, FIG. 12 shows a flowchart of another method for training an image generation model according to an embodiment of the present specification, which is applied to a cloud training platform and includes the following specific steps:

[0241] Step 1202: receiving a model training request sent by a terminal device.

[0242] Step 1204: in response to the model training request, obtaining a plurality of sample original images of sample content and sample description texts of the plurality of sample original images.

[0243] Step 1206: adding noise to the plurality of sample original images respectively to obtain a plurality of sample noise images.

[0244] Step 1208: performing feature encoding on the plurality of sample noise images respectively to obtain a plurality of sample noise features.

[0245] Step 1210: using an initial image generation model to perform inference denoising on the plurality of sample noise features based on the plurality of sample description texts to generate a plurality of predicted images.

[0246] Step 1212: Based on the plurality of sample original images and the plurality of predicted images, the initial image generation model is trained to obtain a target image generation model.

[0247] Step 1214: The target image generation model is fed back to the terminal device.

[0248] Illustratively, the cloud training platform is a high-performance computing platform based on cloud computing technology, dedicated to the training of deep learning models. It is equipped with a large number of computing resources such as GPU, TPU, NPU and VPU, which can efficiently process large-scale data sets and complex model training tasks. The cloud training platform includes a request interface and a response unit. Request Interface: responsible for receiving model training requests from terminal devices. This may include model type, training data set, expected model performance indicators, etc. The request interface is the first entrance for cloud platform to interact with terminal devices, ensuring the safe transmission of data and the effective analysis of requests. Response Unit: After receiving the training request, the response unit begins to process the task, including data preprocessing, model initialization, training process control and result feedback, etc. It coordinates various computing resources to ensure that the training process is efficient and orderly.

[0249] The terminal device is a hardware that initiates the model training request, such as a personal computer, a smartphone, an Internet of Things device, etc. The terminal device is responsible for collecting or generating the data and description text required for training, and then sending these information to the cloud training platform through the network. Once the model training is completed, the terminal device will receive the trained model to perform image generation tasks in the local environment.

[0250] When the terminal device sends a model training request, the request interface of the cloud training platform first receives and analyzes the request, and then triggers the response unit to start executing the training task. The response unit obtains the sample original images and the corresponding description text from the data provided by the terminal device, and then performs operations according to the training process described in Figure 11: adds noise to the sample images, encodes features, and performs inference to generate predicted images, and finally adjusts the model parameters based on the comparison of the sample images and the predicted images until the model reaches the expected performance.

[0251] After the training is completed, the target image generation model is fed back to the terminal device for use in the local environment. This process realizes the efficient cooperation between the cloud training platform and the terminal device, utilizing the powerful computing power of the cloud platform to accelerate model training while maintaining the flexibility and response speed of the terminal device.

[0252] It should be noted that the technical solutions of the embodiments of the specification and the technical solutions of the above-mentioned image generation model training method belong to the same concept. The details of the technical solutions of the embodiments of the specification that are not described in detail can be referred to the description of the technical solutions of the above-mentioned image generation model training method of FIG. 11, and will not be repeated here.

[0253] In the embodiments of the specification, after the plurality of sample original images are respectively added with noise, the plurality of sample noise images are respectively feature-encoded to obtain a plurality of sample noise features. The plurality of sample noise features are simultaneously used as an input feature. An initial image generation model is used to perform inference and denoising based on the sample description texts of the plurality of sample original images to generate a plurality of predicted images. The initial image generation model simultaneously perceives the sample contents in the plurality of sample original images. On this basis, a target image generation model is trained and obtained, so that the image generation model can learn to establish the content correlation between the generated images in the inference and denoising process, avoiding the content fragmentation between the plurality of output images, improving the model generalization capability. Moreover, the target image generation model with high generalization capability can be well adapted to more field image generation applications, reducing the complexity of model training and improving the flexibility of model training. Remote model training is completed on a cloud training platform, reducing the burden of the terminal device and improving the efficiency of model training and the generalization capability of the model itself, which is suitable for a wide range of image generation application scenarios.

[0254] Corresponding to the above method embodiments, the specification also provides cloud training platform embodiments. FIG. 13 shows a structural schematic diagram of a cloud training platform according to an embodiment of the specification. As shown in FIG. 13, the cloud training platform 1300 includes a request interface 1302 and a response unit 1304.

[0255] The request interface 1302 is configured to receive a model training request sent by a terminal device, and obtain a plurality of sample original images of sample contents and sample description texts of the sample original images based on the model training request.

[0256] The response unit 1304 is configured to add noise to each sample original image to obtain a plurality of sample noise images, respectively encode features of each sample noise image to obtain a plurality of sample noise features, and splice the plurality of sample noise features to obtain a spliced noise feature. An initial image generation model is used to perform inference and denoising on the spliced noise feature based on the plurality of sample description texts to generate a plurality of predicted images. The initial image generation model is trained based on the plurality of sample original images and the plurality of predicted images to obtain a target image generation model.

[0257] In this embodiment, the request interface responds to the model training request sent by the terminal device, obtains multiple original sample images of the sample content and sample description text of each original sample image. After adding noise to each original sample image, the response unit performs feature encoding on the multiple noisy sample images to obtain multiple sample noise features, and concatenates the multiple sample noise features to obtain concatenated noise features as a whole input feature. Using the initial image generation model, inference denoising is performed based on the sample description text of each original sample image to generate multiple prediction images. The initial image generation model simultaneously perceives the sample content in each original sample image. Based on this, the target image generation model is trained to obtain the target image generation model. This allows the image generation model to learn to establish content correlation between generated images during the inference denoising process, avoiding content fragmentation between multiple output images, improving the model's generalization ability. Furthermore, the target image generation model with high generalization ability can better adapt to image generation applications in more fields, reducing the complexity of model training, improving the flexibility of model training, and completing interactive training between the cloud training platform and the terminal device, further improving the adaptability and flexibility of model training.

[0258] The above is an illustrative scheme of a cloud training platform according to this embodiment. It should be noted that the technical solution of this cloud training platform and the technical solution of the image generation model training method described above belong to the same concept. For details not described in detail in the technical solution of the cloud training platform, please refer to the description of the technical solution of the image generation model training method described above.

[0259] Corresponding to the above method embodiments, this specification also provides an image generation apparatus embodiment. Figure 14 shows a schematic diagram of an image generation apparatus provided in one embodiment of this specification. As shown in Figure 14, the apparatus includes:

[0260] The first acquisition module 1402 is configured to acquire the description text of the target content;

[0261] The first generation module 1404 is configured to use a target image generation model to perform inference denoising on random noise features based on descriptive text, and generate multiple target images corresponding to the descriptive text of the target content. The target image generation model is trained on an initial image generation model based on multiple predicted images and multiple original sample images of the sample content. The multiple predicted images are generated by inferring denoising on multiple sample noise features based on the sample descriptive text of multiple original sample images. The multiple sample noise features are obtained by adding noise and feature encoding to multiple original sample images.

[0262] Optionally, the plurality of sample noise features are obtained by adding noise to partial image content of a first sample original image and keeping other sample original images, the first sample original image being at least one of the plurality of sample original images.

[0263] Correspondingly, the apparatus further includes an image acquisition module configured to acquire an original image of the target content.

[0264] Correspondingly, the first generation module 1404 is further configured to: use the target image generation model to infer and denoise the random noise feature based on the description text and the original image, to generate a plurality of target images corresponding to the description text of the target content.

[0265] Optionally, the target image generation model includes an inference denoising layer.

[0266] Correspondingly, the first generation module 1404 is further configured to: input the plurality of description texts and the random noise feature into the inference denoising layer, and infer and denoise the random noise feature based on the plurality of description texts to generate a plurality of target images corresponding to the description text of the target content.

[0267] In the embodiments of the present specification, the image generation model can learn to establish content correlation between generated images in the inference denoising process, avoiding content fragmentation between the plurality of output images, improving the model generalization capability, and the target image generation model with high generalization capability can be better adapted to image generation applications in more fields, reducing the complexity of model training, improving the flexibility of model training, thereby realizing the generation of images with high content correlation, improving the efficiency and accuracy of image generation.

[0268] The above is a schematic scheme of an image generation apparatus according to an embodiment of the present specification. It should be noted that the technical scheme of the image generation apparatus belongs to the same concept as the technical scheme of the image generation method described above, and the details of the technical scheme of the image generation apparatus that are not described in detail can be referred to the description of the technical scheme of the image generation method.

[0269] Corresponding to the above method embodiments, the present specification also provides object image generation apparatus embodiments. FIG. 15 shows a structural schematic diagram of an object image generation apparatus according to an embodiment of the present specification. As shown in FIG. 15, the apparatus is applied to a server and includes:

[0270] The second receiving module 1502 is configured to receive a task request sent by a user terminal, wherein the task request carries prompt information of an object image generation task;

[0271] The second determining module 1504 is configured to determine a description text of a target object based on the prompt information.

[0272] The second denoising module 1506 is configured to infer and denoise the random noise features based on the description text to generate a plurality of target images of the target object corresponding to the description text by using a target image generation model, wherein the target image generation model is trained based on the initial image generation model and a plurality of sample original images of sample content, the plurality of prediction images are generated by inferentially denoising a plurality of sample noise features based on a plurality of sample original images and sample description texts, and the plurality of sample noise features are obtained by adding noise to and performing feature encoding on the plurality of sample original images.

[0273] The second feedback module 1508 is configured to feed back the plurality of target images to the user terminal.

[0274] Optionally, the second determination module 1504 is further configured to generate a plurality of description texts of the target object based on the prompt information by using a text generation model.

[0275] In the embodiments of the present specification, the image generation model can learn to establish content correlation between generated images in the inferential denoising process, avoiding content fragmentation between the plurality of output images, improving the model generalization capability, and the target image generation model with high generalization capability can be better adapted to object image generation applications in more fields, reducing the complexity of model training, improving the flexibility of model training, and thus realizing the generation of object images with high content correlation, improving the efficiency and accuracy of object image generation.

[0276] The above is a schematic scheme of the object image generation device in the embodiment. It should be noted that the technical scheme of the object image generation device belongs to the same concept as the technical scheme of the object image generation method described above, and the details of the technical scheme of the object image generation device that are not described in detail can be referred to the description of the technical scheme of the object image generation method.

[0277] Corresponding to the method embodiments described above, the present specification also provides a training device embodiment of an image generation model, and FIG. 16 shows a structural schematic diagram of a training device of an image generation model according to an embodiment of the present specification. As shown in FIG. 16, the device includes:

[0278] The third acquisition module 1602 is configured to acquire a plurality of sample original images of sample content and sample description texts of the plurality of sample original images.

[0279] The third noise adding module 1604 is configured to add noise to the plurality of sample original images respectively to obtain a plurality of sample noise images.

[0280] The third feature encoding module 1606 is configured to perform feature encoding on the plurality of sample noise images respectively to obtain a plurality of sample noise features.

[0281] The third generation module 1608 is configured to generate, by using the initial image generation model, a plurality of predicted images by reasoning and denoising a plurality of sample noise features based on a plurality of sample description texts;

[0282] The third training module 1610 is configured to train the initial image generation model based on a plurality of sample original images and a plurality of predicted images to obtain a target image generation model.

[0283] Optionally, the apparatus further comprises a feature splicing module configured to splice a plurality of sample noise features to obtain spliced noise features.

[0284] Correspondingly, the third generation module 1608 is further configured to generate, by using the initial image generation model, a plurality of predicted images by reasoning and denoising the spliced noise features based on a plurality of sample description texts.

[0285] Optionally, each sample noise feature is a feature sequence, and the spliced noise feature is a feature sequence.

[0286] Correspondingly, the third feature encoding module 1606 is further configured to divide a plurality of sample noise images into image blocks respectively; and encode features of the image blocks of the plurality of sample noise images to obtain a plurality of sample noise feature sequences, wherein each sample noise feature sequence includes feature elements of the image blocks of a sample noise image.

[0287] Correspondingly, the feature splicing module is further configured to splice the feature elements of a plurality of image blocks in the plurality of sample noise feature sequences to obtain a spliced noise feature sequence.

[0288] Optionally, the initial image generation model includes a plurality of noise adding layers, a plurality of feature encoding layers, a feature splicing layer, and a reasoning and denoising layer.

[0289] Correspondingly, the third noise adding module 1604 is further configured to input the plurality of sample original images into the plurality of noise adding layers respectively, and splice a plurality of sample noise features to obtain spliced noise features.

[0290] Correspondingly, the third feature encoding module 1606 is further configured to input the plurality of sample noise images into the plurality of feature encoding layers respectively, and encode features of the plurality of sample noise images respectively to obtain a plurality of sample noise features.

[0291] Correspondingly, the feature splicing module is further configured to input the plurality of sample noise features into the feature splicing layer, and splice the plurality of sample noise features to obtain spliced noise features.

[0292] Correspondingly, the third generation module 1608 is further configured to input the plurality of sample description texts and the splicing noise features into an inference denoising layer, perform inference denoising on the splicing noise features based on the plurality of sample description texts, and generate the plurality of predicted images.

[0293] Optionally, the third noise adding module 1604 is further configured to add different intensities of noise to the plurality of sample original images respectively to obtain the plurality of sample noise images.

[0294] Optionally, the third noise adding module 1604 is further configured to add noise to part of image content of the first sample original image and retain other sample original images to obtain the plurality of sample noise images, where the first sample original image is at least one of the plurality of sample original images.

[0295] Optionally, the third generation module 1608 is further configured to perform attention calculation on the plurality of sample noise features based on the plurality of sample description texts by using the initial image generation model to obtain attention weighted features, and generate the plurality of predicted images based on the attention weighted features.

[0296] Optionally, the third generation module 1608 is further configured to perform cross-attention calculation on the plurality of sample noise features based on the plurality of sample description texts by using the initial image generation model to obtain cross-attention weighted features, perform self-attention calculation on the cross-attention weighted features to obtain attention weighted features, and generate the plurality of predicted images based on the attention weighted features.

[0297] In the embodiments of the present specification, the image generation model can learn to establish content correlation between generated images in the inference denoising process, avoiding content fragmentation between the plurality of output images, improving the model generalization capability, and the target image generation model with high generalization capability can be better adapted to more image generation applications in more fields, reducing the complexity of model training and improving the flexibility of model training.

[0298] The above is a schematic scheme of the training device of the image generation model according to the present embodiment. It should be noted that the technical scheme of the training device of the image generation model belongs to the same concept as the technical scheme of the training method of the image generation model described above, and the details of the technical scheme of the training device of the image generation model that are not described in detail can be referred to the description of the technical scheme of the training method of the image generation model.

[0299] Corresponding to the above method embodiments, the present specification also provides a training device embodiment of the object image generation model, and FIG. 17 shows a structural schematic diagram of another training device of the image generation model according to an embodiment of the present specification. As shown in FIG. 17, the device is applied to a cloud training platform, and includes:

[0300] The fourth receiving module 1702 is configured to receive a model training request sent by the terminal device.

[0301] The fourth obtaining module 1704 is configured to, in response to the model training request, obtain a plurality of sample original images of sample content and sample description texts of the plurality of sample original images.

[0302] The fourth noise adding module 1706 is configured to add noise to the plurality of sample original images respectively to obtain a plurality of sample noise images.

[0303] The fourth feature encoding module 1708 is configured to perform feature encoding on the plurality of sample noise images respectively to obtain a plurality of sample noise features.

[0304] The fourth generating module 1710 is configured to perform inference and denoising on the plurality of sample noise features based on the plurality of sample description texts by using an initial image generation model to generate a plurality of predicted images.

[0305] The fourth training module 1712 is configured to train the initial image generation model based on the plurality of sample original images and the plurality of predicted images to obtain a target image generation model.

[0306] The fourth feedback module 1714 is configured to feed back the target image generation model to the terminal device.

[0307] In the embodiments of the present specification, the image generation model can learn to establish content correlation between generated images in the inference and denoising process, avoid content fragmentation between the output multiple images, improve the model generalization capability, and the target image generation model with high generalization capability can be well adapted to more image generation applications in the field, reduce the complexity of model training, and improve the flexibility of model training. Remote model training is completed on the cloud training platform, which reduces the burden of the terminal device, improves the efficiency of model training and the generalization capability of the model itself, and is suitable for a wide range of image generation application scenarios.

[0308] The above is a schematic scheme of the training device of the object image generation model according to the present embodiment. It should be noted that the technical scheme of the training device of the object image generation model belongs to the same concept as the technical scheme of the training method of the object image generation model described above, and the details of the technical scheme of the training device of the object image generation model that are not described in detail can be referred to the description of the technical scheme of the training method of the object image generation model.

[0309] FIG. 18 illustrates a structural block diagram of a computing device according to an embodiment of the present specification. The components of the computing device 1800 include, but are not limited to, a memory 1810 and a processor 1820. The processor 1820 is connected to the memory 1810 through a bus 1830, and a database 1850 is used to store data.

[0310] The computing device 1800 further includes an access device 1840, which enables the computing device 1800 to communicate via one or more networks 1860. Examples of these networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1840 can include one or more of any type of network interface (e.g., a Network Interface Controller (NIC)), wired or wireless, such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, Near Field Communication (NFC).

[0311] In an embodiment of the present specification, the above-mentioned components of the computing device 1800 and other components not shown in FIG. 18 can also be connected to each other, for example, through a bus. It should be understood that the structural block diagram of the computing device shown in FIG. 18 is for the purpose of example only, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0312] The computing device 1800 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1800 can also be a mobile or stationary server.

[0313] The processor 1820 is configured to execute computer program / instructions to implement the steps of the image generation method, the object image generation method, or the training method of the image generation model.

[0314] The above is a schematic solution of the computing device of the embodiment. It should be noted that the technical solution of the computing device belongs to the same concept as the technical solution of the image generation method, the object image generation method, and the training method of the image generation model. For details of the technical solution of the computing device that are not described in detail, reference can be made to the description of the technical solution of the image generation method, the object image generation method, or the training method of the image generation model.

[0315] An embodiment of the present specification further provides a computer readable storage medium storing computer program / instructions, which, when executed by a processor, implement the steps of the image generation method, the object image generation method, or the training method of the image generation model.

[0316] The above is a schematic solution of the computer readable storage medium of the embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the image generation method, the object image generation method, and the training method of the image generation model. For details of the technical solution of the storage medium that are not described in detail, reference can be made to the description of the technical solution of the image generation method, the object image generation method, or the training method of the image generation model.

[0317] An embodiment of the present specification further provides a computer program product comprising computer program / instructions, which, when executed by a processor, implement the steps of the image generation method, the object image generation method, or the training method of the image generation model.

[0318] The above is a schematic scheme of a computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical schemes of the image generation method, the object image generation method, and the image generation model training method described above belong to the same concept. The technical scheme of the computer program product is not described in detail, and the description of the technical scheme of the image generation method, the object image generation method, or the image generation model training method described above can be referred to.

[0319] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in which they are recited in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or necessary.

[0320] The computer instructions include computer program code, which can be in the form of source code, object code, executable code, or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM for short), random access memory (Random Access Memory, RAM for short), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer readable medium can be appropriately increased or decreased according to the requirements of patent practice, for example, in some regions, according to the patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0321] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited by the order of the described actions, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the embodiments of the present specification.

[0322] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0323] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. Alternative embodiments do not describe all the details and limit the present application to only this specific embodiment. Obviously, many modifications and variations of the embodiments of the present specification can be made in light of the contents of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image generation method, comprising: obtaining a description text of a target content; using a target image generation model, performing inference denoising on random noise features based on the description text to generate a plurality of target images corresponding to the description text of the target content, wherein the target image generation model is trained based on an initial image generation model, a plurality of predicted images, and a plurality of sample original images of sample contents, the plurality of predicted images are generated by performing inference denoising on a plurality of sample noise features based on sample description texts of the plurality of sample original images, and the plurality of sample noise features are obtained by adding noise to and feature encoding the plurality of sample original images.

2. The method of claim 1, wherein the plurality of sample noise features are obtained by adding noise to the plurality of sample raw images and feature encoding, and specifically comprising: The plurality of sample noise features are obtained by adding noise to, feature encoding, and feature splicing the plurality of sample original images, and the method further comprises: performing feature splicing on the plurality of sample noise features to obtain spliced noise features; Correspondingly, the plurality of predicted images are generated by performing inference denoising on the plurality of sample noise features based on the sample description texts of the plurality of sample original images, comprising: The plurality of predicted images are generated by performing inference denoising on the spliced noise features based on the sample description texts of the plurality of sample original images.

3. The method of claim 1, wherein the plurality of sample noise features are obtained by adding noise to a partial image content of a first sample original image and retaining other sample original images, and the first sample original image is at least one of the plurality of sample original images. Before the step of using the target image generation model to perform inference denoising on the random noise features based on the description text to generate the plurality of target images corresponding to the description text of the target content, the method further comprises: obtaining an original image of the target content; The step of using the target image generation model to perform inference denoising on the random noise features based on the description text to generate the plurality of target images corresponding to the description text of the target content, comprises: using the target image generation model to perform inference denoising on the random noise features based on the description text and the original image to generate the plurality of target images corresponding to the description text of the target content.

4. The method of claim 1, wherein the target image generation model comprises an inference denoising layer; The step of using the target image generation model to perform inference denoising on the random noise features based on the description text to generate the plurality of target images corresponding to the description text of the target content, comprises: inputting a plurality of description texts and random noise features into the inference denoising layer, performing inference denoising on the random noise features based on the plurality of description texts to generate the plurality of target images corresponding to the description text of the target content.

5. An object image generation method applied to a server, comprising: receiving a task request sent by a user terminal, wherein the task request carries prompt information of an object image generation task; determining a description text of a target object based on the prompt information; The target image generation model is used to infer and denoise random noise features based on the description text, to generate a plurality of target images of the target object corresponding to the description text, wherein the target image generation model is trained based on an initial image generation model, a plurality of prediction images and a plurality of sample original images of sample content, the plurality of prediction images are generated by inferring and denoising a plurality of sample noise features based on sample description texts of the plurality of sample original images, and the plurality of sample noise features are obtained by adding noise to and feature encoding the plurality of sample original images; The plurality of target images are fed back to the user terminal.

6. The method of claim 5, wherein the description text of the target object is determined based on the prompt information, comprising: a plurality of description texts of the target object are generated based on the prompt information by using a text generation model.

7. A training method of an image generation model, comprising: obtaining a plurality of sample original images of sample content and sample description texts of the plurality of sample original images; adding noise to the plurality of sample original images respectively to obtain a plurality of sample noise images; feature encoding the plurality of sample noise images respectively to obtain a plurality of sample noise features; inferring and denoising the plurality of sample noise features based on a plurality of sample description texts by using an initial image generation model to generate a plurality of prediction images; training the initial image generation model based on the plurality of sample original images and the plurality of prediction images to obtain a target image generation model.

8. The method of claim 7, wherein before the step of inferring and denoising the plurality of sample noise features based on a plurality of sample description texts by using an initial image generation model to generate a plurality of prediction images, the method further comprises: feature splicing the plurality of sample noise features, obtaining spliced noise features, the step of inferring and denoising the plurality of sample noise features based on a plurality of sample description texts by using an initial image generation model to generate a plurality of prediction images, comprising: inferring and denoising the spliced noise features based on the plurality of sample description texts by using the initial image generation model to generate the plurality of prediction images.

9. The method of claim 8, wherein any sample noise feature is a feature sequence, The spliced noise features are feature sequences. the step of feature encoding the plurality of sample noise images respectively to obtain a plurality of sample noise features, comprising: image blocking the plurality of sample noise images respectively; feature encoding image blocks of the plurality of sample noise images to obtain a plurality of sample noise feature sequences, wherein each sample noise feature sequence comprises feature elements of image blocks of a sample noise image; the step of feature splicing the plurality of sample noise features to obtain spliced noise features, comprising: splicing feature elements of a plurality of image blocks in the plurality of sample noise feature sequences to obtain a spliced noise feature sequence.

10. The method of claim 8 or 9, wherein the initial image generation model comprises a plurality of noise adding layers, a plurality of feature encoding layers, a feature splicing layer and an inference denoising layer; the step of adding noise to the plurality of sample original images respectively to obtain a plurality of sample noise images, comprising: The plurality of sample original images are respectively input into the plurality of noise adding layers, and the plurality of sample noise features are respectively subjected to feature splicing, Obtaining spliced noise features; The plurality of sample noise features are respectively subjected to feature encoding, and a plurality of sample noise features are obtained, including: The plurality of sample noise features are respectively input into the plurality of feature encoding layers, and the plurality of sample noise features are respectively subjected to feature encoding to obtain a plurality of sample noise features; The plurality of sample noise features are respectively subjected to feature splicing, and spliced noise features are obtained, including: The plurality of sample noise features are input into the feature splicing layer, and the plurality of sample noise features are subjected to feature splicing to obtain spliced noise features; The initial image generation model is used to infer and denoise the spliced noise features based on the plurality of sample description texts to generate a plurality of predicted images, including: The plurality of sample description texts and the spliced noise features are input into the inference and denoising layer, and the spliced noise features are inferred and denoised based on the plurality of sample description texts to generate a plurality of predicted images.

11. The method of claim 7, wherein the plurality of sample original images are respectively subjected to noise adding to obtain a plurality of sample noise images, including: Different intensities of noise are respectively added to the plurality of sample original images to obtain a plurality of sample noise images.

12. The method of claim 7, wherein the plurality of sample original images are respectively subjected to noise adding to obtain a plurality of sample noise images, including: Part of the image content of a first sample original image is subjected to noise adding, and other sample original images are retained to obtain a plurality of sample noise images, wherein the first sample original image is at least one of the plurality of sample original images.

13. The method of claim 7, wherein the initial image generation model is used to infer and denoise the plurality of sample noise features based on the plurality of sample description texts to generate a plurality of predicted images, including: An initial image generation model is used to perform attention calculation on the plurality of sample noise features based on the plurality of sample description texts to obtain attention weighted features, and the plurality of predicted images are generated based on the attention weighted features.

14. The method of claim 13, wherein the initial image generation model is used to infer and denoise the plurality of sample noise features based on the plurality of sample description texts to generate a plurality of predicted images, including: An initial image generation model is used to perform self-attention calculation on the plurality of sample noise features based on the plurality of sample description texts to obtain attention weighted features, and the plurality of predicted images are generated based on the attention weighted features.

15. The method of claim 13, wherein the initial image generation model is used to infer and denoise the plurality of sample noise features based on the plurality of sample description texts to generate a plurality of predicted images, including: The initial image generation model is used to generate a plurality of predicted images based on the plurality of sample description texts.

16. A training method of an image generation model, applied to a cloud training platform, comprising: receiving a model training request sent by a terminal device; in response to the model training request, obtaining a plurality of sample original images of sample content and sample description texts of the plurality of sample original images; adding noise to the plurality of sample original images respectively to obtain a plurality of sample noise images; performing feature encoding on the plurality of sample noise images respectively to obtain a plurality of sample noise features; using an initial image generation model, performing inference denoising on the plurality of sample noise features based on a plurality of sample description texts to generate a plurality of predicted images; training the initial image generation model based on the plurality of sample original images and the plurality of predicted images to obtain a target image generation model; feeding back the target image generation model to the terminal device.

17. A cloud training platform, comprising a request interface and a response unit; the request interface is configured to receive a model training request sent by a terminal device, and in response to the model training request, obtain a plurality of sample original images of sample content and sample description texts of the plurality of sample original images; the response unit is configured to add noise to the plurality of sample original images respectively to obtain a plurality of sample noise images, perform feature encoding on the plurality of sample noise images respectively to obtain a plurality of sample noise features, use an initial image generation model to perform inference denoising on the plurality of sample noise features based on a plurality of sample description texts to generate a plurality of predicted images, train the initial image generation model based on the plurality of sample original images and the plurality of predicted images to obtain a target image generation model, and feed back the target image generation model to the terminal device.

18. A computing device, comprising: a memory and a processor; the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which realize the steps of the method in any one of claims 1 to 16 when executed by the processor.

19. A computer-readable storage medium storing computer programs / instructions, which realize the steps of the method in any one of claims 1 to 16 when executed by a processor.

20. A computer program product comprising computer programs / instructions, which realize the steps of the method in any one of claims 1 to 16 when executed by a processor.

Images