Image generation method

By training an image generation model to learn for each individual object, generating and storing pre-stored object vectors, the problem of low training efficiency and resource waste caused by multi-concept customized statements is solved, and efficient and stable multi-concept image generation is achieved.

WO2024242624A1PCT designated stage expired Publication Date: 2024-11-28ALIBABA INNOVATION PRIVATE LIMITED

Patent Information

Application Number
PCT/SG2024/050262
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-22
Filing Date
2024-04-23
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing technologies for image generation suffer from low training efficiency and wasted computational resources due to the large number of multi-concept customized statements, making it difficult to efficiently generate multi-concept customized images.

Method used

By training an image generation model to learn about each individual object, pre-stored object vectors are generated and stored, avoiding the need to learn each multi-concept customized statement. Images are generated by combining the pre-stored object vectors with the initial text vectors of the object description text.

Benefits of technology

It improves the training efficiency of image generation models, reduces the waste of computing resources, and can stably generate multi-concept customized images, avoid attribute confusion problems, and provide high-quality image generation capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SG2024050262_28112024_PF_FP_ABST
    Figure SG2024050262_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the embodiments of the present disclosure is an image generation method, the image generation method comprising: inputting an object description text containing at least two objects into an image generation model, and determining an initial text vector of the object description text; determining pre-stored object vectors of the at least two objects, the pre-stored object vectors being generated with respect to the training text vectors of the at least two objects during a training process of the image generation model; and, by means of the pre-stored object vectors, processing the initial text vector to obtain a target text vector and, according to the target text vector, generating an object image containing the at least two objects. Therefore, the present disclosure avoids the problem that a large number of multi-concept customized statements cause the low training efficiency of a neural network model and waste a large number of computing resources, ensures the smooth operation of an image generation model and provides a stable image generation capability.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Image Generation Method Cross-Reference This disclosure claims priority to Chinese patent application No. 202310594509.2, filed with the China Patent Office on May 22, 2023, entitled "Image Generation Method," the entire contents of which are incorporated herein by reference. Technical Field: Embodiments of the present disclosure relate to the field of computer technology, and more particularly, to an image generation method. Background: With the continuous development of artificial intelligence (AI) technology, in the field of image generation, text-based graph technology, which automatically generates images based on conceptual descriptions, has been widely used in various computer service scenarios. Text-based graph technology is based on AI technology and trains neural network models with a large number of samples. This allows the neural network model to automatically generate images simply by inputting concept descriptions. For example, inputting a multi-concept customized sentence containing the concepts "sunglasses" and "male" into a neural network model generates an image of a man wearing sunglasses. In the prior art, when training a neural network model, to enable it to generate multi-concept customized images, the model learns each multi-concept customized sentence separately. However, due to the large number of concept terms in real-world scenarios, the number of multi-concept customized sentences composed of multiple concept terms is very large, resulting in low neural network model training efficiency and a significant waste of computing resources. In view of this, embodiments of the present disclosure provide an image generation method. One or more embodiments of the present disclosure also relate to an image generation model training method, another image generation method, an image generation model training device, two image generation devices, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art. According to a first aspect of an embodiment of the present disclosure, an image generation method is provided, comprising: inputting object description text containing at least two objects into an image generation model to determine an initial text vector for the object description text; determining pre-stored object vectors for the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; processing the initial text vector using the pre-stored object vector to obtain a target text vector, and generating an object image containing the at least two objects based on the target text vector.According to a second aspect of an embodiment of the present disclosure, an image generation device is provided, comprising: a first vector determination module, configured to input object description text containing at least two objects into an image generation model to determine an initial text vector of the object description text; a second vector determination module, configured to determine pre-stored object vectors of the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; and an image generation module, configured to process the initial text vector using the pre-stored object vector to obtain a target text vector, and generate an object image containing the at least two objects based on the target text vector. According to a third aspect of an embodiment of the present disclosure, a method for training an image generation model is provided, comprising: determining a training sample for an image generation model, wherein the training sample includes sample object description text and a reference object image corresponding to the sample object description text; inputting the sample object description text into the image generation model, processing the sample object description text using a text encoding module in the image generation model to obtain a training text vector corresponding to the sample object description text; decoding the training text vector using a vector processing module in the image generation model to obtain a sample object image corresponding to the sample object description text, and generating a pre-stored object vector corresponding to the sample object based on the training text vector; determining a loss function for the image generation model based on the reference object image and the sample object image, and training the image generation model based on the loss function to obtain a trained image generation model. According to a fourth aspect of an embodiment of the present disclosure, an image generation model training apparatus is provided, comprising: a sample determination module, configured to determine training samples for an image generation model, wherein the training samples include sample object description text and a reference object image corresponding to the sample object description text; a vector acquisition module, configured to input the sample object description text into the image generation model, process the sample object description text using a text encoding module in the image generation model, and obtain a training text vector corresponding to the sample object description text; an image acquisition module, configured to decode the training text vector using a vector processing module in the image generation model, obtain a sample object image corresponding to the sample object description text, and generate a pre-stored object vector corresponding to the sample object based on the training text vector; and a loss determination module, configured to determine a loss function of the image generation model based on the reference object image and the sample object image, and train the image generation model based on the loss function to obtain a trained image generation model.According to a fifth aspect of an embodiment of the present disclosure, an image generation method is provided, which is applied to a cloud-side device and includes: receiving object description text sent by an end-side device, wherein the object description text includes at least two objects; inputting the object description text into an image generation model to determine an initial text vector of the object description text; determining pre-stored object vectors of the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; processing the initial text vector using the pre-stored object vector to obtain a target text vector, and generating an object image including the at least two objects based on the target text vector; and sending the object image to the end-side device. According to a sixth aspect of an embodiment of the present disclosure, an image generation apparatus is provided, applied to a cloud-side device, comprising: a text receiving module configured to receive object description text sent by a client-side device, wherein the object description text includes at least two objects; a first vector determination module configured to input the object description text into an image generation model to determine an initial text vector for the object description text; a second vector determination module configured to determine pre-stored object vectors for the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; an image generation module configured to process the initial text vector using the pre-stored object vectors to obtain a target text vector, and generate an object image including the at least two objects based on the target text vector; and an image sending module configured to send the object image to the client-side device. According to a seventh aspect of an embodiment of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is configured to store computer-executable instructions; the processor is configured to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the two image generation methods and the image generation model training method described above. According to an eighth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, storing computer-executable instructions. When executed by a processor, the computer-executable instructions implement the steps of the two image generation methods and the image generation model training method. According to a ninth aspect of the embodiments of the present disclosure, a computer program is provided. When executed on a computer, the computer executes the steps of the two image generation methods and the image generation model training method.The image generation method provided herein includes: inputting object description text containing at least two objects into an image generation model to determine an initial text vector for the object description text; determining pre-stored object vectors for the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; processing the initial text vector using the pre-stored object vectors to obtain a target text vector; and generating an object image containing the at least two objects based on the target text vector. Specifically, this method does not require the image generation model to learn every multi-concept customized sentence. Instead, the image generation model is trained to learn each individual object (concept), thereby avoiding the problem of low training efficiency and significant waste of computing resources in the neural network model due to the large number of multi-concept customized sentences. Simultaneously, during model training, a pre-stored object vector corresponding to each individual object is generated and stored. Therefore, in practical applications, when object description text containing at least two objects is input into the image generation model, the image generation model can determine the pre-stored object vectors corresponding to the at least two objects, combine the pre-stored object vector of each object with the initial text vector of the object description text, and then generate an object image containing the at least two objects based on the combined target text vector, thereby ensuring the smooth operation of the image generation model and providing stable image generation capabilities. BRIEF DESCRIPTION OF THE DRAWINGS FIG1 is a schematic diagram of the structure of a multi-concept customization model provided by an embodiment of the present disclosure; FIG2 is a schematic diagram of an application scenario of an image generation method provided by an embodiment of the present disclosure; FIG3 is a flow chart of an image generation method provided by an embodiment of the present disclosure; FIG4 is a flow chart of the processing process of an image generation method provided by an embodiment of the present disclosure; FIG5 is a flow chart of another image generation method provided by an embodiment of the present disclosure; FIG6 is a flow chart of an image generation model training method provided by an embodiment of the present disclosure; FIG7 is a schematic diagram of the structure of an image generation device provided by an embodiment of the present disclosure; FIG8 is a schematic diagram of the structure of an image generation model training device provided by an embodiment of the present disclosure; FIG9 is a schematic diagram of the structure of another image generation device provided by an embodiment of the present disclosure; FIG10 is a block diagram of the structure of a computing device provided by an embodiment of the present disclosure; FIG11 is a schematic diagram of the structure of a processor provided by an embodiment of the present disclosure. DETAILED DESCRIPTION The following description sets forth numerous specific details to facilitate a thorough understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art may make similar generalizations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific implementations disclosed below.The terminology used in one or more embodiments of the present disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. As used in one or more embodiments of the present disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the terms 'and / or' used in one or more embodiments of the present disclosure refer to and include any or all possible combinations of one or more associated listed items. It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to a determination." It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for the user to choose to authorize or refuse. First, the terms involved in one or more embodiments of the present disclosure are explained. Diffusion model (d Fusion mode: The diffusion model is inspired by non-equilibrium thermodynamics and is a new SOTA in deep generative models. Customized Generation: The task of generating related concepts given by the user. Multi-subject Custom Generation: Generates multiple user-provided customized concepts in a single graph.

[0002] SOTA (state-of-the-art): Used to describe the model that achieves the current best results on a task in machine learning.

[0003] U-Net: An image segmentation network. Since text-based graph technology, implemented using neural network models, is widely used in various computer service scenarios, during the training process, in order for the neural network model to generate multi-concept customized images, each multi-concept customized sentence is learned separately. However, due to the large number of concepts in real-world scenarios, the number of multi-concept customized sentences composed of multiple concepts is very large. As a result, some multi-subject image generation methods are prone to attribute confusion when synthesizing multiple subjects (e.g., concepts such as sunglasses and men), resulting in a decrease in the similarity between the generated image and the reference image. Furthermore, many of these methods train on multi-concept data together, requiring retraining whenever a new object requires customization, resulting in a significant waste of computational resources. For example, since the diffusion model is a parametric neural network, it learns image distributions through gradual denoising. Therefore, to further explore the scalability of the diffusion model, many works have focused on conditional generation based on diffusion, which can be roughly divided into two categories. The first is a classifier-guided approach, which utilizes a classifier to facilitate the sampling process of a pre-trained unconditional model. The second type, known as classifier-free methods, directly collects large amounts of data for joint optimization under the guarantee of conditional probability inference. This approach can produce highly detailed results, but requires a large amount of data and computational resources. Furthermore, due to advances in language and cross-modal grounding models, a large number of text-to-image works using classifier-free techniques have begun to emerge, promoting explicit control of corresponding semantics and style. However, the expressive power of text remains limited, and more work aims to utilize additional, conditional information such as reference images, backgrounds, and sketches to further guide global control. Furthermore, diffusion models for text-to-image generation use textual cues describing image content to guide the denoising process. These models are typically trained by denoising noisy images, resulting in low model training efficiency. It should be noted that the present disclosure provides a multi-concept customization model for diffusion from text to image. The basic idea is to achieve better customized generation effects by fine-tuning the cross-attention mechanism. However, there are also problems with low model training efficiency and the waste of a large amount of computing resources. The purpose of customized generation is to implant a given topic into the diffusion model and bind it to a unique text identifier to indicate its existence; in this way, the model can vividly generate various renditions of the topic under the guidance of text prompts.Specifically, Figure 1 shows a schematic diagram of the structure of a multi-concept customization model provided by one embodiment of the present disclosure. This multi-concept customization model is a diffusion model, U-Net. This model employs a joint training method, directly training the model using multi-concept customized sentences containing multiple concepts. The specific process is as follows: the multi-concept customized sentence "photo of av* dog" is input into a text transformer, which then maps "photo of av* dog" into an embedding vector. It should be noted that the embedding vector output by the text transformer can have a dimension of 77*1024. Some of the dimensions within the 77 dimensions correspond to words in the sentence "photo of av* dog." For example, the first few dimensions within the 77 dimensions correspond to each word in the sentence "photo of av* dog." The word "dog" in "photo of av*dog" represents the 1024 of the fifth dimension. After obtaining the sentence embedding vector, it is input into different attention layers (attention neural network layers). Through the Attention layer and the Residual Network (ResNet), the noisy puppy image is restored to a fresh puppy image. Trainable vectors refer to vectors in the diffusion model that require training and fine-tuning, such as the "v*" vector in Figure 1. Frozen vectors refer to vectors in the diffusion model that are fixed and do not require training, such as the "photo," "of," "a," and "dog" vectors in Figure 1. Based on this, the present disclosure provides an image generation method, an image generation model training method, another image generation method, an image generation model training device, two image generation devices, a computing device, a computer-readable storage medium, and a computer program, each of which is described in detail in the following embodiments. Figure 2 illustrates an application scenario of an image generation method according to an embodiment of the present disclosure. As shown in Figure 2 , a user inputs the text "a cat and a dog" into a terminal. After receiving the text, the terminal inputs it into a diffusion model and uses the encoder in the diffusion model to encode the text to obtain a text embedding vector. Based on the words "cat" and "dog" in the text, the residual vectors for the objects (cat and dog) are determined from pre-cached residual vectors.Adding this residual vector to the text embedding vector can produce corresponding objects such as cats and dogs in the generated image. This results in a multi-concept customized image. Based on this, the image generation method provided by the present disclosure finds a residual vector for each concept (object), which can be reused indefinitely. Furthermore, multiple concepts can be combined without requiring training, unlike other methods that train multiple concepts together, significantly saving computing resources. Figure 3 shows a flowchart of an image generation method provided according to one embodiment of the present disclosure, which specifically includes the following steps. Step 302: Input object description text containing at least two objects into an image generation model to determine an initial text vector for the object description text. The object can be understood as the specific subject to be generated as an image, including but not limited to items, animals, plants, people, landscapes, etc. For example, the object can be sunglasses, a cat, a dog, a man, etc., and this disclosure does not impose specific restrictions on this. The object can be understood as the concept, subject, etc. in the above embodiments. The object description text containing at least two objects can be understood as language used to describe two or more objects. For example, the sentence "a man wearing sunglasses" describes the two objects "sunglasses" and "man"; the sentence "a cat and a kind of dog" describes the two objects "cat" and "dog." In practical applications, the object description text can be customized based on actual needs, and this disclosure does not impose specific restrictions on this. The image generation model can be understood as a model capable of generating an object image containing at least two objects based on the object description text containing at least two objects. For example, the image generation model can be a neural network model or diffusion model that implements text graph technology, and this disclosure does not impose specific restrictions on this. The initial text vector can be understood as a feature vector representing each word in the object description text. For example, the initial text vector is the text embedding vector corresponding to the sentence "a cat and a kind of dog." It should be noted that the dimension of the text embedding vector can be customized based on the actual application scenario. Specifically, the image generation method provided by the present disclosure can obtain object description text containing at least two objects and input this object description text into an image generation model, thereby determining an initial text vector corresponding to the object description text. Furthermore, in one embodiment provided by the present disclosure, to ensure that the image generation model can generate vivid and accurate object images, it is necessary to encode the object description text in a text encoding module within the image generation model, so that accurate object images can be subsequently generated based on the encoded text embedding vectors.Specifically, inputting object description text containing at least two objects into an image generation model and determining an initial text vector for the object description text includes: determining the object description text containing at least two objects; inputting the object description text into the image generation model, and processing the object description text using a text encoding module in the image generation model to obtain an initial text vector corresponding to the object description text. In one embodiment provided by the present disclosure, the object description text containing at least two objects may be provided by a user. Based on this, determining the object description text containing at least two objects includes: receiving an image generation request and obtaining the object description text containing at least two objects from the image generation request. The image generation request can be understood as a request instructing a computing device to generate an image of an object. It should be noted that the image generation method provided by the present disclosure can be applied to computing devices such as computers, mobile phones, clients, mobile terminals, or servers, and this disclosure does not impose specific limitations on this. Based on this, it can be seen that the image generation method provided by the present disclosure is capable of receiving an image generation request sent by a user, wherein the image generation request carries object description text containing at least two objects. Subsequently, in response to the image generation request, the object description text containing at least two objects is input into the image generation model to obtain an object image containing at least two objects, thereby generating corresponding object images according to different user requirements. Furthermore, in some special computer service scenarios, the object description text containing at least two objects may be pre-stored on the computing device used in the image generation method. Upon receiving the image generation request, the locally stored object description text can be quickly obtained to generate the corresponding object image. Specifically, based on this, determining the object description text containing at least two objects includes: obtaining a text identifier from the received image generation request, and determining the object description text corresponding to the text identifier from pre-stored object description text, wherein the object description text contains at least two objects. The text identifier can be understood as information that uniquely identifies an object description text. For example, the object description text "a cat and a dog" can be represented by the character A. Based on this, the image processing method provided by the present disclosure, upon receiving an image generation request from a user or other computing device, obtains the text identifier from the image generation request. The object description text corresponding to the text identifier is quickly determined from locally stored object description text containing at least two objects. The object description text is then input into the image generation model to obtain an object image containing at least two objects.The text encoding module can be understood as a network layer in the image generation model that encodes object description text. For example, the text encoding module can be a text encoder. Specifically, the image generation method provided by the present disclosure can, after determining object description text containing at least two objects, input the object description text into the image generation model. The text encoding module in the image generation model then encodes the object description text, thereby obtaining a word text vector corresponding to each word in the object description text. The word text vector corresponding to each word constitutes the initial text vector corresponding to the object description text. For example, the object description text is "a cat and a dog." Based on this, the object description text is input into the text encoder in the diffusion model, and the text encoder encodes "a cat and a dog" to obtain a text embedding vector corresponding to each word. Step 304: Determine pre-stored object vectors for the at least two objects. The pre-stored object vectors are generated based on the training text vectors for the at least two objects during the training of the image generation model. The pre-stored object vector can be understood as a residual vector corresponding to a single object, generated during the training of the image generation model. Furthermore, the residual vector can be stored in a computing device used in the image generation method. In practical applications, the pre-stored object vector can also be understood as the residual token embedding or residual label embedding of each individual object. The training sample vector can be understood as a text embedding vector generated by encoding each sample object description text during the training of the image generation model. Specifically, in one embodiment provided by the present disclosure, determining the pre-stored object vectors for the at least two objects includes: determining an object identifier in the object description text, wherein the object description text is composed of the object identifier and object feature words other than the object identifier, with each object identifier representing one object; determining, from the pre-stored object vectors, a target pre-stored object vector associated with the object identifier; and determining the target pre-stored object vector as the pre-stored object vector for the at least two objects. The object identifier can be understood as a term in the object description text that directly identifies a specific object. An object identifier directly identifies an object, and an object can be identified through the object identifier. For example, in the object description text "a man wearing sunglasses," the terms "sunglasses" and "man" directly identify the two objects, namely, sunglasses and man. Therefore, these two terms are object identifiers.For another example, in the object description text "a cat and a dog," the words "cat" and "dog" directly represent the two objects, a cat and a dog. Therefore, "cat" and "dog" are object identifiers. Object feature words can be understood as words in the object description text that describe the characteristics of the object. These object feature words can be used to determine some of the attribute characteristics of a specific object. For example, in the object description text "a cat and a dog," the word "a" describes that there are two separate cats and dogs, while the word "and" describes that both the cat and the dog are displayed in the image. Therefore, "a" and "and" are object feature words. It should be noted that, in one embodiment, other words in an object description text besides the object identifier can be object feature words. Specifically, the image generation method provided by the present disclosure utilizes the image generation model to determine the object identifier in the object description text and, from pre-stored object vectors stored locally, determine a target pre-stored object vector associated with the object identifier. The target pre-stored object vector is then determined to be the pre-stored object vector of at least two objects. Continuing with the above example, after the object description text "a cat and a dog" is input into the diffusion model, the diffusion model will identify the two words "cat" and "dog" from "a cat and a dog" and, from the multiple locally stored residual vectors, determine the residual vectors corresponding to the two words "cat" and "dog". These residual vectors are the residual vectors corresponding to the two concepts of cat and dog. Based on this, the image generation method provided by the present disclosure finds and stores a residual vector for each concept (object), allowing for unlimited reuse. Furthermore, the residual vector corresponding to each concept requires only a small storage space, for example, approximately 5 kilobytes. This avoids the problem of other multi-concept customized generation methods requiring full digital precision to store diffusion model parameters, which can be quite costly for large-scale applications on mobile devices. The image generation method provided by the present disclosure only stores the text embedding residual vector for each concept, requiring less storage space and reducing costs. Step 306: Process the initial text vector using the pre-stored object vector to obtain a target text vector, and generate an object image containing the at least two objects based on the target text vector. The target text vector can be understood as a vector obtained by adjusting the initial text vector based on the pre-stored object vector.The object image containing the at least two objects can be understood as an image containing two or more objects, for example, an image containing a cat and a dog, an image of a man wearing sunglasses, etc. In one embodiment of the present disclosure, the image generation method provided herein proposes a method for generating multi-subject images. This method achieves customized synthesis of multiple subjects by learning the residual vector of a single subject, thereby subsequently generating a multi-concept customized image based on the combination of multiple residual vectors. Specifically, the initial text vector is processed using the pre-stored object vector to obtain a target text vector, including: determining, from the initial text vector, vectors to be adjusted corresponding to the at least two objects based on object identifiers in the object description text; and adding the pre-stored object vector to the vectors to be adjusted in the initial text vector to obtain a target text vector. The vectors to be adjusted can be understood as the text embedding vector corresponding to each object. Specifically, the image generation method provided by the present disclosure utilizes an image generation model to determine object identifiers in an object description text. Based on the object identifiers, the method determines vectors to be adjusted corresponding to at least two objects from an initial text vector. The method then adjusts the vectors to be adjusted in the initial text vector based on the pre-stored object vectors to obtain a target text vector. For example, after the diffusion model determines the residual vectors for the two objects "cat" and "dog," the method then uses the text embedding vector of the object description text "a cat and a dog" to determine the text embedding vectors corresponding to the two words "cat" and "dog" as the text embedding vectors corresponding to the two objects "cat" and "dog." The residual vectors for the two objects "cat" and "dog" are then added to the corresponding text embedding vectors, thereby adding the residual vectors to the target text vectors. uThe basic category embedding vector of the object description text "a cat and a dog" is embedded in the text, thereby generating a final text embedding vector containing the object residual vector. The image generated by the final text embedding vector will produce the corresponding themes (cat, dog). In one embodiment provided by the present disclosure, after determining the target text vector, the target text vector is processed by the vector processing module in the image generation model to generate an object image containing at least two objects. Specifically, the object image containing the at least two objects generated according to the target text vector includes: using the vector processing module in the image generation model to process the target text vector to obtain an object image containing the at least two objects. The vector processing module can be understood as a module that decodes the target text vector to generate the object image. For example, the vector processing module can be a U-Net model. For example, after obtaining the final text embedding vector containing the object residual vector, the text embedding vector is input into the U-Net model for decoding processing to obtain an image containing a cat and a dog. Further, in the embodiment provided by the present disclosure, the image generation method provided by the present disclosure, While addressing the high computational resource consumption during model training, this approach also avoids attribute confusion during the generation of multi-subject images (i.e., multi-concept customized images) during model application. Other multi-subject image generation methods are prone to attribute confusion when synthesizing multiple subjects, resulting in a reduced similarity between the generated image and the reference image. For example, a puppy and a kitten may overlap in an image. Therefore, this method proposes a method based on a custom diffusion model that achieves custom synthesis of multiple subjects by learning the residual vector of a single subject, thus avoiding the attribute confusion issue. Specifically, the method utilizes the vector processing module in the image generation model to process the target text vector to obtain an object image containing the at least two objects, including: decoding the target text vector using the vector processing module to obtain image features corresponding to the object description text; adjusting the image features based on object display position parameters of the at least two objects to obtain adjusted image features; and further decoding the adjusted image features to obtain an object image containing the at least two objects. The image features can be understood as the information obtained by the vector processing module during the object image generation based on the target text vector. The attention mechanism is used to determine an attention map for each word in the object description text. In other words, the image feature can be understood as an attention map.The object display position parameter can be understood as a parameter representing the object's display position in the object image. This parameter can be coordinate information or region information. In practical applications, to achieve high-quality multi-topic (i.e., object) generation, this method uses layout (i.e., object display position parameter) as a spatial guide for arranging topics within the attention map. This layout is an abstract and easily accessible prior. Users can define it as a set of topic bounding boxes that describe the spatial composition of the topics. Specifically, this solution emphasizes the required regions corresponding to these topics in the attention map while imposing restrictions on regions associated with other topics based on the layout prior. This results in an adjusted attention map, which is then used to generate accurate object images, avoiding attribute confusion. For example, the image generation method provided in this disclosure, after obtaining a text embedding vector containing a residual vector, first inputs it into a U-net model (i.e., vector processing module). The U-net model decodes the text embedding vector and, through an attention mechanism, determines an attention map for the text embedding vector. Each word corresponds to an attention map. Secondly, based on the user-provided layout for the two objects "cat" and "dog," the two objects are arranged into spatial positions in the attention map to obtain an adjusted attention map. Finally, a U-net model is used to generate an image containing the cat and dog based on the adjusted attention map, thereby ensuring that each object is accurately displayed in the image and avoiding confusion among the object attributes. Furthermore, in an embodiment provided herein, adjusting the image features based on the object display position parameters of the at least two objects to obtain the adjusted image features includes: obtaining the object display position parameters of the at least two objects, and determining target object display position parameters corresponding to a target object from the object display position parameters, wherein the target object is any one of the at least two objects; determining target object image features corresponding to the target object from the image features; determining a target position of the target object within the target object image features based on the object display position parameters, and adjusting the object display weight of the target object at the target position to obtain the adjusted image features. The target object can be understood as any one of the at least two objects. For example, the at least two objects are a cat and a dog, and the target object refers to the cat or dog. The target object image feature refers to the cross-attention map corresponding to the target object in the multiple attention maps. For example, each word in the object description text "a cat and a dog" has a corresponding cross-attention map.The cross-attention maps corresponding to the words "cat" and "dog" are the cross-attention maps corresponding to the two objects, cat and dog. The object display weight can be understood as the weight of the target object displayed in a specific area. Alternatively, it can be understood as the signal of the target object in the image area where the user expects it to appear. The stronger the weight or signal, the more accurately the target is displayed in that image area; the weaker the weight or signal, the less accurately the target is displayed in that image area. Continuing with the above example, the image processing method provided by this disclosure can achieve the first problem of multi-theme customization: some themes may not be displayed as described in the text prompt. This method believes this is caused by insufficient activation values ​​in the cross-attention maps of these subjects. To avoid this, this solution strengthens the signal of the target subject in the area where the user expects it to appear and weakens the signals of irrelevant themes. The second problem in multi-theme customization is attribute confusion: the subject in the generated image may contain features of other themes. This solution believes this is due to overlapping activation areas of different subjects in the cross-attention maps. To avoid this, this method weakens the signal component of each customized object that appears in the area of ​​other customized objects. Based on this, this method obtains the layout for the two objects, cat and dog, sent by the user. From the multiple cross-attention maps corresponding to the text embedding vector for "a cat and a dog," this method determines the cross-attention maps corresponding to cat and dog, specifically the cross-attention maps corresponding to the two words "cat" and "dog." Based on the user-provided layout prior, the method determines the display region of each object in the corresponding cross-attention map. For example, the display region of the dog object in its corresponding cross-attention map is determined; the display region of the cat object in its corresponding cross-attention map is determined. Then, the weight of the corresponding object is strengthened in each region, while the weight of each object in other regions is weakened. For example, the signal (i.e., weight) of the dog object is strengthened in the region corresponding to the dog object, while the signal of the dog object in the display regions of other objects is weakened. This achieves the goal of selectively strengthening the signal of the target subject (object) in the region where the desired object appears, weakening the signal of irrelevant subjects, and selectively weakening the signal component of each customized object (i.e., object) that appears in the region of other customized objects, thereby resolving attribute confusion through the attention mechanism. Specifically, the method of adjusting the target subject signals in different regions of the cross-attention map based on the cross-attention mechanism can be referred to the following formula (1). Formula 1 represents how to enhance the signal component of the target object in its own region and weaken the signal component of the target object in other object regions.

[0004] EditedCAfCA. Service: Consult L ... ⑶} - (W v -

[0005] (Formula 1) Where IMitrfCA represents the cross-attention mechanism, represents the attention map, represents an activation function, represents time, represents the input text embedding (embedding vector), represents the dimension of the object (e.g., dog), represents the average value, and represents the weight matrix. Refers to the numerical value of I ayput provided by the user, for example, I ayput Parameters. In one embodiment of the present disclosure, this method addresses the problem of requiring retraining whenever a new object requires customization during the joint training of multiple concept data. This method proposes a flexible, multi-concept combination method. Simply finding a residual vector for each concept during model training allows for unlimited reuse. Based on this, the training process for the image generation model in this method is detailed as follows. Before inputting object description text containing at least two objects into the image generation model and determining the initial text vector of the object description text, the method further includes: determining training samples for the image generation model, wherein the training samples include sample object description text and a reference object image corresponding to the sample object description text; inputting the sample object description text into the image generation model, processing the sample object description text using a text encoding module in the image generation model to obtain a training text vector corresponding to the sample object description text; decoding the training text vector using a vector processing module in the image generation model to obtain a sample object image corresponding to the sample object description text, and generating a pre-stored object vector corresponding to the sample object based on the training text vector; determining a loss function for the image generation model based on the reference object image and the sample object image, and training the image generation model based on the loss function to obtain a trained image generation model. The sample object description text can be understood as a sample description text for a single object, for example, "an image of a dog," "a photo of a cat," or "a photo of a dog." The reference object image can be understood as the image of the object described in the sample object description text. For example, if the sample object description text is "a photo of a dog," the reference object image is an image of a dog. The training text vector can be understood as the text embedding vector corresponding to the sample object description text. For example, this method provides an image generation model training scheme. The goal of this scheme in training the diffusion model is to generate vivid and accurate new images that contain any combination of these themes. Based on this, this method requires determining the residual vector corresponding to each object. This goal can be achieved by subsequently combining the residual vectors of specific themes with the pre-trained diffusion model and using the layout to guide the generation process. Training the diffusion model includes: given a set of themes (sample object description text) from different perspectives, such as "a photo of a dog," and its corresponding 3-5 sample photos of dogs (reference object images) as training samples.The text "a photo of dog" is input into the diffusion model's text encoder, which outputs a text embedding vector (training text vector) corresponding to "a photo of dog." This text embedding vector is then input into the diffusion model's U-Net model to obtain the corresponding generated image. Simultaneously, this solution generates a residual vector corresponding to the dog object based on the text embedding vector corresponding to "a photo of dog." After obtaining the image generated by the U-Net model, a loss function for the diffusion model is determined based on the generated image and the sample photo. The diffusion model is then trained based on this loss function until training stops and adjustments are made, resulting in a trained diffusion model. In one embodiment provided in the specification, to reduce computing resource waste, it is necessary to determine a pre-stored object vector corresponding to each object. The pre-stored object vector is generated as follows: Generating a pre-stored object vector corresponding to a sample object based on the training text vector includes: determining a first sample object text vector corresponding to the sample object from the training text vector based on a sample object identifier in the sample object description text; inputting the sample object description text into a text encoding module in a pre-trained image generation model to obtain an untrained text vector for the sample object description text; determining a second sample object text vector corresponding to the sample object from the untrained text vector based on the sample object identifier; and using the average difference between the first sample object text vector and the second sample object text vector as the pre-stored object vector for the sample object. A sample object can be understood as an object serving as a sample, which is consistent with the object in the above-described embodiment. A sample object identifier can be understood as a term in the sample description text that directly represents a specific sample object. One sample object identifier directly represents one sample object. For example, in the sample object description text "a photo of dog," the term "dog" directly represents the sample object, dog. The first sample object text vector can be understood as the text embedding vector corresponding to the sample object in the training text vector. The pre-trained image generation model refers to the image generation model before this model training. The untrained text vector refers to the text embedding vector corresponding to the sample object description text obtained by the text encoding module in the pre-trained image generation model. Continuing with the above example, in the text-to-graph model (diffusion model), the user-provided text is encoded by the text encoder to obtain the corresponding text embedding vector. oIn determining the residual vector for each object, this method seeks an average change. This average change is calculated by using a trained text encoder and a frozen text encoder to calculate the average value of the dimension of the customized object (e.g., custom dog) for various texts, such as "a dog on the beach," "a dog sleeping," and so on. Frozen refers to a fixed, untrained encoder in the diffusion model. The average difference between the embedding vectors of these texts output by the two text encoders (i.e., the trained text vector and the untrained text vector) in the dog vector dimension (i.e., the first sample object text vector and the second sample object text vector) is calculated and used as the residual vector. It should be noted that the calculation method for this residual vector can be found in the following formula (2). Formula (2) shows that the method for finding each concept is to calculate the residual vector from the trained text encoder and the original text encoder, from the category to the specific object under different texts:

[0006] A custom Formula (2)

[0007] A custom L, where represents the residual vector of the object "dog," J refers to all dog-related texts and is used to calculate the mean of this change, £ refers to the trained text encoder, and C is the input text embedding. In the embodiments provided herein, during the training of an image generation model, the main content is to provide a small number of images of a custom theme and fine-tune the text encoder to learn a residual embedding based on the basic embedding of the original theme. Specifically, the current data loss calculation method for fine-tuning is as follows. Determining the loss function of the image generation model based on the reference object image and the sample object image includes: determining a first loss function of the image generation model based on the reference object image and the sample object image; determining a second loss function of the image generation model based on the untrained text vector and the trained text vector; and adding the first loss function and the second loss function to obtain the loss function of the image generation model. Where IE refers to averaging, Ψ is the diffusion model, f is the predicted noise, and Ψ is the input text embedding. After determining the first loss function, the embedding vectors output by the trained text encoder and the frozen (trained fixed) text encoder are determined, and other embedding vectors other than the embedding vector corresponding to the sample object are determined from the two embedding vectors. Based on the other embedding vectors, the second loss function is calculated. The loss function can be found in the following formula (4). Formula (4) is used to find the residual of each concept: Finally, the overall diffusion model loss is the sum of the two losses. The image generation method provided by the present disclosure does not require the image generation model to learn every multi-concept customized statement. Instead, it trains the image generation model to learn each individual object (concept). This avoids the low training efficiency of the neural network model and the waste of significant computing resources due to the large number of multi-concept customized statements. Furthermore, during model training, a pre-stored object vector corresponding to each individual object is generated and stored. Therefore, in practical applications, when object description text containing at least two objects is input to the image generation model, the image generation model can determine the pre-stored object vectors corresponding to the at least two objects, combine each pre-stored object vector with the initial text vector of the object description text, and then generate an object image containing the at least two objects based on the combined target text vector. This ensures the smooth operation of the image generation model and provides stable image generation capabilities. The following, combined with FIG4 , further illustrates the image generation method provided by the present disclosure using its application in a multi-concept customized generation scenario within a diffusion model as an example. FIG4 shows a flowchart of the processing process of an image generation method provided by one embodiment of the present disclosure. FIG4 shows that the image generation method provided by the present disclosure includes two phases: a training phase and an application phase. The processing process for the training phase includes: First, a set of text (i.e., the text "a photo of dog" in FIG4 ) is given as training samples from different perspectives, and reference images (2-5 dog images) corresponding to the training samples are determined. The training samples are input into a training text encoder to obtain a text embedding vector corresponding to the training samples. This text embedding vector is input into a frozen U-Net model. The U-Net model generates an image using a noise image and the text embedding vector, thereby obtaining the model output image. The diffusion model includes the text encoder and the U-Net model. Second, the training sample "a photo of dog" is input into the frozen text encoder. Finally, a loss function (e.g., Formula 3 above) is calculated based on the generated image and the reference image output by the U-Net model. Another loss function (e.g., Formula 4 above) is calculated using the text embeddings of the two text embeddings, excluding the dimension "dog." These two loss functions serve as the overall loss function of the diffusion model, and the diffusion model is trained based on this overall loss function to obtain a trained diffusion model. The training process includes the following: First, the user text "a cat and a dog" is received and input into the diffusion model to train a fixed text encoder and obtain a text embedding vector. The text embedding vectors for the two objects "cat" and "dog" are determined from the text embedding vector, and the residual vectors corresponding to the cat and dog (i.e., dog and dog in Figure 4) are obtained from the residual vectors cached locally. The residual vector is added to the text embedding vectors of the two objects, thereby adding the residual vector to the text embedding vector corresponding to "a cat and a dog". Next, the text embedding vectors are processed using an attention mechanism to determine the cross-attention map corresponding to each word in "a cat and a dog". For the cross-attention map of the two objects "cat" and "dog", based on the layout provided by the user (an abstract and easily accessible prior that the user can define as a set of subject bounding boxes), the regions of the two objects in the cross-attention map are determined, the weight of the corresponding object in each region is strengthened, and the weight of each object in other regions is reduced. Finally, after editing the cross-attention map, an image containing a cat and a dog is generated based on the adjusted cross-attention map. Based on the above content, it can be seen that the image generation method provided by the present disclosure, by embedding the residual token of a specific subject (i.e., the residual vector) Combined with a pre-trained diffusion model and using a layout-based word generation process, this approach allows for vivid and accurate generation of new images containing any combination of topics. Each topic is represented as a residual token embedding (i.e., a residual vector) transferred from its base category. Adding this residual token embedding (i.e., a residual vector) to the base category embedding (i.e., the embedding of "a cat and a dog" after passing it through the text encoder) allows the corresponding topic to be generated in the generated image. Furthermore, during inference, this approach overcomes two key issues in multi-topic customized generation: topic non-occurrence and attribute confusion between different topics.Figure 5 shows a flowchart of another image generation method provided according to an embodiment of the present disclosure. This image generation method, applied to a cloud-side device, specifically includes the following steps: Step 502: Receive object description text sent by a client-side device, wherein the object description text includes at least two objects. Step 504: Input the object description text into an image generation model to determine an initial text vector for the object description text. Step 506: Determine pre-stored object vectors for the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during the training process of the image generation model. Step 508: Process the initial text vector using the pre-stored object vectors to obtain a target text vector, and generate an object image including the at least two objects based on the target text vector. Step 510: Send the object image to the client-side device. The cloud-side device can be understood as a device located in the cloud and capable of providing cloud services to the client-side device. For example, the cloud-side device can be one or more servers or one or more hosts. In one embodiment provided by the present disclosure, the cloud-side device may also be composed of a cloud-side computing device and / or a cloud-side storage device. The cloud-side computing device can be understood as a device located in the cloud and capable of providing computing services to the end-side device. For example, the cloud-side computing device may be one or more servers. The cloud-side storage device can be understood as a device located in the cloud and capable of providing storage services to the end-side device. For example, one or more data storage servers, cloud disks, etc. The end-side device can be understood as a device that exists relative to the cloud-side device and is capable of using the cloud services provided by the cloud-side device. The end-side device includes but is not limited to a client, terminal, computer, server, mobile phone, or smart mobile device. Specifically, another image generation method provided by the present disclosure can be applied to a cloud-side device. When receiving object description text sent by a end-side device, the object description text containing at least two objects is input into an image generation model. The text encoding module in the image generation model is used to obtain an initial text vector of the object description text. Then, pre-stored object vectors for at least two objects are determined, and the initial text vector is adjusted based on the pre-stored object vectors, thereby adding the pre-stored object vectors to the initial text vector to obtain a target text vector. The target text vector is then processed using an image generation model to obtain an object image containing at least two objects. After the object image is obtained, it is sent to the client-side device.Based on this, the present disclosure provides another image generation method for cloud-side devices. This method eliminates the need for the image generation model to learn every multi-concept customized statement. Instead, the model is trained for each individual object (concept). This avoids the low training efficiency of the neural network model and the waste of significant computing resources due to the large number of multi-concept customized statements. Furthermore, during model training, a pre-stored object vector corresponding to each individual object is generated and stored. Therefore, in practical applications, after obtaining object description text sent by a client-side device, the object description text containing at least two objects is input into the image generation model. The image generation model can determine the pre-stored object vectors corresponding to the at least two objects, combine the pre-stored object vector for each object with the initial text vector of the object description text, and then generate an object image containing the at least two objects based on the combined target text vector. This object image is then returned to the client-side device. This ensures the smooth operation of the image generation model and enables the cloud-side device to stably provide image generation capabilities to the client-side device. The above is an exemplary embodiment of another image generation method of this embodiment. It should be noted that the technical solution of this alternative image generation method shares the same concept as the technical solution of the aforementioned image generation method. Details not described in detail in the technical solution of the alternative image generation method can be found in the description of the technical solution of the aforementioned image generation method and will not be elaborated upon here. FIG6 shows a flowchart of an image generation model training method according to an embodiment of the present disclosure, specifically comprising the following steps: Step 602: Determine training samples for the image generation model, wherein the training samples include sample object description text and reference object images corresponding to the sample object description text. Step 604: Input the sample object description text into the image generation model, and process the sample object description text using a text encoding module within the image generation model to obtain a training text vector corresponding to the sample object description text. Step 606: Decode the training text vector using a vector processing module within the image generation model to obtain a sample object image corresponding to the sample object description text, and generate a pre-stored object vector corresponding to the sample object based on the training text vector. Step 608: Determine a loss function of the image generation model according to the reference object image and the sample object image, and train the image generation model based on the loss function to obtain a trained image generation model.Specifically, for a description of the technical solution for an image generation model training method, please refer to the corresponding or corresponding description of the aforementioned image generation method, and this disclosure will not elaborate further here. The image generation model training method provided in this disclosure does not require the image generation model to learn every multi-concept customized statement. Instead, the image generation model is trained to learn each individual object (concept). This avoids the low training efficiency of the neural network model and the waste of significant computing resources due to the large number of multi-concept customized statements. Furthermore, during the model training process, a pre-stored object vector corresponding to each individual object is generated and stored. Consequently, in subsequent practical applications, when object description text containing at least two objects is input into the image generation model, the image generation model can determine the pre-stored object vectors corresponding to the at least two objects, combine each pre-stored object vector with the initial text vector of the object description text, and then generate an object image containing the at least two objects based on the combined target text vector. This ensures the smooth operation of the image generation model and provides stable image generation capabilities. The above is a schematic diagram of an image generation model training method according to this embodiment. It should be noted that the technical solution of this image generation model training method shares the same concept as the technical solution of the aforementioned image generation method. For details not described in detail in the technical solution of the image generation model training method, reference can be made to the description of the technical solution of the aforementioned image generation method, and will not be elaborated upon here. Corresponding to the aforementioned method embodiment, the present disclosure also provides an embodiment of an image generation device. FIG7 shows a schematic structural diagram of an image generation device according to one embodiment of the present disclosure. As shown in FIG7 , the apparatus includes: a first vector determination module 702 configured to input object description text containing at least two objects into an image generation model to determine an initial text vector of the object description text; a second vector determination module 704 configured to determine pre-stored object vectors of the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; and an image generation module 706 configured to process the initial text vector using the pre-stored object vectors to obtain a target text vector, and generate an object image containing the at least two objects based on the target text vector.Optionally, the second vector determination module 704 is further configured to: determine an object identifier in the object description text, wherein the object description text is composed of the object identifier and object feature words other than the object identifier, and each object identifier represents one object; determine a target pre-stored object vector associated with the object identifier from pre-stored object vectors; and determine the target pre-stored object vector as the pre-stored object vector for the at least two objects. Optionally, the image generation module 706 is further configured to: determine, based on the object identifier in the object description text, vectors to be adjusted corresponding to the at least two objects from the initial text vector; and add the pre-stored object vector to the vector to be adjusted in the initial text vector to obtain a target text vector. Optionally, the image generation module 706 is further configured to: process the target text vector using a vector processing module in the image generation model to obtain an object image containing the at least two objects. Optionally, the image generation module 706 is further configured to: decode the target text vector using the vector processing module to obtain image features corresponding to the object description text; adjust the image features based on the object display position parameters of the at least two objects to obtain adjusted image features; and continue decoding the adjusted image features to obtain an object image containing the at least two objects. Optionally, the image generation module 706 is further configured to: obtain object display position parameters of the at least two objects, and determine target object position parameters corresponding to a target object from the object display position parameters, wherein the target object is any one of the at least two objects; determine target object image features corresponding to the target object from the image features; determine a target position of the target object within the target object image features based on the object display position parameters, and adjust the object display weight of the target object at the target position to obtain adjusted image features. Optionally, the first vector determination module 702 is further configured to: determine an object description text containing at least two objects; input the object description text into an image generation model, and process the object description text using a text encoding module in the image generation model to obtain an initial text vector corresponding to the object description text.Optionally, the first vector determination module 702 is further configured to: receive an image generation request and obtain object description text containing at least two objects from the image generation request; or obtain a text identifier from the received image generation request and determine an object description text corresponding to the text identifier from pre-stored object description texts, wherein the object description text contains at least two objects. Optionally, the image generation device further includes a model training module, which is configured to: determine a training sample for an image generation model, wherein the training sample includes a sample object description text and a reference object image corresponding to the sample object description text; input the sample object description text into the image generation model, and process the sample object description text using a text encoding module in the image generation model to obtain a training text vector corresponding to the sample object description text; decode the training text vector using a vector processing module in the image generation model to obtain a sample object image corresponding to the sample object description text, and generate a pre-stored object vector corresponding to the sample object based on the training text vector; determine a loss function of the image generation model based on the reference object image and the sample object image, and train the image generation model based on the loss function to obtain a trained image generation model. Optionally, the model training module is further configured to: determine a first sample object text vector corresponding to the sample object from the training text vector based on sample object identifiers in the sample object description text; input the sample object description text into a text encoding module in a pre-trained image generation model to obtain an untrained text vector for the sample object description text; determine a second sample object text vector corresponding to the sample object from the untrained text vector based on the sample object identifiers; and use the average difference between the first sample object text vector and the second sample object text vector as the pre-stored object vector for the sample object. Optionally, the model training module is further configured to: determine a first loss function for the image generation model based on the reference object image and the sample object image; determine a second loss function for the image generation model based on the untrained text vector and the training text vector; and add the first loss function and the second loss function to obtain a loss function for the image generation model. The image generation device provided by the embodiments of the present disclosure does not require the image generation model to learn every multi-concept customized sentence. Instead, the image generation model is trained to learn each individual object (concept). This avoids the problem of low training efficiency of the neural network model and waste of a large amount of computing resources due to the large number of multi-concept customized sentences.Simultaneously, during the model training process, a pre-stored object vector corresponding to each individual object is generated and stored. Therefore, in practical applications, when object description text containing at least two objects is input into the image generation model, the image generation model can determine the pre-stored object vectors corresponding to the at least two objects, combine each pre-stored object vector with the initial text vector of the object description text, and then generate an object image containing the at least two objects based on the combined target text vector. This ensures the smooth operation of the image generation model and provides stable image generation capabilities. The above is a schematic diagram of an image generation device according to this embodiment. It should be noted that the technical solution of this image generation device and the technical solution of the above-mentioned image generation method are based on the same concept. For details not described in detail in the technical solution of the image generation device, please refer to the description of the technical solution of the above-mentioned image generation method. Corresponding to the above-mentioned method embodiment, the present disclosure also provides an embodiment of an image generation model training device. Figure 8 shows a schematic structural diagram of an image generation model training device according to one embodiment of the present disclosure. As shown in FIG8 , the apparatus includes: a sample determination module 802 configured to determine a training sample for an image generation model, wherein the training sample includes sample object description text and a reference object image corresponding to the sample object description text; a vector acquisition module 804 configured to input the sample object description text into the image generation model, process the sample object description text using a text encoding module in the image generation model, and obtain a training text vector corresponding to the sample object description text; an image acquisition module 806 configured to decode the training text vector using a vector processing module in the image generation model to obtain a sample object image corresponding to the sample object description text, and generate a pre-stored object vector corresponding to the sample object based on the training text vector; and a loss determination module 808 configured to determine a loss function of the image generation model based on the reference object image and the sample object image, and train the image generation model based on the loss function to obtain a trained image generation model. The image generation model training device provided in this disclosure eliminates the need for the image generation model to learn every multi-concept customized sentence. Instead, it trains the model to learn each individual object (concept). This avoids the low training efficiency of neural network models and the waste of significant computing resources caused by the large number of multi-concept customized sentences. Furthermore, during model training, a pre-stored object vector corresponding to each individual object is generated and stored.Therefore, in subsequent practical applications, when object description text containing at least two objects is input into the image generation model, the image generation model can determine the pre-stored object vectors corresponding to the at least two objects, combine the pre-stored object vector for each object with the initial text vector of the object description text, and then generate an object image containing the at least two objects based on the combined target text vector. This ensures the smooth operation of the image generation model and provides stable image generation capabilities. The above is a schematic diagram of an image generation model training device according to this embodiment. It should be noted that the technical solution of this image generation model training device and the technical solution of the above-mentioned image generation model training method are based on the same concept. For details not described in detail in the technical solution of the image generation model training device, please refer to the description of the technical solution of the above-mentioned image generation model training method. Corresponding to the above-mentioned method embodiment, the present disclosure also provides another embodiment of an image generation device. Figure 9 shows a schematic structural diagram of another image generation device provided by one embodiment of the present disclosure. As shown in FIG9 , the apparatus is applied to a cloud-side device and includes: a text receiving module 902 configured to receive object description text sent by a terminal-side device, wherein the object description text includes at least two objects; a first vector determination module 904 configured to input the object description text into an image generation model to determine an initial text vector of the object description text; a second vector determination module 906 configured to determine pre-stored object vectors of the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; an image generation module 908 configured to process the initial text vector using the pre-stored object vector to obtain a target text vector, and generate an object image including the at least two objects based on the target text vector; and an image sending module 910 configured to send the object image to the terminal-side device. Another image generation device provided by this disclosure, applied to cloud-side devices, does not require the image generation model to learn every multi-concept customized statement. Instead, it trains the image generation model to learn each individual object (concept). This avoids the low training efficiency of the neural network model and the waste of significant computing resources caused by the large number of multi-concept customized statements. Furthermore, during model training, a pre-stored object vector corresponding to each individual object is generated and stored.Therefore, in actual applications, after obtaining the object description text sent by the end-side device, the object description text containing at least two objects is input into the image generation model. The image generation model can determine the pre-stored object vectors corresponding to the at least two objects, combine the pre-stored object vector for each object with the initial text vector of the object description text, and then generate an object image containing the at least two objects based on the combined target text vector. This object image is then returned to the end-side device, thereby ensuring the smooth operation of the image generation model and enabling the cloud-side device to stably provide image generation capabilities to the end-side device. The above is a schematic diagram of another image generation device according to this embodiment. It should be noted that the technical solution of this another image generation device and the technical solution of the another image generation method described above are based on the same concept. For details of the technical solution of the another image generation device, please refer to the description of the technical solution of the another image generation method described above. Figure 10 shows a block diagram of the structure of a computing device 1000 provided according to one embodiment of the present disclosure. Components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030. A database 1050 is used to store data. The computing device 1000 also includes an access device 1040, which enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include the public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like. In one embodiment of the present disclosure, the aforementioned components of computing device 1000 and other components not shown in FIG. 10 may also be connected to one another, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 10 is for illustrative purposes only and does not limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed.Computing device 1000 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 1000 can also be a mobile or stationary server. Processor 1020 is configured to execute the following computer-executable instructions, which, when executed by processor 1020, implement the steps of the two aforementioned image generation methods and image generation model training method. Embodiments of the present disclosure also provide a processor. Figure 11 shows a schematic diagram of the structure of a processor used to implement embodiments of the present disclosure. As shown in Figure 11, processor 1100 is configured to run a program, wherein, when executed by the processor, the program executes the methods of the aforementioned embodiments. In embodiments of the present disclosure, processor 1100 can execute the program for executing the image generation method. Optionally, in this embodiment, the processor 1100 may be configured to perform the following steps: inputting object description text containing at least two objects into an image generation model to determine an initial text vector of the object description text; determining pre-stored object vectors of the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; processing the initial text vector using the pre-stored object vector to obtain a target text vector, and generating an object image containing the at least two objects based on the target text vector. Optionally, in this embodiment, the processor 1100 may be configured to perform the following steps: determining a training sample for an image generation model, wherein the training sample includes sample object description text and a reference object image corresponding to the sample object description text; inputting the sample object description text into the image generation model, processing the sample object description text using a text encoding module in the image generation model to obtain a training text vector corresponding to the sample object description text; decoding the training text vector using a vector processing module in the image generation model to obtain a sample object image corresponding to the sample object description text, and generating a pre-stored object vector corresponding to the sample object based on the training text vector; determining a loss function of the image generation model based on the reference object image and the sample object image, and training the image generation model based on the loss function to obtain a trained image generation model.Optionally, in this embodiment, the processor 1100 may be configured to perform the following steps: receiving object description text sent by an end-side device, wherein the object description text includes at least two objects; inputting the object description text into an image generation model to determine an initial text vector for the object description text; determining pre-stored object vectors for the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; processing the initial text vector using the pre-stored object vectors to obtain a target text vector, and generating an object image including the at least two objects based on the target text vector; and sending the object image to the end-side device. The above is a schematic scheme of a computing device in this embodiment. It should be noted that the technical scheme of this computing device is based on the same concept as the technical schemes of the two aforementioned image generation methods and image generation model training methods. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical schemes of the two aforementioned image generation methods and image generation model training methods. One embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the two aforementioned image generation methods and image generation model training methods. The above is an illustrative embodiment of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium shares the same concept as the technical solutions of the two aforementioned image generation methods and image generation model training methods. For details not described in detail regarding the technical solution of the storage medium, reference can be made to the description of the technical solutions of the two aforementioned image generation methods and image generation model training methods. The computer-readable storage medium may include a data signal transmitted in baseband or as part of a carrier wave, carrying readable program code. This transmitted data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable storage medium may transmit, disseminate, or transfer a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. The program code contained in the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, radio frequency, or any suitable combination thereof. An embodiment of the present disclosure further provides a computer program, wherein when the computer program is executed in a computer, the computer is instructed to execute the steps of the above two image generation methods and the image generation model training method.Optionally, the computer program, when executed by a processor, implements program code for the following steps: inputting object description text containing at least two objects into an image generation model to determine an initial text vector of the object description text; determining pre-stored object vectors of the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; processing the initial text vector using the pre-stored object vector to obtain a target text vector, and generating an object image containing the at least two objects based on the target text vector. Optionally, the computer program implements program code for the following steps when executed by a processor: determining a training sample for an image generation model, wherein the training sample includes sample object description text and a reference object image corresponding to the sample object description text; inputting the sample object description text into the image generation model, processing the sample object description text using a text encoding module in the image generation model to obtain a training text vector corresponding to the sample object description text; decoding the training text vector using a vector processing module in the image generation model to obtain a sample object image corresponding to the sample object description text, and generating a pre-stored object vector corresponding to the sample object based on the training text vector; determining a loss function of the image generation model based on the reference object image and the sample object image, and training the image generation model based on the loss function to obtain a trained image generation model. Optionally, the computer program, when executed by a processor, implements program code for the following steps: receiving object description text sent by an end-side device, wherein the object description text includes at least two objects; inputting the object description text into an image generation model to determine an initial text vector for the object description text; determining pre-stored object vectors for the at least two objects, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during training of the image generation model; processing the initial text vector using the pre-stored object vectors to obtain a target text vector, and generating an object image including the at least two objects based on the target text vector; and sending the object image to the end-side device. The above is an illustrative embodiment of a computer program according to this embodiment. It should be noted that the technical solution of this computer program is based on the same concept as the technical solutions of the two aforementioned image generation methods and image generation model training methods. For details not described in detail in the technical solution of the computer program, reference can be made to the description of the technical solutions of the two aforementioned image generation methods and image generation model training methods. The above describes specific embodiments of the present disclosure.Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous. The computer instructions include computer program code, which may be in source code form, object code form, executable files, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunications signals, and software distribution media. It should be noted that the content of the computer-readable medium may be appropriately expanded or reduced based on the requirements of legislation and patent practice within a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, computer-readable media does not include electrical carrier signals or telecommunications signals. It should be noted that, for ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present disclosure are not limited to the order of the actions described, as certain steps may be performed in a different order or simultaneously, depending on the embodiments of the present disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules described are not necessarily required for the embodiments of the present disclosure. In the above embodiments, the description of each embodiment has its own emphasis. For portions not described in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. The preferred embodiments disclosed above are merely intended to help illustrate the present disclosure. The optional embodiments do not describe all details in detail, nor do they limit the present disclosure to the specific implementation methods described. Obviously, many modifications and variations are possible based on the content of the embodiments of the present disclosure. The present disclosure selects and describes these embodiments in detail to better explain the principles and practical applications of the embodiments of the present disclosure, thereby enabling those skilled in the art to better understand and utilize the present disclosure. The present disclosure is limited only by the claims and their full scope and equivalents.Industrial Applicability The solution provided by the embodiments of the present disclosure can be applied to the process of image generation. Object description text containing at least two objects is input into an image generation model to determine an initial text vector of the object description text; pre-stored object vectors of the at least two objects are determined, wherein the pre-stored object vectors are generated based on training text vectors for the at least two objects during the training process of the image generation model; the initial text vector is processed using the pre-stored object vector to obtain a target text vector, and an object image containing the at least two objects is generated based on the target text vector, thereby solving the technical problem of wasting a large amount of computing resources.

Claims

Claims 1. A method for generating an image, comprising: Inputting object description text containing at least two objects into an image generation model to determine an initial text vector of the object description text; Determine pre-stored object vectors of the at least two objects, wherein the pre-stored object vectors are generated according to training text vectors for the at least two objects in the training process of the image generation model; process the initial text vector by using the pre-stored object vector to obtain a target text vector, and generate an object image Q containing the at least two objects according to the target text vector.

2. The image generation method according to claim 1, wherein: The determining of the pre-stored object vectors of the at least two objects includes: determining an object identifier in the object description text, wherein the object description text is composed of the object identifier and object feature words other than the object identifier, and one object identifier represents one object; determining a target pre-stored object vector associated with the object identifier from the pre-stored object vector; and determining the target pre-stored object vector as the pre-stored object vector of the at least two objects.

3. The image generation method according to claim 1, wherein: The processing of the initial text vector by using the pre-stored object vector to obtain a target text vector includes: determining, from the initial text vector, vectors to be adjusted corresponding to the at least two objects according to object identifiers in the object description text; and adding the pre-stored object vector to the vector to be adjusted in the initial text vector to obtain the target text vector.

4. The image generation method according to claim 1, wherein: The step of generating an object image including the at least two objects according to the target text vector includes: utilizing a vector processing module in the image generation model to process the target text vector to obtain an object image including the at least two objects.

5. The image generation method according to claim 4, wherein: The method of using the vector processing module in the image generation model to process the target text vector to obtain an object image containing the at least two objects includes: using the vector processing module to decode the target text vector to obtain image features corresponding to the object description text; adjusting the image features according to object display position parameters of the at least two objects to obtain an adjusted image. 29 continuing to decode the adjusted image features to obtain an object image containing the at least two objects.

6. The image generation method according to claim 5, wherein: The step of adjusting the image features according to the object display position parameters of the at least two objects to obtain the adjusted image features includes: acquiring the object display position parameters of the at least two objects, and determining target object position parameters corresponding to a target object from the object display position parameters, wherein the target object is any one of the at least two objects; determining a target object image feature corresponding to the target object from the image features; determining a target position of the target object in the target object image feature based on the object display position parameters, and adjusting an object display weight of the target object at the target position to obtain the adjusted image features.

7. The image generation method according to claim 1, wherein: The step of inputting an object description text including at least two objects into an image generation model to determine an initial text vector of the object description text includes: determining an object description text including at least two objects; inputting the object description text into an image generation model, processing the object description text using a text encoding module in the image generation model, and obtaining an initial text vector corresponding to the object description text.

8. The image generation method according to claim 1, wherein: The determining of the object description text containing at least two objects includes: receiving an image generation request, and obtaining the object description text containing the at least two objects from the image generation request; or obtaining a text identifier from the received image generation request, and determining the object description text corresponding to the text identifier from pre-stored object description texts, wherein the object description text contains the at least two objects.

9. The image generation method according to claim 1, wherein: Before inputting the object description text containing at least two objects into the image generation model and determining the initial text vector of the object description text, the method further includes: determining a training sample for the image generation model, wherein the training sample contains a sample object description text and a reference object image corresponding to the sample object description text; inputting the sample object description text into the image generation model, processing the sample object description text by using a text encoding module in the image generation model, and obtaining a training text vector corresponding to the sample object description text; decoding the training text vector by using a vector processing module in the image generation model, and obtaining a training text vector corresponding to the sample object description text. 30 Obtain a sample object image corresponding to the sample object description text, and generate a pre-stored object vector corresponding to the sample object according to the training text vector; determine a loss function of the image generation model according to the reference object image and the sample object image, and train the image generation model based on the loss function to obtain the trained image generation model.

10. The image generation method according to claim 9, wherein: The generating a pre-stored object vector corresponding to the sample object according to the training text vector includes: determining a first sample object text vector corresponding to the sample object from the training text vector according to a sample object identification word in the sample object description text; inputting the sample object description text into a text encoding module in a pre-trained image generation model to obtain an untrained text vector of the sample object description text; determining a second sample object text vector corresponding to the sample object from the untrained text vector according to the sample object identification word; and using an average difference between the first sample object text vector and the second sample object text vector as the pre-stored object vector of the sample object.

11. The image generation method according to claim 10, wherein: Determining the loss function of the image generation model based on the reference object image and the sample object image includes: determining a first loss function of the image generation model based on the reference object image and the sample object image; determining a second loss function of the image generation model based on the untrained text vector and the trained text vector; and adding the first loss function and the second loss function to obtain the loss function of the image generation model.

12. A method for training an image generation model, comprising: Determine a training sample for an image generation model, wherein the training sample includes a sample object description text and a reference object image corresponding to the sample object description text; input the sample object description text into the image generation model, and process the sample object description text using a text encoding module in the image generation model to obtain a training text vector corresponding to the sample object description text; decode the training text vector using a vector processing module in the image generation model to obtain a sample object image corresponding to the sample object description text, and generate a pre-stored object vector corresponding to the sample object according to the training text vector; determine a loss function of the image generation model according to the reference object image and the sample object image, The image generation model is trained based on the loss function to obtain a trained image generation model.

13. An image generation method, applied to a cloud-side device, comprising: Receive an object description text sent by an end-side device, wherein the object description text contains at least two objects; input the object description text into an image generation model to determine an initial text vector of the object description text; determine pre-stored object vectors of the at least two objects, wherein the pre-stored object vectors are generated according to training text vectors for the at least two objects during a training process of the image generation model; process the initial text vector through the pre-stored object vector to obtain a target text vector, and generate an object image containing the at least two objects according to the target text vector; and send the object image to the end-side device.

14. The image generation method according to claim 13, wherein: The determining of the pre-stored object vectors of the at least two objects includes: determining an object identifier in the object description text, wherein the object description text is composed of the object identifier and object feature words other than the object identifier, and one object identifier represents one object; determining a target pre-stored object vector associated with the object identifier from the pre-stored object vectors; and determining the target pre-stored object vector as the pre-stored object vector of the at least two objects.

15. The image generation method according to claim 13, wherein: The processing of the initial text vector by the pre-stored object vector to obtain a target text vector includes: determining, from the initial text vector, vectors to be adjusted corresponding to the at least two objects according to object identifiers in the object description text; and adding the pre-stored object vector to the vector to be adjusted in the initial text vector to obtain the target text vector.

16. A computer-readable storage medium storing computer-executable instructions, wherein: When the computer executable instructions are executed by a processor, the steps of the image generation method described in any one of claims 1 to 11, the image generation model training method described in claim 12, and the image generation method described in any one of claims 13 to 15 are implemented.

17. An electronic device comprising: one or more processors; And a memory associated with the one or more processors, the memory being configured to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the program instructions execute the steps of the image generation method described in any one of claims 1 to 11, the image generation model training method described in claim 12, and the image generation method described in any one of claims 13 to 15.

18. A computer program product, wherein: Comprising a computer program, which, when executed by a processor, implements the steps of the image generation method described in any one of claims 1 to 11, the image generation model training method described in claim 12, and the image generation method described in any one of claims 13 to 15.

19. A computer program product, wherein: The present invention comprises a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the image generation method according to any one of claims 1 to 11, the image generation model training method according to claim 12, and the steps of the image generation method according to any one of claims 13 to 15 are implemented.

20. A computer program, wherein When the computer program is executed by a processor, it implements the steps of the image generation method described in any one of claims 1 to 11, the image generation model training method described in claim 12, and the image generation method described in any one of claims 13 to 15. 33

Citation Information

Patent Citations

  • Text image generation method, electronic equipment and storage medium

    CN110443863A

Cited By

  • Method and device for generating video from text and storage medium

    CN119697456A

  • Object sample image generation method and device and electronic equipment

    CN119784867A

  • Generating a series of contextually-persistent visual images for text documents utilizing multiple models

    US20250078351A1