Image generation method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of image generation, and provides an image generation method and device, electronic equipment and a medium. According to the method, text description information can be processed through a large language model, and feature vectors of the text description information are obtained; performing semantic feature extraction on the initial image through a first encoder to obtain a multi-modal semantic feature vector of the initial image; performing image feature extraction on the initial image through a second encoder to obtain an image feature vector of the initial image; splicing the two feature vectors to obtain a semantic fusion feature vector; performing fusion processing on the image feature vector and the semantic fusion feature vector of the initial image through a text image adaptation module to obtain an image text fusion feature vector; and performing denoising processing on the image text fusion feature vector based on a preset noise map through a diffusion model to obtain a target image of the to-be-generated object. The quality and efficiency of text-to-image conversion are improved, and the high-quality image consistent with text description is generated.

Description

Technical Field

[0001] The present disclosure relates to the field of image generation technology, and in particular to an image generation method, device, electronic device and medium. Background Art

[0002] With the rapid development of the Internet, users are increasingly inclined to use applications on terminal devices to convert text into images. For example, text descriptions such as "the cat is chasing the mouse" can be converted into corresponding images through these applications. At present, a common approach is to use diffusion models to generate corresponding images based on text content. However, in the process of text-to-image conversion, most of the existing diffusion models rely on CLIP (Contrastive Language-Image Pre-training) as a text encoder. This dependence limits the ability of diffusion models to handle complex text cues, especially in understanding dense cues such as multiple objects, detailed attribute descriptions, complex relationships, and long text alignment. These limitations pose an obstacle to generating high-quality images that are highly semantically consistent with text descriptions. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide an image generation method, device, electronic device and computer-readable storage medium to solve the technical problem of the ability of existing restricted diffusion models to process complex text prompts.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image generation method, including: obtaining text description information of an object to be generated and an initial image of the object to be generated; processing the text description information through a large language model to obtain a feature vector of the text description information; extracting semantic features of the initial image through a first encoder to obtain a multimodal semantic feature vector of the initial image; extracting image features of the initial image through a second encoder to obtain an image feature vector of the initial image; concatenating the feature vector of the text description information and the multimodal semantic feature vector of the initial image to obtain a semantic fusion feature vector; fusing the image feature vector of the initial image and the semantic fusion feature vector through a text-image adaptation module to obtain an image-text fusion feature vector; and denoising the image-text fusion feature vector based on a preset noise map through a diffusion model to obtain a target image of the object to be generated.

[0005] According to a second aspect of an embodiment of the present disclosure, an image generating device is provided, comprising: an acquisition module for acquiring text description information of an object to be generated and an initial image of the object to be generated; a text information processing module for processing the text description information through a large language model to obtain a feature vector of the text description information; a semantic feature extraction module for performing semantic feature extraction on the initial image through a first encoder to obtain a multimodal semantic feature vector of the initial image; an image extraction module for performing image feature extraction on the initial image through a second encoder to obtain an image feature vector of the initial image; a splicing module for splicing the feature vector of the text description information and the multimodal semantic feature vector of the initial image to obtain a semantic fusion feature vector; a fusion module for performing fusion processing on the image feature vector of the initial image and the semantic fusion feature vector through a text image adaptation module to obtain an image-text fusion feature vector; and a denoising module for performing denoising processing on the image-text fusion feature vector based on a preset noise map through a diffusion model to obtain a target image of the object to be generated.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0008] Compared with the prior art, the embodiments of the present disclosure have the following beneficial effects: the embodiments of the present disclosure can extract features from text description information and images by using the first encoder and the second encoder, so as to better process complex texts containing dense prompts such as multiple objects, detailed attribute descriptions, complex relationships, and long text alignment, thereby improving the ability to process complex text prompts. By splicing the feature vector of the text description information with the multimodal semantic feature vector of the initial image, and fusing the image feature vector and the semantic fusion feature vector through the text image adaptation module, a more accurate, high-quality image that is highly consistent with the text description can be generated, thereby enhancing the accuracy and consistency of image generation. By improving the efficiency and quality of text-to-image conversion, it helps to improve the user experience, allowing users to more easily convert text descriptions into images that meet expectations. At the same time, the generated high-quality images are highly consistent with the text descriptions, which will expand the application fields, such as virtual reality, augmented reality, e-commerce, etc. By denoising the image-text fusion feature vector based on a preset noise map through a diffusion model, the quality of the generated image can be further improved, possible noise can be eliminated, and the final generated image can be clearer and more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied;

[0011] Figure 2 is a flowchart of an image generation method provided by an embodiment of the present disclosure;

[0012] Figure 3 is a framework diagram of an image generation model provided by an embodiment of the present disclosure;

[0013] Figure 4 is a framework diagram of an adapter provided by an embodiment of the present disclosure;

[0014] Figure 5 is a flowchart of another image generation method provided by an embodiment of the present disclosure;

[0015] Figure 6 is a block diagram of an image generating device provided by an embodiment of the present disclosure;

[0016] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0018] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0019] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied.

[0020] like Figure 1 As shown, the system architecture 100 may include one or more of a first terminal device 101, a second terminal device 102, and a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0021] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0022] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send data, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, and the like.

[0023] The server 105 may be a server that provides various services. For example, the server 105 may obtain text description information of the object to be generated and the initial image of the object to be generated from the first terminal device 103 (which may also be the second terminal device 102 or the third terminal device 103); process the text description information through the large language model to obtain a feature vector of the text description information; extract semantic features from the initial image through the first encoder to obtain a multimodal semantic feature vector of the initial image; extract image features from the initial image through the second encoder to obtain an image feature vector of the initial image; concatenate the feature vector of the text description information and the multimodal semantic feature vector of the initial image to obtain a semantic fusion feature vector; fuse the image feature vector of the initial image and the semantic fusion feature vector through the text image adaptation module to obtain an image-text fusion feature vector; denoise the image-text fusion feature vector based on a preset noise map through a diffusion model to obtain a target image of the object to be generated.

[0024] In some embodiments, the image generation method provided in the embodiments of the present invention is generally executed by the server 105, and accordingly, the image generation device is generally disposed in the server 105. In other embodiments, some terminal devices may have functions similar to those of the server to execute the method. Therefore, the image generation method provided in the embodiments of the present invention is not limited to being executed on the server side.

[0025] The image generation method and device according to the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Figure 2 is a flowchart of an image generation method provided by an embodiment of the present disclosure. The method provided by an embodiment of the present disclosure can be executed by any electronic device with computer processing capabilities, for example, the electronic device can be Figure 1 The server is shown.

[0027] like Figure 2 As shown, the image generating method includes steps S210 to S270.

[0028] In step S210, text description information of the object to be generated and an initial image of the object to be generated are obtained.

[0029] Step S220: Process the text description information through a large language model to obtain a feature vector of the text description information.

[0030] In step S230, semantic features are extracted from the initial image by a first encoder to obtain a multimodal semantic feature vector of the initial image.

[0031] In step S240, the second encoder extracts image features from the initial image to obtain an image feature vector of the initial image.

[0032] In step S250, the feature vector of the text description information and the multimodal semantic feature vector of the initial image are concatenated to obtain a semantic fusion feature vector.

[0033] In step S260, the image feature vector and the semantic fusion feature vector of the initial image are fused by the text image adaptation module to obtain an image-text fusion feature vector.

[0034] In step S270, the image-text fusion feature vector is denoised based on a preset noise map using a diffusion model to obtain a target image of the object to be generated.

[0035] The method can extract features from text description information and images by using a first encoder and a second encoder, and can better process complex text containing dense prompts such as multiple objects, detailed attribute descriptions, complex relationships, and long text alignment, thereby improving the ability to process complex text prompts. By splicing the feature vector of the text description information with the multimodal semantic feature vector of the initial image, and fusing the image feature vector and the semantic fusion feature vector through the text image adaptation module, a more accurate, high-quality image that is highly consistent with the text description can be generated, and the accuracy and consistency of image generation are enhanced. By improving the efficiency and quality of text-to-image conversion, it helps to improve the user experience, allowing users to more easily convert text descriptions into images that meet their expectations. At the same time, the generated high-quality images are highly consistent with the text descriptions, which will expand the application areas, such as virtual reality, augmented reality, e-commerce, etc. De-noising the image-text fusion feature vector based on a preset noise map through a diffusion model can further improve the quality of the generated image, eliminate possible noise, and make the final generated image clearer and more realistic.

[0036] In some embodiments of the present disclosure, the text description information of the object to be generated may be the text description information of one or more objects to be generated. In the task of text-to-image conversion, the input text description information may cover the description information of one or more objects to be generated. Specifically, single object description: the text description information may only involve the description of a single object to be generated. For example, describing an orange kitten playing on the grass, such a description covers the appearance characteristics and behavior of a single object (kitten). Multiple object description: the text description information may also contain descriptions of multiple objects to be generated. For example, describing a family gathering scene, in which people are dining around a table, various foods are placed on the table, and decorations are hung in the room. This description involves multiple objects (people, tables, food, decorations, etc.) and the relationships between them. When processing such diverse text description information, the method of the present disclosure can accurately understand and interpret information such as objects, attributes, relationships, etc. described in the text, so as to generate an image consistent with the text description.

[0037] In some embodiments of the present disclosure, the initial image of the object to be generated may be a reference image of the object to be generated, and the reference image may be a state of the object to be generated. In the text-to-image conversion task, a reference image containing the object to be generated may be used as the initial image, that is, the initial image may show a certain state or feature of the object to be generated. Specifically, a reference image containing the object to be generated is used as the initial image to better understand the appearance features, morphology and other information of the object to be generated. Such a reference image may provide clues about the basic appearance and features of the object to be generated. A state or feature of the object to be generated may be shown in the reference image. For example, in the case where the text description information is "running cat and mouse", the reference image may contain a cat and mouse in a stationary state. In this case, the information in these reference images may be used to better understand the appearance and features of the object to be generated, so as to generate an image consistent with the text description. For describing the object in the reference image, its appearance features, posture, expression and other information need to be described in detail to ensure that the generated image is consistent with the text description. For example, for a cat and mouse in a stationary state, their color, posture, expression and other features need to be considered in order to generate an image of a running state. By using a reference image containing the object to be generated as the initial image, we can better understand the characteristics and state of the object to be generated, thereby generating an image that is more consistent with the text description. This method helps to improve the accuracy and consistency of the generated image and enhance the effect of text-to-image conversion.

[0038] Reference below Figure 3 The framework diagram of the image generation model is shown to describe Figure 2 The various steps in the embodiment are as follows:

[0039] The above-mentioned large prediction model can generally refer to a pre-trained large-scale language model, such as the GPT (Generative Pre-trained Transformer) series of models. These models have been pre-trained on large-scale text data and have strong text understanding and representation learning capabilities. Specifically, the text description information is input into the pre-trained large language model. The large language model processes the input text and converts the text into a high-dimensional feature vector representation by learning the semantic and contextual information in the text. After processing, the text description information will be represented as a feature vector, which contains the semantic information and contextual associations of the text description. The feature vector of the obtained text description information can be used in subsequent image generation tasks, and spliced ​​or fused with the image feature vector to help the system generate images consistent with the text description. By using the pre-trained large language model to process the text description information, the meaning, context and semantic information of the text can be better understood, thereby extracting rich feature representations and providing strong support for subsequent text-to-image conversion tasks. This method helps to improve the system's ability to understand text description information, thereby generating images that are more consistent with the text description.

[0040] The first encoder may be a BLIP (Bidirectional Latent Intention Prediction) 2 encoder, which is used to extract semantic features of the initial image. The BLIP2 encoder is a multimodal encoder that processes visual information such as images and converts it into a semantic feature vector. For example, information of different modalities (such as images and text) is converted into a shared semantic representation, thereby promoting the learning and understanding of multimodal tasks. By extracting semantic features from the initial image through the BLIP2 encoder, a multimodal semantic feature vector of the initial image can be obtained. This feature vector captures the semantic information of the image, which can better understand the meaning and content of the image. The multimodal semantic feature vector generated by the BLIP2 encoder contains the semantic information of the image, which combines the visual features of the image with the corresponding semantic representation. Such a feature vector can be used in combination with the feature vector of the text description information to more accurately generate an image consistent with the text description.

[0041] The second encoder may be a CLIP encoder, which is used to extract image features of the initial image. CLIP is an alignment-based model for processing the relationship between text and image. It can understand text descriptions and image content simultaneously and map them into a shared embedding space, thereby achieving cross-modal learning and understanding. By performing image feature extraction on the initial image through the CLIP encoder, an image feature vector of the initial image can be obtained. This feature vector contains the visual information of the image and can help the system better understand the content and features of the image. The image feature vector generated by the CLIP encoder captures the visual features and content information of the image and converts it into a high-dimensional vector representation. Such a feature vector can be used in combination with the feature vector of the text description information to improve the performance and accuracy of the system in the text-to-image conversion task. By using the CLIP encoder to extract the image features of the initial image, the image information can be converted into a feature vector representation that can be understood and processed by the model. The generation of such an image feature vector helps to improve the system's ability to understand the image content, thereby providing more accurate and consistent image generation effects in the text-to-image conversion task.

[0042] The feature vector of the text description information and the multimodal semantic feature vector of the initial image are concatenated to obtain a semantic fusion feature vector. This fusion feature vector combines the information of the text description and the image content, and can more comprehensively represent the semantic meaning of the image to be generated. For example, after the text description information is processed by a pre-trained large language model (such as LLM), a feature vector of the text description can be obtained. This feature vector captures the semantic information of the text description and converts the text content into a high-dimensional vector representation. After the initial image is processed by the BLIP2 encoder, a multimodal semantic feature vector is obtained. This vector combines the visual features and semantic representation of the image and represents the semantic information of the image. The feature vector of the text description information and the multimodal semantic feature vector of the initial image are concatenated to merge them into a larger vector. This concatenation operation can simply connect the two vectors together to form a joint representation containing the text description and image information. Through the concatenation process, the semantic fusion feature vector obtained contains the information of the text description and the image content, realizing the fusion of text and image information. This fusion feature vector combines the text description and image features, and can more comprehensively express the semantic meaning and characteristics of the image to be generated. By fusing the text description information and the multimodal semantic feature vector of the initial image, a more comprehensive feature representation can be obtained, which provides richer semantic information for generating images consistent with the text description. The generation of this semantically fused feature vector helps improve the performance and generation effect in text-to-image conversion tasks.

[0043] The image feature vector and the semantic fusion feature vector of the initial image are fused through the text-image adaptation module to obtain an image-text fusion feature vector. For example, the text-image adaptation module is a module for fusing image features and semantic information, aiming to adapt and fuse information from different modalities. This module can better understand the relationship between the text description and the image content, and generate an image that is more consistent with the description. The image feature vector and the semantic fusion feature vector of the initial image are fused through the text-image adaptation module. The image feature vector captures the visual information of the image, while the semantic fusion feature vector combines the information of the text description and the image content. This fusion operation can be achieved through weighted summation, splicing, or other methods to ensure that the image features and semantic information are properly fused. After processing by the text-image adaptation module, an image-text fusion feature vector is obtained. This fusion feature vector combines image features and semantic information, represents the comprehensive features of the image and text description, and provides more comprehensive information for subsequent image generation tasks.

[0044] The image-text fusion feature vector is denoised based on a preset noise map by a diffusion model to obtain a target image of the object to be generated. For example, a diffusion model is a generative model for image generation, which is usually used to generate high-quality and realistic images. This model gradually generates realistic images by iteratively diffusing on a noise image. The preset noise map is the initial image provided as input to the diffusion model. This noise map can be a randomly generated image or a preprocessed image, which is used to initialize the generation process of the diffusion model. The diffusion model performs denoising based on the preset noise map and the image-text fusion feature vector. The diffusion model uses this information to gradually generate more accurate and clearer images, remove noise and blur, and make the generated image more consistent with the characteristics of the target object. Through the iterative generation process of the diffusion model, combined with the preset noise map and the image-text fusion feature vector, the target image of the object to be generated is finally obtained. This target image should be more consistent with the requirements of the text description and have higher quality and realism. Through this process, the diffusion model uses the preset noise map and the image-text fusion feature vector to generate the target image, thereby achieving the text-to-image conversion task. This method combines semantic information and image features to generate high-quality images that are more consistent with the description.

[0045] In some embodiments of the present disclosure, a text-image adaptation module includes a plurality of adapters, and the text-image adaptation module is used to fuse the image feature vector and the semantic fusion feature vector of the initial image to obtain the image-text fusion feature vector, including: processing a preset question text, a preset time step, the image feature vector of the initial image, and the semantic fusion feature vector through the adapter to obtain the processing result of the adapter; using the processing result of the previous adapter and the preset time step, the image feature vector of the initial image, and the semantic fusion feature vector as the input of the next adapter, and processing the processing result of the previous adapter and the preset time step, the image feature vector of the initial image, and the semantic fusion feature vector through the next adapter, until the last adapter outputs the image-text fusion feature vector.

[0046] Reference below Figure 4 The following is a framework diagram of an adapter to describe the process of the adapter processing data, as follows:

[0047] The text image adaptation module contains multiple adapters, each of which is responsible for performing specific processing on the input data to adapt to the needs of the model. These adapters can be set according to different tasks to process different types of information. The adapter processes the preset question text, preset time step, image feature vector of the initial image, and semantic fusion feature vector to obtain the processing result of the adapter. The processing result of the previous adapter, the preset time step, image feature vector of the initial image, and semantic fusion feature vector are used as the input of the next adapter. The processing result of the previous adapter and other inputs are processed by the next adapter. This processing flow is repeated until the last adapter outputs the image text fusion feature vector. The processing process of each adapter can include different operations, such as feature transformation, feature fusion, feature mapping, etc. The design purpose of each adapter is to gradually optimize the input features so that the final image text fusion feature vector is more accurate and complete. Through the iterative processing of the adapter, the input information can be gradually refined and optimized so that the final image text fusion feature vector can better express the relationship between the image and the text, thereby improving the quality and accuracy of the generated image. Through this step-by-step processing approach, the text-image adaptation module can more effectively fuse image features and semantic information, generate an image-text fusion feature vector that is more consistent with the text description, and provide better input features for subsequent image generation tasks.

[0048] Each of the above adapters includes a linear layer, a multi-layer perceptron, a self-attention layer, an adaptive normalization layer, a cross-attention layer, and a feedforward network. The adapter processes the preset question text, the preset time step, the image feature vector of the initial image, and the semantic fusion feature vector, and the processing result of the adapter includes: processing the preset question text through the linear layer to obtain the embedded features of the preset question text, processing the semantic fusion feature vector through the linear layer to obtain the semantic fusion enhancement vector; processing the preset time step through the multi-layer perceptron to obtain the time feature vector, and processing the image feature vector of the initial image through the multi-layer perceptron to obtain the first image feature; fusing the embedded features of the preset question text and the time feature vector to obtain the first fused feature; fusing the time feature vector and the first image feature The first fusion feature is processed by the self-attention layer, and the processing result of the self-attention layer and the second fusion feature are processed by the adaptive normalization layer to obtain the processing result of the adaptive normalization layer; the processing result of the adaptive normalization layer and the semantic fusion enhancement vector are processed by the cross-attention layer to obtain the processing result of the cross-attention layer; the processing result of the cross-attention layer and the first fusion feature are fused through the residual connection to obtain the third fusion feature; the second fusion feature and the third fusion feature are processed through the adaptive normalization layer to obtain the fourth fusion feature; the fourth fusion feature is processed through the feedforward network, and the processing result of the feedforward network and the processing result of the cross-attention layer are fused through the residual connection to obtain the processing result of the adapter.

[0049] The preset question text is a text description of the image to be generated. The preset question text is processed by a linear layer (i.e., LayerNormalization, referred to as LN). This linear layer learns a set of weights to map the input text features to a new feature space to obtain the embedded features of the preset question text. This embedded feature can be regarded as an abstract representation of the input text and contains more advanced semantic information. The semantic fusion feature vector is a comprehensive representation of image features and semantic information obtained in the previous step. The semantic fusion feature vector is processed by a linear layer, which also maps the input features to another space by learning weights to obtain a semantic fusion enhancement vector. This vector can be regarded as a reinforcement or enhancement of the semantic information and may contain more information about the image content and description. Through these two processing steps, the linear layer can linearly transform the input text and semantic information to obtain a feature representation that is more representative and more suitable for model processing. These processed features will be used in subsequent processing steps to help the model better understand and combine text descriptions and image features to generate images that meet the description.

[0050] The preset time step is a vector or sequence describing time information, which is used to guide the process of image generation. The preset time step is processed by a multilayer perceptron (MLP). The multilayer perceptron consists of multiple fully connected hidden layers, each of which contains multiple neurons. The input data is mapped to a higher-dimensional feature space through a nonlinear activation function to obtain a time feature vector, which contains the representation of the time information after being processed by the multilayer perceptron. The image feature vector of the initial image is a vector representation describing the image content. The image feature vector of the initial image is also processed by the multilayer perceptron. This process is similar to processing the time step. Through the nonlinear transformation of the multilayer perceptron, more representative image features can be extracted to obtain the first image feature. This feature vector contains the image feature representation after being processed by the multilayer perceptron. Through the processing of the multilayer perceptron, the model can extract more representative feature representations from the time step and image features. These features will help the subsequent steps to better combine the time information and image content, thereby generating a more satisfactory image.

[0051] The embedded features and time feature vectors of the preset question text contain the semantic information and time information of the text description respectively. The two features are fused in some way (such as splicing, weighted summation, etc.). A common method is to splice the two feature vectors together to form a larger feature vector to obtain the first fused feature. This feature vector contains the combination of the semantic information and time information of the preset question text, which can help the model better understand the context and time background behind the text description.

[0052] The time feature vector and the first image feature are fused to obtain a second fused feature. The time feature vector and the first image feature contain time information and image content information, respectively. The two features are also fused in some way (such as splicing, weighted summation, etc.). A similar method can be used to splice the two feature vectors together to form a more comprehensive feature vector to obtain a second fused feature. This feature vector contains a combination of time information and image content information, which can help the model better understand the impact of time background on image generation. Through these two feature fusion processes, the model can effectively integrate information from different sources, so that the model can more comprehensively consider text descriptions, time information and image content when generating images, thereby improving the accuracy and consistency of generated images. The first fused feature, this feature vector contains a combination of semantic information and time information of the preset question text.

[0053] The self-attention layer is a mechanism that can assign different weights to different parts of the input. Through the self-attention mechanism, the model can learn the importance of different positions in the input sequence, so as to better understand the relationship within the sequence. Here, the self-attention layer is used to weight the first fused feature to better capture the correlation information within the feature vector. The feature representation processed by the self-attention layer is obtained, which takes into account the importance weights between different parts.

[0054] The processing result of the self-attention layer and the second fusion feature are processed by the adaptive normalization layer (Adaptive Layer Normalization, referred to as AdaLN) to obtain the processing result of the adaptive normalization layer. The processing result of the self-attention layer and the second fusion feature respectively contain the weighted representation of the first fusion feature and the combination of time information and image content information. The adaptive normalization layer is a normalization method that can adaptively adjust the mean and variance according to the input data to better control the distribution of the data. Here, the adaptive normalization layer is used to normalize the processing result of the self-attention layer and the second fusion feature to ensure the stability of the data and better training effect. The processing result of the adaptive normalization layer is obtained. This result is a feature representation after normalization, which can help the model better process feature data and improve the performance of the model. Through the processing of the self-attention layer and the adaptive normalization layer, the model can better process the feature representation and control the distribution of the data, thereby improving the learning ability of the model and the quality of the generated results.

[0055] The processing result of the adaptive normalization layer and the semantic fusion enhancement vector. The processing result of the adaptive normalization layer contains the feature representation after normalization, and the semantic fusion enhancement vector is a further fusion enhancement of the initial image features and text description features.

[0056] The cross attention layer is a mechanism that can simultaneously consider the cross relationship between two input sequences. Through the cross attention mechanism, the mutual influence between the two input sequences can be learned to better capture the correlation information between them. Here, the cross attention layer is used to process the processing results of the adaptive normalization layer and the semantic fusion enhancement vector to obtain a richer feature representation. The feature representation after the cross attention layer is obtained, which takes into account the cross relationship between the processing results of the adaptive normalization layer and the semantic fusion enhancement vector. The processing result of the cross attention layer and the first fusion feature are fused through a residual connection to obtain a third fusion feature. The processing result of the cross attention layer and the first fusion feature respectively contain the feature representation after the cross attention layer and the weighted representation of the first fusion feature. Residual connection is a method for solving the problem of gradient vanishing or gradient exploding in deep neural network training. Through the residual connection, the adapter can directly learn the residual (i.e., the difference between the input and the output), so that it is easier to learn the representation of the residual part. Here, the processing result of the cross attention layer and the first fusion feature are fused through a residual connection to retain the information of the first fusion feature while adding the processing result of the cross attention layer. The third fused feature is obtained. This feature vector is the result of residual connection fusion processing. It comprehensively considers the processing results of the cross attention layer and the first fused feature, and has a richer feature representation. Through the processing of cross attention layer and residual connection, the adapter can better integrate information from different sources, improve the richness of feature representation and the expressiveness of the model, thereby improving the accuracy and consistency of the generated image.

[0057] The second fused feature and the third fused feature respectively contain the combination of time information and image content information, as well as the result of the processing of the cross attention layer and the fusion result of the first fused feature. The adaptive normalization layer is used to normalize the two features to ensure data stability and better training effect. The purpose of this step is to adjust the distribution of feature representations to better meet the learning needs of the model. The fourth fused feature is obtained, and the feature vector is the result of processing by the adaptive normalization layer, which contains the normalized feature representation. The fourth fused feature is processed by the feedforward network (FeedForward), and the processing results of the feedforward network and the processing results of the cross attention layer are fused through residual connections to obtain the processing results of the adapter. The feedforward network is a common deep learning structure that learns complex representations of input data through multiple layers of nonlinear transformations. Here, the feedforward network is used to process the fourth fused feature to obtain a more advanced feature representation. The processing results of the feedforward network and the processing results of the cross attention layer are fused through residual connections. Residual connections can help the adapter better learn the representation of the residual part, thereby learning feature representation more effectively. The processing result of the adapter is obtained. This feature vector is the result of processing by the feedforward network and fusion with the processing result of the cross attention layer through residual connection, which has a higher-level feature representation. Through the processing of adaptive normalization layer, feedforward network and residual connection, the distribution of feature representation can be better adjusted, higher-level feature representation can be learned, and information from different sources can be integrated to improve the performance of the adapter and the quality of the generated results.

[0058] Figure 5 FIG. 1 is a flow chart of another image generation method provided by an embodiment of the present disclosure. Figure 5 As shown, before the image feature vector and the semantic fusion feature vector of the initial image are fused by the text image adaptation module, the method further includes steps S510 to S590.

[0059] In step S510, training data is obtained, wherein the training data includes a plurality of training samples, each of which includes text description information of a training object, an initial image of the training object, and a real generated image of the training object.

[0060] In step S520, the text description information of the training object is processed by the large language model to obtain a feature vector of the text description information of the training object.

[0061] In step S530, semantic features are extracted from the initial image of the training object by using the first encoder to obtain a multimodal semantic feature vector of the initial image of the training object.

[0062] In step S540, the image feature extraction is performed on the initial image of the training object by the second encoder to obtain an image feature vector of the initial image of the training object.

[0063] In step S550, the feature vector of the text description information of the training object and the multimodal semantic feature vector of the initial image of the training object are concatenated to obtain a trained semantic fusion feature vector.

[0064] In step S560, the image feature vector of the initial image of the training object and the training semantic fusion feature vector are fused by the initial text image adaptation module to obtain the training image text fusion feature vector.

[0065] In step S570, the training image-text fusion feature vector is denoised based on a preset noise map using a diffusion model to obtain a target image of the training object.

[0066] In step S580, the target loss is determined according to the target image of the training object, the real generated image of the training object, the feature vector of the text description information of the training object, the image feature vector of the initial image of the training object, and the image-text fusion feature vector of the training.

[0067] In step S590, the parameters in the initial text image adaptation module are updated based on the target loss.

[0068] This method can learn effective feature representation from training data based on the pre-trained large oracle model and diffusion model as well as the first encoder and the second encoder, and continuously adjust the parameters in the text image adaptation module according to the target loss to improve the quality and accuracy of the generated image.

[0069] In some embodiments of the present disclosure, when training the above-mentioned initial text image adaptation module, the parameters of the pre-trained large language model, the first encoder, the second encoder, and the diffusion model can be frozen. In this way, only the parameters of the initial text image adaptation module can be updated according to the target loss, thereby improving the training efficiency. Among them, the initial text image adaptation module includes multiple adapters, for example, it can be 5 adapters.

[0070] In some embodiments of the present disclosure, for the training process of the adapter, the data processing process of the pre-trained large language model, the first encoder, the second encoder, the adapter, and the diffusion model is consistent with the data processing process using the adapter. You can refer to the contents described in the above embodiments and will not repeat them here.

[0071] Based on the aforementioned embodiment, the training process of the adapter needs to calculate the loss to update the parameters of each adapter. For example, according to the target image of the training object, the real generated image of the training object, the feature vector of the text description information of the training object, the trained semantic fusion feature vector, and the trained image-text fusion feature vector, determining the target loss includes: calculating the image alignment loss based on the target image of the training object and the real generated image of the training object through the mean square error loss function; calculating the semantic alignment loss based on the target image of the training object, the feature vector of the text description information of the training object, the image feature vector of the initial image of the training object, and the trained image-text fusion feature vector through the cosine similarity loss function; summing the image alignment loss and the semantic alignment loss to obtain the target loss. Specifically, the image alignment loss (i.e., the mean square error loss) is calculated based on the target image of the training object and the real generated image of the training object through the mean square error loss function. The mean square error loss calculates the difference between the predicted image and the real image, and the mean square error loss is expressed as L rebuild , ensuring the accuracy and detail of the generated images.

[0072] Based on the aforementioned embodiment, the semantic alignment loss is calculated by using a cosine similarity loss function based on a target image of a training object, a feature vector of text description information of the training object, an image feature vector of an initial image of the training object, and a trained image-text fusion feature vector, including: calculating a first semantic alignment loss based on a target image of a training object and a feature vector of text description information of the training object through a cosine similarity loss function; calculating a second semantic alignment loss based on an image feature vector of an initial image of a training object and a trained image-text fusion feature vector through a cosine similarity loss function; calculating a third semantic alignment loss based on a feature vector of text description information of a training object and a trained image-text fusion feature vector through a cosine similarity loss function; calculating the semantic alignment loss based on the first semantic alignment loss and the weight of the first semantic alignment loss, the second semantic alignment loss and the weight of the second semantic alignment loss, and the third semantic alignment loss and the weight of the third semantic alignment loss. Specifically, the first semantic alignment loss is calculated by the following formula:

[0073] L gen-text =1-cosine_similarity(CLIP gen_image ,CLIP text_description )

[0074] Among them, L gen-text is the first semantic alignment loss, CLIP gen_image is the target image of the training object, CLIP text_description It is the feature vector of the text description information of the training object.

[0075] The second semantic alignment loss is calculated by the following formula:

[0076] L adapter-ref =1-cosine_similarity(CLIP adapter_output ,CLIP ref_image )

[0077] Among them, L adapter-ref is the second semantic alignment loss, CLIP ref_image is the image feature vector of the initial image of the training object, CLIP adapter_output Fusion feature vector for training image text.

[0078] The third semantic alignment loss is calculated by the following formula:

[0079] L adapter-text =1-cosine_similarity(CLIP adapter_output ,CLIP text_description )

[0080] Among them, L adapter-text is the third semantic alignment loss, CLIP text_description is the feature vector of the text description information of the training object, CLIP adapter_output Fusion feature vector for training image text.

[0081] The semantic alignment loss is calculated by the following formula:

[0082] L align =λ1L gen-text +λ2L adapter-ref +λ3L adapter-text

[0083] Among them, L align is the semantic alignment loss, L gen-text is the first semantic alignment loss, λ1 is the weight of the first semantic alignment loss, L adapter-ref is the second semantic alignment loss, λ2 is the weight of the second semantic loss, L adapter-text is the third semantic alignment loss, and λ3 is the weight of the third semantic loss.

[0084] The above target loss can be obtained by summing the image alignment loss and the semantic alignment loss, that is, target loss = L rebuild +L align Then, the parameters of each adapter are updated according to the target loss. The text-image adaptation module obtained by the above training method can combine the information in the text and the image so that the overall process can accurately identify the relevant information of multiple objects contained in the text and the image, and further make the generated image consistent with the text description.

[0085] Figure 6 It is a block diagram of an image generating device provided by an embodiment of the present disclosure.

[0086] like Figure 6 As shown, the image generating device 600 includes an acquisition module 610 , a text information processing module 620 , a semantic feature extraction module 630 , an image extraction module 640 , a splicing module 650 , a fusion module 660 and a denoising module 670 .

[0087] Specifically, the acquisition module 610 is used to acquire text description information of the object to be generated and an initial image of the object to be generated;

[0088] A text information processing module 620 is used to process the text description information through a large language model to obtain a feature vector of the text description information;

[0089] A semantic feature extraction module 630 is used to extract semantic features from the initial image through a first encoder to obtain a multimodal semantic feature vector of the initial image;

[0090] An image extraction module 640 is used to extract image features from the initial image through a second encoder to obtain an image feature vector of the initial image;

[0091] A splicing module 650 is used to splice the feature vector of the text description information and the multimodal semantic feature vector of the initial image to obtain a semantic fusion feature vector;

[0092] A fusion module 660 is used to fuse the image feature vector and the semantic fusion feature vector of the initial image through the text image adaptation module to obtain an image text fusion feature vector;

[0093] The denoising module 670 is used to perform denoising processing on the image-text fusion feature vector based on a preset noise map through a diffusion model to obtain a target image of the object to be generated.

[0094] The image generation device 600 can extract features from text description information and images using the first encoder and the second encoder, and can better process complex texts containing dense prompts such as multiple objects, detailed attribute descriptions, complex relationships, and long text alignment, thereby improving the ability to process complex text prompts. By splicing the feature vector of the text description information with the multimodal semantic feature vector of the initial image, and fusing the image feature vector and the semantic fusion feature vector through the text image adaptation module, a more accurate, high-quality image that is highly consistent with the text description can be generated, and the accuracy and consistency of image generation are enhanced. By improving the efficiency and quality of text-to-image conversion, it helps to improve the user experience, allowing users to more easily convert text descriptions into images that meet expectations. At the same time, the generated high-quality images are highly consistent with the text descriptions, which will expand the application areas, such as virtual reality, augmented reality, e-commerce, etc. De-noising the image-text fusion feature vector based on a preset noise map through a diffusion model can further improve the quality of the generated image, eliminate possible noise, and make the final generated image clearer and more realistic.

[0095] In some embodiments of the present disclosure, the text-image adaptation module includes multiple adapters, and the fusion module 660 is configured to: process a preset question text, a preset time step, an image feature vector of an initial image, and a semantic fusion feature vector through an adapter to obtain a processing result of the adapter; use the processing result of the previous adapter and the preset time step, the image feature vector of the initial image, and the semantic fusion feature vector as inputs of the next adapter, and process the processing result of the previous adapter and the preset time step, the image feature vector of the initial image, and the semantic fusion feature vector through the next adapter, until the last adapter outputs an image-text fusion feature vector.

[0096] In some embodiments of the present disclosure, the adapter includes a linear layer, a multi-layer perceptron, a self-attention layer, an adaptive normalization layer, a cross-attention layer, and a feedforward network. The adapter processes a preset question text, a preset time step, an image feature vector of an initial image, and a semantic fusion feature vector, and the processing result of the adapter includes: processing the preset question text through a linear layer to obtain an embedded feature of the preset question text, processing the semantic fusion feature vector through a linear layer to obtain a semantic fusion enhancement vector; processing the preset time step through a multi-layer perceptron to obtain a time feature vector, and processing the image feature vector of the initial image through a multi-layer perceptron to obtain a first image feature; fusing the embedded feature of the preset question text and the time feature vector to obtain a first fused feature; fusing the time feature vector and the first The image features are fused to obtain a second fused feature; the first fused feature is processed by a self-attention layer, and the processing result of the self-attention layer and the second fused feature are processed by an adaptive normalization layer to obtain the processing result of the adaptive normalization layer; the processing result of the adaptive normalization layer and the semantic fusion enhancement vector are processed by a cross-attention layer to obtain the processing result of the cross-attention layer; the processing result of the cross-attention layer and the first fused feature are fused through a residual connection to obtain a third fused feature; the second fused feature and the third fused feature are processed through an adaptive normalization layer to obtain a fourth fused feature; the fourth fused feature is processed through a feedforward network, and the processing result of the feedforward network and the processing result of the cross-attention layer are fused through a residual connection to obtain the processing result of the adapter.

[0097] In some embodiments of the present disclosure, before the image feature vector and the semantic fusion feature vector of the initial image are fused by the text image adaptation module, the image generation device 600 is further used to: obtain training data, the training data includes multiple training samples, each training sample includes text description information of the training object, the initial image of the training object, and the real generated image of the training object; process the text description information of the training object by a large language model to obtain the feature vector of the text description information of the training object; extract semantic features from the initial image of the training object by a first encoder to obtain a multimodal semantic feature vector of the initial image of the training object; extract image features from the initial image of the training object by a second encoder to obtain the image feature vector of the initial image of the training object; The feature vector of the text description information of the training object and the multimodal semantic feature vector of the initial image of the training object are spliced ​​to obtain the trained semantic fusion feature vector; the image feature vector of the initial image of the training object and the training semantic fusion feature vector are fused through the initial text image adaptation module to obtain the trained image-text fusion feature vector; the training image-text fusion feature vector is denoised based on a preset noise map through a diffusion model to obtain the target image of the training object; the target loss is determined according to the target image of the training object, the real generated image of the training object, the feature vector of the text description information of the training object, the image feature vector of the initial image of the training object, and the trained image-text fusion feature vector; the parameters in the initial text image adaptation module are updated based on the target loss.

[0098] In some embodiments of the present disclosure, determining the target loss based on the target image of the training object, the real generated image of the training object, the feature vector of the text description information of the training object, the trained semantic fusion feature vector, and the trained image-text fusion feature vector includes: calculating the image alignment loss based on the target image of the training object and the real generated image of the training object through a mean square error loss function; calculating the semantic alignment loss based on the target image of the training object, the feature vector of the text description information of the training object, the image feature vector of the initial image of the training object, and the trained image-text fusion feature vector through a cosine similarity loss function; and summing the image alignment loss and the semantic alignment loss to obtain the target loss.

[0099] In some embodiments of the present disclosure, calculating the semantic alignment loss through a cosine similarity loss function based on a target image of a training object, a feature vector of text description information of the training object, an image feature vector of an initial image of the training object, and a trained image-text fusion feature vector includes: calculating a first semantic alignment loss through a cosine similarity loss function based on the target image of the training object and the feature vector of the text description information of the training object; calculating a second semantic alignment loss through a cosine similarity loss function based on the image feature vector of the initial image of the training object and the trained image-text fusion feature vector; calculating a third semantic alignment loss through a cosine similarity loss function based on the feature vector of the text description information of the training object and the trained image-text fusion feature vector; calculating the semantic alignment loss based on the first semantic alignment loss and the weight of the first semantic alignment loss, the second semantic alignment loss and the weight of the second semantic alignment loss, and the third semantic alignment loss and the weight of the third semantic alignment loss.

[0100] In some embodiments of the present disclosure, the text description information of the object to be generated includes text information of multiple objects to be generated, and the initial image of the object to be generated is an initial image including multiple objects to be generated.

[0101] Figure 7 Schematic diagram of an electronic device 7 provided by an embodiment of the present disclosure. Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 701 executes the computer program 703, the functions of the modules in the above-mentioned device embodiments are implemented.

[0102] The electronic device 7 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 7 may include, but is not limited to, a processor 701 and a memory 702. Those skilled in the art will appreciate that Figure 7 The electronic device 7 is merely an example and does not limit the electronic device 7 , and may include more or less components than those shown in the figure, or different components.

[0103] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0104] The memory 702 may be an internal storage unit of the electronic device 7, for example, a hard disk or memory of the electronic device 7. The memory 702 may also be an external storage device of the electronic device 7, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. The memory 702 may also include both an internal storage unit of the electronic device 7 and an external storage device. The memory 702 is used to store computer programs and other programs and data required by the electronic device.

[0105] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0107] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. An image generation method, characterized in that: include: Obtaining text description information of the object to be generated and an initial image of the object to be generated; Processing the text description information through a large language model to obtain a feature vector of the text description information; Extracting semantic features from the initial image using a first encoder to obtain a multimodal semantic feature vector of the initial image; Extracting image features from the initial image using a second encoder to obtain an image feature vector of the initial image; Performing splicing processing on the feature vector of the text description information and the multimodal semantic feature vector of the initial image to obtain a semantic fusion feature vector; The image feature vector of the initial image and the semantic fusion feature vector are fused by a text image adaptation module to obtain an image-text fusion feature vector; The image-text fusion feature vector is denoised based on a preset noise map through a diffusion model to obtain a target image of the object to be generated.

2. The method according to claim 1, characterized in that The text image adaptation module includes a plurality of adapters, and the image feature vector of the initial image and the semantic fusion feature vector are fused by the text image adaptation module to obtain the image text fusion feature vector, which includes: Processing the preset question text, the preset time step, the image feature vector of the initial image, and the semantic fusion feature vector through the adapter to obtain a processing result of the adapter; The processing result of the previous adapter and the preset time step, the image feature vector of the initial image, and the semantic fusion feature vector are used as the input of the next adapter, and the processing result of the previous adapter and the preset time step, the image feature vector of the initial image, and the semantic fusion feature vector are processed by the next adapter until the last adapter outputs the image-text fusion feature vector.

3. The method according to claim 2, characterized in that The adapter includes a linear layer, a multi-layer perceptron, a self-attention layer, an adaptive normalization layer, a cross-attention layer, and a feedforward network. The adapter processes the preset question text, the preset time step, the image feature vector of the initial image, and the semantic fusion feature vector, and the processing result of the adapter includes: Processing the preset question text through the linear layer to obtain the embedded features of the preset question text, and processing the semantic fusion feature vector through the linear layer to obtain the semantic fusion enhancement vector; Processing a preset time step by the multilayer perceptron to obtain a time feature vector, and processing an image feature vector of the initial image by the multilayer perceptron to obtain a first image feature; Fusing the embedded features of the preset question text and the time feature vector to obtain a first fused feature; fusing the time feature vector and the first image feature to obtain a second fused feature; Processing the first fused feature through the self-attention layer, and processing the processing result of the self-attention layer and the second fused feature through the adaptive normalization layer to obtain the processing result of the adaptive normalization layer; Processing the processing result of the adaptive normalization layer and the semantic fusion enhancement vector through the cross attention layer to obtain the processing result of the cross attention layer; Fusing the processing result of the cross attention layer and the first fused feature through a residual connection to obtain a third fused feature; Processing the second fused feature and the third fused feature through the adaptive normalization layer to obtain a fourth fused feature; The fourth fusion feature is processed by the feedforward network, and the processing result of the feedforward network and the processing result of the cross attention layer are fused through a residual connection to obtain the processing result of the adapter.

4. The method according to claim 1, characterized in that Before the text image adaptation module performs fusion processing on the image feature vector of the initial image and the semantic fusion feature vector, the method further includes: Acquire training data, wherein the training data includes a plurality of training samples, each of which includes text description information of a training object, an initial image of the training object, and a real generated image of the training object; Processing the text description information of the training object by using the large language model to obtain a feature vector of the text description information of the training object; Extracting semantic features from the initial image of the training object by using the first encoder to obtain a multimodal semantic feature vector of the initial image of the training object; Extracting image features from the initial image of the training object by using the second encoder to obtain an image feature vector of the initial image of the training object; Performing splicing processing on the feature vector of the text description information of the training object and the multimodal semantic feature vector of the initial image of the training object to obtain a trained semantic fusion feature vector; The image feature vector of the initial image of the training object and the training semantic fusion feature vector are fused by an initial text image adaptation module to obtain a training image text fusion feature vector; Performing denoising processing on the trained image-text fusion feature vector based on the preset noise map by using the diffusion model to obtain a target image of the training object; Determine a target loss according to the target image of the training object, the real generated image of the training object, the feature vector of the text description information of the training object, the image feature vector of the initial image of the training object, and the image-text fusion feature vector of the training; Parameters in the initial text image adaptation module are updated based on the target loss.

5. The method according to claim 4, characterized in that Determining the target loss according to the target image of the training object, the real generated image of the training object, the feature vector of the text description information of the training object, the trained semantic fusion feature vector, and the trained image-text fusion feature vector comprises: Calculating an image alignment loss based on a target image of the training object and a true generated image of the training object by a mean square error loss function; Calculating semantic alignment loss based on the target image of the training object, the feature vector of the text description information of the training object, the image feature vector of the initial image of the training object, and the image-text fusion feature vector of the training by using a cosine similarity loss function; The image alignment loss and the semantic alignment loss are summed to obtain the target loss.

6. The method according to claim 5, characterized in that Calculating the semantic alignment loss based on the target image of the training object, the feature vector of the text description information of the training object, the image feature vector of the initial image of the training object, and the image-text fusion feature vector of the training by using the cosine similarity loss function includes: Calculating a first semantic alignment loss based on a target image of the training object and a feature vector of text description information of the training object by using the cosine similarity loss function; Calculating a second semantic alignment loss based on the image feature vector of the initial image of the training object and the image-text fusion feature vector of the training by using the cosine similarity loss function; Calculating a third semantic alignment loss based on the feature vector of the text description information of the training object and the image-text fusion feature vector of the training by using the cosine similarity loss function; The semantic alignment loss is calculated based on the first semantic alignment loss and the weight of the first semantic alignment loss, the second semantic alignment loss and the weight of the second semantic alignment loss, and the third semantic alignment loss and the weight of the third semantic alignment loss.

7. The method according to any one of claims 1 to 6, characterized in that: The text description information of the object to be generated includes text information of multiple objects to be generated, and the initial image of the object to be generated is an initial image including multiple objects to be generated.

8. An image generating device, characterized in that: include: An acquisition module, used to acquire text description information of the object to be generated and an initial image of the object to be generated; A text information processing module, used to process the text description information through a large language model to obtain a feature vector of the text description information; A semantic feature extraction module, configured to extract semantic features from the initial image through a first encoder to obtain a multimodal semantic feature vector of the initial image; An image extraction module, configured to extract image features from the initial image through a second encoder to obtain an image feature vector of the initial image; A splicing module, used for splicing the feature vector of the text description information and the multimodal semantic feature vector of the initial image to obtain a semantic fusion feature vector; A fusion module, used for fusing the image feature vector of the initial image and the semantic fusion feature vector through a text image adaptation module to obtain an image-text fusion feature vector; The denoising module is used to perform denoising processing on the image-text fusion feature vector based on a preset noise map through a diffusion model to obtain a target image of the object to be generated.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images