A training method of an image generation model and an image generation method

Abstract

This application relates to the field of image generation technology, and in particular to a training method for an image generation model and an image generation method, which is used to solve the problem that the position of image instances generated by layout control generation methods in related technologies is difficult to align with the semantics of text. The method is as follows: Based on the layout conditions of target instances in image samples in an image sample set, layout guiding noise of the target instance is generated. The image sample includes the original image, image description text, and layout conditions of multiple instances contained in the original image. The layout conditions of the instance include a reference image, instance description text, and position information. The target instance is any one of the multiple instances. The layout guiding noise of each instance in the image sample is used as a layout control constraint. The layout control noise fine-tuning model obtained by coupling the pre-trained basic generation model with the CondLoRA module is fine-tuned by CondLoRA to obtain the target generation model, thereby enhancing the spatial layout accuracy and visual consistency of the instance.

Description

Technical Field

[0001] This application relates to the field of image generation technology, and in particular to a training method for an image generation model and an image generation method. Background Technology

[0002] Image generation has become a significant research area in artificial intelligence in recent years. Diffusion models are widely used in image generation tasks. With further research, a multimodal generation model combining the Transformer architecture and the diffusion model has gained widespread application due to its superior performance in terms of generation quality and sampling speed, achieving significant results in single-instance image generation. However, this model typically samples completely random noise during the generation inference stage and then gradually denoises the noise to obtain the image required by the user. This inherent method of sampling completely random noise makes the generated image instance positions lack effective spatial controllability in multi-instance scenarios, making it difficult to accurately generate images that meet specified layout requirements.

[0003] Subsequently, layout-to-image generation tasks introduce explicit spatial constraints (such as bounding box coordinates) and instance-level conditions (such as local text descriptions) to achieve positional and semantic control of multiple instances in an image, meeting the generation requirements of complex scenes. However, in related technologies, layout-to-image generation (L2I) methods based on this task still suffer from the problem of difficulty in aligning instance positions with text semantics. Summary of the Invention

[0004] This application provides a training method for an image generation model and an image generation method to significantly improve the spatial layout accuracy and instance visual consistency of generated images.

[0005] The specific technical solutions provided in this application are as follows: In a first aspect, embodiments of this application provide a method for training an image generation model, comprising: Obtain an image sample set, wherein a single image sample in the image sample set includes an original image, image description text, and layout conditions of multiple instances contained within the original image, and the layout conditions of a single instance include a reference image, instance description text, and location information; By coupling the conditional low-rank adaptation CondLoRA module into the pre-trained basic generative model, a layout control noise fine-tuning model is obtained. Based on the layout conditions of the target instances included in the image samples in the image sample set, layout guiding noise for the target instances is generated, wherein the target instance is any one of the multiple instances included in the image samples; The layout-guided noise of each instance in the image sample is used as a layout control constraint. The layout control noise fine-tuning model is trained using CondLoRA until the preset convergence condition is met, thus obtaining the target generation model.

[0006] In one possible implementation, acquiring the image sample set includes: The existing image dataset is integrated and expanded to include multiple instance categories. For each of the expanded instance categories, N descriptive phrases are generated. The descriptive phrases of at least two instance categories are randomly combined to obtain multiple image description texts, where N is a positive integer greater than 1. Based on any one of the multiple image description texts, generate a composite image that matches any one of the image description texts, and based on the composite image, determine the layout conditions of each of the multiple instances within the composite image; The synthesized image is taken as the original image, along with the image description text and the layout conditions of each of the multiple instances within the synthesized image, and is taken as one of the image samples in the image sample set.

[0007] In one possible implementation, determining the layout conditions of each of the multiple instances within the synthesized image based on the synthesized image includes: Target detection is performed on the synthesized image to obtain the location information of each instance within the synthesized image; For any instance of the synthesized image, perform the following operation: Segment the sub-image corresponding to the location information of any instance from the synthesized image; Image editing is performed on the sub-image to generate a reference image for any instance, and instance description text for any instance is generated based on the reference image for any instance; The location information, reference image, and instance description text of any instance are used as the layout conditions for any instance.

[0008] In one possible implementation, before using the synthesized image as the original image, the method further includes: It is determined that the reference image of any of the instances contains a single instance; The similarity between the instance description text of any instance and the reference image of any instance is determined to be greater than a first threshold; and the similarity between the reference image of any instance and the sub-image is determined to be greater than a second threshold.

[0009] In one possible implementation, the step of image editing the sub-image to generate a reference image for any instance includes: Based on candidate prompts from multiple preset dimensions, editing instructions are generated. These preset dimensions include any combination of background, pose, or viewpoint dimensions. Based on the editing instructions, the sub-image is edited to generate a reference image for any of the instances.

[0010] In one possible implementation, generating layout-guided noise for the target instance based on the layout conditions of the target instance included in the image sample set includes: The instance visual features corresponding to the reference image of the target instance are fused with the spatial features corresponding to the location information of the target instance to obtain the first fused feature; The instance text features corresponding to the instance description text of the target instance are fused with the spatial features to obtain a second fused feature; A first layout guiding noise for the target instance is generated based on the first fusion feature, and a second layout guiding noise for the target instance is generated based on the second fusion feature.

[0011] In one possible implementation, the step of using the layout-guided noise of each instance in the image sample as a layout control constraint, and performing CondLoRA fine-tuning training on the layout control noise fine-tuning model until a preset convergence condition is reached to obtain the target generation model includes: The first layout guidance noise is superimposed with the instance visual features of the target instance to obtain a first superimposed feature, and the second layout guidance noise is superimposed with the instance text features of the target instance to obtain a second superimposed feature. The first and second superimposed features of each instance in the image sample are used as layout control constraints. Combined with a preset time step, the predicted noise output by the basic generation model is obtained. Based on the predicted noise, the noisy image corresponding to the image sample is subjected to iterative denoising operation. The noisy image is obtained by adding noise to the original image in the image sample. Based on the predicted noise and the real noise in the noisy image, the parameters in the CondLoRA module are optimized until the convergence condition is met, thus obtaining the target generation model.

[0012] In one possible implementation, the step of using the first and second superimposed features of each instance in the image sample as layout control constraints, combined with a preset time step, to obtain the prediction noise output by the basic generative model includes: Masking is performed on the first and second superimposed features between different instances, and, for the first and second superimposed features of each instance, the global text features corresponding to the image description text in the image sample are masked. Self-attention is calculated based on the first and second superimposed features of each instance after masking, as well as the global text features and the global text features after masking, and the predicted noise is obtained by combining the time step.

[0013] In one possible implementation, the CondLoRA module includes a first low-rank trainable matrix, a second low-rank trainable matrix, and a third low-rank trainable matrix; target features correspond to target fusion features. If the target feature is the instance visual feature, then the target fusion feature is the first fusion feature; if the target feature is the instance text feature, then the target fusion feature is the second fusion feature. The target fusion feature is obtained by performing the following operation: The target features are mapped through the first low-rank trainable matrix to obtain the first intermediate features; The spatial features are mapped through the third low-rank trainable matrix to obtain the second intermediate features; The first intermediate feature and the second intermediate feature are fused to obtain the third intermediate feature; The third intermediate feature is reprojected onto the second low-rank trainable matrix to obtain the target fusion feature.

[0014] Secondly, embodiments of this application provide an image generation method, including: Obtain the description information of the target image to be generated. The description information includes the image description information of the target image and the instance description information of multiple instances that need to be included in the target image. The instance description information of a single instance includes the instance description text and location information of the instance. The description information is input into a trained target generation model, and the image output by the target generation model is used as the target image, wherein the target generation model is trained according to any one of the methods described in the first aspect above.

[0015] Thirdly, embodiments of this application provide a training apparatus for an image generation model, comprising: An acquisition unit is used to acquire an image sample set, wherein a single image sample in the image sample set includes an original image, image description text, and layout conditions of multiple instances contained in the original image, and the layout conditions of a single instance include a reference image, instance description text, and location information. The model building unit is used to couple the conditional low-rank adaptation CondLoRA module into the pre-trained base generative model to obtain a layout-controlled noise fine-tuning model. The noise generation unit is used to generate layout guiding noise for the target instance based on the layout conditions of the target instance included in the image sample set, wherein the target instance is any one of a plurality of instances included in the image sample. The fine-tuning unit is used to use the layout-guided noise of each instance in the image sample as a layout control constraint, and to perform CondLoRA fine-tuning training on the layout control noise fine-tuning model until the preset convergence condition is reached to obtain the target generation model.

[0016] In one possible implementation, the acquisition unit is specifically used for: The existing image dataset is integrated and expanded to include multiple instance categories. For each of the expanded instance categories, N descriptive phrases are generated. The descriptive phrases of at least two instance categories are randomly combined to obtain multiple image description texts, where N is a positive integer greater than 1. Based on any one of the multiple image description texts, generate a composite image that matches any one of the image description texts, and based on the composite image, determine the layout conditions of each of the multiple instances within the composite image; The synthesized image is taken as the original image, along with the image description text and the layout conditions of each of the multiple instances within the synthesized image, and is taken as one of the image samples in the image sample set.

[0017] In one possible implementation, the acquisition unit is specifically used for: Target detection is performed on the synthesized image to obtain the location information of each instance within the synthesized image; For any instance of the synthesized image, perform the following operation: Segment the sub-image corresponding to the location information of any instance from the synthesized image; Image editing is performed on the sub-image to generate a reference image for any instance, and instance description text for any instance is generated based on the reference image for any instance; The location information, reference image, and instance description text of any instance are used as the layout conditions for any instance.

[0018] In one possible implementation, before using the synthesized image as the original image, the acquisition unit is further configured to: It is determined that the reference image of any of the instances contains a single instance; The similarity between the instance description text of any instance and the reference image of any instance is determined to be greater than a first threshold; and the similarity between the reference image of any instance and the sub-image is determined to be greater than a second threshold.

[0019] In one possible implementation, the acquisition unit is specifically used for: Based on candidate prompts from multiple preset dimensions, editing instructions are generated. These preset dimensions include any combination of background, pose, or viewpoint dimensions. Based on the editing instructions, the sub-image is edited to generate a reference image for any of the instances.

[0020] In one possible implementation, the noise generation unit is specifically used for: The instance visual features corresponding to the reference image of the target instance are fused with the spatial features corresponding to the location information of the target instance to obtain the first fused feature; The instance text features corresponding to the instance description text of the target instance are fused with the spatial features to obtain a second fused feature; A first layout guiding noise for the target instance is generated based on the first fusion feature, and a second layout guiding noise for the target instance is generated based on the second fusion feature.

[0021] In one possible implementation, the fine-tuning unit is specifically used for: The first layout guidance noise is superimposed with the instance visual features of the target instance to obtain a first superimposed feature, and the second layout guidance noise is superimposed with the instance text features of the target instance to obtain a second superimposed feature. The first and second superimposed features of each instance in the image sample are used as layout control constraints. Combined with a preset time step, the predicted noise output by the basic generation model is obtained. Based on the predicted noise, the noisy image corresponding to the image sample is subjected to iterative denoising operation. The noisy image is obtained by adding noise to the original image in the image sample. Based on the predicted noise and the real noise in the noisy image, the parameters in the CondLoRA module are optimized until the convergence condition is met, thus obtaining the target generation model.

[0022] In one possible implementation, the fine-tuning unit is specifically used for: Masking is performed on the first and second superimposed features between different instances, and, for the first and second superimposed features of each instance, the global text features corresponding to the image description text in the image sample are masked. Self-attention is calculated based on the first and second superimposed features of each instance after masking, as well as the global text features and the global text features after masking, and the predicted noise is obtained by combining the time step.

[0023] In one possible implementation, the CondLoRA module includes a first low-rank trainable matrix, a second low-rank trainable matrix, and a third low-rank trainable matrix; target features correspond to target fusion features. If the target feature is the instance visual feature, then the target fusion feature is the first fusion feature; if the target feature is the instance text feature, then the target fusion feature is the second fusion feature. The target fusion feature is obtained by performing the following operation: The target features are mapped through the first low-rank trainable matrix to obtain the first intermediate features; The spatial features are mapped through the third low-rank trainable matrix to obtain the second intermediate features; The first intermediate feature and the second intermediate feature are fused to obtain the third intermediate feature; The third intermediate feature is reprojected onto the second low-rank trainable matrix to obtain the target fusion feature.

[0024] Fourthly, embodiments of this application provide an image generation model, including: The acquisition unit is used to acquire the description information of the target image to be generated. The description information includes the image description information of the target image and the instance description information of multiple instances that need to be included in the target image. The instance description information of a single instance includes the instance description text and location information of the instance. An image generation unit is configured to input the description information into a trained target generation model and use the image output by the target generation model as the target image, wherein the target generation model is trained according to the method described in any one of the first aspects above.

[0025] Fifthly, embodiments of this application provide an electronic device, including: Memory is used to store computer programs or instructions; A processor for executing a computer program or instructions in the memory, such that the method as described in any one of the first or second aspects above is performed.

[0026] Sixthly, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform the method as described in any one of the first or second aspects above.

[0027] In a seventh aspect, embodiments of this application provide a computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform the method as described in any one of the first or second aspects above.

[0028] In this embodiment, an image sample set is obtained, wherein a single image sample in the image sample set includes an original image, image description text, and layout conditions of multiple instances contained in the original image. The layout conditions of a single instance include a reference image of the instance, instance description text, and location information. A CondLoRA module is coupled into a pre-trained basic generative model to obtain a layout control noise fine-tuning model. Based on the layout conditions of the target instances included in the image samples in the image sample set, layout guidance noise for the target instances is generated, wherein the target instance is any one of the multiple instances contained in the image sample. Then, the layout guidance noise of each instance in the image sample is used as a layout control constraint to perform CondLoRA fine-tuning training on the above layout control noise fine-tuning model until a preset convergence condition is reached to obtain the target generative model.

[0029] In this way, by introducing a reference image, the model can perform reverse denoising learning based on the appearance features of instances within the given reference image during training, thereby improving the visual consistency of the generated instances. At the same time, layout guiding noise is generated based on the layout conditions of the instances, and this layout guiding noise is used as a layout control constraint to guide the model training, effectively enhancing the spatial layout accuracy of the model. Furthermore, the CondLoRA fine-tuning training method is adopted, which effectively reduces additional trainable parameters and lowers computational overhead by processing visual, text, and spatial modal features in a low-rank manner. Attached Figure Description

[0030] Figure 1 This is a schematic diagram illustrating a possible application scenario of an image generation method according to an embodiment of this application; Figure 2 This is a flowchart illustrating a training method for an image generation model according to an embodiment of this application. Figure 3 This is a schematic diagram of an image sample set construction process in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the layout condition determination process for one example of an embodiment of this application; Figure 5 This is a schematic diagram of a data filtering process in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the construction process of an image sample set in an embodiment of this application; Figure 7This is a schematic diagram of the architecture of a pre-trained basic generative model in an embodiment of this application; Figure 8 This is a schematic diagram of the architecture of a layout control noise fine-tuning model in an embodiment of this application; Figure 9 This is a schematic diagram of the specific process of training an image generation model in an embodiment of this application. Figure 1 ; Figure 10 This is a schematic diagram of the specific process of training an image generation model in an embodiment of this application. Figure 2 ; Figure 11 This is a schematic diagram of the specific process of training an image generation model in an embodiment of this application. Figure 3 ; Figure 12 This is a schematic diagram illustrating the constraint of feature interaction range using an attention masking mechanism in an embodiment of this application. Figure 13 This is a flowchart illustrating an image generation method according to an embodiment of this application; Figure 14 This is a schematic diagram of the image generation task and the generated target image in an embodiment of this application; Figure 15 This is a schematic diagram of the logical architecture of a training device for an image generation model according to an embodiment of this application; Figure 16 This is a schematic diagram of the logical architecture of an image generation device according to an embodiment of this application; Figure 17 This is a schematic diagram of the physical architecture of an electronic device according to an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0032] The acquisition, storage, and application of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0034] (1) In the embodiments of this application, the terms “first,” “second,” “third,” etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than those illustrated or described herein.

[0035] (2) In the embodiments of this application, the term "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0036] (3) In the embodiments of this application, the term "instance" usually refers to a specific, independent single object in the picture (image), rather than an abstract category. For example, when the prompt asks to generate "three cats", each cat with a different shape, position and posture is an independent "instance".

[0037] (4) In the embodiments of this application, the term “LoRA” refers to Low-Rank Adaptation, which is a parameter efficient fine-tuning (PEFT) method. Specifically, under the premise of freezing the weights of the pre-trained model, the lightweight adaptation of the pre-trained model is achieved through low-rank matrix factorization.

[0038] CondLoRA is an abbreviation for Conditionally Parameterized LoRA, which teaches a model to generate images under specific structural constraints by using "conditions" as part of the training input.

[0039] In this embodiment, CondLoRA is used for efficient fine-tuning, while controllable noise is intentionally introduced to improve generalization, stability or achieve specific regularization effects, thereby meeting the needs of multi-instance image generation.

[0040] (5) The term "Text-to-Image Generation (T2I)" in the embodiments of this application refers to a generative artificial intelligence task that generates corresponding images based on natural language text descriptions. In related technologies, text-to-image models are used to generate matching images based on input natural language text descriptions to accomplish the above task.

[0041] (6) The term “Diffusion Model” in the embodiments of this application is a generative model that generates samples from a learned data distribution by progressively adding noise to the data and learning the reverse denoising process. It is widely used in image generation tasks.

[0042] (7) The term “Multimodal Diffusion Transformer (MM-DiT)” in the embodiments of this application is a multimodal generation framework that combines the Transformer architecture with the diffusion model. During the denoising process of the diffusion model, image and text feature vectors are introduced simultaneously for self-attention calculation, thereby realizing semantic interaction between text and image (visual) features.

[0043] (8) In the embodiments of this application, the term "multi-instance layout control" refers to a layout control method that controls the spatial arrangement and interactive combination of multiple instances in the generated image by inputting layout conditions of several instances into the model in the image generation task. Each layout condition includes a spatial location represented by bounding box coordinates (referred to as location information in subsequent embodiments) and local prompt words (referred to as instance description text in subsequent embodiments).

[0044] (9) The term “Fourier Embeddings” in the embodiments of this application is an encoding method that maps input signals (such as coordinates or time steps) to a high-dimensional space through trigonometric function bases of different frequencies, which is used to enhance the model’s ability to represent high-frequency components.

[0045] (10) The term “Multi-Layer Perceptron (MLP)” in the embodiments of this application is a feedforward neural network structure consisting of an input layer, multiple hidden layers and an output layer. It enhances the representation ability of the model through a non-linear activation function and is a commonly used basic network structure in deep learning models.

[0046] (11) The term “Rotary Position Embedding (RoPE)” in the embodiments of this application is a position representation method in neural networks. It introduces relative position information by multiplying the feature vector by a rotation matrix in the self-attention mechanism, thereby enhancing the model’s ability to model the relative relationship of sequences or spaces.

[0047] (12) In the embodiments of this application, the term "positive-incentive noise" (pi-noise) "-noise" refers to "beneficial noise" that has greater than zero mutual information with the target task and can reduce task entropy (i.e., reduce uncertainty). Task entropy measures the complexity and uncertainty of a task from an information theory perspective. Noise that can reduce task uncertainty during task modeling or learning has a positive effect on the task and is defined as positive excitation noise, satisfying the following expression:

[0048] in, Represents task entropy. This refers to the target task, such as the image generation task in the embodiments of this application. Represents mutual information, This represents positively excited noise.

[0049] The design concept of the embodiments of this application will be briefly introduced below.

[0050] This application relates to the field of image generation technology, and mainly to a training method for an image generation model and an image generation method.

[0051] In related technologies, layout control generation methods typically define instances using text descriptions combined with spatial coordinates, leading to ambiguity in the generated results. This can result in difficulty aligning instance positions in the generated image with the text semantics, and even issues such as missing entities, attribute swapping, and spatial layout errors. Therefore, this application provides a training method for an image generation model and an image generation method to improve the spatial layout accuracy and instance visual consistency of the generated images.

[0052] This application provides a training method for an image generation model. The method involves acquiring an image sample set, where each individual image sample in the set includes an original image, image description text, and layout conditions for multiple instances contained within the original image. The layout conditions for a single instance include a reference image, instance description text, and location information. A CondLoRA module is coupled into a pre-trained basic generation model to obtain a layout control noise fine-tuning model. Based on the layout conditions of the target instances included in the image samples in the set, layout guidance noise for the target instances is generated, where the target instance is any one of the multiple instances contained in the image sample. Then, the layout guidance noise for each instance in the image sample is used as a layout control constraint to fine-tune the layout control noise fine-tuning model using CondLoRA until a preset convergence condition is met, resulting in a target generation model.

[0053] In this way, by introducing a reference image, the model can perform reverse denoising learning based on the appearance features of instances within the given reference image during training, thereby improving the visual consistency of the generated instances. At the same time, layout guiding noise is generated based on the layout conditions of the instances, and this layout guiding noise is used as a layout control constraint to guide the model training, effectively enhancing the spatial layout accuracy of the model. Furthermore, the CondLoRA fine-tuning training method is adopted, which effectively reduces additional trainable parameters and lowers computational overhead by processing visual, text, and spatial modal features in a low-rank manner.

[0054] In this embodiment, the target generation model can be obtained through the above training method. This model can be deployed in application scenarios where multiple instance images are generated in a controllable manner, implementing the image generation method provided in this embodiment to achieve controllable generation of multiple instance images. For example, in fields such as creative design, virtual scene construction, and game development, users can specify the position and layout of different objects in the image to be generated according to their own needs, thereby generating a high-quality image that meets specific spatial requirements, which will be referred to as the target image.

[0055] The application scenarios described in this application embodiment can be implemented using AI Flow. AI Flow is a key technology for the cross-integration of artificial intelligence (AI), communication (transmission), and networking (network). Through a layered network architecture and based on connectivity and interaction, it enables the transmission and emergence of intelligence. AI Flow proposes a generative intelligent transmission mechanism for environments with extremely narrow bandwidth and limited latency, breaking through the traditional communication paradigm that aims for pixel-level accurate reconstruction and shifting to a new transmission mode that "trades bandwidth for computing power." This mechanism is essentially a task-oriented communication paradigm: the sending end no longer transmits complete data, but rather transmits a small amount of key information according to task requirements; the receiving end relies on the generative model and prior knowledge to reconstruct, complete, and enhance the content, or directly support downstream task understanding and decision-making. Compared with traditional encoding / decoding and transmission methods, generative intelligent transmission avoids the transmission of redundant pixel information, maintains good reconstruction quality even under extremely low bitrate conditions, and effectively guarantees task performance.

[0056] See Figure 1 As shown, this application embodiment provides a possible application scenario for the image generation method, which includes a sending end 10 and a receiving end 20, wherein... The sending end 10 is used to acquire the image to be transmitted or the image generation requirement input by the user, and to determine the description information of the target image to be generated based on the image to be transmitted or the image generation requirement; wherein, the description information includes the image description information of the target image, and the instance description information of multiple instances that need to be contained in the target image; the instance description information of a single instance includes the instance description text and location information of the instance; and is also used to encode the above description information and send it to the receiving end 20. The receiver 20 is equipped with the target generation model described above. The receiver 20 is used to obtain the encoded description information sent by the sender 10; input the encoded description information into the target generation model; use the image output by the target generation model as the target image; and display the target image.

[0057] In some possible implementations, the description information determined by the sending end 10 may also include a specified image of the above-mentioned multiple instances or the encoding features corresponding to the specified image. In this case, the appearance of the multiple instances in the target image generated by the receiving end 20 is consistent with that of the instances in the specified image.

[0058] Thus, in the above application scenarios, the sending end 10 does not need to transmit complete image pixel data, but only semantic information, layout information, and optional instance reference information of the specified image, thereby reducing the amount of data transmitted. In some embodiments, the specified image can be a low-resolution image. The above description information is in text form; the image generation request input by the user can be in text form, voice form, or any other form, and this application does not impose any specific limitations.

[0059] It should be noted that the methods provided in the embodiments of this application are not limited to those described above. Figure 1 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of this application are not limited thereto.

[0060] After introducing one possible application scenario in the embodiments of this application, the preferred embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0061] First, let's explain the training content of the target generation model.

[0062] See Figure 2 As shown in the embodiments of this application, a training method for an image generation model is provided. The specific process of this method includes, but is not limited to, the following steps: Step 200: Obtain an image sample set, wherein a single image sample in the image sample set includes the original image, image description text, and layout conditions of multiple instances contained within the original image. The layout conditions of a single instance include a reference image, instance description text, and location information.

[0063] In this embodiment of the application, the image sample set is constructed based on an existing image dataset. Therefore, when performing step 200, refer to... Figure 3 As shown, the specific process is as follows: Step 2001: Integrate and expand the instance categories in the existing image dataset. For the multiple instance categories obtained after category expansion, generate N descriptive phrases for each category, where N is a positive integer greater than 1.

[0064] In this embodiment of the application, when performing step 2001, the core categories contained in the existing image dataset are first integrated and expanded to enrich the instance categories and scenes of the image sample set. For example, taking the Object365 dataset as the base dataset, in this embodiment of the application, the core categories in the Object365 dataset are integrated, and high-frequency daily objects are expanded through network collection and large models, thereby forming a complete system of 11 super categories and 645 fine-grained categories, covering multiple daily scenes such as human accessories, home furnishings, transportation, and food, so as to better generate high-quality images that meet the needs of users.

[0065] Then, based on the expanded instance categories, a large model is used to generate N differentiated descriptive phrases for each instance category.

[0066] In one possible implementation, the aforementioned large model can be a Generative Pre-trained Transformer (GPT). GPT is an advanced artificial intelligence language model that understands and generates natural language text through deep learning techniques, particularly the Transformer architecture. However, it should be noted that this application does not limit the specific type of the aforementioned large model; any existing or soon-to-emerge model or method with language generation capabilities is acceptable.

[0067] Step 2002: Randomly combine the descriptive phrases of at least two instance categories to obtain multiple image description texts.

[0068] In this embodiment of the application, when performing step 2002, based on the multiple descriptive phrases generated in step 2001, the descriptive phrases of at least two instance categories are randomly combined to construct multiple image description texts.

[0069] In one possible implementation, step 2002 above can be implemented by a large language model. Specifically, this is achieved by providing the large language model with prompt word templates, enabling the large language model to automatically generate any one of the multiple image description texts based on the prompt word templates. The prompt word templates can assign roles and tasks to the large language model. For example, roles and tasks can be assigned to the large language model as follows: "You are a creative promptcomposer for text-to-image generation models"; "Given a list of entity descriptor phrases, you must compose..."<PROMPT_NUM> Detailed and coherent prompts (Given a set of entity description phrases, you must generate <PROMPT_NUM> detailed and coherent prompts).

[0070] To ensure that the large language model performs the task better, the following rule examples can be given to the large language model, such as "Follow these rules": 1. Combine all This combines all descriptor phrases into one prompt. 2. Add appropriate scene context (environment, atmosphere, style, camera views, lighting conditions) 3. Describe spatial relationships or action or interactionrelationships of the descriptor phrases; 4. Keep the output fluent, natural, and suitable for text-to-image models. 5. Avoid listing items — always merge them into a single cohesive scene. 6. Return the prompts in a list. 7. No more than 80 words.

[0071] With the above settings, the large language model can generate corresponding image description text based on the description phrases of at least two instance categories randomly combined in step 2002.

[0072] Step 2003: Based on any one of the above image description texts, generate a composite image that matches the image description text.

[0073] In this embodiment of the application, when performing step 2003, any one of the multiple image description texts can be input into the text-to-image model, and the text-to-image model can generate a synthetic image that matches the image description text based on the image description text.

[0074] In one possible implementation, the text-to-image model could be the FLUX model (no official full name is currently available in Chinese or English), which employs a hybrid architecture based on multimodal and parallel diffusion Transformer blocks (DiT) and incorporates flow matching technology. The FLUX model can generate high-quality images, i.e., the aforementioned synthesized images, based on the image description text input to the FLUX model.

[0075] Step 2004: Based on the synthesized image, determine the layout conditions of each instance within the synthesized image.

[0076] In this embodiment of the application, when performing step 2004, refer to... Figure 4 As shown, the specific process is as follows: Step 20041: Perform target detection on the synthesized image to obtain the location information of each instance within the synthesized image.

[0077] In this embodiment of the application, when performing step 20041, the target detection algorithm / model in the related technology can be used to perform target detection on the synthetic image and accurately mark the bounding box spatial coordinates (i.e., position information) of each instance in the synthetic image.

[0078] In one possible implementation, when performing step 20041, the synthesized image is input into a Transformer-based end-to-end object detection (DETR with Improved deNoising anchOr boxes, DINO) model or a Grounding DINO model, etc., to perform object detection on the synthesized image, thereby obtaining the position information of each instance in the synthesized image.

[0079] Step 20042: Perform the following operation for any instance in the synthesized image: Operation 1: Segment the sub-image corresponding to the location information of the instance from the synthesized image; Step 2: Perform image editing on the sub-image to generate a reference image for this instance; Step 3: Generate instance description text based on the reference image of the instance; Step 4: Use the instance's location information, reference image, and instance description text as the instance's layout conditions.

[0080] In the first embodiment of this application, after obtaining the location information of any instance in the synthesized image, the instance is segmented from the synthesized image using an instance segmentation method based on the location information of the instance.

[0081] In the second operation, in this embodiment of the application, when generating the reference image, an editing instruction is generated based on candidate prompts of multiple preset dimensions, wherein the multiple preset dimensions include any combination of background dimension, pose dimension or viewpoint dimension; then, based on the editing instruction, the sub-image is edited to generate the reference image of the instance.

[0082] For example, assuming multiple preset dimensions include background, pose, and view dimensions, the candidate cue words that enhance the diversity of reference image generation in the background, pose, and view dimensions are shown in Table 1: Table 1. List of candidate cue words for enhancing reference images

[0083] In one possible implementation, during the reference image generation stage, editing instructions are generated by combining candidate prompts for background, pose, and viewpoint dimensions from Table 1 above. The sub-image and the editing instructions are then input into the image editing model, which performs image editing on the sub-image based on the editing instructions, thereby generating the reference image for that instance.

[0084] In some embodiments, the image editing model described above can be a multimodal image generation and editing model, such as the Kontext model. The Kontext model emphasizes a joint understanding of the context of text and reference images, supporting dual input of text and images to achieve context-aware and accurate editing and generation. Its core functions include modifying images through natural language commands, maintaining character consistency (e.g., multiple rounds of editing without changing character features), local editing (pixel-level adjustment of specific areas), style transfer (converting artistic styles), and background replacement.

[0085] In this embodiment, the process of generating reference images is guided by prompts (i.e., editing instructions) to amplify the differences between the reference image and the original image in terms of background, pose, and viewpoint while maintaining the consistency of the entity identity of the instance. This increases the sample diversity of the reference image and breaks through the scene limitations of traditional datasets. As shown in Table 1 above, in this embodiment, the background of the reference image is not limited to a pure white scene, but also includes a contextual environment that matches the semantics. For human instances, the pose is controlled by specially designed candidate prompts, covering various situations such as standing, sitting, and walking.

[0086] In operation three, in this embodiment of the application, the instance in the reference image of the instance is described by a visual model, thereby obtaining the instance description text of the instance.

[0087] In one possible implementation, the reference image of the instance can be input into a Bootstrapped Language-Image Pretraining (BLIP) model and / or a Qwen3 Vision-Language Model (Qwen3-VL model), and the instance description text of the instance can be generated based on the output of the BLIP model and / or the Qwen3-VL model.

[0088] Operation four involves using the triplet annotations (i.e., three modal data) of the reference image, instance description text, and location information of the instance as the layout conditions for the instance.

[0089] In this embodiment of the application, operations one to four are performed for each instance in the synthesized image to obtain the layout conditions of each instance in the synthesized image, complete the data annotation, and at the same time ensure the accurate correspondence of the layout conditions of the multimodal images.

[0090] In this embodiment of the application, after obtaining the layout conditions of each instance in the synthesized image, before performing subsequent step 2005, refer to... Figure 5 As shown, the following data filtering process also needs to be performed: Step 501: Determine that the reference image of this instance contains a single instance; Step 502: Determine that the similarity between the instance description text of the instance and the reference image of the instance is greater than a first threshold; and determine that the similarity between the reference image of the instance and the sub-image is greater than a second threshold.

[0091] In this embodiment of the application, when performing step 501, it is identified whether the reference image of the instance contains only a single instance; when performing step 502, the similarity between the reference image of the instance and the instance description text, and the similarity between the sub-image of the instance and the reference image are calculated, and it is determined that both are greater than their respective thresholds; then, subsequent step 2005 is performed.

[0092] In one possible implementation, when performing step 501, the Very Large Language Model (VLLM) is used to identify whether the reference image of the instance contains only a single instance to ensure that there is no redundant interference.

[0093] In one possible implementation, when performing step 502, the similarity between the reference image of the instance and the instance description text, as well as the similarity between the sub-image of the instance and the reference image, is calculated using a pre-trained Contrastive Language-Image Pre-training (CLIP) model to ensure that both are greater than their respective thresholds, thereby ensuring semantic consistency and local similarity.

[0094] In other embodiments of this application, if either of the conditions in steps 501 and 502 is not met, the condition is discarded, i.e., the subsequent step 2005 is not executed.

[0095] To ensure that the synthesized image and the reference image of the instance are clear, have accurate colors, acceptable noise levels, and complete structure, in this embodiment of the application, before performing step 501, it is also necessary to evaluate the overall image quality of the synthesized image and the reference image of the instance respectively, so as to determine that the overall image quality scores of the synthesized image and the reference image of the instance are both greater than a preset score threshold.

[0096] In one possible implementation, the overall image quality is evaluated using a VLLM model / Qwen3-VL model to eliminate low-quality samples and ensure data quality.

[0097] In this embodiment of the application, by comparing the above-mentioned multi-dimensional data filtering, namely image quality, single entity constraint, semantic consistency, and local similarity, low-quality data can be eliminated, thereby ensuring that each image sample in the final image sample set is of high quality and has high semantic consistency, so as to construct a reliable image sample set.

[0098] In some embodiments of this application, when performing step 502, the similarity between the reference image and the sub-image (i.e., the image of the corresponding local region of the instance in the synthesized image) can also be measured using DINO. Based on determining that the similarity calculated by CLIP is greater than a second threshold, it is further determined that the similarity determined using DINO is also greater than the second threshold, thereby confirming that the similarity between the reference image and the sub-image of the instance is greater than the second threshold. In this way, through redundant settings, the quality and semantic consistency of the image samples are further guaranteed.

[0099] It should be noted that through the multi-dimensional data filtering described above, reference images of instances that do not meet the filtering criteria will be removed. However, a composite image typically contains multiple instances, so some instances can be filtered out. If none of the instances in the composite image meet the filtering criteria, the entire composite image will be filtered out.

[0100] Step 2005: Take the synthesized image as a raw image, and combine it with the image description text and the layout conditions of the multiple instances within the synthesized image as an image sample in the above image sample set.

[0101] In this embodiment of the application, by performing step 2005, an image sample from the above image sample set is obtained.

[0102] For each of the multiple image description texts generated in step 2002, the aforementioned steps 2003 to 2005 are performed to obtain the aforementioned image sample set.

[0103] The following example illustrates the process of constructing an image sample set.

[0104] Figure 6 The specific construction process of the aforementioned image sample set is illustrated. (See also...) Figure 6As shown, the image sample set construction process in this application embodiment may include three parts: the first part, data generation, is used to generate synthetic images; the second part, instance annotation, is used to annotate each instance in each synthetic image with triples; and the third part, data filtering, is used to remove low-quality and semantically inconsistent data to obtain the aforementioned image sample set. Examples of each part are given below.

[0105] See still in Part 1. Figure 6 As shown, suppose we want to generate a composite image containing a person, a teapot, and a rolling pin. Then, the instance categories (names) include person, teapot, and rolling pin. In this embodiment, a large model generates descriptive phrases for each of the above three instance categories, such as... Figure 6 The illustrations show "woman in apron holding a fan", "a rusticclay teapot", and "a rolling pin on a wooden table".

[0106] Then, the descriptive phrases of the three instance categories mentioned above are randomly combined to obtain the image description text of the synthesized image to be generated, such as... Figure 6 The image description shown is: "In a warmly lit kitchen, a woman in an apron holding a fan stands beside a rolling pin...". This image description is input into the FLUX model. Based on the input image description, the FLUX model generates a high-quality image, i.e., the aforementioned synthesized image, such as... Figure 6 The composite image shown is shown in the image.

[0107] In the second part, the synthesized image is input into the Grounding DINO model, and target detection is performed using the Grounding DINO model to obtain the location information of each instance in the synthesized image, such as... Figure 6 The bounding boxes of each instance shown represent their respective spatial coordinates.

[0108] The synthesized image is input into the Kontext model, which extracts instances from the synthesized image to obtain individual sub-images. Then, for any instance, editing instructions are generated by combining candidate cue words for background, pose, and viewpoint dimensions as shown in Table 1 above. Based on the editing instructions, image editing is performed on the sub-image of that instance to generate a reference image for that instance. This process is repeated for each instance to obtain the following result: Figure 6 The image shows a woman on a white background, a rolling pin on a gray background, and a teapot on a white background.

[0109] Then, the reference images of the three instances are input into the BLIP / VLLM model respectively. With the powerful content understanding capabilities of the BLIP / VLLM model, the input reference images are analyzed to obtain the instance description text for each instance.

[0110] This gives us the layout conditions for each instance: reference image, instance description text, and location information.

[0111] See also Part Three. Figure 6 As shown, the overall image quality is evaluated using the VLLM model, and it is determined whether the reference image of each of the three instances contains only a single instance. The similarity between the sub-image and the reference image of each instance is calculated using the CLIP model and compared with a second threshold. The similarity between the reference image and the instance description text of each instance is also calculated and compared with a first threshold. The similarity between the sub-image and the reference image of each instance is calculated using the DINO model and compared with a second threshold. This process eliminates low-quality samples, retains the synthetic image that meets the criteria as an original image, and stores the original image, the corresponding image description text, and the layout conditions of each instance contained in the synthetic image as an image sample in the image sample set.

[0112] In this embodiment, the image sample set is constructed through the three stages of generation, annotation, and filtering in steps 2001 to 2005, thereby achieving a dual improvement in the quality and diversity of the image sample set. This ensures the accuracy of the annotation and enhances the generalization ability of the model through rich scene and modal variations, providing reliable data support for subsequent layout control training.

[0113] After introducing the specific process of constructing (obtaining) the image sample set, the construction process of the model to be trained will be explained below.

[0114] Step 210: Couple the CondLoRA module into the pre-trained base generative model to obtain the layout control noise fine-tuning model.

[0115] In this embodiment of the application, the layout control noise fine-tuning model is built on the basis of a pre-trained basic generative model, wherein the pre-trained basic generative model includes multiple MM-DiT modules.

[0116] In one possible implementation, see [reference] Figure 7 As shown, the pre-trained basic generative model includes multiple dual-stream MM-DiT modules and multiple single-stream MM-DiT modules. The dual-stream MM-DiT module is a core architecture module specifically designed for multimodal generation tasks. Its core innovation lies in processing image latent variables (hereinafter referred to as visual features in this embodiment) and text latent variables (hereinafter referred to as text features in this embodiment) separately through two sets of independent attention mapping parameters, and fusing them only in the attention calculation stage, thereby achieving efficient alignment while maintaining the purity of each modality feature. The single-stream MM-DiT module processes visual features and text features through a set of attention mapping parameters, and fuses them in the attention calculation stage to achieve more refined modality fusion.

[0117] The attention mapping parameters mentioned above include, but are not limited to, linear mapping parameters for query (Q), key (Q), and value (V), as well as attention output mapping parameters.

[0118] In this embodiment of the application, when performing step 210, a CondLoRA module for processing visual features (hereinafter referred to as visual CondLoRA) and a CondLoRA module for processing text features (hereinafter referred to as text CondLoRA) are coupled in each MM-DiT module included in the pre-trained basic generative model to obtain a layout control noise fine-tuning model.

[0119] In one possible implementation, the layout-controlled noise fine-tuning model includes a multimodal encoder, multiple dual-stream MM-DiT modules (e.g., 19 dual-stream MM-DiT modules, etc.), multiple single-stream MM-DiT modules (e.g., 38 single-stream MM-DiT modules, etc.), and visual CondLoRA and text CondLoRA coupled in each MM-DiT module, wherein, A multimodal encoder includes an image encoder, a text encoder, and a spatial encoder. The image encoder encodes the input image into visual features; the text encoder encodes the input text into text features; and the spatial encoder includes a Fourier embedding module and a multilayer perceptron to encode the input location information into spatial features. Visual CondLoRA is used to perform feature fusion (mapping) on ​​the input visual features to obtain a first fused feature, and to generate a first layout guiding noise based on the first fused feature; The text CondLoRA is used to perform feature fusion (mapping) on ​​the input text features to obtain a second fused feature, and to generate a second layout guiding noise based on the second fused feature.

[0120] In some embodiments, the layout control noise fine-tuning model described above further includes a routing module for routing features and dynamically activating the corresponding feature fusion branch.

[0121] In this embodiment, a dual-branch noise fine-tuning structure is constructed to address the heterogeneity of visual, text, and spatial modal features. A dynamic routing strategy is used to achieve precise matching between modalities and branches: for visual features corresponding to the reference image, the visual branch (visual CondLoRA) is activated for feature processing; for text features corresponding to the instance description text, the text branch (text CondLoRA) is activated; and for spatial features, they are used as input to a newly added third low-rank trainable matrix in the CondLoRA module.

[0122] Figure 8 The diagram illustrates the architecture of an optional layout control noise fine-tuning model. It should be noted that this is for ease of subsequent description. Figure 8 Only one dual-stream MM-DiT module and one single-stream MM-DiT module are shown.

[0123] See Figure 8 As shown, in one possible implementation, in the dual-stream MM-DiT module, the routing module ( Figure 8 The "routing" module separates the visual features of an instance and routes them to the spatial features in the visual CondLoRA for fusion with those features. Figure 8 The term "routing" in Chinese refers to the separation of textual features of an instance and routing to the text CondLoRA for fusion with the spatial features of that instance; in the single-stream MM-DiT module, the routing module ( Figure 8 The visual features of an instance are separated and routed to the visual CondLoRA for fusion with the spatial features of that instance. The text features of an instance are separated and routed to the text CondLoRA for fusion with the spatial features of that instance.

[0124] In this embodiment, the standard denoising diffusion loss function is used as the basic optimization objective during the training of the layout control noise fine-tuning model. The weights of the pre-trained base generative model are frozen, and only the parameters of the visual CondLoRA, text CondLoRA, and spatial encoder are updated. The parameters of the multilayer perceptron in the spatial encoder can be learned during training to ensure the accuracy of spatial feature encoding.

[0125] In this embodiment of the application, at any given time step (denoted as t, t∈[0,1]), the original image (denoted as x0) included in the image sample set is progressively noise-added to obtain a noisy image (denoted as x). t This can be expressed by the following formula: x t =(1-t)x0+t

[0126] in, N(0, I) indicates that the noise follows a standard Gaussian distribution, and I represents the identity matrix.

[0127] The task of the layout-controlled noise fine-tuning model is to predict the noise added to the noisy image and restore the original image distribution based on the layout conditions of multiple instances in the image description text and the original image. The loss function in this embodiment can be expressed by the following formula:

[0128] in, Describe the image with text (or global cue words). For layout conditions, This represents the predicted noise (distribution) of the pre-trained base generative model output.

[0129] The following section explains the processing flow during training from the perspective of an image sample.

[0130] Step 220: Based on the layout conditions of the target instances included in the image samples in the above image sample set, generate layout guiding noise for the target instances, wherein the target instance is any one of the multiple instances included in the image sample.

[0131] In this embodiment, the aforementioned noise-adding method is used to add noise to the original images of the image samples in the image sample set to obtain noisy images. These noisy images, together with the image description text in the image samples and the layout conditions of multiple instances, form training samples for the layout control noise fine-tuning model.

[0132] During the training of the layout control noise fine-tuning model, please refer to... Figure 8As shown, the above noisy image, the image description text in the above image sample, and the layout conditions of multiple instances are input into the layout control noise fine-tuning model. For the image modality, the image encoder in the layout control noise fine-tuning model encodes the noisy image and the reference images of multiple instances into noisy visual features and instance visual features, respectively. For the text modality, the text encoder in the layout control noise fine-tuning model encodes the image description text in the image sample and the instance description text of multiple instances into global text features and instance text features, respectively. For the spatial modality, the spatial encoder in the layout control noise fine-tuning model encodes the position information of multiple instances in the image sample into spatial features, respectively.

[0133] In one possible implementation, the image encoder described above is a variational autoencoder (VAE encoder) configured for the pre-trained base generative model. Then, for a reference image, the VAE encoder encodes the reference image in the image sample to generate instance visual features, thus ensuring consistency with the visual domain representation of MM-DiT.

[0134] In one possible implementation, the text encoder described above is a T5 text encoder configured for a pre-trained base generative model. Then, for instance description text, it is transformed into instance text features by the T5 text encoder to ensure consistency with the text domain of MM-DiT.

[0135] In one possible implementation, the Fourier embedding module in the spatial encoder described above is a Fourier embedding module configured for a pre-trained base generative model. Therefore, for the location information of the instance... The spatial features are obtained by passing them through a Fourier embedding module and then inputting them into a multilayer perceptron.

[0136] In this embodiment of the application, when performing step 220, refer to... Figure 9 As shown, the specific process is as follows: Step 2201: Fuse the instance visual features corresponding to the reference image of the target instance with the spatial features corresponding to the location information of the target instance to obtain the first fused feature; and fuse the instance text features corresponding to the instance description text of the target instance with the spatial features to obtain the second fused feature.

[0137] In related technologies, a traditional LoRA module includes a first low-rank trainable matrix (denoted as A) and a second low-rank trainable matrix (denoted as B). In this embodiment, a third low-rank trainable matrix, denoted as C, is added based on the traditional LoRA module, resulting in the CondLoRA module of this embodiment. Figure 8The visual or textual CondLoRA shown is used to process spatial coordinate features, where the third low-rank trainable matrix C is used to process spatial coordinate features.

[0138] In specific implementation, during step 2201, the encoded spatial features are projected onto a low-rank subspace using the third low-rank trainable matrix C. Instance visual features or instance text features undergo low-rank transformation using the original first low-rank trainable matrix A. Subsequently, the low-ranked spatial features are fused with the instance visual features or instance text features through a concatenation operation. The fused features are then mapped back to the original feature space using the second low-rank trainable matrix B, yielding either the first fused feature or the second fused feature. This low-ranking process significantly reduces additional trainable parameters while achieving efficient fusion of three modalities, balancing parameter efficiency and fusion effectiveness.

[0139] For example, taking target features as an example, assuming the target features correspond to the target fusion features, if the target features are the aforementioned instance visual features, then the target fusion features are the aforementioned first fusion features; if the target features are the aforementioned instance text features, then the target fusion features are the aforementioned second fusion features. Specifically, the target fusion features are obtained by performing the following operations: Operation 1: Map the target features through the first low-rank trainable matrix to obtain the first intermediate features; Operation 2: Map the spatial features through the third low-rank trainable matrix to obtain the second intermediate features; Operation 3: Merge the first and second intermediate features to obtain the third intermediate feature; Operation 4: Reproject the third intermediate feature through the second low-rank trainable matrix to obtain the target fused feature.

[0140] For specific implementation details, please refer to... Figure 8 The dashed bounding box (CondLoRA) shown on the right, assuming the position information of the i-th instance is represented as... To eliminate numerical bias caused by different image resolutions and ensure consistent dimensions of spatial features during training and inference, the location information of instances in the image sample is normalized. The normalized representation is as follows: ,in," "Indicates normalized ; "express It contains 4 items, respectively with Includes " "Correspondingly, the value range for each item is [0,1]; Then, the spatial features obtained after passing through the above multilayer perceptron can be expressed as follows:

[0141] in, represents the spatial features of the i-th instance; MLP stands for Multilayer Perceptron, used to further learn the nonlinear combination relationship between spatial location and scale; Fourier represents the mapping operation, used to map coordinates to a higher-dimensional frequency domain representation that is easier for network modeling.

[0142] Then, the encoded spatial features are processed by the third low-rank trainable matrix C mentioned above. Reprojecting to a lower-rank subspace, the resulting target fusion features can be expressed as follows: .

[0143] in, Let A represent the target fusion feature of the i-th instance; A is the first low-rank trainable matrix, representing the low-rank mapping matrix of the semantic branch; B is the second low-rank trainable matrix, representing the reprojection mapping matrix, which undertakes the function of reprojecting the fusion feature; C is the third low-rank trainable matrix, representing the newly added mapping matrix of the spatial branch. Represents the target feature of the i-th instance; This indicates feature fusion.

[0144] Thus, in this embodiment of the application, the results of semantic branching and spatial branching are jointly modeled in the same feature space through the above operations, so as to achieve the synergistic effect of semantic information and spatial constraints.

[0145] Step 2202: Generate first layout guiding noise for the target instance based on the first fusion feature, and generate second layout guiding noise for the target instance based on the second fusion feature.

[0146] In this embodiment of the application, when performing step 2202, the first fusion feature is mapped to obtain the noise distribution parameters of the visual guidance of the target instance, that is, the corresponding noise mean and noise variance are obtained, and the first layout guidance noise adapted to the layout conditions of the target instance is generated by random sampling; and the second fusion feature is mapped to obtain the noise distribution parameters of the text guidance of the target instance, that is, the corresponding noise mean and noise variance are obtained, and the second layout guidance noise adapted to the layout conditions of the target instance is generated by random sampling.

[0147] The above mapping can be achieved through a single-layer mapping network. This single-layer mapping network includes a mean mapping part and a variance mapping part, which are used to map the fused features into the noise mean and noise variance, respectively.

[0148] In one possible implementation, for any fused feature F, a single-layer mapping network can be used to linearly map the fused feature F to obtain the corresponding noise distribution parameters. Specifically, the single-layer mapping network includes a first linear mapping layer for predicting the noise mean and a second linear mapping layer for predicting the noise scale parameter. The mapping process can be represented by the following formula:

[0149] in, Indicates the noise mean; Indicates the noise variance; , , , These represent the weights.

[0150] For example, see the example visual features. Figure 8 The dashed bounding box (CondLoRA) shown on the right still assumes that the position information of the i-th instance is represented as... Its first fusion feature is represented as , then, The noise distribution parameters (noise mean and noise variance) are obtained through mapping, and the generated noise can be represented by the following formula through random sampling:

[0151] in, This represents the first layout guiding noise of the i-th instance; This represents the mean noise value obtained by mapping the i-th instance; This represents the noise variance obtained by mapping the i-th instance; This represents the noise sampled from the i-th instance. N(0, I) indicates that the noise follows a standard Gaussian distribution.

[0152] Step 230: Use the layout-guided noise of each instance in the image sample as a layout control constraint, and perform CondLoRA fine-tuning training on the above layout control noise fine-tuning model until the preset convergence condition is reached to obtain the target generation model.

[0153] In this embodiment of the application, when performing step 230, refer to... Figure 10 As shown, the specific process is as follows: Step 2301: Superimpose the first layout guidance noise with the instance visual features of the target instance to obtain the first superimposed feature; and superimpose the second layout guidance noise with the instance text features of the target instance to obtain the second superimposed feature.

[0154] In this embodiment, both the first layout guidance noise and the second layout guidance noise are positive excitation noise, which is helpful for model training. Therefore, during step 2301, the first layout guidance noise is superimposed on the instance visual features of the target instance to generate a first superimposed feature, and the second layout guidance noise is superimposed on the instance text features of the target instance to generate a second superimposed feature.

[0155] It should be noted that since the above layout control noise fine-tuning model has multiple MM-DiT modules, during the superposition, for each MM-DiT module, the first layout guiding noise / second layout guiding noise is superimposed with the instance visual features / instance text features input to the MM-DiT module to generate a first superimposed feature / second superimposed feature containing multimodal layout constraints, which participates in the self-attention calculation in the MM-DiT module.

[0156] Step 2302: Use the first and second superimposed features of each instance in the image sample as layout control constraints, combine them with a preset time step, obtain the prediction noise output by the above basic generation model, and perform iterative denoising operation on the noisy image corresponding to the image sample based on the prediction noise, wherein the noisy image is obtained by adding noise to the original image in the image sample.

[0157] In this embodiment of the application, when performing step 2302, refer to... Figure 11 As shown, the specific process is as follows: Step 23021: Mask the first and second superimposed features between different instances, and mask the global text features corresponding to the image description text in the image sample for each instance's first and second superimposed features.

[0158] In this embodiment of the application, in order to ensure cross-instance information masking when multiple instances are generated, before performing self-attention calculation, step 23021 is executed to mask the features between different instances so that the features of any instance are not affected by the features of other instances during the self-attention calculation process; and, for the first superimposed feature and the second superimposed feature of each instance, the global text features corresponding to the image description text in the image sample are masked so that the features of any instance are not affected by the global text features during the self-attention calculation process.

[0159] like Figure 12 As shown, Figure 12The diagram illustrates the constraint of the feature interaction range by the attention masking mechanism. In this model, the noisy visual features corresponding to a noisy image are treated as global features and can interact with all modal features. The global text features corresponding to the image description text are only allowed to interact with the noisy visual features and do not directly participate in instance-level feature interactions. Features corresponding to the reference image and instance description text of the same instance can interact with each other, while features from different instances are blocked from interacting through a mask. Compared to existing cross-attention or adapter fusion methods, this embodiment of the application reduces computational overhead while achieving efficient alignment and precise spatial control of multimodal information, effectively mitigating representation bias and information loss problems.

[0160] Step 23022: Perform self-attention calculation based on the first superimposed feature and the second superimposed feature of each instance after masking, as well as the noisy visual features and the global text features after masking corresponding to the noisy image. Combined with the above time steps, the above predicted noise is obtained.

[0161] In this embodiment, masking effectively blocks cross-instance information interaction, ensuring the independence and rationality of the multi-instance layout.

[0162] Step 2303: Based on the predicted noise and the real noise in the noisy image, optimize the parameters in the CondLoRA module until the aforementioned convergence condition is met, and obtain the target generation model.

[0163] In this embodiment of the application, when performing step 2303, based on the standard denoising diffusion loss function (see the aforementioned formula), the loss value is obtained according to the predicted noise and the real noise in the noisy image, and the parameters in the CondLoRA module are optimized based on the loss value, that is, the parameters in the aforementioned first low-rank trainable matrix, second low-rank trainable matrix and third low-rank trainable matrix are adjusted until the aforementioned convergence condition is reached, thereby obtaining the target generation model.

[0164] In this embodiment, the location information is first encoded and then fused with semantic features (visual features or text features). The layout-guided noise is generated through low-rank projection and noise parameterization, and finally applied to the feature update in the denoising stage. That is, the location information (spatial information) is used to participate in noise distribution control. Compared with related technologies that only perform style feature embedding or channel splicing for model training, this significantly improves the spatial layout accuracy and instance visual consistency of the generated image.

[0165] To avoid the model over-reliance on image description text (global cue words) for image samples during training, thus neglecting layout conditions and weakening layout control, this application provides an optimization and enhancement method to improve model learning stability. Specifically, before training, a predetermined proportion of image description text in the aforementioned image sample set is converted to null tokens. This forces the model to rely more on layout conditions during training when dealing with the lack of global cue guidance (image description text), significantly enhancing the model's attention to local layout and effectively alleviating competition between different modalities. Finally, by adjusting hyperparameters (such as learning rate, weight decay, etc.) and comparing the model's performance on the validation set under different settings, the optimal hyperparameter combination is selected to further improve training results.

[0166] In this embodiment, visual and textual features are uniformly encoded by reusing the internal parameters of MM-DiT in the multimodal encoding stage. Compared with related technologies where there is a representational deviation between multimodal conditionally encoded features and the internal features of the basic generative model, this ensures the consistency between multimodal conditionally encoded features and the internal features of the basic generative model, while retaining the prior knowledge of the pre-trained basic generative model. Furthermore, based on the classic LoRA module, a third low-rank trainable matrix C is added for fusing spatial features. Low-rank adaptation technology is used to significantly reduce additional trainable parameters while achieving precise constraints on spatial positions. Conditionally independent attention masks effectively block cross-instance information interaction, ensuring the independence and rationality of multi-instance layouts. Ultimately, this network significantly improves the spatial layout accuracy and instance visual consistency of generated images while reducing computational overhead.

[0167] After introducing the model training process, the following section explains the processing flow after the target generation model is deployed.

[0168] See Figure 13 As shown in the figure, this application provides an image generation method, which includes, but is not limited to, the following steps: Step 1300: Obtain the description information of the target image to be generated, wherein the description information includes the image description information of the target image and the instance description information of multiple instances that need to be contained in the target image. The instance description information of a single instance includes the instance description text and location information of that instance.

[0169] Step 1310: Input the description information into the trained target generation model, and use the image output by the target generation model as the target image.

[0170] The target generation model mentioned above was trained using the aforementioned training method.

[0171] In step 1300, for example, in the field of virtual scene construction, the user inputs the description information of the target image to be generated through the user interaction interface displayed on the user terminal device, so as to obtain the above description information and trigger image generation.

[0172] In step 1310, the description information is input into the target generation model to obtain the target image.

[0173] In this embodiment of the application, the above-mentioned image description information and the above-mentioned instance description information may include descriptive text or voice data, etc. This application does not make specific limitations. If it is voice data, a speech-to-text module can be embedded to facilitate the recognition of the target generation model.

[0174] In some embodiments of this application, when performing step 1310, the above-mentioned description information is input into the target image generation model. The target generation model identifies that the description information does not contain the specified image provided by the user. The specified image contains at least one instance of the multiple instances that the target image needs to contain. Then, based on the instance description information of any instance of the multiple instances in the target image, a reference image containing an object that matches the instance description information of that instance is randomly generated. Based on the reference images and image description information of the multiple instances, the target image is generated according to the position information of the multiple instances.

[0175] In other embodiments of this application, the above-mentioned description information also includes a specified image, which contains at least one instance of a plurality of instances contained in the target image. In this case, when performing step 1310, the description information is input into the target generation model, and instance identification is performed on the specified image. The image containing the instance segmented from the specified image is used as the reference image of the corresponding instance. Based on the instance description information of the other instances among the plurality of instances besides the above-mentioned at least one instance, a reference image containing an object that matches the instance description information of the other instances is randomly generated. Based on the reference images and image description information of the plurality of instances, the target image is generated according to the position information of the plurality of instances.

[0176] Figure 14 The diagram shows descriptions of four image generation tasks and target images generated based on those descriptions. Description information 1 includes image description information (A dog and a sneaker on the grass), as well as instance description information for two instances (dog and shoe) (a dog and a sneaker) and specified image and location information (on the grass). Description information 2 includes image description information (A cat, a dog, and a dog in a room), as well as instance description information (a cat, a dog, a dog) for 3 instances (cat, dog 1, dog 2) and specified image and location information (in a room). Description information 3 includes image description information (A backpack, a clock, and a flower in the jungle), as well as instance description information for 3 instances (backpack, clock, flower) and specified image and location information (in the jungle). Description information 4 includes image description information (A woman wearing a hat, a t-shirt, and atrousers in the jungle), as well as instance description information for 4 instances (woman, hat, t-shirt, trousers) and specified image and location information (A woman wearing).

[0177] Therefore, by inputting the aforementioned description information 1, description information 2, description information 3, and description information 4 into the target generation model, the respective target images are obtained, such as... Figure 14 The target images shown are 1, 2, 3, and 4. In any of these target images, the target generation model can accurately generate a target image that matches the descriptive information, and the layout is reasonable.

[0178] Based on the same inventive concept, see [reference] Figure 15 As shown in the embodiment of this application, a training apparatus for an image generation model is provided, comprising: The acquisition unit 1510 is used to acquire an image sample set, wherein a single image sample in the image sample set includes an original image, image description text, and layout conditions of multiple instances contained in the original image, and the layout conditions of a single instance include a reference image, instance description text, and location information. Model building unit 1520 is used to couple the conditional low-rank adaptation CondLoRA module into the pre-trained basic generative model to obtain a layout control noise fine-tuning model. The noise generation unit 1530 is used to generate layout guiding noise for the target instance based on the layout conditions of the target instance included in the image sample set, wherein the target instance is any one of a plurality of instances included in the image sample. The fine-tuning unit 1540 is used to use the layout-guided noise of each instance in the image sample as a layout control constraint, and to perform CondLoRA fine-tuning training on the layout control noise fine-tuning model until a preset convergence condition is reached to obtain the target generation model.

[0179] In one possible implementation, the acquisition unit 1510 is specifically used for: The existing image dataset is integrated and expanded to include multiple instance categories. For each of the expanded instance categories, N descriptive phrases are generated. The descriptive phrases of at least two instance categories are randomly combined to obtain multiple image description texts, where N is a positive integer greater than 1. Based on any one of the multiple image description texts, generate a composite image that matches any one of the image description texts, and based on the composite image, determine the layout conditions of each of the multiple instances within the composite image; The synthesized image is taken as the original image, along with the image description text and the layout conditions of each of the multiple instances within the synthesized image, and is taken as one of the image samples in the image sample set.

[0180] In one possible implementation, the acquisition unit 1510 is specifically used for: Target detection is performed on the synthesized image to obtain the location information of each instance within the synthesized image; For any instance of the synthesized image, perform the following operation: Segment the sub-image corresponding to the location information of any instance from the synthesized image; Image editing is performed on the sub-image to generate a reference image for any instance, and instance description text for any instance is generated based on the reference image for any instance; The location information, reference image, and instance description text of any instance are used as the layout conditions for any instance.

[0181] In one possible implementation, before using the synthesized image as the original image, the acquisition unit 1510 is further configured to: It is determined that the reference image of any of the instances contains a single instance; The similarity between the instance description text of any instance and the reference image of any instance is determined to be greater than a first threshold; and the similarity between the reference image of any instance and the sub-image is determined to be greater than a second threshold.

[0182] In one possible implementation, the acquisition unit 1510 is specifically used for: Based on candidate prompts from multiple preset dimensions, editing instructions are generated. These preset dimensions include any combination of background, pose, or viewpoint dimensions. Based on the editing instructions, the sub-image is edited to generate a reference image for any of the instances.

[0183] In one possible implementation, the noise generation unit 1530 is specifically used for: The instance visual features corresponding to the reference image of the target instance are fused with the spatial features corresponding to the location information of the target instance to obtain the first fused feature; The instance text features corresponding to the instance description text of the target instance are fused with the spatial features to obtain a second fused feature; A first layout guiding noise for the target instance is generated based on the first fusion feature, and a second layout guiding noise for the target instance is generated based on the second fusion feature.

[0184] In one possible implementation, the fine-tuning unit 1540 is specifically used for: The first layout guidance noise is superimposed with the instance visual features of the target instance to obtain a first superimposed feature, and the second layout guidance noise is superimposed with the instance text features of the target instance to obtain a second superimposed feature. The first and second superimposed features of each instance in the image sample are used as layout control constraints. Combined with a preset time step, the predicted noise output by the basic generation model is obtained. Based on the predicted noise, the noisy image corresponding to the image sample is subjected to iterative denoising operation. The noisy image is obtained by adding noise to the original image in the image sample. Based on the predicted noise and the real noise in the noisy image, the parameters in the CondLoRA module are optimized until the convergence condition is met, thus obtaining the target generation model.

[0185] In one possible implementation, the fine-tuning unit 1540 is specifically used for: Masking is performed on the first and second superimposed features between different instances, and, for the first and second superimposed features of each instance, the global text features corresponding to the image description text in the image sample are masked. Self-attention is calculated based on the first and second superimposed features of each instance after masking, as well as the global text features and the global text features after masking, and the predicted noise is obtained by combining the time step.

[0186] In one possible implementation, the CondLoRA module includes a first low-rank trainable matrix, a second low-rank trainable matrix, and a third low-rank trainable matrix; target features correspond to target fusion features. If the target feature is the instance visual feature, then the target fusion feature is the first fusion feature; if the target feature is the instance text feature, then the target fusion feature is the second fusion feature. The target fusion feature is obtained by performing the following operation: The target features are mapped through the first low-rank trainable matrix to obtain the first intermediate features; The spatial features are mapped through the third low-rank trainable matrix to obtain the second intermediate features; The first intermediate feature and the second intermediate feature are fused to obtain the third intermediate feature; The third intermediate feature is reprojected onto the second low-rank trainable matrix to obtain the target fusion feature.

[0187] Based on the same inventive concept, see [reference] Figure 16 As shown, this application provides an image generation apparatus, including: The acquisition unit 1610 is used to acquire the description information of the target image to be generated. The description information includes the image description information of the target image and the instance description information of multiple instances that need to be included in the target image. The instance description information of a single instance includes the instance description text and location information of the instance. Image generation unit 1620 is used to input the description information into a trained target generation model and use the image output by the target generation model as the target image, wherein the target generation model is trained according to any one of the above-described image generation model training methods.

[0188] See Figure 17 As shown, this application provides an electronic device that can implement the functions of any of the aforementioned methods. (Refer to...) Figure 17 The electronic device includes: At least one processor 171 and a memory 172 connected to at least one processor 171. In this embodiment, the specific connection medium between the processor 171 and the memory 172 is not limited. Figure 17 The example shown is the connection between processor 171 and memory 172 via bus 170. Bus 170 is... Figure 17 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Bus 170 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 17 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, processor 171 can also be called a controller; there is no restriction on the name.

[0189] In this embodiment, memory 172 stores instructions executable by at least one processor 171. By executing the instructions stored in memory 172, at least one processor 171 can perform any of the methods discussed above. Processor 171 can implement the functions of any of the aforementioned devices.

[0190] In one possible implementation, processor 171 may include one or more processing units. Processor 171 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 171. In some embodiments, processor 171 and memory 172 may be implemented on the same chip; in some embodiments, they may be implemented separately on separate chips.

[0191] Processor 171 can be a general-purpose processor, such as a Central Processing Unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of any method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0192] Memory 172, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 172 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 172 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 172 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0193] By designing and programming the processor 171, the code corresponding to any of the methods described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute the steps of any of the methods in the foregoing embodiments during operation. How to design and program the processor 171 is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0194] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform any of the methods described in the above embodiments.

[0195] In some possible implementations, various aspects of the methods provided in this application may also be implemented as a program product comprising program code that, when the program product is run on a device, causes the device to perform the steps of the methods described above according to the various exemplary embodiments of this application.

[0196] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0198] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flows in a flowchart and / or one or more blocks in a block diagram.

[0199] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0200] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A training method for an image generation model, characterized in that, include: Obtain an image sample set, wherein a single image sample in the image sample set includes an original image, image description text, and layout conditions of multiple instances contained within the original image, and the layout conditions of a single instance include a reference image, instance description text, and location information; By coupling the conditional low-rank adaptation CondLoRA module into the pre-trained basic generative model, a layout control noise fine-tuning model is obtained. Based on the layout conditions of the target instances included in the image samples in the image sample set, layout guiding noise for the target instances is generated, wherein the target instance is any one of the multiple instances included in the image samples; The layout-guided noise of each instance in the image sample is used as a layout control constraint. The layout control noise fine-tuning model is trained using CondLoRA until the preset convergence condition is met, thus obtaining the target generation model.

2. The method as described in claim 1, characterized in that, The acquisition of the image sample set includes: The existing image dataset is integrated and expanded to include multiple instance categories. For each of the expanded instance categories, N descriptive phrases are generated. The descriptive phrases of at least two instance categories are randomly combined to obtain multiple image description texts, where N is a positive integer greater than 1. Based on any one of the multiple image description texts, generate a composite image that matches any one of the image description texts, and based on the composite image, determine the layout conditions of each of the multiple instances within the composite image; The synthesized image is taken as the original image, along with the image description text and the layout conditions of each of the multiple instances within the synthesized image, and is taken as one of the image samples in the image sample set.

3. The method as described in claim 2, characterized in that, The step of determining the layout conditions of multiple instances within the synthesized image based on the synthesized image includes: Target detection is performed on the synthesized image to obtain the location information of each instance within the synthesized image; For any instance of the synthesized image, perform the following operation: Segment the sub-image corresponding to the location information of any instance from the synthesized image; Image editing is performed on the sub-image to generate a reference image for any instance, and instance description text for any instance is generated based on the reference image for any instance; The location information, reference image, and instance description text of any instance are used as the layout conditions for any instance.

4. The method as described in claim 3, characterized in that, Before using the synthesized image as the original image, the method further includes: It is determined that the reference image of any of the instances contains a single instance; The similarity between the instance description text of any instance and the reference image of any instance is determined to be greater than a first threshold; and the similarity between the reference image of any instance and the sub-image is determined to be greater than a second threshold.

5. The method as described in claim 3, characterized in that, The step of image editing the sub-image to generate a reference image for any instance includes: Based on candidate prompts from multiple preset dimensions, editing instructions are generated. These preset dimensions include any combination of background, pose, or viewpoint dimensions. Based on the editing instructions, the sub-image is edited to generate a reference image for any of the instances.

6. The method according to any one of claims 1-5, characterized in that, The step of generating layout-guided noise for the target instance based on the layout conditions of the target instance included in the image sample set includes: The instance visual features corresponding to the reference image of the target instance are fused with the spatial features corresponding to the location information of the target instance to obtain the first fused feature; The instance text features corresponding to the instance description text of the target instance are fused with the spatial features to obtain a second fused feature; A first layout guiding noise for the target instance is generated based on the first fusion feature, and a second layout guiding noise for the target instance is generated based on the second fusion feature.

7. The method as described in claim 6, characterized in that, The step of using the layout-guided noise of each instance in the image sample as a layout control constraint, and performing CondLoRA fine-tuning training on the layout control noise fine-tuning model until a preset convergence condition is reached to obtain the target generation model includes: The first layout guidance noise is superimposed with the instance visual features of the target instance to obtain a first superimposed feature, and the second layout guidance noise is superimposed with the instance text features of the target instance to obtain a second superimposed feature. The first and second superimposed features of each instance in the image sample are used as layout control constraints. Combined with a preset time step, the predicted noise output by the basic generation model is obtained. Based on the predicted noise, the noisy image corresponding to the image sample is subjected to iterative denoising operation. The noisy image is obtained by adding noise to the original image in the image sample. Based on the predicted noise and the real noise in the noisy image, the parameters in the CondLoRA module are optimized until the convergence condition is met, thus obtaining the target generation model.

8. The method as described in claim 7, characterized in that, The step of using the first and second superimposed features of each instance in the image sample as layout control constraints, combined with a preset time step, to obtain the prediction noise output by the basic generative model includes: Masking is performed on the first and second superimposed features between different instances, and, for the first and second superimposed features of each instance, the global text features corresponding to the image description text in the image sample are masked. Self-attention is calculated based on the first and second superimposed features of each instance after masking, the noisy visual features corresponding to the noisy image, and the global text features after masking. Combined with the time step, the predicted noise is obtained.

9. The method as described in claim 6, characterized in that, The CondLoRA module includes a first low-rank trainable matrix, a second low-rank trainable matrix, and a third low-rank trainable matrix; the target feature corresponds to the target fusion feature. If the target feature is the instance visual feature, then the target fusion feature is the first fusion feature; if the target feature is the instance text feature, then the target fusion feature is the second fusion feature. The target fusion features are then obtained by performing the following operations: The target features are mapped through the first low-rank trainable matrix to obtain the first intermediate features; The spatial features are mapped through the third low-rank trainable matrix to obtain the second intermediate features; The first intermediate feature and the second intermediate feature are fused to obtain the third intermediate feature; The third intermediate feature is reprojected onto the second low-rank trainable matrix to obtain the target fusion feature.

10. An image generation method, characterized in that, include: Obtain the description information of the target image to be generated. The description information includes the image description information of the target image and the instance description information of multiple instances that need to be included in the target image. The instance description information of a single instance includes the instance description text and location information of the instance. The description information is input into a trained target generation model, and the image output by the target generation model is used as the target image, wherein the target generation model is trained by the method according to any one of claims 1-9.

Images