Diffusion generation method based on implicit multi-modal guided alignment

Abstract

The application provides a diffusion generation method based on implicit multimodal guidance alignment, and relates to the technical field of image processing. In the application, for the diffusion generated image and the expected text prompt, an implicit multimodal guidance feature for aligning the initial image generated by the diffusion model is generated by a multimodal large language model; through an implicit aligner, the features in the initial image generated by the diffusion model that do not match the expected prompt text are aligned by operating the hidden features; the diffusion generation method based on implicit multimodal guidance alignment provided in the application effectively reduces the mismatch between the generated image and the expected text prompt without sacrificing the output quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a diffusion generation method based on implicit multi-modal guided alignment. BACKGROUND

[0002] Recently, diffusion models have become powerful tools for text-to-image (T2I) generation, capable of generating diverse and realistic images. However, diffusion models still require human inspection to find misalignments between input and output. More specifically, diffusion models occasionally miss or misinterpret certain content of the input text prompt, resulting in undesirable visual results with misinterpretations. In related studies, even the latest advanced diffusion model, FLUX, needs further correction in terms of conceptual awareness and aesthetic quality.

[0003] Due to the above challenges, exploring methods to improve the misalignment between the prompt text and the generated image has become an emerging and critical research field. Early work in this field mainly adopts preference-based weight fine-tuning. For example, related studies propose fine-tuning diffusion models based on text prompts and corresponding high-quality images. Related studies also apply reinforcement learning (RLHF) learned from human feedback, exploring reward algorithms and datasets. However, these methods either focus on a small number of prompts and feedback or require a large amount of data to fine-tune model weights. SUMMARY

[0004] The present application provides a diffusion generation method based on implicit multi-modal guided alignment and products to at least partially solve the above problems.

[0005] The first aspect of the present application provides a diffusion generation method based on implicit multi-modal guided alignment, the method comprising:

[0006] generating an initial image by a diffusion model based on an expected text prompt;

[0007] constructing an alignment instruction based on the expected text prompt and the initial image, the alignment instruction instructing a multi-modal large language model to generate an editing instruction that matches the initial image with the expected text prompt based on the initial image and the expected text prompt;

[0008] inputting the alignment instruction into a multi-modal large language model to obtain an implicit multi-modal guided feature for aligning the initial image;

[0009] processing the initial image based on an image prompt encoder to obtain an initial image feature;

[0010] inputting the initial image feature and the implicit multi-modal guided feature into an implicit aligner to obtain an aligned image feature;

[0011] Based on the aligned image features, an aligned image is generated by the diffusion model.

[0012] Optionally, the training step of the multi-modal large language model comprises:

[0013] An original image, an edited image, a text prompt corresponding to the original image, and a text prompt corresponding to the edited image, and an editing instruction for obtaining the edited image based on the original image are obtained.

[0014] The original image, the text prompt corresponding to the edited image, and the editing instruction are taken as a training triple.

[0015] Based on a plurality of training triples, a multi-modal large language model is trained, and during the training process, the original image and the text prompt corresponding to the edited image are taken as input information, and the editing instruction is taken as supervision information.

[0016] Optionally, the alignment instruction is input into the multi-modal large language model to obtain an implicit multi-modal guidance feature for aligning the initial image, comprising:

[0017] The alignment instruction is input into the multi-modal large language model, and the features of the last hidden layer of the multi-modal large language model are extracted as the implicit multi-modal guidance feature.

[0018] Optionally, the implicit aligner is trained by the following steps:

[0019] Obtain triple training data ; wherein, and respectively represent the preferred image and the non-preferred image obtained by the diffusion model based on the expected text prompt ;

[0020] Obtain and corresponding image features: preferred feature and non-preferred feature ;

[0021] Based on the input condition , the preset network is trained as the target, and the formula is as follows:

[0022] ;

[0023] Wherein, h represents the implicit multi-modal guidance feature for aligning the non-preferred image obtained by the multi-modal large language model based on the non-preferred image and the text prompt; f ​θ c ) represents the output result of the preset network.

[0024] Optionally, the training step of the implicit aligner further comprises:

[0025] obtaining a reference aligner f ref ; the reference aligner is obtained by randomly initializing network parameters of a preset network;

[0026] combined with a direct preference optimization (DPO) algorithm and a self-play fine-tuning (SPIN) algorithm, based on the reference aligner f ref and the preset network f θ perform a second training:

[0027] ;

[0028] ;

[0029] In the training process, when f θ c ) is closer to the preferred feature f ref c , the network parameters of f θ are iteratively copied to f ref ; wherein, f ref c ) represents the output result of the reference aligner, is a monotonically decreasing convex function, ( x )= log(1 + e −x ); λ represents a ratio parameter.

[0030] Optionally, the method further comprises:

[0031] constructing a secondary alignment instruction based on the expected text prompt and the alignment image;

[0032] inputting the secondary alignment instruction into a multi-modal large language model to obtain an implicit multi-modal secondary guidance feature for aligning the alignment image;

[0033] processing the alignment image based on an image prompt encoder to obtain an alignment image feature;

[0034] ​​​​​inputting the aligned image feature and the implicit multimodal secondary guidance feature into an implicit aligner to obtain a secondary aligned image feature;

[0035] generating, based on the secondary aligned image feature, a secondary aligned image through a diffusion model.

[0036] The second aspect of the application provides a diffusion generation device based on implicit multimodal guidance alignment, which comprises:

[0037] an image generation module configured to generate an initial image based on an expected text prompt through a diffusion model;

[0038] an instruction construction module configured to construct an alignment instruction based on the expected text prompt and the initial image, the alignment instruction instructing a multimodal large language model to generate an editing instruction that matches the initial image with the expected text prompt based on the initial image and the expected text prompt;

[0039] a guidance feature determination module configured to input the alignment instruction into the multimodal large language model to obtain an implicit multimodal guidance feature for aligning the initial image;

[0040] a feature extraction module configured to process the initial image based on an image prompt encoder to obtain an initial image feature;

[0041] a feature alignment module configured to input the initial image feature and the implicit multimodal guidance feature into an implicit aligner to obtain an aligned image feature;

[0042] an alignment module configured to generate an aligned image based on the aligned image feature through the diffusion model.

[0043] The third aspect of the application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the diffusion generation method based on implicit multimodal guidance alignment as described in the first aspect of the application.

[0044] The fourth aspect of the application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the diffusion generation method based on implicit multimodal guidance alignment as described in the first aspect of the application.

[0045] The fifth aspect of the application provides a computer program product comprising a computer program / instruction, wherein the computer program / instruction is executed by a processor to implement the steps of the diffusion generation method based on implicit multimodal guidance alignment as described in the first aspect of the application.

[0046] The image alignment method of the implicit multi-modal guidance feature and diffusion model provided in the application first uses a multi-modal large language model (MLLM) capable of visual question answering as a detector of the existing misalignment between the initial image and the expected text prompt. Based on the expected text prompt, the initial image is sampled from the diffusion model, and the MLLM analyzes the initial image and identifies potential misalignments. In the present application, it is considered that identifying misalignments is not the default task of the MLLM; therefore, in the present application, the MLLM is trained by the original image, the expected text prompt and the corresponding editing instruction to customize the MLLM.

[0047] In the present application, in order to align the potential misalignment, the initial image is converted into image features using an image prompt encoder. Similar to the image variation task, these features enable the model to reconstruct a picture similar to the initial image based on the features. The goal of the present application is to align the misalignment, not to reconstruct the input. Therefore, in the present application, an implicit aligner is introduced, which is a diffusion model adapter designed to convert the initial image features into aligned image features that better match the expected text prompt. In the present application, the implicit aligner is trained using the preferred image and the non-preferred image based on the human preference obtained from the text prompt. And there is a training target that combines a direct preference optimization algorithm and a self-play fine-tuning algorithm.

[0048] In summary, in the present application, based on the multi-modal large language model (MLLM), the image prompt encoder and the implicit aligner, the images generated by the diffusion model are effectively aligned without the need to fine-tune the original diffusion model weights, resulting in high-quality images that match the expected text prompt. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the description of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0050] Figure 1 is a step flow chart of the diffusion generation method based on implicit multi-modal guidance alignment provided by the present application;

[0051] Figure 2 is an exemplary schematic diagram of the training process of the multi-modal large language model in the diffusion generation method based on implicit multi-modal guidance alignment provided by the present application;

[0052] Figure 3 is an exemplary flowchart of the diffusion generation method based on implicit multi-modal guidance alignment provided by the present application;

[0053] Figure 4is an exemplary schematic diagram of the secondary alignment of the diffusion generation method based on implicit multi-modal guided alignment provided in the present application;

[0054] Figure 5 is an exemplary schematic diagram of the image generation effect of the comparative model in the comparative embodiment of the diffusion generation method based on implicit multi-modal guided alignment provided in the present application. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0056] In recent years, diffusion models have become the main method for text-to-image generation. However, it is still challenging to ensure the precise alignment between the generated images and the input prompts. With the significant progress of multi-modal large language models (MLLMs) in visual understanding, the present application proposes a novel network framework: implicit multi-modal guidance (IMG), which is used to enhance the adherence of the images generated by the diffusion model to the expected prompt text and the aesthetic quality, and realizes the diffusion generation method based on implicit multi-modal guided alignment proposed in the present application. Specifically, in the present application, for the diffusion generated image and the expected text prompt, IMG performs the following operations: a) generates implicit multi-modal guidance features for aligning the initial image generated by the diffusion model through a multi-modal large language model; b) aligns the features in the initial image generated by the diffusion model that do not match the expected prompt text by operating on the hidden features through an implicit aligner; c) formulates a trainable objective based on the alignment target (i.e., the iteratively updated preference target). The present application also conducts extensive qualitative and quantitative evaluations through comparative embodiments, in which the diffusion generation method based on implicit multi-modal guided alignment proposed in the present application effectively reduces the mismatch between the generated image and the expected text prompt without sacrificing the output quality.

[0057] Specifically, a step flowchart of the diffusion generation method based on implicit multi-modal guided alignment is provided in the present application, as shown in Figure 1 Specifically, the diffusion generation method based on implicit multi-modal guided alignment includes the following steps:

[0058] S101, generating an initial image based on an expected text prompt through a diffusion model.

[0059] The diffusion model is an image generation method involving a forward process and a reverse process. The forward process is usually referred to as the progressive process, which converts a data point x 0 into Gaussian noise x T , for example, the data point obtained in the progressive process of any step x tThe classical formulation of the problem is defined as:

[0060] (1)

[0061] where t ~ u (0, T ), ~ N (0, I ) is a random Gaussian noise, α t and σ t is a predefined function of t .

[0062] The reverse process iteratively converts x T into x 0 by training a well-tuned deep neural network θ to evaluate intermediate results x t .

[0063] Considering the text-to-image generation task, given an image x 0 and a text condition c T , θ the training objective can be formulated as:

[0064] (2)

[0065] Currently, diffusion models with image prompts are maturing, which usually reconstruct an image similar to the input image through a frozen T2I model with an additional newly trained adapter that takes visual input. Thus, the training objective is extended to include an additional image condition c I , which is usually the encoded features of x 0:

[0066] (3)

[0067] In the inference process, the user inputs c I and c T (optionally) to create or enhance a content-consistent variant of x 0.

[0068] S102, construct an alignment instruction based on the expected text prompt and the initial image, the alignment instruction instructing a multi-modal large language model to generate an editing instruction that matches the initial image with the expected text prompt based on the initial image and the expected text prompt.

[0069] Diffusion models sometimes misinterpret or ignore parts of the prompt when generating images, resulting in generated images that do not align with the expected text prompt. To address this problem, the present application further proposes to identify these inconsistencies through multi-modal large language models (MLLMs) that are capable of visual question answering. In IMG, the present application addresses this problem by introducing a custom multi-modal large language model, which is trained on instruction-based image data to enable the multi-modal large language model to analyze and respond to potential misalignment features.

[0070] Specifically, in the present application, the training step of the multi-modal large language model includes:

[0071] S1, obtaining an original image, an edited image, a text prompt corresponding to the original image, and a text prompt corresponding to the edited image, and an editing instruction for obtaining the edited image based on the original image;

[0072] S2, taking the original image, the text prompt corresponding to the edited image, and the editing instruction as a training triple;

[0073] S3, training a multi-modal large language model based on a plurality of training triples, and in the training process, taking the text prompts corresponding to the original image and the edited image as input information, and taking the editing instruction as supervision information.

[0074] Specifically, the instruction-based image data includes an original image I 0, an edited image I 1, and respective text prompts T 0 and T 1 of the two images, and an editing instruction for obtaining the edited image based on the original image T E The present application utilizes these instruction-based image data as training triples by taking out I 0, T 1 and T E as training triples of the present application.

[0075] When I 0 and T 1 are provided as input to the multi-modal large language model (MLLM), the model is required to generate an editing instruction that matches the original image I 0> to the expected text prompt T1: <edit instruction T E such a question prompt to describe the alignment situation, and then according to T E the supervision model output. As Figure 2 shown, it shows an exemplary schematic diagram of the training process of a multi-modal large language model.

[0076] S103, input the alignment instruction into the multi-modal large language model to obtain an implicit multi-modal guidance feature for aligning the initial image.

[0077] In the embodiments of the present application, the alignment instruction can be input into the multi-modal large language model, and the features of the last hidden layer of the multi-modal large language model are extracted as the implicit multi-modal guidance feature.

[0078] S104, processing the initial image based on the image prompt encoder to obtain an initial image feature.

[0079] In the present application, the image prompt encoder can use any mature method in related research, for example, the IP-Adapter trained on SDXL and FLUX can be used to realize image prompting and extract image features.

[0080] S105, input the initial image feature and the implicit multi-modal guidance feature into the implicit aligner to obtain an aligned image feature.

[0081] In IMG, the next step needs to align the generated image by removing the inconsistencies between the image detected by the multi-modal large language model and the expected text prompt.

[0082] The implicit aligner of the present application is an adaptive network applied to the original diffusion model, which helps to align the initial image to the image that matches the expected text prompt. The principle behind the design of the present application is to align the implicit features during the diffusion process, rather than explicitly correcting the initial image.

[0083] In the present application, the implicit aligner is composed of a series of cross-attention layers, which takes the hidden state of the multi-language model (MLLM), i.e. the multi-modal guidance feature, as one of the inputs. The other input of the implicit aligner is the initial image feature extracted by the image prompt encoder. The multi-modal guidance feature can promote the alignment of the initial image feature, and generate a feature that better matches the expected text prompt to guide the diffusion model to regenerate the aligned image.

[0084] In the embodiments of the present application, the implicit aligner is obtained by training the following steps:

[0085] Obtain triple training data ; wherein, and respectively represent the preferred image and the non-preferred image obtained by the diffusion model based on the expected text prompt ;

[0086] obtain and corresponding image features: preferred features and non-preferred features ;

[0087] based on the input condition , the preset network is trained with the target , and the formula is as follows:

[0088] (4)

[0089] wherein, h the implicit multi-modal guidance feature obtained by the multi-modal large language model based on the non-preferred image and the text prompt is aligned to the non-preferred image; f θ ( c ) represents the output result of the preset network.

[0090] In the present application, the implicit aligner is trained using the open source human preference dataset commonly used in the field of diffusion model image analysis. Among them, the preferred image and the non-preferred image respectively represent the human preferred image (success image) and the human non-preferred image (failure image) corresponding to the text prompt.

[0091] In the present application, the direction from to represents the image feature alignment improvement track under the condition of c T This track proposes the basic training goal for the implicit aligner of the present application: based on the input condition , the preset network is trained with the target .

[0092] In addition to the basic training goal, the present application also obtains inspiration from the direct preference optimization (DPO) algorithm and the self-play fine-tuning (SPIN) algorithm to further enhance feature alignment. In the present application, the DPO is used to fine-tune the reference aligner f ref (the copy of the preset network f θ pretrained in other fields), encouraging the network output f θ (c)Approaching preferred features while distancing from non-preferred features . SPIN also forces f θ (c) Approaching .

[0093] In the meantime, the present application forces f θ (c) More advantages than . Combining the optimization of dynamic programming algorithm (DPO) and SPIN algorithm, the training objective of the present application is derived as follows:

[0094] (5)

[0095] where is a monotonically decreasing convex function, which is usually implemented as ( x ):= log(1 + e −x ). In the present application, p θ and p ref are defined as Gaussian distributions with mean and constant variance σ predicted by f θ and f ref i.e. p(c I |c） = N(c I ； f (c), σ) By some transformations, the objective in equation (5) can be simplified as:

[0096] (6)

[0097] In this case, the final training objective of the preset model in the present application is L base and L pref combined with the ratio parameter λ:

[0098] (7)

[0099] The DPO algorithm and SPIN algorithm in related researches need a fixed reference network f ref whose weights are frozen. In the present application, the implicit aligner can be trained from scratch, and the reference network can be updated at runtime. Specifically, can be first randomly initialized f ref , and then in​f θ Superior f ref Iteratively f θ Copy to f ref In practice, for the k-th consecutive iteration, in f θ ( c )Compare f ref ( c Closer to preferred features In this case, perform parameter replacement, and f θ Network parameters are iteratively copied to f ref Subsequently, the objective function L is determined as the training target in the iterative update process.

[0100] S106, Based on the aligned image features, an aligned image is generated using the diffusion model.

[0101] In this application, the alignment image features are re-input into the diffusion model, so that the diffusion model can obtain an alignment image that matches the expected text prompt.

[0102] Figure 3 An exemplary flowchart of the diffusion generation method based on implicit multimodal guided alignment provided in this application is shown. Figure 3 As shown, the expected text prompt is: "One morning I chased an elephant in my pajamas." The initial image generated by the diffusion model is shown in the figure. It can be seen that the initial image does not match the expected text prompt. In this application, based on the expected text prompt, an MLLM instruction, i.e., an alignment instruction, is constructed. This MLLM instruction is input into a multimodal large language model to obtain multimodal guidance features. In this application, the initial image features of the initial image can also be obtained based on the image prompt encoder. The initial image features and the multimodal guidance features are input together into an implicit aligner to obtain aligned image features. The aligned image features are then input into the diffusion model to obtain a new aligned image, as shown in the figure. It can be seen that the aligned image obtained by the method provided in this application can match the expected text prompt well.

[0103] In this embodiment of the application, the aligned image can also be further aligned. Specifically, the method further includes:

[0104] A secondary alignment instruction is constructed based on the expected text prompt and the alignment image;

[0105] inputting the secondary alignment instruction into the multimodal large language model to obtain an implicit multimodal secondary guide feature for aligning the alignment image;

[0106] processing the alignment image based on an image prompt encoder to obtain an alignment image feature;

[0107] inputting the alignment image feature and the implicit multimodal secondary guide feature into an implicit aligner to obtain a secondary alignment image feature;

[0108] generating a secondary alignment image based on the secondary alignment image feature through a diffusion model.

[0109] In the present application, the secondary alignment process is similar to the above steps S101-S106, which will not be repeated here.

[0110] In the present application, IMG can be used as an iterative framework to generate images that match the expected text prompts through multiple rounds of alignment. As shown in FIG. 1, Figure 4 it shows an exemplary schematic diagram of secondary alignment, which can be seen that the details that are not completely matched in the first round of IMG, such as the small radish-shaped figure wearing a one-piece suit and the action of diving, the second round of IMG will further improve these details.

[0111] In the present application, the image can also be aligned multiple times based on IMG until the generated alignment image meets the actual demand.

[0112] In the present application, a comparative example is also provided to verify the beneficial effects of the method provided in the present application. The comparative example compares the image generation effects of IMG and SDXL, as well as IMG and SDXL-DPO which obtains the best performance through diffusion fine-tuning. For MLLM, the method of the present application fine-tunes LLaVA 1.5-13b on Instruct-Pix2Pix dataset and extracts the features of the last hidden layer as guide features. The present application uses IP-Adapter trained on SDXL and FLUX to realize image prompts and extract image features, Figure 5 shows the image generation effect schematic diagram of IMG and the comparative model in the present comparative example.

[0113] In this application, the implicit aligner of IMG is trained for 100,000 iterations on the Pick-a-Pic training set (same as the training set of SDXL-DPO). The training data contains 851,000 pairs of preferred and non-preferred images under specific prompts. In formula 7, the scale parameter λ is set to 1. The reference model update step k is set to 10. This application determines the optimal hyperparameters by the average Pick score of the generated images under 500 Pick-a-Pic test set prompts.

[0114] In this application, the image generation effects of several models are shown in FIG. 5, which shows that a) the IMG provided in this application has effectiveness in solving various alignment problems from different aspects, such as conceptual understanding (e.g., penguin with robot body and fox playing musical instrument), aesthetic quality (e.g., well-built city on Mars), object addition (e.g., comet in the sky), and object correction (e.g., rectangular mirror), and b) the IMG proposed in this application can be operated with SDXL-DPO without additional training, which is flexible. The last row also shows that SDXL and SDXL-DPO are difficult to deal with, while IMG shows obvious improvement in alignment in both models.

[0115] In summary, in this application, based on a multi-modal large language model (MLLM), an image prompt encoder, and an implicit aligner, the images generated by the diffusion model are effectively aligned without fine-tuning the original diffusion model weights, resulting in high-quality images that match the expected text prompts. Moreover, the IMG proposed in this application can be used as a flexible plug-and-play adapter compatible with image alignment methods based on fine-tuning in related research.

[0116] Based on the same inventive concept, the application also provides a diffusion generation device based on implicit multi-modal guided alignment, which comprises:

[0117] An image generation module for generating an initial image based on an expected text prompt through a diffusion model;

[0118] An instruction construction module for constructing an alignment instruction based on the expected text prompt and the initial image, the alignment instruction instructing a multi-modal large language model to generate an editing instruction that matches the initial image with the expected text prompt based on the initial image and the expected text prompt;

[0119] A guide feature determination module for inputting the alignment instruction into a multi-modal large language model to obtain an implicit multi-modal guide feature for aligning the initial image;

[0120] The feature extraction module is used to process the initial image based on the image cue encoder to obtain the initial image features;

[0121] The feature alignment module is used to input the initial image features and the implicit multimodal guided features into the implicit aligner to obtain aligned image features;

[0122] An alignment module is used to generate an aligned image based on the features of the aligned image using the diffusion model.

[0123] Optionally, the training steps of the multimodal large language model include:

[0124] The system acquires an original image, an edited image, text prompts corresponding to the original image and the edited image, and obtains editing instructions for the edited image based on the original image.

[0125] The original image, the text prompt corresponding to the edited image, and the editing instruction are used as a training triple;

[0126] The multimodal large language model is trained based on multiple training triples. During the training process, the text prompts corresponding to the original image and the edited image are used as input information, and the editing instructions are used as supervision information.

[0127] Optionally, the guidance feature determination module is used to:

[0128] The alignment instructions are input into the multimodal large language model, and the features of the last hidden layer of the multimodal large language model are extracted as implicit multimodal guiding features.

[0129] Optionally, the implicit aligner is trained using the following steps:

[0130] Obtain triplet training data ;in, and These represent diffusion models based on expected text prompts. The preferred and non-preferred images obtained;

[0131] Get and The corresponding image features: preferred features Non-preferred features ;

[0132] Based on input conditions ,by For the target of the preset network For the first training exercise, the formula is as follows:

[0133] ;

[0134] wherein, h the multimodal large language model obtains implicit multimodal guided features for aligning the non-preferred image based on the non-preferred image and the text prompt; f θ ( c ) represents the output result of the preset network.

[0135] Optionally, the training step of the implicit aligner further comprises:

[0136] obtaining a reference aligner f ref ; the reference aligner is obtained by randomly initializing network parameters of a preset network;

[0137] combined with a direct preference optimization (DPO) algorithm and a self-play fine-tuning (SPIN) algorithm, based on the reference aligner f ref and the preset network f θ perform a second training:

[0138] ;

[0139] ;

[0140] In the training process, when f θ ( c ) is closer to the preferred feature f ref ( c ) than , the network parameters of f θ are iteratively copied to f ref ; wherein, f ref ( c ) represents the output result of the reference aligner, is a monotonically decreasing convex function, ( x )= log(1 + e −x ); λ represents a ratio parameter.

[0141] Optionally, the instruction construction module is further configured to construct a secondary alignment instruction based on the expected text prompt and the aligned image;

[0142] The guiding feature determination module is further configured to input the secondary alignment instruction into a multi-modal large language model to obtain implicit multi-modal secondary guiding features for aligning the alignment image.

[0143] The feature extraction module is further configured to process the alignment image based on an image prompt encoder to obtain alignment image features.

[0144] The feature alignment module is further configured to input the alignment image features and the implicit multi-modal secondary guiding features into an implicit aligner to obtain secondary alignment image features.

[0145] The alignment module is further configured to generate a secondary alignment image based on the secondary alignment image features through a diffusion model.

[0146] Based on the same inventive concept, the present application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the diffusion generation method based on implicit multi-modal guiding alignment according to any one of the above embodiments when executed.

[0147] Based on the same inventive concept, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements the steps of the diffusion generation method based on implicit multi-modal guiding alignment according to any one of the above embodiments when executed by a processor.

[0148] Based on the same inventive concept, the present application provides a computer program product comprising computer programs / instructions, wherein the computer programs / instructions implement the steps of the diffusion generation method based on implicit multi-modal guiding alignment according to any one of the above embodiments when executed by a processor.

[0149] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0150] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device, or computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0151] The computer program instructions can also be loaded onto a computer or other programmable terminal device to cause a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process such that the instructions which execute on the computer or other programmable terminal device provide steps for implementing the functions specified in the flowchart or flows and / or block or blocks in the block diagram. Figure 1 one or more flows and / or blocks. Figure 1 one or more blocks.

[0152] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable terminal device to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flows and / or block or blocks in the block diagram. Figure 1 one or more flows and / or blocks. Figure 1 one or more blocks.

[0153] These computer program instructions can also be loaded onto a computer or other programmable terminal device to cause a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process such that the instructions which execute on the computer or other programmable terminal device provide steps for implementing the functions specified in the flowchart or flows and / or block or blocks in the block diagram. Figure 1 one or more flows and / or blocks. Figure 1 one or more blocks.

[0154] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure. Therefore, the appended claims are intended to cover all such additional variations and modifications as fall within the scope of this application.

[0155] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying that there is any such actual relationship or order between these entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof are intended to cover non-exclusive inclusions, so that a process, method, article, or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or terminal device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0156] The above describes in detail the diffusion generation method based on implicit multi-modal guidance alignment provided by the present application, and the principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application range; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A diffusion generation method based on implicit multi-modal guidance alignment, characterized in that, The method comprises: generating an initial image based on an expected text prompt through a diffusion model; constructing alignment instructions based on the expected text prompt and the initial image, the alignment instructions instructing a multimodal large language model to generate editing instructions that make the initial image match the expected text prompt based on the initial image and the expected text prompt; inputting the alignment instructions into the multimodal large language model to obtain implicit multimodal guidance features for aligning the initial image; processing the initial image based on an image prompt encoder to obtain initial image features; inputting the initial image features and the implicit multimodal guidance features into an implicit aligner to obtain aligned image features; generating an aligned image based on the aligned image features through the diffusion model.

2. The implicit multi-modal guidance alignment based diffusion generation method of claim 1, wherein, The training steps of the multimodal large language model comprise: obtaining an original image, an edited image, a text prompt corresponding to the original image, a text prompt corresponding to the edited image, and editing instructions for obtaining the edited image based on the original image; taking the original image, the text prompt corresponding to the edited image, and the editing instructions as a training triple; training the multimodal large language model based on multiple training triples, and in the training process, taking the original image and the text prompt corresponding to the edited image as input information, and taking the editing instructions as supervision information.

3. The implicit multi-modal guidance alignment based diffusion generation method of claim 1, wherein, Inputting the alignment instructions into the multimodal large language model to obtain implicit multimodal guidance features for aligning the initial image comprises: inputting the alignment instructions into the multimodal large language model, and extracting features of a last hidden layer of the multimodal large language model as the implicit multimodal guidance features.

4. The implicit multi-modal guidance alignment based diffusion generation method of claim 1, wherein, The implicit aligner is trained by the following steps: Obtaining triple training data ; wherein, and respectively represent the preferred image and the non-preferred image obtained by the diffusion model based on the expected text prompt ; acquire and corresponding image features: preferred features and non-preferred features ; Based on input conditions To Target the preset network First training, formula as follows: ； wherein, h represents that the multimodal large language model obtains implicit multimodal guided features for aligning the non-preferred image based on the non-preferred image and the text prompt; f θ c represents an output result of the preset network.​ 5. The implicit multi-modal guidance alignment based diffusion generation method of claim 4, wherein, The training steps of the implicit aligner further comprise: Acquiring reference aligner f ref The reference aligner is obtained by randomly initializing network parameters of a preset network. In combination with a direct preference optimization (DPO) algorithm and a self-play-in-iteration (SPIN) algorithm, based on the reference aligner f ref and the preset network f θ performing second training: ； ； During the training process, f θ ( c )Compare f ref ( c Closer to preferred features In the case of, f θ Network parameters are iteratively copied to f ref ;in, f ref ( c () indicates the output of the reference aligner. It is a monotonically decreasing convex function. ( x ) = log(1 + e) −x ); λ represents the ratio parameter.

6. The implicit multi-modal guidance alignment based diffusion generation method according to any one of claims 1-5, characterized in that, The method further comprises: constructing secondary alignment instructions based on the expected text prompt and the aligned image; inputting the secondary alignment instructions into the multimodal large language model to obtain implicit multimodal secondary guidance features for aligning the aligned image; processing the aligned image based on an image prompt encoder to obtain aligned image features; inputting the aligned image features and the implicit multimodal secondary guidance features into an implicit aligner to obtain secondary aligned image features; generating a secondary aligned image through a diffusion model based on the secondary aligned image features.

7. A diffusion generation device based on implicit multi-modal guidance alignment, characterized in that, The diffusion generation device based on implicit multimodal guidance alignment comprises: an image generation module configured to generate an initial image based on an expected text prompt through a diffusion model; an instruction construction module configured to construct alignment instructions based on the expected text prompt and the initial image, the alignment instructions instructing a multimodal large language model to generate editing instructions that make the initial image match the expected text prompt based on the initial image and the expected text prompt; a guidance feature determination module configured to input the alignment instructions into the multimodal large language model to obtain implicit multimodal guidance features for aligning the initial image; a feature extraction module configured to process the initial image based on an image prompt encoder to obtain initial image features; a feature alignment module, configured to input the initial image features and the implicit multi-modal guidance features into an implicit aligner to obtain aligned image features; an alignment module, configured to generate an aligned image by the diffusion model based on the aligned image features.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the implicit multi-modal guidance alignment-based diffusion generation method in any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the implicit multi-modal guidance alignment-based diffusion generation method in any one of claims 1-6.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps in the implicit multi-modal guidance alignment-based diffusion generation method in any one of claims 1-6.

Images