Diffusion generation method based on implicit multi-mode guide alignment

Abstract

The invention provides a diffusion generation method based on implicit multi-mode guide alignment, and relates to the technical field of image processing. In the application, for an image generated by diffusion and an expected text prompt, an implicit multi-modal guide feature for aligning an initial image generated by a diffusion model is generated through a multi-modal large language model; through an implicit aligner, features, which are not matched with the expected prompt text, in the initial image generated by the diffusion model are aligned by operating the hidden features; according to the implicit multi-mode guide alignment-based diffusion generation method provided by the invention, the mismatching between the generated image and the expected text prompt is effectively reduced under the condition that the output quality is not sacrificed.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a diffusion generation method based on implicit multimodal guided alignment. Background Art

[0002] Recently, diffusion models have emerged as powerful tools for text-to-image (T2I) generation, capable of generating diverse and realistic images. However, diffusion models currently still require human-based inspection to detect input-output misalignments. More specifically, diffusion models occasionally ignore or misinterpret certain contents of the input textual cues, resulting in poor visual results with misunderstandings. In related research, even the latest state-of-the-art diffusion model FLUX needs further correction in terms of conceptual awareness and aesthetic quality.

[0003] Due to the above challenges, exploring methods to improve the misalignment between prompt text and generated images has become an emerging and critical research area. Early work in this field mainly adopted preference-based weight fine-tuning. For example, related studies proposed fine-tuning the diffusion model based on text prompts and corresponding high-quality images. Related studies also applied reinforcement learning learned from human feedback (RLHF) to explore reward algorithms and datasets. However, these methods either focus on only a few prompts and feedback, or require a large amount of data to fine-tune the model weights. Summary of the invention

[0004] The present application provides a diffusion generation method and product based on implicit multimodal guided alignment to at least partially solve the above problems.

[0005] In a first aspect, the present application provides a diffusion generation method based on implicit multimodal guided alignment, the method comprising: Based on the expected textual cues, an initial image is generated through a diffusion model; constructing an alignment instruction based on the expected text prompt and the initial image, the alignment instruction instructing the multimodal large language model to generate an editing instruction that matches the initial image with the expected text prompt based on the initial image and the expected text prompt; Inputting the alignment instruction into a multimodal large language model to obtain an implicit multimodal guided feature for aligning the initial image; Processing the initial image based on an image hint encoder to obtain initial image features; Inputting the initial image features and the implicit multimodal guided features into an implicit aligner to obtain aligned image features; Based on the aligned image features, an aligned image is generated by the diffusion model.

[0006] Optionally, the training step of the multimodal large language model includes: Acquire an original image, an edited image, a text prompt corresponding to the original image and a text prompt corresponding to the edited image, and obtain an editing instruction of the edited image based on the original image; The original image, the text prompt corresponding to the edited image and the edit instruction are used as a training triplet; The multimodal large language model is trained based on multiple training triplets. During the training process, the text prompts corresponding to the original image and the edited image are used as input information, and the editing instructions are used as supervision information.

[0007] Optionally, inputting the alignment instruction into a multimodal large language model to obtain an implicit multimodal guided feature for aligning the initial image comprises: The alignment instruction is input into a multimodal large language model, and features of a last hidden layer of the multimodal large language model are extracted as implicit multimodal guided features.

[0008] Optionally, the implicit aligner is trained by following steps: Get triplet training data ;in, and Respectively represent the diffusion model based on the expected text prompt The resulting preferred and non-preferred images; Get and Corresponding image features: preferred features and non-preferred features ; Based on input conditions ,by Preset network for target pair For the first training, the formula is as follows: ; in, h Representing a multimodal large language model based on the non-preferred image and the text prompt, obtaining an implicit multimodal guided feature for aligning the non-preferred image; f θ ( c ) represents the output result of the preset network.

[0009] Optionally, the implicit aligner training step further includes: Get Reference Aligner f ref ; The reference aligner is obtained by randomly initializing the network parameters of the preset network; Combining the direct preference optimization DPO algorithm and the self-play fine-tuning SPIN algorithm, based on the reference aligner f ref and the preset network f θ Conduct the second training: ; ; During the training process, f θ ( c )Compare f ref ( c ) is closer to the preferred characteristics In the case of f θ The network parameters are iteratively copied to f ref ;in, f ref ( c ) represents the output of the reference aligner, is a monotonically decreasing convex function, ( x ) = log(1 + e −x ); λ represents the ratio parameter.

[0010] Optionally, the method further comprises: constructing a secondary alignment instruction based on the expected text prompt and the alignment image; Inputting the secondary alignment instruction into a multimodal large language model to obtain an implicit multimodal secondary guided feature for aligning the aligned images; Processing the aligned image based on an image hint encoder to obtain aligned image features; Inputting the aligned image features and the implicit multimodal secondary guided features into an implicit aligner to obtain secondary aligned image features; Based on the quadratic alignment image features, a quadratic alignment image is generated through a diffusion model.

[0011] A second aspect of the present application provides a diffusion generation device based on implicit multi-modal guided alignment, the diffusion generation device based on implicit multi-modal guided alignment comprising: An image generation module, for generating an initial image based on the expected text prompt through a diffusion model; An instruction construction module is used to construct an alignment instruction based on the expected text prompt and the initial image, wherein the alignment instruction instructs the multimodal large language model to generate an editing instruction that matches the initial image with the expected text prompt based on the initial image and the expected text prompt; A guide feature determination module, used for inputting the alignment instruction into a multimodal large language model to obtain an implicit multimodal guide feature for aligning the initial image; A feature extraction module, used for processing the initial image based on an image prompt encoder to obtain initial image features; A feature alignment module, used for inputting the initial image features and the implicit multimodal guided features into an implicit aligner to obtain aligned image features; An alignment module is used to generate an alignment image through the diffusion model based on the alignment image features.

[0012] The third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the diffusion generation method based on implicit multimodal guided alignment as described in the first aspect of the present invention is implemented.

[0013] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the diffusion generation method based on implicit multimodal guided alignment as described in the first aspect of the present invention.

[0014] A fifth aspect of the present application provides a computer program product, including a computer program / instruction, which is used by a processor to implement the steps in the diffusion generation method based on implicit multi-modal guided alignment as described in the first aspect of the present invention.

[0015] In the image alignment method of implicit multimodal guided features and diffusion model provided in the present application, a multimodal large language model (MLLM) capable of visual question answering is first used as a detector of misalignment between the initial image and the expected text prompt. The initial image is sampled from the diffusion model based on the expected text prompt, and the MLLM analyzes the initial image and identifies potential misalignments. In the present application, considering that identifying misalignment is not the default task of the MLLM; therefore, the MLLM is customized in the present application by training the MLLM with the original image, the expected text prompt, and the corresponding editing instructions.

[0016] In this application, in order to align potential misalignments, an image prompt encoder is used to convert the initial image into image features. Similar to the image variation task, these features enable the model to reconstruct pictures similar to the initial image based on the features. The goal of this application is to align misalignments rather than reconstruct the input. Therefore, in this application, an implicit aligner is introduced, which is a diffusion model adapter that aims to convert initial image features into aligned image features that better match the expected text prompts. In this application, the implicit aligner is trained using preferred images and non-preferred images of human preferences obtained based on text prompts. And there is a training goal that combines a direct preference optimization algorithm with a self-play fine-tuning algorithm.

[0017] In summary, in this application, based on the multimodal large language model (MLLM), image prompt encoder and implicit aligner, the images generated by the diffusion model are effectively aligned without fine-tuning the original diffusion model weights to obtain high-quality images that match the expected text prompts. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a flowchart of the steps of the diffusion generation method based on implicit multimodal guided alignment provided by the present application; Figure 2 is an exemplary schematic diagram of the training process of a multimodal large language model in the diffusion generation method based on implicit multimodal guided alignment provided by the present application; Figure 3 is an exemplary flow chart of a diffusion generation method based on implicit multimodal guided alignment provided by the present application; Figure 4 is an exemplary schematic diagram of secondary alignment of the diffusion generation method based on implicit multimodal guided alignment provided by the present application; Figure 5 It is a schematic diagram of the image generation effect of the comparison model in the comparison embodiment of the diffusion generation method based on implicit multimodal guided alignment provided in the present application. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] In recent years, diffusion models have become the main method for text-to-image generation. However, ensuring accurate alignment between the generated image and the input prompt remains challenging. With the significant progress of multimodal large language models (MLLMs) in visual understanding, this application proposes a novel network framework: implicit multimodal guidance (IMG), which is used to enhance the compliance and aesthetic quality of the image generated by the diffusion model to the expected prompt text, and realize the diffusion generation method based on implicit multimodal guidance alignment proposed in this application. Specifically, in this application, for the diffusion-generated image and the expected text prompt, IMG performs the following operations: a) implicit multimodal guidance features are generated by the multimodal large language model to align the initial image generated by the diffusion model; b) through the implicit aligner, the features in the initial image generated by the diffusion model that do not match the expected prompt text are aligned by operating the hidden features; c) based on the alignment target (i.e., the iteratively updated preference target) is formulated as a trainable target. This application also conducts extensive qualitative and quantitative evaluations through comparative examples, in which the diffusion generation method based on implicit multimodal guidance alignment proposed in this application effectively reduces the mismatch between the generated image and the expected text prompt without sacrificing output quality.

[0022] Specifically, the present application provides a flowchart of the steps of a diffusion generation method based on implicit multimodal guided alignment, such as Figure 1 As shown, specifically, the diffusion generation method based on implicit multimodal guided alignment includes the following steps: S101, based on the expected text prompt, an initial image is generated through a diffusion model.

[0023] The diffusion model is an image generation method that involves a forward process and a reverse process. The forward process is often called a gradual process. After t steps, the data points x 0 is converted to Gaussian noise x T , for example, any step in the asymptotic process yields x t The classic formula for is defined as: (1) in t ~ u (0, T ), ~ N (0, I ) is random Gaussian noise, α t and σ t yes t Predefined functions of .

[0024] The reverse process iteratively converts xT Convert to x 0, by training a well-trained deep neural network θ Evaluating intermediate results x t .

[0025] Considering the text-to-image generation task, given an image x 0 and text conditions c T , θ The training objective can be expressed as: (2) Currently, diffusion models with image cues are gradually maturing. Such diffusion models usually reconstruct images similar to the input image through a frozen T2I model to which a newly trained adapter is attached that accepts visual input. Therefore, the training objective is extended to include an additional image condition c I , it is usually x 0: Encoding features: (3) During the inference process, the user input c I as well as c T (optionally) to create or enhance x Content consistent variant of 0.

[0026] S102, constructing an alignment instruction based on the expected text prompt and the initial image, wherein the alignment instruction instructs the multimodal large language model to generate an editing instruction that matches the initial image with the expected text prompt based on the initial image and the expected text prompt.

[0027] Diffusion models sometimes misunderstand or ignore parts of the prompt when generating images, resulting in the generated image being inconsistent with the expected text prompt. To address this issue, the present application further proposes to identify these inconsistencies through multimodal large language models (MLLMs) capable of visual question answering. In IMG, the present application solves this problem by introducing a customized multimodal large language model, which is trained using instruction-based image data, enabling the multimodal large language model to analyze and respond to potential misaligned features.

[0028] Specifically, in this application, the training steps of the multimodal large language model include: S1, acquiring an original image, an edited image, a text prompt corresponding to the original image and a text prompt corresponding to the edited image, and obtaining an editing instruction of the edited image based on the original image; S2, taking the original image, the text prompt corresponding to the edited image and the edit instruction as a training triplet; S3, training a multimodal large language model based on multiple training triplets. During the training process, the text prompts corresponding to the original image and the edited image are used as input information, and the editing instructions are used as supervision information.

[0029] Specifically, the image data based on the instruction includes the original image I 0. Edit the image I 1. And the text hints for the two images T 0 and T 1, and an editing instruction for obtaining an edited image based on the original image T E This application is taken out by I 0. T 1 and T E These instruction-based image data are utilized as training triplets for this application.

[0030] Will I 0 and T 1 is provided as input to a multimodal large language model (MLLM), requiring the model to pass such as “how to make < the original image I 0>Match expected text prompt T 1: <Edit command T E >?" to describe the alignment, and then T E Supervised model output. Figure 2 As shown, it shows an exemplary schematic diagram of the training process of a multimodal large language model.

[0031] S103: Input the alignment instruction into a multimodal large language model to obtain an implicit multimodal guided feature for aligning the initial image.

[0032] In an embodiment of the present application, the alignment instruction may be input into a large multimodal language model, and features of the last hidden layer of the large multimodal language model may be extracted as implicit multimodal guided features.

[0033] S104, processing the initial image based on the image hint encoder to obtain initial image features.

[0034] In this application, the image prompt encoder can adopt any mature method in related research. For example, the IP-Adapter trained on SDXL and FLUX can be used to implement image prompts and extract image features.

[0035] S105, inputting the initial image features and the implicit multimodal guided features into an implicit aligner to obtain aligned image features.

[0036] In IMG, the next step requires aligning the generated images by removing the inconsistencies between the images and the expected text cues detected by the multimodal large language model.

[0037] The implicit aligner of this application is an adaptive network applied to the original diffusion model, which helps to align the initial image to an image that matches the expected text prompt. The principle behind the design of this application is to align implicit features during the diffusion process instead of explicitly correcting the initial image.

[0038] In this application, the implicit aligner consists of a series of cross-attention layers, which takes the hidden state of the multi-language model (MLLM), i.e., the multimodal guided features, as one of the inputs, and the other input of the implicit aligner is the initial image features extracted by the image prompt encoder. The multimodal guided features can promote the alignment of the initial image features, produce features that better match the expected text prompts to guide the diffusion model to regenerate the aligned image.

[0039] In the embodiment of the present application, the implicit aligner is trained by following the steps below: Get triplet training data ;in, and Respectively represent the diffusion model based on the expected text prompt The resulting preferred and non-preferred images; Get and Corresponding image features: preferred features and non-preferred features ; Based on input conditions ,by Preset network for target pair For the first training, the formula is as follows: (4) in, h Representing a multimodal large language model based on the non-preferred image and the text prompt, obtaining an implicit multimodal guided feature for aligning the non-preferred image; f θ( c ) represents the output result of the preset network.

[0040] In this application, an implicit aligner is trained using an open source human preference dataset commonly used in the field of diffusion model image analysis, where the preferred image and the non-preferred image represent the human preferred image (successful image) and human non-preferred image (failed image) corresponding to the text prompt, respectively.

[0041] In this application, from arrive The direction is indicated by c T Improved trajectory for conditional image feature alignment. This trajectory proposes a basic training goal for the implicit aligner of this application: based on the input condition ,by Preset network for target pair Conduct training.

[0042] In addition to the basic training objectives, this application also draws inspiration from the Direct Preference Optimization (DPO) algorithm and the Self-Play Fine-Tuning (SPIN) algorithm to further enhance feature alignment. f ref (Pre-trained networks in other fields f θ copies of ) to fine-tune and encourage the network to output f θ (c) Close to preferred features , while with non-preferred features Keep your distance. SPIN also forces f θ (c) near .

[0043] At the same time, this application forces f θ (c) Compare Combining the optimization of the dynamic programming algorithm (DPO) and the SPIN algorithm, the training objectives of this application are derived as follows: (5) in, is a monotonically decreasing convex function, usually implemented as ( x ):= log(1 + e −x ). In this application, p θ and p ref is defined as havingf θ and f ref The predicted Gaussian distribution with mean and constant variance σ, That is, p(c I |c） = N(c I ； f (c), σ) Through some transformations, the objective in formula (5) can be simplified to: (6) In this case, in this application, the final training target of the preset model is L base and L pref Combination with ratio parameter λ: (7) The DPO algorithm and SPIN algorithm in related research require a fixed reference network f ref , whose weights are frozen, in this application, the implicit aligner can be trained from scratch, and the reference network can be updated at runtime. Specifically, we can first randomly initialize f ref , then f θ Better than f ref Iteratively f θ Copy to f ref In practice, for the kth consecutive iteration, f θ ( c )Compare f ref ( c ) is closer to the preferred characteristics In the case of f θ The network parameters are iteratively copied to f ref . Thereafter, the objective function L is determined as the training target in the iterative updating process.

[0044] S106: Generate an aligned image through the diffusion model based on the aligned image features.

[0045] In the present application, the aligned image features are re-input into the diffusion model, and the diffusion model can obtain an aligned image that matches the expected text prompt.

[0046] Figure 3FIG. 1 shows an exemplary flow chart of a diffusion generation method based on implicit multimodal guided alignment provided by the present application. Figure 3 As shown, the expected text prompt is: "One morning I was chasing an elephant in my pajamas", and the initial image generated by the diffusion model is shown in the figure. It can be seen that the initial image does not match the expected text prompt. In this application, based on the expected text prompt, an MLLM instruction, that is, an alignment instruction, is constructed. The MLLM instruction is input into a multimodal large language model to obtain a multimodal guided feature. In this application, the initial image features of the initial image can also be obtained based on the image prompt encoder. The initial image features and the multimodal guided features are input into the implicit aligner together to obtain the aligned image features. The aligned image features are input into the diffusion model to obtain the aligned image again. The aligned image is shown in the figure. It can be seen that the aligned image obtained by the method provided in this application can well match the expected text prompt.

[0047] In the embodiment of the present application, the aligned images may also be aligned again. Specifically, the method further includes: constructing a secondary alignment instruction based on the expected text prompt and the alignment image; Inputting the secondary alignment instruction into a multimodal large language model to obtain an implicit multimodal secondary guided feature for aligning the aligned images; Processing the aligned image based on an image hint encoder to obtain aligned image features; Inputting the aligned image features and the implicit multimodal secondary guided features into an implicit aligner to obtain secondary aligned image features; Based on the quadratic alignment image features, a quadratic alignment image is generated through a diffusion model.

[0048] In the present application, the secondary alignment process is similar to the above steps S101 to S106, and will not be described in detail in this application.

[0049] In this application, IMG is used as an iterative framework to enhance the matching degree between the generated image and the expected text prompt through multiple rounds of alignment. Figure 4 As shown, it shows an exemplary schematic diagram of the secondary alignment. It can be seen that the details that were not fully matched in the first round of IMG, such as the human shape of the little carrot in a jumpsuit and the diving action, will be further improved by the second round of IMG.

[0050] In this application, the image can also be aligned multiple times based on IMG until the generated aligned image meets the actual needs.

[0051] In this application, a comparative example is also provided to verify the beneficial effects of the method provided by this application. The comparative example compares the image generation effects of IMG and SDXL, and IMG and SDXL-DPO with the best performance obtained by diffusion fine-tuning. For MLLM, the method of this application fine-tunes LLaVA 1.5-13b on the Instruct-Pix2Pix dataset and extracts the features of the last hidden layer as guiding features. This application uses IP-Adapter trained on SDXL and FLUX to implement image prompts and extract image features. Figure 5 A schematic diagram of the image generation effect of IMG and the comparison model in this comparative embodiment is shown.

[0052] In this application, IMG's implicit aligner is trained for 100,000 iterations on the Pick-a-Pic training set (the same training set as SDXL-DPO). The training data contains 851,000 pairs of preferred and non-preferred images under specific prompts. In Equation 7, the scale parameter λ is set to 1. The reference model update step k is set to 10. This application determines the optimal hyperparameters by averaging the Pick scores of images generated under 500 Pick-a-Pic test set prompts.

[0053] In this application, the image generation effects of several models are shown in Figure 5, which shows that a) the IMG provided by this application is effective in solving various alignment problems from different aspects, such as concept understanding (e.g., penguins with robotic bodies and foxes playing musical instruments), aesthetic quality (e.g., carefully built cities on Mars), object addition (e.g., comets in the sky), and object correction (e.g., rectangular mirrors), and b) the IMG proposed in this application can operate in collaboration with SDXL-DPO without additional training, which is flexible. The last row also shows the situation that both SDXL and SDXL-DPO have difficulty in dealing with, while IMG shows obvious improvements in alignment in both models.

[0054] In summary, in this application, based on the multimodal large language model (MLLM), image prompt encoder and implicit aligner, the images generated by the diffusion model are effectively aligned without fine-tuning the original diffusion model weights, and high-quality images matching the expected text prompts are obtained. In addition, the IMG proposed in this application can be used as a flexible plug-and-play adapter, which is compatible with the fine-tuning-based image alignment method in related research.

[0055] Based on the same inventive concept, the present application also provides a diffusion generation device based on implicit multi-modal guided alignment, and the diffusion generation device based on implicit multi-modal guided alignment includes: An image generation module, for generating an initial image based on the expected text prompt through a diffusion model; An instruction construction module is used to construct an alignment instruction based on the expected text prompt and the initial image, wherein the alignment instruction instructs the multimodal large language model to generate an editing instruction that matches the initial image with the expected text prompt based on the initial image and the expected text prompt; A guide feature determination module, used for inputting the alignment instruction into a multimodal large language model to obtain an implicit multimodal guide feature for aligning the initial image; A feature extraction module, used for processing the initial image based on an image prompt encoder to obtain initial image features; A feature alignment module, used for inputting the initial image features and the implicit multimodal guided features into an implicit aligner to obtain aligned image features; An alignment module is used to generate an alignment image through the diffusion model based on the alignment image features.

[0056] Optionally, the training step of the multimodal large language model includes: Acquire an original image, an edited image, a text prompt corresponding to the original image and a text prompt corresponding to the edited image, and obtain an editing instruction of the edited image based on the original image; The original image, the text prompt corresponding to the edited image and the edit instruction are used as a training triplet; The multimodal large language model is trained based on multiple training triplets. During the training process, the text prompts corresponding to the original image and the edited image are used as input information, and the editing instructions are used as supervision information.

[0057] Optionally, the guidance feature determination module is used to: The alignment instruction is input into a multimodal large language model, and features of a last hidden layer of the multimodal large language model are extracted as implicit multimodal guided features.

[0058] Optionally, the implicit aligner is trained by following steps: Get triplet training data ;in, and Respectively represent the diffusion model based on the expected text prompt The resulting preferred and non-preferred images; Get and Corresponding image features: preferred features and non-preferred features ; Based on input conditions ,by Preset network for target pair For the first training, the formula is as follows: ; in, h Representing a multimodal large language model based on the non-preferred image and the text prompt, obtaining an implicit multimodal guided feature for aligning the non-preferred image; f θ ( c ) represents the output result of the preset network.

[0059] Optionally, the implicit aligner training step further includes: Get Reference Aligner f ref ; The reference aligner is obtained by randomly initializing the network parameters of the preset network; Combining the direct preference optimization DPO algorithm and the self-play fine-tuning SPIN algorithm, based on the reference aligner f ref and the preset network f θ Conduct the second training: ; ; During the training process, f θ ( c )Compare f ref ( c ) is closer to the preferred characteristics In the case of f θ The network parameters are iteratively copied to f ref ;in, f ref ( c ) represents the output of the reference aligner, is a monotonically decreasing convex function, ( x ) = log(1 + e −x ); λ represents the ratio parameter.

[0060] Optionally, the instruction construction module is further used to construct a secondary alignment instruction based on the expected text prompt and the alignment image; The guiding feature determination module is further used to input the secondary alignment instruction into a multimodal large language model to obtain an implicit multimodal secondary guiding feature for aligning the aligned images; The feature extraction module is further used to process the aligned image based on the image prompt encoder to obtain aligned image features; The feature alignment module is further used to input the aligned image features and the implicit multimodal secondary guided features into an implicit aligner to obtain secondary aligned image features; The alignment module is further used to generate a secondary alignment image based on the secondary alignment image features through a diffusion model.

[0061] Based on the same inventive concept, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps in the diffusion generation method based on implicit multimodal guided alignment as described in any of the above embodiments are implemented.

[0062] Based on the same inventive concept, the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the diffusion generation method based on implicit multimodal guided alignment described in any of the above embodiments are implemented.

[0063] Based on the same inventive concept, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the diffusion generation method based on implicit multimodal guided alignment described in any of the above embodiments.

[0064] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0065] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0066] The present application is described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (apparatus), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0067] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0068] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0069] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0070] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0071] The above is a detailed introduction to a diffusion generation method based on implicit multimodal guided alignment provided by the present invention. In this article, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A diffusion generation method based on implicit multimodal guided alignment, characterized in that: The method comprises: Based on the expected textual cues, an initial image is generated through a diffusion model; constructing an alignment instruction based on the expected text prompt and the initial image, the alignment instruction instructing the multimodal large language model to generate an editing instruction that matches the initial image with the expected text prompt based on the initial image and the expected text prompt; Inputting the alignment instruction into a multimodal large language model to obtain an implicit multimodal guided feature for aligning the initial image; Processing the initial image based on an image hint encoder to obtain initial image features; Inputting the initial image features and the implicit multimodal guided features into an implicit aligner to obtain aligned image features; Based on the aligned image features, an aligned image is generated by the diffusion model.

2. The diffusion generation method based on implicit multimodal guided alignment according to claim 1, characterized in that: The training steps of the multimodal large language model include: Acquire an original image, an edited image, a text prompt corresponding to the original image and a text prompt corresponding to the edited image, and obtain an editing instruction of the edited image based on the original image; The original image, the text prompt corresponding to the edited image and the edit instruction are used as a training triplet; The multimodal large language model is trained based on multiple training triplets. During the training process, the text prompts corresponding to the original image and the edited image are used as input information, and the editing instructions are used as supervision information.

3. The diffusion generation method based on implicit multimodal guided alignment according to claim 1, characterized in that: Inputting the alignment instruction into a multimodal large language model to obtain an implicit multimodal guided feature for aligning the initial image includes: The alignment instruction is input into a multimodal large language model, and features of a last hidden layer of the multimodal large language model are extracted as implicit multimodal guided features.

4. The diffusion generation method based on implicit multimodal guided alignment according to claim 1, characterized in that: The implicit aligner is trained using the following steps: Get triplet training data ;in, and Respectively represent the diffusion model based on the expected text prompt The resulting preferred and non-preferred images; Get and Corresponding image features: preferred features and non-preferred features ; Based on input conditions ,by Preset network for target pair For the first training, the formula is as follows: ； in, h Representing a multimodal large language model based on the non-preferred image and the text prompt, obtaining an implicit multimodal guided feature for aligning the non-preferred image; f θ ( c ) represents the output result of the preset network.

5. The diffusion generation method based on implicit multimodal guided alignment according to claim 4 is characterized in that: The training step of the implicit aligner also includes: Get Reference Aligner f ref ; The reference aligner is obtained by randomly initializing the network parameters of the preset network; Combining the direct preference optimization DPO algorithm and the self-play fine-tuning SPIN algorithm, based on the reference aligner f ref and the preset network f θ Conduct the second training: ； ； During the training process, f θ ( c )Compare f ref ( c ) is closer to the preferred characteristics In the case of f θ The network parameters are iteratively copied to f ref ;in, f ref ( c ) represents the output of the reference aligner, is a monotonically decreasing convex function, ( x ) = log (1 + e −x ); λ represents the ratio parameter.

6. The diffusion generation method based on implicit multimodal guided alignment according to any one of claims 1 to 5, characterized in that: The method further comprises: constructing a secondary alignment instruction based on the expected text prompt and the alignment image; Inputting the secondary alignment instruction into a multimodal large language model to obtain an implicit multimodal secondary guided feature for aligning the aligned images; Processing the aligned image based on an image hint encoder to obtain aligned image features; Inputting the aligned image features and the implicit multimodal secondary guided features into an implicit aligner to obtain secondary aligned image features; Based on the quadratic alignment image features, a quadratic alignment image is generated through a diffusion model.

7. A diffusion generation device based on implicit multi-modal guided alignment, characterized in that: The diffusion generation device based on implicit multi-modal guided alignment includes: An image generation module, for generating an initial image based on the expected text prompt through a diffusion model; An instruction construction module is used to construct an alignment instruction based on the expected text prompt and the initial image, wherein the alignment instruction instructs the multimodal large language model to generate an editing instruction that matches the initial image with the expected text prompt based on the initial image and the expected text prompt; A guiding feature determination module, used for inputting the alignment instruction into a multimodal large language model to obtain an implicit multimodal guiding feature for aligning the initial image; A feature extraction module, used for processing the initial image based on an image prompt encoder to obtain initial image features; A feature alignment module, used for inputting the initial image features and the implicit multimodal guided features into an implicit aligner to obtain aligned image features; An alignment module is used to generate an alignment image through the diffusion model based on the alignment image features.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the diffusion generation method based on implicit multimodal guided alignment described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the diffusion generation method based on implicit multimodal guided alignment described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps in the diffusion generation method based on implicit multimodal guided alignment described in any one of claims 1 to 6 are implemented.

Images