A multimodal coding alignment method and apparatus based on a diffusion model

Abstract

This invention relates to a multimodal coding alignment method and apparatus based on a diffusion model, belonging to the field of multimodal coding alignment technology. The method includes: acquiring multimodal data to be semantically encoded; using a pre-trained multimodal coding alignment model to conditionally diffuse the multimodal data to be semantically encoded to generate text encoding; inputting the text encoding into a large-scale model decoder or as RAG encoding for downstream tasks; wherein the generated text encoding is used as the semantic encoding of the multimodal data and is semantically aligned with the text encoding. In this invention, the pre-trained multimodal coding alignment model utilizes the latent multimodal data encoding information learnable by the diffusion model, simultaneously completing multimodal semantic encoding through a diffusion model structure, while using a large-scale language model encoding as the multi-task optimization direction, making the pre-trained multimodal coding alignment model results more applicable to downstream tasks.

Description

Technical Field

[0001] This invention belongs to the field of multimodal coding alignment technology, and particularly relates to a multimodal coding alignment method and apparatus based on a diffusion model. Background Technology

[0002] With the continuous development of generative artificial intelligence technology, more and more large language models are no longer limited to single-modal data understanding. The premise of this data understanding task is that multimodal data has the same semantic encoding. Previous methods usually use contrastive learning to map multimodal information pairs (such as text-image, text-audio) to the same vector space. This method usually requires a large amount of text alignment dataset. Although this method can achieve unified encoding of multimodal data, a single model usually only supports the encoding of one type of multimodal pair. Moreover, when connecting to downstream tasks, it usually introduces information loss, affecting the performance of downstream semantic understanding tasks. Summary of the Invention

[0003] In view of the shortcomings of the prior art, the purpose of the invention is to provide a multimodal coding alignment method and apparatus based on a diffusion model. The pre-trained multimodal coding alignment model utilizes the latent multimodal data encoding information that the diffusion model can learn, and simultaneously completes multimodal semantic encoding through a diffusion model structure. At the same time, it uses large language model encoding as a multi-task optimization direction, making the pre-trained multimodal coding alignment model results more applicable to downstream tasks.

[0004] In a first aspect, the present invention proposes a multimodal coding alignment method based on a diffusion model, comprising:

[0005] Acquire multimodal data to be semantically encoded;

[0006] A pre-trained multimodal coding alignment model is used to conditionally diffuse the multimodal data to be semantically encoded to generate text codes;

[0007] Input the text encoding into a large model decoder or use it as RAG encoding for downstream tasks;

[0008] The generated text encoding is used as a semantic encoding for multimodal data and is semantically aligned with the text encoding.

[0009] Furthermore, in the aforementioned multimodal coding alignment method based on a diffusion model, the training steps of the pre-trained multimodal coding alignment model include:

[0010] Retrieve the text data and the corresponding modality matching data;

[0011] The text data and the corresponding modality matching data are preprocessed.

[0012] The preprocessed text data and the corresponding modality matching data are encoded into a shallow space and used as input to the constructed multimodal joint diffusion model.

[0013] A multimodal data joint noise distribution prediction model is trained using the regression loss of a multimodal joint diffusion model.

[0014] Using a multimodal data joint noise distribution prediction model as a pre-trained model, a pre-trained multimodal coding alignment model is obtained by performing bidirectional conditional diffusion between text data and other modal data, and multi-task fine-tuning learning between other modal data and text data through minimizing the coding distance of a large language model.

[0015] Furthermore, in the aforementioned multimodal coding alignment method based on a diffusion model, the regression loss of the multimodal joint diffusion model is used to train the multimodal data joint noise distribution prediction model, including:

[0016] Add noise to multimodal data pairs;

[0017] A multimodal data joint noise distribution prediction model is obtained by simultaneously training a single diffusion model to predict the joint noise distribution of spliced ​​features of multimodal data.

[0018] Furthermore, in the aforementioned multimodal coding alignment method based on a diffusion model, the overall loss of the pre-trained multimodal coding alignment model is:

[0019]

[0020] in, This represents the overall loss of the pre-trained multimodal coding alignment model, where γ and φ are the weight parameters of the multi-task loss. This represents the diffusion loss from other modal data to text data. This represents the diffusion loss from text data to other modalities. This represents the distance loss between the generation of other modal data and the original encoding of the large language model.

[0021] Furthermore, in the aforementioned multimodal coding alignment method based on a diffusion model,

[0022] The process of adding noise to multimodal data pairs can be represented as follows:

[0023]

[0024] Where, x b Let x represent the original input, t represent the time step t∈T, and x represent the time step. t x t-1 Let q(x) represent the data input at different time steps.t |x t-1 ) indicates to x t-1 Add Gaussian noise to the input, q(x) 1:T |x b ) indicates that the original input x is added step by step. b Gaussian noise, β t Indicates noise interpolation, α t =1-β t N(0,I) represents the standard normal distribution, and T represents the total duration.

[0025] Furthermore, in the aforementioned multimodal coding alignment method based on a diffusion model, the regression loss of the joint diffusion, in the ability to simultaneously train and predict the joint noise distribution of multimodal data splicing features using a single diffusion model, is expressed as:

[0026]

[0027] Among them, t x ,t y These represent the time steps of different modal data. Representing different modal data t x ,t y Encoded input at time step, Represents the joint noise distribution, ∈ x ,∈ y This represents the true noise in data from different modalities. This represents the regression loss due to the joint diffusion of the two modalities x0 and y0.

[0028] Furthermore, in the aforementioned multimodal encoding alignment method based on a diffusion model, the distance between the generated modal data and the original encoding of the large language model is learned by the encoding transformation layer based on the transformer attention method.

[0029] A second aspect of the present invention also proposes a multimodal coding alignment device based on a diffusion model, comprising:

[0030] Acquisition module: Used to acquire multimodal data to be semantically encoded;

[0031] The generation module is used to generate text codes by conditionally diffusing the multimodal data to be semantically encoded using a pre-trained multimodal coding alignment model.

[0032] Input module: used to input the text encoding into the large model decoder or as RAG encoding for downstream tasks;

[0033] The generated text encoding is used as a semantic encoding for multimodal data and is semantically aligned with the text encoding.

[0034] A third aspect of the present invention also provides an electronic device comprising: a processor and a memory;

[0035] The processor executes any of the above-mentioned multimodal coding alignment methods based on a diffusion model by calling programs or instructions stored in memory.

[0036] In a fourth aspect, the present invention also provides a computer-readable storage medium that stores a program or instructions that cause a computer to execute a diffusion-based multimodal coding alignment method as described above.

[0037] The beneficial effects of this invention are as follows: This invention acquires multimodal data to be semantically encoded, and uses a pre-trained multimodal encoding alignment model to conditionally diffuse the multimodal data to be semantically encoded to generate text encoding; the text encoding is then input into a large model decoder or used as RAG encoding for downstream tasks; wherein, the generated text encoding is used as multimodal data semantic encoding and is semantically aligned with the text encoding. In this invention, the pre-trained multimodal encoding alignment model utilizes the latent multimodal data encoding information learnable by the diffusion model, simultaneously completing multimodal semantic encoding through a diffusion model structure, while using a large language model encoding as the multi-task optimization direction, making the results of the pre-trained multimodal model encoding alignment model more applicable to downstream tasks. Attached Figure Description

[0038] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0039] Figure 1 A diagram illustrating a multimodal coding alignment method based on a diffusion model provided in an embodiment of the present invention;

[0040] Figure 2 A diagram illustrating a training method for a pre-trained multimodal coding alignment model provided in an embodiment of the present invention;

[0041] Figure 3 A method diagram for predicting multimodal data joint noise distribution provided in an embodiment of the present invention is shown.

[0042] Figure 4 A diagram of a multimodal coding alignment device based on a diffusion model is provided in an embodiment of the present invention;

[0043] Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0045] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0046] In the description of this invention, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0048] This invention proposes a multimodal coding alignment method, device, electronic device, and storage medium based on a diffusion model. It utilizes the latent multimodal data encoding information that the diffusion model itself can learn, and simultaneously completes multimodal semantic coding through a diffusion model structure. At the same time, it uses large language model coding as a multi-task optimization direction, making the multimodal model coding results more applicable to downstream tasks.

[0049] Before introducing the embodiments of the present invention, the technical terms involved in the present invention will be introduced first.

[0050] Diffusion Model: The diffusion model is a generative model consisting of a forward (diffusion) process and an inverse process. The forward process progressively adds pixel-by-pixel noise to the image until the image satisfies Gaussian noise; the inverse process reconstructs the image through denoising.

[0051] Encoding alignment: The process of ensuring consistency and comparability of data from different modalities in the vector space.

[0052] Large Language Models (LLMs) are deep learning models trained on large amounts of text data that can generate natural language text or understand the meaning of language text. LLMs can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0053] Method Implementation Examples

[0054] Figure 1 A diagram illustrating a multimodal coding alignment method based on a diffusion model provided in an embodiment of the present invention.

[0055] In a first aspect, this invention proposes a multimodal coding alignment method based on a diffusion model, combined with... Figure 1 It includes three steps: S1 to S3.

[0056] S1: Obtain the multimodal data to be semantically encoded.

[0057] Specifically, in this embodiment of the invention, the multimodal data to be semantically encoded can be image data, voice data, text data, and video data, etc.

[0058] S2: Use a pre-trained multimodal coding alignment model to perform conditional diffusion to generate text codes from the multimodal data to be semantically encoded.

[0059] Specifically, in this embodiment of the invention, the training steps of the pre-trained multimodal coding alignment model are described in detail below. The pre-trained multimodal coding alignment model is used to perform conditional diffusion on the multimodal data to be semantically encoded. The resulting text code can then be used as the semantic code for multimodal data. Here, the code is semantically aligned with the text code.

[0060] S3: Input the text encoding into the large model decoder or use it as RAG encoding for downstream tasks;

[0061] Specifically, in this embodiment of the invention, the generated text encoding can be directly input into a large model decoder or used as RAG encoding for downstream tasks.

[0062] Figure 2 This diagram illustrates a training method for a pre-trained multimodal coding alignment model provided in an embodiment of the present invention.

[0063] Furthermore, in the aforementioned multimodal coding alignment method based on a diffusion model, the training steps of the pre-trained multimodal coding alignment model are combined with... Figure 2 ,include:

[0064] S21: Obtain the text data and the other modality matching data corresponding to the text data.

[0065] Specifically, in this embodiment of the invention, the text data and the other modality matching data corresponding to the text data are a large amount of text data and the image data, voice data and video data corresponding to the text data.

[0066] S22: Preprocess the text data and the other modality matching data corresponding to the text data.

[0067] Specifically, in this embodiment of the invention, the means of preprocessing a large amount of text data and the corresponding image data, audio data and video data are not limited. For example, text data may need to be processed by word segmentation, stop word removal, word embedding, etc.; image data may need to be processed by scaling, cropping, color standardization, etc.

[0068] S23: Encode the preprocessed text data and the corresponding modality matching data into a shallow space as input to the constructed multimodal joint diffusion model.

[0069] Specifically, in this embodiment of the invention, the preprocessed text data and the other modality matching data corresponding to the text data are encoded into the shallow space by an encoder. For example, the preprocessed text data is encoded into the shallow space by an LLM encoder, the preprocessed image data is encoded into the shallow space by a VAE encoder, and the other modality matching data is encoded into the shallow space by other modality encoders as input to the multimodal joint diffusion model.

[0070] S24: Train a multimodal data joint noise distribution prediction model using regression loss from a multimodal joint diffusion model.

[0071] Specifically, in this embodiment of the invention, noise is added to the multimodal data pairs, and a single diffusion model is used to simultaneously train the ability to predict the joint noise distribution of the spliced ​​features of the multimodal data, thereby obtaining a multimodal data joint noise distribution prediction model.

[0072] S25: Using the multimodal data joint noise distribution prediction model as a pre-trained model, perform bidirectional conditional diffusion between text data and other modal data, and perform multi-task fine-tuning learning between other modal data and text data by minimizing the coding distance of the large language model, to obtain a pre-trained multimodal coding alignment model.

[0073] Specifically, in this embodiment of the invention, the final multimodal coding alignment model is achieved by conditional diffusion transformation of training text data and other modal data and minimizing the original coding distance between the generated other modal data and the large language model.

[0074] Figure 3 A diagram illustrating a method for predicting the joint noise distribution of multimodal data provided in an embodiment of the present invention.

[0075] Furthermore, in the aforementioned multimodal coding alignment method based on a diffusion model, the regression loss of the multimodal joint diffusion model is used to train the multimodal data joint noise distribution prediction model, combined with... Figure 3 It includes two steps, S31 and S32:

[0076] S31: Add noise to multimodal data pairs.

[0077] S32: Use a single diffusion model to simultaneously train the ability to predict the joint noise distribution of spliced ​​features of multimodal data, thereby obtaining a prediction model for the joint noise distribution of multimodal data.

[0078] Specifically, in this embodiment of the invention, the process of adding noise to multimodal data pairs is represented as follows:

[0079]

[0080] Where, x b Let x represent the original input, t represent the time step t∈T, and x represent the time step. t x t-1 Let q(x) represent the data input at different time steps. t |x t-1 ) indicates to x t-1 Add Gaussian noise to the input, q(x) 1:T |x b ) indicates that the original input x is added step by step. b Gaussian noise, β t Indicates noise interpolation, α t =1-β t N(0,I) represents the standard normal distribution, and T represents the total duration.

[0081] In existing technologies, diffusion models perform well in generating multimodal data, and some diffusion models can incorporate conditional guidance to complete text-to-image and text-to-speech tasks. This indicates that diffusion models have learned the intrinsic meaning of multimodal data well during the denoising process. This application utilizes the potential multimodal information understanding capability of diffusion models to complete the semantic encoding alignment task of multimodal data using a single model architecture, empowering downstream large language model inference or encoding retrieval tasks (such as RAG).

[0082] The following describes the process of obtaining the ability to simultaneously train and predict the joint noise distribution of spliced ​​features of multimodal data using a single diffusion model, thereby obtaining a joint noise distribution prediction model for multimodal data.

[0083] The optimal mean of the maximum likelihood estimate in the reverse generation process of a single diffusion model is:

[0084]

[0085] Where, ∈ x This indicates adding to data x t The noise, E[∈ x |x t Let β be the expected noise added to the input data at time t. t Indicates noise interpolation, α t =1-β t x t For data at different time steps, Let α represent the optimal mean. i This represents the value of α when noise is added for the i-th time. Let α represent the product of all time steps up to t, where i represents the i-th time step up to t.

[0086] The data generation process can be reconstructed using noise at different time steps in the noise prediction model. The unconditional single diffusion model is expressed as the formula for optimizing the minimum regression loss as follows:

[0087]

[0088] Where, ∈ θ Describes an unconditional multimodal joint diffusion model with parameter θ, ∈ θ (x t (,t) represents the noise result predicted by the unconditional multimodal joint diffusion model at time t, where x0 represents the original input data, ∈ x This represents the noise actually added to x0. This represents the regression loss of the model with known modal data at time t;

[0089] The conditional single diffusion model can be expressed as the formula for optimizing the minimum regression loss as follows:

[0090]

[0091] Where x0, y0 represent two known modalities, ∈ θ (x t (y0,t) represents the original input data x at time t given the modal data y0. t The conditional multimodal joint diffusion model predicts noise results, ∈ xThis represents the noise actually added to x0. This represents the regression loss of the model predicting x under known conditions, where the two modal data at time t are x0 and y0.

[0092] Furthermore, in the aforementioned multimodal coding alignment method based on a diffusion model, the regression loss of the joint diffusion, in the ability to simultaneously train and predict the joint noise distribution of multimodal data splicing features using a single diffusion model, is expressed as:

[0093]

[0094] Among them, t x ,t y These represent the time steps of different modal data. Representing different modal data t x ,t y Encoded input at time step, Represents the joint noise distribution, ∈ x ,∈ y This represents the true noise in data from different modalities. This represents the regression loss due to the joint diffusion of data from the two modalities x0 and y0.

[0095] Specifically, in this embodiment of the invention, in order to enable the diffusion model to learn multimodal codes simultaneously, the codes are mapped to the same vector space for alignment. A single diffusion model is used to simultaneously predict the joint noise distribution of multiple modal concatenation features. Taking two modalities as an example, the regression loss of joint diffusion is:

[0096]

[0097] Furthermore, in the aforementioned multimodal coding alignment method based on a diffusion model, the overall loss of the pre-trained multimodal coding alignment model is:

[0098]

[0099] in, This represents the overall loss of the pre-trained multimodal coding alignment model, where γ and φ are the weight parameters of the multi-task loss. This represents the diffusion loss from other modal data to text data. This represents the diffusion loss from text data to other modalities. This represents the distance loss between the generation of other modal data and the original encoding of the large language model.

[0100] Specifically, in this embodiment of the invention, the diffusion loss from other modal data to text data... The calculation formula is as follows:

[0101]

[0102] Where x_text0 represents the text data input, and x_m0 represents the other modal data input, the above formula indicates that the text vector is generated when the other modal data is known.

[0103] Indicates t x_text The text data noise prediction result output by the time step model, t x_m This indicates the time step of the remaining modal data.

[0104] Represents the loss of the noise prediction model from other modal data to text data, ∈ x_text Represents real noise;

[0105] Text data to other modal data diffusion loss The calculation formula is as follows:

[0106]

[0107] Where x_text0 is the text data input, x_m0 is the other modal data input, and the above formula indicates that the other modal vectors are generated when the text data is known. x_m This indicates the time step of the remaining modal data. This represents the loss of the noise prediction model from text data to other modal data. Indicates t x_text The text data noise prediction results output by the time step model, ∈ x_text This is real noise;

[0108] The formulas for calculating the distance loss between the generated modal data and the original encoding of the large language model are as follows:

[0109]

[0110] Where, d embedding Generate codes for a multimodal data joint noise distribution prediction model, d llm_text The result of encoding text data through a large language model encoder, d embedding , W represents the encoded vector output by the encoding conversion layer. Q W K W V Let represent the weight parameter matrix of the attention mechanism, MLP represents multilayer perceptron, Att represents transformer attention computation, i represents the i-th dimension of the encoding vector, and N represents the dimension of the encoding vector.

[0111] Furthermore, in the aforementioned multimodal encoding alignment method based on a diffusion model, the distance between the generated modal data and the original encoding of the large language model is learned by the encoding transformation layer based on the transformer attention method.

[0112] Specifically, in this embodiment of the invention, in order to make the encoding generation of the diffusion model more similar to the encoding of the large model, an encoding transformation layer based on the transformer attention method is designed to learn the distance between the multimodal data generation and the original encoding of the large language model.

[0113] Device Examples

[0114] Figure 4 This is a diagram of a multimodal coding alignment device based on a diffusion model, provided in an embodiment of the present invention.

[0115] In a second aspect, the present invention also proposes a multimodal coding alignment device based on a diffusion model, combined with Figure 4 ,include:

[0116] Acquisition module 41: Used to acquire multimodal data to be semantically encoded.

[0117] Specifically, in this embodiment of the invention, the multimodal data to be semantically encoded acquired by the acquisition module 41 can be image data, voice data, text data, and video data, etc.

[0118] Generation Module 42: Used to generate text data encoding by conditional diffusion of the multimodal data to be semantically encoded using a pre-trained multimodal coding alignment model.

[0119] Specifically, in this embodiment of the invention, a pre-trained multimodal coding alignment model is used to perform conditional diffusion on the multimodal data to be semantically encoded. The text encoding finally generated by the generation module 42 can be used as the semantic encoding of the multimodal data. Here, the encoding is semantically aligned with the text encoding.

[0120] Input module 43: used to encode the text data and input it into a large model decoder or as RAG encoding for downstream tasks.

[0121] Specifically, in this embodiment of the invention, the generated text encoding can be directly input into the large model decoder through the input module 43 or used as RAG encoding for downstream tasks.

[0122] A third aspect of the present invention also provides an electronic device comprising: a processor and a memory;

[0123] The processor executes any of the above-mentioned multimodal coding alignment methods based on a diffusion model by calling programs or instructions stored in memory.

[0124] In a fourth aspect, the present invention also provides a computer-readable storage medium that stores a program or instructions that cause a computer to execute a diffusion-based multimodal coding alignment method as described above.

[0125] Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention.

[0126] like Figure 5 As shown, the electronic device includes at least one processor 501, at least one memory 502, and at least one communication interface 503. The various components in the electronic device are coupled together via a bus system 504. The communication interface 503 is used for information transmission with external devices. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 5 The general designated all buses as Bus System 504.

[0127] It is understood that the memory 502 in this embodiment can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0128] In some implementations, memory 502 stores elements such as executable units or data structures, or subsets thereof, or extended sets thereof: operating systems and applications.

[0129] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application programs include various applications, such as media players and browsers, used to implement various application functions. A program implementing any method in the diffusion-based multimodal coding alignment method provided in this embodiment of the invention can be included in the application programs.

[0130] In this embodiment of the invention, the processor 501 executes the steps of various embodiments of a multimodal coding alignment method based on a diffusion model provided by the present invention by calling the program or instructions stored in the memory 502, specifically, the program or instructions stored in the application program.

[0131] Acquire multimodal data to be semantically encoded;

[0132] A pre-trained multimodal coding alignment model is used to conditionally diffuse the multimodal data to be semantically encoded to generate text data encoding;

[0133] Encode the text data and input it into a large model decoder or use it as RAG encoding for downstream tasks;

[0134] The generated text data encoding is used as a multimodal data semantic encoding and is semantically aligned with the text data encoding.

[0135] Any method in the multimodal coding alignment method based on a diffusion model provided in this embodiment of the invention can be applied to, or implemented by, processor 501. Processor 501 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 501 or through software instructions. Processor 501 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor.

[0136] The steps of any method in the multimodal coding alignment method based on a diffusion model provided in this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software units in the decoding processor. The software units can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 502, and processor 501 reads the information in memory 502 and combines it with hardware to complete the steps of the method.

[0137] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention and form different embodiments.

[0138] Those skilled in the art will understand that the descriptions of the various embodiments have different focuses, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0139] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention. All such modifications and variations fall within the scope defined by the appended claims. The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0140] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-modal encoding alignment method based on a diffusion model, characterized in that, The method comprises the following steps: obtaining multi-modal data to be semantically encoded; using a pre-trained multi-modal encoding alignment model to perform conditional diffusion on the multi-modal data to be semantically encoded to generate text encoding; inputting the text encoding into a large model decoder or as RAG encoding for a downstream task; wherein the generated text encoding is used as multi-modal data semantic encoding and is semantically aligned with the text encoding; the training steps of the pre-trained multi-modal encoding alignment model comprise: obtaining text data and matching data of the remaining modal corresponding to the text data; preprocessing the text data and the matching data of the remaining modal corresponding to the text data; encoding the preprocessed text data and the matching data of the remaining modal corresponding to the text data into a shallow space as input of a constructed multi-modal joint diffusion model; training a multi-modal data joint noise distribution prediction model using the regression loss of the multi-modal joint diffusion model; using the multi-modal data joint noise distribution prediction model as a pre-training model, performing bidirectional conditional diffusion of the text data and the matching data of the remaining modal, and multi-task fine-tuning learning of minimizing the encoding distance of the matching data of the remaining modal and the text data through a large language model to obtain the pre-trained multi-modal encoding alignment model.

2. The multi-modal encoding alignment method based on diffusion model according to claim 1, characterized in that, Training a multi-modal data joint noise distribution prediction model using the regression loss of the multi-modal joint diffusion model comprises: adding noise to the multi-modal data pair; using a single diffusion model to simultaneously train the joint noise distribution prediction ability of the multi-modal data splicing features, thereby obtaining the multi-modal data joint noise distribution prediction model.

3. The multi-modal encoding alignment method based on diffusion model according to claim 1, characterized in that, The overall loss of the pre-trained multi-modal encoding alignment model is: wherein, denotes the overall loss of the pre-trained multi-modal encoding alignment model, γ, φ are weight parameters of multi-task loss, denotes the remaining modal data to text data diffusion loss, denotes the text data to the remaining modal data diffusion loss, denotes the remaining modal data generation and large language model original encoding distance loss.

4. The multi-modal encoding alignment method based on diffusion model according to claim 1, characterized in that, the process of adding noise to the multi-modal data pair is represented as: Where, x b Let x represent the original input, t represent the time step t∈T, and x represent the time step. t x t-1 Let q(x) represent the data input at different time steps. t |x t-1 ) indicates to x t-1 Add Gaussian noise to the input, q(x) 1:T |x b ) indicates that the original input x is added step by step. b Gaussian noise, β t Indicates noise interpolation, α t =1-β t N(0,I) represents the standard normal distribution, and T represents the total duration.

5. The multi-modal encoding alignment method based on diffusion model according to claim 2, characterized in that, in the process of using a single diffusion model to simultaneously train the joint noise distribution prediction ability of the multi-modal data splicing features, the regression loss of the joint diffusion is represented as: where t x , t y denote the time steps of different modal data, denote different modal data, t x , t y encoded inputs at time steps, denote the joint noise distribution, ∈ x , ∈ y denote the real noise of different modal data, denote the regression loss of joint diffusion of x 0, y0.

6. The multi-modal encoding alignment method based on diffusion model according to claim 3, characterized in that, the generation of the matching data of the remaining modal and the original encoding distance of the large language model are based on the coding transformation layer learning of the transformer attention mode.

7. A multimodal coding alignment device based on a diffusion model, characterized in that, The method comprises the following steps: an acquisition module for acquiring multi-modal data to be semantically encoded; a generation module for using a pre-trained multi-modal encoding alignment model to perform conditional diffusion on the multi-modal data to be semantically encoded to generate text encoding; an input module for inputting the text encoding into a large model decoder or as RAG encoding for a downstream task; wherein the generated text encoding is used as multi-modal data semantic encoding and is semantically aligned with the text encoding; the training steps of the pre-trained multi-modal encoding alignment model comprise: obtaining text data and matching data of the remaining modal corresponding to the text data; preprocessing the text data and the matching data of the remaining modal corresponding to the text data; encoding the preprocessed text data and the matching data of the remaining modal corresponding to the text data into a shallow space as input of a constructed multi-modal joint diffusion model; training a multi-modal data joint noise distribution prediction model using the regression loss of the multi-modal joint diffusion model; Taking the multi-modal data joint noise distribution prediction model as a pre-training model, bidirectional conditional diffusion of text data and the remaining modal data and multi-task fine-tuning learning of the remaining modal data and the text data through the encoding distance minimization of a large language model are performed to obtain a pre-trained multi-modal encoding alignment model.

8. An electronic device, comprising: The method comprises the following steps: a processor and a memory; the processor is configured to execute the multi-modal encoding alignment method based on the diffusion model according to any one of claims 1 to 6 by calling programs or instructions stored in the memory.

9. A computer-readable storage medium, characterized in that, the computer readable storage medium stores programs or instructions, and the programs or instructions enable the computer to execute the multi-modal encoding alignment method based on the diffusion model according to any one of claims 1 to 6.

Images