Multimodal coding alignment method and device based on diffusion model

Abstract

The invention relates to a multi-modal coding alignment method and device based on a diffusion model, and belongs to the technical field of multi-modal coding alignment. The method comprises the following steps: acquiring multi-modal data to be subjected to semantic coding, performing conditional diffusion on the multi-modal data to be subjected to semantic coding by using a pre-trained multi-modal coding alignment model to generate a text code; the text codes are input into a large model decoder or used as RAG codes for downstream tasks; wherein the generated text code is used as a multi-modal data semantic code, and is in semantic alignment with the text code. According to the method, the pre-trained multi-modal coding alignment model utilizes potential multi-modal data coding information which can be learned by a diffusion model, multi-modal semantic coding is completed at the same time through a diffusion model structure, and meanwhile, large language model coding is used as a multi-task optimization direction; and a pre-trained multi-modal model coding alignment model result is more suitable for downstream tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal coding alignment, and in particular relates to a multimodal coding alignment method and device based on a diffusion model. Background Art

[0002] With the continuous development of generative artificial intelligence technology, more and more large language models are no longer limited to single-modal data understanding. The premise of this data understanding task is that multimodal data has the same semantic encoding. Previous methods usually use contrastive learning to map multimodal information pairs (such as text-image, text-audio) to the same vector space. This method usually requires a large number of text alignment data sets. Although this method can achieve unified encoding of multimodal data, a single model usually only supports one multimodal pair encoding, and when connecting to downstream tasks, it usually introduces information loss, affecting the performance of downstream semantic understanding tasks. Summary of the invention

[0003] In view of the above deficiencies in the prior art, the purpose of the invention is to provide a multimodal coding alignment method and device based on a diffusion model. The pre-trained multimodal coding alignment model utilizes the potential multimodal data coding information that can be learned by the diffusion model, and simultaneously completes multimodal semantic coding through a diffusion model structure. At the same time, the large language model coding is used as a multi-task optimization direction, so that the pre-trained multimodal model coding alignment model results are more suitable for downstream tasks.

[0004] In a first aspect of the present invention, a multimodal coding alignment method based on a diffusion model is proposed, comprising:

[0005] Acquire multimodal data to be semantically encoded;

[0006] Use the pre-trained multimodal encoding alignment model to conditionally diffuse the multimodal data to be semantically encoded to generate text encoding;

[0007] Input text encoding into large model decoder or as RAG encoding for downstream tasks;

[0008] Among them, the generated text encoding is used as the semantic encoding of multimodal data and is semantically aligned with the text encoding.

[0009] Furthermore, in the above-mentioned diffusion model-based multimodal code alignment method, the training step of the pre-trained multimodal code alignment model includes:

[0010] Obtain text data and other modal matching data corresponding to the text data;

[0011] Preprocessing the text data and other modality matching data corresponding to the text data;

[0012] Encode the preprocessed text data and the other modal matching data corresponding to the text data into the shallow space as the input of the constructed multimodal joint diffusion model;

[0013] Use the regression loss of the multimodal joint diffusion model to train the multimodal data joint noise distribution prediction model;

[0014] Taking the multimodal data joint noise distribution prediction model as the pre-training model, bidirectional conditional diffusion of text data and other modal data and multi-task fine-tuning learning of minimizing the encoding distance between other modal data and text data through a large language model are performed to obtain a pre-trained multimodal encoding alignment model.

[0015] Furthermore, in the above-mentioned multimodal encoding alignment method based on diffusion model, the multimodal data joint noise distribution prediction model is trained using the regression loss of the multimodal joint diffusion model, including:

[0016] Add noise to multimodal data pairs;

[0017] A single diffusion model is used to simultaneously train the ability to predict the joint noise distribution of multimodal data splicing features, thereby obtaining a multimodal data joint noise distribution prediction model.

[0018] Furthermore, in the above-mentioned multimodal encoding alignment method based on the diffusion model, the overall loss of the pre-trained multimodal encoding alignment model is:

[0019]

[0020] in, represents the overall loss of the pre-trained multimodal encoder alignment model, γ, φ are the weight parameters of the multi-task loss, represents the diffusion loss from other modal data to text data, represents the diffusion loss from text data to other modal data, Represents the distance loss between the remaining modal data generation and the original encoding of the large language model.

[0021] Furthermore, in the above-mentioned multimodal coding alignment method based on diffusion model,

[0022] The process of adding noise to multimodal data pairs is expressed as:

[0023]

[0024] Among them, x b represents the original input, t represents the time step t∈T, x t 、x t-1 Represents the data input at different time steps, q(xt ∣x t-1 ) indicates the direction to x t-1 Gaussian noise is added to the input, q(x 1:T ∣x b ) represents the gradual addition of the original input x b Gaussian noise, β t represents noise interpolation, α t =1-β t , N(0,I) represents the standard normal distribution, and T represents the total duration.

[0025] Furthermore, in the above-mentioned multimodal encoding alignment method based on the diffusion model, a single diffusion model is used to simultaneously train the ability to predict the joint noise distribution of the splicing features of multimodal data, and the regression loss of the joint diffusion is expressed as:

[0026]

[0027] Among them, t x ,t y Respectively represent the time steps of different modal data, Represents different modal data t x ,t y The encoded input at time steps, represents the joint noise distribution, τ x ,τ y Represents the real noise of different modal data, Represents the regression loss of the joint diffusion of the two modal data x0y0.

[0028] Furthermore, in the above-mentioned multimodal encoding alignment method based on the diffusion model, the distance between the generation of the remaining modal data and the original encoding of the large language model is learned by the encoding conversion layer based on the transformer attention method.

[0029] In a second aspect of the present invention, a multimodal coding alignment device based on a diffusion model is also proposed, comprising:

[0030] Acquisition module: used to acquire multimodal data to be semantically encoded;

[0031] Generation module: used to generate text encoding by conditional diffusion of multimodal data to be semantically encoded using a pre-trained multimodal encoding alignment model;

[0032] Input module: used to input the text encoding into a large model decoder or as RAG encoding for downstream tasks;

[0033] Among them, the generated text encoding is used as the semantic encoding of multimodal data and is semantically aligned with the text encoding.

[0034] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0035] The processor is used to execute any one of the above diffusion model-based multimodal coding alignment methods by calling the program or instruction stored in the memory.

[0036] In a fourth aspect of the present invention, a computer-readable storage medium is further proposed, wherein the computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute any of the above multimodal coding alignment methods based on a diffusion model.

[0037] The beneficial effects of the present invention are as follows: the present invention obtains multimodal data to be semantically encoded, and uses a pre-trained multimodal encoding alignment model to conditionally diffuse the multimodal data to be semantically encoded to generate text encoding; the text encoding is input into a large model decoder or used as a RAG encoding for downstream tasks; wherein the generated text encoding is used as a multimodal data semantic encoding, and is semantically aligned with the text encoding. The pre-trained multimodal encoding alignment model in the present invention utilizes the potential multimodal data encoding information that can be learned by the diffusion model, and simultaneously completes the multimodal semantic encoding through a diffusion model structure, and uses the large language model encoding as a multi-task optimization direction, so that the pre-trained multimodal model encoding alignment model results are more suitable for downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, the remaining drawings can also be obtained based on these drawings.

[0039] Figure 1 A multimodal coding alignment method based on a diffusion model provided by an embodiment of the present invention;

[0040] Figure 2 A diagram of a training method for a pre-trained multimodal coding alignment model provided in an embodiment of the present invention;

[0041] Figure 3 A method diagram of a multimodal data joint noise distribution prediction model provided by an embodiment of the present invention;

[0042] Figure 4 A diagram of a multimodal coding alignment device based on a diffusion model provided by an embodiment of the present invention;

[0043] Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should belong to the scope of protection of the present invention.

[0045] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0046] In the description of the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be internal communication between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0048] The present invention proposes a multimodal coding alignment method, device, electronic device and storage medium based on a diffusion model, which utilizes the potential multimodal data encoding information that can be learned by the diffusion model itself, and simultaneously completes multimodal semantic encoding through a diffusion model structure. At the same time, large language model encoding is used as a multi-task optimization direction, so that the multimodal model encoding results are more suitable for downstream tasks.

[0049] Before introducing the embodiments of the present invention, professional terms involved in the present invention are first introduced.

[0050] Diffusion Model: Diffusion model is a generative model, which is divided into forward (diffusion) process and inverse process. The forward process gradually adds pixel-by-pixel noise to the image until the image satisfies Gaussian noise; the inverse process reconstructs the image by denoising.

[0051] Encoding Alignment: The process of ensuring that data from different modalities are consistent and comparable in vector space.

[0052] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.

[0053] Method Embodiment

[0054] Figure 1 A multimodal coding alignment method based on a diffusion model is provided in an embodiment of the present invention.

[0055] In the first aspect of the present invention, a multimodal coding alignment method based on a diffusion model is proposed. Figure 1 , including three steps from S1 to S3:

[0056] S1: Obtain multimodal data to be semantically encoded.

[0057] Specifically, in the embodiment of the present invention, the multimodal data to be semantically encoded may be image data, voice data, text data, video data, etc.

[0058] S2: Use the pre-trained multimodal encoding alignment model to conditionally diffuse the multimodal data to be semantically encoded to generate text encoding.

[0059] Specifically, in an embodiment of the present invention, the training steps of the pre-trained multimodal coding alignment model are described in detail below. The pre-trained multimodal coding alignment model is used to perform conditional diffusion on the multimodal data to be semantically encoded, and the text encoding finally generated can be used as the semantic encoding of the multimodal data. Here, the encoding is semantically aligned with the text encoding.

[0060] S3: Input the text encoding into the large model decoder or as RAG encoding for downstream tasks;

[0061] Specifically, in an embodiment of the present invention, the generated text encoding can be directly input into a large model decoder or used as RAG encoding for downstream tasks.

[0062] Figure 2 A diagram of a training method for a pre-trained multimodal coding alignment model provided in an embodiment of the present invention.

[0063] Furthermore, in the above-mentioned multimodal code alignment method based on diffusion model, the training step of the pre-trained multimodal code alignment model is combined with Figure 2 ,include:

[0064] S21: Obtain text data and other modality matching data corresponding to the text data.

[0065] Specifically, in the embodiment of the present invention, the text data and other modal matching data corresponding to the text data are a large amount of text data and image data, voice data and video data corresponding to the text data.

[0066] S22: Preprocess the text data and other modal matching data corresponding to the text data.

[0067] Specifically, in the embodiments of the present invention, the means for preprocessing a large amount of text data and the image data, voice data and video data corresponding to the text data are not limited here. For example, text data may need to be processed by word segmentation, stop word removal, word embedding, etc.; image data may need to be scaled, cropped, color normalized, etc.

[0068] S23: Encode the preprocessed text data and the remaining modal matching data corresponding to the text data into a shallow space as input to the constructed multimodal joint diffusion model.

[0069] Specifically, in an embodiment of the present invention, encoding of the preprocessed text data and the remaining modal matching data corresponding to the text data into a shallow space is achieved through an encoder, such as: encoding the preprocessed text data into a shallow space through an LLM encoder, encoding the preprocessed image data into a shallow space through a VAE encoder, and encoding the remaining modal matching data into a shallow space through the remaining modal encoder as input to a multimodal joint diffusion model.

[0070] S24: Use the regression loss of the multimodal joint diffusion model to train a multimodal data joint noise distribution prediction model.

[0071] Specifically, in an embodiment of the present invention, noise is added to multimodal data pairs, and a single diffusion model is used to simultaneously train the ability to predict the joint noise distribution of multimodal data splicing features, thereby obtaining a multimodal data joint noise distribution prediction model.

[0072] S25: Using the multimodal data joint noise distribution prediction model as the pre-training model, perform bidirectional conditional diffusion of text data and other modal data, and multi-task fine-tuning learning of minimizing the encoding distance between the other modal data and the text data through a large language model to obtain a pre-trained multimodal encoding alignment model.

[0073] Specifically, in an embodiment of the present invention, the final multimodal encoding alignment model is completed by conditional diffusion transformation of the training text data and the remaining modal data and minimizing the original encoding distance between the remaining modal data and the large language model.

[0074] Figure 3 A method diagram of a multimodal data joint noise distribution prediction model provided by an embodiment of the present invention.

[0075] Furthermore, in the above-mentioned multimodal encoding alignment method based on diffusion model, the multimodal data joint noise distribution prediction model is trained using the regression loss of the multimodal joint diffusion model, and combined with Figure 3 , including two steps from S31 to S32:

[0076] S31: Add noise to multimodal data pairs.

[0077] S32: Use a single diffusion model to simultaneously train the ability to predict the joint noise distribution of multimodal data splicing features, thereby obtaining a multimodal data joint noise distribution prediction model.

[0078] Specifically, in the embodiment of the present invention, the process of adding noise to the multimodal data pair is expressed as:

[0079]

[0080] Among them, x b represents the original input, t represents the time step t∈T, x t 、x t-1 Represents the data input at different time steps, q(x t ∣x t-1 ) indicates the direction to x t-1 Gaussian noise is added to the input, q(x 1:T ∣x b ) represents the gradual addition of the original input x b Gaussian noise, β t represents noise interpolation, α t =1-β t , N(0,I) represents the standard normal distribution, and T represents the total duration.

[0081] In the prior art, diffusion models perform well in generating multi-modal data, and some diffusion models can add conditional guidance to complete text-to-image and text-to-speech tasks, which shows that the diffusion model has learned the intrinsic meaning of multi-modal data very well during the denoising process. This application utilizes the potential multi-modal information understanding ability of the diffusion model and uses a single model architecture to complete the semantic encoding alignment task of multi-modal data, enabling downstream large language model reasoning or encoding retrieval tasks (such as RAG).

[0082] The following describes the process of obtaining the ability to simultaneously train a single diffusion model to predict the joint noise distribution of multimodal data splicing features, thereby obtaining a multimodal data joint noise distribution prediction model.

[0083] The optimal mean of the maximum likelihood estimate for the reverse generation process of a single diffusion model is:

[0084]

[0085] Among them, ∈ x Indicates adding to data x t The noise, E[τ x ∣x t ] is the noise expectation added to the input data at time t, β t represents noise interpolation, α t =1-β t , x t is the data at different time steps, represents the optimal mean, α i represents the α value of adding noise for the i-th time, represents the product of α of all time steps before t, and i represents the i-th time step before t;

[0086] The noise at different time steps of the noise prediction model can restore the data generation process. The unconditional single diffusion model is expressed as the formula for optimizing the minimum regression loss as follows:

[0087]

[0088] Among them, τ θ represents the unconditional multimodal joint diffusion model with parameters θ, ∈ θ (x t ,t) represents the noise result predicted by the unconditional multimodal joint diffusion model at time t, x0 represents the original input data, ∈ x represents the noise actually added to x0, Indicates the model prediction regression loss of x0 at time t for known modal data;

[0089] The conditional single diffusion model is expressed as the formula for optimizing the minimum regression loss as follows:

[0090]

[0091] Among them, x0, y0 represent two known modal data, ∈ θ (x t ,y0,t) represents the original input data x at time t under the known modal data y0 t The conditional multimodal joint diffusion model predicts noisy outcomes, ∈ xrepresents the noise actually added to x0, It represents the regression loss of the model predicting x under the condition that the two modal data x0 and y0 are known at time t.

[0092] Furthermore, in the above-mentioned multimodal encoding alignment method based on the diffusion model, a single diffusion model is used to simultaneously train the ability to predict the joint noise distribution of the splicing features of multimodal data, and the regression loss of the joint diffusion is expressed as:

[0093]

[0094] Among them, t x ,t y Respectively represent the time steps of different modal data, Represents different modal data t x ,t y The encoded input at time steps, represents the joint noise distribution, τ x ,τ y Represents the real noise of different modal data, Represents the regression loss of the joint diffusion of two modal data x0 and y0.

[0095] Specifically, in the embodiment of the present invention, in order to enable the diffusion model to learn multi-modal encoding at the same time, the encoding is mapped to the same vector space for alignment, and a single diffusion model is used to simultaneously predict the joint noise distribution of multiple modal splicing features. Taking two modalities as an example, the regression loss of the joint diffusion is:

[0096]

[0097] Furthermore, in the above-mentioned multimodal encoding alignment method based on the diffusion model, the overall loss of the pre-trained multimodal encoding alignment model is:

[0098]

[0099] in, represents the overall loss of the pre-trained multimodal encoder alignment model, γ, φ are the weight parameters of the multi-task loss, represents the diffusion loss from other modal data to text data, represents the diffusion loss from text data to other modal data, Represents the distance loss between the remaining modal data generation and the original encoding of the large language model.

[0100] Specifically, in the embodiment of the present invention, the diffusion loss from other modal data to text data is The calculation formula is as follows:

[0101]

[0102] Among them, x_text0 represents the text data input, x_m0 represents the other modal data input, and the above formula means that the other modal data are known to complete the generation of text vectors.

[0103] Indicates t x _ text The text data noise prediction results output by the time step model, t x _ m represents the time step of the remaining modal data,

[0104] represents the noise prediction model loss of the rest modality data to text data, ∈ x _ text represents the real noise;

[0105] Diffusion loss from text data to other modal data The calculation formula is as follows:

[0106]

[0107] Among them, x_text0 is the text data input, x_m0 is the other modal data input, and the above formula means that the text data is known to complete the generation of the other modal vectors, t x _ m represents the time step of the remaining modal data, represents the noise prediction model loss from text data to other modal data, Indicates t x _ text The prediction result of the text data noise output by the time step model, τ x _ text is the real noise;

[0108] The calculation formula for the distance loss between the remaining modal data generation and the original encoding of the large language model is as follows:

[0109]

[0110] Among them, d embedding Generates code for multimodal data joint noise distribution prediction model, d llm_text Encode the text data through a large language model encoder, d embedding , represents the encoding vector output by the encoding conversion layer, W Q , WK , W V is the attention mechanism weight parameter matrix, MLP represents multi-layer perceptron, A tt Represents the transformer attention calculation, i represents the i-th dimension of the encoding vector, and N represents the encoding vector dimension.

[0111] Furthermore, in the above-mentioned multimodal encoding alignment method based on the diffusion model, the distance between the generation of the remaining modal data and the original encoding of the large language model is learned by the encoding conversion layer based on the transformer attention method.

[0112] Specifically, in an embodiment of the present invention, in order to make the encoding generation of the diffusion model closer to the large model encoding, a transformer-based attention encoding conversion layer is designed to learn the distance between multimodal data generation and the original encoding of the large language model.

[0113] Device Embodiment

[0114] Figure 4 A diagram of a multimodal coding alignment device based on a diffusion model provided in an embodiment of the present invention.

[0115] In a second aspect of the present invention, a multi-modal coding alignment device based on a diffusion model is also proposed. Figure 4 ,include:

[0116] Acquisition module 41: used for acquiring multimodal data to be semantically encoded.

[0117] Specifically, in the embodiment of the present invention, the multimodal data to be semantically encoded acquired by the acquisition module 41 may be image data, voice data, text data, video data, etc.

[0118] Generation module 42: used to generate text data encoding by conditionally diffusing multimodal data to be semantically encoded using a pre-trained multimodal encoding alignment model.

[0119] Specifically, in an embodiment of the present invention, a pre-trained multimodal coding alignment model is used to perform conditional diffusion on multimodal data to be semantically encoded, and the text encoding finally generated by the generation module 42 can be used as the semantic encoding of the multimodal data, where the encoding is semantically aligned with the text encoding.

[0120] Input module 43: used to encode the text data into a large model decoder or as RAG encoding for downstream tasks.

[0121] Specifically, in the embodiment of the present invention, the generated text encoding can be directly input into the large model decoder through the input module 43 or used as RAG encoding for downstream tasks.

[0122] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0123] The processor is used to execute any one of the above diffusion model-based multimodal coding alignment methods by calling the program or instruction stored in the memory.

[0124] In a fourth aspect of the present invention, a computer-readable storage medium is further proposed, wherein the computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute any of the above multimodal coding alignment methods based on a diffusion model.

[0125] Figure 5 It is a schematic block diagram of an electronic device provided by an embodiment of the present invention.

[0126] like Figure 5 As shown, the electronic device includes: at least one processor 501, at least one memory 502 and at least one communication interface 503. The various components in the electronic device are coupled together through a bus system 504. The communication interface 503 is used to transmit information between external devices. It can be understood that the bus system 504 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 504 .

[0127] It can be understood that the memory 502 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0128] In some implementations, the memory 502 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0129] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. A program that implements any method in a multimodal coding alignment method based on a diffusion model provided in an embodiment of the present invention can be included in the application.

[0130] In an embodiment of the present invention, the processor 501 calls a program or instruction stored in the memory 502, specifically, a program or instruction stored in an application, and the processor 501 is used to execute the steps of each embodiment of a multimodal coding alignment method based on a diffusion model provided in an embodiment of the present invention.

[0131] Acquire multimodal data to be semantically encoded;

[0132] Use the pre-trained multimodal encoding alignment model to conditionally diffuse the multimodal data to be semantically encoded to generate text data encoding;

[0133] Encode text data into a large model decoder or as RAG encoding for downstream tasks;

[0134] Among them, the generated text data encoding is used as the multimodal data semantic encoding, which is aligned with the text data encoding semantics.

[0135] Any method in the multimodal coding alignment method based on a diffusion model provided in an embodiment of the present invention can be applied to the processor 501, or implemented by the processor 501. The processor 501 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 501 or an instruction in the form of software. The above-mentioned processor 501 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0136] The steps of any method in a multimodal coding alignment method based on a diffusion model provided by an embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software units in a decoding processor. The software unit can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502, and completes the steps of the method in combination with its hardware.

[0137] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0138] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0139] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

[0140] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multimodal coding alignment method based on a diffusion model, characterized in that: include: Acquire multimodal data to be semantically encoded; Use the pre-trained multimodal encoding alignment model to conditionally diffuse the multimodal data to be semantically encoded to generate text encoding; Input the text encoding into a large model decoder or as RAG encoding for downstream tasks; Among them, the generated text encoding is used as the semantic encoding of multimodal data and is semantically aligned with the text encoding.

2. According to the multimodal coding alignment method based on diffusion model in claim 1, it is characterized in that: The training steps of the pre-trained multimodal encoding alignment model include: Obtain text data and other modal matching data corresponding to the text data; Preprocessing the text data and other modality matching data corresponding to the text data; Encode the preprocessed text data and the other modal matching data corresponding to the text data into the shallow space as the input of the constructed multimodal joint diffusion model; Use the regression loss of the multimodal joint diffusion model to train the multimodal data joint noise distribution prediction model; Taking the multimodal data joint noise distribution prediction model as the pre-training model, bidirectional conditional diffusion of text data and other modal data and multi-task fine-tuning learning of minimizing the encoding distance between other modal data and text data through a large language model are performed to obtain a pre-trained multimodal encoding alignment model.

3. The multimodal coding alignment method based on diffusion model according to claim 2, characterized in that: The multimodal data joint noise distribution prediction model is trained using the regression loss of the multimodal joint diffusion model, including: Add noise to multimodal data pairs; A single diffusion model is used to simultaneously train the ability to predict the joint noise distribution of multimodal data splicing features, thereby obtaining a multimodal data joint noise distribution prediction model.

4. The multimodal coding alignment method based on diffusion model according to claim 2, characterized in that: The overall loss of the pre-trained multimodal encoder alignment model is: in, represents the overall loss of the pre-trained multimodal encoder alignment model, γ, φ are the weight parameters of the multi-task loss, represents the diffusion loss from other modal data to text data, represents the diffusion loss from text data to other modal data, Represents the distance loss between the remaining modal data generation and the original encoding of the large language model.

5. The multimodal coding alignment method based on diffusion model according to claim 2, characterized in that: The process of adding noise to multimodal data pairs is expressed as: Among them, x b represents the original input, t represents the time step t∈T, x t 、x t-1 Represents the data input at different time steps, q(x t ∣x t-1 ) indicates the direction to x t-1 Gaussian noise is added to the input, q(x 1:T ∣x b ) represents the gradual addition of the original input x b Gaussian noise, β t represents noise interpolation, α t =1-β t , N(0,I) represents the standard normal distribution, and T represents the total duration.

6. The multimodal coding alignment method based on diffusion model according to claim 3 is characterized in that: In the joint noise distribution capability of multimodal data concatenation features trained using a single diffusion model, the regression loss of the joint diffusion is expressed as: Among them, t x ,t y Respectively represent the time steps of different modal data, Represents different modal data, t x ,t y The encoded input at time steps, represents the joint noise distribution, τ x ,τ y Represents the real noise of different modal data, Represents x 0, y0 is the regression loss for the joint diffusion of the two modal data.

7. The multimodal coding alignment method based on diffusion model according to claim 4, characterized in that: The remaining modal data generation and the original encoding distance of the large language model are learned based on the encoding conversion layer of the transformer attention method.

8. A multimodal coding alignment device based on a diffusion model, characterized in that: include: Acquisition module: used to acquire multimodal data to be semantically encoded; Generation module: used to generate text encoding by conditional diffusion of multimodal data to be semantically encoded using a pre-trained multimodal encoding alignment model; Input module: used to input the text encoding into a large model decoder or as RAG encoding for downstream tasks; Among them, the generated text encoding is used as the semantic encoding of multimodal data and is semantically aligned with the text encoding.

9. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the multimodal coding alignment method based on a diffusion model as described in any one of claims 1 to 7 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute a multimodal coding alignment method based on a diffusion model as described in any one of claims 1 to 7.

Images