Multi-modal data integration, matching and interpolation method based on optimal transmission driving

Abstract

The invention provides a multi-modal data integration, matching and interpolation method based on optimal transmission driving. According to the method, a Wasserstein auto-encoder model driven by an optimal transmission theory is constructed. The model can capture complex distribution of different modal data, and effectively deals with a feature sparsity problem through regularization of a potential space. And meanwhile, the similarity between the same kind of modals is enhanced by the model through comparative learning based on sample categories, so that an encoder with strong feature extraction and matching capabilities is trained, the data integration and matching process is optimized, and the accuracy of data interpolation is also remarkably improved. The innovative method successfully deals with the core challenge of multi-modal data integration and analysis in the field of data science, and shows wide application prospects and remarkable actual values especially in the fields of medical images, automatic driving, sentiment analysis and the like.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal data integration calculation and analysis of artificial intelligence, and specifically relates to a multimodal data integration, matching and interpolation method based on optimal transmission drive. Background Art

[0002] Multimodal data refers to data from different types of sensors or data sources, which have different forms and characteristics, such as images, text, audio, video, etc. These data sources provide different perspectives and information on the same phenomenon or object. By combining medical images (such as X-rays and MRI) with clinical data (such as patient history and laboratory test results), more comprehensive diagnostic information can be obtained and the accuracy of disease detection can be improved; using data from different sensors such as cameras, radars, and lidars, the surrounding environment can be more fully perceived, improving the safety and navigation accuracy of vehicles. Combining text, audio, and video data for sentiment analysis can more accurately understand the user's emotional state. By integrating users' browsing history, purchase history, and social media data, more personalized and accurate recommendations can be provided. With the continuous expansion of the application of multimodal data in various fields, how to effectively process and utilize this data has become an important issue.

[0003] Effective data integration, matching, and interpolation are key steps to achieve this goal. Combining data from different modalities into a unified representation helps provide a more comprehensive representation, thereby making more accurate decisions and predictions. Through data matching, data from different modalities can be effectively utilized to avoid information loss or duplicate processing, thereby improving the efficiency of data use. For missing, damaged, or difficult-to-obtain data, interpolation technology can ensure data integrity.

[0004] However, existing interpolation methods often require multimodal data with known matching relationships, which may not be applicable to all data types or application scenarios. In addition, data of different modalities usually have different characteristics and representations. This heterogeneity makes data integration complex and difficult. At the same time, the alignment of data of different modalities in time, space or semantics is also a challenge. Existing matching algorithms use simple similarity metrics, which may not capture the complex relationship between modalities, resulting in errors in matching results. Summary of the invention

[0005] In order to solve the shortcomings of the prior art, the present invention proposes a multimodal data integration, matching and interpolation method based on optimal transport (OT), which can simultaneously handle the integration and pairing tasks of multimodal data, and effectively interpolate the data of one modality by using the data of another modality. This method is applicable to various types of multimodal data and has efficient feature extraction capabilities. It can be widely used in various downstream tasks. It not only optimizes the data integration and pairing process, but also realizes the accurate filling of missing data, thereby promoting the development of multimodal data processing technology and bringing significant application value to related fields.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] The multimodal data integration, matching and interpolation method based on optimal transmission drive is implemented based on the following steps:

[0008] 1) Obtain multimodal training data:

[0009] 1.1) The user selects the multimodal dataset for training locally and reads two datasets, Data X and Data Y, as well as related metadata information;

[0010] 1.2) Random sampling of data, including random sampling of samples, feature dimensionality reduction using principal component analysis, and label processing;

[0011] 1.3) Data preprocessing, including data standardization and missing value processing, to obtain two modalities Modality X and Modality Y from data sets Data X and Data Y;

[0012] 2) Build the model:

[0013] 2.1) Use a modality-specific joint encoder consisting of three layers of multi-layer perceptrons to learn the latent representations of the two modalities separately. The encoder is driven by optimal transfer theory and trained by contrastive learning methods to capture the key structures and features within the data and perform preliminary multimodal data alignment;

[0014] 2.2) Driven by the optimal transmission theory, the maximum bipartite matching problem is transformed into an optimal transmission problem, and the soft matching matrix between modes is obtained;

[0015] 2.3) Using the soft matching matrix, linearly combine the two latent spaces at the corresponding data to obtain an aggregated latent space; for the aggregated latent space, constrain the modal information after aggregation to be similar to that before aggregation, and the corresponding samples after aggregation are also similar;

[0016] 2.4) Reconstruct the model through a decoder composed of three layers of MLP to build a multimodal data integration, matching and interpolation model; and save the model when the loss function converges to the minimum through continuous iterative learning;

[0017] 3) Multimodal data integration, matching and interpolation:

[0018] 3.1) Load the trained model, extract the latent space, and train the K nearest neighbor classifier to classify the test data to detect the model effect;

[0019] 3.2) Input the latent space into the matching algorithm to obtain the matching relationship between the multimodal data; cross-modal data interpolation can be performed by using the encoder of one modality and the decoder of another modality for the data of one modality;

[0020] 3.3) Users adjust network parameters according to their own data characteristics to achieve the best detection effect.

[0021] Further, step 2.1) specifically includes the following steps:

[0022] 2.1.1) Use encoder for feature extraction:

[0023] Feature embeddings are generated using two modality-specific encoders; the encoder consists of three MLP layers, and the data enters the encoder as follows:

[0024] In the datasets Data X and Data Y, for the modality Where n1 is the number of samples of mode X, m1 is the feature dimension of mode X; Where n2 is the number of modal Y samples, and m2 is the feature dimension of modal Y; then:

[0025] X1=f1(ω1X+b1),

[0026] X2=f2(ω2X1+b2),

[0027] L X =f3(ω3X2+b3),

[0028] Y1=g1(η1Y+c1),

[0029] Y2=g2(η2Y1+c2),

[0030] L Y =g3(η3Y2+c3),

[0031] Among them, L X and L Yis the result of the distribution of modality X and modality Y after the encoder, representing the final latent space; ω1, ω2, ω3, η1, η2, η3 are the weights in the network layer, b1, b2, b3, c1, c2, c3 are the bias terms in the network layer; X1 and X2 are the feature matrices obtained by modality X after the first and second layers of the encoder, respectively, and Y1 and Y2 are the feature matrices obtained by modality Y after the first and second layers of the encoder; f1(.), f2(.), f3(.) and g1(.), g2(.), g3(.) are all activation functions LeakyReLU, the subscript represents the i-th layer, i=1, 2, 3;

[0032] 2.1.2) Adopting optimal transfer theory and category-based contrastive learning method to the latent space L X ,L Y To impose restraints;

[0033] The optimal transport is to move from one distribution to another to obtain the minimum amount of movement, by finding an optimal solution π so that the distribution P(x) is The transfer cost is minimized, where P(x) is the probability function of X, for The probability function of is the reconstructed data of mode X;

[0034] In the latent space Z, the decoder learns a mapping G such that z∈Z, the conditional probability density is That is, random variable The probability density under the condition of random variable Z = z; the following transforms the original optimal transmission problem into an expectation express:

[0035]

[0036] in, is any measurable cost function, is the transportation plan that minimizes the total transportation cost. is the joint probability density; x∈X, denote a single sample in the original data space and a single sample in the reconstructed data space, respectively, and Ε(·) denotes taking the expectation;

[0037] Find all possible joint distributions So that its edge representation is:

[0038]

[0039] in, That is, random variable The probability density under the condition of random variable X = x, That is, the random variable X is in the random variable The probability density under the condition.

[0040] and

[0041]

[0042] Q(z)=∫Q(z|x)P(x)dx=P(z),

[0043] P(x,z)=∫ z Q(x)Q(z|x)dz,

[0044] The second equation is to ensure that the marginal distribution of the conditional distribution Q(z) is P(z), where P(z) is the probability function of the random variable Z; P(x,z) represents the joint distribution of X and Z, represents X and The joint distribution of, Q(x) represents the marginal probability density of X, Q(z|x) is the probability density of the random variable Z under the condition that the random variable X=x;

[0045] Because the assumption Z does not change the marginal distribution under the condition of given X; the theorem transforms the original formula into:

[0046]

[0047] The last equation is because the joint distribution in π is replaced by the conditional distribution Q(z|x), thereby transforming the optimized coupling into the optimized random encoder Q(z|x). Through this appropriate Q(z|x) and generator, the reconstructed data is close to the original data; P x represents the probability density of the random variable X, P z represents the probability density of the random variable Z; P x,z , denote the joint probability density between random variables X and Z, Z and The joint probability density between ; c(X,G(z)) represents any measurable cost function;

[0048] If the data of the two modalities are represented as X1, X2, the latent variables of the two modalities are represented as Z1, Z2, and the reconstructed data is represented as Then the loss functions of the two modalities based on the optimal transmission are combined and expressed as:

[0049]

[0050] Among them, L OT represents the combined loss function, MMD represents the maximum mean difference; P X is the prior probability distribution of mode X, PY is the prior probability distribution of mode Y, Q(Z X |X) is the posterior probability distribution of the first mode, Q(Z Y |Y) is the posterior probability distribution of the second mode, k(z X ,·),k(z Y ,·) is a positive definite reproducing kernel function, H K is a real-valued function of the reproducing kernel Hilbert space, is the probability density of the random variable Z corresponding to the X variable, is the probability density of the random variable Z under the Y variable, is the marginal distribution of the random variable Z under the X variable, is the marginal distribution of the random variable Z under the Y variable; is the reconstructed data of mode Y;

[0051] In order to allow the model to learn inter-modal information and generate high-quality latent codes in the subsequent matching process, the training process learns inter-modal similarities through type-based contrastive learning. In this process, the embeddings of cells of the same type between modalities are brought closer, and the embeddings of different cell types are pushed further away; contrastive learning is expressed as follows:

[0052] similarity(encoder(x i ),encoder(y i ))>>similarity(encoder(x i ),encoder(y j )),

[0053] Among them, x i ,y i is the data from the two modalities of the ith cell type, y j are the data from the two modalities of the jth cell type; x i Relative to y i It is a positive sample, with the same cell type; x j Relative to y i It is a negative sample, the cell type is inconsistent; encoder(x i ) represents x i The feature matrix after the encoder, encoder(y i ) represents y i The feature matrix similarity(encoder(x i ),encoder(y i )) means encoder(x i ) and encoder(y i), the constructed loss function hopes to maximize the similarity between the multimodal similarities of the same sample and minimize the similarity between the multimodal similarities of different samples; it is expressed as

[0054]

[0055] Among them, L InfoNCE represents the alignment loss, represents the alignment loss function from modality X to modality Y, represents the alignment loss function from modality Y to modality X,

[0056] τ is the temperature scale parameter;

[0057] This step gives us the latent space L of the two modalities X ,L Y .

[0058] Further, step 2.2) specifically includes the following steps:

[0059] 2.2.1) By relaxing the constraints of the maximum weighted bipartite matching problem, it is converted into an optimal transmission problem, and finally the effect of finding a soft match is achieved; the maximum weighted bipartite matching problem in the form of integer linear programming is considered to solve the problem that the original multimodal data may not be paired. Its goal is to find the optimal transmission problem in two sets L. X ,L Y Find the optimal match in the bipartite graph to maximize the total weight of the matching edges. The weight matrix is It is expressed as:

[0060] subject to:

[0061]

[0062] If you remove The integer constraint of , so that it can be any non-negative real number, take for And L X and L Y is a uniform distribution, then the probability density function is n is the number of samples of X and Y; That belongs to L X ,L Y ; represents the real number weight, represents any measurable cost function, represents the joint probability density;

[0063] Then the above equation becomes an optimal transmission problem, and soft matching can be performed, that is, each With multiple match:

[0064] subject to:

[0065]

[0066] And introduce the entropy regularization term to make the transportation plan more decentralized:

[0067]

[0068] Among them, ε is a hyperparameter with a value of 0.01; finally, the matching relationship matrix A between the two modes is obtained.

[0069] Furthermore, in step 2.3), the matching relationship matrix A between the two modalities is obtained to linearly aggregate the different modal latent spaces to achieve the fusion of the latent spaces of different modalities at the alignment point, condense the information of the two modalities, and obtain the aggregated latent space C X ,C Y .

[0070] Furthermore, in step 2.4), by defining the optimal transmission loss, alignment loss and fusion loss, the pre-processed data is trained using the constructed model, and the loss function is converged to the minimum through multiple iterative learning, and the optimal model parameters are saved; specifically, it includes:

[0071] 2.4.1) Set the parameters of the model, including the dimension of latent variables, the dimension of PCA dimensionality reduction, and the number of training iterations;

[0072] 2.4.2) Calculate the loss; the loss consists of the following:

[0073] L Total =αL OT +βL InfoNCE +γL M +ξL A ,

[0074] Among them, L Total is the total loss function, α, β, γ, ξ are hyper parameters, L OT is the optimal transmission loss, L A and L InfoNCE is the alignment loss, L M For combined loss.

[0075] Furthermore, the loss calculation in step 2.4.2) specifically includes:

[0076] 2.4.2.1) In order to make the corresponding samples in the aggregated latent space have similar semantic information between different modalities, that is, to drive in the same direction during optimization, the alignment loss is used:

[0077]

[0078] Among them, C X ,C Y They represent the aggregated latent space of modality X and modality Y respectively, A represents the optimal matching matrix obtained by step 2.2), representing the pairing relationship between the two modalities; g is a hyperparameter, i.e., the dimension of the specified latent space; n X is the number of samples of mode X; Usually represents the l2 norm of the Frobenius norm;

[0079] 2.4.2.3) In order to make the aggregated latent space maintain the important structure of the original latent space, the preservation loss is used:

[0080]

[0081] Among them, L X ,L Y represents the latent space of X modality and Y modality|, n Y is the number of samples of the Y mode;

[0082] Finally, through continuous iterative learning, when the value of the loss function converges to the minimum, the model is saved.

[0083] The present invention has the following beneficial effects:

[0084] 1. The present invention uses Wasserstein distance to capture the complex distribution of each modality data and solves the feature sparsity problem through regularized cell embedding. This method not only fully considers the differences between different cell types, but also does not need to pre-assume the distribution type of the data.

[0085] 2. The model provided by the present invention is trained by contrastive learning. When learning the embedding representation between two modalities, the similarity between pairs of samples of the same type is maximized, and the similarity between pairs of samples of different types is minimized. For large-scale data sets, pre-training can be used to speed up the training of the model and further improve efficiency.

[0086] 3. The present invention transforms the maximum linear bipartite matching problem into an optimal transmission problem, so that the model can dynamically perform sample matching during the learning process, without the need to obtain matching information between modalities in advance or use paired data for training, thereby enhancing the flexibility and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a structural diagram of the present invention;

[0088] Figure 2 UMAP (Uniform Manifold Approximation and Projection) visualization of single-cell simulation data;

[0089] Figure 3 UMAP visualization of integrated single-cell multimodal data;

[0090] Figure 4 A matching relationship diagram obtained by the present invention;

[0091] Figure 5 This is the interpolation effect diagram of the present invention. DETAILED DESCRIPTION

[0092] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not intended to limit the scope of use of the present invention.

[0093] like Figure 1-5 As shown, the present invention proposes a multimodal data integration, matching and interpolation method driven by optimal transport (OT). The core innovation of this method lies in the construction of a Wasserstein Auto-Encoders (WAE) framework driven by optimal transport theory, which enables the model to focus on the characteristics of different types of samples more effectively. During the training process, by introducing the contrastive learning method, the model can learn the similarities between modalities, in which the embeddings of cells of the same type are brought closer, while the embeddings of cells of different types are pushed away. In addition, this method uses optimal transport theory instead of the traditional maximum bisection matching algorithm to generate the matching matrix. This method not only optimizes the data integration and matching process, but also significantly improves the accuracy of data interpolation. This innovative method provides a new solution for multimodal data processing, especially in the fields of medical imaging, autonomous driving and sentiment analysis, and has broad application prospects and significant practical value. Specifically, the method is implemented based on the following steps:

[0094] Step 1) Obtain multimodal training data

[0095] 1.1) Obtain multimodal training data: The user selects the multimodal dataset used for training locally and reads two datasets, Data X and Data Y, as well as related metadata information;

[0096] 1.2) And randomly sample data, including random sampling of samples, feature dimensionality reduction using principal component analysis, and processing of labels;

[0097] 1.3) Then the data is preprocessed, including data standardization and missing value processing, to obtain the input features Modality X and Modality Y of the two modalities (for example, image modality and text modality from the same sample).

[0098] Step 2) Build the model

[0099] 2.1) A modality-specific joint encoder consisting of three layers of multilayer perceptrons (MLP) is used to learn the latent representations of the two modalities separately. The encoder is driven by optimal transfer theory and trained by contrastive learning methods to capture the key structures and features within the data and perform preliminary multimodal data alignment.

[0100] 2.1.1) Use encoder for feature extraction:

[0101] Two modality-specific encoders are used to generate feature embeddings. The encoder consists of three MLP layers, and the process of how the data enters the encoder will be described:

[0102] Given Data X and Data Y, for Among them, n1 is the number of samples of modality X, m1 is the feature dimension of modality X, and for Among them, n2 is the number of modal Y samples, m2 is the feature dimension of modal Y, then:

[0103] X1=f1(ω1X+b1),

[0104] X2=f2(ω2X1+b2),

[0105] L X =f3(ω3X2+b3),

[0106] Y1=g1(η1Y+c1),

[0107] Y2=g2(η2Y1+c2),

[0108] L Y =g3(η3Y2+c3),

[0109] Among them, L X and L Yare the results of the distribution of modality X and modality Y after the encoder, which represent the final latent space, ω1, ω2, ω3, η1, η2, η3 are the weights in the network layer, b1, b2, b3, c1, c2, c3 are the bias terms in the network layer; X1 and X2 are the feature matrices of modality X obtained after the first and second layers of the encoder, respectively, and Y1 and Y2 are the feature matrices of modality Y obtained after the first and second layers of the encoder; f1(.), f2(.), f3(.) and g1(.), g2(.), g3(.) are activation functions (LeakyReLU), the subscript represents the i-th layer, i=1,2,3.

[0110] 2.1.2) In order to obtain an encoder with strong feature learning and alignment capabilities, the optimal transmission theory and category-based contrastive learning method are used to transform the latent space L X ,L Y To constrain.

[0111] Optimal transmission can better adapt to data of different types and structures, thereby improving the effect and reliability of data analysis and model training. It attempts to move from one distribution to another to obtain the minimum amount of movement. It tries to find an optimal solution π so that the distribution P(x) (the probability function of X) is ( The transfer cost is minimized. is the reconstructed data of mode X.

[0112] In the latent space Z, the decoder learns a mapping G such that The conditional probability density is (i.e., random variable Under the condition of probability density of random variable Z = z), the following transforms the original optimal transmission problem into an expectation express:

[0113]

[0114] in, is any measurable cost function, is the transportation plan that minimizes the total transportation cost. is the joint probability density; x∈X, They represent a single sample in the original data space and a single sample in the reconstructed data space respectively, and Ε(·) means taking the expectation.

[0115] We need to find all possible joint distributions So that its edge representation is:

[0116]

[0117] in, That is, random variable The probability density under the condition of random variable X = x, That is, the random variable X is in the random variable The probability density under the condition.

[0118] and

[0119]

[0120] Q(z)=∫Q(z|x)P(x)dx=P(z),

[0121] P(x,z)=∫ z Q(x)Q(z|x)dz,

[0122] The second equation is to ensure that the marginal distribution of the conditional distribution Q(z) is P(z) (the probability function of the random variable Z), where P(x,z) represents the joint distribution of X and Z. represents X and The joint distribution of; Q(x) represents the marginal probability density of X, Q(z|x) is the probability density of the random variable Z under the condition that the random variable X=x.

[0123] Because the assumption Z does not change the marginal distribution under the condition of given X. According to the theorem, the original formula can be transformed into:

[0124]

[0125] The last equation is because the joint distribution in π is replaced by the conditional distribution Q(z|x), thereby transforming the optimized coupling into the optimized random encoder Q(z|x). Through this appropriate Q(z|x) and generator, the reconstructed data is close to the original data; P x represents the probability density of the random variable X, P z represents the probability density of the random variable Z, P x,z , denote the joint probability density between random variables X and Z, Z and The joint probability density between ; c(X,G(z)) represents any measurable cost function.

[0126] Importantly, approximating the posterior Q(z|x) to the prior P(z) will cause different cell types to have the same distribution, thus producing latent codes close to each other, which will undoubtedly introduce bias. z (z)=∫Q(z|x)P x (x)dx enables different cell types to have different distributions, which can obtain more heterogeneous latent codes and thus reconstruct the gene expression matrix more accurately.

[0127] In general, if the data of the two modalities are represented as X1, X2, the latent variables of the two modalities are represented as Z1, Z2, and the reconstructed data is represented as Then the loss functions of the two modes based on optimal transmission can be combined and expressed as:

[0128]

[0129] Among them, L OT represents the combined loss function, MMD represents the maximum mean difference; P X is the prior probability distribution of mode X, P Y is the prior probability distribution of mode Y, Q(Z X |X) is the posterior probability distribution of the first mode, Q(Z Y |Y) is the posterior probability distribution of the second mode, k(z X ,·),k(z Y ,·) is a positive definite reproducing kernel function, H K is a real-valued function of the reproducing kernel Hilbert space, is the probability density of the random variable Z corresponding to the X variable, is the probability density of the random variable Z under the Y variable, is the marginal distribution of the random variable Z under the X variable, is the marginal distribution of the random variable Z under the Y variable; is the reconstructed data of mode Y.

[0130] In order to enable the model to learn inter-modal information and generate high-quality latent codes for the subsequent matching process, inter-modal similarities are learned through type-based contrastive learning during the training process. In this process, the embeddings of cells of the same type between modalities are brought closer, and the embeddings of different cell types are pushed further away.

[0131] In general, contrastive learning can be expressed as follows:

[0132] similarity(encoder(x i ),encoder(y i ))>>similarity(encoder(x i ),encoder(y j )),

[0133] Among them, x i ,y i is the data from the two modalities of the ith cell type, y j are the data from the two modalities of the jth cell type. i Relative to y iis a positive sample (consistent cell type), x j Relative to y i is a negative sample (the cell type is inconsistent). encoder(x i ) represents x i The feature matrix after the encoder, encoder(y i ) represents y i The feature matrix similarity(encoder(x i ),encoder(y i )) means encoder(x i ) and encoder(y i ) similarity.

[0134] The constructed loss function hopes to maximize the similarity between the multimodal samples of the same sample, while minimizing the similarity between the multimodal samples of different samples; it can be expressed as

[0135]

[0136] Among them, L InfoNCE represents the alignment loss, represents the alignment loss function from modality X to modality Y, represents the alignment loss function from modality Y to modality X,

[0137] τ is the temperature scale parameter. This step gives the latent space L of the two modalities X ,L Y .

[0138] 2.2) Driven by the optimal transmission theory, the maximum bipartite matching problem is transformed into an optimal transmission problem, and the soft matching matrix between modes is obtained.

[0139] By relaxing the constraints of the maximum weighted bipartite matching problem and converting it into an optimal transmission problem, we can finally achieve the effect of finding a soft match (it is not necessary for the modes to be one-to-one corresponding). Consider the maximum weighted bipartite matching problem in the form of integer linear programming to solve the problem that the original multimodal data may not be paired. The goal is to find the optimal transmission problem in two sets L. X ,L Y Find the best match in the bipartite graph to maximize the total weight of the matching edges (the weight matrix is ). It is represented as:

[0140] subject to:

[0141]

[0142] If you remove The integer constraint of , so that it can be any non-negative real number, take for And L X and L Y is a uniform distribution, then the probability density function is (n is the number of samples of X and Y); That belongs to L X ,L Y ; represents the real number weight, represents any measurable cost function, represents the joint probability density.

[0143] Then the above equation becomes an optimal transmission problem, and soft matching can be performed, that is, each Can be used with multiple match:

[0144] subject to:

[0145]

[0146] And introduce the entropy regularization term to make the transportation plan more decentralized:

[0147]

[0148] Among them, ε is a hyperparameter with a value of 0.01. Finally, the matching relationship matrix A between the two modalities can be obtained.

[0149] 2.3) Use the soft matching matrix to linearly combine the two latent spaces at the corresponding data to obtain an aggregated latent space; for the aggregated latent space, constrain the modal information after aggregation to be similar to that before aggregation, and the corresponding samples after aggregation are also as similar as possible.

[0150] By using the matching relationship matrix A between the two modalities, the latent spaces of different modalities are linearly aggregated to achieve the fusion of the latent spaces of different modalities at the alignment point, condense the information of the two modalities, and obtain the latent space C based on the aggregation. X ,C Y .

[0151] 2.4) Finally, the decoder composed of three layers of MLP is used for reconstruction. In this way, a multimodal data integration, matching and interpolation method is constructed; through continuous iterative learning, when the value of the loss function converges to the minimum, the model is saved. That is:

[0152] By defining the optimal transmission loss, alignment loss, and fusion loss, the pre-processed data is trained using the constructed model, and the loss function is converged to the minimum through multiple iterative learning, saving the best model parameters. Specifically:

[0153] 2.4.1) Set the parameters of the model, including the dimension of latent variables, the dimension of PCA dimensionality reduction, and the number of training iterations.

[0154] 2.4.2) Calculate the loss: The loss consists of the following components:

[0155] L Total =αL OT +βL InfoNCE +γL M +ξL A ,

[0156] Among them, L Total is the total loss function, α, β, γ, ξ are hyper parameters, L OT is the optimal transmission loss, L A and L InfoNCE is the alignment loss, L M is the combined loss. The specific calculation includes:

[0157] 2.4.2.1) In order to make the corresponding samples in the aggregated latent space have similar semantic information between different modalities, that is, to drive in the same direction during optimization, the alignment loss is used:

[0158]

[0159] Among them, C X ,C Y They represent the aggregated latent space of modality X and modality Y respectively, A represents the optimal matching matrix obtained by step 2.2), representing the pairing relationship between the two modalities; g is a hyperparameter, i.e., the dimension of the specified latent space; n X is the number of samples of mode X; Usually represents the l2 norm of the Frobenius norm.

[0160] 2.4.2.3) In order to make the aggregated latent space maintain the important structure of the original latent space, the preservation loss is used:

[0161]

[0162] Among them, L X ,L Y represents the latent space of X modality and Y modality, n Y is the number of samples of the Y mode;

[0163] Finally, through continuous iterative learning, when the value of the loss function converges to the minimum, the model is saved.

[0164] Step 3: Multimodal data integration, matching and interpolation:

[0165] 3.1) Load the trained model, extract the latent space, and train the K nearest neighbor classifier to classify the test data to detect the model effect.

[0166] 3.2) Input the latent space into the matching algorithm to obtain the matching relationship between multimodal data; cross-modal data interpolation can be performed by using the encoder of one modality and the decoder of another modality for one modality data.

[0167] 3.3) Users adjust network parameters according to their own data characteristics to achieve the best detection effect

[0168] from Figure 3 It can be seen that the present invention not only achieves the alignment of different modal latent spaces, but also successfully integrates multimodal latent spaces while retaining the most critical information in the data. Each type of cell is tightly clustered in the feature space, accurately capturing the internal and interrelationships of multimodal data. Figure 4 The visualization of the matching results shows that the model can accurately identify the correct matching sample pairs, showing high accuracy. Figure 5 The model further demonstrates its excellent performance in interpolation tasks.

[0169] The present invention proposes a multimodal data integration, matching and interpolation method driven by optimal transmission. A WAE framework driven by optimal transmission theory is constructed, and a comparative learning method based on sample type is introduced, which can effectively learn the similarities between modalities and generate high-quality latent codes. Compared with traditional methods, this invention can not only optimize the data integration and matching process, but also provide more accurate interpolation results when facing missing data. By using optimal transmission theory instead of the traditional maximum bisection matching algorithm, the resulting matching matrix is ​​more accurate, thereby improving the overall performance of the model and data processing efficiency.

[0170] The present invention has broad application prospects in multiple fields, especially in scenarios where complex multimodal data needs to be processed. In the integration of medical images and clinical data, the method can effectively integrate image data such as X-rays and MRI with clinical data such as the patient's medical history and laboratory test results, providing doctors with a more comprehensive diagnosis basis, thereby improving the accuracy of disease detection. For autonomous driving systems, the method can fuse data from different types of sensors such as cameras, radars, and lidars to generate a more accurate environmental model, improve the safety and navigation accuracy of autonomous driving, and especially when sensor data is partially lost or interfered with, it can make up for the missing data through interpolation technology to ensure stable operation of the system. In the field of sentiment analysis, the method can more accurately capture and understand the user's emotional state by integrating text, audio and video data, which is particularly critical for applications such as intelligent customer service and sentiment computing. In addition, in personalized recommendation systems, the present invention can integrate users' browsing history, purchase history and social media data to provide more accurate recommendations and improve user satisfaction. In general, the present invention has important practical application value and broad market prospects in the fields of medicine, autonomous driving, sentiment analysis and recommendation systems due to its excellent performance in integrating, matching and interpolating multimodal data.

[0171] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A multimodal data integration, matching and interpolation method based on optimal transmission drive, characterized by: The method is implemented based on the following steps: 1) Obtain multimodal training data: 1.1) The user selects the multimodal dataset for training locally and reads two datasets, Data X and Data Y, as well as related metadata information; 1.2) Random sampling of data, including random sampling of samples, feature dimensionality reduction using principal component analysis, and label processing; 1.3) Data preprocessing, including data standardization and missing value processing, to obtain two modalities Modality X and Modality Y from data sets Data X and Data Y; 2) Build the model: 2.1) Use a modality-specific joint encoder consisting of three layers of multi-layer perceptrons to learn the latent representations of the two modalities separately. The encoder is driven by optimal transfer theory and trained by contrastive learning methods to capture the key structures and features within the data and perform preliminary multimodal data alignment; 2.2) Driven by the optimal transmission theory, the maximum bipartite matching problem is transformed into an optimal transmission problem, and the soft matching matrix between modes is obtained; 2.3) Using the soft matching matrix, linearly combine the two latent spaces at the corresponding data to obtain an aggregated latent space; for the aggregated latent space, constrain the modal information after aggregation to be similar to that before aggregation, and the corresponding samples after aggregation are also similar; 2.4) Reconstruct the model through a decoder composed of three layers of MLP to build a multimodal data integration, matching and interpolation model; and save the model when the loss function converges to the minimum through continuous iterative learning; 3) Multimodal data integration, matching and interpolation: 3.1) Load the trained model, extract the latent space, and train the K nearest neighbor classifier to classify the test data to detect the model effect; 3.2) Input the latent space into the matching algorithm to obtain the matching relationship between the multimodal data; cross-modal data interpolation can be performed by using the encoder of one modality and the decoder of another modality for the data of one modality; 3.3) Users adjust network parameters according to their own data characteristics to achieve the best detection effect.

2. The optimal transmission driven multimodal data integration, matching and interpolation method according to claim 1, characterized in that: Step 2.1) Specific The following steps are involved: 2.1.1) Use encoder for feature extraction: Feature embeddings are generated using two modality-specific encoders; the encoder consists of three MLP layers, and the data enters the encoder as follows: In the datasets Data X and Data Y, for the modality Where n1 is the number of samples of mode X, m1 is the feature dimension of mode X; Where n2 is the number of modal Y samples, and m2 is the feature dimension of modal Y; then: X1=f1(ω1X+b1), X2=f2(ω2X1+b2), <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> X <h2 style=";text-align:left;direction:ltr"> =f3(ω3X2+b3), Y1=g1(η1Y+c1), Y2=g2(η2Y1+c2), L Y =g3(η3Y2+c3), Among them, L X and L Y is the result of the distribution of modality X and modality Y after the encoder, representing the final latent space; ω1, ω2, ω3, η1, η2, η3 are the weights in the network layer, b1, b2, b3, c1, c2, c3 are the bias terms in the network layer; X1 and X2 are the feature matrices obtained by modality X after the first and second layers of the encoder, respectively, and Y1 and Y2 are the feature matrices obtained by modality Y after the first and second layers of the encoder; f1(.), f2(.), f3(.) and g1(.), g2(.), g3(.) are all activation functions LeakyReLU, the subscript represents the i-th layer, i=1,2,3; 2.1.2) Adopting optimal transfer theory and category-based contrastive learning method to the latent space L X ,L Y To impose restraints; The optimal transport is to move from one distribution to another to obtain the minimum amount of movement, by finding an optimal solution π so that the distribution P(x) is The transfer cost is minimized, where P(x) is the probability function of X, for The probability function of is the reconstructed data of mode X; In the latent space Z, the decoder learns a mapping G such that z∈Z, the conditional probability density is That is, random variable The probability density under the condition of random variable Z = z; the following transforms the original optimal transmission problem into the expected L OT (P(x), express: in, is any measurable cost function, is the transportation plan that minimizes the total transportation cost. is the joint probability density; denote a single sample in the original data space and a single sample in the reconstructed data space, respectively, and Ε(·) denotes taking the expectation; Find all possible joint distributions So that its edge representation is: in, That is, random variable The probability density under the condition of random variable X = x, That is, the random variable X is in the random variable Probability density under the conditions; and Q(z)=∫Q(z|x)P(x)dx=P(z), P(x,z)=∫ z Q(x)Q(z|x)dz, The second equation is to ensure that the marginal distribution of the conditional distribution Q(z) is P(z), where P(z) is the probability function of the random variable Z; P(x,z) represents the joint distribution of X and Z, represents X and The joint distribution of, Q(x) represents the marginal probability density of X, Q(z|x) is the probability density of the random variable Z under the condition that the random variable X=x; Because the assumption Z does not change the marginal distribution under the condition of given X; the theorem transforms the original formula into: The last equation is because the joint distribution in π is replaced by the conditional distribution Q(z|x), thereby transforming the optimized coupling into the optimized random encoder Q(z|x). Through this appropriate Q(z|x) and generator, the reconstructed data is close to the original data, P x represents the probability density of the random variable X, P z represents the probability density of the random variable Z; P x,z , denote the joint probability density between random variables X and Z, Z and The joint probability density between ; c(X,G(z)) represents any measurable cost function; If the data of the two modalities are represented as X1, X2, the latent variables of the two modalities are represented as Z1, Z2, and the reconstructed data are represented as Then the loss functions of the two modalities based on the optimal transmission are combined and expressed as: Among them, L OT represents the combined loss function, MMD represents the maximum mean difference; P X is the prior probability distribution of mode X, P Y is the prior probability distribution of mode Y, Q(Z X |X) is the posterior probability distribution of the first mode, Q(Z Y |Y) is the posterior probability distribution of the second mode, k(Z X ,·)、k(Z Y ,·) is a positive definite reproducing kernel function, H K is a real-valued function of the reproducing kernel Hilbert space, is the probability density of the random variable Z corresponding to the X variable, is the probability density of the random variable Z under the Y variable, is the marginal distribution of the random variable Z under the X variable, is the marginal distribution of the random variable Z under the Y variable; is the reconstructed data of mode Y; In order to allow the model to learn inter-modal information and generate high-quality latent codes in the subsequent matching process, the training process learns inter-modal similarities through type-based contrastive learning. In this process, the embeddings of cells of the same type between modalities are brought closer, and the embeddings of different cell types are pushed further away; contrastive learning is expressed as follows: similarity(encoder(x i ),encoder(y i ))>>similarity(encoder(x i ),encoder(y j )), Among them, x i ,y i is the data from the two modalities of the ith cell type, y j are the data from the two modalities of the jth cell type; x i Relative to y i It is a positive sample, with the same cell type; x j Relative to y i It is a negative sample, the cell type is inconsistent; encoder(x i ) represents x i The feature matrix after the encoder, encoder(y i ) represents y i The feature matrix similarity(encoder(x i ),encoder(y i )) means encoder(x i ) and encoder(y i ), the constructed loss function hopes to maximize the similarity between the multimodal similarities of the same sample and minimize the similarity between the multimodal similarities of different samples; it is expressed as Among them, L InfoNCE represents the alignment loss, represents the alignment loss function from modality X to modality Y, represents the alignment loss function from modality Y to modality X, τ is the temperature scale parameter; This step gives us the latent space L of the two modalities X ,L Y .

3. The optimal transmission driven multimodal data integration, matching and interpolation method according to claim 1, characterized in that: Step 2.2) specifically includes the following steps: 2.2.1) By relaxing the constraints of the maximum weighted bipartite matching problem, it is converted into an optimal transmission problem, and finally the effect of finding a soft match is achieved; the maximum weighted bipartite matching problem in the form of integer linear programming is considered to solve the problem that the original multimodal data may not be paired. Its goal is to find the optimal transmission problem in two sets L. X ,L Y Find the best match in the bipartite graph to maximize the total weight of the matching edges; Its weight matrix is It is expressed as: If you remove The integer constraint of , so that it can be any non-negative real number, take for And L X and L Y is a uniform distribution, then the probability density function is n is the number of samples of X and Y; That belongs to L X ,L Y ; represents the real number weight, represents any measurable cost function, represents the joint probability density; Then the above equation becomes an optimal transmission problem, and soft matching can be performed, that is, each With multiple match: And introduce the entropy regularization term to make the transportation plan more decentralized: Among them, ε is a hyperparameter with a value of 0.01; finally, the matching relationship matrix A between the two modes is obtained.

4. The optimal transmission driven multimodal data integration, matching and interpolation method according to claim 3, characterized in that: In step 2.3), the matching relationship matrix A between the two modalities is obtained to linearly aggregate the latent spaces of different modalities to achieve the fusion of the latent spaces of different modalities at the alignment point, condense the information of the two modalities, and obtain the latent space C based on the aggregation. X ,C Y .

5. The optimal transmission driven multimodal data integration, matching and interpolation method according to claim 1, characterized in that: In step 2.4), by defining the optimal transmission loss, alignment loss and fusion loss, the pre-processed data is trained using the constructed model, and the loss function is converged to the minimum through multiple iterative learning, and the optimal model parameters are saved; specifically, it includes: 2.4.1) Set the parameters of the model, including the dimension of latent variables, the dimension of PCA dimensionality reduction, and the number of training iterations; 2.4.2) Calculate the loss; the loss consists of the following: L Total =αL OT +βL InfoNCE +γL M +ξL A , Among them, L Total is the total loss function, α, β, γ, ξ are hyper parameters, L OT is the optimal transmission loss, L A and L InfoNCE is the alignment loss, L M For combined loss.

6. The optimal transmission driven multimodal data integration, matching and interpolation method according to claim 5, characterized in that: The loss calculation in step 2.4.2) specifically includes: 2.4.2.1) In order to make the corresponding samples in the aggregated latent space have similar semantic information between different modalities, that is, to drive in the same direction during optimization, the alignment loss is used: Among them, C X ,C Y They represent the aggregated latent space of modality X and modality Y respectively, A represents the optimal matching matrix obtained by step 2.2), representing the pairing relationship between the two modalities; g is a hyperparameter, i.e., the dimension of the specified latent space; n X is the number of samples of mode X; Usually represents the l2 norm of the Frobenius norm; 2.4.2.3) In order to make the aggregated latent space maintain the important structure of the original latent space, the preservation loss is used: in, L X ,L Y represents the latent space of X modality and Y modality, n Y is the number of samples of the Y mode; Finally, through continuous iterative learning, when the value of the loss function converges to the minimum, the model is saved.

Images