Multi-source data fusion method and system based on optimal transmission algorithm

Abstract

The present application relates to the field of data processing technology, and in particular to a multi-source data fusion method and system based on an optimal transmission algorithm, the method comprising obtaining an input data source and extracting a feature vector of each data source; merging the feature vectors of each data source and performing a clustering operation to generate a prototype set, defining a joint weight distribution of multi-path jump paths by calculating the jump probability between a cross-modal vector and a vector in the prototype set, obtaining an entropy-weighted multi-path prototype bridging cost that fuses multiple semantic path costs based on the joint weight distribution; constructing a sparse optimal transmission matrix based on the optimal transmission algorithm and the entropy-weighted multi-path prototype bridging cost; and utilizing the sparse optimal transmission matrix to perform cross-modal weighted alignment and enhanced representation construction on the extracted feature vectors of each data source to generate a multi-modal fusion representation. The present application can reduce the consumption of computing resources while ensuring the accuracy and efficiency of data fusion.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-source data fusion method and system based on an optimal transmission algorithm. Background Art

[0002] In multi-source data fusion applications, especially when data comes from multiple heterogeneous sources, efficiently integrating these data in different forms and origins has become a key challenge in current research and application. With the continuous development of big data technologies and intelligent computing models, an increasing number of application scenarios require processing and fusing information from multiple data sources, such as sensors, images, text, and audio. While some classic methods in the field of data fusion, such as weighted averaging and collaborative filtering, can integrate multi-source data to a certain extent, they often overlook the complex interrelationships between data sources, resulting in low accuracy of fusion results and bottlenecks in computational efficiency when the data scale is large.

[0003] To overcome technical limitations, optimal transmission algorithms have been introduced into the field of data fusion. By minimizing the transmission costs between different data sources, optimal matching between them is achieved, significantly improving the quality of data integration in multi-source data fusion. The core advantage of optimal transmission algorithms lies in their ability to find the best data matching method by minimizing transmission costs, thereby eliminating differences between different data sources and optimizing error propagation during the data integration process.

[0004] However, in practical applications, optimal transmission algorithms also face challenges, especially when processing large amounts of data, where computational complexity and resource consumption remain prominent. Traditional optimal transmission algorithms often require full computation of all data points, which is computationally intensive and difficult to meet the demands of real-time processing. Therefore, the current challenge is to ensure the accuracy and efficiency of data fusion while reducing its consumption of computing resources. Summary of the Invention

[0005] This application provides a multi-source data fusion method and system based on an optimal transmission algorithm, which can reduce the consumption of computing resources while ensuring the accuracy and efficiency of data fusion. This application provides the following technical solutions:

[0006] In a first aspect, the present application provides a multi-source data fusion method based on an optimal transmission algorithm, the method comprising:

[0007] Obtaining an input data source, performing data preprocessing on the input data source and extracting a feature vector of each data source;

[0008] The feature vectors of each data source are combined and clustered to generate a prototype set. The joint weight distribution of multi-path jump paths is defined by calculating the jump probability between the cross-modal vector and the vectors in the prototype set. Based on the joint weight distribution, the entropy-weighted multi-path prototype bridging cost that integrates the costs of multiple semantic paths is obtained.

[0009] Constructing a sparse optimal transmission matrix based on the optimal transmission algorithm and the entropy-weighted multipath prototype bridging cost;

[0010] The sparse optimal transfer matrix is ​​used to perform cross-modal weighted alignment and enhanced representation construction on the feature vectors extracted from each data source, and a multimodal fusion representation is generated based on the Wasserstein centroid fusion strategy.

[0011] In a specific implementation scheme, preprocessing the input data source and extracting the feature vector of each data source includes:

[0012] For text data sources, the large-scale pre-trained language model RoBERTa is called to extract text feature vectors;

[0013] For image data sources, a multimodal pre-trained model based on contrastive learning is called to extract image feature vectors;

[0014] For video data sources, a large model architecture combined with time series modeling capabilities is called to extract video feature vectors. During the extraction process of video feature vectors, a temporal convolutional network is introduced as an auxiliary module.

[0015] In a specific implementation scheme, merging the feature vectors of each data source and performing a clustering operation to generate a prototype set, and defining the joint weight distribution of the multi-path jump paths by calculating the jump probability between the cross-modal vector and the vectors in the prototype set includes:

[0016] Merge the feature sets of all modalities into a feature pool, and use the K-means clustering algorithm to cluster the merged feature pool to generate a cluster center set. ;

[0017] For any pair of cross-modal vectors , introduce the prototype vector , construct the path , and calculate from arrive and from arrive The jump probability as well as as follows:

[0018] ;

[0019] ;

[0020] in, Represents a vector and vector The distance metric function between is the temperature parameter, and the The joint weights of the transit paths are as follows:

[0021] ;

[0022] in, .

[0023] In a specific implementation scheme, the entropy-weighted multi-path prototype bridging cost obtained by fusing multiple semantic path costs based on the joint weight distribution includes:

[0024] The calculation formula of the entropy-weighted multipath prototype bridging cost is as follows:

[0025] ;

[0026] in, It is The cumulative cost of the transit paths.

[0027] In a specific implementation scheme, the constructing of a sparse optimal transmission matrix based on the optimal transmission algorithm and the entropy-weighted multipath prototype bridging cost includes:

[0028] The two sets of modal feature sets to be connected are respectively recorded as as well as , using the approximate nearest neighbor technique to quickly screen the two sets of feature vectors to be docked, and return and Recent index collection and ;

[0029] According to the index collection and Constructing a sparse mask matrix :

[0030] ;

[0031] in, Represents a sparse mask matrix Middle Rank The elements of the column connect the sparse mask matrix with the entropy-weighted multipath prototype bridge cost Multiply element-wise to get a sparse cost matrix :

[0032] .

[0033] In a specific implementation scheme, the constructing of a sparse optimal transmission matrix based on the optimal transmission algorithm and the entropy-weighted multipath prototype bridging cost further includes:

[0034] With sparse cost matrix As input, together with the normalized marginal distributions of the two sets of modal eigenvectors and , together we construct the entropy regularized optimal transmission problem:

[0035] ;

[0036] Among them, the vector is the first set of modal features The normalized marginal distribution of The second set of modal features The normalized marginal distribution of is the transmission matrix, is the entropy regularization weight, and The lengths are and The set of real vectors of , is the transfer matrix The entropy of For a set of transmission matrices that satisfy row and column constraints, is the weighted total cost;

[0037] According to the sparse cost matrix Calculate the kernel matrix :

[0038] ;

[0039] Initialization Vector , , updated repeatedly through Sinkhorn iterations:

[0040] , ;

[0041] in, and is the scaling vector obtained by Sinkhorn iteration, is the kernel matrix The transposed matrix of and At the same time, the convergence conditions under the edge distribution constraints are met, and the sparse optimal transmission matrix is ​​obtained as follows:

[0042] ;

[0043] in, and They are and The diagonal matrix of .

[0044] In a specific implementation scheme, the method of performing cross-modal weighted alignment and enhanced representation construction on the extracted feature vectors of each data source using the sparse optimal transfer matrix, and generating a multimodal fusion representation based on the Wasserstein centroid fusion strategy includes:

[0045] Text feature vector collection , image feature vector set And the video feature vector set The corresponding sparse optimal transmission matrices are 、 as well as ;

[0046] The above sparse optimal transfer matrix is ​​used to bridge and weight the eigenvectors between the modalities, and the text and video information are fused into the image space. The weighted summary vector of the text and video in the image space can be calculated as follows:

[0047] ;

[0048] ;

[0049] in, is the sparse optimal transmission matrix Middle Rank Elements of the column, is the sparse optimal transmission matrix Middle Rank Elements of the column, In text mode feature vectors, In video mode feature vectors, and Representing the The image vector carries weighted information from the text and video modalities. In the same way, the weighted summary vectors of image and video, image and text can be calculated in the text and video spaces respectively. 、 、 、 ;

[0050] The original vector is concatenated or fused with the aligned vectors collected from other modalities to obtain a representation with enhanced cross-modal information in each modality. The fused vector of the image modality is:

[0051] ;

[0052] Constructed in the same way and ;

[0053] Using the Wasserstein centroid weighted aggregation strategy, all enhanced modal vectors are fused according to the preset weights to obtain a globally unified multimodal fusion representation:

[0054] ;

[0055] in, 、 、 The preset weight.

[0056] In a second aspect, the present application provides a multi-source data fusion system based on an optimal transmission algorithm, which adopts the following technical solutions:

[0057] A multi-source data fusion system based on an optimal transmission algorithm comprises:

[0058] A feature vector extraction module is used to obtain an input data source, perform data preprocessing on the input data source and extract a feature vector of each data source;

[0059] The feature vector clustering module is used to merge the feature vectors of each data source and perform clustering operations to generate a prototype set. The joint weight distribution of multi-path jump paths is defined by calculating the jump probability between the cross-modal vector and the vectors in the prototype set. Based on the joint weight distribution, the entropy-weighted multi-path prototype bridging cost that integrates the costs of multiple semantic paths is obtained.

[0060] An optimal transmission matrix construction module, configured to construct a sparse optimal transmission matrix based on an optimal transmission algorithm and the entropy-weighted multipath prototype bridging cost;

[0061] The multimodal feature fusion module is used to use the sparse optimal transfer matrix to perform cross-modal weighted alignment and enhanced representation construction on the feature vectors extracted from each data source, and generate a multimodal fusion representation based on the Wasserstein centroid fusion strategy.

[0062] In a third aspect, the present application provides an electronic device comprising a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement a multi-source data fusion method based on an optimal transmission algorithm as described in the first aspect.

[0063] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the storage medium stores a program, and when the program is executed by a processor, it is used to implement a multi-source data fusion method based on an optimal transmission algorithm as described in the first aspect.

[0064] In summary, the beneficial effects of this application include at least:

[0065] (1) By introducing the entropy-weighted multi-path prototype bridging cost and the sparse optimal transfer matrix, fine alignment and weighted fusion between different modalities are effectively achieved. Prototype clustering not only reduces the feature dimension and number, but also retains key semantic information, ensuring the accuracy of cross-modal matching. The sparse transfer matrix avoids the redundancy of full matrix calculation by screening key transfer paths, greatly enhancing the robustness and adaptability of the algorithm when facing large-scale heterogeneous data. The Wasserstein centroid fusion strategy combined with dynamic weights can fully integrate the complementary information of text, image and video modalities, significantly improving the expressive power of multimodal fusion representation, thereby providing a more reliable and accurate input basis for downstream tasks such as retrieval, classification and decision-making.

[0066] (2) By combining the entropy regularized optimal transmission algorithm with a sparse matrix construction strategy, the bottleneck of high computational complexity and high resource consumption in traditional cross-modal feature fusion is broken through. The design of the sparse cost matrix avoids the global calculation of all feature pairs, reduces the amount of computation and memory usage, and greatly improves the execution speed and real-time performance of the algorithm. The entropy weighting mechanism balances computational complexity and fusion quality, ensuring efficient calculation without sacrificing fusion effect. In addition, the overall solution adopts a modular design and is easy to deploy on edge devices or distributed systems, achieving efficient real-time fusion in large-scale, multimodal data environments, and meeting the high performance requirements under resource-constrained conditions in practical applications.

[0067] For three types of heterogeneous modal data, text, images, and videos, the high-dimensional feature vectors of each are first extracted through deep pre-training models to achieve unified feature mapping between modalities. To address the problem of high computational complexity in cross-modal feature alignment, prototype clustering technology is used to compress a large number of high-dimensional features into a small number of prototypes, and an entropy-weighted multi-path bridging strategy is used to calculate the cross-modal matching cost, which not only ensures the accuracy of matching but also significantly reduces the computational burden. Furthermore, a sparse cost matrix is ​​constructed through approximate nearest neighbor screening, and the sparse transfer matrix is ​​solved by combining the entropy-regularized optimal transfer algorithm, effectively reducing the consumption of computing resources while maintaining high matching accuracy. Finally, these sparse transfer matrices are used to realize cross-modal weighted information transfer and fusion, and a globally unified multimodal fusion representation is obtained through a dynamic weighting strategy. This method effectively solves the technical problem of significantly reducing computing resource consumption while ensuring data fusion accuracy and efficiency, and is suitable for the real-time fusion and application of large-scale complex data.

[0068] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and to implement it in accordance with the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flowchart of a multi-source data fusion method based on an optimal transmission algorithm in an embodiment of the present application.

[0070] Figure 2 It is a schematic diagram of the overall process of the multi-source data fusion method based on the optimal transmission algorithm in an embodiment of the present application.

[0071] Figure 3 It is a structural block diagram of a multi-source data fusion system based on an optimal transmission algorithm in an embodiment of the present application.

[0072] Figure 4 It is a block diagram of an electronic device for multi-source data fusion based on an optimal transmission algorithm in an embodiment of the present application. DETAILED DESCRIPTION

[0073] The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0074] Optionally, the present application uses the multi-source data fusion method based on the optimal transmission algorithm provided in each embodiment as an example for explanation in an electronic device, where the electronic device is a terminal or a server. The terminal can be a mobile phone, a computer, a tablet computer, etc. This embodiment does not limit the type of electronic device.

[0075] Reference Figure 1, is a flow chart of a multi-source data fusion method based on an optimal transmission algorithm provided by an embodiment of the present application. The method includes at least the following steps:

[0076] Step S101: Acquire input data sources, perform data preprocessing on the input data sources, and extract feature vectors of each data source.

[0077] In step S101, first, data sources from multiple heterogeneous inputs are obtained. In this application, heterogeneous input data sources refer to data sets from different modalities, different forms or different sources. It should be noted that in this application, different modalities mainly refer to text, images and videos. Subsequently, data preprocessing is performed on the input data source, including but not limited to data cleaning and denoising, normalization processing and data completion verification, such as removing abnormal pixels and duplicate frames from images, removing blank records and invalid characters from text, filtering damaged frames from videos and other data processing operations. After completing the above preprocessing, a deep embedding model is finally applied to each modality to extract high-dimensional feature vectors for each data source. For text data sources, the large-scale pre-trained language model RoBERTa is called to extract text feature vectors. As an optimized version of the BERT model, the RoBERTa model has stronger language understanding capabilities, can more accurately capture contextual semantic information, and provide high-quality language embedding representations for subsequent cross-modal alignment. For image data sources, a multimodal pre-trained model based on contrastive learning is called to extract image feature vectors, such as the CLIP or BLIP model. For video data sources, a large model architecture with time series modeling capabilities, such as VideoBERT or UniVL, is used to extract video feature vectors. Furthermore, a temporal convolutional network can be introduced as an auxiliary module during the video feature vector extraction process to enhance the ability to capture information in the temporal dimension. Finally, the extracted feature vectors for each data source are stored in the Milvus vector database.

[0078] Optionally, the models called for extracting the feature vectors of the data source are all existing model structures. This application can also use other methods for extracting feature vectors. This application does not limit the specific method of extracting feature vectors.

[0079] Step S102: Merge the feature vectors of each data source and perform clustering operations to generate a prototype set. Define the joint weight distribution of the multi-path jump path by calculating the jump probability between the cross-modal vector and the vectors in the prototype set. Based on the joint weight distribution, obtain the entropy-weighted multi-path prototype bridging cost that integrates the costs of multiple semantic paths.

[0080] In step S102, first call the text feature vector set stored in the vector database , image feature vector set And the video feature vector set Given such high-dimensional and large-scale data, directly constructing pairwise cost matrices between images and text, text and video, and images and video would be computationally intensive. Therefore, this application maps the original feature vector set to a smaller prototype set, significantly reducing the cost of constructing the cost matrix. In practice, the feature sets of all modalities can be merged into a single feature pool as follows:

[0081] , ;

[0082] Then, the K-means clustering algorithm is used to cluster the merged feature pool in the vector database to generate a cluster center set. ,in , that is, the prototype set, the prototype vector in each prototype set It represents a representative direction or central position in the semantic space and is used to establish a transit path between modalities. The prototype vector acts as an intermediary, eliminating the need to directly calculate all pairing costs of the vectors to be matched. Instead, a prototype jump mechanism is introduced to estimate the semantic matching cost between data.

[0083] Next, for any pair of cross-modal vectors , instead of directly calculating their matching costs, we introduce an intermediate jump point , construct a three-hop path , that is, first To a prototype vector , and then from arrive , use this indirect path to approximate the direct path semantic distance.

[0084] And calculate from the following formula respectively arrive and from arrive The jump probability as well as as follows:

[0085] ;

[0086] ;

[0087] in, Represents a vector and vector The distance measurement function between them can usually be Euclidean distance, cosine distance or Mahalanobis distance, etc. The specific selection depends on the nature of the feature space. It is a temperature parameter used to adjust the hardness of the softmax function.

[0088] However, there may be multiple prototypes, each path provides a possible explanation, so it is necessary to consider the importance of each path and then define the first The joint weights of the transit paths are as follows:

[0089] ;

[0090] The weight reflects the path The relative importance of the entire cross-modal docking and satisfying On this basis, the final cross-modal cost is no longer the distance of a single path, but the expected value of the cost on all paths, which is called the entropy-weighted multi-path prototype bridging cost. ,The calculation formula of entropy-weighted multipath prototype bridging cost is as follows:

[0091] ;

[0092] in, It is The cumulative cost of the transit paths, and the weight Then, a weighted average is performed on each path according to its credibility. This method combines the expected costs of all possible transfers. Compared to simple minimum or soft minimum methods, it can smoothly integrate information from multiple semantic pathways and reduce the interference of single-path noise on matching results. By constructing a shared prototype set and using the aforementioned path soft assignment mechanism, a computationally efficient and noise-resistant cross-modal matching cost estimation strategy is implemented. This strategy not only avoids the enormous burden of directly calculating the full pairing cost matrix, but also effectively integrates multi-path semantic information through an "entropy weighting" approach, laying a solid foundation for subsequent sparse optimal transfer matrix construction and cross-modal feature alignment.

[0093] Step S103: constructing a sparse optimal transmission matrix based on the optimal transmission algorithm and the entropy-weighted multipath prototype bridging cost.

[0094] In step S103, the goal is to maintain the entropy weighted multipath prototype bridging cost in step S102 To ensure high accuracy, we introduce the Optimal Transport (OT) algorithm to establish a sparse transport mapping relationship between cross-modal matches, and combine it with approximate nearest neighbor technology to further reduce computational complexity. This step not only completes the core optimization solution for cross-modal docking, but also enhances the selectivity of semantic mapping through sparse structure, constructing an efficient and discriminative sparse transport matrix.

[0095] First, following the completion of the entropy-weighted multi-path prototype bridging cost After the construction of , in order to further reduce the number of pairs involved in the optimal transmission calculation while maintaining the accuracy of the aforementioned entropy-weighted multipath cost, this scheme uses the approximate nearest neighbor (ANN) technology to quickly screen the two sets of feature vectors to be docked. Specifically, let the two sets of modal feature sets to be docked be respectively denoted as as well as In the vector database and Establish ANN index separately, through each The corresponding neighbor query quickly returns its nearest Index collection Similarly for each Returns the index collection The above index structure can use HNSW, IVF+PQ or other industrial-grade approximate search algorithms, which can be completed in milliseconds and maintain low latency even in scenarios with millions of vectors.

[0096] Then, according to the index collection and Constructing a sparse mask matrix :

[0097] ;

[0098] in, Represents a sparse mask matrix Middle Rank The elements of the column are used to represent the first Features Hedi Features , whether it is allowed to participate in the optimal transmission calculation as a matching pair, specifically, Indicates that this pair of features is considered to be one of the closest neighbors of each other in the approximate nearest neighbor query, so this pair is retained as a candidate matching pair. , it means that this pair of features is not in the corresponding neighbor index set, so it is masked and excluded from the matching calculation. The mask only retains all vector pairs within the neighbor range, and all other pairs are masked. Then, the sparse mask matrix is ​​combined with the entropy weighted multipath prototype bridge cost Multiply element-wise to get a sparse cost matrix :

[0099] ;

[0100] This thinning operation will be possible up to The matching pairs of records are reduced to approximately This greatly reduces the burden on the subsequent entropy regularized optimal transmission solution. It not only retains the multipath accuracy of the prototype bridging cost, but also filters out most of the irrelevant pairs under the effect of the approximate nearest neighbor mask.

[0101] Next, take the output sparse cost matrix As input, together with the normalized marginal distributions of the two sets of modal eigenvectors and , together we construct the entropy regularized optimal transmission problem:

[0102] ;

[0103] Among them, the vector is the first set of modal features The normalized marginal distribution of , which represents the weight or probability of each feature, vector The second set of modal features The normalized marginal distribution of , representing the weight or probability of each feature. is the transmission matrix, is the entropy regularization weight, and The lengths are and The set of real vectors . is the transfer matrix entropy. For a set of transmission matrices that satisfy row and column constraints, , ensuring the transmission matrix The row and column sums are equal to and ; is the weighted total cost, represents the first Rank Elements of the column, Represents the first Rank Elements of the column; is the entropy term, Used to control the degree of smoothing.

[0104] In the implementation, we first calculate the cost matrix Calculate the kernel matrix :

[0105] ;

[0106] Then initialize the vector , , and Respectively express and The summation operation of the modal features is repeatedly updated through Sinkhorn iteration:

[0107] , ;

[0108] in, and is the scaling vector obtained by Sinkhorn iteration, is the kernel matrix The transposed matrix of and At the same time, the convergence conditions under the edge distribution constraints are met, and the sparse optimal transmission matrix is ​​finally obtained as follows:

[0109] ;

[0110] in, and They are and The diagonal matrix of .

[0111] In practice, the entropy-weighted multi-path prototype bridging cost can more meticulously and comprehensively measure the distance relationship between different modal features, fully utilize multiple path information, avoid the deviation of single path calculation, and thus improve the accuracy of cross-modal matching. The sparse optimal transfer matrix uses approximate nearest neighbor mask screening to retain only feature pairs with high semantic relevance, filter out most irrelevant matches, and improve the model's sensitivity and discriminability for key semantic mappings. Traditional optimal transfer methods have high computational costs, especially in large-scale modal feature sets, where the amount of computation increases quadratically. Introducing approximate nearest neighbor screening to construct a sparse cost matrix significantly reduces the number of matching pairs involved in the calculation, reducing computing time and resource consumption.

[0112] Step S104: Use the sparse optimal transfer matrix to perform cross-modal weighted alignment and enhanced representation construction on the feature vectors extracted from each data source, and generate a multimodal fusion representation based on the Wasserstein centroid fusion strategy.

[0113] In step S104, the text feature vector set , image feature vector set And the video feature vector set The corresponding sparse optimal transmission matrices are 、 as well as , which represent the optimal transport weights between each mode.

[0114] In implementation, the sparse optimal transfer matrix is ​​first used to bridge and weight the feature vectors between the modalities. For example, to fuse text and video information into the image space, the weighted summary vector of text and video in the image space can be calculated as follows:

[0115] ;

[0116] ;

[0117] in, is the sparse optimal transmission matrix Middle Rank Elements of the column, is the sparse optimal transmission matrix Middle Rank Elements of the column, In text mode feature vectors, In video mode feature vectors, and Representing the The image vector carries weighted information from both text and video modalities. In the same way, the weighted summary vectors of image and video, image and text can be calculated in the text and video spaces respectively. 、 、 、 .

[0118] After completing the cross-modal transfer weighting, the original vector is concatenated or fused with the alignment vectors collected from other modalities to obtain a representation of each modality with enhanced cross-modal information. For example, the fused vector of the image modality is:

[0119] ;

[0120] Similarly, construction and , at this time, each enhancement vector That is, it integrates the semantics of the modality itself and the information of other modalities, preparing for the final fusion calculation. Then, using the weighted aggregation strategy of Wasserstein centroid, all enhanced modal vectors are fused according to the preset weights to obtain a globally unified multimodal fusion representation:

[0121] ;

[0122] Among them, the preset weight 、 、 It can be dynamically adjusted according to the reliability of each mode or its importance to the application scenario, ensuring that the global representation fully reflects the key modal information. This is the multimodal fusion representation required by this application, which can be directly input into the multimodal large model for retrieval, matching or decision support.

[0123] In addition, as a preference, in another feasible embodiment, the fusion calculation formula of the multimodal fusion representation may also be as follows:

[0124] ;

[0125] in, For modal Middle cross-modal augmentation vectors, For modal The overall fusion weight of is the local modal confidence coefficient, which is used to measure the reliability of the current vector for the fusion result in this mode. It is calculated as follows:

[0126] ;

[0127] in, Indicates modality and modal The weights in the sparse optimal transfer matrix, such as the transfer weights from text to image, is a normalization constant. A weighted aggregation strategy based on local confidence modulation is employed to introduce sample-by-sample dynamic confidence adjustment into each modal enhancement vector while maintaining controllable modal weights. Specifically, for each enhanced feature vector within a modality, a local confidence coefficient is estimated based on the sparse transfer strength between it and other modalities, and this coefficient is used as a weighting factor in the fusion calculation. The resulting global fusion representation not only integrates the semantic information of each modality but also has the ability to adaptively regulate differences in information reliability, further improving the expressive quality of cross-modal information integration.

[0128] In summary, combined with Figure 2, this application targets three types of heterogeneous modal data, text, image and video. First, the high-dimensional feature vectors of each are extracted through a deep pre-training model to achieve unified feature mapping between modalities. In order to solve the problem of high computational complexity in cross-modal feature alignment, prototype clustering technology is used to compress a large number of high-dimensional features into a small number of prototypes, and the matching cost between modalities is calculated based on the entropy-weighted multi-path bridging strategy, which not only ensures the accuracy of matching, but also significantly reduces the computational burden. Furthermore, a sparse cost matrix is ​​constructed through approximate nearest neighbor screening, and the sparse transmission matrix is ​​solved by combining the entropy regularized optimal transmission algorithm, which effectively reduces the consumption of computing resources while maintaining high matching accuracy. Finally, these sparse transmission matrices are used to realize cross-modal weighted information transfer and fusion, and a globally unified multi-modal fusion representation is obtained through a dynamic weighting strategy. This method effectively solves the technical problem of significantly reducing the consumption of computing resources while ensuring data fusion accuracy and efficiency, and is suitable for real-time fusion and application of large-scale complex data.

[0129] Figure 3 This is a structural block diagram of a multi-source data fusion system based on an optimal transmission algorithm provided by an embodiment of the present application. The system includes at least the following modules:

[0130] A feature vector extraction module is used to obtain input data sources, perform data preprocessing on the input data sources and extract feature vectors of each data source;

[0131] The feature vector clustering module is used to merge the feature vectors of each data source and perform clustering operations to generate a prototype set. The joint weight distribution of multi-path jump paths is defined by calculating the jump probability between the cross-modal vector and the vectors in the prototype set. Based on the joint weight distribution, the entropy-weighted multi-path prototype bridging cost that integrates the costs of multiple semantic paths is obtained.

[0132] Optimal transmission matrix construction module, used to construct a sparse optimal transmission matrix based on the optimal transmission algorithm and entropy-weighted multipath prototype bridging cost;

[0133] The multimodal feature fusion module is used to perform cross-modal weighted alignment and enhanced representation construction on the feature vectors extracted from each data source using a sparse optimal transfer matrix, and to generate a multimodal fusion representation based on the Wasserstein centroid fusion strategy.

[0134] For relevant details, please refer to the above method embodiment.

[0135] Figure 4 4 is a block diagram of an electronic device provided in one embodiment of the present application. The device includes at least a processor 401 and a memory 402.

[0136] Processor 401 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 401 may be implemented in hardware using at least one of the following: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content displayed on the display screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0137] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is executed by the processor 401 to implement the multi-source data fusion method based on the optimal transmission algorithm provided in the method embodiment of the present application.

[0138] In some embodiments, the electronic device may optionally include a peripheral device interface and at least one peripheral device. The processor 401, memory 402, and peripheral device interface may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface via a bus, signal lines, or circuit boards. Illustratively, the peripheral devices include, but are not limited to, radio frequency circuitry, a touchscreen display, audio circuitry, and a power supply.

[0139] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0140] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored. The program is loaded and executed by a processor to implement the multi-source data fusion method based on the optimal transmission algorithm of the above method embodiment.

[0141] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored. The program is loaded and executed by a processor to implement the multi-source data fusion method based on the optimal transmission algorithm of the above method embodiment.

[0142] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multi-source data fusion method based on an optimal transmission algorithm, characterized in that: The method comprises: Obtaining an input data source, performing data preprocessing on the input data source and extracting a feature vector of each data source; Merging the feature vectors of each data source and performing a clustering operation to generate a prototype set, defining a joint weight distribution of multi-path jump paths by calculating the jump probability between the cross-modal vector and the vectors in the prototype set, and obtaining an entropy-weighted multi-path prototype bridging cost that fuses multiple semantic path costs based on the joint weight distribution; merging the feature vectors of each data source and performing a clustering operation to generate a prototype set, and defining a joint weight distribution of multi-path jump paths by calculating the jump probability between the cross-modal vector and the vectors in the prototype set includes: Merge the feature sets of all modalities into a feature pool, and use the K-means clustering algorithm to cluster the merged feature pool to generate a cluster center set. ; For any pair of cross-modal vectors , introduce the prototype vector , construct the path , and calculate from arrive and from arrive The jump probability as well as as follows: ; ; in, Represents a vector and vector The distance metric function between is the temperature parameter, and the The joint weights of the transit paths are as follows: ； in, ; Constructing a sparse optimal transmission matrix based on the optimal transmission algorithm and the entropy-weighted multipath prototype bridging cost; The sparse optimal transfer matrix is ​​used to perform cross-modal weighted alignment and enhanced representation construction on the feature vectors extracted from each data source, and a multimodal fusion representation is generated based on the Wasserstein centroid fusion strategy.

2. The multi-source data fusion method based on the optimal transmission algorithm according to claim 1 is characterized in that: The performing data preprocessing on the input data source and extracting the feature vector of each data source comprises: For text data sources, the large-scale pre-trained language model RoBERTa is called to extract text feature vectors; For image data sources, a multimodal pre-trained model based on contrastive learning is called to extract image feature vectors; For video data sources, a large model architecture combined with time series modeling capabilities is called to extract video feature vectors. During the extraction process of video feature vectors, a temporal convolutional network is introduced as an auxiliary module.

3. The multi-source data fusion method based on the optimal transmission algorithm according to claim 1 is characterized in that: The entropy-weighted multi-path prototype bridging cost obtained by integrating multiple semantic path costs based on the joint weight distribution includes: The calculation formula of the entropy-weighted multipath prototype bridging cost is as follows: ； in, It is The cumulative cost of the transit paths.

4. The multi-source data fusion method based on the optimal transmission algorithm according to claim 3 is characterized in that: The constructing of a sparse optimal transmission matrix based on the optimal transmission algorithm and the entropy-weighted multipath prototype bridging cost includes: The two sets of modal feature sets to be docked are respectively recorded as as well as , using the approximate nearest neighbor technique to quickly screen the two sets of feature vectors to be docked, and return and Recent index collection and ; According to the index collection and Constructing a sparse mask matrix : ； in, Represents a sparse mask matrix Middle Rank The elements of the column connect the sparse mask matrix with the entropy-weighted multipath prototype bridge cost Multiply element-wise to get a sparse cost matrix : 。 5. The multi-source data fusion method based on the optimal transmission algorithm according to claim 4 is characterized in that: The constructing of a sparse optimal transmission matrix based on the optimal transmission algorithm and the entropy-weighted multipath prototype bridging cost further includes: With sparse cost matrix As input, together with the normalized marginal distributions of the two sets of modal eigenvectors and , together we construct the entropy regularized optimal transmission problem: Among them, the vector is the first set of modal features The normalized marginal distribution of The second set of modal features The normalized marginal distribution of is the transmission matrix, is the entropy regularization weight, and The lengths are and The set of real vectors of , is the transfer matrix The entropy of For a set of transmission matrices that satisfy row and column constraints, is the weighted total cost; According to the sparse cost matrix Calculate the kernel matrix : ； Initialization Vector , updated repeatedly through Sinkhorn iterations: ； in, and is the scaling vector obtained by Sinkhorn iteration, is the kernel matrix The transposed matrix of and At the same time, the convergence conditions under the edge distribution constraints are met, and the sparse optimal transmission matrix is ​​obtained as follows: ； in, and They are and The diagonal matrix of .

6. The multi-source data fusion method based on the optimal transmission algorithm according to claim 5 is characterized in that: The method of using the sparse optimal transfer matrix to perform cross-modal weighted alignment and enhanced representation construction on the feature vectors extracted from each data source, and generating a multimodal fusion representation based on the Wasserstein centroid fusion strategy includes: Text feature vector collection , image feature vector set And the video feature vector set The corresponding sparse optimal transmission matrices are 、 as well as ; The above sparse optimal transfer matrix is ​​used to bridge and weight the eigenvectors between the modalities, and the text and video information are fused into the image space. The weighted summary vector of the text and video in the image space can be calculated as follows: ； ； in, is the sparse optimal transmission matrix Middle Rank Elements of the column, is the sparse optimal transmission matrix Middle Rank Elements of the column, For the text mode feature vectors, In video mode feature vectors, and Representing the The image vector carries weighted information from the text and video modalities. In the same way, the weighted summary vectors of image and video, image and text can be calculated in the text and video spaces respectively. 、 、 、 ; The original vector is concatenated or fused with the aligned vectors collected from other modalities to obtain a representation with enhanced cross-modal information in each modality. The fused vector of the image modality is: ； Constructed in the same way and ; Using the Wasserstein centroid weighted aggregation strategy, all enhanced modal vectors are fused according to the preset weights to obtain a globally unified multimodal fusion representation: ； in, 、 、 The preset weight.

7. A multi-source data fusion system based on an optimal transmission algorithm, characterized in that: include: A feature vector extraction module is used to obtain an input data source, perform data preprocessing on the input data source and extract a feature vector of each data source; A feature vector clustering module is used to merge the feature vectors of each data source and perform a clustering operation to generate a prototype set, define the joint weight distribution of multi-path jump paths by calculating the jump probability between the cross-modal vector and the vectors in the prototype set, and obtain the entropy-weighted multi-path prototype bridging cost that integrates the costs of multiple semantic paths based on the joint weight distribution; the step of merging the feature vectors of each data source and performing a clustering operation to generate a prototype set, and defining the joint weight distribution of multi-path jump paths by calculating the jump probability between the cross-modal vector and the vectors in the prototype set includes: Merge the feature sets of all modalities into a feature pool, and use the K-means clustering algorithm to cluster the merged feature pool to generate a cluster center set. ; For any pair of cross-modal vectors , introduce the prototype vector , construct the path , and calculate from arrive and from arrive The jump probability as well as as follows: ; ; in, Represents a vector and vector The distance metric function between is the temperature parameter, and the The joint weights of the transit paths are as follows: ； in, ; An optimal transmission matrix construction module, configured to construct a sparse optimal transmission matrix based on an optimal transmission algorithm and the entropy-weighted multipath prototype bridging cost; The multimodal feature fusion module is used to use the sparse optimal transfer matrix to perform cross-modal weighted alignment and enhanced representation construction on the feature vectors extracted from each data source, and generate a multimodal fusion representation based on the Wasserstein centroid fusion strategy.

8. An electronic device, characterized in that: The device includes a processor and a memory; the memory stores a program, and the program is loaded and executed by the processor to implement the multi-source data fusion method based on the optimal transmission algorithm as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores a program, which, when executed by a processor, is used to implement a multi-source data fusion method based on an optimal transmission algorithm as claimed in any one of claims 1 to 6.

Images