A Multimodal Multi-Source Heterogeneous Data Fusion Method and System

Abstract

The present invention relates to the technical field of data fusion, and discloses a multi-modal multi-source heterogeneous data fusion method and system. Among them, a multi-modal multi-source heterogeneous data fusion method includes: constructing a multi-modal prototype network, extracting different modal features and establishing semantic mapping relationships between modalities; constructing a memory module, storing multi-modal representations of historical key examples through sample value evaluation; performing diversity sampling to retain rare abnormal patterns; implementing modal balance processing to dynamically adjust the weight of different modal gradient contributions; applying knowledge distillation to extract general fusion knowledge from data-rich scenarios to assist small-sample decision-making. The present invention solves the problems of low fusion efficiency in small-sample and cold-start scenarios, catastrophic forgetting in incremental learning, imbalance in modal learning rates, and retention of rare patterns under long-tailed distributions, and is applicable to scenarios such as predictive maintenance in manufacturing, personalized recommendation, medical health monitoring, and scenarios that require small-sample learning and incremental updates.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and more specifically, it relates to a multi-modal multi-source heterogeneous data fusion method and system. Background Art

[0002] With the rapid development of Internet of Things, big data, and artificial intelligence technologies, the fusion analysis of multi-source heterogeneous data has become the core technology of intelligent decision-making systems. In the fields of manufacturing predictive maintenance, personalized recommendation services, medical health monitoring, etc., it is necessary to simultaneously process multi-modal data from different sensors or channels, such as machine vibration signals, temperature data, sound recordings, image data, text records, etc., in order to obtain more comprehensive and accurate information.

[0003] Existing multi-modal data fusion methods mainly include techniques such as feature-level fusion based on deep neural networks, selective fusion based on attention mechanisms, and cross-modal mapping based on transfer learning. These methods perform well in data-rich scenarios, but still face multiple technical challenges in practical applications: in cold start scenarios such as new device deployment or new user access, the available labeled data is extremely limited, and traditional fusion methods are difficult to adapt quickly; in continuously running systems, the continuously generated new data will cause the model to over-adapt to new patterns and forget the important knowledge learned historically; the complexity and learning difficulty of different modal data (such as images, texts, sounds) vary significantly, which easily leads to certain modalities dominating the decision-making process during fusion and ignoring the key information of other modalities; real-world data usually follows a long-tail distribution, and the samples of key abnormal patterns (such as device fault features) are scarce and difficult to be effectively learned and identified.

[0004] The above problems severely restrict the application effect of multi-modal data fusion technology in small sample learning and incremental update scenarios, and a new fusion method that can quickly and effectively learn under limited data conditions, prevent catastrophic forgetting, balance the contributions of different modalities, and effectively protect rare patterns is needed. Summary of the Invention

[0005] The present invention provides a multi-modal multi-source heterogeneous data fusion method and system to solve the technical problems of small sample learning, catastrophic forgetting, modal imbalance, and rare pattern protection in related technologies.

[0006] The present invention provides a multi-modal multi-source heterogeneous data fusion method, including the following steps:

[0007] Construct a multi-modal prototype network, extract different modal features, and establish semantic mapping relationships between modalities;

[0008] Based on the multi-modal prototype network, construct a memory module, and store the multi-modal representations of historical key examples through sample value evaluation;

[0009] Leveraging memory modules, we perform diversity sampling to ensure that a wide range of data patterns are included while preserving rare, unusual patterns.

[0010] Based on the diversity sampling results, modal balance processing is implemented to dynamically adjust the contribution weights of different modal gradients;

[0011] Combined with the results of modal balance processing, knowledge distillation is applied to extract general fusion knowledge from data-rich scenarios to assist small sample decision-making.

[0012] In a preferred embodiment, the constructing of a multimodal prototype network includes:

[0013] Construct a corresponding feature extraction network for each modal data;

[0014] Compute prototype representations for each category of each modality;

[0015] Learn the mapping function between different modal feature spaces;

[0016] A model-independent meta-learning algorithm is used to optimize the entire prototype network.

[0017] In a preferred embodiment, the construction memory module includes:

[0018] Initialize the memory storage structure to save historical data samples and historical data sample feature representation;

[0019] Calculate the retention value of each candidate sample;

[0020] Implement a memory update mechanism based on sample value;

[0021] Introduce a time-based decay mechanism to adjust sample value.

[0022] In a preferred embodiment, performing diversity sampling includes:

[0023] Define the diversity measurement function of the sample set;

[0024] Implement a diversity sampling strategy based on entropy regularization;

[0025] Build abnormal mode protection mechanism;

[0026] Assign importance weights to the selected samples.

[0027] In a preferred embodiment, the performing of the modal balancing process comprises:

[0028] Assess the learning difficulty of each modality;

[0029] Calculate the contribution weight of each modal gradient;

[0030] Achieve balanced gradient updates;

[0031] Introduce modality-specific adaptive learning rates.

[0032] In a preferred embodiment, the application of knowledge distillation includes:

[0033] Construct a teacher-student knowledge distillation architecture;

[0034] Implement feature-level, relationship knowledge, attention, and output distillation;

[0035] Use a generative adversarial network to synthesize rare pattern samples;

[0036] Comprehensively apply the results of knowledge distillation and adversarial generation to update the target model.

[0037] In a preferred embodiment, the sample value evaluation is determined by calculating a weighted combination of the difference between the sample feature representation and the current prototype set and the information gain of the sample for the task, where the weight between the difference and the information gain can be dynamically adjusted.

[0038] In a preferred embodiment, the modality gradient contribution weight is calculated by a softmax function, which converts the learning difficulty index of each modality into the corresponding contribution weight to ensure that modalities with higher difficulty obtain more learning resources.

[0039] In a preferred embodiment, diversity sampling is achieved by optimizing an objective function that comprehensively considers the diversity metric of the sample batch, the entropy value of the sample batch, and the abnormality degree of the sample to ensure that the sampling results include both a wide range of data distributions and rare abnormal patterns.

[0040] In a preferred embodiment, a multi-modal multi-source heterogeneous data fusion system for performing a multi-modal multi-source heterogeneous data fusion method includes:

[0041] A multi-modal prototype network module for extracting different modality features and establishing semantic mapping relationships between modalities;

[0042] A memory storage module for storing the multi-modal representations of historical key examples through sample value evaluation;

[0043] A diversity sampling module for ensuring the inclusion of a wide range of data patterns and the retention of rare abnormal patterns;

[0044] A modality balance processing module for dynamically adjusting the modality gradient contribution weights;

[0045] A knowledge distillation module for extracting general fusion knowledge from data-rich scenarios to assist small-sample decision-making.

[0046] The beneficial effects of the present invention are as follows:

[0047] Significantly enhanced small-sample learning ability: The prototype network and meta-learning optimization strategy adopted in the present invention, combined with the hierarchical knowledge distillation technology, enable the model to quickly learn the mapping relationship between modalities from a small number of samples.

[0048] The catastrophic forgetting problem is effectively alleviated: Through the construction of a memory module and a time decay mechanism, the present invention can retain historical key knowledge during the continuous learning process.

[0049] Significantly improved modality balance performance: The modality balance processing mechanism based on modality difficulty assessment and dynamic adjustment of gradient contribution weights enables different modalities to be treated equally during the fusion process.

[0050] Outstanding rare pattern protection effect: The diversity sampling strategy and abnormal pattern protection mechanism, combined with the method of synthesizing rare samples using a generative adversarial network, improve the model's sensitivity to rare patterns.

[0051] Greatly improved computational efficiency: Compared with traditional deep fusion models, the student model of the present invention has fewer parameters while maintaining or improving the fusion performance. Description of the Drawings

[0052] Figure 1 is a flowchart of a multi-modal multi-source heterogeneous data fusion method of the present invention;

[0053] Figure 2 is a line graph comparing the small-sample learning performance of the present invention;

[0054] Figure 3 is a multi-bar graph and line graph of the historical knowledge retention rate during the incremental learning process of the present invention. Detailed Embodiments

[0055] Now, the subject matter described herein will be discussed with reference to example embodiments. It should be understood that discussing these embodiments is only to enable those skilled in the art to better understand and thus implement the subject matter described herein. Without departing from the scope of protection of the content of this specification, changes can be made to the functions and arrangements of the elements discussed. Each example can omit, substitute, or add various processes or components as needed. Additionally, the features described in some examples can also be combined in other examples.

[0056] In at least one embodiment of the present invention, a multi-modal multi-source heterogeneous data fusion method is disclosed, as Figure 1 shown, including the following steps:

[0057] Step 1, construct a multi-modal prototype network, extract different modality features, and establish a semantic mapping relationship between modalities;

[0058] Specifically, it includes the following sub-steps:

[0059] Step 1.1, Feature Extraction Network Construction;

[0060] For each type of modal data, construct a corresponding feature extraction network to map heterogeneous data of different modalities into a unified feature space. Suppose there are types of different modalities, then construct corresponding feature extraction networks , where , , respectively represent the feature extraction networks designed for modalities , , , and represents the total number of modalities.

[0061] For the input data of modality , through the feature extraction network obtain the feature representation:

[0062] ;

[0063] where is the feature representation vector of modality , representing the features obtained after converting the original data; is the feature extraction network corresponding to modality , including convolutional layers, recurrent layers, or Transformer layers, etc., and the specific structure is selected according to the characteristics of the data modality, and is used to map the original samples into the feature space; is the original input data of modality , which can be different types of data such as images, texts, audios, etc.; is the index of the modality, used to distinguish different data sources. For example, for the image modality, can be a pre-trained convolutional neural network; for the text modality, can be an encoder based on BERT.

[0064] Step 1.2, Modal Prototype Calculation;

[0065] For each category or task of each modality, calculate its prototype representation. Suppose for modality , category has support samples , where , , respectively represent the st, rd samples of category , , support samples, representing the number of samples of the category in the support set , the prototype representation of this category is calculated as follows:

[0066] ;

[0067] where, is the prototype representation of the category in modality , obtained by averaging the feature representations of all support samples of this category; represents the number of samples of the category in the support set; represents the -th support sample of the category in modality ; represents the feature extraction network corresponding to modality , used to map the original sample to the feature space; represents the feature representation obtained by passing the -th support sample through the feature extraction network. This method can effectively capture the core features of the category when the number of samples is limited, providing a stable category representation for few-shot learning.

[0068] Step 1.3, cross-modal mapping function learning;

[0069] Construct a cross-modal mapping function to establish a correspondence between the feature spaces of different modalities.

[0070] For the mapping between modality and modality , learn the mapping function :

[0071] ;

[0072] where, is the feature representation of modality predicted from the feature representation of modality , representing the target modality feature obtained after conversion by the mapping function; is the feature representation of the source modality , that is, the feature vector input to the mapping function; is the mapping function from modality to modality , used to convert the source modality feature space to the target modality feature space.

[0073] Mapping function It can be implemented as a multi - layer perceptron or an attention mechanism network, and its parameters are optimized by minimizing the distance between the predicted representation and the true representation:

[0074] ;

[0075] where, is the loss function for cross - modal mapping, used to measure the quality of the mapping effect; represents a pair of corresponding samples from different modalities, is the sample of modality ; is the sample of modality ; is the dataset containing the corresponding sample pairs of modality and modality ; and are the feature extraction networks of modality and modality respectively; is the mapping function from modality to modality ; is the distance metric function in the feature space, such as Euclidean distance or cosine distance, used to calculate the difference between the predicted feature and the true feature. This loss function optimizes the parameters of the cross - modal mapping function by minimizing the feature differences after mapping between all sample pairs.

[0076] Step 1.4, Meta - learning optimization;

[0077] Adopt the model - agnostic meta - learning algorithm to optimize the entire prototype network and improve its generalization ability under few - shot conditions. For each meta - learning task, divide the dataset into a support set and a query set . First, calculate the prototype representation based on the support set , and then evaluate the performance on the query set and update the model parameters.

[0078] The objective function of meta - learning is:

[0079] ;

[0080] where, represents the overall loss function of meta - learning, used to optimize the overall performance of the model on multiple tasks; represents the mathematical expectation symbol, indicating the average over all sampled tasks; is the task distribution, representing the set of all possible learning tasks; represents sampling from the task distribution Specific tasks sampled, including support sets and query sets; Denotes the support set, which contains a small number of labeled samples for the model to quickly adapt; Denotes the query set, which is used to evaluate the performance of the model on the current task; Denotes that the parameter is The feature extraction network, which is responsible for mapping the input data to the feature space; Denotes the set of learnable parameters of the model, which are optimized through the meta-learning process; Denotes the loss function calculated on the query set, which measures the performance of the model on the current task.

[0081] By optimizing this objective function, the model can learn how to quickly adapt to new tasks using a small number of samples, which is especially suitable for cold start scenarios. This method enables the model to have the ability to "learn how to learn" and can quickly adapt to new data distributions or task requirements with only a small number of samples.

[0082] The output of Step 1 is a trained multi-modal prototype network, which includes feature extractors for each modality, prototype calculation methods, and cross-modal mapping functions, laying a foundation for subsequent memory enhancement and diversity protection.

[0083] Step 2, based on the multi-modal prototype network, constructs a memory module to store the multi-modal representations of historical key examples through sample value evaluation;

[0084] Specifically, it includes the following sub-steps:

[0085] Step 2.1, initialize the memory storage structure;

[0086] Establish an explicit memory storage structure , which is used to save historical data samples and their feature representations.

[0087] The memory storage structure can be represented as a set of tuples:

[0088] ;

[0089] Among them, Denotes the entire memory storage structure, Denotes the original multi-modal data sample, which contains the original data content from different modalities (such as images, texts, audios, etc.); Denotes the feature representation of the sample, which is generated by the feature extraction network and is a high-dimensional feature vector obtained by converting the original data; Denotes the label or task-related information of the sample, which is the target value for supervised learning and evaluating the performance of the model; Denotes the timestamp when the sample is added to the memory, which is used to implement the time decay mechanism and track the "age" of the sample; Represents the usage frequency counter of a sample, recording the number of times the sample is accessed or used for training by the model, which affects the retention value of the sample; Represents the upper limit of the memory capacity, which is set according to the application scenario and computational resource constraints, and controls the maximum number of samples that the memory module can store.

[0090] To ensure the efficient use of the memory module, initialize the upper limit of the memory capacity And set up an index-based fast retrieval mechanism, using approximate nearest neighbor search algorithms such as Locality-Sensitive Hashing (LSH) to accelerate the retrieval process.

[0091] Step 2.2, sample value evaluation;

[0092] Calculate the retention value for each candidate sample , which measures the potential contribution of the sample to the model performance:

[0093] ;

[0094] Among them, Represents the sample Retention value score of, Represents the retention value weight, which is used to evaluate the potential contribution of the sample to the model performance; Represents the sample Feature representation of, which is a vector generated by the feature extraction network; Represents the set of prototypes stored in the current memory, representing the learned class centers; Represents the sample Information gain for the task, which measures the degree of improvement in the model's prediction ability after adding this sample; Represents the label space, which contains all possible class labels; Represents the sample feature representation And the set of prototypes in the current memory Degree of difference, calculated as the distance to the nearest prototype:

[0095] ;

[0096] Among them, Represents the sample feature representation And the set of prototypes in the current memory Degree of difference of, Represents the Euclidean distance between the sample feature and the prototype; Represents the sample Feature representation of, which is a vector generated by the feature extraction network; Represents the set of prototypes stored in the current memory, representing the learned class centers; Represents an operator for finding the minimum value in the prototype set ; Represents the prototype set ; a single prototype vector in it

[0097] The information gain of the sample pair for the task, measuring the improvement in the model's prediction ability after adding this sample:

[0098] ;

[0099] Among them, Represents the information gain of the sample pair for the task, is the entropy of the label distribution, representing the uncertainty of the label distribution; is the conditional entropy under the condition of the given sample feature representation, representing the uncertainty of the label distribution after knowing the sample features; Represents the label space, containing all possible class labels; Represents the sample ; the feature representation of the sample, a vector generated by the feature extraction network

[0100] Step 2.3, implementation of the memory update mechanism;

[0101] Implement the memory update mechanism, including strategies for adding new samples and eliminating old samples:

[0102] Adding new samples: For the new sample , if the memory is not full or its retention value is higher than the sample with the lowest retention value in the memory, then add it to the memory

[0103] Elimination strategy: When the memory reaches the capacity limit , adopt an elimination strategy based on the sample value, eliminating the sample with the lowest retention value:

[0104] ;

[0105] Among them, Represents the sample to be removed; Represents the independent variable when the objective function reaches the minimum value; Represents the samples in the memory module; Represents the entire memory storage structure; Represents the sample ; the retention value of the sample, jointly determined by the difference degree and the information gain

[0106] Memory integration: Regularly perform clustering analysis on the samples in the memory module, merge highly similar samples, and release space to save more diverse samples:

[0107] ;

[0108] Among them, represents the integrated memory module, which contains the sample set after clustering processing; represents the clustering algorithm function, which is used to merge or group similar samples; represents the original memory module, which contains all samples to be integrated; represents the sample similarity threshold, which is used to control the clustering granularity. A smaller threshold will generate more clusters, and a larger threshold will group more samples into the same class.

[0109] Step 2.4, time decay mechanism;

[0110] Introduce a time-based decay mechanism so that the memory module can adapt to changes in the data distribution while retaining long-term valuable knowledge.

[0111] For each sample in the memory, its value is adjusted over time:

[0112] ;

[0113] Among them, represents the adjusted retention value of sample after considering the time factor, which is updated over time; represents the initial retention value of sample , which is jointly determined by the difference degree and information gain; represents the current timestamp, indicating the time point when the system is currently running; represents the timestamp when sample is added to the memory, which is used to calculate the duration of the sample in the memory; represents the time decay factor, which controls the rate of sample value decay over time. The larger the value, the faster the decay; represents the time decay term, which decreases as the sample exists in the memory for a longer time; represents the usage frequency counter of sample , which records the number of times the sample is accessed or used for training by the model; represents the usage frequency influence factor, which controls the gain effect of usage frequency on the sample value. The larger the value, the more significant the frequency influence; represents the usage frequency gain term. Frequently used samples obtain value improvement, and the logarithmic function ensures that the gain does not grow infinitely; represents the natural logarithm function.

[0114] This time decay mechanism ensures that the memory module can gradually adapt to changes in the data distribution while retaining long-term valuable knowledge.

[0115] Step 3: Use the memory module to perform diversity sampling to ensure that a wide range of data patterns are included and rare abnormal patterns are retained;

[0116] Specifically, it includes the following sub-steps:

[0117] Step 3.1: Define diversity metrics;

[0118] Define a diversity metric function for the sample set , which is used to evaluate the extent to which the sample set covers the feature space:

[0119] ;

[0120] Among them, represents the sample set to be evaluated, which contains multiple data samples; represents the sample set in the th sample; represents the sample set in the th sample, and represents a sample that is not the same as ; represents the feature representation of the sample , that is, the vector that maps the original sample to the feature space; represents the feature representation of the sample ; represents the similarity function between feature representations, which is used to calculate the similarity degree of two feature vectors; represents the natural logarithm function, which is used to convert the similarity into a diversity metric; represents the diversity metric value of the sample set , and the larger the value, the higher the diversity of the sample set.

[0121] The similarity function can be the cosine similarity:

[0122] ;

[0123] Among them, represents the first feature vector, corresponding to , representing the representation of the i-th sample in the feature space; represents the second feature vector, corresponding to , representing the representation of the j-th sample in the feature space; represents the dot product of two vectors, representing the correlation between vectors, and the larger the value, the more similar the directions of the two feature vectors; 、 respectively represent the vectors and the vector of The norm (Euclidean norm), i.e., the vector length, is used to normalize the eigenvector; represents the cosine similarity function, with a value range of [-1, 1]. A value of 1 indicates that the directions of two vectors are exactly the same, a value of -1 indicates that the directions are exactly opposite, and a value of 0 indicates orthogonality (no correlation).

[0124] This metric function encourages the selection of samples that are significantly different from each other, thereby increasing the diversity of the sample set.

[0125] Step 3.2, implementation of the entropy regularization sampling strategy;

[0126] Implement the diversity sampling strategy based on entropy regularization, and select a subset of samples from the memory module such that this subset maximizes diversity while maintaining the representation of the original distribution:

[0127] ;

[0128] where, represents the sampled batch of samples, which is a subset of samples selected from the memory module; represents the optimal batch of samples, i.e., the subset of samples that maximizes the objective function; represents the entire memory storage structure, containing all stored samples; represents the batch size, specifying the number of samples to be selected; represents the diversity measure of the sample batch, used to evaluate the dispersion degree of the sample set in the feature space; represents the entropy of the sample batch, measuring the balance of the sample class distribution. The higher the value, the more balanced the class distribution; represents the first balance factor, used to adjust the relative importance of the diversity measure and entropy in the objective function; represents the independent variable when the objective function reaches its maximum value, i.e., finding the subset of samples that maximizes the expression within the square brackets.

[0129] ;

[0130] where, represents the entropy value of the sample batch used to measure the balance of the sample class distribution; represents the set of all possible classes, indicating all classes present in the data; represents the sample 's class label; represents the batch belonging to the class in the sample quantity; represents the batch The total number of samples; Indicates the category The proportion in the batch, that is, the probability of the category of.

[0131] ;

[0132] Among them, Indicates the optimally selected sample, that is, the sample that maximizes the objective function; Indicates the independent variable when the objective function reaches the maximum value; Indicates the sample from the memory module but not in the current batch ; Indicates adding the sample to the current batch to form a new batch; Indicates adding the sample the diversity measure of the batch after addition; Indicates adding the sample the entropy value of the batch after addition; Indicates the current batch the diversity measure of; Indicates the current batch the entropy value of; Indicates the second balance factor;

[0133] Add the selected sample to the batch ;

[0134] [[ID= 60]]Repeat steps 2 to 3 until the batch size reaches .

[0135] Step 3.3, Abnormal Pattern Protection Mechanism;

[0136] To ensure that rare but important abnormal patterns are protected, the following mechanism is introduced:

[0137] Error-Sensitive Resampling: For samples misclassified by the model, increase their weights in the sampling process:

[0138] ;

[0139] Among them, Indicates the importance weight of the sample used to adjust the probability of the sample being selected in the sampling process; Indicates the th sample in the memory module; Denotes the error sensitivity parameter, which controls the degree of increasing the weight for misclassified samples. The larger the value, the stronger the protection for misclassified samples. Denotes the indication operation, which takes the value of 1 when the model prediction result is inconsistent with the true label and 0 when they are consistent. Denotes the model's prediction label for the sample ; Denotes the true label of the sample ;

[0140] Cluster density awareness: Based on the sample density in the feature space, preferentially select samples in low-density regions (which may contain abnormal patterns):

[0141] ;

[0142] Among them, Denotes the cluster density index of the sample . The larger the value, the more likely the sample is in a low-density region (more likely to be an abnormal sample); Denotes the representation vector of the sample in the feature space; Denotes the representation vector of the sample in the feature space; Denotes the Euclidean distance between the sample and the sample in the feature space; Denotes the index set of the nearest neighbor samples of the sample in the feature space; Denotes the number of nearest neighbor samples to be considered, which is a preset hyperparameter; Denotes the summation of all nearest neighbor samples of the sample ; Denotes the normalization of the summation result to calculate the average distance.

[0143] Anomaly score calculation: Calculate the anomaly score for each sample:

[0144] ;

[0145] Among them, Denotes the anomaly score of the sample , which is used to quantify the anomaly degree and importance of the sample; Denotes the cluster density value of the sample , which reflects the sparsity of the sample in the feature space. The larger the value, the more likely the sample is in a low-density region (abnormal region); Denotes the sample Error-sensitive resampling weights assign higher weights to samples misclassified by the model.

[0146] Modify the objective function of diversity sampling to incorporate an outlier protection mechanism:

[0147] ;

[0148] where represents the optimal sample batch, i.e., the subset of samples that maximizes the objective function; represents the sampled sample batch, which is a subset of samples selected from the memory module; represents the entire memory storage structure, containing all stored samples; represents the batch size, specifying the number of samples to be selected; represents the diversity measure of the sample batch, used to evaluate the dispersion of the sample set in the feature space; represents the entropy of the sample batch, measuring the balance of the sample class distribution; represents the third balance factor, used to adjust the relative importance of the diversity measure and entropy in the objective function; represents the outlier protection intensity parameter, controlling the importance weight of outlier samples in the sampling process; represents the sample 's outlier score, reflecting the rarity and importance of the sample; represents the sum of the outlier scores of all samples in the batch, used to ensure that rare but important outlier patterns are selected into the batch; represents the independent variable when the objective function reaches its maximum value, i.e., finding the subset of samples that maximizes the expression within the square brackets.

[0149] Step 3.4, Sample importance weighting;

[0150] According to the results of diversity sampling, assign importance weights to each selected sample, which are used to adjust its contribution in the subsequent learning process:

[0151] ;

[0152] where represents the importance weight of the sample , indicating the degree of contribution of this sample in the subsequent learning process; represents the outlier score of the sample , calculated from the previous steps, reflecting the rarity and importance of the sample; represents the temperature parameter, controlling the smoothness of the weight distribution. A smaller value will make the weight distribution steeper, highlighting the importance of outlier samples; a larger The value will make the weight distribution smoother, ensuring that all samples can receive a certain degree of attention; Represents the exponential function, used to convert the anomaly score to a non - negative value; Represents the currently selected sample batch, containing all samples participating in the calculation; Represents the batch Sums all samples in it, used for normalization to ensure that the sum of all weights is 1.

[0153] The output of step 3 is a sample batch with optimized diversity and the corresponding sample importance weights , which maximizes the feature space coverage and class distribution balance, while protecting rare anomaly patterns, providing high - quality training data for subsequent modal balance processing.

[0154] Step 4, based on the diversity sampling results, implement modal balance processing, dynamically adjusting the weight of different modal gradient contributions;

[0155] Specifically, it includes the following sub - steps:

[0156] Step 4.1, modal difficulty assessment;

[0157] Evaluate the learning difficulty of each modality, based on the model's performance and convergence speed on that modality:

[0158] ;

[0159] Among them, Represents the learning difficulty index of modality , indicating the ratio of the current modality's learning difficulty to the historical average difficulty; Represents the loss of the current model on modality , Represents the part of the model on modality , Represents the sample set of the current batch; Represents the model in the previous training steps, at the step on modality historical loss value; Represents the size of the considered historical time window, used to calculate the average historical loss of modality ; Represents the average loss value of modality in the historical training steps; Represents a specific modality index, indicating a specific modality in multi - modal data (such as images, text, audio, etc.).

[0160] Learning Difficulty Index It reflects the ratio of the current loss to the historical average loss. A value greater than 1 indicates an increase in the current learning difficulty, and a value less than 1 indicates a decrease in the learning difficulty.

[0161] Step 4.2, Gradient Contribution Weight Calculation;

[0162] Based on the modal learning difficulty, calculate the contribution weights of the gradients of each modality:

[0163] ;

[0164] Among them, represents the gradient contribution weight of modality , indicating the contribution ratio of modality to the total gradient during the multi-modal fusion process; , respectively represent the learning difficulty indexes of modality and modality , reflecting the current learning difficulty of this modality. The larger the value, the more difficult the learning; represents the temperature parameter, which controls the smoothness of the weight distribution. A smaller value will make the weight distribution steeper (the more difficult-to-learn modality gets a higher weight), and a larger value will make the weight distribution more uniform; represents the total number of modalities, referring to the number of different data modalities included in the system (such as images, texts, audios, etc.); represents the exponential function, which is used to non-linearly map the difficulty index to the positive value range; represents the summation variable, which traverses all modalities for normalization to ensure that the sum of the weights of all modalities is 1.

[0165] This way of weight calculation will assign a larger gradient contribution weight to the modality with a higher learning difficulty, prompting the model to pay more attention to these difficult-to-learn modalities, thereby balancing the learning progress of different modalities.

[0166] Step 4.3, Modal Balanced Gradient Update;

[0167] Based on the calculated modal contribution weights, achieve balanced gradient update:

[0168] ;

[0169] Among them, represents the total gradient of the model parameters, indicating the final gradient value used to update the model parameters after integrating all modalities; represents the gradient contribution weight of modality , controlling the influence degree of this modality in the total gradient; represents the gradient generated by modality , indicating the gradient of the model parameters with respect to modality Partial derivatives of the loss function; represents the total number of modalities, which indicates the number of different data modalities contained in the system; Represents the summation of weighted gradients of all modes; Indicates the modal index, from 1 to , representing a specific data modality (such as image, text, audio, etc.).

[0170] In specific implementation, each mode The loss function The weighted summation gives the total loss function:

[0171] ;

[0172] in, Represents the total loss function, which is used to integrate the losses of all modalities and guide the update of model parameters; Indicates that all The summation operation is performed on each mode; Indicates the modal index, from 1 to , representing a specific data modality (such as image, text, audio, etc.); represents the total number of modalities, the number of different data modalities contained in the system; Indicates modality The gradient contribution weight determines the importance of the mode in the total loss; Indicates modality The loss function value reflects the performance of the model on this specific modality.

[0173] Then the gradient is calculated based on the total loss function and the model parameters are updated:

[0174] ;

[0175] in, represents the updated model parameters; represents the model parameters before the update, and represents the parameter values in the current training iteration; Represents the basic learning rate, which controls the step size of parameter updates; Represents the total loss function Model parameters The gradient of , which indicates the direction and magnitude of change of the loss function in the parameter space; Represents the weighted total loss function, which is composed of the loss functions of each modality according to their contribution weights.

[0176] Step 4.4, adaptive learning rate adjustment;

[0177] To further balance the learning processes of different modalities, a modality-specific adaptive learning rate is introduced:

[0178] ;

[0179] where, represents the adaptive learning rate of modality , a specific learning rate dynamically adjusted according to the learning difficulty of this modality; represents the base learning rate, the initial learning rate value shared by all modalities, serving as a benchmark for adaptive adjustment; represents the learning rate adjustment factor, controlling the magnitude of the learning rate adjustment according to the learning difficulty, and the larger the value, the more obvious the adjustment; represents the learning difficulty index of modality , indicating the ratio of the learning difficulty of the current modality to the historical average difficulty. A value greater than 1 indicates an increase in difficulty, and a value less than 1 indicates a decrease in difficulty.

[0180] When , the learning rate of modality will increase to accelerate the learning of difficult modalities;

[0181] When , the learning rate will decrease to avoid overfitting of simple modalities.

[0182] To ensure balanced learning among modalities, a regularization term is introduced to limit the performance differences between modalities:

[0183] ;

[0184] where, represents the modality balance regularization loss, used to reduce the differences in loss function values between different modalities; represents the regularization strength parameter, controlling the influence degree of the regularization term on the total loss. The larger the value, the more emphasis is placed on the balance between modalities; represents the loss function value of modality , reflecting the current learning state of modality ; represents the loss function value of modality , paired with to calculate the difference; represents the total number of modalities, the number of different data modalities included in the system; represents the square difference between the loss function values of modality and modality , punishing the performance differences between different modalities; represents double summation, calculating the total sum of the loss differences between all modality pairs.

[0185] This regularization term is added to the total loss function:

[0186] ;

[0187] Among them, represents the final total loss function, which is the optimization objective for model parameter update; represents the sum of all modal weighted losses; represents the modal balance regularization term, which is used to limit the performance difference between different modalities.

[0188] The output of step 4 is a gradient adjustment mechanism that achieves modal balance, including a modal difficulty evaluation method, a gradient contribution weight calculation formula, and an adaptive learning rate adjustment strategy, enabling the model to balance the processing of modal data with different difficulties and learning rates, avoiding a single modality from dominating the fusion process, and improving the comprehensiveness and accuracy of the fusion result.

[0189] Step 5, combining the modal balance processing results, applies knowledge distillation to extract general fusion knowledge from data-rich scenarios to assist small-sample decision-making;

[0190] Specifically, it includes the following sub-steps:

[0191] Step 5.1, constructing the teacher-student model;

[0192] Construct a teacher-student knowledge distillation architecture, where the teacher model is pre-trained on a data-rich source domain, and the student model is fine-tuned on a data-scarce target domain:

[0193] Teacher model training: Using data-rich source domain data Train the teacher model , so that it masters rich modal fusion knowledge:

[0194] ;

[0195] Among them, represents the optimal parameters of the teacher model; represents finding the parameters that minimize the following expression ; represents the loss function for a specific task; represents the parameter as of the model function; represents the data-rich source domain dataset.

[0196] Student model initialization: Initialize the student model using the same network structure as the teacher model but with fewer parameters , or use the same structure as the teacher model but apply regularization techniques such as dropout.

[0197] Step 5.2, implementing hierarchical knowledge distillation;

[0198] Implement hierarchical knowledge distillation, migrating the knowledge of the teacher model comprehensively from low-level features to high-level semantics:

[0199] Feature-level distillation: Align the feature representations of the teacher model and the student model at each level:

[0200] ;

[0201] Among them, represents the feature-level distillation loss function; represents the layer index of the network; represents the total number of layers of the network; represents the weight coefficient of the feature distillation of the -th layer, used to control the importance of feature distillation at different layers; represents the feature output of the teacher model for the input at the -th layer; represents the feature output of the student model for the input at the -th layer; represents the feature adaptation function, used to adjust the feature dimension of the teacher model to match that of the student model;

[0202] Relational knowledge distillation: Preserve the distance relationships between samples in the teacher model:

[0203] ;

[0204] Among them, represents the relational knowledge distillation loss, used to preserve the similarity relationships between samples; represents the Gram matrix generated by the teacher model; represents the Gram matrix generated by the student model; represents the distance metric function.

[0205] is the Gram matrix calculated based on sample features:

[0206] ;

[0207] Among them, represents the similarity between samples and ; represents the similarity calculation function, used to measure the similarity degree of two feature vectors; , respectively represent the model's outputs for the input samples and The extracted feature representation.

[0208] Attention knowledge distillation: Transfer the attention distribution of the teacher model:

[0209] ;

[0210] where, represents the attention knowledge distillation loss, represents the attention weight distribution of the teacher model for the input , represents the attention weight distribution of the student model for the same input , is the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions. By minimizing this divergence, the student model can learn the attention mechanism of the teacher model.

[0211] Output distillation: Learn the soft label output of the teacher model:

[0212] ;

[0213] where, represents the output distillation loss, which is used to measure the difference between the output distributions of the teacher model and the student model; represents the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions; represents the logits output of the teacher model (the raw prediction value without softmax); represents the logits output of the student model (the raw prediction value without softmax); represents the temperature parameter, which controls the smoothness of the soft label. A higher temperature value will produce a smoother probability distribution, which helps to transfer the similarity relationship between classes in the teacher model; represents the function that converts logits to a probability distribution.

[0214] Total distillation loss:

[0215] ;

[0216] where, represents the total distillation loss; represents the feature-level distillation loss function; represents the relational knowledge distillation loss; represents the attention knowledge distillation loss; represents the output distillation loss; , , , They respectively represent the weight coefficients of the feature-level distillation loss, relationship knowledge distillation loss, attention knowledge distillation loss, and output distillation loss. These weight coefficients jointly determine the relative importance of knowledge distillation at different levels.

[0217] The total training objective of the student model is:

[0218] ;

[0219] Among them, is the total loss function of the student model, is the loss function for a specific task, represents the student model, is the limited data in the target domain, is the weight coefficient of the distillation loss (used to balance the importance of the task loss and the distillation loss), is the knowledge distillation loss function defined above (including four parts: feature-level distillation, relationship knowledge distillation, attention knowledge distillation, and output distillation).

[0220] Step 5.3, the generative adversarial network synthesizes rare samples;

[0221] Introduce a generative adversarial network (GAN), and generate synthetic samples based on the existing rare pattern samples to expand the training data:

[0222] Generator construction: Construct a conditional generator , which accepts random noise and conditional information (such as class labels or key features) and generates synthetic samples.

[0223] Discriminator construction: Construct a discriminator , which determines whether the sample is a real sample or a generated synthetic sample.

[0224] Adversarial training: Optimize the generator and discriminator through a min-max game:

[0225] ;

[0226] Among them, represents the generator whose goal is to minimize the objective function, while the discriminator whose goal is to maximize the objective function, reflecting the game process of adversarial training; represents taking the expectation of samples sampled from the real data distribution ; represents the discriminator's evaluation of real samples The logarithmic probability output that the discriminator hopes to maximize; Denotes the expectation of joint sampling from the noise distribution and the conditional distribution ; Denotes the random noise vector input to the generator, which serves to generate diverse samples; Denotes the conditional information, such as class labels or key features, used to control specific attributes of the generated samples; Denotes the synthetic samples generated by the generator based on the noise and conditional information; Denotes the probability score of the discriminator for the generated samples, representing the probability that the discriminator believes the samples are real; Denotes that the generator hopes to minimize this term, i.e., hopes the discriminator misclassifies the generated samples as real samples.

[0227] Feature matching: Ensuring that the feature distribution of the generated samples is similar to that of the real samples:

[0228] ;

[0229] Among them, Denotes the feature matching loss, which is used to ensure the similarity between the generated samples and the real samples in the feature space; Denotes the feature extraction function of the teacher model, which is used to extract feature representations from the input samples; Denotes the real samples, which come from the original dataset; Denotes the generator network, which is used to generate synthetic samples; Denotes the random noise vector, which serves as the input source of the generator; Denotes the conditional information, such as class labels or key features, which is used to guide the generator to generate specific types of samples; Denotes the square of the L2 norm, which is used to calculate the squared Euclidean distance between two feature vectors and measure the degree of difference in the feature space.

[0230] Diversity enhancement: Encouraging the generator to produce diverse samples:

[0231] ;

[0232] Among them, Denotes the diversity loss function, which is used to encourage the generator to produce diverse samples; Denotes the number of samples in the memory bank; Denotes the number of all possible sample pairs; Denotes the th random noise vector; Denotes the th random noise vector; Represent conditional information (such as category labels or key features); Represents the generator based on Samples generated by noise vectors and conditional information; Represents the generator based on Samples generated by noise vectors and conditional information; Represents the distance metric function between samples, which is used to calculate the difference between two generated samples; the negative sign indicates that minimizing this loss is equivalent to maximizing the distance between generated samples, thereby increasing sample diversity.

[0233] Semantic consistency assurance: For multimodal data, ensure that the generated samples maintain semantic consistency across different modalities:

[0234] ;

[0235] in, Represents semantic consistency loss, which is used to ensure that samples generated by different modalities are semantically consistent; Indicates the total number of modes; and They are modal and modal Generator of; represents the random noise input to the generator; Represent conditional information, such as category labels or key features; and They are modal and modal feature extractor; Indicates that the modal To Modal The mapping function is used to transform the features of one modality into the feature space of another modality; Represents the square of the L2 norm, which is used to calculate the Euclidean distance between two feature vectors.

[0236] Step 5.4, knowledge synthesis and model updating;

[0237] The results of knowledge distillation and adversarial generation are comprehensively applied to target model update:

[0238] Fusion dataset construction: The original data, the data obtained by diversity sampling and the generated synthetic data are combined into an enhanced dataset:

[0239] ;

[0240] in, Represents the enhanced fusion dataset, which includes original data and various augmented data; The original dataset representing the target domain, which contains limited real samples; Represents a data subset selected through a diverse sampling strategy to ensure that various patterns and edge cases are included; Represents a rare sample dataset synthesized by a generative adversarial network, which is used to enhance the model's ability to recognize rare patterns; Represents a union operation.

[0241] Comprehensive training objectives:

[0242] ;

[0243] in, represents the final comprehensive training loss function used to optimize the student model; Represents the main loss function for a specific task, measuring the performance of the model on the augmented dataset; represents the student model, that is, the target model to be optimized; Represents an enhanced dataset, which includes original data, diverse sampled data, and generated synthetic data; The weight coefficient representing the knowledge distillation loss controls the strength of knowledge transfer from the teacher model; represents the knowledge distillation loss, which includes the combination of feature hierarchy, relational knowledge, attention, and output distillation; Represents the weight coefficient of the modal balance regularization term, which adjusts the influence of the modal balance mechanism; represents the modal balance regularization term, which is used to balance the contributions of different modes.

[0244] Progressive knowledge distillation: First distill general knowledge on a broader task, and then gradually focus on specific tasks:

[0245] ;

[0246] in, Represents the knowledge distillation loss weight that changes dynamically as training progresses; Represents the initial weight value of the knowledge distillation loss, which controls the intensity of the distillation process in the early stage of training; represents the base of natural logarithms; Represents the decay factor, which controls the rate at which the weight decreases over time. The larger the value, the faster the decay. represents the number of training steps and the progress of model training; this formula implements progressive knowledge distillation, which makes the model rely more on the knowledge of the teacher model in the early stage of training. As the training progresses, it gradually reduces its dependence on the teacher's knowledge and relies more on the target domain data for learning, thus achieving a smooth transition from teacher guidance to autonomous learning.

[0247] Model Fine-tuning and Update: Optimize the student model using the final training objective, and regularly update and save the model parameters:

[0248] ;

[0249] Among them, represents the parameters of the student model after update; represents the parameters of the student model before update; represents the base learning rate, which controls the step size of parameter update; represents the final loss function for the gradient of the model parameters indicating the direction and magnitude of parameter update.

[0250] The output of Step 5 is a fusion model enhanced by knowledge distillation and adversarial generation techniques. This model can effectively fuse multi-modal heterogeneous data in a small-sample scenario, while maintaining the ability to recognize rare patterns, providing reliable support for final decision-making analysis.

[0251] Application Example of This Embodiment:

[0252] To verify the effectiveness of the present invention, the following provides a detailed description of this method through an application example in a predictive maintenance scenario of the manufacturing industry.

[0253] A certain intelligent manufacturing factory needs to monitor the status and predict faults of core production equipment to avoid production losses caused by unplanned downtime. The core equipment of this factory is equipped with a variety of sensors to collect the following multi-modal data:

[0254] Vibration signal: Data from a triaxial acceleration sensor with a sampling rate of 20 kHz;

[0255] Sound data: Ambient sound recordings sampled at 16 kHz;

[0256] Temperature data: Temperatures of various parts of the equipment sampled once per second;

[0257] Infrared thermal imaging: Thermal images of the equipment at one frame per minute;

[0258] Operation logs: Equipment operation records and status information

[0259] This scenario faces the following challenges:

[0260] Lack of sufficient fault samples during the deployment of new equipment (cold start problem);

[0261] The fault types show a long-tailed distribution, and key fault examples are scarce;

[0262] The characteristics and acquisition rates of different sensor data vary significantly (modal imbalance problem);

[0263] As the device operating parameters are adjusted, the data distribution continuously changes (catastrophic forgetting problem).

[0264] Method implementation process:

[0265] Prototype network construction:

[0266] For different modal data, corresponding feature extraction networks are constructed:

[0267] Vibration signal: A network combining 1D-CNN and BiLSTM is used to map the original vibration signal into a 128-dimensional feature vector;

[0268] Sound data: A Mel spectrogram combined with the ResNet18 structure is used to extract 96-dimensional sound features;

[0269] Temperature data: 1D-TCN (temporal convolutional network) is applied to extract 64-dimensional temperature features;

[0270] Infrared thermal imaging: The lightweight MobileNetV3 is used to extract 128-dimensional image features;

[0271] Operation log: A variant of the BERT model is used to extract text features and compress them to 64 dimensions.

[0272] For the newly deployed Type A devices, there are only 5 faulty samples as the support set. Taking the vibration and temperature modalities as examples, first calculate the prototype representations for each fault type (normal, bearing fault, gear fault, etc.):

[0273] For the bearing fault class, calculate the vibration modality prototype:

[0274] ;

[0275] Then learn the mapping function from the vibration modality to the temperature modality , so that the temperature features predicted from the vibration features are as close as possible to the actual temperature features.

[0276] Through meta-learning optimization, the prototype network can identify different fault types with only 5 samples, achieving a detection accuracy of 85%, while the traditional method only has an accuracy of 52% under the same conditions.

[0277] Memory module construction and diversity sampling:

[0278] The initial capacity of the memory module is 500 samples, and samples with high retention value are preferentially saved. Taking the bearing fault class as an example, a special early fault sample has a high degree of difference ( ) and high information gain ( ) from the existing prototypes, and the retention value is calculated is preferentially stored in memory.

[0279] When adopting the entropy-regularized diversity sampling strategy, a sample set with a batch size of 32 is selected from historical data for training. By maximizing sample diversity and class distribution balance, it is ensured that the sample batches cover the entire fault evolution process from slight wear to severe damage.

[0280] For rare fault modes (such as pitting faults in the inner ring of bearings, which only account for 2% of the total fault samples), through the abnormal mode protection mechanism, its sampling probability is increased from the original 0.02 to 0.15, significantly enhancing the model's recognition ability for such faults.

[0281] Modal balance processing:

[0282] In the initial stage of training, the learning difficulty index of the model for vibration modes is significantly higher than that of temperature modes , indicating that vibration signals are more difficult to learn.

[0283] Through modal balance gradient adjustment, the gradient contribution weight of vibration modes is calculated as , which is much higher than that of temperature modes , making the model pay more attention to the difficult-to-learn vibration modes.

[0284] At the same time, an adaptive learning rate is applied. The learning rate of vibration modes is adjusted from the base value of 0.001 to 0.00145, while the learning rate of temperature modes is adjusted to 0.00083, further balancing the learning progress of different modes.

[0285] The results show that after modal balance processing, the difference in the utilization degree of vibration and temperature modal features by the model is reduced from the original 76% to 21%, and the fusion accuracy is improved by 24%.

[0286] Knowledge distillation application:

[0287] Train a powerful teacher model from data-rich Class B devices (with more than 2000 fault samples) and transfer the knowledge to the student model of the target Class A devices through hierarchical knowledge distillation.

[0288] In feature-level distillation, the student model successfully replicates 98.7% of the intermediate layer feature distributions of the teacher model. In relational knowledge distillation, 94.3% of the similarity relationships between samples are retained.

[0289] For extremely rare bearing inner ring fault modes (there are only 2 samples in Class A devices), 15 high-quality synthetic samples are synthesized through a generative adversarial network, and the detection rate of such faults is increased from the original 43% to 92%.

[0290] The generated synthetic samples maintain a high degree of semantic consistency among multiple modalities. The mutual prediction error between the vibration modality and the temperature modality is reduced by 78%, ensuring the authenticity and effectiveness of the synthetic data.

[0291] Verification of technical effects:

[0292] Verification of few-shot learning ability:

[0293] As shown in Table 1, the fault detection accuracies of this method and three mainstream fusion methods under different sample numbers are compared:

[0294] Table 1: Comparison of fault detection accuracies of this method and three mainstream fusion methods under different sample numbers;

[0295]

[0296] This method achieves an accuracy of 87.3% with only 5 samples, which is a 65.9% improvement compared to the best comparative method, verifying its excellent few-shot learning ability. More importantly, this method can achieve the performance level that other methods need 15 to 20 samples to reach with only 5 samples, significantly reducing the data collection cost.

[0297] Verification of rare pattern protection effect:

[0298] For four different fault types (sorted from most to least sample number), as shown in Table 2, the detection performances of each method are tested:

[0299] Table 2: Detection performance test of each method;

[0300]

[0301] It can be seen that this method still maintains a high detection rate of 91.4% for the rare fault pattern (pitting of the inner ring of the bearing) with only a 2% proportion, which is a 40.0% improvement compared to the average of the comparative methods, verifying its excellent rare pattern protection ability. More importantly, while protecting rare patterns, this method does not significantly sacrifice the detection performance of common patterns.

[0302] As Figure 2 、 Figure 3 shown, the comparison results of few-shot learning performance and the retention rate of historical knowledge during incremental learning are respectively presented.

[0303] The actual application effect shows that this method can establish an effective fault prediction model only 5 days after the initial deployment of the equipment, while traditional methods require more than 30 days of data accumulation. In addition, this method successfully predicted a key bearing fault, issued a warning 15 days in advance, and avoided a sudden shutdown accident that was expected to cause a production loss of 2 million yuan, fully verifying the practical value of the present invention.

[0304] In summary, the multi-modal multi-source heterogeneous data fusion method proposed in this embodiment comprehensively solves key technical problems such as few-shot learning, catastrophic forgetting, modality imbalance, and rare pattern protection by innovatively combining meta-learning, memory enhancement, and experience replay techniques, greatly improving the performance and application scope of multi-modal data fusion, and providing strong technical support for intelligent decision-making in fields such as industry, healthcare, and personalized recommendation.

[0305] The embodiments of the present invention have been described above, but these embodiments are not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative rather than restrictive. Under the inspiration of this embodiment, those of ordinary skill in the art can also make more equivalent embodiments in various forms, all of which fall within the protection scope of this embodiment.

Claims

1. A multi-modal multi-source heterogeneous data fusion method, characterized in that, It includes the following steps: Construct a multi-modal prototype network, extract different modal features and establish semantic mapping relationships between modalities: Construct a corresponding feature extraction network for each type of modal data; Calculate the prototype representation of each category for each modality; Learn the mapping function between different modal feature spaces; Optimize the entire prototype network using the model-agnostic meta-learning algorithm; Based on the multi-modal prototype network, construct a memory module, and store the multi-modal representations of historical key examples through sample value evaluation: Initialize the memory storage structure for saving historical data samples and the feature representations of historical data samples; Calculate the retention value of each candidate sample; Implement a memory update mechanism based on sample value; Introduce a time-based decay mechanism to adjust the sample value; Utilize the memory module to perform diversity sampling to ensure inclusion of a wide range of data patterns and retention of rare abnormal patterns; Based on the diversity sampling results, implement modal balance processing to dynamically adjust the contribution weights of different modal gradients; Combined with the results of modal balance processing, apply knowledge distillation to extract general fusion knowledge from data-rich scenarios to assist few-shot decision-making: Construct a teacher-student knowledge distillation architecture; Implement feature-level, relational knowledge, attention, and output distillation; Use a generative adversarial network to synthesize rare pattern samples.

2. The multimodal multi-source heterogeneous data fusion method according to claim 1, characterized in that, The execution of diversity sampling includes: Define the diversity metric function for the sample set; Implement a diversity sampling strategy based on entropy regularization; Construct an abnormal pattern protection mechanism; Assign importance weights to the selected samples.

3. A multimodal multi-source heterogeneous data fusion method according to claim 1, characterized in that The implementation of modal balance processing includes: Evaluate the learning difficulty of each modality; Calculate the contribution weights of the gradients of each modality; Implement balanced gradient updates; Introduce modality-specific adaptive learning rates.

4. A multimodal multi-source heterogeneous data fusion method according to claim 1, characterized in that The sample value evaluation is determined by calculating the weighted combination of the difference between the sample feature representation and the current prototype set and the information gain of the sample for the task, where the weight between the difference and the information gain can be dynamically adjusted.

5. A multimodal multi-source heterogeneous data fusion method according to claim 1, characterized in that The modal gradient contribution weights are calculated through a normalized exponential function, which converts the learning difficulty indicators of each modality into corresponding contribution weights.

6. A multimodal multi-source heterogeneous data fusion method according to claim 1, characterized in that, Diversity sampling is achieved by optimizing the objective function, which comprehensively considers the diversity metric of the sample batch, the entropy value of the sample batch, and the abnormality degree of the sample to ensure that the sampling results include both a wide range of data distributions and retain rare abnormal patterns.

7. A multimodal multi-source heterogeneous data fusion system for performing the multimodal multi-source heterogeneous data fusion method according to any one of claims 1-6, characterized in that, It includes: A multi-modal prototype network module for extracting different modal features and establishing semantic mapping relationships between modalities; A memory storage module for storing the multi-modal representations of historical key examples through sample value evaluation; A diversity sampling module for ensuring inclusion of a wide range of data patterns and retention of rare abnormal patterns; A modal balance processing module for dynamically adjusting the contribution weights of different modal gradients; A knowledge distillation module for extracting general fusion knowledge from data-rich scenarios to assist few-shot decision-making.

Images