Multi-modal feature dynamic fusion method based on uncertainty estimation
By independently modeling each modality using a Gaussian process regression model and dynamically adjusting the weights, the heterogeneity and missing data issues in multimodal data fusion are resolved, achieving high-precision and stable fusion results, and making it suitable for environments with limited computing resources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-03-31
AI Technical Summary
Existing multimodal data fusion methods face problems such as high heterogeneity, insufficient modal quality assessment, and unstable model prediction when data is missing. In particular, it is difficult to achieve high-precision fusion under conditions of limited computing resources and bandwidth.
A dynamic fusion method for multimodal features based on uncertainty estimation is adopted. Each mode is independently modeled by a Gaussian process regression model to capture nonlinear relationships, dynamically adjust mode weights, and perform weighted fusion using prediction uncertainty and data quality. The weights are also automatically adjusted when modes are missing.
It improves the prediction accuracy and stability of multimodal data fusion, enables real-time and efficient fusion under limited computing resources, adapts to data quality fluctuations and missing data, and provides reliable prediction results and uncertainty assessment.
Smart Images
Figure CN121765620A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing, and more specifically, to a method for dynamic fusion of multimodal features based on uncertainty estimation. Background Technology
[0002] With the development of the Internet of Things (IoT), smart devices, and sensor technologies, modern data acquisition methods can obtain various types of data from different sources. This data typically exists in multimodal form, encompassing multiple dimensions such as images, text, sound, and sensor data. In many practical applications, effective fusion and prediction based on this multimodal data has become crucial for improving the intelligence level of systems. For example, in the medical field, fusing multiple modalities such as imaging data, clinical records, and genetic data can provide more comprehensive information for disease prediction and diagnosis; in autonomous driving, fusing data from different sensors, such as radar, lidar, and camera data, can achieve more accurate obstacle detection and path planning.
[0003] Existing multimodal data fusion methods primarily rely on deep learning techniques, typically inputting data from different modalities into a unified model for training. However, these methods often face the following problems: First, multimodal data usually exhibits significant heterogeneity, with substantial differences in feature spaces between different modalities, making direct fusion prone to information loss or inconsistency; second, traditional methods lack effective assessment of the data quality for each modality, failing to effectively mitigate the impact of low-quality modalities when dealing with modality quality differences; finally, existing technologies often fail to adequately account for missing data, easily leading to model instability and decreased accuracy when some modal data is missing.
[0004] These shortcomings limit the accuracy of multimodal data fusion. Especially under conditions of limited computing resources and bandwidth, there is an urgent need to develop dynamic adaptive multimodal data fusion methods that can achieve high-precision data fusion even under limited computing resources and bandwidth. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for dynamic fusion of multimodal features based on uncertainty estimation.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] A dynamic fusion method for multimodal features based on uncertainty estimation includes the following steps:
[0008] Step S1: Preprocess the data from different modalities and convert them into a unified format to eliminate differences between modalities;
[0009] Step S2: Independently establish a Gaussian process regression model for each mode to capture the nonlinear relationships in the modal data;
[0010] Step S3: Use the trained Gaussian process regression model to predict each mode and output the predicted value and uncertainty measure.
[0011] Step S4: Dynamically adjust the modality weights based on the uncertainty measure and data quality of each modality prediction, and use the weights to perform a weighted calculation on the predicted values of each modality to obtain the final prediction result.
[0012] Furthermore, in step S1, the preprocessing of different modal data includes standardization and normalization, encoding, missing data processing, and outlier detection and processing.
[0013] Furthermore, in step S1, the modal data are integrated by splicing or weighted fusion after preprocessing.
[0014] Furthermore, step S2 includes the following steps:
[0015] Step S21: The preprocessed modal data are used to form training datasets for each modality. Training dataset for each modality , It is the input data. This corresponds to the target value;
[0016]
[0017] In the formula, For input data The potential function value; Noise term;
[0018] Step S22: A radial basis function kernel is used as the covariance function to capture the smooth nonlinear relationship of the input data;
[0019] The radial basis function kernel has the following form:
[0020]
[0021] In the formula, and For any two data points; It is a length scale parameter hyperparameter;
[0022] Step S23: The optimal hyperparameters are obtained by maximizing the marginal likelihood function;
[0023] The marginal likelihood function has the following form:
[0024]
[0025] In the formula, This is the noise variance hyperparameter; For input data The covariance matrix between them; The identity matrix is T; T is the transpose operator. This represents the number of samples.
[0026] Furthermore, step S3 includes the following steps:
[0027] Step S31, for the first Input data for each modality and the corresponding target value Construct the joint distribution using Gaussian process regression:
[0028]
[0029] In the formula, Indicates input data The potential function value; For input data The covariance matrix between them; and These are transposes of each other, and are used as test data. and input data The covariance matrix between them; This is the noise variance hyperparameter; It is the identity matrix; The symbol for a multivariate Gaussian distribution;
[0030] Step S32: Use the joint distribution to obtain the predicted values of the Gaussian process regression. :
[0031]
[0032] In the formula, For test data With input data The covariance matrix between them; For input data The inverse of the covariance matrix;
[0033] Step S33, based on test data Predicted value Calculate its forecast uncertainty measure :
[0034]
[0035] In the formula, For the first Test data for each modality A measure of predictive uncertainty; For test data The covariance matrix between them; For test data and input data The covariance matrix between them; For input data The covariance matrix; T is the transpose operator.
[0036] Furthermore, step S4 includes the following steps:
[0037] Step S41: Calculate the quality metric of the mode based on the predicted value of the mode and the uncertainty metric;
[0038]
[0039] In the formula, For the first A quality metric for each modality; For the first Test data for each modality A measure of predictive uncertainty; It is a constant;
[0040] Step S42: Convert the quality metrics into weights, ensuring that the sum of all weights is 1;
[0041]
[0042] In the formula, The total number of modes; It is the sum of all modal quality measures; For the first The weights of each modality;
[0043] Step S43: Perform weighted fusion to obtain the final fusion prediction result;
[0044]
[0045]
[0046] In the formula, For the first Predicted values for each modality; This is the final fusion prediction value; This is a measure of the uncertainty of the final fused prediction.
[0047] Furthermore, the availability of data for each modality is monitored in real time. When data for one modality is missing, the weight of that modality is automatically set to zero, while the weights of other modalities are adjusted.
[0048] Furthermore, with each new data input, the weight of each modality is calculated in real time, and the weights are adjusted based on the real-time calculated measurement of prediction uncertainty.
[0049] The beneficial effects of this invention are:
[0050] 1. In this invention, Gaussian process regression is used to independently model each modality. On the one hand, it can fully explore the characteristics of each modality, flexibly capture the nonlinear relationship between it and the target variable, and effectively improve the prediction accuracy of multimodal data fusion. Especially when dealing with complex, high-dimensional data, it can provide more accurate prediction results than traditional methods. On the other hand, it makes the prediction results of each modality not only output the predicted value, but also the corresponding uncertainty assessment, which increases the interpretability and reliability of the model.
[0051] 2. In this invention, the heuristic weighted fusion mechanism is a dynamic weighting mechanism based on the uncertainty measurement and data quality provided by Gaussian process regression. The weight of each modality is automatically adjusted according to its prediction uncertainty measurement and data quality, thereby ensuring the stability and high quality of the multimodal data fusion results.
[0052] 3. This invention employs the sparse approximation technique of Gaussian process regression model, which reduces the complexity of model calculation, enabling the method to operate efficiently under conditions of large-scale datasets and limited computing resources. By optimizing the calculation process, it can achieve real-time multimodal data fusion and meet the needs of real-time prediction. Attached Figure Description
[0053] Figure 1 This is a flowchart of a multimodal feature dynamic fusion method based on uncertainty estimation in this embodiment;
[0054] Figure 2 This is a flowchart of multimodal data preprocessing in this embodiment;
[0055] Figure 3 This is a flowchart illustrating the combination of Gaussian process regression and heuristic weighted fusion mechanism in this embodiment;
[0056] Figure 4 This is a flowchart of a heuristic weighted fusion mechanism in this embodiment;
[0057] Figure 5 This is a flowchart of multimodal data missing and compensation processing in this embodiment. Detailed Implementation
[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0059] Example: A method for dynamic fusion of multimodal features based on uncertainty estimation, such as... Figures 1-5 As shown, this method models each mode independently through Gaussian process regression, thereby avoiding information loss or inconsistency that may occur when directly fusing data from different modes. Gaussian process regression can flexibly capture the nonlinear relationships between different modes and improve the accuracy of prediction.
[0060] Secondly, this method proposes a heuristic weighted fusion mechanism, which dynamically calculates the weight of each modality and weights it according to the prediction uncertainty metric and data quality of the modality. This makes the contribution of high-quality modalities to the final prediction result greater, while the influence of low-quality modalities is effectively reduced, thereby ensuring the reliability and stability of the fusion result.
[0061] Furthermore, when faced with missing data, the modality weights are automatically adjusted through a heuristic weighted fusion mechanism. When a modality is missing, the data is not discarded directly, but compensated based on the prediction results of other modalities, ensuring the robustness and accuracy of the model.
[0062] This method employs a sparse approximation method based on Gaussian process regression, which reduces the computational complexity of the model, enabling it to run efficiently on large datasets and adapt to the needs of real-time prediction, especially suitable for scenarios with limited computing resources.
[0063] Specifically, the method includes the following steps:
[0064] Step S, as Figure 2 As shown, data from different modalities are preprocessed and converted into a unified format to eliminate differences between modalities.
[0065] Because multimodal data comes from a wide range of sources, including numerical, categorical, image, and text data, and each modality has different data characteristics and formats, it is necessary to preprocess the data from different modalities to transform them into a unified format and scope to facilitate subsequent modeling and prediction, and ensure effective data fusion.
[0066] In this embodiment, preprocessing includes standardization and normalization, encoding, missing data handling, and outlier detection and handling.
[0067] 1.1 Standardization and Normalization Processing
[0068] In multimodal data, numerical data often have different dimensions or ranges; for example, some modal data may represent temperature in degrees Celsius, while other modal data may represent weight in grams. Because of the inconsistency in dimensions and ranges, directly processing this data may lead to biased results. Therefore, this embodiment processes the numerical data using standardization or normalization methods.
[0069] Standardization typically adjusts data according to its mean and standard deviation, making the mean 0 and the standard deviation 1. Normalization, on the other hand, linearly transforms the data according to its minimum and maximum values, mapping it to a specified range (usually 0 to 1).
[0070] The choice between standardization and normalization depends on the characteristics of the data and is typically applied to numerical data. Through processing, dimensional differences in the data are eliminated, ensuring the comparability of data from different modalities.
[0071] 1.2 Encoding Processing
[0072] For categorical data (such as text labels or discrete category data), directly inputting it into the model is inappropriate. To enable the algorithm to process this data, it needs to be encoded. Common encoding methods include one-hot encoding and label encoding.
[0073] One-hot encoding and label encoding can transform non-numerical data into numerical data suitable for algorithm processing. For text data, methods such as the bag-of-words model, TF-IDF, and Word2Vec can be used for processing; the specific choice depends on the size of the text data and the task requirements.
[0074] 1.3, Missing Data Handling
[0075] Missing data is a common phenomenon in multimodal data processing. Some modal data may be missing due to sensor malfunction, data loss, acquisition delay, or other reasons. This embodiment uses interpolation and data from other modalities to supplement the missing data.
[0076] Interpolation is a common method for imputing missing data. Common interpolation methods include linear interpolation and spline interpolation. For time series data, linear interpolation is a simple and commonly used method that estimates missing values based on the linear relationship between two adjacent data points.
[0077] Furthermore, this embodiment also employs a supplementation method based on data from other modalities. If data for a certain modality is missing, it can be supplemented using data from other modalities. For example, by using multimodal data fusion, the prediction results of other modalities can be used to fill in the missing data. The specific implementation of this method depends on the correlation between each modality, and is usually accomplished by training a regression model or using Gaussian process regression.
[0078] 1.4 Outlier Detection and Handling
[0079] Multimodal data may contain outliers, which may be caused by sensor malfunctions or data entry errors. To improve the robustness of the model and prevent outliers from having an excessive impact on the prediction results, outlier detection is necessary.
[0080] This embodiment uses a statistical method to detect outliers. For numerical data, boxplots can be used to identify outliers, typically defined as values exceeding 1.5 times the upper or lower quartile. For categorical data, outliers can be identified by the distribution of category frequencies; if a category's frequency is too low, an outlier may be present.
[0081] Once outliers are detected, they can be processed by methods such as deleting them, replacing them with the median or mean, depending on the characteristics of the data and the requirements of the task.
[0082] Data from all modalities is preprocessed and then integrated into a unified feature vector. Data from each modality is converted to a uniform format and dimension, and the preprocessing results from all modalities are merged into a multidimensional vector for input into subsequent models for prediction.
[0083] For data from different modalities, concatenation or weighted fusion are typically used for integration. Concatenation involves piecing together the processing results of each modality into a large vector in dimensional order. Weighted fusion, on the other hand, adds corresponding weights to the feature vectors of each modality to indicate the importance of each modality in the final prediction.
[0084] By employing various preprocessing techniques, including standardization, normalization, encoding, missing data handling, and outlier detection, data from different modalities can be effectively processed uniformly, eliminating intermodal differences and providing reliable input for subsequent Gaussian process regression and weighted fusion. These preprocessing techniques ensure data consistency and reliability, thus laying the foundation for the successful implementation of multimodal data fusion.
[0085] Step S2, as follows Figure 3As shown, a Gaussian process regression model is independently built for each modality to capture the nonlinear relationships in the modal data. The final prediction is achieved by fusing the regression results from different modalities. Gaussian process regression can not only model the nonlinear relationships in the data, but also provide an assessment of the uncertainty of the prediction results, which is helpful for the subsequent weighted fusion process.
[0086] Specifically, step S2 includes the following steps:
[0087] Step S21, Prepare training data
[0088] Each preprocessed modality data is then used to form its own training dataset; for the first... Training dataset for each modality , It is the input data. This corresponds to the target value;
[0089]
[0090] In the formula, For input data The potential function value; For noise terms, , This is the noise variance hyperparameter. It follows a multivariate Gaussian distribution;
[0091] Step S22, Configure the Gaussian process regression model
[0092] For the Input data for each modality and target value Modeling it using Gaussian process regression, assuming that... For each sample, the input data The target value is After modeling using Gaussian process regression, a model based on the input data can be obtained. and target value The probability distribution, thus allowing for the analysis of new input data. (i.e., test data) is used to make predictions.
[0093] In Gaussian process regression, the kernel function determines the similarity and correlation between data points. To adapt to the characteristics of different modal data, this embodiment selects the radial basis function kernel (RBF) as the covariance function to capture smooth nonlinear relationships.
[0094] The radial basis function kernel has the following form:
[0095]
[0096] In the formula, and For any two data points, that is, any two points in the input data, it represents their positions in the input space; It is a length scale parameter hyperparameter used to control the rate at which the similarity between data points decays.
[0097] Radial basis function kernels can effectively capture the nonlinear relationships in input data and possess smoothing properties, enabling Gaussian process regression to adapt to variations in different modalities. Therefore, Gaussian process regression models using radial basis function kernels can handle various types of data well, including numerical and time series data.
[0098] Step S23: The optimal hyperparameters are obtained by maximizing the marginal likelihood function.
[0099] The training process of Gaussian process regression mainly includes determining the hyperparameters of the kernel function, such as the length scaling parameter. and noise variance hyperparameter This process is typically achieved by maximizing the marginal likelihood function. Maximizing the marginal likelihood function yields the optimal kernel function hyperparameters, enabling the training of a Gaussian process regression model, which in turn provides the best fit to the training data.
[0100] The marginal likelihood function has the following form:
[0101]
[0102] In the formula, This is the noise variance hyperparameter; For input data The covariance matrix between them; The identity matrix is T; T is the transpose operator. This represents the number of samples.
[0103] In step S2, by establishing a Gaussian process regression model, the characteristics of each modality can be fully modeled, and a corresponding measure of prediction uncertainty is provided. Gaussian process regression can effectively handle nonlinear relationships and complex dependencies between data in multimodal data fusion, providing accurate input for subsequent weighted fusion. This design considers both the potential structure of the data and provides a clear quantitative assessment of the contribution of each modality, laying a solid foundation for the final prediction results.
[0104] Step S3, as follows Figure 3 As shown, a trained Gaussian process regression model is used to predict each mode, and the predicted value and uncertainty measure are output.
[0105] In this embodiment, for the input data of each modality An independent Gaussian process regression model is established for each modality, and the predicted value for each modality is obtained by using this model to predict the data. In the process of multimodal data fusion, the prediction results of all modalities are weighted and fused to obtain the final prediction result.
[0106] Specifically, step S3 includes the following steps:
[0107] Step S31, for the first Training dataset for each modality , It is the input data. The corresponding target value is used to construct a joint distribution through Gaussian process regression:
[0108]
[0109] In the formula, Indicates input data The potential function value; For input data The covariance matrix between them; and These are transposes of each other, and are used as test data. and input data The covariance matrix between them; This is the noise variance hyperparameter; It is the identity matrix; It follows a multivariate Gaussian distribution.
[0110] In Gaussian process regression, all latent function values Both input and output data follow a joint distribution, which describes the relationship between the input and output data. The test data can be obtained through this joint distribution. The conditional distribution is used to predict unknown data.
[0111] Step S32: Use the joint distribution to obtain the predicted values of the Gaussian process regression. :
[0112]
[0113] In the formula, For test data With input data The covariance matrix between them; For input data The inverse of the covariance matrix.
[0114] The test data can be obtained using the above formula. Predicted value Furthermore, it can calculate the uncertainty measure of the prediction results. This allows us to obtain a measure of the uncertainty of all modal prediction results.
[0115] Step S33, based on test data Predicted value Calculate its forecast uncertainty measure :
[0116]
[0117] In the formula, For the first Test data for each modality A measure of predictive uncertainty; For test data The covariance matrix between them; For test data and input data The covariance matrix between them; For input data The covariance matrix; T is the transpose operator.
[0118] In the formula, uncertainty measure Quantify the test data The prediction uncertainty reflects the model's uncertainty regarding the test data. The level of confidence. Uncertainty measure. The smaller the value, the better the model performs on the test data. The more confident you are in the output results, the less confident you are in the uncertainty measure. The larger the value, the better the model's performance on the test data. The reliability of the output results is low.
[0119] In step S3, the Gaussian process regression for each modality not only provides the predicted value, but also gives the variance of the prediction result (i.e., the uncertainty measure), which will play a key role in the subsequent weighted fusion process.
[0120] Step S4, as follows Figure 4 As shown, the weights of modes are dynamically adjusted based on the uncertainty measure and data quality of each mode prediction. The weights are then used to perform a weighted calculation on the predicted values of each mode to obtain the final prediction result.
[0121] Traditional weighting methods typically rely on pre-set fixed weights, while this embodiment employs a heuristic weighted fusion mechanism that dynamically adjusts the weights of modes based on the uncertainty measure of each mode's prediction and the data quality. This method ensures that higher-quality modes are given more weight during prediction, thereby reducing the interference of low-quality modes on the final result. Modes with larger weights indicate that the model's prediction results for that mode are more reliable, while modes with smaller weights indicate that the model has lower confidence in that mode.
[0122] Specifically, step S4 includes the following steps:
[0123] Step S41: Calculate the quality metric of the mode based on the predicted value of the mode and the uncertainty metric;
[0124]
[0125] In the formula, For the first A quality metric for each modality; For the first Test data for each modality A measure of predictive uncertainty; It is a very small constant used to prevent division-by-zero errors when the prediction uncertainty is zero.
[0126] In the heuristic weighted fusion mechanism, the weight of each mode is calculated from the quality metric of that mode. The quality metric is based on the prediction uncertainty of the mode; modes with lower prediction uncertainty receive higher weights, while modes with higher prediction uncertainty receive lower weights.
[0127] Step S42: Convert the quality metrics into weights, ensuring that the sum of all weights is 1;
[0128]
[0129] In the formula, The total number of modes; It is the sum of all modal quality measures; For the first Weights of each modality.
[0130] Using the above formula, the weights of all modes are normalized to their prediction uncertainty measure. Modes with larger weights indicate higher data quality and thus carry more weight in the final prediction, while modes with smaller weights indicate lower data quality and thus contribute less to the final prediction.
[0131] Step S43: Once the weight of each modality is calculated, the final fusion prediction result can be obtained by weighted averaging. Specifically, the predicted values of all modalities are weighted according to their corresponding weights to obtain the final fusion prediction result.
[0132]
[0133]
[0134] In the formula, For the first Predicted values for each modality; This is the final fusion prediction value; This is a measure of the uncertainty of the final fused prediction.
[0135] This weighted averaging method adjusts the final prediction result according to the weight of each modality, thereby making full use of the prediction results of high-quality modalities and reducing the interference of low-quality modalities on the final result. In this way, the prediction accuracy and stability of multimodal data fusion can be effectively improved.
[0136] In practical applications, the quality of modalities may fluctuate due to changes in data source quality and environment. To address this, the heuristic weighted fusion mechanism employs a dynamic adjustment mechanism. With each new data input, the weight of each modality is calculated in real time, and the weights are adjusted based on the real-time calculated prediction uncertainty metric. Specifically, when the prediction result of a modality becomes unreliable, its corresponding weight is automatically reduced, thereby decreasing the modality's impact on the final result; conversely, when the prediction result of a modality becomes more reliable, its weight is increased, increasing the modality's contribution to the final prediction.
[0137] This dynamic weight adjustment mechanism enables this embodiment to consistently output accurate prediction results even when data quality fluctuates or certain modal data are missing.
[0138] In practical applications, some modal data may be missing. Traditional weighted fusion methods may directly ignore the missing modalities, resulting in information loss. However, in this embodiment, such as... Figure 5 As shown, the heuristic weighted fusion mechanism can automatically adjust weights when modal data is missing, supplementing it with data from other modalities. Specifically, when data for a certain modality is missing, the weight of that modality is automatically set to zero, while the weights of other modalities are adjusted to ensure that the impact of the missing modality on the final result is minimized.
[0139] This mechanism enables this embodiment to handle cases where data is missing without forcibly discarding modalities or performing complex interpolation calculations.
[0140] The heuristic weighted fusion mechanism dynamically calculates the weight of each modality, adjusting the contribution of each modality based on prediction uncertainty metrics and data quality. By weightedly fusing the prediction results of each modality, it effectively improves the accuracy and stability of multimodal data fusion while handling cases where modality data is missing. This mechanism provides higher reliability and interpretability for the final prediction results and is suitable for various practical application scenarios.
[0141] Furthermore, in the prediction process of multimodal data fusion, in addition to outputting predicted values, assessing the uncertainty of the prediction structure is equally crucial. Uncertainty assessment helps quantify the model's confidence in the prediction results, thereby providing more comprehensive information. This is particularly important for decision-making processes, especially in applications requiring high reliability and transparency, such as medical diagnosis, financial forecasting, and weather forecasting.
[0142] In this embodiment, the Gaussian process regression model provides a metric for prediction uncertainty, which is calculated based on the prediction variance of each mode. By combining the prediction uncertainty metric for each mode, this embodiment can fully consider the reliability of each mode in the final fused prediction result and provide a clear quantitative assessment of the contribution of each mode. Specifically:
[0143] Gaussian process regression has a built-in uncertainty measurement mechanism. Whenever a Gaussian process regression model makes a prediction for a certain test data, it not only gives the predicted value, but also provides a variance (i.e. uncertainty measure) associated with the predicted value. The uncertainty measure measures the model's confidence in the predicted value. The smaller the uncertainty measure, the more confident the model is in the prediction result. The larger the uncertainty measure, the lower the model's confidence in the prediction result.
[0144] Meanwhile, uncertainty metrics also play a crucial role in the heuristic weighted fusion process. The weight of each modality is calculated using its quality metric, which is closely related to the prediction uncertainty metric of each modality. Specifically, the quality metric of a modality is the reciprocal of the prediction uncertainty metric; a smaller prediction uncertainty metric corresponds to a larger quality metric, meaning that the prediction result of that modality contributes more to the final fusion result.
[0145] By combining the prediction results of each modality with the corresponding uncertainty metric, the weights of each modality can be dynamically adjusted during the multimodal data fusion process, ensuring that higher-quality modalities have a greater impact on the final prediction results.
[0146] Assessing the uncertainty of forecast results is of great significance in practical applications. First, uncertainty measurement can help identify which modes contribute significantly to the forecast results and which modes produce unreliable results. For example, in weather forecasting using multimodal data fusion, if the data from some sensors is inaccurate or of poor quality, the model can reduce the influence of these modes through weighting mechanisms, thereby obtaining more reliable forecast results.
[0147] Secondly, predictive uncertainty can also be used for risk assessment. In applications such as financial forecasting and medical diagnosis, assessing the uncertainty of forecast results can provide more information for decision-making, helping decision-makers make more prudent judgments. For example, in the medical field, when the variance of the model's predicted results is large, doctors can use this uncertainty measure to decide whether further testing or verification is needed.
[0148] In this embodiment, the uncertainty measurement mechanism inherent in the Gaussian process regression model provides a detailed uncertainty assessment for the prediction results of each modality. During the multimodal data fusion process, the weights of the modalities are dynamically adjusted to fully consider the prediction uncertainty of each modality, thereby ensuring the accuracy and stability of the final prediction results. Through this mechanism, the model not only provides accurate prediction values but also provides confidence levels and confidence intervals for each prediction result, thus enhancing the model's transparency and interpretability and improving its decision support capabilities in application scenarios.
[0149] Meanwhile, in this embodiment, Gaussian process regression employs a sparse approximation method, which significantly reduces computational complexity while maintaining high accuracy, enabling this embodiment to run efficiently even in edge computing environments with limited computing resources. In addition to Gaussian process regression, this embodiment can also utilize lightweight models (such as YOLONano), image compression, and downsampling techniques to make data processing not only accurate but also efficient, further reducing the computational burden of data processing and ensuring good performance even with limited hardware resources.
[0150] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A method for dynamic fusion of multimodal features based on uncertainty estimation, characterized in that, Includes the following steps: Step S1: Preprocess the data from different modalities and convert them into a unified format to eliminate differences between modalities; Step S2: Independently establish a Gaussian process regression model for each mode to capture the nonlinear relationships in the modal data; Step S3: Use the trained Gaussian process regression model to predict each mode and output the predicted value and uncertainty measure. Step S4: Dynamically adjust the modality weights based on the uncertainty measure and data quality of each modality prediction, and use the weights to perform a weighted calculation on the predicted values of each modality to obtain the final prediction result.
2. The multimodal feature dynamic fusion method based on uncertainty estimation according to claim 1, characterized in that, In step S1, the preprocessing of different modal data includes standardization and normalization, encoding, missing data handling, and outlier detection and handling.
3. The multimodal feature dynamic fusion method based on uncertainty estimation according to claim 1, characterized in that, In step S1, the modal data are preprocessed and then integrated by splicing or weighted fusion.
4. The multimodal feature dynamic fusion method based on uncertainty estimation according to claim 1, characterized in that, Step S2 includes the following steps: Step S21: The preprocessed modal data are used to form training datasets for each modality. Training dataset for each modality , It is the input data. This corresponds to the target value; In the formula, For input data The potential function value; Noise term; Step S22: A radial basis function kernel is used as the covariance function to capture the smooth nonlinear relationship of the input data; The radial basis function kernel has the following form: In the formula, and For any two data points; It is a length scale parameter hyperparameter; Step S23: The optimal hyperparameters are obtained by maximizing the marginal likelihood function; The marginal likelihood function has the following form: In the formula, This is the noise variance hyperparameter; For input data The covariance matrix between them; It is the identity matrix; T is the transpose operator; This represents the number of samples.
5. The multimodal feature dynamic fusion method based on uncertainty estimation according to claim 1, characterized in that, Step S3 includes the following steps: Step S31, for the first Input data for each modality and the corresponding target value Construct the joint distribution using Gaussian process regression: In the formula, Indicates input data The potential function value; For input data The covariance matrix between them; and These are transposes of each other, and are used as test data. and input data The covariance matrix between them; This is the noise variance hyperparameter; It is the identity matrix; The symbol for a multivariate Gaussian distribution; Step S32: Use the joint distribution to obtain the predicted values of the Gaussian process regression. : In the formula, For test data With input data The covariance matrix between them; For input data The inverse of the covariance matrix; Step S33, based on test data Predicted value Calculate its forecast uncertainty measure : In the formula, For the first Test data for each modality A measure of predictive uncertainty; For test data The covariance matrix between them; For test data and input data The covariance matrix between them; For input data The covariance matrix; T is the transpose operator.
6. The multimodal feature dynamic fusion method based on uncertainty estimation according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: Calculate the quality metric of the mode based on the predicted value of the mode and the uncertainty metric; In the formula, For the first A quality metric for each modality; For the first Test data for each modality A measure of predictive uncertainty; It is a constant; Step S42: Convert the quality metrics into weights, ensuring that the sum of all weights is 1; In the formula, The total number of modes; It is the sum of all modal quality measures; For the first The weights of each modality; Step S43: Perform weighted fusion to obtain the final fusion prediction result; In the formula, For the first Predicted values for each modality; This is the final fusion prediction value; This is a measure of the uncertainty of the final fused prediction.
7. The multimodal feature dynamic fusion method based on uncertainty estimation according to claim 6, characterized in that, The availability of data for each modality is monitored in real time. When data for one modality is missing, the weight of that modality is automatically set to zero, and the weights of other modalities are adjusted simultaneously.
8. The multimodal feature dynamic fusion method based on uncertainty estimation according to claim 6, characterized in that, Each time new data is input, the weight of each modality is calculated in real time, and the weights are adjusted based on the real-time calculated measurement of prediction uncertainty.