A multimodal data fusion method, device, equipment and medium

Abstract

The present application discloses a multimodal data fusion method, device, equipment and medium, which relates to the field of data processing technology, including: obtaining multimodal data fusion requirements, and judging the type of the pre-processed data to be fused in the data fusion requirements; if the data type of the data to be fused is an accurate data type, then the fused data is treated from the perspective of information theory to perform data fusion processing to obtain a fusion result; if the data type of the data to be fused is a fuzzy data type, then the data to be fused is cleaned into accurate data based on DS evidence theory to obtain the cleaned data to be fused, and then jump to perform data fusion processing from the perspective of information theory to treat the fused data to obtain a fusion result. The performance of data fusion is improved in many aspects, and more accurate and reliable fusion results can be provided for data fusion applications in different fields.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal data fusion method, device, equipment and medium. Background Art

[0002] As an important means of data processing, data fusion technology has a wide range of applications, such as target detection, medical diagnosis, remote sensing mapping, fault diagnosis, intelligent manufacturing, etc. In the field of intelligent manufacturing, data fusion technology has effectively improved people's processing capabilities and utilization efficiency of industrial big data. Through data fusion technology, massive, high-dimensional, multi-source heterogeneous noisy industrial data can be cleaned, denoised, integrated modeled, and multi-scale classified, providing reliable data resource support for subsequent correlation analysis, performance prediction, and optimization decision-making. Therefore, data fusion technology has strongly promoted the development of application fields.

[0003] Data association is a key step in data fusion, which involves determining the correspondence between a set of observations and a specific target. Common data association techniques include nearest neighbor method, K-means clustering, probabilistic data association, joint probabilistic data association, and multiple hypothesis testing. In addition, the data fusion system needs to make decisions. Decision fusion techniques usually use symbolic information and consider uncertainty and constraints in the fusion process. At present, deep learning technology provides new possibilities for processing complex multimodal data, but there is currently a lack of theoretical analysis methods to associate data information with model performance. And although fuzzy logic performs well in processing multimodal data, it still has the following defects in multimodal data fusion: How to effectively process fuzzy information from different modalities. How to effectively represent and reason about fuzzy information. Current methods cannot fully capture all the nuances of fuzzy information, and more advanced models are needed to improve the expression and processing capabilities of fuzzy information. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a multimodal data fusion method, device, equipment and medium, which can fully utilize the fuzzy information of fuzzy data to improve the model performance of multimodal data fusion. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a multimodal data fusion method, comprising:

[0006] Obtaining multimodal data fusion requirements, and determining the type of preprocessed data to be fused in the data fusion requirements;

[0007] If the data type of the data to be fused is an accurate data type, performing data fusion processing on the data to be fused according to the information theory perspective to obtain a fusion result;

[0008] If the data type of the data to be fused is a fuzzy data type, the data to be fused is cleaned into precise data based on the DS evidence theory to obtain the cleaned data to be fused, and then the data fusion processing of the data to be fused is performed from the perspective of information theory to obtain the fusion result.

[0009] Optionally, performing data fusion processing on the data to be fused according to the information theory perspective to obtain a fusion result includes:

[0010] Performing noise elimination processing on the data to be fused, and extracting decision features from the data to be fused after noise elimination, so as to obtain corresponding decision features; wherein the decision features include any one or more of the attribute features, time series features, spatial location features, and association relationship features of the data to be fused;

[0011] Calculating the information entropy of each of the decision features, and calculating the mutual information between the decision features;

[0012] Calculating the conditional entropy of the decision feature using the information entropy and the mutual information, and calculating the difference between the conditional entropy and the information entropy to obtain the information contribution provided by a single modality or a combination of modalities;

[0013] Based on the information entropy and the mutual information, corresponding initial weights are assigned to the corresponding decision features, and the initial weights are reversely updated using the difference between the predicted results and the actual fusion results generated by the data fusion process of the decision features with the information contribution and the assigned initial weights, so as to obtain the target weights of the decision features;

[0014] The BJS divergence calculation is performed on the joint entropy determined based on the information entropy, the mutual information, and the target weight, so as to optimize the selection of decision features and the adjustment of target weight in the data fusion process according to the BJS divergence calculation result.

[0015] Optionally, the cleaning of the data to be fused into accurate data based on DS evidence theory to obtain cleaned data to be fused includes:

[0016] Using an effective label set covering algorithm, the data to be fused carrying a plurality of labels is divided into data groups to obtain a plurality of target data groups; wherein each target data group carries only one effective label;

[0017] Inputting each of the target data sets into a classifier trained based on DS evidence theory, so that the classifier outputs the category probability of each category label under the valid label of the target data set;

[0018] The target label of each of the to-be-fused data in the target data group is determined according to the conversion processing result of each of the category probabilities, so as to obtain the cleaned to-be-fused data carrying the target label.

[0019] Optionally, determining the target label of each of the to-be-fused data in the target data group according to the conversion processing result of each of the category probabilities to obtain the cleaned to-be-fused data carrying the target label includes:

[0020] Performing pignistic probability transformation on the probabilities of each category to obtain a single category probability result;

[0021] The category label of the single category probability result with the highest probability is used as the target label to obtain the cleaned data to be fused with the target label.

[0022] Optionally, before determining the type of the preprocessed data to be fused in the data fusion requirement, the method further includes:

[0023] Build and train a preset data classification model for determining whether a data type is a precise data type or a fuzzy data type.

[0024] Optionally, the performing type judgment on the preprocessed data to be fused in the data fusion requirement includes:

[0025] Inputting the data fusion requirement into a preset data classification model, so as to convert the multimodal data in the data fusion requirement into data to be fused in a matrix form through the preset data classification model;

[0026] The data features of the data to be fused are extracted through the preset data classification model, so as to make a type judgment on the data to be fused according to the data features, and output a corresponding judgment result.

[0027] Optionally, the step of constructing and training a preset data classification model for determining whether a data type is a precise data type or a fuzzy data type includes:

[0028] The initial model of the Transformer architecture is trained for classification using a data training set that carries precise data type labels and fuzzy data type labels, so as to obtain a preset data classification model that has learned the corresponding relationship between the corresponding data type labels and the data classification results.

[0029] In a second aspect, the present application discloses a multimodal data fusion device, comprising:

[0030] A type judgment module is used to obtain multimodal data fusion requirements and perform type judgment on the pre-processed data to be fused in the data fusion requirements;

[0031] A first fusion module is used for, if the data type of the data to be fused is an accurate data type, performing data fusion processing on the data to be fused according to the information theory perspective to obtain a fusion result;

[0032] The second fusion module is used to clean the data to be fused into precise data based on DS evidence theory if the data type of the data to be fused is a fuzzy data type, so as to obtain the cleaned data to be fused, and then jump to execute the data fusion processing of the data to be fused from the perspective of information theory to obtain the fusion result.

[0033] In a third aspect, the present application discloses an electronic device, comprising:

[0034] Memory, used to store computer programs;

[0035] The processor is used to execute the computer program to implement the steps of the multimodal data fusion method disclosed above.

[0036] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the multimodal data fusion method disclosed above are implemented.

[0037] It can be seen that the present application discloses a multimodal data fusion method, including: obtaining a multimodal data fusion requirement, and performing type judgment on the pre-processed data to be fused in the data fusion requirement; if the data type of the data to be fused is a precise data type, then performing data fusion processing on the data to be fused from the perspective of information theory to obtain a fusion result; if the data type of the data to be fused is a fuzzy data type, then based on the DS evidence theory, the data to be fused is cleaned into precise data to obtain the cleaned data to be fused, and then jumps to execute the data fusion processing on the data to be fused from the perspective of information theory to obtain a fusion result. It can be seen that by determining whether the current data to be fused is precise data or fuzzy data through data type judgment, and executing corresponding processing procedures for different data types, it is possible to process both precise data and fuzzy data, and is suitable for a variety of complex data scenarios, thereby improving the versatility and comprehensiveness of the data fusion method. In the branch where the data to be fused is precise data, accurate weight allocation and effective measurement of uncertainty and information loss are performed on the data to be fused from the perspective of information theory, thereby realizing the fusion processing of accurate data. In the branch where the data to be fused is fuzzy data, the fuzzy data is cleaned and processed through DS evidence theory and effective label information of fuzzy data to obtain precise data, and then the data fusion process of precise data is executed. The performance of data fusion is improved in many aspects, and more accurate and reliable fusion results can be provided for data fusion applications in different fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A flow chart of a multimodal data fusion method disclosed in this application;

[0040] Figure 2 A flow chart of an accurate data fusion method disclosed in this application;

[0041] Figure 3 A flow chart of a fuzzy data processing method disclosed in this application;

[0042] Figure 4 This is a schematic diagram of the structure of a multimodal data fusion device disclosed in this application;

[0043] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] As an important means of data processing, data fusion technology has a wide range of applications, such as target detection, medical diagnosis, remote sensing mapping, fault diagnosis, intelligent manufacturing, etc. In the field of intelligent manufacturing, data fusion technology has effectively improved people's processing capabilities and utilization efficiency of industrial big data. Through data fusion technology, massive, high-dimensional, multi-source heterogeneous noisy industrial data can be cleaned, denoised, integrated modeled, and multi-scale classified, providing reliable data resource support for subsequent correlation analysis, performance prediction, and optimization decision-making. Therefore, data fusion technology has strongly promoted the development of application fields.

[0046] Data association is a key step in data fusion, which involves determining the correspondence between a set of observations and a specific target. Common data association techniques include nearest neighbor method, K-means clustering, probabilistic data association, joint probabilistic data association, and multiple hypothesis testing. In addition, the data fusion system needs to make decisions. Decision fusion techniques usually use symbolic information and consider uncertainty and constraints in the fusion process. At present, deep learning technology provides new possibilities for processing complex multimodal data, but there is currently a lack of theoretical analysis methods to associate data information with model performance. And although fuzzy logic performs well in processing multimodal data, it still has the following defects in multimodal data fusion: How to effectively process fuzzy information from different modalities. How to effectively represent and reason about fuzzy information. Current methods cannot fully capture all the nuances of fuzzy information, and more advanced models are needed to improve the expression and processing capabilities of fuzzy information.

[0047] To this end, the present invention provides a multimodal data fusion solution, which can fully utilize the fuzzy information of fuzzy data to improve the model performance of multimodal data fusion.

[0048] Reference Figure 1 As shown, an embodiment of the present invention discloses a multimodal data fusion method, comprising:

[0049] Step S11: obtaining multimodal data fusion requirements, and determining the type of preprocessed data to be fused in the data fusion requirements.

[0050] In this embodiment, the multimodal data fusion requirements include the data to be fused, which may include any one or more of precise data and fuzzy data, wherein precise data is obtained through precise measurement tools and clear rules, and fuzzy data comes from subjective evaluation, imprecise description or concepts that are inherently ambiguous.

[0051] In this embodiment, before the type judgment is performed on the preprocessed data to be fused in the data fusion requirement, it also includes: constructing and training a preset data classification model for judging whether the data type is an accurate data type or a fuzzy data type. Specifically, the initial model of the Transformer architecture is classified and trained using a data training set carrying an accurate data type label and a fuzzy data type label to obtain a preset data classification model that has learned the corresponding relationship between the corresponding data type label and the data classification result. It can be understood that by pre-classifying the large language model for accurate data and fuzzy data to obtain the preset data classification model, first, data samples containing various types of data are collected, and these data samples should cover a variety of situations of accurate data and fuzzy data. For example, in the text field, news articles and academic papers are collected as accurate data samples, and user comments, chat records and other texts that may contain fuzzy expressions are collected as fuzzy data samples; in terms of numerical data, scientific experimental data with accurate measurements are collected as accurate data, and data with estimates and ranges are collected as fuzzy data; in addition, data samples include not only samples in the above text field and numerical field, but also samples in different fields such as audio field and image field. Data samples in the appropriate field are selected according to the actual data fusion requirements, and no specific limitation is made to this. The collected data samples are labeled to clarify which are accurate data and which are fuzzy data. For fuzzy data, the types of fuzziness can be further labeled, such as semantic fuzziness, numerical range fuzziness, etc. Manual labeling or existing labeling tools can be used to assist labeling. For example, for text data, by formulating labeling rules, the labeler marks the text with fuzzy words (such as "probably", "possibly", "about", etc.) as fuzzy data. Then select a suitable open source large model according to the type of data sample (such as text, image, numerical value, etc.) and the characteristics of the task (judgment of accuracy or fuzziness). Taking text data as an example, if the data is natural language text, the present invention selects the initial model of the Transformer architecture, such as open source models such as Bert and RoBERTa. These models have good performance in natural language processing tasks and can effectively learn features such as semantics of text. The collected data samples are preprocessed to meet the input requirements of the selected open source large model. For text data, operations such as word segmentation and encoding may be included. For example, when using the Bert model, the text needs to be segmented and converted into a vector representation that the model can accept. According to the model documentation and task requirements, set appropriate training parameters, such as learning rate, batch size, number of training rounds, etc. Generally speaking, the learning rate can be set to a smaller value first, such as 1×10 -5The batch size can be adjusted according to the hardware resources and data scale, and the number of training rounds can be determined in the appropriate range through preliminary experiments. The labeled precise data samples and fuzzy data samples are input into the initial model and trained in a supervised learning manner. The goal of the model is to learn the features that can distinguish precise and fuzzy data, that is, to learn the correspondence between precise data and the corresponding unique precise data label, and the correspondence between fuzzy data and multiple corresponding fuzzy labels. It mainly learns the differences between precise data and fuzzy data in feature expression, format characteristics, semantic expression, etc. based on the learning mechanism of the initial model itself and the corresponding training algorithm. For example, precise data may be more standardized in numerical features, with a clear range of values ​​and accuracy requirements, while fuzzy data often has uncertain and relatively vague vocabulary at the semantic level. The large model grasps these rules and forms corresponding judgment capabilities by analyzing a large number of samples. During the training process, the loss function between the model output and the label, such as the cross entropy loss function, is calculated, and the parameters of the model are updated through the back propagation algorithm so that the output of the model gradually approaches the correct label.

[0052] In addition, if the selected open source model is a pre-trained model, you can use its pre-trained weights for fine-tuning. For example, the Bert model has been pre-trained on large-scale text data. When training tasks to judge precise and fuzzy data, you can freeze the parameters of some layers and only fine-tune some layers. This can speed up training and improve model performance. Then use the trained, verified, and tested model as the preset data classification model.

[0053] In this embodiment, the data fusion requirement is input into a preset data classification model so that the multimodal data in the data fusion requirement is converted into data to be fused in matrix form through the preset data classification model; the data features of the data to be fused are extracted through the preset data classification model to judge the type of the data to be fused according to the data features, and the corresponding judgment results are output. It can be understood that after obtaining the data fusion requirement, the data fusion requirement is input into the preset data classification model, and the multimodal data in the data fusion requirement is format converted through the classification model to obtain the data to be fused in matrix format, and then the model further extracts the data features of the data to be fused, for example: when facing the data that needs to be judged, it is input into the preset data classification model, and the preset data classification model will analyze and identify based on the previously learned features and patterns. If the data presents features such as a clear numerical format, a clear unit, and a range of values ​​that meet the precise measurement standards, it is consistent with the learned precise data feature performance, and it is determined to be precise data. For example, a data representing the length of an object as "5.23 cm" is input, and the large model judges that it is precise data based on its precise numerical value and clear unit. If the data appears in the form of text labels, including words such as "probably", "perhaps", "relatively" that reflect uncertainty and fuzzy semantics, or data corresponding to expressions based on subjective feelings that are difficult to measure with accurate numerical values, the preset data classification model will identify the characteristic patterns that conform to fuzzy data and then determine it as fuzzy data. For example, if you input data related to "this solution feels okay", the preset data classification model will determine that it is fuzzy data.

[0054] Step S12: If the data type of the data to be fused is an accurate data type, data fusion processing is performed on the data to be fused from the perspective of information theory to obtain a fusion result.

[0055] In this embodiment, if the data type of the data to be fused is an accurate data type, the following steps are performed in sequence: preprocessing and feature extraction, information entropy and mutual information calculation, weight allocation and adjustment, and calculation of BJS divergence, as follows: Figure 2 As shown:

[0056] Performing noise elimination processing on the data to be fused, and extracting decision features from the data to be fused after noise elimination, so as to obtain corresponding decision features; wherein the decision features include any one or more of the attribute features, time series features, spatial location features, and association relationship features of the data to be fused;

[0057] Calculating the information entropy of each of the decision features, and calculating the mutual information between the decision features;

[0058] Calculating the conditional entropy of the decision feature using the information entropy and the mutual information, and calculating the difference between the conditional entropy and the information entropy to obtain the information contribution provided by a single modality or a combination of modalities;

[0059] Based on the information entropy and the mutual information, corresponding initial weights are assigned to the corresponding decision features, and the initial weights are reversely updated using the difference between the predicted results and the actual fusion results generated by the data fusion process of the decision features with the information contribution and the assigned initial weights, so as to obtain the target weights of the decision features;

[0060] The BJS divergence calculation is performed on the joint entropy determined based on the information entropy, the mutual information, and the target weight, so as to optimize the selection of decision features and the adjustment of target weight in the data fusion process according to the BJS divergence calculation result.

[0061] It is understandable that the purpose of necessary preprocessing operations on the data to be fused of precise data types is to eliminate the noise and irrelevant information in them, make the data more "pure" and focus on the key content, so as to better perform the fusion processing later. On this basis, decision features that are helpful for fusion decision making are extracted from the preprocessed data. These decision features will serve as an important basis for subsequent calculations and analysis, and prepare for the accurate evaluation of the value and relationship of data in the fusion process.

[0062] Furthermore, for each extracted decision feature, its information entropy is calculated. Information entropy can intuitively reflect the degree of uncertainty of the information contained in the feature. At the same time, the mutual information between different decision features is calculated. The mutual information can clarify the degree of dependence between decision features, that is, to judge the close relationship between different decision features. With the help of the difference between information entropy and conditional entropy (by calculating the difference between conditional entropy and information entropy in the absence of data), the amount of information provided by a single modality or modal combination can be quantified, which helps to accurately evaluate the contribution of different modalities to the prediction performance of the entire model, so as to plan the fusion strategy more reasonably. Among them, the information entropy calculation formula of the decision feature is as follows:

[0063] ;

[0064] in, represents a random variable, represents a decision feature in the present invention, n represents the number of values ​​of the decision feature, represents the i-th decision feature The probability information of Represents information entropy.

[0065] For example, suppose there is a decision feature of the data to be fused that represents the product quality level (divided into four levels: excellent, good, medium, and poor). Through statistical historical data, the probabilities of excellent, good, medium, and poor are 0.2, 0.3, 0.4, and 0.5 respectively. Then the information entropy of the decision feature of this product quality level is

[0066] ;

[0067] By calculating this information entropy, we can intuitively understand the uncertainty of the information contained in the data feature. The larger the information entropy, the higher the uncertainty of the feature.

[0068] Furthermore, after calculating the information entropy of the decision feature, the formula for calculating the mutual information between different decision features is as follows:

[0069] ;

[0070] in, and are two different decision features. express and The joint probability of and They are and The marginal probability of .

[0071] For example, there are product quality grades ( ) and production equipment operating time ( ) These two decision features. First, calculate the joint probability of different quality levels and different running time combinations , and the marginal probabilities of quality level and runtime, respectively and , and then substitute it into the formula to calculate the mutual information. Mutual information measures the degree of mutual dependence between two decision features. The larger the mutual information, the closer the relationship between the two features.

[0072] Finally, the difference between the conditional entropy and the information entropy in the absence of data is calculated to quantify the amount of information provided by a single modality or modal combination and obtain the information contribution. The calculation formula is as follows:

[0073] ;

[0074] in, It is under the condition Down The amount of information provided by a single mode or combination of modes (Assumption is a conditional modality). For example, in a model containing product quality grades ( ) and production equipment operating time ( ) in the system, first calculate (information entropy of product quality level) and (Conditional entropy of product quality grade under the condition of known production equipment operating time). The difference between the two is the amount of information provided by the mode of production equipment operating time for the mode of product quality grade. This amount of information helps to evaluate the contribution of different modes to the prediction performance of the model.

[0075] After completing the calculation process of information entropy, mutual information, and information contribution, the uncertainty and correlation reflected in the above information are used to assign corresponding initial weights to each decision feature. Specifically, for decision features with lower information entropy, since their own uncertainty is small and they can provide more stable information, relatively higher initial weights can be assigned. For example, in a bimodal data fusion scenario involving product quality and production equipment status, if the information entropy of the product quality index is 0.5 and the information entropy of the production equipment status index is 1.2 (assuming that the value range of information entropy allows such comparison), when allocating initial weights, a relatively large weight can be assigned to the product quality index, such as 0.6 for the product quality index and 0.4 for the production equipment status index. Considering the influence of mutual information, if the mutual information between two decision features is high, it means that their joint contribution to the decision is large. In this case, the weights of the two features can be increased at the same time or the weights can be assigned according to their joint contribution. For example, in market trend forecasting, if the mutual information between market price fluctuations and macroeconomic indicators is high, they can be regarded as an important combination when allocating weights. Assuming that the information entropy of individual market price fluctuations is low, the original weight assigned is 0.4, and the information entropy of macroeconomic indicators is high, the original weight assigned is 0.3. Due to the high mutual information between them, after considering the joint contribution, the weight of market price fluctuations can be adjusted to 0.5, and the weight of macroeconomic indicators can be adjusted to 0.4. At the same time, the weights of other data features with lower correlation can be appropriately reduced.

[0076] It should be noted that in the process of assigning weights, the information entropy and mutual information of all decision features need to be considered comprehensively. For example, in a multimodal data fusion system, there are multiple data features. After calculating their information entropy and mutual information with other features, the weights are initially assigned according to the above principles. Finally, in order to ensure that the sum of all weights is 1, normalization is required. The normalized weights, in this way, obtain the initial weights for data fusion decision-making, so that the contribution of each data feature in the fusion process can be reasonably quantified based on its own uncertainty and correlation with other features.

[0077] These weights reflect the importance of each decision feature in the fusion process. Then, the initial weights are further reversely adjusted based on the information contribution and the difference between the predicted fusion results and the actual fusion results output by the optimization algorithm to minimize the difference between the predicted fusion results and the actual fusion results, and the target weights of each decision feature are obtained.

[0078] BJS divergence is used to measure the uncertainty and information loss of data. BJS divergence is a symmetrical divergence. The formula for calculating BJS divergence in the present invention is as follows:

[0079] ;

[0080] in, It represents the joint entropy, which combines the advantages of Jensen-Shannon divergence and Bhattacharyya distance and improves the sensitivity to uncertainty in the data fusion process. From the perspective of information theory, the calculation formula of the joint entropy is as follows:

[0081] ;

[0082] in, yes The information entropy of yes The information entropy of yes and The mutual information of .

[0083] It should be noted that Also represents the joint entropy, but is different from The difference between is that the order is different when calculating the joint probability distribution. In practical applications, if the probability distributions of and are symmetrical, the two are equal, but in the present invention, the order has a specific meaning. In causal relationship analysis, if Cause variable, is the outcome variable, The uncertainty of the cause and the result can be inferred from the result. The uncertainty of the cause and the result can be measured more comprehensively by taking both into account in the calculation of BJS divergence. and differences and uncertainties between them.

[0084] In this way, the precise data is preprocessed to eliminate irrelevant information such as noise, and features that are helpful for decision-making are extracted. The information entropy of each data feature is calculated to evaluate uncertainty, and the mutual information between different data features is used to determine the degree of mutual dependence, so as to quantify the contribution of different modes to the model prediction performance. The initial weights are assigned according to the information entropy and mutual information results, and then the weights are adjusted using an optimization algorithm to reduce the difference between the fusion results and the actual situation. The BJS divergence is used to measure the uncertainty and information loss of the data. Its combined advantages increase the sensitivity to uncertainty and help the precise data fusion process.

[0085] Step S13: If the data type of the data to be fused is a fuzzy data type, the data to be fused is cleaned into precise data based on the DS evidence theory to obtain the cleaned data to be fused, and then the data fusion processing of the data to be fused is performed from the perspective of information theory to obtain the fusion result.

[0086] In this embodiment, if the data type of the data to be fused is a fuzzy data type, the fuzzy data processing flow is executed, specifically including: basic classifier application and data set construction, effective label set cover operation, and subsequent processing, specifically as follows Figure 3 As shown:

[0087] Using an effective label set covering algorithm, the data to be fused carrying a plurality of labels is divided into data groups to obtain a plurality of target data groups; wherein each target data group carries only one effective label;

[0088] Inputting each of the target data sets into a classifier trained based on DS evidence theory, so that the classifier outputs the category probability of each category label under the valid label of the target data set;

[0089] The target label of each of the to-be-fused data in the target data group is determined according to the conversion processing result of each of the category probabilities, so as to obtain the cleaned to-be-fused data carrying the target label.

[0090] Specifically, determining the target label of each of the to-be-fused data in the target data group according to the conversion processing result of each of the category probabilities to obtain the cleaned to-be-fused data carrying the target label includes:

[0091] Performing pignistic probability transformation on the probabilities of each category to obtain a single category probability result;

[0092] The category label of the single category probability result with the highest probability is used as the target label to obtain the cleaned data to be fused with the target label.

[0093] It can be understood that the effective label set coverage algorithm is used to divide the fuzzy type of data to be fused into multiple target data groups. Each target data group represents a valid label set, which can cover the entire category space and ensure the diversity and representativeness of the data. The division rule is to distinguish based on the description of different aspects of each data to be fused, and then fuse them after distinction to obtain a more comprehensive and accurate description of the data to be fused. For example, in a fuzzy data set of user evaluation of a product, the evaluation labels may be "good appearance but average function" and "powerful function but slightly higher price". The effective label set can be a set of evaluation labels around "appearance", "function" and "price". Through the algorithm, the fuzzy label data involving appearance evaluation is divided into a small data group, and the data involving function evaluation is divided into another group, and so on. This division is based on the different entity aspects involved in the data label. After the division, a target data group about "appearance evaluation" will be obtained, in which the data are all fuzzy evaluations of the product appearance by users, such as "fashionable appearance" and "a bit outdated appearance"; a target data group of "function evaluation" will also be obtained, which contains fuzzy evaluations of product functions, such as "very practical functions" and "somewhat complicated functions". These target data groups provide more targeted data subsets for subsequent data fusion based on DS evidence theory, so as to comprehensively describe entities from different aspects and ultimately achieve a more comprehensive and accurate understanding of the entities.

[0094] Furthermore, for each target data group, a classifier based on DS evidence theory is designed and trained. These classifiers will use fuzzy labels as evidence of uncertainty information and assign probabilities to each category label through BPA. Advanced machine learning techniques such as logistic regression, support vector machine or neural network are used for classifier training. After the classifier design and training is completed, the classifier is used to process the target data group. For the target data group, Gaussian distribution data containing five super-category labels are generated, and random noise and interdependence between features are used to simulate the complexity of the real world. Five categories with precise category labels are selected and associated with the Gaussian distribution data of super-category labels.

[0095] Specifically, a machine learning method is used to train a fuzzy data fusion model containing several classifiers. Through a large amount of training data, the model will learn how to make the best decision based on weights and information content. In addition, an independent test set is used to evaluate the generalization ability and performance of the model.

[0096] Given y as a set of existing labels. In order to implement k different classifiers, it is necessary to find k valid label set covers denoted as c1, c2, ci, …, ck, and add fuzzy labels of unselected elements to select multiple hypotheses until the conditions of valid label set covers are met. After deduplication and simplification operations, the valid label set covers can be obtained, which is represented as the set of valid label set covers U={c1, c2, …, ck}.

[0097] According to the number of elements in (yi), the existing tag set y is split into |S| subsets Rj. Secondly, all initial tag set covers are found and retained in the set U. The valid tag set covers in U are simplified. Finally, the set of valid tag set covers U is deduplicated.

[0098] If there is a high degree of conflict between the evidence, fusion through evidence fusion synthesis rules may lead to counterintuitive results in the normalization process. Therefore, to avoid such a phenomenon, the present invention adopts the following strategies:

[0099] If a BPA is more supported by other BPAs, its credibility is higher and the evidence has a greater impact on the final fusion result, thus avoiding over-reliance on a single piece of evidence.

[0100] If there are n classifiers, in order to increase the confidence of the dominant subset and provide convergence after weighted averaging, the weighted average evidence is combined with itself n-1 times. There is no risk of potentially counterintuitive combination results due to high conflict. It can also provide convergence to facilitate subsequent decisions.

[0101] If there is only a single value in the fusion result m_wa, the category with the highest probability can be simply selected. However, if there are fuzzy labels in the outputs of two different classifiers, there are multiple hypotheses in the fusion result. This paper proposes a solution to distribute the probabilities of multiple hypotheses to a single case according to the cardinality, which is called probability transformation. It is a transformation method similar to maximum entropy. The Pignistic probability transformation is defined in the transferable belief model. The model is a two-layer structure, including a trust layer and a Pignistic layer. These two layers are usually indistinguishable, and the probability function is used to quantify the confidence of the two layers. The confidence is transferred between the evidence of the trust layer and converted into the Pignistic probability through the Pignistic probability transformation. After the Pignistic probability transformation, the fusion result m_wa is transformed into m_da, that is, the cleaned data to be fused with the target label is obtained. After the above processing, the fuzzy data in the data set has been converted into precise data. Next, information fusion is performed through an additional fully connected layer to generate the final prediction result.

[0102] In this way, by combining DS evidence theory and information theory, a fusion framework that can process fuzzy data is constructed, and the fusion framework's ability to process and fuse fuzzy information is improved to extract richer information.

[0103] It can be seen that the present application discloses a multimodal data fusion method, including: obtaining a multimodal data fusion requirement, and performing type judgment on the pre-processed data to be fused in the data fusion requirement; if the data type of the data to be fused is a precise data type, then performing data fusion processing on the data to be fused from the perspective of information theory to obtain a fusion result; if the data type of the data to be fused is a fuzzy data type, then based on the DS evidence theory, the data to be fused is cleaned into precise data to obtain the cleaned data to be fused, and then jumps to execute the data fusion processing on the data to be fused from the perspective of information theory to obtain a fusion result. It can be seen that by determining whether the current data to be fused is precise data or fuzzy data through data type judgment, and executing corresponding processing procedures for different data types, it is possible to process both precise data and fuzzy data, and is suitable for a variety of complex data scenarios, thereby improving the versatility and comprehensiveness of the data fusion method. In the branch where the data to be fused is precise data, accurate weight allocation and effective measurement of uncertainty and information loss are performed on the data to be fused from the perspective of information theory, thereby realizing the fusion processing of accurate data. In the branch where the data to be fused is fuzzy data, the fuzzy data is cleaned and processed through DS evidence theory and effective label information of fuzzy data to obtain precise data, and then the data fusion process of precise data is executed. The performance of data fusion is improved in many aspects, and more accurate and reliable fusion results can be provided for data fusion applications in different fields.

[0104] Reference Figure 4 As shown, the present invention also discloses a multimodal data fusion device, including:

[0105] The type judgment module 11 is used to obtain the multimodal data fusion requirement and perform type judgment on the pre-processed data to be fused in the data fusion requirement;

[0106] A first fusion module 12 is used for performing data fusion processing on the data to be fused according to the information theory perspective to obtain a fusion result if the data type of the data to be fused is an accurate data type;

[0107] The second fusion module 13 is used to clean the data to be fused into precise data based on DS evidence theory if the data type of the data to be fused is a fuzzy data type, so as to obtain the cleaned data to be fused, and then jump to execute the data fusion processing of the data to be fused from the perspective of information theory to obtain the fusion result.

[0108] It can be seen that the present application discloses the method of obtaining the multimodal data fusion requirement, and performing type judgment on the pre-processed data to be fused in the data fusion requirement; if the data type of the data to be fused is a precise data type, the data to be fused is subjected to data fusion processing from the perspective of information theory to obtain a fusion result; if the data type of the data to be fused is a fuzzy data type, the data to be fused is cleaned into precise data based on the DS evidence theory to obtain the cleaned data to be fused, and then the data fusion processing of the data to be fused from the perspective of information theory is jumped to execute to obtain a fusion result. It can be seen that by determining whether the current data to be fused is precise data or fuzzy data through data type judgment, and executing corresponding processing procedures for different data types, it is possible to process both precise data and fuzzy data, and is suitable for a variety of complex data scenarios, thereby improving the versatility and comprehensiveness of the data fusion method. In the branch where the data to be fused is precise data, accurate weight allocation and effective measurement of uncertainty and information loss are performed on the data to be fused from the perspective of information theory, thereby realizing the fusion processing of accurate data. In the branch where the data to be fused is fuzzy data, the fuzzy data is cleaned and processed through DS evidence theory and effective label information of fuzzy data to obtain precise data, and then the data fusion process of precise data is executed. The performance of data fusion is improved in many aspects, and more accurate and reliable fusion results can be provided for data fusion applications in different fields.

[0109] Furthermore, the present application also discloses an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0110] Figure 5 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the multimodal data fusion method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0111] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0112] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0113] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0114] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the multimodal data fusion method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data transmitted from an external device received by the electronic device, the data 223 can also include data collected by its own input and output interface 25, etc.

[0115] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the multimodal data fusion method disclosed above. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0116] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0117] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly with hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory RAM (Random Access Memory), memory, read-only memory ROM (Read Only Memory), electrically programmable EPROM (Electrically Programmable Read Only Memory), electrically erasable programmable EEPROM (ElectricErasable Programmable Read Only Memory), register, hard disk, removable disk, CD-ROM (CompactDisc-Read Only Memory), or any other form of storage medium known in the technical field.

[0118] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0119] The scheme provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multimodal data fusion method, characterized in that: include: Obtaining multimodal data fusion requirements, and determining the type of pre-processed data to be fused in the data fusion requirements; data samples of multimodal data include text, numerical data, audio, and images; If the data type of the data to be fused is an accurate data type, performing data fusion processing on the data to be fused according to the information theory perspective to obtain a fusion result; If the data type of the data to be fused is a fuzzy data type, the data to be fused is cleaned into precise data based on the DS evidence theory to obtain the cleaned data to be fused, and then the data fusion processing of the data to be fused according to the information theory perspective is performed to obtain the fusion result; The step of performing data fusion processing on the data to be fused according to the information theory perspective to obtain a fusion result includes: Performing noise elimination processing on the data to be fused, and extracting decision features from the data to be fused after noise elimination, so as to obtain corresponding decision features; wherein the decision features include any one or more of the attribute features, time series features, spatial location features, and association relationship features of the data to be fused; Calculating the information entropy of each of the decision features, and calculating the mutual information between the decision features; Calculating the conditional entropy of the decision feature using the information entropy and the mutual information, and calculating the difference between the conditional entropy and the information entropy to obtain the information contribution provided by a single modality or a combination of modalities; Based on the information entropy and the mutual information, corresponding initial weights are assigned to the corresponding decision features, and the initial weights are reversely updated using the difference between the predicted results and the actual fusion results generated by the data fusion process of the decision features with the information contribution and the assigned initial weights, so as to obtain the target weights of the decision features; The BJS divergence calculation is performed on the joint entropy determined based on the information entropy, the mutual information, and the target weight, so as to optimize the selection of decision features and the adjustment of target weight in the data fusion process according to the BJS divergence calculation result.

2. The multimodal data fusion method according to claim 1, characterized in that: The step of cleaning the data to be fused into accurate data based on the DS evidence theory to obtain cleaned data to be fused includes: Using an effective label set covering algorithm, the data to be fused carrying a plurality of labels is divided into data groups to obtain a plurality of target data groups; wherein each target data group carries only one effective label; Inputting each of the target data sets into a classifier trained based on DS evidence theory, so that the classifier outputs the category probability of each category label under the valid label of the target data set; The target label of each of the to-be-fused data in the target data group is determined according to the conversion processing result of each of the category probabilities, so as to obtain the cleaned to-be-fused data carrying the target label.

3. The multimodal data fusion method according to claim 2, characterized in that: The step of determining the target label of each of the to-be-fused data in the target data group according to the conversion processing result of each of the category probabilities to obtain the cleaned to-be-fused data carrying the target label includes: Performing pignistic probability transformation on the probabilities of each category to obtain a single category probability result; The category label of the single category probability result with the highest probability is used as the target label to obtain the cleaned data to be fused with the target label.

4. The multimodal data fusion method according to any one of claims 1 to 3, characterized in that: Before determining the type of the pre-processed data to be fused in the data fusion requirement, the method further includes: Build and train a preset data classification model for determining whether a data type is a precise data type or a fuzzy data type.

5. The multimodal data fusion method according to claim 4, characterized in that: The type determination of the pre-processed data to be fused in the data fusion requirement includes: Inputting the data fusion requirement into a preset data classification model, so as to convert the multimodal data in the data fusion requirement into data to be fused in a matrix form through the preset data classification model; The data features of the data to be fused are extracted through the preset data classification model, so as to make a type judgment on the data to be fused according to the data features, and output a corresponding judgment result.

6. The multimodal data fusion method according to claim 4, characterized in that: The constructing and training of a preset data classification model for determining whether a data type is a precise data type or a fuzzy data type includes: The initial model of the Transformer architecture is trained for classification using a data training set that carries precise data type labels and fuzzy data type labels, so as to obtain a preset data classification model that has learned the corresponding relationship between the corresponding data type labels and the data classification results.

7. A multimodal data fusion device, characterized in that: include: A type judgment module is used to obtain multimodal data fusion requirements and perform type judgment on the pre-processed data to be fused in the data fusion requirements; data samples of multimodal data include text, numerical data, audio, and images; A first fusion module is used for, if the data type of the data to be fused is an accurate data type, performing data fusion processing on the data to be fused according to the information theory perspective to obtain a fusion result; The second fusion module is used for, if the data type of the data to be fused is a fuzzy data type, cleaning the data to be fused into precise data based on DS evidence theory to obtain cleaned data to be fused, and then jumping to execute the data fusion processing of the data to be fused from the perspective of information theory to obtain a fusion result; The first fusion module is specifically used to perform noise elimination processing on the data to be fused, and to extract decision features from the data to be fused after noise elimination, so as to obtain corresponding decision features; wherein the decision features include any one or more of the attribute features, time series features, spatial position features, and association relationship features of the data to be fused; calculate the information entropy of each of the decision features, and calculate the mutual information between each of the decision features; calculate the conditional entropy of the decision features using the information entropy and the mutual information, and calculate the difference between the conditional entropy and the information entropy to obtain the information contribution provided by a single mode or a combination of modes; assign corresponding initial weights to the corresponding decision features based on the information entropy and the mutual information, and reversely update the initial weights using the difference between the predicted results and the actual fusion results generated by the data fusion process of the decision features assigned with the information contribution and the initial weights, so as to obtain the target weights of each decision feature; perform BJS divergence calculation on the joint entropy determined based on the information entropy, the mutual information, and the target weight, so as to optimize the selection of decision features and the adjustment of target weights in the data fusion process according to the BJS divergence calculation results.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the multimodal data fusion method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the multimodal data fusion method according to any one of claims 1 to 6 are implemented.

Images