A classification system based on multimodal credible progressive fusion
By introducing multimodal trusted gradual fusion technology in the multimodal classification system, gradually integrating and optimizing multimodal features, the problem of insufficient fusion at the feature level of existing systems is solved, and the efficiency and robustness of data processing are improved. , improves the processing capability of multimodal data.
Patent Information
- Application Number
- CN202411132392.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-19
AI Technical Summary
The existing multimodal classification system fails to effectively integrate at the feature level, resulting in insufficient generalization ability and low robustness when processing diversified data.
Adopting a classification system based on multimodal trusted progressive fusion, through multimodal data preprocessing, feature extraction and screening, neural network module, feature fusion module, uncertainty modeling module and evidence fusion module, gradually integrate and optimize multiple Modal features, using techniques such as attention mechanism and Dirichlet distribution, dynamically adjust the importance of features and evaluate uncertainty.
Improve the efficiency and accuracy of multimodal data processing, enhance the generalization ability and robustness of the system, significantly improve the processing ability of multimodal data, and realize dynamic adjustment of the importance of different modal data characteristics. and optimization integration.
Smart Images

Figure CN118656683B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence and data processing technology, and in particular to a classification system based on multi-modal credible progressive fusion. Background Art
[0002] Multimodal fusion refers to the process of integrating information using data sources from different sensors, time or space, with the aim of improving the decision-making process, enhancing the interpretability of data or improving the performance of the system through such integration. In the field of machine learning and artificial intelligence, multimodal fusion technology enables the system to process and understand different types of data inputs, such as text, images, sounds, etc., to achieve more accurate and effective decision support or prediction.
[0003] Credible progressive fusion is a relatively new concept that emphasizes the gradual integration of information from different sources during the fusion process, and constantly evaluates and improves the credibility of the data and the reliability of the fusion results in the process. This approach pays special attention to how to effectively manage and utilize uncertainty at different stages to ensure that the final decision or classification result is both accurate and reliable.
[0004] Currently, a new method called Trusted Multi-View Classification (TMC) is emerging. This method dynamically integrates information from different views or modalities at the evidence level rather than at the feature or output level. Specifically, it uses the Dirichlet distribution to simulate the distribution of class probabilities, which are parameterized based on the evidence provided by different views or modalities, and combines the Dempster-Shafer theory to improve the reliability and robustness of classification. The core of this method is to accurately estimate the uncertainty of each view or modality, and accordingly improve the model's ability to handle abnormal samples, thereby improving accuracy, reliability, and robustness. Although this method improves the reliability of decision-making based on the evidence-based fusion strategy, it does not fuse multimodal data at the feature level.
[0005] Therefore, there is an urgent need for a multimodal credible progressive fusion classification system to solve the above problems. Summary of the invention
[0006] The purpose of the present invention is to provide a classification system based on multimodal credible progressive fusion in view of the deficiencies of the prior art.
[0007] The object of the present invention is achieved through the following technical solution: a classification system based on multimodal credible progressive fusion, comprising:
[0008] The multimodal data preprocessing module is used to preprocess the collected multimodal data of the object to be tested, so as to convert it into a format that can be processed by the feature extraction and screening module, and obtain the preprocessed multimodal data;
[0009] The feature extraction and screening module is used to extract the corresponding features from the preprocessed multimodal data and screen the extracted features to obtain the key features related to the classification task;
[0010] A neural network module, comprising a plurality of fully connected layers and a plurality of activation functions, each fully connected layer being connected to an activation function, and the neural network module is used to process the key features output by the feature extraction and screening module and the multimodal fusion features output by the feature fusion module to obtain deep feature representation;
[0011] The feature fusion module is used to fuse the deep feature representations of different modalities output by the neural network module using a fusion strategy based on the attention mechanism to obtain multimodal fusion features;
[0012] An uncertainty modeling module, which is used to model the uncertainty of the deep feature representation of each single modality and the multimodal fusion feature by using Dirichlet distribution and combining the outputs of the neural network module and the feature fusion module, so as to obtain the classification prediction probability and uncertainty of multiple modalities; and
[0013] The evidence fusion module is used to use evidence theory to fuse the classification prediction probabilities and uncertainties of multiple modalities output by the uncertainty modeling module to obtain a comprehensive evidence metric, and determine the prediction category of the classification based on the comprehensive evidence metric; wherein the comprehensive evidence metric includes a comprehensive prediction probability and overall uncertainty.
[0014] Furthermore, the multimodal data preprocessing module is specifically implemented through the following steps:
[0015] (1.1) Collecting multimodal data of the object to be tested, integrating and aligning the multimodal data of the same individual according to the classification category and the individual test object to form a set of multimodal samples; wherein the multimodal data includes image and text type data;
[0016] (1.2) Preprocessing the image-type multimodal data to convert it into a format that can be processed by the feature extraction and screening module, specifically including: first marking the region of interest in the multimodal image, and saving the annotation result of the region of interest in the form of a binary image as an annotated image; then calculating the enclosed area of the rectangular frame based on the annotated image, and setting the edge extension value; then intercepting the rectangular frames of the original image and the annotated image; adjusting the pixel values of the original image and standardizing it to the interval [0,1] to complete the image size adjustment; and finally obtaining the preprocessed image-type multimodal data;
[0017] (1.3) Preprocess the text-type multimodal data to convert it into a format that can be processed by the feature extraction and screening module. Specifically, the text data is first cleaned to remove HTML tags, special characters and punctuation marks; then the cleaned text data is segmented using a byte pair encoding method to merge character sequences or word fragments in the text data into a single symbol; finally, the text data is vectorized by converting each word into a dense vector of a predefined size using the Word2Vec word embedding model as the vector representation of the word, and the vector representations of all words are averaged to generate a one-dimensional vector representing the entire sentence or text fragment as the vector representation of the sentence.
[0018] Furthermore, the expression of the vector representation of the sentence is:
[0019]
[0020] in, Represents the vector representation of sentence S, the set of subwords contained in sentence S , represents the kth word in sentence S, and K represents the total number of K subwords in sentence S; Representing words The vector representation of is the output of the Word2Vec word embedding model; , d is the dimension of the vector space.
[0021] Furthermore, the feature extraction and screening module is specifically implemented through the following steps:
[0022] (2.1) Extracting features from preprocessed multimodal data, specifically: for preprocessed image-type multimodal data, using convolutional neural networks or radiomics methods to extract visual features of image data; for preprocessed text-type multimodal data, using natural language processing tools to extract semantic features of text data; for preprocessed audio-type multimodal data, using acoustic models to extract acoustic features of audio data;
[0023] (2.2) The features extracted in step (2.1) are screened by using variance analysis, Spearman correlation analysis, and least absolute shrinkage and selection operator methods. By evaluating the strength of association between the features and the classification task, the most predictive key features related to the classification task are screened out, and the key feature vector output by the feature extraction and screening module is obtained.
[0024] Furthermore, the radiomics method includes the following feature extraction algorithm:
[0025] Histogram oriented gradients, scale-invariant feature transforms or accelerated robust features: used to detect and describe local features in images;
[0026] Local binary pattern: used to describe texture features that focus on illumination changes;
[0027] pyRadiomics library: used to extract texture features of regions of interest in medical images.
[0028] Furthermore, the feature fusion module is specifically implemented through the following steps:
[0029] (4.1) Mapping the deep feature representations of each modality output by the neural network module to a common feature space through a feature mapping layer to obtain the features of each mapped modality; wherein the feature mapping layer includes a fully connected layer and an activation function, and the weights and biases of the fully connected layer corresponding to each modality and the ReLU nonlinear activation function are used to enhance the expressiveness of the features;
[0030] (4.2) Perform feature concatenation on all mapped modal features to obtain the total feature tensor after feature concatenation;
[0031] (4.3) The total feature tensor is fed into the attention mechanism layer to evaluate the relative importance of each modality feature and assign corresponding attention weights;
[0032] (4.4) The deep feature representations of each modality output by the neural network module are concatenated and multiplied with the attention weight to obtain the weighted multimodal fusion features;
[0033] (4.5) In a single progressive iteration process, the multimodal fusion features are fed back to the neural network module, and the multimodal fusion features are concatenated with the input feature vectors of each modality of the neural network module to obtain the input feature vector after feature concatenation, which is used as the current input feature vector of the neural network module, and the deep feature representation corresponding to each modality is output; the deep feature representation corresponding to each modality is used as the input of the feature fusion module, and steps (4.1) to (4.4) are repeated to complete the update of the multimodal fusion features;
[0034] (4.6) Repeat step (4.5) and continuously update and optimize the multimodal fusion features through multiple iterations to finally obtain a comprehensive multimodal fusion feature.
[0035] Furthermore, the uncertainty modeling module is specifically implemented through the following steps:
[0036] (5.1) The deep feature representation of each single modality output by the neural network module or the multimodal fusion feature output by the feature fusion module is recorded as , , where V represents the total number of single modalities output by the neural network module, and c represents the fusion modality. Through the sigmoid activation function, the corresponding evidence vector is obtained, which is expressed as:
[0037]
[0038] in, express is the evidence vector for category q, represents the sigmoid activation function, represents the deep feature representation of the first modality output by the neural network module, represents the deep feature representation of the second modality output by the neural network module, represents the deep feature representation of the Vth modality output by the neural network module, Represents the multimodal fusion features output by the feature fusion module;
[0039] (5.2) The evidence vector obtained in step (5.1) is converted into Dirichlet distribution parameters by the following calculation formula:
[0040]
[0041] in, Represents the evidence vector The corresponding Dirichlet distribution parameters;
[0042] (5.3) Based on the Dirichlet distribution parameters, the Dirichlet intensity is calculated by the following formula:
[0043]
[0044] in, is the Dirichlet intensity, q represents the category, and Q represents the total number of categories;
[0045] (5.4) Calculate the predicted probability and uncertainty of each class according to the Dirichlet intensity:
[0046]
[0047]
[0048] in, represents the predicted probability that the mode v is the category q, represents the uncertainty of the mode v.
[0049] Furthermore, the evidence fusion module is specifically implemented through the following steps:
[0050] (6.1) For each mode v and category q, define the basic probability distribution , Indicates the degree to which the evidence provided by modality v supports category q. The basic probability distributions from each modality are combined using the combination rules of evidence theory to obtain the fused basic probability distribution;
[0051] (6.2) Based on the fused basic probability distribution, repeat the fusion process in step (6.1) until the basic probability distribution of all modes of category q is fused, and the comprehensive prediction probability of category q is obtained. and overall uncertainty U;
[0052] (6.3) Repeat steps (6.1) to (6.2) until the comprehensive prediction probability and overall uncertainty of all categories are obtained, and the prediction category of the classification is determined based on this.
[0053] Furthermore, in step (6.1), it is assumed that there are two modes and , and its basic probability distribution for category q is and , the combination rule of evidence theory is used to fuse the basic probability distributions from each modality. The fused basic probability distribution is calculated by the following formula:
[0054]
[0055]
[0056] Among them, X and Y are sets of evidence under different modes, is the conflict coefficient.
[0057] Furthermore, in step (6.2), the comprehensive prediction probability is the sum of all basic probability assignments about category q, and the overall uncertainty U is expressed as the sum of basic probability assignments that do not belong to any specific category, and their expressions are:
[0058]
[0059]
[0060] in, is the sum of the basic probability assignments of all supporting evidence sets X containing category q, is the sum of all the basic probability assignments that do not support any particular class.
[0061] Compared with the prior art, the present invention has the following beneficial effects:
[0062] (1) The present invention not only focuses on how to extract and integrate evidence information from different modalities, but also integrates them progressively at the feature level. Compared with fusing the features of all modalities at one time, progressive fusion can gradually integrate the features according to their influence on the classification results, give priority to the most informative features, and improve the efficiency of feature utilization; the present invention can better understand and utilize the complementary information between modalities by progressively learning the correlation between different modalities, thereby demonstrating stronger generalization ability when facing diverse data; in the progressive fusion process, the system can evaluate the contribution and reliability of each modality before integrating more modal information, which helps to effectively identify and reduce the negative impact of abnormal modalities.
[0063] (2) The present invention effectively improves the efficiency and accuracy of multimodal data processing by introducing a progressive feature fusion mechanism, which is beneficial to improving feature utilization efficiency, system generalization ability and robustness. It not only enhances the performance of the classification system, but also provides new ideas for the development of multimodal data processing technology in the future.
[0064] (3) The present invention adopts a method of multimodal trusted progressive fusion, which effectively improves the accuracy and reliability of the classification system in processing complex multimodal data and realizes the dynamic adjustment and optimal integration of the importance of features of different modal data; the present invention utilizes an advanced attention mechanism to enable the system to adaptively identify and enhance key information, thereby significantly improving the processing capability of multimodal data; the implementation method of the present invention is simple and flexible, does not require major modifications to the existing data processing framework, and can maintain high performance in various application scenarios; in addition, the progressive fusion strategy of the present invention further enhances the system's ability to focus on information during the iteration process, ensuring the performance improvement of the system in a continuous learning environment. In general, the present invention not only significantly reduces the difficulty of processing complex multimodal data, but also ensures efficient and accurate classification results, providing a reliable and efficient solution for multimodal data analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 It is an architectural flow chart of a classification system based on multimodal trusted progressive fusion of the present invention;
[0066] Figure 2 is a flow chart of the architecture of the feature fusion module of the present invention;
[0067] Figure 3 is an architectural flow chart of the uncertainty modeling module of the present invention;
[0068] Figure 4 is a flow chart of the architecture of the neural network module of the present invention;
[0069] Figure 5 It is a flowchart of the architecture of the progressive iteration of multimodal fusion of the present invention. DETAILED DESCRIPTION
[0070] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Obviously, the drawings used in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can also be obtained based on these drawings without paying creative labor. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present invention. On the contrary, they are only examples of devices and methods consistent with some aspects of the present invention as detailed in the attached claims.
[0071] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0072] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0073] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.
[0074] Example 1
[0075] See also Figure 1 The classification system based on multimodal credible progressive fusion of the present invention specifically includes a multimodal data preprocessing module, a feature extraction and screening module, a neural network module, a feature fusion module, an uncertainty modeling module and an evidence fusion module.
[0076] In this embodiment, the multimodal data preprocessing module is used to preprocess the collected multimodal data of the object to be tested, so as to convert it into a format that can be processed by the feature extraction and screening module, thereby obtaining preprocessed multimodal data.
[0077] Furthermore, the multimodal data preprocessing module is specifically implemented through the following steps:
[0078] (1.1) Collect multimodal data of the object to be tested, and integrate and align the multimodal data of the same individual according to the classification category and individual test object to form a set of multimodal samples. Among them, multimodal data includes images, text, audio and other types of data.
[0079] (1.2) Preprocessing the image-type multimodal data to convert it into a format that can be processed by the feature extraction and screening module, specifically including: first marking the region of interest (ROI) in the multimodal image, and saving the annotation result of the ROI in the form of a binary image as an annotated image, where 1 represents the foreground and 0 represents the background; then calculating the enclosed area of the rectangular frame based on the annotated image, and setting the edge extension value; then intercepting the rectangular frames of the original image and the annotated image; adjusting the pixel values of the original image and standardizing it to the [0,1] interval to complete the image size adjustment; and finally obtaining the preprocessed image-type multimodal data.
[0080] It should be noted that the annotated image includes the marked region of interest. When marking the region of interest in the image, manual annotation or computer algorithm can be used to automatically outline and manually correct the region of interest.
[0081] (1.3) Preprocess the multimodal data of text type to convert it into a format that can be processed by the feature extraction and screening module. Specifically, the text data is first cleaned to remove HTML tags, special characters and punctuation marks that may interfere with the processing flow; then the Byte Pair Encoding (BPE) method is used to segment the cleaned text data to merge common character sequences or word fragments in the text data into a single symbol; finally, the text data is vectorized, which is the last and most critical step of text data preprocessing. Specifically, the Word2Vec word embedding model is used to convert each word into a dense vector of a predefined size as the vector representation of the word, and the vector representation of all words is averaged to generate a one-dimensional vector representing the entire sentence or text fragment as the vector representation of the sentence for subsequent processes. The vector representation of the sentence obtained at this time is a numerical form that can be processed by the feature extraction and screening module.
[0082] It should be understood that the BPE method for word segmentation can better handle rare words and spelling variations than traditional word segmentation methods because it breaks down unknown or uncommon words into smaller, recognizable segments.
[0083] It should be noted that the Word2Vec word embedding model is used to convert each word into a dense vector of a predefined size. The Word2Vec word embedding model generates a vector representation that captures word meaning and grammatical relations by learning the context of the word in a large corpus. The vector representation of each word can be combined, usually by averaging the vector representations of these words, to generate a one-dimensional vector representing the entire sentence or text fragment. The resulting vector encodes the key semantic information of the original text and is ready to be used as input to a machine learning or deep learning model.
[0084] Furthermore, the expression of the sentence vector representation is:
[0085] (1)
[0086] in, Represents the vector representation of sentence S, the set of subwords contained in sentence S , represents the kth word in sentence S, and K represents the total number of K subwords in sentence S; Representing words The vector representation of is the output of the Word2Vec word embedding model; , d is the dimension of the vector space. The average operation assumes that each subword contributes equally to the semantics of the sentence.
[0087] (1.4) Preprocess other special types of multimodal data. Specifically, make corresponding adjustments based on the data itself and finally process it into a one-dimensional vector.
[0088] For example, clinical information and data in medicine, including but not limited to blood routine, various biochemical test data, patients' physical clinical data and related epidemiological data. These data usually come from patients' medical records, laboratory reports and epidemiological questionnaires. After collecting the data, the data is cleaned to remove invalid data, outliers and duplicate records. This step ensures the quality of the data and the accuracy of the analysis. The unstructured data is then converted into a format that can be processed by the machine learning model. For example, information such as symptoms and medical history is encoded as numeric or categorical variables. Numerical clinical features are standardized, such as standardizing indicators such as red blood cell counts and white blood cell counts in blood routine to a unified range, usually the [0,1] interval or standard normal distribution, to eliminate the influence of different dimensions and ranges.
[0089] It should be understood that the purpose of preprocessing the collected multimodal data, including but not limited to the above-mentioned images, text and other special types of data, is to convert the collected multimodal data into a format suitable for subsequent feature extraction and screening modules to process, which may include image resizing, text segmentation, audio sampling rate conversion, etc.
[0090] In this embodiment, the feature extraction and screening module is used to extract corresponding features from the preprocessed multimodal data, and screen the extracted features to obtain key features related to the classification task.
[0091] It should be understood that the key features output by the feature extraction and screening module are the features that are most helpful for the classification task. By screening the extracted features, the dimension can be reduced and redundant and irrelevant information can be eliminated.
[0092] Furthermore, the feature extraction and screening module is implemented specifically through the following steps:
[0093] (2.1) Extracting features from preprocessed multimodal data, including but not limited to: for preprocessed image-type multimodal data, using convolutional neural networks (CNN) or radiomics to extract visual features of image data; for preprocessed text-type multimodal data, using natural language processing (NLP) tools to extract semantic features of text data; for preprocessed audio-type multimodal data, using acoustic models to extract acoustic features of audio data; etc.
[0094] Furthermore, the radiomics method includes the following feature extraction algorithms: Histogram of Oriented Gradient (HOG), Scale-Invariant Feature Transform (SIFT) or Speeded Up Robust Features (SURF): used to detect and describe local features in images; Local Binary Patterns (LBP): a powerful texture description feature that is robust to illumination changes; if the image is a medical image, the pyRadiomics library is used to extract the texture features of the tumor ROI, such as the gray-level co-occurrence matrix (GLCM), gray-level dependency matrix (GLDM), gray-level run length matrix (GLRLM) and gray-level zone size matrix (GLSZM), etc., and 1218 visual features are extracted for each imaging modality.
[0095] It should be understood that in the current step (2.1), existing methods can be used to extract features, such as using a pre-trained visual feature extraction network I3D to extract visual features of the image. Of course, other CNNs can also be used to extract visual features of the image.
[0096] (2.2) The features extracted in step (2.1) are screened using methods such as analysis of variance (ANOVA), Spearman correlation analysis, and least absolute shrinkage and selection operator (LASSO). The correlation strength between the features and the classification task is evaluated using the above statistical and machine learning methods, and the key features with the most predictive ability related to the classification task are screened out to obtain the key feature vector output by the feature extraction and screening module.
[0097] Furthermore, the ANOVA method is used to detect whether there are significant differences in the means of characteristics between different groups. The calculation formula is:
[0098] (2)
[0099] Among them, F represents the calculated variance of the group. If the calculated F value is greater than the critical value or the p value is less than the set significance level (such as 0.05), it is considered that there are significant differences between different groups. The larger the F value, the greater the difference. Represents the within-group variance, which is used to measure the deviation of the sample values in each group from the mean of the group; It represents the between-group variance, which is used to measure the difference between the means of different groups.
[0100] Furthermore, the Spearman correlation analysis method is used to calculate the Pearson correlation coefficient. The Pearson correlation coefficient is a non-parametric indicator to measure the degree of association between two level variables. After converting the variables into levels, the Pearson correlation coefficient between the two level variables is calculated. The calculation formula is:
[0101] (3)
[0102] in, represents the Pearson correlation coefficient between two ordinal variables, It represents the difference in the levels of the corresponding sample point i in the two variables, and n is the vertical number of sample points. The value is between -1 and 1. Values close to 1 indicate a strong positive correlation. Values close to -1 indicate a strong negative correlation. Values closer to 0 indicate smaller correlation.
[0103] Furthermore, the LASSO method is a regression analysis method that produces a sparser solution by imposing an L1 penalty (i.e., the sum of the absolute values of the coefficients) on the coefficient estimates. Its objective function is:
[0104] (4)
[0105] Where β is the coefficient, is the initial value of the coefficient, j is the index of the coefficient, p is the total number of features, i is the index of the sample, n is the total number of samples, is the response variable, is the predictor variable and λ is the regularization parameter, which controls the intensity of the penalty.
[0106] In this embodiment, the neural network module includes multiple fully connected layers (FCs) and multiple activation functions. Each fully connected layer is connected to an activation function. The neural network module is used to process the key features output by the feature extraction and screening module and the multimodal fusion features output by the feature fusion module to obtain deep feature representation.
[0107] It should be noted that the neural network module is designed as a multi-level architecture that can capture and represent complex patterns and relationships in data, which is critical for achieving deep feature fusion and extracting high-level abstract features. Among them, the fully connected layer is the most common network layer in the neural network. Each fully connected layer is connected to all the activation outputs of the previous layer and can learn nonlinear and complex patterns in the input data.
[0108] For example, in Figure 4 In the neural network module shown, a three-layer structure is designed for each modal feature, including three connection layers and three activation functions. The first fully connected layer is connected with a ReLU activation function, which is then connected to the second fully connected layer. The second fully connected layer is connected with a ReLU activation function, which is then connected to the third fully connected layer. The third fully connected layer is connected with a sigmoid activation function, which is designed to further refine and enhance the representation ability of the features. For the input feature vector of the ath modality , first transform it through the first fully connected layer and the ReLU activation function to obtain:
[0109] (5)
[0110] The feature vector after the first layer transformation Pass it to the second fully connected layer and the ReLU activation function for transformation, and we get:
[0111] (6)
[0112] Similarly, the feature vector after the second layer transformation Pass it to the third fully connected layer and the sigmoid activation function, and we get:
[0113] (7)
[0114] in, It represents the feature vector after the third layer transformation, and also represents the deep feature representation output by the neural network module; represents the fully connected layer, and denote the weight and bias of the fully connected layer respectively, represents the ReLU activation function, Represents the sigmoid activation function.
[0115] In this three-layer fully connected network, each layer is designed to extract higher-order feature expressions and enhance the interaction between different features. This series of nonlinear transformations enables the neural network module to learn the deeper structure of the input features and ultimately output a highly abstract deep feature representation. .
[0116] It should be noted that the deep feature representation finally output by the neural network module is used in the subsequent fusion and classification steps by adjusting the weights and bias As well as the type of activation function, the neural network module can ensure that effective features are learned from multimodal inputs and model complex data relationships.
[0117] In this embodiment, the feature fusion module is used to fuse the deep feature representations of different modalities output by the neural network module using a fusion strategy based on the attention mechanism to obtain multimodal fusion features.
[0118] It should be understood that the feature fusion module is a key component module of the classification system described in the present invention. Its goal is to integrate information from different data sources, that is, to integrate features of multiple modalities (such as text, images, sounds, etc.) to improve the system's understanding of the inherent correlation of multimodal data.
[0119] Furthermore, if Figure 2 As shown in the figure, the feature fusion module is implemented through the following steps:
[0120] (4.1) The deep feature representation of each modality output by the neural network module is mapped to a common feature space through the feature mapping layer to obtain the features of each mapped modality. The feature mapping layer includes a fully connected layer and an activation function, which uses the weights and biases of the fully connected layer corresponding to each modality and the ReLU nonlinear activation function to enhance the expressiveness of the features.
[0121] For example, the deep feature representation of the ath modality output by the neural network module is As an example, the mapping process of the feature mapping layer can be expressed as:
[0122] (8)
[0123] in, represents the ath modal feature after mapping, and Represents the fully connected layer in the feature mapping layer The weights and biases of is a non-linear activation function applied to the mapped result.
[0124] (4.2) All the mapped modal features are concatenated to obtain the total feature tensor after feature concatenation. The feature concatenation process can be expressed as:
[0125] (9)
[0126] Among them, Z represents the total feature tensor, Represents feature concatenation.
[0127] (4.3) The total feature tensor is fed into the attention mechanism layer to evaluate the relative importance of each modality feature and assign corresponding attention weights, specifically through an adaptive calibration function Transform the total feature tensor Z to obtain the corresponding attention weight, which can be expressed as:
[0128] (10)
[0129] in, represents the attention weight; Represents the sigmoid activation function, the purpose is to make the value between 0-1; It can be any deep learning layer or network structure. In this embodiment, a fully connected layer is used. is its weight parameter.
[0130] It should be understood that in the feature fusion module, the relative importance of different modal input features can be dynamically adjusted through the attention mechanism layer, which significantly enhances the feature fusion module's ability to identify key information. By using the attention mechanism, the feature fusion module not only improves the model's adaptability and dynamic adjustment capabilities for the importance of different modal features, but also can effectively extract and integrate key features when facing challenging multimodal data. This attention mechanism shows its powerful performance especially when processing complex or incompletely aligned multimodal information. The attention weight calculated in the current step can be used to guide the subsequent feature fusion process.
[0131] (4.4) The deep feature representations of each modality output by the neural network module are concatenated and multiplied with the attention weight to obtain the weighted multimodal fusion feature, which can be expressed as:
[0132] (11)
[0133] Among them, C represents the weighted multimodal fusion features.
[0134] It should be understood that the weighted multimodal fusion features significantly enhance the ability to identify key information and improve the accuracy and reliability of classification.
[0135] (4.5) If Figure 5 As shown, in a single progressive iteration process, the multimodal fusion features are fed back to the neural network module, and the multimodal fusion features are concatenated with the input feature vectors of each modality of the neural network module to obtain the input feature vector after feature concatenation, which is used as the current input feature vector of the neural network module, and the deep feature representation corresponding to each modality is output; the deep feature representation corresponding to each modality is used as the input of the feature fusion module, and steps (4.1) to (4.4) are repeated to complete the update of the multimodal fusion features, so that the feature fusion module can introduce multimodal mutual information in the early stage, and refine the feature fusion module's processing capabilities for different modal data.
[0136] Specifically, the single progressive iteration process and the neural network module are the same process, with only the input of the first fully connected layer being different. For the input feature vector of the ath mode , the vector After feature concatenation with the multimodal fusion feature C, it is transformed through the first fully connected layer of the neural network module and the ReLU activation function, which can be expressed as:
[0137] (12)
[0138] in, represents the multimodal fusion features in the t-th iteration process, It represents the feature vector after the transformation of the first fully connected layer and the ReLU activation function in the tth iteration. The subsequent steps are the same as those in formula (6) to (11).
[0139] (4.6) Repeat step (4.5) and continuously update and optimize the multimodal fusion features through multiple iterations. Finally, a comprehensive multimodal fusion feature is obtained, which helps to more effectively capture the complementary information from different modalities and ensure the continuous improvement of performance.
[0140] It should be understood that the above process embodies a dynamic feature fusion strategy implemented through the attention mechanism, which not only improves the adaptability and dynamic adjustment ability of the feature fusion module to the importance of different modal data, but also introduces multimodal mutual information at an early stage through a progressive fusion method, thereby continuously improving the overall performance.
[0141] It should be noted that this progressive fusion method can not only adaptively weight different modal data, but also refine the focus of the feature fusion module on information in each progressive fusion step to ensure the continuous improvement of its performance; through this strategy, it is possible to more effectively capture and integrate information from different modalities to improve the accuracy and reliability of classification. The final input in the feature fusion module is the fused feature, and the multimodal fusion feature finally output by the feature fusion module and the deep feature representation of each single modality output by the neural network module are used as the input of the uncertainty modeling module.
[0142] In this embodiment, the uncertainty modeling module is used to adopt Dirichlet distribution, combined with the output of the neural network module and the feature fusion module, to model the uncertainty of the deep feature representation of each single modality and the multimodal fusion feature, so as to evaluate the prediction uncertainty of the quantitative system and obtain the classification prediction probability and uncertainty of multiple modalities. The uncertainty modeling module is the key to understanding the prediction confidence and is very helpful for improving the reliability and robustness of the system.
[0143] It should be understood that the Dirichlet distribution is a continuous multivariate probability distribution, which is often used to represent the situation where the sum of the probabilities of multiple random variables is 1, so it is suitable for probability modeling as the output of this uncertainty modeling module.
[0144] Furthermore, if Figure 3 As shown in Figure 1, the uncertainty modeling module is implemented through the following steps:
[0145] (5.1) The deep feature representation of each single modality output by the neural network module or the multimodal fusion feature output by the feature fusion module is recorded as , , where V represents the total number of single modalities output by the neural network module, and c represents the fusion modality. Through the sigmoid activation function, the corresponding evidence vector is obtained, which is expressed as:
[0146] (13)
[0147] in, express is the evidence vector for category q, represents the sigmoid activation function, represents the deep feature representation of the first modality output by the neural network module, represents the deep feature representation of the second modality output by the neural network module, represents the deep feature representation of the Vth modality output by the neural network module, Represents the multimodal fusion features output by the feature fusion module.
[0148] (5.2) The evidence vector obtained in step (5.1) is converted into Dirichlet distribution parameters by the following calculation formula:
[0149] (14)
[0150] in, Represents the evidence vector The corresponding Dirichlet distribution parameter.
[0151] (5.3) Based on the Dirichlet distribution parameters, the Dirichlet intensity is calculated by the following formula:
[0152] (15)
[0153] in, is the Dirichlet intensity, q represents the category, and Q represents the total number of categories.
[0154] (5.4) Calculate the predicted probability and uncertainty of each class according to the Dirichlet intensity:
[0155] (16)
[0156] (17)
[0157] in, represents the predicted probability that the mode v is the category q, represents the uncertainty of the modality v. The predicted probability reflects the evaluation of each category, while the uncertainty measures the uncertainty of the system about its prediction.
[0158] It should be appreciated that the uncertainty modeling process described above provides a probability estimate for each classification along with an accompanying uncertainty measure, which enables not only probabilistic predictions but also quantification of the confidence in the predictions, which is very valuable for decision making and risk assessment, especially when more interpretability of the classification results is required.
[0159] In this embodiment, the evidence fusion module is used to fuse the classification prediction probabilities and uncertainties of multiple modalities output by the uncertainty modeling module using evidence theory to obtain a comprehensive evidence metric, and determine the prediction category of the classification according to the comprehensive evidence metric. The comprehensive evidence metric includes the comprehensive prediction probability and the overall uncertainty.
[0160] It should be understood that the evidence fusion module can incorporate the knowledge of uncertainty into the final decision-making process. Evidence fusion integrates information from different modalities and neural network layers by using evidence (Dempster-Shafer, DS) theory, so that the final classification decision is not only based on the original data features, but a collection of information that integrates all available evidence and uncertainty estimates, and finally obtains the integrated overall uncertainty and various classification prediction outputs. Among them, DS theory provides a framework for reasoning and decision-making in the presence of uncertainty and ambiguity. In the context of multimodal data fusion, DS theory is used to merge evidence from different sources to improve the prediction probability estimate for classification tasks and provide a comprehensive measure of uncertainty.
[0161] Furthermore, the evidence fusion module is implemented through the following steps:
[0162] (6.1) For each mode v and category q, define a basic probability assignment (BPA) for it: , It represents the degree to which the evidence provided by modality v supports category q. The basic probability distributions from each modality are combined using the combination rules of evidence theory to obtain the fused basic probability distribution.
[0163] Furthermore, suppose there are two modes and , and its basic probability distribution for category q is and , the combination rule of evidence theory is used to fuse the basic probability distributions from each modality. The fused basic probability distribution is calculated by the following formula:
[0164] (18)
[0165] Among them, X and Y are sets of evidence under different modes, is the conflict coefficient, calculated as the sum of the products of the basic probability assignments of all mutually exclusive evidences, expressed as:
[0166] (19)
[0167] (6.2) Based on the fused basic probability distribution, repeat the fusion process in step (6.1) until the basic probability distribution of all modes of category q is fused, and the comprehensive prediction probability of category q is obtained. and the overall uncertainty U.
[0168] Furthermore, the comprehensive prediction probability is the sum of all basic probability assignments about category q, and the overall uncertainty U is expressed as the sum of basic probability assignments that do not belong to any specific category, and their expressions are:
[0169] (20)
[0170] (twenty one)
[0171] in, is the sum of the basic probability assignments of all supporting evidence sets X containing category q, is the sum of all the basic probability assignments that do not support any particular class, After softmax, the final prediction output is obtained.
[0172] It should be understood that the fusion rules described in the above process can be extended to more modal cases. For example, if there are three modalities, the basic probability distribution of category q is and , then first use formula (18) and formula (19) to get the mode and Basic probability distribution after fusion , and then use formula (18) and formula (19) to and modal The basic probability distribution of Similarly, it can be extended to more modal cases.
[0173] (6.3) Repeat steps (6.1) to (6.2) until the comprehensive prediction probability and overall uncertainty of all categories are obtained, and the prediction category of the classification is determined based on this.
[0174] It should be understood that the evidence fusion module considers information from all modalities and takes uncertainty into account when making predictions, thereby improving the reliability and robustness of the decision-making process.
[0175] In summary, through the above technical solutions, each module in the present invention is designed for its specific purpose and plays a specific role in the entire system. The multimodal data preprocessing module, feature extraction and screening module, neural network module, feature fusion module, uncertainty modeling module and evidence fusion module work together to ensure that the information extracted from the multimodal data can be fully utilized and converted into reliable classification decisions. In addition, through the uncertainty modeling module and the evidence fusion module, the system can also provide a credibility assessment of its decision, which is particularly important for application scenarios that require high credibility output.
[0176] Example 2
[0177] The following uses multimodal ovarian cancer early screening data as an implementation case to explain in detail the classification system based on multimodal credible progressive fusion of the present invention, which specifically includes the following steps:
[0178] (1) Multimodal data preprocessing module: The collected ovarian cancer-related ultrasound (US), magnetic resonance imaging (MRI) and computed tomography (CT) images are preprocessed, and the region of interest (ROI) is marked in the multimodal image. The annotation result of the ROI is saved in the form of a binary image, which is named the annotated image, where 1 represents the foreground and 0 represents the background. The enclosed area of the rectangular box of the annotated image is calculated, and the edge extension value is set. Then, the rectangular boxes of the original image and the annotated image are intercepted. The pixel values of the original image are adjusted and standardized to the [0,1] interval.
[0179] (2) Feature extraction and screening module: The pyRadiomics library was used to extract the texture features of the tumor ROI, such as the gray-level co-occurrence matrix (GLCM), gray-level dependency matrix (GLDM), gray-level run length matrix (GLRLM), and gray-level zone size matrix (GLSZM). 1218 features were extracted for each image modality. After the features were extracted, the extracted features were screened using analysis of variance (ANOVA), Spearman correlation analysis, and least absolute shrinkage and selection operator (LASSO) methods to select the features that were most helpful for the classification task. Finally, 7 key features were extracted from US images, 26 key features were extracted from MRI-DWI images, 9 key features were extracted from MRI-T2 images, and 5 key features were extracted from CT images.
[0180] (3) Neural network module processing: The features of each modality obtained in step (2) are input into an independent neural network module consisting of three fully connected layers to further process the filtered features. The input and output sizes of this step remain unchanged.
[0181] (4) Feature fusion module: The deep feature representations of different modalities obtained in step (3) are fused to obtain multimodal fusion features. This step is the core of the present invention and is divided into the following sub-steps:
[0182] (4.1) Feature mapping: For ovarian cancer image data, the input features from different modalities (ultrasound, MRI, CT) are mapped into a common feature space. This process is implemented through a fully connected layer, using modality-specific weights and bias parameters, and using the ReLU nonlinear activation function to enhance the expressiveness of each modality feature.
[0183] (4.2) Mapping feature concatenation: Perform feature concatenation on all mapped modal features to obtain the total feature tensor after feature concatenation.
[0184] (4.3) Attention weight calculation: The attention mechanism is applied to the total feature tensor after feature concatenation to dynamically evaluate and adjust the relative importance of each modality feature. Especially in ovarian cancer image analysis, this step calculates the attention weight of each modality feature to optimize the subsequent feature fusion effect.
[0185] (4.4) Feature fusion: After the input features from different modalities (ultrasound, MRI, CT) are concatenated, they are multiplied using the calculated attention weights to form a comprehensive multimodal fusion feature. This step significantly enhances the neural network model's ability to identify key information in ovarian cancer image data processing, thereby improving the accuracy and reliability of classification.
[0186] (4.5) Progressive fusion: The features fused through the attention mechanism are fed back to the neural network module, so that each iterative update is based on the fused multimodal information, further refining the processing capabilities of ovarian cancer image data.
[0187] (4.6) Iterative update: Through multiple iterations, the multimodal fusion features are continuously updated and optimized, so that the system can more effectively capture complementary information from different modalities. In the processing of ovarian cancer image data, this step ensures continuous improvement of performance, thereby improving the accuracy of ovarian cancer diagnosis.
[0188] (5) Uncertainty modeling: Uncertainty modeling is performed on US, MRI, CT features and fusion features obtained through feature fusion. This process aims to evaluate and quantify the prediction uncertainty of the integrated multimodal fusion features in ovarian cancer classification decisions. By connecting the parameters of the Dirichlet distribution, the system can provide a quantitative assessment of uncertainty for each modality and fusion feature. This comprehensive uncertainty modeling method enables the classification system to not only understand the confidence of single modality data, but also evaluate the comprehensive confidence of the fused features, thereby providing a more comprehensive and reliable uncertainty assessment for the final classification decision.
[0189] (6) Evidence fusion: In the evidence fusion stage, the uncertainty knowledge of a single modality and the uncertainty knowledge of the fused features are integrated into the final decision-making process. The Dempster-Shafer theory or Bayesian method is used to effectively integrate the predictive evidence from each modality and the uncertainty assessment of the fused features. This comprehensive evidence fusion method further enhances the information basis for classification decisions, ensuring that the final decision is not only based on each individual modality and their combined features, but also comprehensively considers the uncertainty of the overall prediction, greatly improving the accuracy of ovarian cancer classification and the reliability of the overall system.
[0190] Exemplarily, in order to verify the effectiveness of the method described in the present invention, a comparative experiment was conducted on a real data set. The deep learning framework used in the experiment is Pytorch, and a deep model related to the present invention is constructed. The data set used was provided by a hospital. It is an ovarian cancer multi-classification data set. The category labels are divided into two categories: borderline and malignant, with a total of 419 data. The data set is divided into a training set of 251 cases, 186 malignant cases, and 65 borderline cases; a validation set of 84 cases, 63 malignant cases, and 21 borderline cases; a test set of 84 cases, 63 malignant cases, and 21 borderline cases. The results are shown in Table 1.
[0191] Table 1: Comparative experimental results
[0192]
[0193] From the experimental results shown in Table 1, it can be concluded that the comprehensive performance of the method of the present invention on the test set exceeds the existing technology, which is specifically reflected in the classification accuracy, sensitivity and comprehensive uncertainty evaluation brought about by the fusion of different modal data. Through the above steps, a highly accurate and reliable method and system can be provided for the early screening of ovarian cancer, which is of great significance to improving the early diagnosis rate and reducing the misdiagnosis rate.
[0194] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A classification system based on multimodal credible progressive fusion, characterized in that: include: A multimodal data preprocessing module is used to preprocess the collected multimodal data of the object to be tested, so as to convert it into a format that can be processed by the feature extraction and screening module, and obtain preprocessed multimodal data; wherein the multimodal data includes image and text type data; The feature extraction and screening module is used to extract the corresponding features from the preprocessed multimodal data and screen the extracted features to obtain the key features related to the classification task; A neural network module, comprising a plurality of fully connected layers and a plurality of activation functions, each fully connected layer being connected to an activation function, and the neural network module is used to process the key features output by the feature extraction and screening module and the multimodal fusion features output by the feature fusion module to obtain deep feature representation; The feature fusion module is used to fuse the deep feature representations of different modalities output by the neural network module using a fusion strategy based on the attention mechanism to obtain multimodal fusion features; The feature fusion module is implemented in the following steps: (4.1) Mapping the deep feature representations of each modality output by the neural network module to a common feature space through a feature mapping layer to obtain the features of each mapped modality; wherein the feature mapping layer includes a fully connected layer and an activation function, and the weights and biases of the fully connected layer corresponding to each modality and the ReLU nonlinear activation function are used to enhance the expressiveness of the features; (4.2) Perform feature concatenation on all mapped modal features to obtain a total feature tensor after feature concatenation; (4.3) The total feature tensor is fed into the attention mechanism layer to evaluate the relative importance of each modality feature and assign corresponding attention weights; (4.4) The deep feature representations of each modality output by the neural network module are concatenated and multiplied with the attention weight to obtain the weighted multimodal fusion features; (4.5) In a single progressive iteration process, the multimodal fusion features are fed back to the neural network module, and the multimodal fusion features are concatenated with the input feature vectors of each modality of the neural network module to obtain the input feature vector after feature concatenation, which is used as the current input feature vector of the neural network module, and the deep feature representation corresponding to each modality is output; the deep feature representation corresponding to each modality is used as the input of the feature fusion module, and steps (4.1) to (4.4) are repeated to complete the update of the multimodal fusion features; (4.6) Repeat step (4.5) and continuously update and optimize the multimodal fusion features through multiple iterations, and finally obtain a comprehensive multimodal fusion feature; An uncertainty modeling module, which is used to model the uncertainty of the deep feature representation of each single modality and the multimodal fusion feature by using Dirichlet distribution and combining the outputs of the neural network module and the feature fusion module, so as to obtain the classification prediction probability and uncertainty of multiple modalities; and The evidence fusion module is used to use evidence theory to fuse the classification prediction probabilities and uncertainties of multiple modalities output by the uncertainty modeling module to obtain a comprehensive evidence metric, and determine the prediction category of the classification based on the comprehensive evidence metric; wherein the comprehensive evidence metric includes a comprehensive prediction probability and overall uncertainty.
2. The classification system based on multimodal credible progressive fusion according to claim 1 is characterized in that: The multimodal data preprocessing module is specifically implemented through the following steps: (1.1) Collect multimodal data of the object to be tested, integrate and align the multimodal data of the same individual according to the classification category and individual test object to form a set of multimodal samples; (1.2) Preprocessing the image-type multimodal data to convert it into a format that can be processed by the feature extraction and screening module, specifically including: first marking the region of interest in the multimodal image, and saving the annotation result of the region of interest in the form of a binary image as an annotated image; then calculating the enclosed area of the rectangular frame according to the annotated image, and setting the edge extension value; then intercepting the rectangular frames of the original image and the annotated image; adjusting the pixel value of the original image and standardizing it to the [0,1] interval to complete the image size adjustment; finally obtaining the preprocessed image-type multimodal data; (1.3) Preprocessing the text-type multimodal data to convert it into a format that can be processed by the feature extraction and screening module, specifically including: first, cleaning the text data to remove HTML tags, special characters and punctuation marks; then using the byte pair encoding method to segment the cleaned text data to merge the character sequences or word fragments in the text data into a single symbol; finally, vectorizing the text data, specifically using the Word2Vec word embedding model to convert each word into a dense vector of a predefined size as the vector representation of the word, and averaging the vector representations of all words to generate a one-dimensional vector representing the entire sentence or text fragment as the vector representation of the sentence.
3. The classification system based on multimodal credible progressive fusion according to claim 2 is characterized in that: The expression of the vector representation of the sentence is: Where Vec(S) represents the vector representation of sentence S, and the subword set S contained in sentence S = {w1,w2,…,w k ,…,w K },w k represents the kth word in sentence S, K represents the total number of K subwords in sentence S; Vec(w k ) represents word w k The vector representation of is the output of the Word2Vec word embedding model; d is the dimension of the vector space.
4. The classification system based on multimodal credible progressive fusion according to claim 1 is characterized in that: The feature extraction and screening module is specifically implemented through the following steps: (2.1) extracting features from the preprocessed multimodal data, specifically including: for preprocessed image-type multimodal data, using convolutional neural networks or radiomics methods to extract visual features of image data; for preprocessed text-type multimodal data, using natural language processing tools to extract semantic features of text data; for preprocessed audio-type multimodal data, using acoustic models to extract acoustic features of audio data; (2.2) The features extracted in step (2.1) are screened by using variance analysis, Spearman correlation analysis, and least absolute shrinkage and selection operator methods. By evaluating the strength of association between the features and the classification task, the most predictive key features related to the classification task are screened out, and the key feature vector output by the feature extraction and screening module is obtained.
5. The classification system based on multimodal credible progressive fusion according to claim 4 is characterized in that: The radiomics method includes the following feature extraction algorithms: Histogram oriented gradients, scale-invariant feature transforms or accelerated robust features: used to detect and describe local features in images; Local binary pattern: used to describe texture features that focus on illumination changes; pyRadiomics library: used to extract texture features of regions of interest in medical images.
6. The classification system based on multimodal credible progressive fusion according to claim 1 is characterized in that: The uncertainty modeling module is specifically implemented through the following steps: (5.1) The deep feature representation of each single modality output by the neural network module or the multimodal fusion feature output by the feature fusion module is recorded as h v , v = 1, 2, ..., V, c, where V represents the total number of single modalities output by the neural network module, c represents the fusion modality, and h v Through the sigmoid activation function, the corresponding evidence vector is obtained, which is expressed as: in, Indicates h v is the evidence vector for category q, σ(·) represents the sigmoid activation function, h1 represents the deep feature representation of the first modality output by the neural network module, h2 represents the deep feature representation of the second modality output by the neural network module, and h V represents the deep feature representation of the Vth modality output by the neural network module, h c Represents the multimodal fusion features output by the feature fusion module; (5.2) The evidence vector obtained in step (5.1) is converted into Dirichlet distribution parameters by the following calculation formula: in, Represents the evidence vector The corresponding Dirichlet distribution parameters; (5.3) Based on the Dirichlet distribution parameters, the Dirichlet intensity is calculated by the following formula: Among them, L v is the Dirichlet intensity, q represents the category, and Q represents the total number of categories; (5.4) Calculate the predicted probability and uncertainty of each category according to the Dirichlet intensity: in, represents the predicted probability of mode v being category q, u v represents the uncertainty of the mode v.
7. The classification system based on multimodal credible progressive fusion according to claim 1 is characterized in that: The evidence fusion module is specifically implemented through the following steps: (6.1) For each mode v and category q, define the basic probability distribution m v ({q}), Indicates the degree to which the evidence provided by modality v supports category q. The basic probability distributions from each modality are combined using the combination rules of evidence theory to obtain the fused basic probability distribution; (6.2) Based on the fused basic probability distribution, repeat the fusion process in step (6.1) until the basic probability distribution of all modes of category q is fused, and the comprehensive prediction probability b(q) and overall uncertainty U of category q are obtained; (6.3) Repeat steps (6.1) to (6.2) until the comprehensive prediction probability and overall uncertainty of all categories are obtained, and the prediction category of the classification is determined based on this.
8. The classification system based on multimodal credible progressive fusion according to claim 7 is characterized in that: In step (6.1), it is assumed that there are two modes v1 and v2, and their basic probability distributions for category q are respectively and Then, the combination rule of evidence theory is used to fuse the basic probability distributions from each modality. The fused basic probability distribution is calculated by the following formula: Among them, X and Y are evidence sets under different modes, and γ is the conflict coefficient.
9. The classification system based on multimodal credible progressive fusion according to claim 8, characterized in that: In step (6.2), the comprehensive prediction probability b(q) is the sum of all basic probability distributions for category q, and the overall uncertainty U is expressed as the sum of basic probability distributions that do not belong to any specific category, and their expressions are: b(q)=Σ x:q∈X m(X) Where m(X) is the sum of the basic probability assignments of all supporting evidence sets X containing category q, is the sum of all the basic probability assignments that do not support any particular class.
