Image fusion classification method based on minimum spanning tree and evidence theory

Abstract

The application provides an image fusion classification method based on a minimum spanning tree and evidence theory, initial classification results of each modality image are regarded as nodes, conflict coefficients between the modality images are regarded as weights of edges connecting the nodes, a non-directional connected weighted graph is constructed, and a minimum spanning tree of the graph is generated by using a minimum spanning tree algorithm; two nodes connected on the minimum spanning tree are fused by using the evidence theory to obtain multiple new nodes, a new non-directional connected weighted graph is constructed, and the above process is repeated until all nodes are fused; and the classification of the image is completed by using the fusion result of all modes. The application is verified by experiments on two multi-modality medical image data sets, and the experimental results show that the synthesis method based on the minimum spanning tree and the evidence theory can obtain better classification performance compared with a traditional decision-level fusion algorithm and an original evidence theory fusion method.

Description

Technical Field

[0001] This invention belongs to the field of image fusion and image classification technology, specifically relating to an image fusion and classification method based on minimum spanning tree and evidence theory. Background technology:

[0002] In image classification tasks, single-modal images often provide limited information, insufficient to meet the large amount of information required for classification. Multimodal image fusion methods can effectively integrate complementary information between different modalities, thereby improving classification accuracy. Multimodal image fusion techniques can be mainly divided into three categories at the fusion level: pixel-level fusion, feature-level fusion, and decision-level fusion. Decision-level fusion has advantages over the other two methods, including strong anti-interference capability, high computational efficiency, and high flexibility. Among various classic decision-level fusion algorithms, evidence theory is superior to other algorithms. Specifically, compared to other decision-level fusion algorithms, evidence theory can accurately represent and quantify the uncertainties of each modality of the image, thus providing more accurate decision fusion results.

[0003] However, evidence theory synthesis algorithms typically fuse multimodal images sequentially according to their input order, lacking a specific analysis of the conflicts between the modalities and a more rational fusion order. If the initial two modalities to be fused have significant or even near-complete conflict, the fusion of these two modalities will yield poor or even counterintuitive results, and the final result after further fusion with the remaining modalities will likely deviate significantly from the true labels. Furthermore, this sequential fusion approach fails to minimize the overall conflict in multimodal fusion, thus failing to achieve optimal classification performance when using evidence theory. Therefore, a method is needed to improve the evidence theory synthesis method to better fuse information from various modalities, thereby improving image classification accuracy. Summary of the Invention

[0004] This invention proposes an image fusion classification method based on minimum spanning trees and evidence theory. Simply put, it improves the evidence theory synthesis method by utilizing the idea that a minimum spanning tree "contains all nodes in the graph and has the lowest sum of edge weights". The technical solution adopted by this invention is as follows:

[0005] An image fusion classification method based on minimum spanning tree and evidence theory is proposed. This method treats the initial classification results of each modality as nodes, and the conflict coefficient between each modality as the weight of the edges connecting the nodes. An undirected connected weighted graph is constructed, and its minimum spanning tree is generated. The pairs of nodes connected to the minimum spanning tree are then fused using evidence theory to complete the image classification. The specific steps of this method include:

[0006] (a) Data preprocessing: Image denoising and image segmentation are performed on images of each modality;

[0007] (II) Basic Feature Extraction: Extracting features from each modality of the image;

[0008] (III) Evidence Body Generation: For the features extracted from each modality image, two fully connected layers are used to condense the information contained in the features and reduce the dimensionality to the dimension required for the specific classification problem; at the same time, activation functions are used to obtain the specific information of each modality evidence body, that is, the confidence probability for each category, to complete the evidence body generation process for each modality.

[0009] (IV) Decision Fusion Based on Minimum Spanning Tree and Evidence Theory: First, the probabilistic information contained in the evidence bodies of each modality is regarded as nodes, and the conflict coefficient between any two evidence bodies is regarded as the weight of the edge connecting these two nodes, constructing an undirected connected weighted graph; then, the minimum spanning tree of this graph is obtained by generating the minimum spanning tree method. At this point, the graph includes all evidence body nodes and the sum of the weights between each node is minimized, that is, the overall conflict between each modality is minimized; then, the pair of nodes connected to each other on this minimum spanning tree is combined using the synthesis rules of evidence theory. Assuming that the two evidence body nodes are m1 and m2, the synthesis process can be represented as follows: A represents a subset of the two pieces of evidence {A i} and {B j The intersection of the two evidence sets, m(A) is the result of combining the two evidence sets. The conflict coefficient between the evidence bodies is used; each pair of nodes that has been combined forms a new node. After all the new nodes have been generated, the conflict coefficient between each pair of new nodes is recalculated and used as the edge weight of the node pair. A new undirected connected weighted graph is reconstructed, and the minimum spanning tree of the graph is obtained by generating the minimum spanning tree. The above process is repeated until the constructed new graph has only two nodes. At this point, the evidence theory synthesis rules are used to combine these two nodes to complete the entire decision fusion process.

[0010] (v) Backpropagation: Calculate the loss between the decision result after fusion in step (iv) and the true label using the loss function, and backpropagate the loss to train the model;

[0011] (vi) Classification: Output the final classification result using the trained model.

[0012] The beneficial effects of this invention include:

[0013] 1) By using the minimum spanning tree algorithm, the sum of the conflict coefficients between the modalities is minimized, that is, the overall conflict when the modalities are fused is minimized, so that the fusion is carried out in a more reasonable order, ensuring the best classification effect when using evidence theory;

[0014] 2) By using the minimum spanning tree algorithm, the connection relationship between the most conflicting modes is removed, thereby avoiding the situation where the initial fusion modes have a large or even close to complete conflict to the greatest extent, thus avoiding the final fusion result being extremely poor or even counterintuitive. Attached Figure Description

[0015] Figure 1 This invention proposes an image fusion classification method based on minimum spanning tree and evidence theory.

[0016] Figure 2 It is a decision fusion process based on minimum spanning tree and evidence theory. Detailed Implementation

[0017] The process of the image fusion classification method based on minimum spanning tree and evidence theory proposed in this invention is as follows: Figure 1 As shown, the process of this method will be explained in detail below through a specific example. Suppose that in a medical multimodal image classification problem, there are images from four modalities:

[0018] (i) Perform image registration on images of each modality, remove artifacts and noise from the images by filtering, and then segment the lesion areas in the images;

[0019] (ii) Extracting shape features and statistical features of each modality of images using medical imaging omics technology;

[0020] (III) Two fully connected layers are used to condense the information contained in the features of each modality, reducing the dimensionality to the level required for classification problems; simultaneously, the Softmax function is used to obtain the specific information of the evidence body of each modality, i.e., the confidence probability for each category of the task, thus completing the generation process of the evidence body for each modality; the evidence body of each modality is represented by m1, ...

[0021] m2, m3, and m4 represent the evidence, and the conflict coefficients between each pair of evidence are denoted by k. 12 k 13 k 14 k 23 k 24 and

[0022] k 34 Let k represent the value of k. 12 It is the conflict coefficient calculated from m1 and m2, that is...

[0023] The same applies to the rest;

[0024] (iv) Treating the four pieces of evidence m1, m2, m3, and m4 as four nodes, and the conflict coefficients between any two pieces of evidence as the weights of the edges connecting the two nodes, construct an undirected connected weighted graph G = (V, E, K), where E is the six edges connecting the four nodes of graph G, V = {m1, m2, m3, m4} represents the set of all the nodes, and K = ...

[0025] {k 12 ,k 13 ,k 14 ,k 23 ,k 24 ,k 34} represents the conflict coefficient between each pair of pieces of evidence, such as Figure 2 (a);

[0026] (V) Assume that the relationship between the six conflict coefficients in K is k 14 <k 13 <k 34 <k 12 <k 23 <k 24 The minimum spanning tree of graph G can be obtained using Kruskal's algorithm, a classic method for generating minimum spanning trees, as follows: Figure 2 (b);

[0027] (vi) Combine the interconnected node pairs of the spanning tree, namely {m1,m2}, {m1,m3}, and {m1,m4}, using evidence theory. The resulting new evidence bodies are denoted as m. 12 m 13 and m 14 , where m 12 That is, the result of combining the node pair {m1, m2}, i.e. The same applies to the others, such as... Figure 2 (c);

[0028] (vii) The three new pieces of evidence m 12 m 13 and m 14 Treating each pair of evidence as a node, and considering the conflict coefficient between them as the weight of the edge connecting the two nodes, we construct a new undirected connected weighted graph G1 = (V1, E1, K1), where...

[0029] V1={m 12 ,m 13 ,m 14}, E1={(m 12 ,m 13 ),(m 12 ,m 14 ),(m13 ,m 14 Let K1 = {k} be the three edges connecting the three nodes of graph G1. 12_13 ,k 12_14 ,k 13_14};

[0030] (viii) Assume that the relationship between the three conflict coefficients in K1 is k 12_13 <k 12_14 <k 13_14 The minimum spanning tree of graph K1 is generated using Kruskal's algorithm, such as... Figure 2 (d);

[0031] (ix) Connect the nodes of the spanning tree to each other, i.e., {m 12 ,m 13}、{m 12 ,m 14 Using evidence theory, the resulting new evidence is denoted as m. 12_13 and m 12_14 , where m 12_13 That is, {m 12 ,m 13 This node represents the combined result, m 12_14 Similarly, such as Figure 2 (e);

[0032] (x) The two new pieces of evidence m 12_13 and m 12_14 The evidence theory is used to combine the results to obtain the final decision fusion result. The loss function is used to calculate the loss between the fused decision result and the true label. The loss is backpropagated and the Adam optimizer is used to train the model. The trained model is then used to output the final classification result.

[0033] The effects of this invention can be further illustrated by experimental results:

[0034] The datasets used in the experiments were the training sets of Brats2017 and Brats2021, which were used to classify two molecular phenotypes of glioma using multimodal medical images. All experiments employed five-fold cross-validation to train and predict the final results. Specifically, the dataset was randomly divided into five equal parts. One part was used as the test set to evaluate the model's classification performance, while the remaining four parts were used as the training set to train the model to fit the data distribution. The classification performance of each fusion algorithm was evaluated using five metrics: accuracy (ACC), area under the receiver operating characteristic (AUC), precision, recall, and F1 score. Table 1 shows the classification performance of each algorithm on the Brats2017 dataset, and Table 2 shows the classification performance of each algorithm on the Brats2021 dataset.

[0035] Table 1 Comparison of experimental results on the Brats2017 dataset

[0036]

[0037] As can be seen from Table 1, compared with various traditional decision-level fusion algorithms and original evidence theory synthesis methods, the synthesis method based on minimum spanning tree and evidence theory proposed in this invention achieves better results in all other metrics except for a slight disadvantage in AUC. This basically proves the superiority of this method in the Brats2017 dataset classification task.

[0038] Table 2 Comparison of experimental results for the Brats2021 dataset

[0039]

[0040] As can be seen from Table 2, compared with various traditional decision-level fusion algorithms and original evidence theory synthesis methods, the synthesis method based on minimum spanning tree algorithm and evidence theory proposed in this invention achieves better results in all other metrics except for a slight disadvantage in accuracy. This basically proves the superiority of this method in the Brats2021 dataset classification task.

Claims

1. An image fusion classification method based on minimum spanning tree and evidence theory. This method treats the initial classification results of each modality as nodes, the conflict coefficient between each modality as the weight of the edges connecting the nodes, constructs an undirected connected weighted graph and generates the minimum spanning tree of the graph, and fuses the pairs of nodes connected on the minimum spanning tree using evidence theory to complete the image classification. The specific steps of this method include: (a) Data preprocessing: Image denoising and image segmentation are performed on images of each modality; (II) Basic Feature Extraction: Extracting features from each modality of the image; (III) Evidence Body Generation: For the features extracted from each modality image, two fully connected layers are used to condense the information contained in the features and reduce the dimensionality to the dimension required for the specific classification problem; at the same time, activation functions are used to obtain the specific information of each modality evidence body, that is, the confidence probability for each category, to complete the evidence body generation process for each modality. (IV) Decision Fusion Based on Minimum Spanning Tree and Evidence Theory: First, the probabilistic information contained in the evidence bodies of each pair of modalities is regarded as nodes, and the conflict coefficient between the evidence bodies is regarded as the weight of the edge connecting the two nodes, constructing an undirected connected weighted graph; then, the minimum spanning tree of this graph is obtained by generating the minimum spanning tree method. At this time, the graph includes all evidence body nodes and the sum of the weights between the nodes is minimized, that is, the overall conflict between the modalities is minimized; then, the pair of nodes connected to each other on this minimum spanning tree is combined by the synthesis rules of evidence theory. Let the two evidence body nodes be respectively and The synthesis process is represented as , Describing a subset of two pieces of evidence and The intersection between them This is the result of combining the two pieces of evidence. The conflict coefficient between the evidence bodies is used; each pair of nodes that has been combined forms a new node. After all the new nodes have been generated, the conflict coefficient between each pair of new nodes is recalculated and used as the edge weight of the node pair. A new undirected connected weighted graph is reconstructed, and the minimum spanning tree of the graph is obtained by generating the minimum spanning tree. The above process is repeated until the constructed new graph has only two nodes. At this point, the evidence theory synthesis rules are used to combine these two nodes to complete the entire decision fusion process. (V) Backpropagation: Calculate the loss between the decision result after fusion in step (IV) and the true label using the loss function, and backpropagate the loss to train the model; (vi) Classification: Output the final classification result using the trained model.

Images