A multi-source uncertainty driven medical image classification method

Abstract

The present application relates to the field of computer multi-source medical image processing, and proposes a multi-source uncertainty driven medical image classification method, comprising the following steps: first, feature extraction is performed on different medical image views, and an evidence representation of each view is generated; second, an uncertainty-aware evidence dynamic fusion strategy is constructed to obtain a unified multi-view evidence representation after fusion; then, by introducing a multi-source uncertainty measure, it is ensured that the multi-view classification performance will not decrease with the increase of views during the fusion process; finally, the unified multi-view evidence representation after fusion is mapped to a Dirichlet distribution, and the disease prediction result of the case and the multi-source uncertainty degree of the prediction result are output; for the case with high multi-source uncertainty, it is handed over to the expert for further diagnosis and treatment. The present application successfully solves the defects that the traditional multi-view classification method based on deep learning cannot measure the conflict and unknown uncertainty between views, and the multi-view fusion performance decreases.

Description

Technical Field

[0001] This invention relates to the field of computer-based multi-source medical image processing. Background Technology

[0002] With the rapid development of computer science and medical imaging technology, multi-source medical image-assisted diagnosis is becoming increasingly popular in clinical applications. Currently, multi-view fusion technology has achieved significant results in the field of medical image classification. However, traditional deep learning-based multi-view classification methods face some limitations when dealing with multi-source uncertainties between views. One of the main challenges is the uncertainty caused by conflicting classification decisions due to feature differences between multiple views, and the unknown uncertainty caused by inconsistent distributions of test and training data across different views. These factors lead to a decrease in the reliability of disease diagnosis after model fusion, which in turn affects patient treatment processes and endangers patients' lives.

[0003] This problem becomes particularly acute in open-world, multi-source medical image-assisted diagnosis because different medical image data sources are not perfect and are often affected by factors such as image artifacts, missing medical image views due to variations in patient diagnosis frequency, and out-of-distribution medical image data related to rare diseases. Therefore, ensuring robust medical image diagnostic results is crucial for addressing the current imperfect open-world medical image data. This requires not only ensuring that diagnostic accuracy does not decrease due to anomalous views during image view fusion, but also providing an uncertainty metric for each diagnostic result; high uncertainty indicates that the current diagnostic result is not worth considering.

[0004] Recent studies have attempted to use uncertainty estimation methods to assess the reliability of model diagnostic results. However, due to the inability to effectively resolve conflicting uncertainties between decisions made from different perspectives, these methods have failed to ensure improved accuracy and reliability after fusing different perspectives. This may lead to increased uncertainty in the final decision, thereby reducing the reliability of medical-assisted diagnosis and posing a high risk to diagnostic results.

[0005] To address these challenges, we propose a more robust medical multiview image classification method that integrates multisource uncertainty measures. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies: they cannot measure conflicts and unknown uncertainties among multiple medical image views in an open world; and they cannot resolve the problem of imperfect multi-view image fusion, leading to a decline in final classification performance (i.e., increased empirical risk). By introducing a Dirichlet distribution prior, this invention proposes a robust deep multi-view medical image classification method for open worlds that integrates multi-source uncertainty. The auxiliary diagnostic method mainly consists of three steps: generating evidence representations for different image views, dynamically fusing uncertainty-aware evidence from multiple image views, and predicting multi-view medical image categories and a unified multi-source uncertainty measurement.

[0007] Technical solution of the present invention:

[0008] A medical image classification method driven by multi-source uncertainty, comprising the following steps:

[0009] Step 1: Dataset and Model Preparation

[0010] 1.1 Multi-view images and corresponding category labels were collected from hospitals and publicly available online datasets. Through preprocessing, data cleaning, and organization, a multi-view classification image dataset was established, including training and testing data; among which, multi-view medical image data... Corresponding tags The dataset is denoted as D = {X, Y}, where N is the number of samples and M is the number of views;

[0011] 1.2 Obtain a pre-trained neural network trained on a natural image dataset for image feature extraction.

[0012] 1.3 Construct a multi-view medical image classification neural network, including: a pre-trained neural network, an evidence generation network, and an uncertainty-aware evidence dynamic fusion module; wherein, each view corresponds to one pre-trained neural network and one evidence generation network, and the multi-view images are processed by the corresponding pre-trained neural network and evidence generation network, and finally integrated in the uncertainty-aware evidence dynamic fusion module.

[0013] Step 2: Model Training and Optimization

[0014] 2.1 Use the pre-trained neural network corresponding to each view to extract features from the multi-view training image data obtained in step 1.1.

[0015] 2.2 The different view features extracted in step 2.1 are used to generate initial evidence representations for the corresponding views through an evidence generation network.

[0016] 2.3 The multiple initial view evidences obtained in step 2.2 are fused through the uncertainty-aware evidence dynamic fusion module to obtain a unified multi-view evidence representation.

[0017] 2.4 Map the unified multi-view evidence representation obtained in step 2.3 to the Dirichlet distribution.

[0018] 2.5 Derive the class prediction, unknown uncertainty, and conflict uncertainty of the training samples from the Dirichlet distribution obtained in step 2.4.

[0019] 2.6 Based on the category labels in the multi-view image dataset obtained in step 1.1, and combined with the category prediction and multi-source uncertainty measure obtained in step 2.5, calculate the evidence fusion classification loss and multi-source uncertainty regularization loss as evaluation indicators.

[0020] 2.7 Combining the evidence fusion classification loss and multi-source uncertainty regularization loss obtained in 2.6, end-to-end training is performed on the pre-trained neural network and the evidence generation network.

[0021] Step 3: Category prediction and unified multi-source uncertainty measurement:

[0022] 3.1 Input the multi-view image test data into the trained multi-view medical image classification neural network to obtain multi-view evidence representation.

[0023] 3.2 Mapping the multi-view evidence obtained in step 3.1 to the corresponding Dirichlet distribution.

[0024] 3.3 From the Dirichlet distribution obtained in step 3.2, the category prediction, unknown uncertainty and conflict uncertainty of the test sample are derived, and a unified multi-source uncertainty measure is formed.

[0025] 3.4 Compare the magnitude of the preset threshold with the unified multi-source uncertainty metric in step 3.3 to determine whether the category prediction in step 3.3 is reliable.

[0026] 3.4.1 If the unified multi-source uncertainty metric is greater than the preset threshold, the classification prediction is determined to be abnormal, and expert doctors should be requested to intervene in the diagnosis and treatment.

[0027] 3.4.2 If the unified multi-source uncertainty metric is less than or equal to the preset threshold, the classification prediction is determined to be normal, and routine treatment is carried out according to the predicted category.

[0028] The beneficial effects of this invention are:

[0029] The goal of this invention is to reduce the impact of imperfect multi-source medical images in open worlds on the performance of deep multi-view classification. Through this invention, we aim to introduce more advanced and reliable technical solutions to the field of multi-view medical image processing, reducing the risk of misdiagnosis caused by imperfect multi-source medical image data in open environments, while improving the ability to detect unknown diseases. This invention aims to provide doctors with safer and more reliable diagnostic information, thereby improving the accuracy, reliability, and interpretability of medical diagnoses.

[0030] This invention is easy to implement and successfully addresses the shortcomings of traditional deep learning-based multi-view classification methods, which cannot measure inter-view conflicts and unknown uncertainties, leading to a decline in multi-view fusion performance. Simultaneously, this invention significantly improves the detection capability of unknown diseases in open environments, further enhancing the robustness of intelligent medical auxiliary diagnosis in clinical applications. Through this invention, the risk of misdiagnosis caused by imperfect multi-view data sources in open environments is effectively reduced in intelligent medical auxiliary diagnosis, without increasing the computational complexity of the original multi-view deep classification network. Attached Figure Description

[0031] Figure 1 This is the overall flowchart of the method of the present invention.

[0032] Figure 2 This is a flowchart illustrating the training optimization process of this invention.

[0033] Figure 3 This is a flowchart illustrating the medical image category prediction and multi-source uncertainty measurement implemented in this invention.

[0034] Figure 4(a) is a schematic diagram of the network workflow of the present invention.

[0035] Figure 4(b) is a schematic diagram of the workflow of a traditional deep multi-view classification neural network.

[0036] Figure 5(a) is a schematic diagram of network training according to an embodiment of the present invention.

[0037] Figure 5(b) is a schematic diagram of requesting doctor / expert intervention for abnormal medical image view cases in an open world according to an embodiment of the present invention.

[0038] Figure 5(c) is a schematic diagram of routine treatment performed on a normal medical image view in an open world according to the predicted disease category in an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions provided in this application will be further described below with reference to specific embodiments and accompanying drawings. The advantages and features of this application will become clearer from the following description.

[0040] This invention models information extracted from multiple medical image views as evidence and introduces Dirichlet priors into category prediction, thereby allowing the model to capture multi-source uncertainties (i.e., inter-view conflict uncertainty and unknown uncertainty) arising from imperfect multi-view images in open worlds. By fusing multi-view information at the evidence level and constraining multi-source uncertainties during the fusion process, this invention aims to mitigate the impact of imperfect medical images on multi-view classification results in open worlds, preventing the performance of multi-view classification from declining with the increase in the number of views, thus reducing the risk of clinical diagnosis and improving the accuracy and reliability of medical auxiliary diagnosis. The technical solution of this invention is itself only a tool for diagnosis and decision support, not a technical solution directly used for treating and saving lives.

[0041] like Figure 1 As shown, the workflow of the open-world robust depth multi-view medical image classification method that integrates multi-source uncertainty is as follows:

[0042] Step 1, Dataset and Model Preparation, the specific implementation process is as follows:

[0043] 1.1 Collect multi-view medical image data from hospitals or public datasets and their corresponding tags Create a dataset, denoted as D = {X, Y}. Here, N is the number of samples, and M is the number of views.

[0044] 1.2 For each different medical image view, collect the corresponding pre-trained neural network trained on the natural image dataset.

[0045] 1.3 A multi-view medical image classification neural network was constructed, which consists of a pre-trained neural network, an evidence generation network, and an uncertainty-aware evidence dynamic fusion module. Each view corresponds to one pre-trained neural network and one evidence generation network, which are ultimately integrated in the fusion module.

[0046] Step 2, model training and optimization, the specific process is as follows: Figure 2 As shown, the specific implementation process is as follows:

[0047] 2.1 Take a sample of the i-th view from the dataset D. This sample Input the corresponding pre-trained neural network for feature extraction to obtain features.

[0048] 2.2 Features extracted from step 2.1 Input evidence to generate a neural network, and obtain the network output corresponding to the category. The ReLU activation function is used to make... Transform into nonnegative evidence The formula for calculating the ReLU(·) function is as follows:

[0049] ReLU(x) = max(0,x)

[0050] 2.3 The samples of the M views are processed through steps 2.1-2.2 to obtain the M pieces of evidence corresponding to each view. The input is fed into the uncertainty-aware evidence dynamic fusion module to obtain the fused multi-view evidence e. j :

[0051] Two view evidence and In the case of uncertainty perception, the dynamic fusion process of evidence as follows:

[0052]

[0053] in, For the j-th sample in view Figure 1 The uncertainty measure is K, where K is the number of classes to which the current sample belongs.

[0054] Similarly, in the case of M view evidences, the dynamic fusion process of evidence based on uncertainty perception is as follows:

[0055]

[0056] 2.4 The fused K-dimensional multi-view evidence e obtained in step 2.3 j =(e j1 ,...,e jK Mapped to parameter α j =(α j1 ,...,α jK Dirichlet distribution Dir(μ|α) j Its probability density can be expressed as:

[0057]

[0058] Where μ is a random variable of Dirichlet distribution, K is the number of classes in the classification task, and B(e j +1) is the polynomial Beta function.

[0059] 2.5 Dirichlet distribution Dir(μ|α) obtained from step 2.4 j Derive the category prediction value p in multi-source uncertainty j Conflict uncertainty U conflict (α j ) and unknown uncertainty U unknown (α j ).

[0060]

[0061]

[0062]

[0063] in It is the balance function.

[0064] 2.6 Combine the category labels y from the dataset in step 1 j Using the Dirichlet distribution parameters α obtained in step 2.4 j The multi-source uncertainty U obtained in step 2.5 conflict (α j ) and U unknown (α j ), calculate the category mean squared error loss L cls (α j ) and multi-source uncertainty regularized loss L unc (α j The category mean squared error loss (L) is used as an evaluation metric for training a multi-view medical image classification neural network. cls (α j The formula for calculating ) is:

[0065]

[0066] in Dir(μ|α) is the Dirichlet distribution. j The mean of ). Multi-source uncertainty regularized loss L unc (α j The formula for calculating ) is:

[0067]

[0068] Where λ1 and λ2 are hyperparameters.

[0069] 2.7 Combined Category Mean Square Error Loss L cls (α j ) and multi-source uncertainty regularized loss L unc (α j The gradient descent algorithm is used to train and optimize the pre-trained neural network and the evidence generation network.

[0070] Step 3 involves multi-view image category prediction and multi-source uncertainty measurement for the test data. The process is as follows: Figure 3 As shown, the specific implementation process is as follows:

[0071] 3.1 Input the test image into the multi-view medical image classification neural network trained in step 2 to obtain the Dirichlet distribution Dir(μ|α) learned from the training dataset. j The derived classification prediction p and the unknown uncertainty U unknown (α), Conflict uncertainty U conflict (α) and a unified multi-source uncertainty measure U.

[0072] U=λ1U conflict (α)+λ2U unknown (α)

[0073] Where α is the Dirichlet distribution parameter measured on any test dataset in the open world, α j The Dirichlet distribution parameters are the ones that the network learns from the training dataset in step 1.

[0074] 3.3 Given a threshold τ, make a decision by comparing the unified multi-source uncertainty metric U with the preset threshold τ.

[0075] 3.3.1 If U > τ, the multi-view medical imaging data of the case is determined to be abnormal, the classification result is unreliable, and expert doctors are requested to intervene in the diagnosis and treatment.

[0076] 3.3.2 If U≤τ, then the multi-view medical imaging data of the case is considered normal, and routine treatment should be carried out according to the final classification prediction p.

[0077] The comparison between the present invention and traditional deep learning-based multi-view medical image classification methods is shown in Figures 4(a) and 4(b):

[0078] Traditional deep learning-based multi-view medical image classification methods fuse information from multiple views at the feature level and then output class probabilities through a Softmax layer (as shown in Figure 4(b)). These methods cannot handle multi-source medical image classification problems in open worlds. In particular, when there are conflicts or unknown uncertainties between views, the performance of multi-view classification decreases with the increase in views (i.e., empirical risk increases with the increase in views), leading to a higher misdiagnosis rate and posing a significant risk to patients.

[0079] This invention models features extracted by deep neural networks as evidence and performs uncertainty-aware dynamic evidence fusion at the evidence level (as shown in Figure 4(a)). By representing features from different views as evidence and introducing Dirichlet priors into classification prediction, the model can measure multi-source uncertainty from between views. By constructing an effective uncertainty-aware dynamic evidence fusion strategy, this invention can avoid the problem in open worlds where multi-view classification performance decreases with the increase of the number of views due to conflicts and unknowns between views, avoid the interference of abnormal image views on prediction results, improve the robustness of prediction, and reduce the risk of assisted diagnosis.

[0080] Example

[0081] This embodiment takes an Alzheimer's disease detection task based on multi-view medical images as an example, using magnetic resonance imaging (MRI) and positron emission tomography (PET) as two types of multi-view medical images. As shown in Figure 5(a), during training, the collected MRI and PET images need to be input into the multi-view medical image classification neural network. This network consists of three parts: a pre-trained neural network, an evidence generation network, and an uncertainty-aware evidence dynamic fusion module.

[0082] First, PET and MRI views are input into pre-trained neural networks 1 and 2, respectively, for feature extraction. These features capture low- to mid-level semantic information in different image views, such as edge contours, grayscale textures, and shape sizes, providing strong support for Alzheimer's disease classification and detection. Next, the feature vectors are passed to an evidence generation neural network to generate evidence representations for each view. These evidence representations reflect high-level semantic information in different views, especially supporting the total amount of information for each category. Subsequently, evidence 1 and evidence 2 are fused by an uncertainty-aware evidence dynamic fusion module to obtain a unified multi-view evidence representation. By calculating the Dirichlet distribution corresponding to the multi-view evidence, we can derive the corresponding category probabilities and the unified multi-source uncertainty in the views. Combining data labels, we can calculate the category probability classification loss and the multi-source uncertainty regularization loss, and then update the network parameters using a gradient descent algorithm. This process is repeated multiple times until the network training is complete.

[0083] During the testing phase, as shown in Figures 5(b) and 5(c), MRI and PET images were input as multi-view image test samples into a trained multi-view medical image classification neural network. By calculating the Dirichlet distribution corresponding to the multi-view evidence, the probability of having Alzheimer's disease and a multi-source uncertainty measure were derived. In Figure 5(b), due to the noise in the PET view and the presence of a glioma image in the MRI view, there were conflicts and unknowns, resulting in the multi-source uncertainty of this case exceeding the preset threshold; therefore, medical intervention was recommended. In Figure 5(c), however, both the PET and MRI views in this case were normal Alzheimer's medical images, so conventional treatment could be directly implemented based on the classification results. This process, through the multi-source uncertainty measure, helps determine whether further diagnosis by a specialist is needed, improving the reliability of the classification results.

[0084] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.

Claims

1. A medical image classification method driven by multi-source uncertainty, characterized by comprising the following steps: Step 1: Dataset and Model Preparation 1.1 Multi-view images and corresponding category labels were collected from hospitals and publicly available online datasets. Through preprocessing, data cleaning, and organization, a multi-view classification image dataset was established, including training and testing data. Multi-view medical imaging data Corresponding tags The dataset is denoted as , For the sample size, Number of views; 1.2 Obtain a pre-trained neural network trained on a natural image dataset for image feature extraction; 1.3 Construct a multi-view medical image classification neural network, including: a pre-trained neural network, an evidence generation network, and an uncertainty-aware evidence dynamic fusion module; wherein, each view corresponds to one pre-trained neural network and one evidence generation network, and the multi-view images are processed by the corresponding pre-trained neural network and evidence generation network, and finally integrated in the uncertainty-aware evidence dynamic fusion module; Step 2: Model Training Optimization 2.1 Use the pre-trained neural network corresponding to each view to extract features from the multi-view training image data obtained in step 1.1; 2.2 The different view features extracted in step 2.1 are used to generate initial evidence representations for the corresponding views through an evidence generation network; 2.3 The multiple initial view evidences obtained in step 2.2 are fused through the uncertainty-aware evidence dynamic fusion module to obtain a unified multi-view evidence representation; 2.4 Map the unified multi-view evidence representation obtained in step 2.3 to the Dirichlet distribution; 2.5 Derive the class prediction, unknown uncertainty, and conflict uncertainty of the training samples from the Dirichlet distribution obtained in step 2.4; 2.6 Based on the category labels in the multi-view image dataset obtained in step 1.1, and combined with the category prediction and multi-source uncertainty measure obtained in step 2.5, calculate the evidence fusion classification loss and multi-source uncertainty regularization loss as evaluation indicators; 2.7 Combining the evidence fusion classification loss and multi-source uncertainty regularization loss obtained in 2.6, end-to-end training is performed on the pre-trained neural network and the evidence generation network; Step 3: Category prediction and unified multi-source uncertainty measurement: 3.1 Input the multi-view image test data into the trained multi-view medical image classification neural network to obtain multi-view evidence representation; 3.2 Mapping the multi-view evidence obtained in step 3.1 to the corresponding Dirichlet distribution; 3.3 From the Dirichlet distribution obtained in step 3.2, the category prediction, unknown uncertainty and conflict uncertainty of the test samples are derived, and a unified multi-source uncertainty measure is formed; 3.4 Compare the magnitude of the preset threshold with the unified multi-source uncertainty metric in step 3.3 to determine whether the category prediction in step 3.3 is reliable; 3.4.1 If the unified multi-source uncertainty metric is greater than the preset threshold, the classification prediction is determined to be abnormal, and expert doctors should be consulted for diagnosis and treatment. 3.4.2 If the unified multi-source uncertainty metric is less than or equal to the preset threshold, the classification prediction is determined to be normal, and routine treatment is carried out according to the predicted category.

2. The medical image classification method driven by multi-source uncertainty as described in claim 1, characterized in that step 2.2 specifically involves: extracting features from step 2.

1. Input evidence to generate a neural network, and obtain the network output corresponding to the category. ,pass Activation function Transform into nonnegative evidence ;in The formula for calculating the function is as follows: 。 3. The medical image classification method driven by multi-source uncertainty as described in claim 1, characterized in that, Step 2.3 involves the following steps: The sample of each view is processed through steps 2.1-2.2 to obtain the view corresponding to the sample. One piece of evidence The data is input into the uncertainty-aware evidence dynamic fusion module to obtain fused multi-view evidence. ,as follows: Two view evidence and In the case of uncertainty perception, the dynamic fusion process of evidence as follows: in, For the first The uncertainty measure of a sample on view 1 This represents the number of categories to which the current sample belongs. In the case of individual view evidence, the dynamic evidence fusion process for uncertainty perception is as follows: 。 4. The medical image classification method driven by multi-source uncertainty as described in claim 1, characterized in that, Step 2.4 specifically involves processing the fused material obtained in step 2.

3. Multiview Evidence Mapping to parameters Dirichlet distribution Its probability density can be expressed as: in, Let be a random variable with a Dirichlet distribution. The number of categories in the classification task. This is a polynomial Beta function.

5. The medical image classification method driven by multi-source uncertainty as described in claim 1, characterized in that, Step 2.5 specifically involves processing the Dirichlet distribution obtained in step 2.

4. Deriving Class Predictions in Multi-Source Uncertainty Conflict uncertainty and unknown uncertainty ; in , It is the balance function.

6. The medical image classification method driven by multi-source uncertainty as described in claim 1, characterized in that, Step 2.6 specifically involves combining the category labels from the dataset in Step 1. Using the Dirichlet distribution parameters obtained in step 2.4 Compared with the multi-source uncertainty obtained in step 2.5 and Calculate the category mean squared error loss With multi-source uncertainty regularization loss As an evaluation metric, it is used to train a multi-view medical image classification neural network; the category mean squared error loss. The formula for calculation is: in For Dirichlet distribution The mean; multi-source uncertainty regularization loss The calculation formula is: in, and This is a hyperparameter.

7. The medical image classification method driven by multi-source uncertainty as described in claim 1, characterized in that, Step 2.7 specifically involves: combining the category mean squared error loss. With multi-source uncertainty regularization loss The pre-trained neural network and evidence generation network are trained and optimized using the gradient descent algorithm.

8. The medical image classification method driven by multi-source uncertainty as described in claim 1, characterized in that, Step 3.1 specifically involves inputting the test image into the multi-view medical image classification neural network trained in Step 2 to obtain the Dirichlet distribution learned from the training dataset. Exported classification prediction and a unified multi-source uncertainty measure ;as follows: ， in, The Dirichlet distribution parameters are measured for any test dataset in an open world. The parameters of the Dirichlet distribution learned by the network from the training dataset in step 1. For the unknown and uncertain, This is due to the uncertainty of conflict.

Images