A method for detecting out-of-distribution data in image classification of deep learning uncertainty based on super-opinion evidence.

Abstract

This method provides a super-opinion-based deep learning uncertainty image classification out-of-distribution data detection method that innovatively extends and improves the uncertainty method framework based on conventional evidence theory. It can extract common evidence in images that conventional deep learning evidence theory has ignored, enabling more comprehensive evidence extraction. By fully utilizing the rich similarity features contained in complex images, it significantly improves the ability to detect out-of-distribution data, enhances the reliability of machine learning aid applications, and greatly improves the recognition effect of models on out-of-distribution data. [Solution] The method includes step 1 of preparing a dataset, step 2 of building a super-opinion network and optimizing its training, and step 3 of performing image category prediction and out-of-distribution detection.

Description

[Technical Field]

[0001] The present invention relates to the field of computer image processing, and more specifically to a method for detecting out-of-distribution data in deep learning uncertainty based on super-opinion evidence in image classification. [Background technology]

[0002] Image classification is one of the core research directions in the field of computer vision, and its core task is to accurately classify a given image. This technique plays an important role in many practical applications. However, current image classification methods mainly rely on limited training data to learn feature representations of object categories, which limits performance when faced with categories not present in the training data, resulting in the so-called out-of-distribution data detection problem.

[0003] In real-world applications, this problem is particularly common, as the number of object categories is vast and highly variable, and collecting sufficient data for all possible categories is difficult. Therefore, how to improve the generalization ability of target detection models and enable them to effectively handle out-of-distribution data that deviates from the training distribution is a challenging and practical research topic. This problem has even greater research significance in high-risk fields such as medical image classification.

[0004] Some researchers have proposed out-of-distribution detection methods based on density models, preprocessing techniques, and outlier exposure strategies. However, the performance of these methods is susceptible to the quality of training data, and additional data may be required for correction. Furthermore, these methods face problems such as high computational complexity, difficulty in convergence, and weak mobility, which limit their robustness in practical applications. Therefore, although these methods are theoretically attractive, their effectiveness and reliability in practical applications need to be further validated and improved.

[0005] Deep learning-based uncertainty estimation methods, such as evidence deep learning and its derivative models, demonstrate significant advantages in terms of computational performance, efficiency, and scalability. However, existing evidence deep learning-based out-of-distribution detection methods can only construct evidence within a polynomial opinion framework and extract clear evidence for each category, failing to adequately consider ambiguous evidence shared across multiple categories. This limitation makes it difficult for these methods to address the challenges posed by high-similarity images in complex images. In complex images with high similarity, a large amount of shared ambiguous evidence is ignored, significantly reducing the reliability of models on in-distribution data.

[0006] In image category prediction and out-of-distribution detection, misclassification of in-distribution samples poses an extremely high risk, potentially leading to erroneous decision-making in high-risk applications (e.g., auxiliary diagnosis in the medical field). [Overview of the project] [Problems that the invention aims to solve]

[0007] In light of the limitations of existing technologies, this invention discloses a method for detecting out-of-distribution data in image classification with deep learning uncertainty based on super-opinion evidence, aiming to solve the following three main problems in the auxiliary application field: strong reliance on large amounts of training and additional data, overconfidence in the deep learning model regarding the auxiliary application results, and the inability of evidence-based deep learning to fully utilize the large amount of similar information in images. For this reason, we propose a method for detecting out-of-distribution data in image classification with deep learning uncertainty based on super-opinion evidence. [Means for solving the problem]

[0008] A method for detecting out-of-distribution data in deep learning uncertainty based on super-opinion evidence, Step 1 involves preparing a dataset where a large amount of image data and corresponding category labels are collected from publicly available online datasets, and then, after data cleansing and data preprocessing, an image classification dataset D is constructed with K sample categories, and the dataset samples (x,y)∈D are defined as x being the image data and y being the category label. The super opinion network includes a feature extractor, a super opinion evidence generation module, and an opinion projection module. Here, the feature extractor is obtained through pre-training on a large, general-purpose dataset. Image data is sent to the feature extractor to obtain feature vectors that reflect the grayscale, texture, and other feature information of the image. The feature vector is input to the super opinion evidence generation module, super opinion evidence values ​​corresponding to each composite set are generated, and corresponding super opinion beliefs are generated. Then, these super-opinion beliefs and corresponding collective information are sent to the opinion projection module, converted into polynomial opinion beliefs for each category after projection, and bijected to the Dirichlet distribution parameters corresponding to the polynomial opinions. Step 2 involves constructing a super-opinion network and optimizing its training, which is done by using the expected category probability value obtained using the Dirichlet distribution as the network prediction, calculating the evaluation metrics using the category label y of the dataset and the digamma evidence loss ψ(·), and training the entire super-opinion network until the network training reaches the expected stable state. The images to be predicted are sequentially input into the optimal super-opinion network trained in step 2, the Dirichley distribution parameters of the polynomial opinions are obtained, the category prediction p and uncertainty u of the test samples are obtained through the Dirichley distribution, an uncertainty threshold T is specified, and the uncertainty u and T are compared as follows: If u > T, the sample does not belong to any of the training sets and is determined to be out-of-distribution data, prompting experts to intervene for further decision-making. If u < T, it includes step 3 of performing image category prediction and out-of-distribution detection that outputs the category prediction p as the sample prediction result.

Advantages of the Invention

[0009] The present invention has the following beneficial effects compared with the prior art. The method for detecting out-of-distribution data of image classification of deep learning uncertainty based on the super opinion evidence proposed by the present invention innovatively expands and improves the uncertainty method framework based on the conventional evidence theory, can extract the common evidence of images ignored by the conventional deep learning evidence theory, thereby enabling more comprehensive evidence extraction, and by fully utilizing the rich similar features contained in complex images, significantly improving the detection ability of out-of-distribution data and enhancing the reliability of machine learning-assisted applications.

[0010] In the present invention, the proposed uncertainty measurement technology can effectively reduce the risk of misjudging out-of-distribution data as known categories due to incomplete training data in the machine learning support application process. By extracting evidence more comprehensively, this method significantly improves the discrimination effect of out-of-distribution data.

Brief Description of the Drawings

[0011] [Figure 1] is the overall flowchart of the present invention. [Figure 2] is the detailed flowchart of step 2 of the present invention. [Figure 3] is the detailed flowchart of step 3 of the present invention. [Figure 4] is the schematic diagram of the module structure and workflow of the present invention. [Figure 5] is the schematic diagram of the training flow on the true data in the medical skin disease classification inspection field of the embodiment of the present invention. [Figure 6] is the result diagram of predicting the classification result diagram on the true data in the medical skin disease classification inspection field of the embodiment of the present invention. [Figure 7] is a schematic diagram of the uncertainty measurement test out-of-distribution data on the true data in the medical dermatosis classification test field of the embodiment of the present invention. [Figure 8] is a schematic diagram comparing the present invention with traditional evidence deep learning methods in actual classification detection examples. [Figure 9] is a comparison diagram of the effects of the embodiment of the present invention and other methods.

Embodiments for Carrying out the Invention

[0012] To make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer, the present invention will be described in detail below in combination with the accompanying drawings and embodiments. It should be noted that the specific embodiments described in this specification are only used to explain the present invention and are not used to limit the present invention. Products that can achieve the same functions belong to equivalent substitutions and improvements within the protection scope of the present invention.

[0013] FIG. 1 is an overall flowchart for classifying medical images to determine whether they are out-of-distribution samples and measuring their uncertainty (Steps 1, 2, 3).

[0014] FIG. 2 is a detailed flowchart of Step 2 (implementation of the training optimization process) of the present invention.

[0015] FIG. 3 is an implementation flowchart of Step 3 (related to image category prediction and out-of-distribution detection) of the present invention.

[0016] As shown in FIG. 1, a deep learning uncertainty image classification out-of-distribution data detection method based on super opinion evidence, comprising: Step 1, preparing a data set; A large amount of image data and corresponding category labels are collected from publicly available online datasets. After data cleansing and preprocessing, an image classification dataset D is constructed with K sample categories, where (x,y)∈D is the image data and y is the category label.

[0017] Step 2: Build a super-opinion network and optimize its training. The aforementioned super-opinion network includes a feature extractor, a super-opinion evidence generation module, and an opinion projection module, where the feature extractor is prior art, obtained through pre-training on a large general-purpose dataset, and image data is sent to the feature extractor to obtain feature vectors that reflect the grayscale, texture, and other feature information of the image. These feature vectors are input into the super opinion evidence generation module, which generates super opinion evidence values ​​corresponding to each composite set, and generates corresponding super opinion beliefs. These super-opinion beliefs and corresponding collective information are then sent to the opinion projection module, where they are converted into polynomial opinion beliefs for each category after projection, and then bijected to the Dirichlet distribution parameters corresponding to the polynomial opinions.

[0018] The concepts of polynomial opinions and super-opinions originate from Subjective Logic theory. Traditional deep learning of evidence uses polynomial opinions to model evidence, but the modeled evidence is incomplete, leading to inaccurate uncertainty estimations. This invention uses super-opinions to model more comprehensive evidence, thereby obtaining more accurate uncertainty estimations and better out-of-distribution data detection effects.

[0019] Building a super-opinion network and optimizing its training also involves the following steps:

[0020] Step 2.1: Extract features from the image data obtained in Step 1 using a feature extractor. A sample (x,y)∈D is extracted from the dataset, and the image data x is input into a pre-trained feature extractor to perform feature extraction and obtain the feature f.

[0021] Step 2.2, input the feature f into the super opinion evidence generation module, and super evidence e H Obtained,

[0022]

number

[0023] The ReLU activation function guarantees the non-negativity of the evidence, which is given by the following equation.

[0024]

number

[0025] Step 2.3, the super opinion evidence generation module further models the super opinion evidence from Step 2.2 to obtain a super opinion, in which the super opinion is given a prior probability α H Super opinion belief b H It consists of three groups: and uncertainty u, and the modeling method is as follows:

[0026]

number

[0027]

number

[0028] S is calculated using the following formula:

[0029]

number

[0030] Here, K represents the number of sample categories, X represents the dataset category domain, and R(X) represents the superdomain corresponding to the dataset category domain (i.e., all non-empty subsets containing domain X). Step 2.4, the opinion projection module is the super-opinion belief b obtained in Step 2.3. H An opinion projection is performed on the result, and the parameter α of the Dirichley distribution Dir for the polynomial opinion is output.

[0031] The opinion projection module includes a projection coefficient calculation module and a polynomial opinion processing module, where the projection coefficient calculation module includes a single fully connected neural network for projecting super-opinions onto polynomial opinions (steps 2.4.1, 2.4.2), and the polynomial opinion processing module is used to biject the polynomial opinions to the corresponding Dilikray distribution parameters α (step 2.4.3).

[0032] in particular, Step 2.4.1, Super-opinion / belief b H The following is input to the projection coefficient calculation module, where W is the fully connected layer parameter of the projection coefficient calculation module and is the matrix W corresponding to the composite set category of each superopposition. H Calculate.

[0033]

number

[0034] Step 2.4.2, Based on the lack of prior knowledge of relative foundational probabilities, super opinion belief b H We distribute them evenly to obtain a polynomial opinion / belief b,

[0035]

number

[0036] Thereby obtaining polynomial opinions, the polynomial opinions are composed of a three - element group of prior probability α, super - opinion belief b H and uncertainty u, where α = 1.

[0037] Note that based on the projection from super - opinions to polynomial opinions in subjective logic, each super - opinion is distributed to the polynomial opinions of its corresponding category based on relative prior probabilities, and when there is no relative prior - probability prior knowledge, it is expressed as an even distribution.

[0038] Step 2.4.3, The polynomial - opinion processing module bijectively maps the polynomial opinions to the Dirichlet - distribution parameter α corresponding to the polynomial opinions according to the following formula,

[0039]

Number

[0040]

Number

[0041]

Number

[0042] Here, B(α) represents the K - dimensional polynomial beta function, K represents the number of sample categories, and P i represents the probability of the category corresponding to the i - th parameter derived from the Dirichlet distribution.

[0043] The expected value of the predicted - category probability obtained using the Dirichlet distribution is used as the network prediction, and the category label y of the dataset described in Step 1 and the digamma evidence loss ψ(·) are calculated as evaluation indicators, and the entire super - opinion network is trained. The calculation formula of the loss function is shown as follows,

[0044]

Number

[0045] Repeat the above operations until the network training reaches a predicted stable state.

[0046] Step 3: As shown in Figure 3, perform image category prediction and out-of-distribution detection. Step 3.1: Sequentially input the prediction target image into the optimal super opinion network trained in Step 2 to obtain the Dirichlet distribution parameters of the polynomial opinion.

[0047] Step 3.2: Through the Dirichlet distribution, obtain the category prediction p and uncertainty u of the test sample, specify an uncertainty threshold T, and compare the uncertainty u with T.

[0048]

Number

[0049]

Number

[0050] Step 3.3: Specify an uncertainty threshold T, where the value range of T is 0 to 1 and is manually selected based on the task and data. Compare the uncertainty u with T, as follows: If u > T, it is determined that the sample does not belong to any in the training set and is out-of-distribution data, prompting an expert to intervene for further decision-making. If u < T, output the category prediction p as the sample prediction result.

[0051] Figure 4 is an overall module frame diagram, with the upper half being the module body frame and the lower two parts being specific module details. The data from step 2.1 is processed through a feature extractor to obtain features, which are then modeled into super-evidence through step 2.2, super-opinions are generated through step 2.3, and polynomial opinions are projected onto the corresponding Dirichley distribution parameters through step 2.4. In step 2.5, predicted probabilities and uncertainties are obtained and optimized through the loss function.

[0052] Within this process, steps 2.2 and 2.3 are super-opinion evidence generation modules (where features are first activated as super-opinion evidence, and then super-opinion beliefs are generated based on the super-opinion evidence); step 2.4 is the opinion projection module (projecting the super-opinion beliefs as polynomial opinions and corresponding Diliklay distribution parameters); and step 2.5 is module optimization (calculating the loss, and then backpropagating to optimize the entire super-opinion network).

[0053] Training stage: As shown in Figure 5, the training process begins with inputting the collected image dataset into the super-opinion network. The super-opinion network consists of three core modules: a feature extractor, a super-opinion evidence generation module, and an opinion projection module.

[0054] First, image data is sent to a feature extractor to obtain feature vectors that reflect characteristic information such as image gradation and texture, providing strong support for subsequent classification diagnoses. These feature vectors are then input to a super opinion evidence generation module, which generates super opinion evidence values ​​corresponding to each composite set and generates corresponding super opinion beliefs. These super opinion beliefs and corresponding set information are then sent to an opinion projection module, which converts them into polynomial opinion beliefs for each category after projection. Finally, the true labels of the data are combined based on the Dirichley distribution corresponding to the polynomial opinions, a double gamma evidence loss function is calculated, and the network parameters are updated accordingly. The above operations are repeated until the network training reaches the expected stable state.

[0055] Testing phase: As shown in Figures 6 and 7, the test sample is input into the super-opinion network and outputs a corresponding super-opinion evidence representation. Next, the Dilikrei distribution corresponding to the polynomial opinion evidence after opinion projection is calculated, and the corresponding predicted classification result and uncertainty are derived. In Figure 6, the uncertainty of the test sample is below the set threshold, so the classification result is output directly. On the other hand, in Figure 7, the uncertainty of the test sample is above the set threshold, indicating that the model's classification result for this sample is uncertain, and recommending that a physician intervene for further diagnosis. This process ensures that the model can take appropriate action when faced with highly uncertain situations, thereby increasing the safety and reliability of the real-world application of the deep learning model.

[0056] Figure 8 is a schematic diagram comparing the present invention with traditional evidence-based deep learning methods in actual classification detection examples.

[0057] Figure 9 is a graph comparing the effectiveness of the actual classification method of the present invention with other methods for detecting out-of-distribution datasets. In the comparison of the effectiveness of the actual classification method with other methods for detecting out-of-distribution datasets, a lower value (↓) indicates better performance, and a higher value (↑) indicates better performance.

[0058] For comparison, there are classical methods such as MSP (Maximum Category Probability), ODIN (Out-of-Distribution Detector), and MCDropout (Monte Carlo Dropout), as well as some of the currently mainstream methods, such as openGAN, GradNorm, VIM, KNN, DICE, RankFeat, ASH, SHE, GEN, G-ODIN, CSI, MOS, VOS, LogitNorm, and the traditional evidence deep learning method EDL and its variant RED.

[0059] Three metrics were used as indicators for detecting out-of-distribution data, and each of them was used as follows: 1) FPR95: The value of the false negative rate (FPR) when the total recall rate (TPR) is equal to 95%. The lower the score, the better the representation. 2) AUROC (area under the Receiver Operating Characteristic (ROC) curve): The area under the ROC curve can be interpreted as the probability that a positive ID sample has a higher detection score than a negative OOD sample. A larger AUROC value indicates better performance. 3) AUPR (area under the Precision-Recall (PR) curve): The PR curve is plotted by showing the relationship between accuracy and recall rate. A larger AUPR value indicates better performance.

[0060] AUROC is the most common metric, and it is used as the primary metric for OOD detection performance to accurately measure the performance of ID samples. At the same time, in-distribution classification accuracy is used to measure the accuracy of the model's classification. The goal is to detect more OOD samples while maintaining ID classification performance. Compared with multiple methods, experimental results show that the present invention method guarantees classification accuracy while simultaneously improving the ability to detect out-of-distribution data and surpassing current mainstream methods.

[0061] The above description is intended solely to describe preferred embodiments of the present application and does not imply any limitation of the scope of the present application. Any modifications or alterations made by a general expert in the art based on the technical content disclosed above should be considered equivalent valid embodiments and shall fall within the scope of protection of the present invention.

Claims

1. A method for detecting out-of-distribution data in image classification of deep learning uncertainty based on hyper-opinion evidence, Step 1 involves preparing a dataset where a large amount of image data and corresponding category labels are collected from publicly available online datasets, and then, after data cleansing and data preprocessing, an image classification dataset D is constructed with K sample categories, and the dataset samples (x, y) ∈ D are defined as x being the image data and y being the category label. The hyper-opinion network includes a feature extractor, a hyper-opinion evidence generation module, and an opinion projection module. Here, the feature extractor is obtained through pre-training on a large, general-purpose dataset. Image data is sent to the feature extractor to obtain feature vectors that reflect the grayscale, texture, and other feature information of the image. The feature vector is input to the hyperopinion evidence generation module, and hyperopinion evidence values ​​corresponding to each composite set are generated, and corresponding hyperopinion beliefs are generated. Then, these hyper-opinion beliefs and corresponding collective information are sent to the opinion projection module, converted into polynomial opinion beliefs for each category after projection, and bijected to the Dirichlet distribution parameters corresponding to the polynomial opinions. Step 2 involves constructing a hyperopinion network and optimizing its training, which involves using the Dirichlet distribution to obtain the expected category probability value as the network prediction, calculating the evaluation metrics using the category label y of the dataset and the digamma evidence loss ψ(•), and training the entire hyperopinion network until the network training reaches the expected stable state. The target images are sequentially input into the optimal hyperopinion network trained in step 2 to obtain the Dirichley distribution parameters of the polynomial opinions. Through the Dirichley distribution, the category prediction p and uncertainty u of the test samples are obtained. An uncertainty threshold T is specified, and the uncertainty u and T are compared as follows: If u > T, the sample is determined to be out-of-distribution data, not belonging to any of the training sets, and prompts experts to intervene for further decision-making. Step 3 includes performing image category prediction and out-of-distribution detection, where if u < T, the sample prediction result is output using category prediction p. In Step 2, constructing a hyper-opinion network and optimizing its training is necessary. Step 2.1 involves extracting features from the image data obtained in Step 1 using a feature extractor, which involves taking a sample (x, y) ∈ D from the dataset, inputting the image data x into a pre-trained feature extractor to perform feature extraction and obtain the feature f. Step 2.2 involves inputting the aforementioned feature f into the hyper opinion evidence generation module to obtain hyper evidence e H, The hyperopinion evidence generation module further models the hyperopinion evidence from step 2.2 to obtain a hyperopinion. In this hyperopinion, the hyperopinion consists of a ternary group: prior probability αH, hyperopinion belief bH, and uncertainty u. The modeling method is as follows: [Number 14] [Number 15] S is calculated by the following formula: [Number 16] Here, K represents the number of sample categories, X represents the dataset category domain, and R(X) represents the hyperdomain corresponding to the dataset category domain. Step 2.3 The opinion projection module performs opinion projection on the hyperopinion belief b H obtained in step 2.3 and outputs it as a parameter α of the Dirichley distribution Dir of the polynomial opinion in step 2.

4. The opinion projection module includes a projection coefficient calculation module and a polynomial opinion processing module, where the projection coefficient calculation module includes a single fully connected neural network for projecting hyperopinions onto polynomial opinions, and the polynomial opinion processing module is used to biject the polynomial opinions to the corresponding Dirichley distribution parameter α. Specifically, step 2.4 is, Step 2.4.1 involves inputting the hyperopinion belief b H into the projection coefficient calculation module, where W is the fully connected layer parameter of the projection coefficient calculation module, and calculating the matrix W H corresponding to the composite set category of each hyperopinion, [Number 17] Based on the lack of relative foundational probability prior knowledge, the hyper-opinion belief b H is distributed averagely to obtain the polynomial opinion belief b. [Number 18] This yields a polynomial opinion, which consists of a ternary group of prior probability α, hyperopinion belief b H, and uncertainty u, where α = 1 in step 2.4.2, The polynomial opinion processing module bijects the polynomial opinion to the corresponding Dilikley distribution parameter α according to the following equation, [Number 19] [Number 20] [Math 21] Step 2.4.3 includes, where B(α) represents the K-dimensional polynomial beta function, K represents the number of sample categories, and Pi represents the probability of the category corresponding to the i-th parameter derived from the Dirichley distribution. The expected value of the predicted category probability obtained using the Dirichlet distribution is used as the network prediction, and the category label y of the dataset described in Step 1 and the digamma evidence loss ψ(•) are used as evaluation metrics for calculation, and the entire hyperopinion network is trained accordingly. The formula for calculating the loss function is shown below: [Number 22] A method for detecting out-of-distribution data in image classification of deep learning uncertainty based on hyper-opinion evidence, comprising step 2.5, which involves repeating the above operation until the network training reaches a expected stable state.

2. The aforementioned hyperevidence e Ｈ The formula for calculating this is as follows: [Number 23] The method for detecting out-of-distribution data in image classification of deep learning uncertainty based on hyper-opinion evidence according to claim 1, characterized in that the non-negativity of the evidence is guaranteed by the ReLU activation function, where the ReLU function is given by the following formula. [Number 24]

3. Performing image category prediction and out-of-distribution detection as described in Step 3 is, Step 3.1 involves inputting the image to be predicted into the optimal hyper-opinion network trained in step 2 to obtain the Dirichley distribution parameters of the polynomial opinion, Step 3.2 involves obtaining the category prediction p and uncertainty u of the test sample using the Dirichley distribution, [Number 25] [Number 26] As follows, If u > T, the sample is determined to be out-of-distribution data, not belonging to any of the training sets, and the expert is prompted to intervene in further decision-making in step 3.3.

1. 3.3.2 Specify an uncertainty threshold T, which includes step 3.3.2 outputting the sample prediction result using the category prediction p if u < T, where the range of T is 0 to 1, manually selected based on the task and data, and step 3.3 comparing the uncertainty u with T. The method for detecting out-of-distribution data in image classification of deep learning uncertainty based on hyper-opinion evidence according to claim 1, characterized by including the above.

Images