Out-of-distribution target detection method based on sample sampling

Abstract

The invention discloses an out-of-distribution target detection method based on sample sampling, and the method comprises the steps: S1, evaluating in-distribution data DID, so as to determine an out-of-distribution score threshold gamma for correctly recognizing a set percentage of in-distribution samples; s2, evaluating the auxiliary out-of-distribution data DAUX to identify samples with out-of-distribution scores lower than a threshold gamma, and selecting the samples as auxiliary out-of-distribution data # imgabs0 # in the training process; S3, performing ascending sorting on the data in the auxiliary data # imgabs1 # according to the out-of-distribution scores, a subset # imgabs4 # S4 is selected from the top of the # imgabs3 #, and the subset # imgabs5 # is adopted to carry out training with the goal of minimizing the empirical risk # imgabs6 #; s5, returning the trained f <. Theta). According to the method, the training data are selected from the out-of-distribution data set by using the out-of-distribution scores, and the model is adjusted and optimized only by using a small part of most effective data, so that the training cost can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and more specifically, to an out-of-distribution target detection method based on sample sampling. Background Art

[0002] When machine learning models are deployed in real-world environments, they often encounter inputs from out-of-distribution data that exhibit semantic shifts from the training data distribution. The distribution of the training data is often referred to as in-distribution, while a distribution that deviates significantly from the training data is referred to as out-of-distribution. Models must avoid making predictions on out-of-distribution data, especially in safety-critical domains such as autonomous driving and medical diagnostics. In these domains, making arbitrary predictions on out-of-distribution data can lead to catastrophic consequences. This concern has recently sparked extensive research on out-of-distribution detection within the field of machine learning reliability.

[0003] Out-of-distribution detection primarily involves two tasks: accurately predicting in-distribution data and effectively detecting out-of-distribution data. Existing methods can be broadly categorized into two types based on whether or not additional training data is introduced. The first type uses only in-distribution data to train the model for accurate classification and then utilizes post-processing methods based on the model output to distinguish between in-distribution and out-of-distribution data. These methods include: designing scoring functions to better capture the distinct patterns of in-distribution and out-of-distribution data; using different distance metrics to distinguish in-distribution and out-of-distribution data based on model-extracted features; and relying on gradient-based information for distinction. However, due to the lack of explicit out-of-distribution knowledge, these methods, despite their theoretical guarantees, often fail to achieve strong out-of-distribution detection performance. The second type of methods utilizes additional auxiliary data to enhance the pre-trained model's ability to detect out-of-distribution data. For example, these methods employ various out-of-distribution data generation methods to create diverse out-of-distribution datasets for model training; and expanding the coverage of out-of-distribution data through various methods to improve the model's ability to detect out-of-distribution data. These methods generally outperform the first type in out-of-distribution detection. However, these methods focus on obtaining more out-of-distribution data for training, but pay little attention to fully utilizing the potential of existing out-of-distribution data. Motivated by this gap, it is indeed necessary to propose a sample sampling-based out-of-distribution target detection method that effectively utilizes existing data to train the model. Summary of the Invention

[0004] The purpose of the present invention is to provide an out-of-distribution target detection method based on sample sampling to overcome the defects of the prior art.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A method for detecting out-of-distribution targets based on sample sampling, comprising the following steps:

[0007] S1. Evaluate data within the distribution D ID , to determine the out-of-distribution score threshold γ that correctly identifies a set percentage of in-distribution samples;

[0008] S2. Evaluate auxiliary out-of-distribution data D AUX , to identify samples with out-of-distribution scores lower than the threshold γ, and select these samples as auxiliary out-of-distribution data for the training process

[0009] S3. Auxiliary data according to out-of-distribution scores Sort the data in ascending order to get the set From the collection Select a subset of the top

[0010] S4. Using the Subset To minimize experience risk Train for a goal;

[0011] S5. Return the trained model f(·; θ), where θ is the model parameter.

[0012] Furthermore, the distributed data D in step S1 ID Defined as:

[0013]

[0014] Where, and Respectively represent the input space and label space corresponding to the data distribution within the distribution, for Distribution within distribution, Represents the data sample within the distribution, y 1 represents the data sample label within the distribution, n represents the number of data samples within the distribution, Indicates that the data distribution is sampled independently and identically from the distribution.

[0015] Furthermore, in step S1, the data D in the distribution is evaluated. ID The specific formula is:

[0016] P(s(x ID ; f)≥γ)=1

[0017] Where x ID is a sample within the distribution, f is a pre-trained classifier, s(x ID ; f) means x ID Maps to a value.

[0018] Furthermore, the auxiliary distribution data D in step S2AUX Defined as:

[0019]

[0020] Where, is an auxiliary distribution outside the distribution, represents the sample of data outside the auxiliary distribution, m represents the number of samples outside the auxiliary distribution, Indicates that the data distribution is independently and identically sampled from the auxiliary distribution.

[0021] Furthermore, in step S2, the auxiliary out-of-distribution data D is evaluated. AUX The formula is:

[0022]

[0023] Where, is an auxiliary out-of-distribution sample, is the output of the out-of-distribution scoring function, where s is the scoring function, which is based on the model f. The formula represents the filtering of the auxiliary out-of-distribution data set. , samples with out-of-distribution scores below a threshold γ.

[0024] Furthermore, the empirical risk in step S4 The formula for (f;θ) is:

[0025]

[0026] Where,

[0027]

[0028] Where, l ID Represents the multi-classification loss function, specifically the cross entropy loss function, The output of the model for the input sample x and the sample label y are used as the input of the loss function, where θ is the model parameter; l OOD is the loss function for distinguishing ID and OOD, The output given by the model for the input sample x is used as the input of the loss function.

[0029] Compared with the existing technology, the advantages of the present invention are: the present invention provides an out-of-distribution target detection method based on sample sampling, uses out-of-distribution scores to select training data from out-of-distribution datasets, and only uses a small part of the most effective data to tune the model, which can reduce training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 This is a flow chart of the out-of-distribution target detection method based on sample sampling of the present invention;

[0032] Figure 2 The present invention analyzes which data are more important for out-of-distribution detection training through experimental results, (a) the time evolution diagram of the indicators during the training process, (b) the comparative effect evaluation diagram of different training methods;

[0033] Figure 3 For the experimental results, ResNet-18 is used as the backbone network;

[0034] Figure 4 Experimental results of out-of-distribution scores, (a) results on the CIFAR-10 benchmark, (b) results on the CIFAR-100 benchmark;

[0035] Figure 5 The experimental results involve the selection of auxiliary out-of-distribution data for training and its subsequent testing using different scoring methods. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the protection scope of the present invention.

[0037] See Figure 1 As shown, this embodiment discloses a method for detecting out-of-distribution targets based on sample sampling, comprising the following steps:

[0038] Step S1: Evaluate the data within the distribution D ID , to determine the out-of-distribution score threshold γ that correctly identifies a set percentage (e.g., 95%) of in-distribution samples.

[0039] Step S2: Evaluate auxiliary out-of-distribution data D AUX , to identify samples with out-of-distribution scores lower than the threshold γ, and select these samples as auxiliary out-of-distribution data for the training process

[0040] Step S3: sort the auxiliary data according to the out-of-distribution score. Sort the data in ascending order to get the set From the collection Select a subset of the top

[0041] Step S4: Using the subset To minimize experience risk Train for a goal;

[0042] Step S5: Return the trained model f(·; θ), where θ is a model parameter. The out-of-distribution target detection method in this embodiment is to train such a model that can perform out-of-distribution detection.

[0043] The algorithm of the out-of-distribution target detection method based on sample sampling is shown in Table 1.

[0044] Table 1

[0045]

[0046] In this embodiment, consider the multi-class classification task, let and Respectively represent the input space and label space corresponding to the data distribution within the distribution. Above, definition For the distribution within the distribution above, For the distribution outside the distribution above, let for A random variable in space, that is, and is a random variable in the label space outside the unknown distribution. For the formula This means that the sample x ID and out-of-distribution samples x OOD With different label spaces, the data within the distribution D can then be ID Defined as:

[0047]

[0048] Represents the data sample within the distribution, y 1 represents the data sample label within the distribution, n represents the number of data samples within the distribution, Indicates that the data distribution is sampled independently and identically from the distribution.

[0049] In step S1, the data D in the distribution is evaluated. ID The specific formula is:

[0050] P(s(x ID ; f)≥γ)=1#(1)

[0051] Where x ID is a sample within the distribution, f is a pre-trained classifier, s(xID ; f) means x ID Maps to a value.

[0052] In this embodiment, the target of out-of-distribution detection can be defined as follows:

[0053] Definition 1 (OOD Detection). Out-of-distribution detection mainly accomplishes two tasks: (a) correctly classifying in-distribution samples and (b) enabling the model to detect out-of-distribution samples that may be encountered during the inference phase.

[0054] These two goals are achieved by designing different out-of-distribution scoring functions. The formal definition of the out-of-distribution scoring function is given below:

[0055] Definition 2 (OOD Scoring Function). Using the pre-trained classifier f, the out-of-distribution scoring function s(·; f) maps the input x to a value, i.e. In this way, the input data can be classified as in-distribution or out-of-distribution data by level set estimation, that is:

[0056]

[0057] Where h(x) is the binary decision between ID and OOD, and λ is a predefined parameter.

[0058] In this embodiment, the external scoring function is It is based on outlier exposure, given the training data D in the distribution ID and additional auxiliary out-of-distribution dataset D AUX , defined as follows:

[0059]

[0060] Where, For auxiliary distribution outside the distribution, this embodiment uses instead of Because the auxiliary distribution is distributed outside May not completely match the true distribution Outlier exposure aims to tune the model to enhance its ability to detect out-of-distribution samples. represents the sample of data outside the auxiliary distribution, m represents the number of samples outside the auxiliary distribution, Indicates that the data distribution is independently and identically sampled from the auxiliary distribution.

[0061] In this embodiment, the evaluation of the auxiliary out-of-distribution data D in step S2 is AUX The formula is:

[0062]

[0063] Where, is an auxiliary out-of-distribution sample, is the output of the out-of-distribution scoring function, where s is the scoring function, which is based on the model f. The formula means that the auxiliary out-of-distribution data set is screened out. , samples with out-of-distribution scores below a threshold γ.

[0064] In this embodiment, the empirical risk in step S4 The formula is:

[0065]

[0066] in This improves the model's ability to classify data within the distribution, which can be written as:

[0067]

[0068] and The model's ability to detect out-of-distribution data is enhanced and can be written as:

[0069]

[0070] Where, l ID Represents the multi-classification loss function, specifically the cross entropy loss function, The output of the model for the input sample x and the sample label y are used as the input of the loss function, where θ is the model parameter; l OOD It is a loss function that distinguishes ID and OOD. It is flexible in implementation. You can choose the binary cross entropy loss function or the cross entropy of the model output and the uniform distribution of 1 / c. The output given by the model for the input sample x is used as the input of the loss function.

[0071] This example uses CIFAR-100 as in-distribution data and the texture dataset as auxiliary out-of-distribution data to conduct experiments and systematically analyze which types of data are most effective for out-of-distribution detection training. Figure 2 The main results are summarized. Figure 2 In order to analyze which data is more important for out-of-distribution detection training through experimental results, D all Indicates that the complete out-of-distribution sample set is used for training, D c Indicates that only out-of-distribution samples that are correctly identified by the pre-trained model are used for training, D w Indicates that only out-of-distribution samples that are misclassified by the pre-trained model are used for training. (a) The time evolution of the indicator during the training process, and (b) the comparative effect evaluation diagram of different training methods.

[0072] Given that metric 1 is optimized with a low value, the experimental results reveal an interesting phenomenon: when training with more challenging out-of-distribution data, training performance begins to deteriorate after multiple iterations. Conversely, when using out-of-distribution data that the pre-trained model can correctly detect, the model converges faster than when training with all out-of-distribution data. The final results further show that training with the most difficult out-of-distribution data yields the worst performance, while training with only out-of-distribution data that the pre-trained model can correctly detect outperforms training with all auxiliary out-of-distribution data.

[0073] This embodiment can infer from these experimental data that, in the auxiliary out-of-distribution data, the samples that are more beneficial to model training are relatively simple out-of-distribution examples that the pre-trained classifier can accurately detect. This is likely because correctly detected out-of-distribution data has more unique features than in-distribution data, allowing the model to benefit from learning these patterns. In contrast, out-of-distribution data that the model fails to detect exhibits lower distinguishability from in-distribution data, potentially weakening the model's ability to learn to distinguish in-distribution and out-of-distribution data.

[0074] Therefore, it is sufficient to identify and exploit simpler out-of-distribution data that the pre-trained model can accurately distinguish.

[0075] This example shows the performance of the method in various out-of-distribution detection scenarios. The experimental results show that the method of the present invention has good efficiency. The experimental conditions are as follows:

[0076] Dataset

[0077] In-distribution datasets: CIFAR-10 and CIFAR-100

[0078] Out-of-distribution datasets: SVHN, Places365, LSUN-Resize, LSUN-Crop and 192Textures

[0079] measure

[0080] FPR95: The probability of misclassifying an out-of-distribution sample as an in-distribution sample when the true positive rate reaches 95%, which is represented by indicator 1 in this embodiment.

[0081] AUROC: area under the ROC curve, referred to as indicator 2 in this embodiment.

[0082] ID ACC: classification accuracy of data within the distribution, represented by indicator three in this embodiment.

[0083] Compare to baseline

[0084] ·Post-hoc methods: GODIN, MSP, ReAct, Energy, Mahalanobis, 199MaxLogit, ODIN, KNN, DICE, and ASH.

[0085] ·Fine-tuning methods:OE,CSI,SSD+,MixOE,and VOS.

[0086] backbone network

[0087] In the experiment, Wide-ResNet-40-2 is used as the backbone network, and ResNet-18 is used for ablation study.

[0088] Table 2 compares our method with state-of-the-art methods on the CIFAR-10 benchmark. ↑ (or ↓) indicates that the larger (or smaller) value is better, and the results are expressed as percentages.

[0089] Table 2

[0090]

[0091]

[0092] CIFAR benchmark test: The experimental results on the CIFAR benchmark are shown in Tables 2 and 3. The proposed method outperforms all other methods, thus demonstrating its reliability and superiority. Two key points are emphasized: First, the method proposed in this embodiment uses only a small amount of auxiliary out-of-distribution data, indicating that more training data does not necessarily lead to better performance. Using a limited but more useful and relevant subset of data can achieve performance comparable to or even exceeding that of using a larger dataset; second, while enhancing the model's out-of-distribution detection capabilities, the method of this embodiment also maintains a high classification accuracy on in-distribution data. This is because the auxiliary out-of-distribution data used in this embodiment has more distinguishable features than in-distribution data, and therefore does not interfere with the training of in-distribution tasks.

[0093] To evaluate the performance of this method in the more challenging near-OOD detection scenario, experiments were conducted using CIFAR-10 as in-distribution data and CIFAR-100 as out-of-distribution data. The experimental results are shown in Table 4. This method again demonstrates its superiority over other methods, verifying its robustness and reliability under extreme conditions. This is due to the same principle: the auxiliary out-of-distribution data selected by the proposed method exhibits more discriminative features compared to the in-distribution data, which enables the model to more effectively learn to distinguish between in-distribution and out-of-distribution data.

[0094] Table 3. Comparison of our method with state-of-the-art methods on the CIFAR-100 benchmark. ↑ (or ↓) indicates that the larger (or smaller) value is better, and the results are expressed as percentages.

[0095] Table 3

[0096]

[0097]

[0098] Table 4. Comparison of FIS with other state-of-the-art methods on challenging out-of-distribution settings. ↑ (or ↓) indicates that larger or smaller values ​​are better.

[0099] Table 4

[0100]

[0101] Backbone network, in order to evaluate the performance of the method of this embodiment on different models, an ablation study was conducted using ResNet-18 as the backbone network. The experimental results are shown in Table 4 and Figure 3 The results show that the proposed method maintains its advantage when ResNet-18 is used as the model architecture, thus confirming the robustness of the proposed method under different model configurations.

[0102] The main idea of ​​the method in this embodiment is to select important data that is more conducive to out-of-distribution detection training from the auxiliary out-of-distribution dataset. Although the previous experiments were based on the MLP score, the method in this embodiment is not limited to this specific out-of-distribution score. In order to further enrich the solution of this embodiment, this embodiment experiments with different out-of-distribution scores on two CIFAR benchmarks. The results are shown in the figure. Figure 4 As shown in Figure 3, on two in-distribution datasets, using MLP, MSP, and energy scores to filter out the more critical out-of-distribution detection data yielded better performance than using the entire dataset. It is worth noting that the method of this embodiment is independent of the scoring function; different scoring functions produce consistent results.

[0103] Furthermore, the scoring function used for data selection may be different from the scoring function used in the testing phase, e.g. Figure 5 The experimental results are shown in Figure 2. The MSP+MLP results, which use the MSP score for data selection and the MLP score for testing, not only outperform the results using all out-of-distribution data, but also outperform the results using the MLP score for both data selection and testing. This demonstrates that the data selection and training processes are distinct; the sole purpose of data selection is to identify data that is more beneficial for model training.

[0104] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner may make various changes or modifications within the scope of the appended claims. As long as they do not exceed the scope of protection described in the claims of the present invention, they should be within the scope of protection of the present invention.

Claims

1. A method for detecting out-of-distribution targets based on sample sampling, characterized in that: The following steps are involved: S1. Evaluate data within the distribution D ID , to determine the out-of-distribution score threshold γ that correctly identifies a set percentage of in-distribution samples; S2. Evaluate auxiliary out-of-distribution data D AUX , to identify samples with out-of-distribution scores lower than the threshold γ, and select these samples as auxiliary out-of-distribution data for the training process S3. Auxiliary data according to out-of-distribution scores Sort the data in ascending order to get the set From the collection Select a subset of the top S4. Using the Subset To minimize experience risk Train for a goal; S5. Return the trained model f(·; θ), where θ is the model parameter.

2. The out-of-distribution target detection method based on sample sampling according to claim 1, characterized in that: The distributed data D in step S1 ID Defined as: Where, and Respectively represent the input space and label space corresponding to the data distribution within the distribution, for Distribution within distribution, Represents the data sample within the distribution, y 1 represents the data sample label within the distribution, n represents the number of data samples within the distribution, Indicates that the data distribution is sampled independently and identically from the distribution.

3. The out-of-distribution target detection method based on sample sampling according to claim 1, characterized in that: In step S1, the data D in the distribution is evaluated. ID The specific formula is: P(s(x ID ;f)≥γ)=1 Where x ID is a sample within the distribution, f is a pre-trained classifier, s(x ID ; f) means x ID Maps to a value.

4. The out-of-distribution target detection method based on sample sampling according to claim 1, characterized in that: In step S2, the auxiliary distributed external data D AUX Defined as: Where, is an auxiliary distribution outside the distribution, represents the sample of data outside the auxiliary distribution, m represents the number of samples outside the auxiliary distribution, Indicates that the data distribution is independently and identically sampled from the auxiliary distribution.

5. The out-of-distribution target detection method based on sample sampling according to claim 1, characterized in that: In step S2, the auxiliary out-of-distribution data D is evaluated. AUX The formula is: Where, is an auxiliary out-of-distribution sample, is the output of the out-of-distribution scoring function, where s is the scoring function, which is based on the model f. The formula represents the filtering of the auxiliary out-of-distribution data set. , samples with out-of-distribution scores below a threshold γ.

6. The out-of-distribution target detection method based on sample sampling according to claim 1, characterized in that: The empirical risk in step S4 The formula is: Where, Where, l ID Represents the multi-classification loss function, specifically the cross entropy loss function, The output of the model for the input sample x and the sample label y are used as the input of the loss function, where θ is the model parameter; l OOD is the loss function for distinguishing ID and OOD, The output given by the model for the input sample x is used as the input of the loss function.

Images