A distribution outlier detection method based on k nearest neighbor graph

Abstract

The application relates to the technical field of computer vision and deep learning, and discloses a distribution-out detection method based on a K nearest neighbor graph, which comprises the following steps: firstly, using a feature extractor in a neural network to perform feature extraction on each sample x in a data set, the extracted features being features of a penultimate layer of the neural network; using the features of a training data set of in-distribution data to initialize the K nearest neighbor graph, using each node z in the graph to represent the features of each sample, i being an id number of the sample in the in-distribution data, and letting each feature node randomly point to another K different feature nodes. i The distribution-out detection method based on the K nearest neighbor graph accelerates the speed of searching for the nearest neighbor node by constructing the K nearest neighbor graph; when the feature distance is calculated, the weight of the global representation of the features is increased on the basis of the cosine similarity, the features with better global representation of the in-distribution are fully utilized, and the performance of the distribution-out detection is greatly improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to an out-of-distribution detection method based on K-nearest neighbor graphs. Background Technology

[0002] In recent years, with the development of technology, deep neural networks have demonstrated superior performance in fields such as computer vision and text processing. In closed systems where training and testing data are sampled from the same distribution, neural networks can exhibit excellent performance. However, when deploying neural networks in real-world applications, they often receive a significant amount of data outside the categories found in the training dataset, leading to unexpected outputs. Recent research indicates that even for completely unrecognizable or irrelevant inputs, neural networks tend to make high-confidence predictions. In fields like autonomous driving and medicine, if neural networks provide incorrect outputs with high confidence for never-before-seen inputs, there are significant risks. Ideally, neural networks should proactively recognize inputs they cannot classify or process, assigning significant uncertainty to these out-of-distribution examples and delegating the task to human intervention.

[0003] When a classifier encounters a new type of input, namely out-of-distribution examples, it is crucial that it recognizes uncertainty. Therefore, the ability to accurately detect out-of-distribution examples is an important part of visual recognition tasks.

[0004] The existing technical solutions mainly include:

[0005] 1. Distance-based methods: These methods utilize feature embeddings extracted from the model to calculate the distance between the test data and the training data, thereby enabling out-of-distribution detection.

[0006] 2. Density-based methods: Detecting out-of-distribution data by modeling data density.

[0007] 3. Classification-based methods: out-of-distribution detection is performed by relying on the classification scores given by the classifier.

[0008] However, the aforementioned existing technical solutions have the following drawbacks: Distance-based methods all make a strong distributional assumption that the underlying feature space follows a conditional Gaussian distribution. However, this assumption is not necessarily accurate, as the distribution of the feature space is often high-dimensional, and assuming a Gaussian distribution may limit out-of-distribution detection performance. Density-based methods require modeling the distribution of the dataset within the distribution; however, since datasets within the distribution are diverse and their distribution is high-dimensional, accurately modeling the data distribution requires significant computational resources and time. Classification-based methods rely heavily on the performance of the classification model for out-of-distribution detection, resulting in unstable performance and necessitating improvements. Summary of the Invention

[0009] (a) Technical problems to be solved

[0010] To address the shortcomings of existing technologies, this invention provides an out-of-distribution detection method based on K-nearest neighbor graphs. This method is advantageous in handling massive amounts of diverse data, requiring no assumptions about the data distribution for out-of-distribution detection; leveraging the most globally representative features of the data within the distribution to better define decision boundaries; and saving computational resources and time. This achieves a faster, non-parametric, and efficient out-of-distribution detection method that can better define decision boundaries and more quickly find the most important features of the data within the distribution. It addresses the problem that distance-based methods often make a strong assumption that the underlying feature space is a conditionally Gaussian distribution. However, this assumption is not necessarily accurate because the feature space distribution is often high-dimensional, and assuming a Gaussian distribution may limit out-of-distribution detection performance. Density-based methods require modeling the distribution of the dataset within the distribution, but since datasets are diverse and their distributions are high-dimensional, modeling the data distribution effectively requires significant computational resources and time. Classification-based methods rely heavily on the performance of the classification model for out-of-distribution detection, resulting in unstable performance.

[0011] (II) Technical Solution

[0012] To address the challenges of handling massive amounts of diverse data, out-of-distribution detection needs to be performed without making assumptions about the data distribution. Furthermore, it requires leveraging the most globally representative features of the data within the distribution to better define decision boundaries, while conserving computational resources and time. Therefore, this invention aims to provide a faster, non-parametric, and efficient out-of-distribution detection method that can better define decision boundaries and more quickly identify the most important features of the data within the distribution.

[0013] An out-of-distribution detection method based on K-nearest neighbor graphs includes the following steps:

[0014] S1. First, use the feature extractor in the neural network to extract features for each sample x in the dataset. The extracted features are the features of the second to last layer of the neural network.

[0015] S2. Initialize the K-nearest neighbor graph using features from the training dataset of the in-distribution data, and use each node z in the graph... i Let i represent the feature of each sample, where i is the ID number of the sample in the distributed dataset. Let each feature node randomly point to another K different feature nodes. At the same time, calculate the distance between the features of two nodes that have a pointing relationship, and randomly initialize the K nearest neighbor graph. The distance uses cosine similarity S. After initialization, each sample will point to K neighbor nodes.

[0016] S3. Optimize and reconstruct the initialized K-nearest neighbor graph, and randomly select a node z. i The features are used to calculate the cosine similarity with the features of K random nodes to search for nodes that are closer to the feature node. If a node is found that is closer to the feature node z, then the similarity is calculated. i For closer feature nodes, z i The more distant node that a node points to is updated to the more recent node;

[0017] S4. Then repeat step 3 multiple times. If no closer node is found after more than 5 attempts, the K nearest neighbor graph is considered to have been reconstructed.

[0018] Preferably, in step S1, the parameter x has the following formula:

[0019] z = h(x).

[0020] Preferably, in step S2, the mathematical expression for cosine similarity is as follows, z i and z j ′ represent the features of different samples:

[0021]

[0022] Preferably, in step S4, during the testing phase, the features of the test sample x are also treated as a new node. First, it points to K random nodes in the K nearest neighbor graph, and the global representative weighted feature distance L is calculated between each node and its features. Sample x repeatedly calculates the distance L with the K nearest neighbor nodes of its nearest neighbor node, and updates the operation of node x pointing to closer nodes, until finally finding the K nodes closest to node x. The formula for calculating distance L is as follows:

[0023] L(x,z i )=S(x,z i )*R(z i ).

[0024] Preferably, R is the global representativeness of a feature node, calculated from the in-degree of that node. The greater the global representativeness, the more representative the feature represented by that node is of the data within the distribution. Global representativeness is expressed by the following formula, where I i It is the in-degree of the node with id i in the K-nearest neighbor graph. It is all I greater than 0 i Average value:

[0025]

[0026] Preferably, in step S4, the average distance between the test sample x and the K nearest training data of sample x is used as the score for out-of-distribution detection. If the distance is greater than or equal to the threshold, sample x is identified as out-of-distribution data, and the node representing sample x and the K edges pointing to the K neighboring nodes are deleted from the graph. If the distance is less than the threshold, sample x is considered to be in-distribution data, and the node representing sample x and the edges pointing to the K neighboring nodes are retained. The distance of the edges is changed from distance L to cosine similarity.

[0027] Preferably, the threshold setting varies depending on the different training datasets. The threshold setting needs to ensure that 95% of the ID data are correctly classified. The expression for the out-of-distribution detection score is as follows, where the K nearest distances to the test sample are {L1, L2, L3, ..., L...}. k}:

[0028]

[0029] (III) Beneficial Effects

[0030] Compared with existing technologies, this invention provides an out-of-distribution detection method based on K-nearest neighbor graphs, which has the following advantages:

[0031] 1. This out-of-distribution detection method based on K-nearest neighbor graph accelerates the search for nearest neighbor nodes by constructing a K-nearest neighbor graph.

[0032] 2. This out-of-distribution detection method based on K-nearest neighbor graphs adds a weight of global representativeness of features to the cosine similarity when calculating feature distance, making full use of features that are more globally representative within the distribution, thus greatly improving the performance of out-of-distribution detection. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating the testing of the distributed detection system according to the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Please see Figure 1 The present invention provides the following technical solution:

[0036] S1. First, use the feature extractor in the neural network to extract features for each sample x in the dataset. The extracted features are those from the penultimate layer of the neural network.

[0037] z = h(x).

[0038] S2. Initialize the K-nearest neighbor graph using features from the training dataset within the distribution. Use each node z in the graph... i Let represent the features of each sample, and ... i and z j ′ represent the features of different samples:

[0039]

[0040] S3. Optimize and reconstruct the initialized K-nearest neighbor graph. Randomly select a node z. i The features of the feature node are used to calculate cosine similarity with the features of K random nodes in order to search for nodes that are closer to the feature node. If a node is found that is closer to the feature node z... i For closer feature nodes, z i The more distant node that a node points to is updated to the more recent node.

[0041] S4. Then repeat step 3 multiple times. If no closer node is found after more than 5 attempts, the K nearest neighbor graph is considered to have been reconstructed.

[0042] Furthermore, during the testing phase, the features of the test sample x are also treated as new nodes. First, they are pointed to K random nodes in the K nearest neighbor graph, and the globally representative weighted feature distance L is calculated between each node and its features. Sample x repeatedly calculates the distance L with the K nearest neighbor nodes of its nearest neighbor node, updating the operation of pointing to even closer nodes. This continues until the K nearest nodes to node x are finally found. The formula for calculating distance L is as follows:

[0043] L(x,z i )=S(x,z i )*R(z i ).

[0044] R is the global representativeness of a feature node, calculated from its in-degree. A higher global representativeness indicates that the feature represented by that node is more representative of the data within the distribution. Global representativeness is expressed by the following formula, where I... i It is the in-degree of the node with id i in the K-nearest neighbor graph. It is all I greater than 0 i Average value:

[0045]

[0046] Furthermore, out-of-distribution detection is performed using the mean of the distances between the test sample x and its K nearest training data points as the out-of-distribution detection score. If the distance is greater than or equal to a threshold, sample x is identified as out-of-distribution data, and the node representing sample x and its K edges pointing to its K neighboring nodes are deleted from the graph. If the distance is less than the threshold, sample x is considered in-distribution data, and the node representing sample x and its edges pointing to its K neighboring nodes are retained, with the edge distances changed from distance L to cosine similarity. The threshold setting varies depending on the different training datasets. The threshold setting needs to ensure that 95% of the ID data are correctly classified. The expression for the out-of-distribution detection score is as follows, where the K nearest distances to the test sample are {L1, L2, L3, ..., L...}. k}:

[0047]

[0048] This out-of-distribution detection method based on K-nearest neighbor graph accelerates the search for nearest neighbor nodes by constructing a K-nearest neighbor graph. When calculating feature distance, it increases the weight of the global representativeness of features on the basis of cosine similarity, making full use of features that are more globally representative within the distribution, thus greatly improving the performance of out-of-distribution detection.

[0049] Meanwhile, many current out-of-distribution detection methods make strong assumptions about the underlying feature space being a conditionally Gaussian distribution. Some of these methods consume significant computational resources and time, and their performance largely depends on the classifier's performance. This invention aims to propose a faster and more stable non-parametric density estimation out-of-distribution detection method. By using the K-nearest neighbor (KNN) out-of-distribution detection approach, the strong assumption about the underlying feature space being a conditionally Gaussian distribution can be avoided. Furthermore, using a KNN graph can significantly accelerate computation. This invention only requires calculating the feature distance with the training set within the distribution to quickly calculate the out-of-distribution detection score.

[0050] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for out-of-distribution detection based on K nearest neighbor graph, characterized in that, The method comprises the following steps: S1, first, feature extraction is performed on each sample x in the data set using a feature extractor in a neural network, and the extracted features are features of a penultimate layer of the neural network; S2, initialize the K-Nearest Neighbor graph using the features of the training dataset of in-distribution data, with each node z in the graph i S2, initialize the K-Nearest Neighbor graph using the features of the training dataset of in-distribution data, with each node z in the graph i S2, initialize the K-Nearest Neighbor graph using the features of the training dataset of in-distribution data, with each node z in the graph i S2, initialize the K-Nearest Neighbor graph using the features of the training dataset of in-distribution data, with each node z in the graph i S2, initialize the K-Nearest Neighbor graph using the features of the training dataset of in-distribution data, with each node z in the graph i S2, initialize the K-Nearest Neighbor graph using the features of the training dataset of in-distribution data, with each node z in the graph i S2, initialize the K-Nearest Neighbor graph using the features of the training dataset of in-distribution data, with each node z in the graph <000 S3, optimize and rebuild the K-Nearest Neighbor graph after initialization, randomly select a node z i characteristics, and calculate the cosine similarity with the characteristics of the random K nodes to search for nodes closer to itself, if a characteristic node closer to z i is found, update the nodes pointed to by z i to the closer nodes; S4, then, the operation of S3 is repeatedly performed multiple times, and if no closer node is found after more than 5 times, it is considered that the K-nearest neighbor graph has been reconstructed; In the test phase, the features of the test sample x are also taken as a new node, first, the new node is directed to K random nodes in the K-nearest neighbor graph, and the global representative weight feature distance L is calculated respectively, the sample x repeatedly calculates the L distance with the K neighbors of the nearest neighbor nodes, and the operation of updating the node x to point to closer nodes until the K nodes closest to the node x are finally found, and the calculation formula of the distance L is as follows: L(x, z i ) = S(x, z i ) * R(z i ); R is a global representation of a feature node, which is calculated by the in-degree of the node, the greater the global representation, the more the feature represented by the node can represent the data in the distribution, the global representation is represented by the following formula, wherein I i is the in-degree of the node with id i in the K nearest neighbor graph, is the average value of all I i greater than 0:

2. The out-of-distribution detection method based on K nearest neighbor graph according to claim 1, characterized in that, In the step S1, the parameter x has the following formula: z = h(x).

3. The out-of-distribution detection method based on K nearest neighbor graph according to claim 1, characterized in that, The mathematical expression of the cosine similarity is as follows in the step S2, i and z j represent the features of different samples, respectively:

4. The out-of-distribution detection method based on K nearest neighbor graph according to claim 1, characterized in that, In the step S4, the mean of the distances between the test sample x and the K nearest training data is used as the score of the out-of-distribution detection, the out-of-distribution detection is performed, if the distance is greater than or equal to a threshold value, the sample x is identified as out-of-distribution data, and the node representing the sample x and the K edges pointing to the K neighbor nodes are deleted from the graph; if the distance is less than the threshold value, the sample x is considered to be in-distribution data, and the node representing the sample x and the K edges pointing to the K nodes are retained, and the distance of the edge is modified from the distance L to the cosine similarity.

5. The out-of-distribution detection method based on K nearest neighbor graph according to claim 4, characterized in that, The setting of the threshold varies for different training data sets, and the setting of the threshold needs to ensure that 95% of the ID data is correctly classified, and the expression of the score of out-of-distribution detection is as follows, where the K distances closest to the test sample are {L1, L2, L3, …, L k} :

Images