Image OOD detection method research based on feature reconstruction

Through feature reconstruction and separation of semantic and non-semantic features, combined with the autoencoder, the problem of insufficient recognition ability of deep learning models on unknown distributed data is solved, the accuracy and robustness of OOD detection is improved, and the detection is simplified process.

CN118918402APending Publication Date: 2024-11-08HARBIN UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410979259.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-20
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing deep learning models have reduced their ability to identify and classify when facing unknown distributed data, and cannot effectively identify samples that do not match or corrupt the distribution of the training data, resulting in inaccurate predictions or errors.

Method used

Using the image OOD detection method based on feature reconstruction, by constructing a new data set and semantic and non-semantic features, feature information is extracted using CNN and ResNet-50, and the autoencoder is introduced to learn further feature information, combined with weighted reconstruction The error is OOD detection.

Benefits of technology

It significantly improves the recognition ability of the model on unseen distributed data, enhances the robustness and safety and reliability of the classification model, improves the accuracy and inference speed of OOD detection, and simplifies the detection process of unknown data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure QLYQS_1
    Figure QLYQS_1
  • Figure BDA0004955547830000021
    Figure BDA0004955547830000021
  • Figure BDA0004955547830000031
    Figure BDA0004955547830000031
Patent Text Reader

Abstract

The invention discloses an image OOD detection method research based on feature reconstruction, and belongs to the field of deep learning image application. Because a traditional deep learning task generally assumes that a training sample and a test sample have independent identical distribution attributes, that is, probability distribution of each variable in random variables is the same, the random variables are mutually independent, a training model is used for predicting a test set sample, and when the independent identical distribution of a training set and a test set is not established, the training model is used for predicting the test set sample. And the accuracy of the detection model is sharply reduced, and even high confidence is given to a wrong prediction result. In order to solve the problem, the research method comprises the steps of preprocessing an image type public data set, extracting semantic features and non-semantic features through feature separation, combining a special reconstruction method, solving the negative effect of covariable offset of the data set and solving the problem of a detection method, and compared with methods of classification, density, distance and the like, the research method has better performance improvement. According to the method, the problem that feature information cannot be fully utilized in an existing method is solved, and the performance of image OOD detection is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the application fields of computer vision and deep learning, and designs an image OOD detection method based on feature reconstruction, especially the application of image processing and deep learning technology in OOD detection. It is used to improve the recognition and classification capabilities of the model when facing unknown distribution data, provide more comprehensive and reliable security guarantees for the application fields of deep learning image classification, and improve the accuracy of image OOD detection and the security and reliability of classification models. Background Art

[0002] With the development of deep learning technology, image processing models based on convolutional neural networks (CNNs) have achieved remarkable results in multiple fields, including image recognition, object detection, and image classification. These models are usually trained on large-scale annotated datasets to learn the ability to extract effective features from input images. However, when the model encounters data or data distributions that have not been seen during training, its performance often drops sharply. This data is called OOD data and is generated due to a variety of reasons, such as different shooting conditions, new object categories, or image damage.

[0003] The purpose of OOD detection is to enable the model to identify these samples that do not match the distribution of training data or are damaged, which is crucial to improving the robustness of the model and its reliability in practical applications. However, most existing deep learning models or methods do not directly address the problem of OOD data identification, which may lead to inaccurate or even wrong predictions when encountering OOD samples.

[0004] This paper proposes an image OOD detection method based on feature reconstruction. Through the designed data processing and training process, the model's recognition ability when facing unseen distribution data is significantly improved. This method not only enhances the robustness of the model, but also provides a new idea for deep learning model recognition in complex environments. Summary of the invention

[0005] The present invention studies the image OOD detection method based on feature reconstruction. The purpose is to solve the negative impact of covariate shift and detection method problems in the current OOD detection. A method for image OOD detection based on feature reconstruction is proposed. A method of constructing a new dataset image version and separating features is adopted. CNN and ResNet-50 are used to separate semantic features and non-semantic features, and these two feature information are fully utilized to perform OOD detection of images.

[0006] This method helps the model better understand image details and information and improve the accuracy of OOD detection. At the same time, it introduces an autoencoder to learn further feature information and combines the OOD detection model with semantic features and non-semantic features to better ensure the good performance of the model.

[0007] In order to achieve the above goals, this study provides the following research plan: Research on image OOD detection method based on feature reconstruction, including the following steps:

[0008] Step 1: Collect image datasets, including digital datasets MNIST, USPS, notminst, and object datasets CIFAR-10, CIFAR-100, and CIFAR-10-C. MNIST and CIFAR-10 are used as training sets for the task, and the remaining datasets are used as prediction test datasets. Some features of the training sets are transformed to create new image versions, thereby enriching the out-of-distribution datasets used for prediction.

[0009] Step 3: Build an autoencoder, reconstruct the feature information extracted in step 2, obtain the reconstructed semantic feature x' and non-semantic feature y', calculate the reconstruction error respectively, weight the two reconstruction errors, and train the autoencoder. The training objective function is defined as follows:

[0010]

[0011] Where L represents the weighted reconstruction error, α and β represent weighting coefficients and α+β=1, which is used to balance the weights of semantic feature and non-semantic feature errors. By minimizing the weighted reconstruction error, the autoencoder is trained to effectively reconstruct semantic features and non-semantic features.

[0012] Step 4: Use the test data set to extract semantic features and non-semantic feature information through two sub-networks of feature extraction, calculate the weighted reconstruction error through the trained autoencoder, and determine whether it is greater than the preset threshold to perform OOD detection of the image and generate a prediction result.

[0013] Step 5: Based on the prediction results, use AUROC to evaluate the performance of the model.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] (1) In the preprocessing stage, the present invention creates a new image version by transforming some features of the training set data, performing splicing and noise processing, simulating different features and lighting conditions, enhancing the detection capability in the simulated display scene, and reducing the possibility of misclassification.

[0016] (2) Starting from the perspective of features, the present invention is different from classification, density, distance and other methods. By separating and extracting features, the present invention targets semantic features and non-semantic features respectively, and makes full use of feature information at different levels. CNN focuses on extracting semantic features such as the shape and size of objects, while ResNet-50 extracts non-semantic features such as image style, thereby improving the expression ability of neural networks.

[0017] (3) By introducing a feature reconstruction method, the present invention is independent of a specific model structure compared to the OOD detection method that directly uses multi-dimensional features. It can be used in combination with various deep learning models and feature extraction methods, which simplifies the detection process of unknown data to a certain extent, effectively reduces the dimension and computational complexity of the features, and improves the reasoning speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of the image OOD detection method based on feature reconstruction of the present invention.

[0019] Figure 2 This is a schematic diagram of the pretreatment operation of step 1 of the present invention.

[0020] Figure 3 This is the structural diagram of the feature separation model in step 2 of the present invention.

[0021] Figure 4 This is a structural diagram of the network model of the present invention.

[0022] Figure 5 This is the ResNet-50 residual block structure diagram of the present invention.

[0023] Figure 6 It is a comparative experimental result diagram of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific implementation described herein is only used to explain the present invention and is not limited to the present invention.

[0025] like Figure 1 As shown, the present invention provides a method for image OOD detection based on feature reconstruction, which specifically includes the following steps:

[0026] Step 1: Prepare the data set, obtain the digital data set and the object data set, and perform certain preprocessing operations on the data set.

[0027] Specifically, step 1 includes the following:

[0028] Step 1-1 Obtain the digital data sets MNIST, USPS, notminst, and the object data sets CIFAR-10, CIFAR-100, and CIFAR-10-C. The following uses CIFAR-10 as an example.

[0029] Step 1-2 converts the training set of the CIFAR-10 dataset into a tensor, uses splicing and noise processing to generate processed data containing the same amount as the training set, randomly splices the images by row or column within the class, compares the brightness of each RGB color channel with the mean of the image, and performs noise processing on the image to generate data with the same amount as the original training set but with more changes, and converts the tensors processed by the two methods into images of the same size for storage.

[0030] Step 2, feature separation and extraction, such as Figure 2 shown.

[0031] Specifically, step 2 includes the following:

[0032] Step 2-1 builds CNN and ResNet-50 models respectively.

[0033] In step 2-2, the training set data is passed through the CNN and ResNet-50 models respectively. This dual-model strategy makes feature extraction more comprehensive and helps to more accurately identify OOD samples. CNN is used to learn low-level feature statistics to capture semantic features x, and the deep learning model ResNet-50 is used to capture non-semantic features y for semantic features to obtain effective feature information.

[0034] In step 2-3, the extracted features are standardized, and the standard deviation and mean are calculated respectively. The standardized semantic features and non-semantic features are used as inputs for subsequent tasks. This allows different features to vary within the same range, improving the comparability of features and the model's ability to interpret features. The standardization formula is shown below.

[0035]

[0036] where z s Represents the extracted semantic and non-semantic feature tensors, μ s represents the mean of the feature tensor, σ s Represents the standard deviation of the feature tensor. This method adjusts the feature data by subtracting the mean and dividing it by the standard deviation to ensure that each dimension contributes evenly to the result. This can not only improve the convergence speed of the model, but also enhance the model's sensitivity to abnormal data points, which is particularly critical for OOD detection.

[0037] Step 3: Train the autoencoder.

[0038] Specifically, step 3 includes the following:

[0039] Step 3-1 constructs an autoencoder, which consists of an encoder and a decoder. These two parts work together to achieve effective encoding and reconstruction of features.

[0040] Step 3-2 trains the autoencoder and inputs the standardized semantic features and non-semantic features into the autoencoder for training.

[0041] Step 3-3 reconstructs the semantic features and non-semantic features of the autoencoder to perform feature reconstruction errors, calculates the total weighted reconstruction error as the objective function, and optimizes the autoencoder. In particular, the concept of weighted reconstruction error is introduced. Different weights are assigned to the reconstruction errors of different features to reflect the importance of each feature in OOD recognition and enhance the model's sensitivity to changes in key features.

[0042] Step 4: predict the results.

[0043] Step 4-1: Load the CNN and ResNet-50 models trained in step 2 and load the prediction dataset.

[0044] In step 4-2, features are extracted. CNN is used to extract semantic features, and ResNet-50 is used to extract non-semantic features. The same normalization process is performed as in step 2 to obtain the standardized feature information.

[0045] Step 4-3 loads the autoencoder trained in step 3, reconstructs the standardized feature information, calculates the weighted reconstruction error, and defines the threshold of OOD detection according to the objective function training result of step 3.

[0046] Step 4-4 uses the weighted reconstruction error of the test set data to compare with the threshold, generates a prediction result, saves and outputs the detection result, and adopts dynamic threshold setting technology to automatically adjust the OOD judgment threshold according to the statistical information in the training data set. The threshold is automatically optimized according to different application scenarios and data distribution to avoid the limitations and deviations caused by artificially designed thresholds.

[0047] Step 5, based on the prediction results, AUROC is used to evaluate the performance of the model. In addition to using the AUROC evaluation method, the present invention intends to introduce new performance indicators, such as FPR@95 and mean square error, to provide a more comprehensive evaluation perspective for OOD detection, especially in the case of data imbalance, to more accurately reflect the performance of the model.

[0048] A pre-randomly selected test set containing data samples from a different and independent distribution than the training set is used to comprehensively evaluate the performance.

[0049] The model weights are not updated during the model evaluation phase, so gradient calculations need to be disabled, which is achieved by using the torch.no_grad() context manager.

[0050] When evaluating the model, the number of ID samples and OOD samples in OOD detection is extremely unbalanced. The impact of the number of samples needs to be considered in the evaluation indicators. Commonly used measurement indicators for OOD detection include TNR@95%TRP (true positive rate), FPR@95%TRP (false positive rate), and area under the receiver operating characteristic curve (Area Under the Receiver Operating Characteristic curve, AUROC). The following is a specific description of the above indicators.

[0051] TNR@95%TRP represents the true negative rate when the true positive rate TPR is as high as 95%.

[0052] FPR@95%TRP represents the false positive rate when the true positive rate TPR is as high as 95%.

[0053] The True Positive Rate (TPR) represents the ratio of the number of positive samples predicted by the model to the total number of true positive samples, where TP stands for true positive examples, FN stands for false negative examples, FP stands for false positive examples, and TN stands for true negative examples. The formula is as follows.

[0054]

[0055] The True Negative Rate (TPR) indicates the ratio of correctly predicted negative samples to all negative samples. The formula is as follows.

[0056]

[0057] The false positive rate (FPR) is the ratio of negative samples predicted as positive samples to the total negative samples, also known as the false detection rate or false alarm rate. The formula is as follows.

[0058]

[0059] AUROC is an important indicator for evaluating OOD detection models. It represents the area under the ROC (Receiver Operating Characteristic) curve with FPR as the horizontal coordinate and TPR as the vertical coordinate. AUROC is a value between 0 and 1. When the value is close to 1, it means that the model has good classification ability.

[0060] This method is implemented in the Pytorch framework. Figure 6The experimental results show that the method proposed in the present invention has achieved significant improvement in OOD detection methods. The comparison results of this method with several other OOD detection technologies are presented in detail. This method shows significant performance improvement on multiple recognized challenging datasets, especially when processing image data with complex backgrounds and noise, showing higher robustness and accuracy.

[0061] Among them, MSP represents the maximum Softmax probability method model, ODIN represents the temperature interference method model, EBO represents the energy method model, MDS represents the Mahalanobis distance score model, SEM represents the semantic-based energy method model, and Ours represents the image OOD detection method model based on feature reconstruction (this method). The following is a detailed description and analysis of the background and differences of OOD detection methods.

[0062] MSP (Maximum Softmax Probability): A method model based on the maximum Softmax probability. It is a simple and intuitive method that uses the maximum value of the Softmax output of the last layer of the neural network as an indicator of confidence. If the maximum Softmax probability is low, the sample is considered to be an OOD sample. The main advantage of this method is that it is simple to implement, but it may be sensitive to data with uneven confidence distribution.

[0063] ODIN (Out-of-DIstribution Detector for Neural networks): A method model based on temperature disturbance, which improves the model's ability to distinguish between in-distribution and OOD samples by introducing temperature parameters and adding small input disturbances before the Softmax layer. ODIN does not require retraining the model, but only needs to adjust the temperature parameters and input disturbances to achieve a balance between performance and flexibility.

[0064] EBO (Energy-Based Outlier Detection): A model based on energy methods that uses the energy score of the model output to evaluate whether a sample is OOD. Low energy values ​​usually correspond to samples that are within the distribution, while high energy values ​​indicate that the sample may be OOD. This method provides a more comprehensive metric for OOD detection by evaluating the activity level of the overall output layer.

[0065] MDS (Mahalanobis Distance Score): A model based on Mahalanobis distance score, which determines whether a sample is OOD by calculating the Mahalanobis distance between the sample features and the training data features. This method is particularly suitable for those occasions where the feature space has obvious distinction, and can effectively use the statistical properties of the data to enhance the detection effect.

[0066] SEM (Semantic Energy Model): A semantic-based energy method model that combines semantic information and energy scoring. This model takes into account the anomalies of OOD samples at the semantic level and improves the recognition rate of OOD by analyzing the relationship between the representation of the semantic level and the energy value. This method places special emphasis on understanding the content and context of the sample, making it more effective in dealing with complex or ambiguous OOD scenarios.

[0067] Ours (Our Method): The image OOD detection method model based on feature reconstruction, that is, this method, mainly uses the autoencoder to reconstruct the input features and calculate the reconstruction error. The core idea of ​​this method is that if a sample can be accurately reconstructed by the model, it is considered to belong to the training distribution; conversely, samples with large reconstruction errors are considered OOD. The advantage of this method is that it can simultaneously utilize the intrinsic characteristics of the sample and the reconstruction ability of the model, so as to more accurately identify and distinguish OOD samples.

[0068] Analysis and comparison

[0069] Each method has its own unique advantages and applicable scenarios. For example, MSP and ODIN are suitable for scenarios that require rapid deployment, while EBO and MDS focus more on in-depth analysis using statistical properties and energy distribution. SEM and our method attempt to improve the accuracy of OOD detection by understanding the deeper characteristics of the sample. The selection and application of these methods should be based on specific business needs and data characteristics.

[0070] The present invention may also have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of the present invention.

Claims

1. Research on image OOD detection method based on feature reconstruction, which features include the following steps: Step 1: Collect image datasets, including digital datasets MNIST, USPS, notminst, and object datasets CIFAR-10, CIFAR-100, and CIFAR-10-C. MNIST and CIFAR-10 are used as training sets for the task, and the remaining datasets are used as prediction test datasets. Some features of the training sets are transformed to create new image versions, thereby enriching the out-of-distribution datasets used for prediction. Step 2: Use the feature separation method on the preprocessed training set, and use two sub-networks to extract semantic features and non-semantic features respectively. That is, use CNN to learn low-level feature statistics to capture semantic features x, and use the deep learning model ResNet-50 to capture non-semantic features y. Remove redundant information through standardization to obtain effective feature information. Step 3: Build an autoencoder, reconstruct the feature information extracted in step 2, obtain the reconstructed semantic feature x' and non-semantic feature y', calculate the reconstruction error respectively, weight the two reconstruction errors, and train the autoencoder. The training objective function is defined as follows: Where L represents the weighted reconstruction error, α and β represent weighting coefficients and α+β=1, which is used to balance the weights of semantic feature and non-semantic feature errors. By minimizing the weighted reconstruction error, the autoencoder is trained to effectively reconstruct semantic features and non-semantic features. Step 4: Use the test data set to extract semantic features and non-semantic feature information through two sub-networks of feature extraction, calculate the weighted reconstruction error through the trained autoencoder, and determine whether it is greater than the preset threshold to perform OOD detection of the image and generate a prediction result. Step 5: Based on the prediction results, use AUROC to evaluate the performance of the model.

2. According to the research on the image OOD detection method based on feature reconstruction according to claim 1, it is characterized in that In step 1, the image is converted into a tensor, and two processing methods are used: splicing processing and noise processing. Perform stitching: create a new combined image for each category, stitch the halves of the two images together, and use random stitching by row or column; perform noise processing: process each image, calculate the average value of each RGB color channel separately, create three new images, set the part with brightness lower than the average value to 0, and set the part with brightness higher than the average value to 1. Through this random cropping, stitching and color adjustment, different lighting and special conditions are simulated, which enhances the detection capability in simulating actual scenes and reduces detection errors caused by environmental changes.

3. The method for detecting OOD in an image based on feature reconstruction according to claim 1 is characterized in that The step 2 includes using CNN and ResNet-50 for feature separation and extraction. Semantic features are extracted by leveraging low-level feature statistics learned in CNN. These low-level feature statistics have been proven to be effective in representing image styles that are essentially unrelated to semantics. The ResNet-50 deep learning model is used for non-semantic features, and its powerful feature extraction capabilities are used to capture details and low-level features in the image, rather than high-level semantic information. Only the first layer of ResNet-50 is used, which mainly captures low-level features of the image such as edges and textures, rather than high-level semantic information. The low-level network can better learn good low-level feature representations and flatten the extracted non-semantic features into vectors for subsequent detection tasks.