Out-of-distribution detection method based on multi-scale diffusion

Abstract

The invention discloses an out-of-distribution detection method based on multi-scale diffusion, and belongs to the field of deep learning and computer vision. According to the method, firstly, a multi-scale random shielding strategy is adopted for an input image, and partial shielding processing is carried out on the image by designing a plurality of shielding blocks with different scales; and then, reconstructing the shaded image by using a diffusion model pre-trained on the in-distribution image. As the diffusion model can accurately reconstruct the in-distribution image, and the reconstruction precision of the shielding part of the out-distribution image is low, the reconstruction error difference between the in-distribution image and the out-distribution image is obviously increased. And finally, realizing detection and identification of the out-of-distribution sample through reconstruction error calculation. The method not only is excellent in detection of semantic offset samples, but also has excellent detection performance when processing samples with feature offset but without semantic offset. According to the method, the robustness and the accuracy of out-of-distribution detection can be remarkably improved, and the method has good application prospects and practical benefits.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and relates to an innovative method for out-of-distribution (OOD) detection, which is an out-of-distribution detection method based on multi-scale diffusion. Background Art

[0002] OOD detection is one of the important research topics in deep learning. Traditional deep learning tasks usually assume that training samples and test samples are independent and identically distributed. When the independent and identically distributed assumption of the training set and the test set does not hold, the accuracy of the model established by machine learning (such as the classifier) ​​drops sharply, and even gives high confidence to the wrong prediction results. With the development of deep learning, image recognition and natural language processing have been widely used, and the objects that need to be identified are increasing, and even unknown types are generated at any time. When facing an open environment, the deep learning classification model shows overconfidence in the samples to be detected of unknown categories, and predicts them as a certain category in the training set (that is, a known category). In scenarios with high security requirements, this wrong prediction will have serious consequences.

[0003] When machine models are deployed in the real world, a reliable classifier should not only accurately classify known in-distribution (ID) samples, but also identify OOD inputs from unknown categories, which can be achieved by using an OOD detector in combination with a classification model. At its core, OOD detection can be formulated as a binary classification problem. At test time, the goal is to determine whether the input at test time comes from ID (OOD). Depending on the data availability and model training methods, OOD detection can be divided into supervised OOD detection and unsupervised OOD detection. Supervised OOD detection uses labeled ID samples and OOD samples to transform OOD detection modeling into a supervised classification problem. When the information is most sufficient, it can be assumed that representative OOD samples can be accessed, and OOD auxiliary samples can be used to join the training set and participate in model training. Since real OOD samples are difficult to obtain, generating OOD to participate in training is an intuitive method.

[0004] In many real-world applications, OOD data can be highly diverse and unpredictable. Ideally, we would like to build an OOD detector that only requires unlabeled ID data during training. Unsupervised OOD detection trains the model using only ID samples and designs the evaluation method to make the difference between ID samples and OOD samples as large as possible without any additional auxiliary samples.

[0005] Among the unsupervised methods, the reconstruction-based unsupervised OOD detection method uses unlabeled ID data to train the generative model. Compared with ID samples, the reconstruction error of OOD samples is larger, and the reconstruction error can be used to detect OOD samples. Although the reconstruction-based method has shown good results in multiple scenarios, existing work mostly focuses on detecting semantically shifted samples, and samples that have undergone feature shift but no semantic shift are mostly excluded from the test set. In the real world, OOD detection must be able to detect not only new categories of images that have not been seen in the training set (semantic shift), but also images whose features have changed (feature shift). Therefore, they still face a key limitation: for samples that have undergone feature shift but no semantic shift, such as images that are only flipped or slightly deformed, the generative model has good generalization and reconstruction capabilities for abnormal images, thereby minimizing the reconstruction error, increasing the difficulty of detecting OOD through reconstruction error, and making OOD detection particularly challenging.

[0006] The present invention proposes an out-of-distribution detection method based on multi-scale diffusion, which performs multi-scale masking processing on the image before inputting the diffusion model for reconstruction, and reconstructs the image only through the unmasked part. Therefore, the possibility of accurately reconstructing the masked pixels of the OOD image through the neighborhood pixels is very low, and the diffusion model is trained on the ID image, so the input masked ID image can still be accurately reconstructed. In this way, the difference in reconstruction error between the ID image and the OOD image increases, thereby improving the accuracy of OOD detection. Summary of the invention

[0007] The purpose of the present invention is to provide an out-of-distribution detection method based on multi-scale diffusion to solve the problem of poor detection performance of OOD samples with feature offset in existing reconstruction-based unsupervised OOD detection methods.

[0008] In order to achieve the above object, the present invention provides the following scheme:

[0009] The present invention provides a multi-scale diffusion based out-of-distribution detection method, comprising the following steps:

[0010] S1: Obtain an input image to be detected, perform multi-scale random masking on it, and generate multiple groups of masked images. Each group of masked images has a different masking scale and ensures that each area of ​​the image has an equal masking probability;

[0011] S2: Use the diffusion model pre-trained on the ID dataset to reconstruct the masked image, and use the model's learning ability on the ID data to reconstruct the masked area;

[0012] S3: The reconstructed images of each scale are stitched together respectively, and only the reconstruction of the masked area is retained. An additional reconstructed image consisting of the area with the worst reconstruction effect is obtained at each scale, further increasing the distance between the original image and the reconstructed image of the OOD image;

[0013] S4: Calculate the learned perceptual image patch similarity (LPIPS) between all reconstructed images and the original images, derive the OOD score for each scale based on the difference, and combine the results of different scales to determine whether the input image is an OOD sample.

[0014] Furthermore, in S1, the input image is obtained and multi-scale random masking is performed as follows:

[0015] The input image is masked so that part of the image pixels will be completely removed, so that the model can only repair and reconstruct the image through the unmasked parts, thereby increasing the difficulty of image reconstruction.

[0016] For the specific masking method, several issues are considered. One is that masking blocks of multiple scales should be used to generate multiple reconstructions of a single image, because if the masking block size is too large (or the ratio of the masked area to the complete image is too high), the reconstruction distance of the ID image and the OOD image will be too large. On the contrary, if the masking block size is too small (or the ratio of the masked area to the complete image is too low), the reconstruction distance of the ID image and the OOD image will be too small, and the size of the salient feature area of ​​different images may be different; another issue is that the image area should have an equal probability of being masked, because the salient features of the OOD image may appear in any area of ​​the image. In order to meet these requirements, a multi-scale random mask is used.

[0017] The image is divided into a×a grids, and the black area represents the area to be removed. By changing the value of a, we can get k sets of masks of different scales. Each scale has n masks, and the proportion of black area in these n masks is the same, which is And the union of the black areas can just be spliced ​​into a completely black mask, which ensures that each image area has an equal probability of being removed. Then, a multi-scale random mask is applied to the image, so that nk different masked versions of an image are obtained, preparing for the next reconstruction step.

[0018] Furthermore, in S2, the diffusion model is used to reconstruct the masked image as follows:

[0019] A classic formulation of the diffusion model is called the Denoise Diffusion Probabilistic Model (DDPM). Specifically, starting from the data sample x0, Gaussian noise is injected into each step t = 1, 2, ..., T of the diffusion process. where β t follows a fixed variance table. The prior distribution is x t The denoising process of DDPM learning with ~N(0,1) is as follows: θ (x t-1 |x t )=N(x t-1 ; μ0(x t ,t),∑ θ (x t ,t)), where μ0(x t ,t) and ∑ θ (x t ,t) are learned through a neural network with θ as parameter.

[0020] Since the diffusion model can be reconstructed without retraining, the present invention uses the diffusion model to reconstruct the masked image. The diffusion model is pre-trained on a large amount of ID data, and it can understand and simulate the distribution of ID data. In this step, given the original image I and the binary mask M i,j (0 indicates the area to be deleted, otherwise 1), represents the jth mask (the n value determines how many masks there are at each scale) at the i-th scale (a×a grid, different a values ​​represent different scales), and the parameters θ of the diffusion model. The goal is to generate the inpainted image First, we sample the image from the noise distribution For initial input. Then, at each time step t (from T to 1), the original image I is forward diffused to generate a diffuse version I′ of the corresponding time step t-1: I′=diffuse(I;θ,t-1); at the same time, the current image Perform denoising to generate a denoised version corresponding to time step t-1: Next, through the mask M i,j Perform weighted combination of the diffusion image and the denoised image to update the repaired image in is the binary mask M i,j This process is repeated in each time step until t-1. Finally, the image This is the restored image.

[0021] This operation selectively uses the pixel values ​​of the diffusion image and the denoised image to update the inpainted image. This weighted combination process ensures that we only update the masked area at each time step, while keeping the pixels of the unmasked area unchanged, thereby gradually restoring the masked part of the image. Since the diffusion model is pre-trained on the ID data, it can accurately reconstruct the masked part of the ID image. In contrast, for OOD images, the diffusion model trained only on ID data has a low probability of accurately reconstructing the masked pixels of the OOD image through the neighborhood pixels. Therefore, the distance between the original image and the reconstructed image of the OOD image will be significantly larger than that of the ID image.

[0022] Furthermore, in S3, the splicing process of the reconstructed images of each scale is as follows:

[0023] After the image reconstruction is completed, n reconstructed images are obtained at each scale. In order to further increase the distance between the original image and the reconstructed image of the OOD image, we splice the reconstructed images of each scale separately, retaining only the reconstruction of the masked area (that is, each image only contributes to the area with the worst reconstruction effect), and obtain the spliced ​​image Right now Through the above operations, for each scale, in addition to the n reconstructed images output by the diffusion model repair and reconstruction, an additional reconstructed image consisting of the area with the worst reconstruction effect is also obtained.

[0024] Furthermore, in S4, the LPIPS between all reconstructed images and the original images is calculated as follows:

[0025] We adopt the LPIPS metric, a standard and robust metric that captures the perceptual differences between images. Since LPIPS assigns higher values ​​to more different images, we calculate the LPIPS between the original and reconstructed images and use it directly as the OOD score. The DDPM inpainting and reconstruction process inherently involves multiple sampling steps. Occasionally, due to randomness, DDPM may provide dissimilar reconstructions for ID data or similar reconstructions for OOD data. This may make the reconstruction distances of ID and OOD images difficult to separate, resulting in suboptimal OOD detection performance. To reduce randomness, we use the average of the LPIPS between the original image and n+1 reconstructed images as the final result d at the i-th scale. i : Where d represents the LPIPS value. Finally, the results of different scales are combined and the average results of i scales are calculated to obtain the final OOD score S: S = average (d i ).

[0026] The quality of reconstruction is evaluated by calculating the reconstructed distance between the spliced ​​image and the original image. A larger reconstruction distance means that the image is more likely to be regarded as OOD data by the model. In order to test the sensitivity of the model to masked blocks of different scales and ensure the stability of the experimental results, the scale of masking processing is further expanded. In addition to the original 8×8 scale, a 16×16 grid scale is introduced. For each scale, the previous masking, reconstruction, and splicing steps are repeated to generate two sets of reconstructed images. Finally, the average distance between all these images and the original image is calculated. This includes not only the two complete images after splicing, but also the 2n reconstructed images generated in the intermediate process. By calculating the average distance of all these images, the reconstruction ability of the model and the performance of OOD detection can be more accurately evaluated. This method reduces the contingency that may be caused by a single scale and a single mask, making the experimental results more stable and reliable.

[0027] Compared with the prior art, the present invention has achieved the following beneficial technical effects:

[0028] 1. The out-of-distribution detection method based on multi-scale diffusion provided by the present invention partially masks the image at multi-scales, and uses the diffusion model trained on the ID image to reconstruct the complete image using only the unmasked partial image. While ensuring that the ID image can still be accurately reconstructed, the possibility of accurate reconstruction of the OOD image is reduced, thereby increasing the difference in reconstruction errors between the ID and OOD images.

[0029] 2. The out-of-distribution detection method based on multi-scale diffusion provided by the present invention is a comprehensive OOD detection method. OOD detection not only detects images with semantic displacement, but also detects images with feature displacement. Compared with traditional reconstruction methods, this method shows higher accuracy in detecting images with feature displacement (such as flipped ID images).

[0030] 3. The out-of-distribution detection method based on multi-scale diffusion provided by the present invention combines the diffusion model with the multi-scale masking strategy, significantly enhances the detection effect of OOD samples with semantic offset and feature offset, and improves the accuracy and robustness of OOD detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flow chart of an out-of-distribution detection method based on multi-scale diffusion provided by an embodiment of the present invention;

[0032] Figure 2 A schematic diagram of multi-scale masking processing provided by an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of a reconstruction process based on a diffusion model provided by an embodiment of the present invention;

[0034] Figure 4A schematic diagram of stitching reconstructed images of each scale provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.

[0036] The core of the present invention is to provide an out-of-distribution detection method based on multi-scale diffusion, which can effectively detect OOD samples by enhancing the difficulty of image reconstruction and improving the reconstruction error difference between ID and OOD images.

[0037] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0038] Figure 1 A flow chart of a method for out-of-distribution detection based on multi-scale diffusion provided by an embodiment of the present invention. The method mainly includes the following steps:

[0039] S1: Get the input image and perform multi-scale random masking.

[0040] In this step, the input image is masked so that part of the image pixels will be completely removed, so that the model can only repair and reconstruct the image through the unmasked parts, thereby increasing the difficulty of image reconstruction.

[0041] For the specific masking method, several issues are considered. One is that masking blocks of multiple scales should be used to generate multiple reconstructions of a single image, because if the masking block size is too large (or the ratio of the masked area to the complete image is too high), the reconstruction distance of the ID image and the OOD image will be too large. On the contrary, if the masking block size is too small (or the ratio of the masked area to the complete image is too low), the reconstruction distance of the ID image and the OOD image will be too small, and the size of the salient feature area of ​​different images may be different; another issue is that the image area should have an equal probability of being masked, because the salient features of the OOD image may appear in any area of ​​the image. To meet these requirements, we use multi-scale random masks.

[0042] Specifically, Figure 2 As shown in , in this embodiment, the image is divided into a×a grids, and the black area represents the area to be removed. By changing the value of a, we can get k sets of masks of different scales. Each scale has n masks, and the proportion of black area in these n masks is the same, which is And the union of the black areas can just be spliced ​​into a completely black mask, which ensures that each image area has an equal probability of being removed. Then, a multi-scale random mask is applied to the image, so that nk different masked versions of an image are obtained, preparing for the next reconstruction step.

[0043] S2: Reconstruct the masked image using the diffusion model.

[0044] A classic formula of the diffusion model DDPM, specifically, starting from the data sample x0, each step of the diffusion process t = 1, 2, ..., T injects Gaussian noise where βt follows a fixed variance table. The prior distribution is x t The denoising process of DDPM learning with ~N(0,1) is as follows: θ (x t-1 |x t )=N(x t-1 ; μ0(x t ,t),∑ θ (x t ,t)), where μ0(x t ,t) and ∑ θ (x t ,t) are learned through a neural network with θ as parameter.

[0045] Specifically, Figure 3 As shown in FIG. 1 , since the diffusion model can be reconstructed without retraining, the present invention uses the diffusion model to reconstruct the masked image. The diffusion model is pre-trained on a large amount of ID data, and it can understand and simulate the distribution of ID data. In this step, given the original image I and the binary mask M i,j (0 indicates the area to be deleted, otherwise 1), represents the jth mask (the n value determines how many masks there are at each scale) at the i-th scale (a×a grid, different a values ​​represent different scales), and the parameters θ of the diffusion model. The goal is to generate the inpainted image First, we sample the image from the noise distribution For initial input. Then, at each time step t (from T to 1), the original image I is forward diffused to generate a diffuse version I′ of the corresponding time step t-1: I′=diffuse(I;θ,t-1); at the same time, the current image Perform denoising to generate a denoised version corresponding to time step t-1: Next, through the mask M i,j Perform weighted combination of the diffusion image and the denoised image to update the repaired image in is the binary mask M i,j This process is repeated in each time step until t-1. Finally, the image This is the restored image.

[0046] This operation selectively uses the pixel values ​​of the diffusion image and the denoised image to update the inpainted image. This weighted combination process ensures that we only update the masked area at each time step, while keeping the pixels of the unmasked area unchanged, thereby gradually restoring the masked part of the image. Since the diffusion model is pre-trained on the ID data, it can accurately reconstruct the masked part of the ID image. In contrast, for OOD images, the diffusion model trained only on ID data has a low probability of accurately reconstructing the masked pixels of the OOD image through the neighborhood pixels. Therefore, the distance between the original image and the reconstructed image of the OOD image will be significantly larger than that of the ID image.

[0047] S3: stitching the reconstructed images at each scale.

[0048] Specifically, Figure 4 As shown in Figure 2, after the image reconstruction is completed, n reconstructed images are obtained at each scale. In order to further increase the distance between the original image and the reconstructed image of the OOD image, we splice the reconstructed images of each scale separately, retaining only the reconstruction of the masked area (that is, each image only contributes to the area with the worst reconstruction effect), and obtain the spliced ​​image Right now Through the above operations, for each scale, in addition to the n reconstructed images output by the diffusion model repair and reconstruction, an additional reconstructed image consisting of the area with the worst reconstruction effect is also obtained.

[0049] S4: Calculate the LPIPS between all reconstructed images and the original images.

[0050] We adopt the LPIPS metric, a standard and robust metric that captures the perceptual differences between images. Since LPIPS assigns higher values ​​to more different images, we calculate the LPIPS between the original and reconstructed images and use it directly as the OOD score. The DDPM inpainting and reconstruction process inherently involves multiple sampling steps. Occasionally, due to randomness, DDPM may provide dissimilar reconstructions for ID data or similar reconstructions for OOD data. This may make the reconstruction distances of ID and OOD images difficult to separate, resulting in suboptimal OOD detection performance. To reduce randomness, we take the average of the LPIPS between the original image and the n+1 reconstructed images as the final result d of the i-th scale. i : Where d represents the LPIPS value. Finally, the results of different scales are combined and the average results of i scales are calculated to obtain the final OOD score S: S = average (d i ).

[0051] The quality of reconstruction is evaluated by calculating the reconstructed distance between the spliced ​​image and the original image. A larger reconstruction distance means that the image is more likely to be regarded as OOD data by the model. In order to test the sensitivity of the model to masked blocks of different scales and ensure the stability of the experimental results, the scale of masking processing is further expanded. In addition to the original 8×8 scale, a 16×16 grid scale is introduced. For each scale, the previous masking, reconstruction, and splicing steps are repeated to generate two sets of reconstructed images. Finally, the average distance between all these images and the original image is calculated. This includes not only the two complete images after splicing, but also the 2n reconstructed images generated in the intermediate process. By calculating the average distance of all these images, the reconstruction ability of the model and the performance of OOD detection can be more accurately evaluated. This method reduces the contingency that may be caused by a single scale and a single mask, making the experimental results more stable and reliable.

[0052] The above describes in detail the out-of-distribution detection method based on multi-scale diffusion provided by the present invention. In the specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other.

Claims

1. A multi-scale diffusion based out-of-distribution detection method, characterized in that: The following steps are involved: S1: Obtain an input image to be detected, perform multi-scale random masking on it, and generate multiple groups of masked images. Each group of masked images has a different masking scale and ensures that each area of ​​the image has an equal masking probability; S2: Use the diffusion model pre-trained on the In-Distribution (ID) dataset to reconstruct the masked image, and use the model's learning ability on the ID data to reconstruct the masked area; S3: stitching the reconstructed images at each scale separately, retaining only the reconstruction of the masked area, and obtaining an additional reconstructed image consisting of the area with the worst reconstruction effect at each scale, further widening the distance between the original image and the reconstructed image of the Out-of-Distribution (OOD) image; S4: Calculate the learned perceptual image patch similarity (LPIPS) between all reconstructed images and the original images, derive the OOD score for each scale based on the difference, and combine the results of different scales to determine whether the input image is an OOD sample.

2. The out-of-distribution detection method based on multi-scale diffusion according to claim 1, characterized in that: In step S1: Multi-scale masking processing is achieved by generating combinations of masking masks of different scales, dividing the image into a×a grids. By changing the value of a, multiple groups of masks of different scales can be obtained to adapt to the distinguishing feature areas of ID and OOD data sets of different sizes. The mask covers the image in grid units, and the union of each group of masks can just be spliced ​​into a mask covering a complete image, ensuring that each image area has an equal probability of being removed.

3. The out-of-distribution detection method based on multi-scale diffusion according to claim 1, characterized in that: In step S2: The diffusion model is a denoise diffusion probabilistic model (DDPM). The forward process generates a diffusion image by adding Gaussian noise at each time step t, and the reverse process gradually restores the image by removing noise. The diffusion model is pre-trained on a large amount of ID data. It can understand and simulate the distribution of ID data and reconstruct without retraining. The specific method of using the diffusion model to reconstruct the masked image is as follows: The pixel values ​​of the diffusion image and the denoised image are selectively used to update the repaired image. This weighted combination process ensures that only the masked area is updated at each time step, while keeping the pixels of the unmasked area unchanged, thereby gradually restoring the masked part of the image. Since the diffusion model is pre-trained on the ID data, it can accurately reconstruct the masked part of the ID image. On the contrary, for the OOD image, the diffusion model trained only on the ID data has a low probability of accurately reconstructing the masked pixels of the OOD image through neighborhood pixels. Therefore, the distance between the original image and the reconstructed image of the OOD image will be significantly greater than that of the ID image.

4. The out-of-distribution detection method based on multi-scale diffusion according to claim 1, characterized in that: In step S3: The n reconstructed images generated for each masking scale are stitched together. Only the reconstruction results of the masked areas in each reconstructed image are retained, and the areas with the worst reconstruction effects in each reconstructed image are combined to generate a stitched image. The stitched image further enlarges the reconstruction error between the ID image and the OOD image through the reconstruction error differences of each masked area. Through the above operations, for each scale, in addition to the n reconstructed images output by the diffusion model repair and reconstruction, an additional reconstructed image consisting of the areas with the worst reconstruction effect is also obtained.

5. The out-of-distribution detection method based on multi-scale diffusion according to claim 1, characterized in that: In step S4: LPIPS is used as a measure of perceptual difference to capture the visual difference between the original image and the reconstructed image, and the difference value is used as the basis of the OOD score; Calculate the LPIPS value between the original image and the n+1 reconstructed images at each scale, and take the average of these LPIPS values ​​as the final perceptual difference result d at the i-th scale i : Perceptual difference results for all different occlusion scales d i Take the weighted average S = average(d i ), and get the final OOD detection score S.

Images