Noise-driven small sample distribution external detection method and system

Abstract

The invention discloses a noise-driven small sample distribution external detection method and system, and the method comprises the following steps: 1, obtaining a data set which comprises a small sample training set and a test set; 2, constructing a detection model, wherein the detection model comprises a data coding module, a noise extraction module and a fusion detection module; 3, performing noise enhancement on the noise region set, constructing a loss function in combination with the clean region set, training the detection model by adopting a gradient back propagation algorithm according to the loss function, and testing the detection model by adopting the test set; 4, performing small sample distribution external detection according to the tested detection model; according to the method, the problem of poor noise robustness of a current small sample distribution external detection method is solved, and the ID classification and OOD detection performance of a small sample learning model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of small sample out-of-distribution detection, and in particular to a noise-driven small sample out-of-distribution detection method and system. Background Art

[0002] For machine learning models deployed in open environments, it is crucial to have the ability to detect out-of-distribution (OOD) data. Few-Shot OOD Detection (FSOD), which aims to quickly learn the ability to identify in-distribution (ID) samples and OOD data from a small number of training samples, has received widespread attention in recent years. With the rise of pre-trained multimodal large models represented by CLIP, using prompt learning strategies to achieve rapid generalization of pre-trained general knowledge on target small sample tasks has become an effective way to solve small sample OOD detection.

[0003] However, existing methods such as FSOD add prompt words consisting of a small number of learnable parameters to the pre-trained CLIP model, and use the ID classification performance on the target task and the detection level of unknown OOD data under the premise of fixing the CLIP model parameters. Since the distribution space of unknown OOD data is vast and difficult to measure accurately, the models trained by such methods from a small number of "clean" ID samples are prone to misidentifying ID (or OOD) data with spectral background noise as OOD (or ID) data during the test phase. Summary of the invention

[0004] The present invention aims at solving the problems in the prior art and provides a noise-driven small sample out-of-distribution detection method and system.

[0005] The technical solution adopted by the present invention is:

[0006] A noise-driven small sample out-of-distribution detection method comprises the following steps:

[0007] Step 1: Obtain a data set, including a small sample training set and a test set;

[0008] Step 2: Build a detection model, which includes a data encoding module, a noise extraction module, and a fusion detection module;

[0009] The data encoding module is used to encode the text data and image data in the data to obtain the text category prototype and the local area feature set;

[0010] The noise extraction module calculates the similarity based on the text category prototype and the image region feature set, and divides the region into a clean region set and a noise region set according to the similarity and the set threshold;

[0011] The fusion detection module obtains the visual category prototype based on the clean area set, builds a scoring function based on the visual category prototype and the text category prototype, and determines whether the input data is an in-distribution image based on the result calculated by the scoring function and the threshold;

[0012] Step 3: Perform noise enhancement on the noisy region set, build a loss function based on the clean region set, train the detection model using the gradient back propagation algorithm based on the loss function, and test the detection model using the test set;

[0013] Step 4: Perform small sample out-of-distribution detection based on the tested detection model.

[0014] Furthermore, in the data encoding module in step 2, a text category prototype is obtained according to the text data according to a text encoder;

[0015] In the image data, a local area is randomly cropped for each picture and input into the visual encoder to obtain the local area feature set.

[0016] Furthermore, the process of calculating the similarity based on the text category prototype and the image region feature set in step 2 is as follows:

[0017] The similarity between the i-th image region feature and its corresponding text category prototype is calculated based on the cosine similarity.

[0018] Furthermore, the noise enhancement process for the noise region set in step 3 is as follows:

[0019] A critical noise generator is used to generate critical noise based on the noise region characteristics.

[0020] Furthermore, the scoring function S in step 2 MCM++ as follows:

[0021]

[0022] Where: p * (y=m|q) is the fusion detection probability that image q belongs to category m, where m is the visual category and the corresponding text category number;

[0023] in:

[0024]

[0025] Where: z is the global visual feature extracted by the visual encoder from sample q in the test set, the feature in the local area feature set, and h m and h m′ are the text category prototypes corresponding to categories m and m′, μ m and μ m′are the visual category prototypes corresponding to categories m and m′, τ is the temperature parameter, sim(z,h m ) for z and h m The cosine similarity, sim(z,h m′ ) for z and h m′ The cosine similarity of sim(z,μ m ) are z and μ m The cosine similarity of sim(z,μ m′ ) are z and μ m′ cosine similarity; m′ is the visual category and the corresponding text category number, and M is the number of visual category prototypes and text category prototypes.

[0026] Furthermore, the process of determining whether the input data is an in-distribution image is as follows:

[0027]

[0028] Where: λ (q) is the discriminant function, which indicates the category of the input image q, λ is the set threshold, ID is within the distribution, and OOD is out of the distribution.

[0029] Furthermore, the loss function in step 3 is for:

[0030]

[0031] Where: Learn a loss for the prompt constructed from the set of clean regions; is the critical enhancement loss constructed based on the clean region set and the enhanced noisy region set, and ξ is a hyperparameter.

[0032] Furthermore, the prompt learning loss is:

[0033]

[0034] Where: p(y=m|x) is the probability that x belongs to category m, y m is the mth value of the one-hot category label vector to which x belongs;

[0035] in:

[0036]

[0037] Furthermore, the critical enhancement loss is:

[0038]

[0039] Where: is the predicted probability vector of the regional feature r, r belongs to the enhanced noise region set, is the predicted probability vector of the regional feature z, z belongs to the clean region set, is the expectation, ε(·) is the entropy function, F + is the clean area set, is the enhanced noise region set.

[0040] A detection system for a noise-driven small sample out-of-distribution detection method, comprising a detection model, wherein the detection model comprises a data editing module, a noise extraction module, and a fusion classification module;

[0041] The data encoding module is used to encode the text data and image data in the data to obtain the text category prototype and the local area feature set;

[0042] The noise extraction module calculates the similarity based on the text category prototype and the image region feature set, and divides the region into a clean region set and a noise region set according to the similarity and the set threshold;

[0043] The fusion detection module obtains the visual category prototype based on the clean area set, builds a scoring function based on the visual category prototype and the text category prototype, and determines whether the input data is an in-distribution image based on the result calculated by the scoring function and the threshold;

[0044] It also includes a noise enhancement module for performing noise enhancement on a noise region set;

[0045] The enhanced noise enhancement set is used in combination with the clean area set to construct the loss function, and the detection model is trained using the gradient back propagation algorithm based on the loss function.

[0046] The beneficial effects of the present invention are:

[0047] (1) The present invention divides the image area into a clean area and a noisy area, adopts a critical noise generator, generates ID-OOD critical noise based on the noisy area, and approaches the unknown OOD data distribution space as much as possible; constructs a critical noise enhancement loss function based on information entropy according to the noise data set and the clean area set, which can improve the noise robustness and OOD detection level of the model;

[0048] (2) The present invention is based on multimodal prototype fusion reasoning, sets a scoring function based on visual category prototypes and text category prototypes, fuses the visual category prototypes and text category prototypes, and improves the ID classification and OOD detection performance of the constructed model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0051] like Figure 1 As shown: A noise-driven small sample out-of-distribution detection method, comprising the following steps:

[0052] Step 1: Obtain a data set, including a small sample training set and a test set;

[0053] Step 2: Build a detection model, which includes a data encoding module, a noise extraction module, and a fusion detection module;

[0054] The data encoding module is used to encode the text data and image data in the data to obtain the text category prototype and the local area feature set;

[0055] The noise extraction module calculates the similarity based on the text category prototype and the image region feature set, and divides the region into a clean region set and a noise region set according to the similarity and the set threshold;

[0056] The fusion detection module obtains the visual category prototype based on the clean area set, builds a scoring function based on the visual category prototype and the text category prototype, and determines whether the input data is an in-distribution image based on the result calculated by the scoring function and the threshold;

[0057] Step 3: Perform noise enhancement on the noisy region set, build a loss function based on the clean region set, train the detection model using the gradient back propagation algorithm based on the loss function, and test the detection model using the test set;

[0058] Step 4: Perform small sample out-of-distribution detection based on the tested detection model.

[0059] The present invention can be implemented based on the existing CLIP model, and the specific process is as follows:

[0060] Step 1: Obtain a data set, including a small sample training set and a test set;

[0061] Step 2: Build a detection model, which includes a data encoding module, a noise extraction module, and a fusion detection module;

[0062] The data encoding module is used to edit the text data and image data in the data to obtain the text category prototype and the local area feature set;

[0063] The noise extraction module calculates the similarity based on the text category prototype and the image region feature set, and divides the region into a clean region set and a noise region set according to the similarity and the set threshold;

[0064] The fusion detection module obtains the visual category prototype based on the clean area set, builds a scoring function based on the visual category prototype and the text category prototype, and determines whether the input data is an in-distribution image based on the result calculated by the scoring function and the threshold;

[0065] Step 3: Train the model; the model can adopt the existing CLIP model.

[0066] The input is the pre-trained CLIP model, the small sample training set S, the prompt word vector Ω, and the maximum number of iterations maxEpoch.

[0067] S1: treat the target as an alias {CLS1,…,CLS M} to "a photo of a", enter the text editor in CLIP T Get M text category prototypes {h1,…,h M};

[0068] S2: Define the visual category prototype storage set V = {}, initialize the hyperparameters (J, ρ, σ 2 ,ξ), the number of iterations t=0;

[0069] S3: t = t + 1;

[0070] S4: From the small sample training set Sampling a set of training samples

[0071] S5: For each image x in B i Randomly crop k J×J local regions and input them into the visual encoder f in CLIP V , and obtain the local area feature set where z i is the regional visual feature, y i ∈Y∶={1,...,M}, corresponding to the category of the original image.

[0072] S6: Calculate the image region feature z i Its corresponding text category prototype h i The similarity of the similarity set {s1,…,s κ×M}, and divide the regions with higher and lower scores in F into clean region sets F according to the threshold + and the noise region set F - .

[0073] S7: Using the image region feature F of each category + , introduce momentum parameter γ = 0.2, calculate visual category prototype (Class Center), and add or update V:

[0074]

[0075] S8: F - Do critical noise enhancement, with z i ∈F - , and sample η=3 times from the multivariate Gaussian distribution centered at (i=1,…,U) to obtain {e1,…,e η};

[0076]

[0077] Among them, σ 2 is the variance, I represents the dimension and z i The same unit vector, the enhanced noise region set is obtained

[0078] S9: Exploitation Calculating Hint Learning Loss

[0079]

[0080] Where: p(y=m|x) is the probability that x belongs to category m, y m is the mth value of the one-hot category label vector to which x belongs;

[0081] in:

[0082]

[0083] Where: sim(·,·) is the cosine similarity, τ is the temperature parameter, h m =f T (t m ), there is t m ={ω1,…,ω l ,CLS m} represents the text prompt of the mth category.

[0084] S10: According to F + and Calculating critical enhancement loss

[0085]

[0086] Where: is the predicted probability vector of the regional feature r, r belongs to the enhanced noise region set, is the predicted probability vector of the regional feature z, z belongs to the clean region set, is the expectation, ε(·) is the entropy function, F + is the clean area set, is the enhanced noise region set. Where o represents r or z.

[0087] S11: Calculate the training loss and perform gradient direction propagation to update the prompt vector Ω, Ω={ω1,…,ω l};

[0088]

[0089] Where: Learn a loss for the prompt constructed from the set of clean regions; is the critical enhancement loss constructed based on the clean region set and the enhanced noisy region set, and ξ is a hyperparameter.

[0090] S12: Determine whether the preset maxEpoch is reached. If so, end the training. Otherwise, return to S3.

[0091] Output: Target task specific prompts Visual category prototypes {μ1,…,μ M}.

[0092] After the training is completed, the test is carried out. The existing OOD detection method based on prompt learning usually uses the MCM scoring function to construct the OOD detector in the reasoning stage. However, this method has weak robustness.

[0093] In view of the problems existing in the prior art, the present invention constructs a new scoring function, and fuses the visual category prototype and the text category prototype to obtain a new MCM scoring function.

[0094] as follows:

[0095]

[0096] Where: p * (y=m|q) is the fusion detection probability that image q belongs to category m, where m is the visual category and the corresponding text category number;

[0097] in:

[0098]

[0099] Where: z is the global visual feature extracted by the visual encoder of the test sample q, h m and h m′ are the text category prototypes corresponding to categories m and m′, μ m and μ m′ are the visual category prototypes corresponding to categories m and m′, τ is the temperature parameter, sim(z,h m ) for z and h mThe cosine similarity, sim(z,h m′ ) for z and h m′ The cosine similarity of sim(z,μ m ) are z and μ m The cosine similarity of sim(z,μ m′ ) are z and μ m′ cosine similarity; m′ is the visual category and the corresponding text category number, and M is the number of visual category prototypes and text category prototypes.

[0100] The process of judging whether the input data is an in-distribution image is as follows:

[0101]

[0102] Where: λ (q) is the discriminant function, which indicates the category of the input image q, λ is the set threshold, ID is within the distribution, and OOD is out of the distribution.

[0103] λ is the ID-OOD decision threshold. In order to further improve the ID sample classification performance by using multimodal prototype knowledge, p * (y=m|q) is used to calculate the classification probability of the model for the input test image q.

[0104] Step 4: Perform small sample out-of-distribution detection based on the tested detection model.

[0105] The detection process is as follows:

[0106] The visual features of the input image are extracted through the visual encoder module, and then the detection result is calculated with the text category prototype and the stored clean area visual feature set; the final detection result is obtained through the fusion detection module.

[0107] In order to illustrate the effect of the present invention, the method of the present invention and the existing small sample out-of-distribution detection methods such as LoCoOp (Miyai A, Yu Q, Irie G, et al. Locoop: Few-shot out-of-distribution detection via prompt learning. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2024, 37) and ID-like (Bai Y, Han Z, Cao B, et al. ID-Like Prompt Learning for Few-Shot Out-of-Distribution Detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024, 17480-17489) are used to obtain the average OOD detection performance on the four OOD test sets iNaturalist, SUN, Places and Texture when ImageNet-1K is used as the ID data set. The results are shown in Table 1. Among them, 16-shot, only 16 training samples are sampled for each category.

[0108] Table 1. Performance comparison between the present invention and the existing method (16-shot)

[0109]

[0110] FPR95 is used to measure the misjudgment of OOD samples when the model maintains a high recognition rate (95%) for ID samples; AUROC is used to measure the overall ability of the model to classify positive and negative samples. The smaller the FPR95 value, the larger the AUROC value, and the stronger the model's OOD detection ability.

[0111] It can be seen from Table 1 that the proposed method has achieved better performance than the existing methods in both noise robustness and OOD detection tasks.

[0112] The method of the present invention divides the image area in the training sample into clean area and noise area, turning the noise area into treasure, and uses it to compact the ID category distribution space, thereby improving the ID classification and OOD detection performance of the small sample learning model. It solves the problem of poor noise robustness of the current existing FSOD method.

[0113] The present invention has a powerful visual-text alignment capability with the help of the CLIP model, and divides the image area in the sample into clean areas and noise areas; generates ID-OOD critical features through a critical noise generator, and obtains a loss function based on information entropy based on noise features, which is used to compact the ID category distribution space and expand the distribution interval of ID categories and OOD data. Through the multimodal prototype fusion reasoning mechanism, a new MCM scoring function is constructed to promote the complementary advantages of category prototypes in the visual feature space and the text feature space, and improve the ID classification and OOD detection level of the model. And the critical enhancement loss function constructed by it can be used to compact the ID category representation and improve the classification performance, while also helping to avoid the model from making overconfident predictions about OOD noise during the testing phase.

Claims

1. A noise-driven small sample out-of-distribution detection method, characterized in that: The following steps are involved: Step 1: Obtain a data set, including a small sample training set and a test set; Step 2: Build a detection model, which includes a data encoding module, a noise extraction module, and a fusion detection module; The data encoding module is used to encode the text data and image data in the data to obtain the text category prototype and the local area feature set; The noise extraction module calculates the similarity based on the text category prototype and the image region feature set, and divides the region into a clean region set and a noise region set according to the similarity and the set threshold; The fusion detection module obtains the visual category prototype based on the clean area set, builds a scoring function based on the visual category prototype and the text category prototype, and determines whether the input data is an in-distribution image based on the result calculated by the scoring function and the threshold; Step 3: Perform noise enhancement on the noisy region set, build a loss function based on the clean region set, train the detection model using the gradient back propagation algorithm based on the loss function, and test the detection model using the test set; Step 4: Perform small sample out-of-distribution detection based on the tested detection model.

2. A noise-driven small sample out-of-distribution detection method according to claim 1, characterized in that: In the data encoding module in step 2, a text category prototype is obtained according to the text data according to the text encoder; In the image data, a local area is randomly cropped for each picture and input into the visual encoder to obtain the local area feature set.

3. A noise-driven small sample out-of-distribution detection method according to claim 2, characterized in that: The process of calculating the similarity based on the text category prototype and the image region feature set in step 2 is as follows: The similarity between the i-th image region feature and its corresponding text category prototype is calculated based on the cosine similarity.

4. A noise-driven small sample out-of-distribution detection method according to claim 3, characterized in that: The noise enhancement process for the noise region set in step 3 is as follows: A critical noise generator is used to generate critical noise based on the noise region characteristics.

5. A noise-driven small sample out-of-distribution detection method according to claim 4, characterized in that: In step 2, the scoring function S MCM++ as follows: Where: p * (y=m|q) is the fusion detection probability that image q belongs to category m, where m is the visual category and the corresponding text category number; in: Where: z is the global visual feature extracted by the visual encoder from sample q in the test set, the feature in the local area feature set, and h m and h m′ are the text category prototypes corresponding to categories m and m′, μ m and μ m′ are the visual category prototypes corresponding to categories m and m′, τ is the temperature parameter, sim(z,h m ) for z and h m The cosine similarity, sim(z,h m′ ) for z and h m′ The cosine similarity of sim(z,μ m ) is the cosine similarity between z and μm, sim(z,μ m′ ) is z and μm ′ cosine similarity; m′ is the visual category and the corresponding text category number, and M is the number of visual category prototypes and text category prototypes.

6. A noise-driven small sample out-of-distribution detection method according to claim 5, characterized in that: The process of determining whether the input data is an in-distribution image is as follows: Where: λ (q) is the discriminant function, which indicates the category of the input image q, λ is the set threshold, ID is within the distribution, and OOD is out of the distribution.

7. A noise-driven small sample out-of-distribution detection method according to claim 6, characterized in that: The loss function in step 3 for: Where: Learn a loss for the prompt constructed from the set of clean regions; is the critical enhancement loss constructed based on the clean region set and the enhanced noisy region set, and ξ is a hyperparameter.

8. The noise-driven small sample out-of-distribution detection method according to claim 7, characterized in that: The hint learning loss is: Where: p(y=m|x) is the probability that x belongs to category m, y m is the mth value of the one-hot category label vector to which x belongs; in:

9. The noise-driven small sample out-of-distribution detection method according to claim 7, characterized in that: The critical enhancement loss is: Where: is the predicted probability vector of the regional feature r, r belongs to the enhanced noise region set, is the predicted probability vector of the regional feature z, z belongs to the clean region set, is the expectation, ε(·) is the entropy function, F + is the clean area set, is the enhanced noise region set.

10. A detection system using the noise-driven small sample out-of-distribution detection method according to any one of claims 1 to 9, characterized in that: Including a detection model, the detection model includes a data editing module, a noise extraction module, and a fusion classification module; The data encoding module is used to encode the text data and image data in the data to obtain the text category prototype and the local area feature set; The noise extraction module calculates the similarity based on the text category prototype and the image region feature set, and divides the region into a clean region set and a noise region set according to the similarity and the set threshold; The fusion detection module obtains the visual category prototype based on the clean area set, builds a scoring function based on the visual category prototype and the text category prototype, and determines whether the input data is an in-distribution image based on the result calculated by the scoring function and the threshold; It also includes a noise enhancement module for performing noise enhancement on a noise region set; The enhanced noise enhancement set is used in combination with the clean area set to construct the loss function, and the detection model is trained using the gradient back propagation algorithm based on the loss function.

Images