Detection score-based out-of-distribution detection adaptive attack method
By adaptively adjusting the attack target and direction, highly targeted adversarial samples are generated, which solves the problem of insufficient comprehensive and powerful adversarial samples in the prior art, and realizes an effective evaluation of the robustness of the off-distribution detector.
Patent Information
- Application Number
- CN202510856981.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-25
- Publication Date
- 2025-08-19
AI Technical Summary
When evaluating the robustness of the externally distributed detector, the generated adversarial samples are not comprehensive and powerful enough, resulting in the safety of the externally distributed detection model not being effectively evaluated.
By collecting data from internal and external detectors, adaptively adjusting the attack target and direction, setting the optimal number of iteration steps and step length, using an adversarial optimization algorithm to generate adaptive attack adversarial samples, and attacking against different sample types.
The generated adversarial samples are more comprehensive and powerful to attack the out-of-distribution detector, which can effectively evaluate its robustness and verify the security of the model.
Smart Images
Figure CN120512296A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning security technology, and in particular to a method for detecting out-of-distribution adaptive attacks based on detection scores. Background Art
[0002] When deploying deep learning systems in open scenarios, the models inevitably encounter input data that is inconsistent with the training distribution, known as out-of-distribution samples. Conventional deep learning models struggle to process such samples and often exhibit overconfidence, posing a serious threat to model security. Out-of-distribution detectors can distinguish between out-of-distribution samples, enhancing model security to a certain extent. Out-of-distribution detection distinguishes in-distribution and out-of-distribution samples in open scenarios, preemptively filtering out out-of-distribution samples that subsequent models cannot process, which is crucial for system security. However, conventional out-of-distribution detection algorithms only study the ability of out-of-distribution detectors to distinguish out-of-distribution samples in clean samples, or only consider some common covariate shifts (such as lighting and style changes), without considering the existence of adversarial examples. If adversarial examples can be fabricated that can cause out-of-distribution detectors to make significant errors, they pose a significant threat to out-of-distribution detection models in open scenarios. If an attacker applies malicious adversarial noise to the input, causing the out-of-distribution detector to make significant errors, a large number of in-distribution samples will be rejected, while a large number of out-of-distribution samples will be incorrectly fed into subsequent models, seriously compromising system security.
[0003] In the field of adversarial robustness, conventional adversarial examples are generated against classifiers. That is, the attacker generates adversarial examples by applying adversarial noise to the input, causing the classifier to classify the input into the wrong category. However, the attack target of adversarial examples targeting out-of-distribution detectors is different. Their goal is to make the out-of-distribution detector recognize in-distribution inputs as out-of-distribution, or to recognize out-of-distribution inputs as in-distribution. In this case, there are two types of adversarial examples: adversarial in-distribution samples and adversarial out-of-distribution samples. The semantics of adversarial in-distribution samples are in-distribution. After the attacker applies noise, the out-of-distribution detector detects them as out-of-distribution, thereby incorrectly rejecting the input. The semantics of adversarial out-of-distribution samples are out-of-distribution. After the attacker applies noise, the out-of-distribution detector detects them as in-distribution, thereby passing the input to the subsequent model. However, out-of-distribution inputs cannot be processed by the subsequent model, which leads to security issues.
[0004] Some research has explored the impact of adversarial examples in out-of-distribution detection and used them to evaluate the robustness of out-of-distribution detectors. However, the adversarial examples generated by these methods are not comprehensive or robust enough, hindering the robustness of out-of-distribution detectors. Some methods fail to consider both in-distribution and out-of-distribution examples, some use classification losses instead of out-of-distribution scoring functions when generating adversarial examples, and some evaluate the robustness of out-of-distribution detectors only under relatively small perturbations. These shortcomings hinder the robustness of out-of-distribution detectors. Summary of the Invention
[0005] The purpose of the present invention is to provide an out-of-distribution detection adaptive attack method based on detection scores. This method can solve the problem that the adversarial samples generated by the existing technology when evaluating the robustness of out-of-distribution detectors are not comprehensive and powerful enough. It provides a powerful and effective solution for the robustness evaluation of out-of-distribution detectors, which is conducive to verifying the security of out-of-distribution detection models.
[0006] The purpose of the present invention is achieved through the following technical solutions:
[0007] A method for detecting out-of-distribution adaptive attacks based on detection scores, the method comprising:
[0008] Step 1: Collect in-distribution detector data and out-of-distribution detector data as raw samples and annotate them according to the input resolution of the out-of-distribution detector;
[0009] Step 2: Adaptively adjust the attack target based on the detection scores that the out-of-distribution detector relies on;
[0010] Step 3: Adaptively adjust the attack direction according to the different sample types input;
[0011] Step 4: According to the impact of the number of iteration steps and the iteration step length on the attack strength, set the optimal number of iteration steps and the iteration step length parameters;
[0012] Step 5: Based on the set attack target and attack direction, use the adversarial optimization algorithm to iteratively generate adaptive attack adversarial samples.
[0013] It can be seen from the technical solution provided by the above-mentioned present invention that the above-mentioned method can solve the problem that the adversarial samples generated by the existing technology when evaluating the robustness of out-of-distribution detectors are not comprehensive and powerful enough, and provides a powerful and effective solution for the robustness evaluation of out-of-distribution detectors, which is conducive to verifying the security of the out-of-distribution detection model. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0015] Figure 1 A schematic diagram of a flow chart of a method for detecting out-of-distribution adaptive attacks based on detection scores provided by an embodiment of the present invention;
[0016] Figure 2 This is a schematic diagram of the analysis results of the iterative step length according to an embodiment of the present invention;
[0017] Figure 3 Schematic diagram of analysis results of the number of iteration steps according to an embodiment of the present invention;
[0018] Figure 4 Schematic diagram of the results of the adaptive attack described in an embodiment of the present invention. DETAILED DESCRIPTION
[0019] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, and do not constitute a limitation of the present invention. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0020] like Figure 1 FIG. 1 is a flow chart of a method for detecting out-of-distribution adaptive attacks based on detection scores according to an embodiment of the present invention. The method includes:
[0021] Step 1: Collect in-distribution detector data and out-of-distribution detector data as raw samples and annotate them according to the input resolution of the out-of-distribution detector;
[0022] In this step, the original samples are obtained by using public datasets, network acquisition or actual shooting. The original samples are unified through preprocessing according to the input resolution of the out-of-distribution detector;
[0023] Collect all the data that are judged as in-distribution samples by the out-of-distribution detector as the in-distribution dataset, denoted as D id ;
[0024] All data judged as out-of-distribution samples by the out-of-distribution detector are collected as the out-of-distribution dataset, denoted as D ood .
[0025] Step 2: Adaptively adjust the attack target based on the detection scores that the out-of-distribution detector relies on;
[0026] In this step, the detection scores that the out-of-distribution detector relies on include the MSP (Maximum Softmax Probability) score, the energy score, or the Mahalanobis distance, where:
[0027] If an out-of-distribution detector with MSP score is used, the attack target is set as:
[0028]
[0029] Where S(x) is the out-of-distribution detection score; f j (x) is the output of the classifier in dimension j; K is the number of categories, and the maximum output dimension of the classifier is recorded as k;
[0030] If an out-of-distribution detector with energy fraction is used, the attack target is set as:
[0031]
[0032] Where T represents the temperature coefficient;
[0033] If the out-of-distribution detector of Mahalanobis distance is used, the attack target is set as:
[0034]
[0035] Where φ(x) represents the selected feature layer; p c is the category center of category c; ∑ represents the covariance matrix at the selected feature layer.
[0036] Step 3: Adaptively adjust the attack direction according to the different sample types input;
[0037] In this step, for a batch of input data X batch , adaptively adjust the attack direction according to the different sample types input, specifically:
[0038] For in-distribution samples in the input data, the attack goal is to minimize the out-of-distribution detection score, so the attack loss is:
[0039] L(x)=S(x)
[0040] The meaning of this formula is: the optimization goal of the loss L(x) is to reduce the out-of-distribution detection score S(x). The out-of-distribution detector G(x) will compare the detection score (x) corresponding to the input x with the threshold λ. If (x)>λ, the input x will be judged as an in-distribution sample. Therefore, by optimizing the above loss, the detection score is reduced. When the detection score S(x) is lower than the threshold λ, the out-of-distribution detector will mistakenly judge the input x as an out-of-distribution sample, thereby attacking the out-of-distribution detector.
[0041] For out-of-distribution samples in the input data, the attack goal is to maximize the out-of-distribution detection score, so the attack loss is:
[0042] L(x)=-S(x)
[0043] The meaning of this formula is: the optimization goal of the loss L(x) is to increase the out-of-distribution detection score S(x). The out-of-distribution detector G(x) will compare the detection score S(x) corresponding to the input x with the threshold λ. If (x) < λ, the input x will be judged as an out-of-distribution sample. Therefore, by optimizing the above loss, the detection score is increased. When the detection score S(x) is higher than the threshold λ, the out-of-distribution detector will mistakenly judge the input x as an in-distribution sample, thereby attacking the out-of-distribution detector.
[0044] Through adaptive adjustment, the generated adversarial samples are made more comprehensive for out-of-distribution detection, which can comprehensively evaluate the robustness of the out-of-distribution detection algorithm.
[0045] Step 4: According to the impact of the number of iteration steps and the iteration step length on the attack strength, set the optimal number of iteration steps and the iteration step length parameters;
[0046] In this step, if Figure 2 The figure shows the analysis results of the iteration step according to the embodiment of the present invention. The data points in the figure are 0.5 / 255, 1 / 255, 2 / 255, 3 / 255, 4 / 255, 5 / 255, 6 / 255, 7 / 255, and 8 / 255, respectively. For the iteration step α, it is similar to the learning rate. If it is too large, it will cause the solution to jump around the optimal solution. If it is too small, it will not converge to the optimal point within a limited number of steps. Figure 2 The results show that when the iteration step is set to 2 / 255, the AUROC index of the out-of-distribution detector is the smallest, and the attack strength is the largest at this time. Therefore, in order to obtain a stronger attack strength, it is appropriate to set the iteration step α to 2 / 255.
[0047] like Figure 3The figure shows the analysis results of the number of iterations according to the embodiment of the present invention. The data points in the figure are 1, 2, 5, 10, 25, 50, 100, 200, and 500, respectively. For the number of iterations t, the larger the number of iterations, the more conducive it is to converge to the optimal solution. However, too large a number of iterations will result in a long time to generate adversarial samples. Therefore, it is necessary to select a suitable value according to Figure 3 The results show that when the number of iterations is less than 10, the AUROC indicator of the out-of-distribution detector does not reach its minimum, and there is still room for the attack strength to increase. When the number of iterations is greater than 10, the AUROC indicator gradually converges, and when it is greater than 200, the AUROC indicator hardly decreases. However, too large an iteration number will incur a large time cost. Therefore, in order to achieve a strong attack strength and balance the time cost, setting the number of iterations to 10 to 200 is more appropriate.
[0048] Step 5: Based on the set attack target and attack direction, use the adversarial optimization algorithm to iteratively generate adaptive attack adversarial samples.
[0049] In this step, the optimization problem to be solved is:
[0050]
[0051] stx adv =x+δ,||δ||<∈
[0052] Where δ represents the adversarial perturbation; ∈ is the perturbation boundary, which is used to limit the adversarial perturbation to a bounded area; x is a clean sample, that is, the original input without perturbation; x adv It is an adversarial sample, which is composed of the original input and the copied perturbation superimposed;
[0053] The optimization goal of this optimization problem is to find the optimal adversarial perturbation δ so that the adversarial sample x generated based on this perturbation adv It can minimize the loss function, which has different forms depending on the input type, as described in step 3; the first constraint of the optimization problem represents the adversarial sample x adv It is composed of the original input x and the adversarial perturbation δ. The second constraint condition indicates that the amplitude of the adversarial perturbation δ is controlled by the boundary ∈ constraint. This constraint ensures the semantic invariance of the adversarial sample and prevents excessive perturbation from changing the semantic information of the input.
[0054] Use PGD (Projected Gradient Descent) to solve the optimization problem in multiple steps. Specifically, the adversarial sample x adv The iterative solution formula is:
[0055]
[0056] Where α represents the iteration step size; t represents the number of iteration steps;
[0057] After multiple projection iterations, adversarial samples are obtained, including samples within the adversarial distribution. and adversarial out-of-distribution samples
[0058] In a specific implementation, the strength of the adaptive attack method described in the embodiment of the present invention is verified on a common out-of-distribution detection method, such as Figure 4 The figure shows the result of the adaptive attack described in the embodiment of the present invention. It is observed that the adaptive attack of the present invention will significantly reduce the performance of the out-of-distribution detection method, which shows the power of the method and is an effective out-of-distribution detection robustness verification method.
[0059] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the method.
[0060] An embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method.
[0061] It should be noted that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.
[0062] To sum up, the method described in the embodiment of the present invention adaptively adjusts the attack direction through different sample types, and the generated adversarial samples are more comprehensive for out-of-distribution detection, and can comprehensively evaluate the robustness of the out-of-distribution detection algorithm; this method adaptively adjusts the attack target according to different detection scores, and the generated adversarial samples are more targeted and powerful for out-of-distribution detection; at the same time, this method analyzes the influence of the number of iteration steps and the iteration step size on the attack intensity, and gives a suitable parameter setting range, which provides an effective reference for the robustness evaluation of out-of-distribution detection.
[0063] In addition, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.
[0064] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.
Claims
1. A method for detecting out-of-distribution adaptive attacks based on detection scores, characterized in that: The method comprises: Step 1: Collect in-distribution detector data and out-of-distribution detector data as raw samples and annotate them according to the input resolution of the out-of-distribution detector; Step 2: Adaptively adjust the attack target based on the detection scores that the out-of-distribution detector relies on; Step 3: Adaptively adjust the attack direction according to the different sample types input; Step 4: According to the impact of the number of iteration steps and the iteration step length on the attack strength, set the optimal number of iteration steps and the iteration step length parameters; Step 5: Based on the set attack target and attack direction, use the adversarial optimization algorithm to iteratively generate adaptive attack adversarial samples.
2. The out-of-distribution detection adaptive attack method based on detection score according to claim 1, characterized in that: In step 1, the original samples are obtained by using public datasets, network acquisition or actual shooting. The original samples are unified through preprocessing according to the input resolution of the out-of-distribution detector; Collect all the data that are judged as in-distribution samples by the out-of-distribution detector as the in-distribution dataset, denoted as D id ; All data judged as out-of-distribution samples by the out-of-distribution detector are collected as the out-of-distribution dataset, denoted as D ood .
3. The out-of-distribution detection adaptive attack method based on detection score according to claim 1, characterized in that: In step 2, the detection scores that the out-of-distribution detector relies on include MSP scores, energy scores, or Mahalanobis distances, where: If an out-of-distribution detector with MSP score is used, the attack target is set as: Where S(x) is the out-of-distribution detection score; f j (x) is the output of the classifier in dimension j; K is the number of categories, and the maximum output dimension of the classifier is recorded as k; If an out-of-distribution detector with energy fraction is used, the attack target is set as: Where T represents the temperature coefficient; If the out-of-distribution detector of Mahalanobis distance is used, the attack target is set as: Where φ(x) represents the selected feature layer; p c is the category center of category c; ∑ represents the covariance matrix at the selected feature layer.
4. The out-of-distribution detection adaptive attack method based on detection score according to claim 1, characterized in that: In step 3, for a batch of input data X batch , adaptively adjust the attack direction according to the different sample types input, specifically: For in-distribution samples in the input data, the attack goal is to minimize the out-of-distribution detection score S(x), so the attack loss is: L(x)=S(x) The meaning of this formula is that the optimization goal of the loss L(x) is to reduce the out-of-distribution detection score S(x). The out-of-distribution detector G(x) compares the detection score S(x) corresponding to the input x with the threshold λ. If S(x)>λ, the input x is judged as an in-distribution sample. For out-of-distribution samples in the input data, the attack goal is to maximize the out-of-distribution detection score S(x), so the attack loss is: L(x)=-S(x) The meaning of this formula is: the optimization goal of the loss L(x) is to increase the out-of-distribution detection score S(x). The out-of-distribution detector G(x) will compare the detection score S(x) corresponding to the input x with the threshold λ. If S(x)<λ, the input x will be judged as an out-of-distribution sample.
5. The out-of-distribution detection adaptive attack method based on detection score according to claim 1, characterized in that: In step 5, the optimization problem to be solved is: s.t.x adv =x+δ,||δ||<∈ Where δ represents the adversarial perturbation; ∈ is the perturbation boundary, which is used to limit the adversarial perturbation to a bounded area; x is a clean sample, that is, the original input without perturbation; x adv It is an adversarial sample, which is composed of the original input and the copied perturbation superimposed; The optimization goal of this optimization problem is to find the optimal adversarial perturbation δ so that the adversarial sample x generated based on this perturbation adv Can minimize the loss function; the first constraint of the optimization problem represents the adversarial sample x adv It is composed of the original input x and the adversarial perturbation δ. The second constraint condition indicates that the amplitude of the adversarial perturbation δ is controlled by the boundary ∈ constraint. This constraint ensures the semantic invariance of the adversarial sample and prevents excessive perturbation from changing the semantic information of the input. Use PGD multi-step iteration to solve the optimization problem. Specifically, the adversarial sample x adv The iterative solution formula is: Where α represents the iteration step size; t represents the number of iteration steps; After multiple projection iterations, adversarial samples are obtained, including samples within the adversarial distribution. and adversarial out-of-distribution samples 6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 5.
7. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 5.