A model safety testing method, apparatus, device, and storage medium

Abstract

This application discloses a model security testing method, apparatus, device, and storage medium, relating to the field of artificial intelligence technology. The method includes: acquiring test samples consisting of test images and corresponding object detection labels, and a visual object detection model to be tested for performing an object detection task; generating adversarial examples based on the test samples using the visual object detection model to be tested; inputting the test samples and adversarial examples into the visual object detection model to be tested respectively to obtain corresponding prediction results, thereby determining the target test scores of target neurons in the visual object detection model to be tested, and generating security test results. By comparing the results of adversarial examples and test samples to quantify neuron scores, sensitive neurons in the model can be identified, which can more accurately determine model security and improve the model's anti-interference ability and security stability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model security testing method, apparatus, device, and storage medium. Background Technology

[0002] In recent years, with the development of artificial intelligence, the safety and reliability of AI systems have become increasingly important. Interpretability, generalizability, and security and trustworthiness are the three core pillars supporting the large-scale deployment of AI. For high-risk application scenarios such as medical image diagnosis, autonomous driving, industrial quality inspection, and smart security, visual target detection models undertake missions such as lesion detection, traffic entity recognition, defect identification, and abnormal behavior monitoring. The stability and interpretability of their output are directly related to personal safety and public interest.

[0003] Taking autonomous driving as an example, models must accurately detect pedestrians, vehicles, and traffic signs under conditions such as rain, snow, fog, haze, changes in day and night lighting, occlusion interference, and even malicious attacks. In medical image analysis, they need to accurately locate lesion areas on low-contrast, high-noise images. If a model makes a false detection or a missed detection due to a small input perturbation, it may lead to serious consequences such as traffic accidents or misdiagnosis, and will be deemed "unreliable" in standardized assessments and compliance reviews. With the successive implementation of industry standards for AI (Artificial Intelligence) medical device registration, functional safety of intelligent connected vehicles, and industrial intelligent equipment, relying solely on "high accuracy" black-box models is no longer sufficient to meet the comprehensive review requirements of regulatory agencies for the interpretability and credibility of algorithms. How to identify the weak links within the model in order to understand the model's safety risks and inherent defects has become an unresolved problem in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a model security testing method, apparatus, device, and storage medium. By comparing the results of adversarial examples and test samples, the neuron scores are quantified to identify sensitive neurons in the model, thereby more accurately detecting model security. The specific solution is as follows:

[0005] Firstly, this application provides a model security testing method, including:

[0006] Based on the model testing task, corresponding test samples and visual object detection models to be tested are obtained; the test samples consist of test images and corresponding object detection labels, and the visual object detection models to be tested are pre-trained neural networks used to perform object detection tasks;

[0007] The visual target detection model under test is used to generate corresponding adversarial examples based on the test samples.

[0008] The test sample and the adversarial sample are respectively input into the visual target detection model to be tested to obtain the corresponding test sample prediction results and adversarial sample prediction results. Based on the test sample prediction results and the adversarial sample prediction results, the target test score of the target neuron in the visual target detection model to be tested is determined.

[0009] The security test results of the visual target detection model under test are generated based on the target test score.

[0010] Optionally, generating corresponding adversarial examples based on the test samples using the visual target detection model under test includes:

[0011] The pixel values ​​of the test image in the test sample are mapped to a preset pixel range to obtain a standardized test image;

[0012] Using the visual target detection model to be tested, the adversarial sample is generated based on a preset loss function and the standardized test image; the preset loss function is a loss function constructed based on the attack target of a preset adversarial attack type.

[0013] Optionally, generating the adversarial examples based on a preset loss function and the standardized test image includes:

[0014] Determine the gradient direction of the preset loss function for the standardized test image, and determine the corresponding cumulative momentum based on the gradient direction;

[0015] Adversarial noise is generated based on the gradient direction and the accumulated momentum, and the adversarial noise is added to the normalized test image to generate an initial sample;

[0016] The adversarial sample is generated based on the initial sample.

[0017] Optionally, generating the adversarial sample based on the initial sample includes:

[0018] The initial sample is detected by the visual target detection model under test, and the attack success rate of the initial sample is determined based on the detection results.

[0019] Determine the target norm between the initial sample and the test sample; the target norm is used to characterize the degree of difference between the initial sample and the test sample.

[0020] Based on the attack success rate and the target norm, target samples that meet the preset conditions are selected from the initial samples, and the target samples are used as adversarial samples.

[0021] Optionally, determining the target test score of the target neuron in the visual target detection model under test based on the test sample prediction result and the adversarial example prediction result includes:

[0022] Identify each neuron to be tested in the visual target detection model under test;

[0023] Determine the first activation value corresponding to the prediction result of the test sample for the neuron under test, and determine the second activation value corresponding to the prediction result of the test sample for the neuron under test;

[0024] Determine the target difference between the first activation value and the second activation value;

[0025] The initial test score of the neuron under test is determined based on the absolute value of the target difference, and the target neuron in the neuron under test is determined based on the initial test score; the initial test score is used to characterize the neuron's sensitivity to sample detection.

[0026] Determine the target test score corresponding to the target neuron.

[0027] Optionally, determining the target test score corresponding to the target neuron includes:

[0028] Determine the relative entropy between the prediction results of the test samples and the prediction results of the adversarial samples, and determine the corresponding adversarial loss based on the relative entropy;

[0029] The gradient weights of the first activation value of the visual target detection model under test with respect to the adversarial loss are determined, and the target test score of the target neuron is determined based on the gradient weights.

[0030] Optionally, determining the target test score of the target neuron based on the gradient weights includes:

[0031] A target index is generated based on the product of the gradient weight and the second activation value of the target neuron; the target index is used to quantify the sensitivity of the visual target detection model under test to each image region in the sample image of the adversarial example, and the target index is a three-dimensional matrix.

[0032] The target test score of the target neuron is determined based on the value of the target dimension in the target index.

[0033] Secondly, this application provides a model safety testing apparatus, comprising:

[0034] The sample acquisition module is used to acquire corresponding test samples and visual target detection models to be tested based on the model testing task; the test samples consist of test images and corresponding target detection labels, and the visual target detection models to be tested are pre-trained neural networks used to perform target detection tasks;

[0035] The sample generation module is used to generate corresponding adversarial samples based on the test samples using the visual target detection model to be tested.

[0036] The score determination module is used to input the test sample and the adversarial sample into the visual target detection model to be tested, respectively, to obtain the corresponding test sample prediction results and adversarial sample prediction results, and to determine the target test score of the target neuron in the visual target detection model to be tested based on the test sample prediction results and the adversarial sample prediction results.

[0037] The result generation module is used to generate the security test results of the visual target detection model under test based on the target test score.

[0038] Thirdly, this application provides an electronic device, which includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the aforementioned model security testing method.

[0039] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the aforementioned model security testing method.

[0040] This application can obtain corresponding test samples and a visual object detection model to be tested based on a model testing task. The test samples consist of test images and corresponding object detection labels. The visual object detection model to be tested is a pre-trained neural network used to perform the object detection task. The application uses the visual object detection model to be tested to generate corresponding adversarial samples based on the test samples. Then, the test samples and adversarial samples are respectively input into the visual object detection model to be tested to obtain the corresponding test sample prediction results and adversarial sample prediction results. Based on the test sample prediction results and adversarial sample prediction results, the target test score of the target neuron in the visual object detection model to be tested is determined. Finally, the security test result of the visual object detection model to be tested is generated based on the target test score. After obtaining test samples and the visual target detection model to be tested, this application generates adversarial samples that can deceive the model based on the test samples. Normal test samples and adversarial samples are then input into the visual target detection model to be tested, resulting in two sets of prediction results. The difference between these two sets of results is used to calculate the target test score of the target neurons in the visual target detection model to quantify the sensitivity of neurons to perturbations, thereby outputting security test results. This addresses the problem of traditional models only detecting and outputting accuracy, failing to pinpoint specific weaknesses in the model, leading to one-sided and inaccurate detection results. By comparing the results of adversarial samples and test samples to quantify neuron scores, sensitive neurons in the model are identified. The abstract concept of model security is transformed into target test scores and specific sensitive neurons, allowing for a more intuitive and accurate determination of model security. This precisely identifies vulnerable neurons and related feature regions that are highly sensitive to input perturbations, revealing weaknesses within the network. This facilitates users' systematic assessment of the model's security risks and potential defects, guiding model iteration and improving the model's anti-interference ability and security stability. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 A flowchart of a model safety testing method provided in this application;

[0043] Figure 2 A flowchart for model security testing provided in this application;

[0044] Figure 3 This application provides a schematic diagram of an adversarial example generation method.

[0045] Figure 4 A schematic diagram illustrating the sensitivity score calculation process for each neuron in a neural network, provided in this application;

[0046] Figure 5 This application provides a schematic diagram of the structure of a model safety testing device.

[0047] Figure 6 This application provides a structural diagram of an electronic device. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0050] With the development of artificial intelligence, relying solely on black-box models is no longer sufficient to meet the comprehensive review requirements of regulatory agencies regarding the interpretability and credibility of algorithms when testing target detection models. This application quantifies neuron scores by comparing the results of adversarial examples and test samples, identifies sensitive neurons in the model, and transforms the abstract model security into target test scores and specific sensitive neurons, which can more intuitively and accurately detect model security.

[0051] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] See Figure 1 As shown, this embodiment of the invention discloses a model safety testing method, including:

[0053] Step S11: Obtain the corresponding test samples and the visual target detection model to be tested based on the model testing task; the test samples consist of test images and corresponding target detection labels, and the visual target detection model to be tested is a pre-trained neural network used to perform the target detection task.

[0054] In this embodiment, firstly as follows Figure 2 As shown, test samples and a visual object detection model to be tested are obtained based on the model testing task. The test samples consist of test images and corresponding object detection labels, while the visual object detection model to be tested is a pre-trained neural network used to perform the object detection task. It can be understood that the above model testing task is a test task constructed based on the object detection task and the corresponding visual object detection model to be tested. Specifically, the test sample images for the object detection task consist of two parts: images acquired by acquisition devices such as cameras, or images processed by upstream algorithms, and manually labeled object detection labels. Correspondingly, the neural network used to obtain the test set is a pre-trained object detection neural network, and the algorithm input includes network weights and the network architecture.

[0055] Step S12: Generate corresponding adversarial samples based on the test samples using the visual target detection model to be tested.

[0056] In this embodiment, as Figure 2 As shown, an adversarial sample set corresponding to the image test samples under the visual object detection model to be tested can be generated. Specifically, the pixel values ​​of each image in the test samples can be mapped to a preset pixel range to obtain a standardized test image. Then, adversarial samples are generated using the visual object detection model to be tested, based on a preset loss function and the standardized test image. The preset loss function is a loss function constructed based on the attack target of a preset adversarial attack type. Figure 3 As shown, based on the input data, pre-trained model, and corresponding label information from the above steps, the target direction of the attack is clarified. Specifically, adversarial examples can be constructed through image standardization and adversarial attack loss to ensure that subsequent attacks can be carried out on standardized data. First, the input image is standardized by mathematically transforming the pixel values ​​to a specific range to improve the numerical stability and convergence efficiency of the attack algorithm. In this embodiment, a minimum-maximum scaling method can be used to scale the original pixel values ​​from a range... Linear mapping to target interval Its formula is defined as:

[0057] ;

[0058] Where x is the original pixel value. Given the standardized pixel values, if the pixel values ​​are compressed from [0, 255] to [0, 1], the formula simplifies to: .

[0059] Then, the adversarial attack loss is constructed based on the attack target. For attacks without a target, only the real label needs to be provided. This allows the model output to be any error category; for targeted attacks, the target label must be specified. This can cause the model to misclassify the label as belonging to a specific category. Labels are typically represented using one-hot vectors, such as the true label. Target tag Furthermore, the cross-entropy loss function is often chosen as the above loss function:

[0060] ;

[0061] in, The loss value represents the degree of deviation between the model's current prediction and the set attack target. Here are the parameters of the neural network to be tested, including core parameters such as weights and biases from the model's training, used to calculate the model's prediction results for the input image; x is the image currently input to the model, i.e., the image being tested; y is the target label of the attack, i.e., the category we want the model to recognize. For untargeted attacks, y = The goal is to cause the model to deviate from the true class, with a targeted attack on y= The goal is to make the model accurately identify this error category; in other words, when there is a targeted attack, the loss function needs to be maximized to make the model tend towards the target category. Untargeted attacks minimize the loss function, causing the model to deviate. y is represented using one-hot encoding; This represents the sum of the differences across all categories; i represents the category number. That is, the value of the i-th class in the target label y; Let be the probability predicted by the model for the i-th class of input x.

[0062] Furthermore, when generating adversarial examples based on a preset loss function and standardized test images, it is necessary to first determine the gradient direction of the preset loss function with respect to the standardized test images, and then determine the corresponding accumulated momentum based on the gradient direction. Adversarial noise is then generated based on the gradient direction and accumulated momentum, and added to the standardized test images to generate initial samples. Adversarial examples are then generated based on these initial samples. In this embodiment, the processed image data and corresponding labels obtained above can be used to construct an adversarial attack loss function, and adversarial noise can be generated through iterative optimization. When generating adversarial noise, multiple gradient calculations and noise updates can be performed to generate adversarial examples that are sufficient to deceive the model while having minimal perturbation. It is understood that a balance between attack strength and stealth needs to be struck during the noise addition process. In this embodiment, iterative methods such as PGD (Projected Gradient Descent, an optimization algorithm for adversarial example generation and adversarial training) can be used to achieve optimization by constraining the perturbation norm. Specifically, gradient calculation and momentum accumulation are performed first. During gradient calculation, the partial derivative of the loss function with respect to the input data is first obtained. This indicates the direction in which the loss function grows fastest. Taking a targeted attack as an example, to maximize the model's loss on the target label, the gradient direction is... Furthermore, to improve stability, this embodiment introduces a momentum term to simulate physical inertia and avoid local oscillations. The momentum accumulation formula is:

[0063] ;

[0064] in The cumulative gradient at the current time. The momentum coefficient, For the gradient direction of the previous step, the denominator is... Norm normalization ensures that the update direction is normalized. In this way, the momentum mechanism described above can accelerate convergence and improve the success rate of attacks.

[0065] Then, adversarial noise perturbations are added to the original input image to update the generated adversarial examples. The noise addition generates perturbations based on the gradient direction, and the perturbation amplitude is constrained through a projection operation. Taking PGD as an example, the update formula for the t-th iteration is:

[0066] ;

[0067] in Step size, The function takes the sign of the gradient (generating ±1 perturbations). Indicates the original sample Centered on, with radius norm sphere, projection operation Ensure that the perturbed samples meet the requirements This refers to the invisibility of the perturbation. It is understood that the iterative process in this embodiment continues until the attack is successful or the maximum number of iterations is reached, at which point it stops.

[0068] Furthermore, such as Figure 3As shown, the initial samples can be detected by the visual target detection model under test, and the attack success rate of the initial samples can be determined based on the detection results. Then, the target norm between the initial samples and the test samples is determined. The target norm is used to characterize the degree of difference between the initial samples and the test samples. Target samples that meet preset conditions in the initial samples are selected based on the attack success rate and the target norm, and these target samples are used as adversarial samples. That is, in this embodiment, during the generation of adversarial samples, it is necessary to evaluate the attack success rate of the generated adversarial samples and the stealth of the injected attack noise. Commonly used indicators include attack success rate (ASR) and stealth measure. In this embodiment, the attack success rate is defined as the proportion of samples that successfully deceive the model out of the total samples. For example, the test set contains M samples, and an adversarial sample set is generated. Its label is The formula for calculating ASR is:

[0069] ;

[0070] in This is an indicator function; it takes the value 1 when the model predicts incorrectly, and 0 otherwise. Furthermore, for targeted attacks, it must strictly satisfy the following condition: The higher the ASR, the more effective the attack.

[0071] Then, the stealth of the adversarial attack is evaluated. In this embodiment, stealth is measured by the degree of difference between the adversarial example and the original example using... Norm, its formula is:

[0072] ;

[0073] in, It is the norm, used to calculate the overall difference between the adversarial example and the original sample; x is the original test sample; For adversarial examples; That is, the value of the i-th pixel in the original sample, where i is the pixel index; That is, the value of the i-th pixel in the adversarial sample, and correspond.

[0074] Then, the generated set of adversarial examples that meets the evaluation criteria can be saved. Based on the above steps, this embodiment first constructs a test sample set with learnable noise perturbation, and inputs the perturbation sample set into the visual target detection model to be tested to obtain the network output prediction results under the perturbation samples. Then, the adversarial attack loss is calculated using the model's prediction results under the perturbation samples. After that, the weights of the test network are frozen, and gradient backpropagation is performed through the adversarial attack loss to optimize the perturbation noise, thus obtaining the adversarial example set.

[0075] Step S13: Input the test sample and the adversarial sample into the visual target detection model to be tested, respectively, to obtain the corresponding test sample prediction results and adversarial sample prediction results, and determine the target test score of the target neuron in the visual target detection model to be tested based on the test sample prediction results and the adversarial sample prediction results.

[0076] In this embodiment, test samples and adversarial samples are input into the visual target detection model under test, respectively, to obtain the corresponding test sample prediction results and adversarial sample prediction results. Based on the test sample prediction results and adversarial sample prediction results, the target test score of the target neuron in the visual target detection model under test is determined. Specifically, firstly, each neuron to be tested in the visual target detection model under test needs to be identified, and the first activation value corresponding to the test sample prediction result and the second activation value corresponding to the test sample prediction result need to be determined. Then, the target difference between the first activation value and the second activation value is determined, and the initial test score used to characterize the neuron's sensitivity to sample detection is determined based on the absolute value of the target difference. Then, the target neuron in the neuron to be tested is determined based on the initial test score, and then the corresponding target test score of the target neuron is determined. That is, as shown... Figure 4 As shown, in this embodiment, the sensitive regions of the visual target detection model under test in the adversarial example set can be located, and the sensitivity scores of each neuron in the neural network can be calculated. First, the output prediction results of the model and the activation values ​​of each neuron in the network layer are calculated under normal samples and adversarial samples obtained in step S12. The formula is as follows:

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] in For the target detection neural network, This refers to the set of neurons in the target layer of the model. For the model to adapt to the original input The prediction results For the model against adversarial examples The prediction results Indicates the first The location of each neuron in the feature map activation value, Indicates the first The location of each neuron in the feature map The activation values ​​under adversarial example input. It should be noted that the above model prediction results... and This refers to the logits vector directly output by the network, not the post-processed logits output from the algorithm. Logits are the real-valued vectors generated by the last linear transformation layer of the deep learning classification model before the final activation function normalization, representing the raw scores of each category before probabilistic processing. Then, the absolute value of the difference between the activation values ​​of neurons in the tested neural network under normal samples and those under adversarial sample inputs is calculated as a quantitative indicator to measure whether the neuron has undergone a significant change before and after an adversarial attack. The formula is as follows:

[0082] ;

[0083] in, That is, the first The sensitivity score of a neuron indicates that its activation value has changed significantly after being subjected to adversarial attacks, making it a candidate sensitive neuron, or target neuron. Specifically, it can be determined whether the sensitivity score exceeds a set threshold; if it does, it is classified as a potentially sensitive and vulnerable neuron. The formula is as follows:

[0084] ;

[0085] T is a hyperparameter used to control the screening threshold for sensitive neurons, and N is the set of potentially sensitive and vulnerable neurons.

[0086] It is understandable that a neural network sensitive neuron refers to a neuron in a neural network model whose activation value is easily changed significantly when the input is slightly perturbed, thus affecting the final output result; a neural network sensitive region refers to an area in the input of a neural network model where a slight perturbation to the pixels or feature values ​​within the region can cause a significant change in the activation value of some neurons, thus affecting the final output result.

[0087] Furthermore, when determining the target test score of the target neuron, the relative entropy between the prediction results of the test samples and the prediction results of the adversarial samples can be determined. Based on the relative entropy, the corresponding adversarial loss can be determined. Then, the gradient weight of the first activation value of the visual object detection model under test with respect to the adversarial loss can be determined, and the target test score of the target neuron can be determined based on the gradient weight. In this embodiment, the KL divergence, i.e., the relative entropy, between the output result of the neural network under normal samples and the model output result under adversarial sample input is first calculated as the adversarial loss under test. Its formula is as follows:

[0088] ;

[0089] in Represents the calculation of KL divergence. This is the adversarial loss under testing. Understandably, KL divergence is typically used to measure the difference between two vectors; the greater the difference, the greater the KL divergence. Through this... The function can calculate the difference in output of the model before and after it suffers an adversarial attack. Then, when the adversarial example is input, the test network is frozen, and backpropagation is performed using the adversarial loss under the test as the starting point for gradient direction propagation. The gradient of the activation values ​​of the test neural network with respect to the adversarial loss under the test is calculated and used as the gradient weight coefficients. The formula is as follows:

[0090] ;

[0091] in, The neuron activation value. For feature map sequence number, These are the position coordinates on the feature map; Indicates the first Location on each feature map The gradient value of the neuron quantifies the impact of small changes in neuron activation values ​​on the overall loss. The larger the gradient value, the more significant the contribution of the neuron to the change in model decision and the higher its sensitivity. The smaller the gradient value, the less sensitive the neuron is to perturbations and the stronger its robustness.

[0092] Then, based on the product of the gradient weights and the second activation value of the target neuron, a target index is generated. The target index quantifies the sensitivity of the visual object detection model under test to each image region in the sample image of adversarial examples. The target index is a three-dimensional matrix including three dimensions: C, H, and W. The target test score of the target neuron is then determined based on the value of the target dimension in the target index. In this embodiment, the weights obtained in the above steps are multiplied by the activation values ​​of potentially sensitive and vulnerable neurons. Values ​​less than 0 are removed from the calculation result through numerical truncation, and normalization is performed to obtain the final sensitive region quantification index, i.e., the target index. This is then indexed by channel to obtain C two-dimensional matrices with dimensions H and W. Higher values ​​in the two-dimensional matrix indicate that the texture of the corresponding region is more likely to cause confusion in the test model, representing a sensitive and vulnerable region of the model. The formula is as follows:

[0093] ;

[0094] ;

[0095] ;

[0096] in This indicates that under adversarial example input, the first... The location of each neuron in the feature map The result after being weighted by gradient weight coefficients; represent The function is used to exclude values ​​less than 0 from the calculation result. and They represent in The dimension is taken as the minimum and maximum values ​​of the input matrix; To obtain the final normalized quantification index of the sensitive region, the neuron can be obtained through pseudo-color image processing. A heatmap of sensitive regions is visualized using the current adversarial attack test sample. The sensitivity scores for each neuron channel are then obtained by summing the quantification indicators of the sensitive regions along both the H and W dimensions. A higher score indicates weaker robustness of the neuron under adversarial attacks, classifying it as a sensitive and vulnerable neuron. The formula is as follows:

[0097] ;

[0098] in For the first Sensitivity score of each neuron.

[0099] Step S14: Generate the security test result of the visual target detection model under test based on the target test score.

[0100] In this embodiment, the security test results of the visual target detection model under test can be generated based on the aforementioned target test scores. Specifically, this is achieved by locating the sensitive neurons of the visual target detection model under test and the sensitive regions under the adversarial example set. First, a hyperparameter U is defined. All channel neurons of the layer to be analyzed in the neural network under test are sorted from high to low according to their sensitivity scores. The top U% of neurons are selected as sensitive neurons, and their layer and channel numbers are recorded. A visual heatmap of their sensitive regions is then output.

[0101] This embodiment first obtains test samples of images for the target detection task and the neural network to be tested, and generates an adversarial example set corresponding to the test samples of the visual target detection model to be tested. Then, the corresponding method is used to locate the sensitive regions of the visual target detection model to be tested under the adversarial example set, and the sensitivity score of each neuron in the neural network is calculated to locate the sensitive neurons of the visual target detection model to be tested and the sensitive regions under the adversarial example set, thus obtaining the security test results of the model. Based on the above technical solution, this embodiment takes a deep learning-based target detection model as the research object. By comparing the activation differences of neurons before and after perturbation injection, the change process of the model's decision path can be dynamically restored, which can deeply analyze the robustness and stability of each neuron in the model when facing adversarial attacks, and the impact on the overall decision process of the model. Furthermore, the change trajectory of the model's decision process is intuitively presented through the sensitive neuron and sensitive region localization algorithm to accurately locate the vulnerable neurons in the model that are sensitive to perturbation, thereby accurately locking the vulnerable neurons and related feature regions that are extremely sensitive to input perturbation, revealing the weak links inside the network, so as to deeply understand the security risks and inherent defects of the model, and facilitate users to systematically evaluate the security risks and potential defects of the model.

[0102] See Figure 5 As shown in the embodiments, this application also discloses a model safety testing device, including:

[0103] The sample acquisition module 11 is used to acquire corresponding test samples and a visual target detection model to be tested based on the model testing task; the test samples consist of test images and corresponding target detection labels, and the visual target detection model to be tested is a pre-trained neural network used to perform the target detection task;

[0104] The sample generation module 12 is used to generate corresponding adversarial samples based on the test samples using the visual target detection model to be tested.

[0105] The score determination module 13 is used to input the test sample and the adversarial sample into the visual target detection model to be tested, respectively, to obtain the corresponding test sample prediction results and adversarial sample prediction results, and to determine the target test score of the target neuron in the visual target detection model to be tested based on the test sample prediction results and the adversarial sample prediction results.

[0106] The result generation module 14 is used to generate the security test result of the visual target detection model under test based on the target test score.

[0107] In this embodiment, after obtaining test samples and the visual target detection model to be tested, adversarial samples that can deceive the model are generated based on the test samples. Normal test samples and adversarial samples are then input into the visual target detection model to be tested, resulting in two sets of prediction results. The difference between these two sets of results is used to calculate the target test score of the target neurons in the visual target detection model to quantify the sensitivity of neurons to perturbations, thereby outputting the security test results. This solves the problem that traditional models only detect and output accuracy, failing to locate specific weak points in the model, leading to one-sided and inaccurate detection results. By comparing the results of adversarial samples and test samples to quantify neuron scores, sensitive neurons in the model are identified. The abstract model security is transformed into target test scores and specific sensitive neurons, allowing for a more intuitive and accurate determination of model security. This enables precise identification of vulnerable neurons and related feature regions that are highly sensitive to input perturbations, revealing weaknesses within the network and facilitating users' systematic assessment of the model's security risks and potential defects.

[0108] In some specific embodiments, the sample generation module 12 includes:

[0109] The image processing submodule is used to map each pixel value of the test image in the test sample to a preset pixel range to obtain a standardized test image;

[0110] The sample generation submodule is used to generate the adversarial sample based on the visual target detection model to be tested, a preset loss function, and the standardized test image; the preset loss function is a loss function constructed based on the attack target of a preset adversarial attack type.

[0111] In some specific embodiments, the sample generation submodule specifically includes:

[0112] The parameter determination unit is used to determine the gradient direction of the preset loss function for the standardized test image, and to determine the corresponding cumulative momentum based on the gradient direction.

[0113] A noise addition unit is used to generate adversarial noise based on the gradient direction and the accumulated momentum, and to add the adversarial noise to the normalized test image to generate an initial sample;

[0114] A sample generation unit is used to generate the adversarial sample based on the initial sample.

[0115] In some specific embodiments, the sample generation unit is specifically used to: detect the initial sample using the visual target detection model to be tested, and determine the attack success rate of the initial sample based on the obtained detection results; determine the target norm between the initial sample and the test sample; the target norm is used to characterize the degree of difference between the initial sample and the test sample; filter target samples in the initial sample that meet preset conditions based on the attack success rate and the target norm, and use the target samples as the adversarial samples.

[0116] In some specific embodiments, the score determination module 13 specifically includes:

[0117] The neuron determination submodule is used to determine each neuron to be tested in the visual target detection model under test;

[0118] The activation value determination submodule is used to determine the first activation value corresponding to the prediction result of the neuron under test and the test sample, and to determine the second activation value corresponding to the prediction result of the neuron under test and the test sample.

[0119] The difference determination submodule is used to determine the target difference between the first activation value and the second activation value;

[0120] The neuron screening submodule is used to determine the initial test score of the neuron to be tested based on the absolute value of the target difference, and to determine the target neuron among the neurons to be tested based on the initial test score; the initial test score is used to characterize the neuron's sensitivity to sample detection;

[0121] The score determination submodule is used to determine the target test score corresponding to the target neuron.

[0122] In some specific embodiments, the score determination submodule specifically includes:

[0123] The loss determination unit is used to determine the relative entropy between the prediction results of the test samples and the prediction results of the adversarial samples, and to determine the corresponding adversarial loss based on the relative entropy;

[0124] A score determination unit is used to determine the gradient weight of the first activation value of the visual target detection model under test with respect to the adversarial loss, and to determine the target test score of the target neuron based on the gradient weight.

[0125] In some specific embodiments, the score determination unit is specifically used to generate a target index based on the product of the gradient weight and the second activation value of the target neuron; the target index is used to quantify the sensitivity of the visual target detection model under test to each image region in the sample image of the adversarial example, and the target index is a three-dimensional matrix; the target test score of the target neuron is determined based on the value of the target dimension in the target index.

[0126] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0127] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the model security testing method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0128] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0129] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0130] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the model security testing method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0131] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed model security testing method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0132] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0133] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0135] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0136] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model security testing method characterized by comprising: The method comprises: acquiring corresponding test samples and a visual target detection model to be tested based on a model test task; the test samples are composed of test images and corresponding target detection labels, and the visual target detection model to be tested is a pre-trained neural network used for performing a target detection task; generating corresponding adversarial samples based on the test samples by using the visual target detection model to be tested; inputting the test samples and the adversarial samples into the visual target detection model to be tested respectively to obtain corresponding test sample prediction results and adversarial sample prediction results, and determining target test scores of target neurons in the visual target detection model to be tested based on the test sample prediction results and the adversarial sample prediction results; generating a security test result of the visual target detection model to be tested based on the target test scores; wherein the determination of the target test scores of the target neurons in the visual target detection model to be tested based on the test sample prediction results and the adversarial sample prediction results comprises: determining each neuron to be tested in the visual target detection model to be tested; determining first activation values corresponding to the test sample prediction results of the neurons to be tested, and determining second activation values corresponding to the test sample prediction results of the neurons to be tested; determining target difference values between the first activation values and the second activation values; determining initial test scores of the neurons to be tested based on absolute values of the target difference values, and determining the target neurons in the neurons to be tested based on the initial test scores; the initial test scores are used to represent detection sensitivity of the neurons to samples; determining the target test scores of the target neurons; and the determination of the target test scores of the target neurons comprises: determining relative entropy between the test sample prediction results and the adversarial sample prediction results, and determining a corresponding adversarial loss based on the relative entropy; determining gradient weights of the first activation values of the visual target detection model to be tested with respect to the adversarial loss, and determining the target test scores of the target neurons based on the gradient weights; the determination of the target test scores of the target neurons based on the gradient weights comprises: generating target indicators based on products of the gradient weights and the second activation values of the target neurons; the target indicators are used to quantify sensitivity of the visual target detection model to be tested to each image region in sample images of the adversarial samples, and the target indicators are three-dimensional matrices; determining the target test scores of the target neurons based on values of target dimensions in the target indicators.

2. The model security testing method of claim 1, wherein, the generation of the corresponding adversarial samples based on the test samples by using the visual target detection model to be tested comprises: mapping each pixel value of the test images in the test samples to a preset pixel interval to obtain standardized test images; The adversarial sample is generated based on a preset loss function and the standardized test image; the preset loss function is a loss function constructed based on an attack target of a preset adversarial attack type.

3. The model security testing method of claim 2, wherein, The adversarial sample is generated based on the preset loss function and the standardized test image, including: A gradient direction of the preset loss function with respect to the standardized test image is determined, and a corresponding accumulated momentum is determined based on the gradient direction; Adversarial noise is generated based on the gradient direction and the accumulated momentum, and the adversarial noise is added to the standardized test image to generate an initial sample; The adversarial sample is generated based on the initial sample.

4. The model security testing method of claim 3, wherein, The adversarial sample is generated based on the initial sample, including: The initial sample is detected by the visual object detection model to be tested, and an attack success rate of the initial sample is determined based on a detection result obtained; A target norm between the initial sample and the test sample is determined; the target norm is used to represent a difference degree between the initial sample and the test sample; A target sample meeting a preset condition is screened from the initial sample based on the attack success rate and the target norm, and the target sample is taken as the adversarial sample.

5. A model security testing apparatus characterized by comprising: including: A sample acquisition module is configured to acquire a test sample and a visual object detection model to be tested based on a model test task; The test sample is composed of a test image and a corresponding target detection label, and the visual object detection model to be tested is a pre-trained neural network used to perform a target detection task; A sample generation module is configured to generate a corresponding adversarial sample based on the test sample by using the visual object detection model to be tested; A score determination module is configured to input the test sample and the adversarial sample into the visual object detection model to be tested respectively to obtain a test sample prediction result and an adversarial sample prediction result, and determine a target test score of a target neuron in the visual object detection model to be tested based on the test sample prediction result and the adversarial sample prediction result; A result generation module is configured to generate a security test result of the visual object detection model to be tested based on the target test score. The score determination module includes: A neuron determination submodule is configured to determine each neuron to be tested in the visual object detection model to be tested; An activation value determination submodule is configured to determine a first activation value corresponding to the test sample prediction result of the neuron to be tested, and determine a second activation value corresponding to the test sample prediction result of the neuron to be tested; A difference value determination submodule is configured to determine a target difference value between the first activation value and the second activation value; A neuron screening submodule is configured to determine an initial test score of the neuron to be tested based on an absolute value of the target difference value, and determine the target neuron in the neuron to be tested based on the initial test score; the initial test score is used to represent a detection sensitivity of the neuron to a sample; A score determination submodule is configured to determine the target test score of the target neuron. And the score determination submodule comprises: A loss determination unit is configured to determine a relative entropy between the test sample prediction result and the adversarial sample prediction result, and determine a corresponding adversarial loss based on the relative entropy; A score determination unit is configured to determine a gradient weight of the first activation value of the to-be-tested visual target detection model with respect to the adversarial loss, and determine the target test score of the target neuron based on the gradient weight; And the score determination unit is specifically configured to generate a target index based on a product of the gradient weight and the second activation value of the target neuron; the target index is used to quantify a sensitivity of the to-be-tested visual target detection model to each image region in a sample image of the adversarial sample, and the target index is a three-dimensional matrix; the target test score of the target neuron is determined based on a value of a target dimension in the target index.

6. An electronic device, comprising: The electronic device comprises a processor and a memory; wherein the memory is configured to store a computer program, the computer program is loaded and executed by the processor to implement the model security test method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer program is stored, and the computer program is executed by a processor to implement the model security test method according to any one of claims 1 to 4.

Images