Deep neural network black box testing method based on key region analysis

Abstract

The invention discloses a deep neural network black box test method based on key region analysis, comprising the following steps: firstly, proposing a partition strategy to guide initialization of an image region and dynamic adjustment in an iteration process for an original input image; secondly, performing regional importance analysis on an input image by designing a discrimination function, and automatically positioning a key region on which a model decision depends; then, dynamically adjusting the initialized region, carrying out fine-grained analysis by utilizing a discriminant function, and carrying out iteration until the region is subdivided into a predefined size; and finally, based on the regional importance score, preferentially applying semantic preserving disturbance to the key region, and generating a targeted test sample through iterative query and feedback. According to the method, the decision dependency of the deep neural network is utilized, key region positioning can be achieved through model input and output, and compared with a traditional black box test method, the query frequency is effectively reduced; and a self-adaptive disturbance strategy is adopted, so that the sample semantic rationality is maintained while the test effectiveness is ensured. Meanwhile, the generated input image can be used for finely adjusting a target model, and the robustness of the model is improved.

Description

Technical Field

[0001] This invention belongs to the field of software testing technology, and in particular relates to a black-box testing method for deep neural networks based on critical region analysis. Background Technology

[0002] Due to their outstanding performance in processing large-scale, high-dimensional data, deep neural networks (DNNs) are increasingly being integrated into numerous security and security-sensitive fields, including image and speech recognition, malware detection, medical diagnostics, and autonomous driving systems. Despite their impressive capabilities, DNNs are inherently vulnerable and highly sensitive to often imperceptible perturbations in their input data. These minute changes can cause models to produce incorrect or even dangerous outputs, posing significant risks when deployed in real-world scenarios. Given the crucial role DNNs currently play in modern society, rigorous and systematic testing is not only beneficial but also essential for ensuring their robustness, reliability, and security. Effective testing can help uncover hidden flaws and ultimately enhance confidence in deploying DNN-based solutions in mission-critical applications.

[0003] Most existing testing methods for deep neural networks are designed to operate in white-box testing scenarios. In white-box testing, testers have full access to the model's internal architecture and parameters. These methods typically rely on predefined neuron selection strategies to maximize neuron activation and coverage. The overall goal of these strategies is to achieve high neuron coverage and detect input cases that trigger erroneous or unexpected model behavior. While these white-box testing methods have proven effective in identifying defects and robustness issues in neural network models, their reliance on complete and unrestricted access to detailed internal model information, including architecture, weights, and activation patterns, severely limits their applicability in the real world. Furthermore, some recent studies have shown that neuron coverage is not strongly correlated with the ability to detect defects, making these methods relatively inefficient. Therefore, there is an urgent need for alternative testing strategies that can ensure model reliability while respecting these access limitations.

[0004] To address the inherent limitations of white-box testing methods, black-box testing offers a reliable alternative for evaluating DNN models, especially when internal access is restricted. Typical black-box testing strategies identify defects or inconsistencies by iteratively querying selected inputs and analyzing the corresponding output predictions. Based on this feedback loop, input samples are refined through continuous mutation to reveal unexpected or erroneous behavior. However, one of the main challenges of black-box testing lies in the efficiency of its query process. Due to the lack of visibility into the model's internal knowledge, the discovery of erroneous inputs often requires a large number of inference queries. This reliance on extensive queries leads to significant computational costs and long testing cycles, especially for large models. Furthermore, many existing black-box testing techniques tend to generate candidate test inputs by operating on the entire feature space of the input data, such as globally modifying pixel values ​​or applying extensive perturbations to the original samples. While this brute-force strategy increases the diversity of generated samples, it often leads to the overuse of pixel-level variations, which not only reduces the semantic relevance and plausibility of the test data but also impairs the optimization efficiency of the search process. Therefore, the number of queries required increases, reducing the likelihood of successfully inducing model errors within a limited query budget. This trade-off underscores the need for more targeted and efficient input generation strategies that can improve the effectiveness of black-box testing while minimizing the cost of model queries. Summary of the Invention

[0005] Purpose of the invention: To address the issue of the effectiveness of testing strategies in black-box testing of deep learning, this invention aims to provide a black-box testing method for deep neural networks based on key region analysis. This method combines the decision dependencies of deep learning models to guide the generation of test samples with a more efficient black-box testing approach.

[0006] Technical solution: A black-box testing method for deep neural networks based on key region analysis, comprising: (1) Formulate a partitioning strategy to partition the original input image and generate a discriminant map for model testing; (2) Construct a discriminant function to perform region importance analysis on the input image. The discriminant function is defined as follows:

[0007] in, This indicates the original image After inputting the target model, the target model outputs the value of the true label category; Indicates in the original image A certain partition Add small perturbations The output value of the post-target model on the true label category; The discriminant function is used to evaluate the degree of influence of different regions on the target model decision in step (1), thereby identifying and automatically locating the key regions on which the target model decision depends; (3) The initial region is dynamically adjusted based on the output of the discriminant function, including fine-grained analysis and iteration of the original image until the region is subdivided into a predefined size; (4) Based on the regional importance score, semantically preserved perturbations are applied to key regions, and targeted test samples are generated through iterative queries and feedback.

[0008] Furthermore, the partitioning strategy described in step (1) divides the original image into segments of size [size missing]. The non-overlapping regions are square partitions, which initialize the original input image into several consecutive partitions determined by the coordinates of the top left corner and the side length.

[0009] The square partitions are as follows: For height, width, and number of color channels respectively , , Input image Square partition set The following formula conditions must be met:

[0010] in, This indicates the number of partitions in the original input image. Represents any partition; the first part of the above formula The set of all initial partitions represents all pixels of the original input image; Part Two This indicates that any two initial partitions do not overlap.

[0011] Furthermore, the dynamic adjustment process described in step (3) involves dividing the partition into smaller parts according to rules, generating a more refined discriminative image, and then using a discriminative function for analysis; The dynamic adjustment process is described as follows:

[0012] in, Indicates the first The region selected in the next iteration. This indicates that the size of each region is one-quarter of the size of the region in the previous iteration.

[0013] The specific steps in step (4) to generate targeted test samples through iterative querying and feedback are as follows: (41) Initialize the mutation vector, which is a random perturbation output according to a predetermined rule. The magnitude of the perturbation is within the specified Euclidean norm range, and the perturbed image maintains semantic consistency with the original image. (42) After adding the mutation vector to the original test input, analyze the output of the target model and determine whether a test case revealing a specific type of error in the model has been generated by the value of the objective function. If not, update the value of the mutation vector; if so, reinitialize the mutation vector. (43) After the number of queries for the current category is exhausted, the target category is switched, the objective function is changed accordingly, and the test cases that reveal errors in the next category are explored. (44) Repeat the above steps until all categories have been explored.

[0014] Furthermore, in step (41) above, each mutation vector is a perturbation block whose change value decreases from the center to the surrounding area, controlled by the mutation vector function. Their size is consistent with the partition size. The mutation vectors are represented as follows: , in, Represents the coordinates of a pixel within the mutated vector block; Indicates the center coordinates of the mutated vector block; , These represent two parameters that control the size of the mutation vector.

[0015] The objective function constructed in step (42) is defined as follows: , in, Indicates the target model's response to the perturbated image. The output probability distribution is a vector of predicted probabilities for each class; the operator This indicates that the predicted probability value corresponding to a specific category is indexed from the probability distribution vector; Indicates the input image The original tag, Indicates the target label.

[0016] The objective function represents the difference between the original label prediction value and the target label prediction value after adding perturbation. When the objective function value is less than 0, it indicates that an error-induced test case has been generated.

[0017] Beneficial effects: This invention is a black-box testing method for deep neural networks. By utilizing the decision dependencies of deep neural networks, key regions can be located through model input and output, effectively reducing the number of queries compared to traditional black-box testing methods. An adaptive perturbation strategy is employed to maintain the semantic rationality of the samples while ensuring test effectiveness. Simultaneously, the generated input images can be used to fine-tune the target model, improving its robustness. Attached Figure Description

[0018] Figure 1 This is a framework diagram of the method described in this invention. Detailed Implementation

[0019] This invention provides a black-box testing method for deep neural networks based on key region analysis. First, a partitioning strategy is applied to the original input image to guide the initialization of image regions and their dynamic adjustment during the iteration process. Then, a discriminant function is designed to perform region importance analysis on the input image, automatically locating key regions upon which the model's decisions depend. This operation identifies the key regions of the original image. For these key regions, the initialized regions are further dynamically adjusted, and fine-grained analysis is performed using the discriminant function, iterating until the regions are subdivided to a predefined size. Finally, based on the region importance score, semantically preserving perturbations are preferentially applied to the key regions, and targeted test samples are generated through iterative queries and feedback.

[0020] Furthermore, this invention addresses the technical shortcomings of existing black-box testing frameworks that fail to effectively utilize the decision-dependency characteristics of deep neural networks by proposing a discriminant function based on region sensitivity analysis. Traditional black-box testing methods suffer from limited efficiency due to the inability to access internal model parameters and structure, making it difficult to directly calculate internal feedback information such as gradients. This invention introduces the finite difference approximation concept from derivative calculation into the black-box testing scenario, constructing the discriminant function by measuring the rate of change of the target network's decisions under regional perturbations. This method achieves a quantitative assessment of regional importance by applying controllable micro-perturbations to different image regions and observing the relative changes in the model's output class probabilities. This design mathematically simulates the gradient calculation process in white-box testing, but only relies on the model's input-output mapping relationship, requiring no internal parameter information and strictly satisfying the constraints of black-box testing. By systematically comparing the numerical values ​​of the discriminant functions for each region, the key regions with the greatest impact on model decisions can be accurately identified, providing a theoretical basis for subsequent targeted testing. Compared to traditional methods, this technical solution maintains the applicability of black-box testing while significantly improving testing efficiency and targeting.

[0021] The main process and specific steps of implementing this invention are described below.

[0022] like Figure 1The diagram illustrates a framework for a deep neural network black-box testing method based on key region analysis. The technical input of this invention is the original test cases, and the output is the error-inducing test cases. The process of this invention consists of two important stages: region analysis and test case generation.

[0023] Region Analysis: Deep neural networks, especially those in image classification, tend to rely heavily on specific regions of an image containing the most relevant features for prediction. These regions are often referred to as "critical" or "salient" areas and are where the model's decisions are most sensitive. Based on this characteristic of deep neural networks, this invention proposes a region analysis method for deep learning image recognition systems. The intuition behind this method is that by quickly identifying these critical areas and applying perturbations to them, more effective test inputs can be generated that might trigger incorrect behavior in the model. This region analysis part is mainly used for domain-level image analysis. To this end, the method designs a partitioning strategy to divide the image into regions and further generate a series of discriminant images. A carefully designed discriminant function is used to evaluate the discriminant images to select regions containing important features. The specific implementation steps are as follows: Step 1: Initialize partitions and generate a series of discriminant images. During the initialization phase, the original image is divided into multiple non-overlapping regions. In this process, the geometric characteristics of the partitions play a crucial role in determining the quality of the generated test cases and the computational resources required for generation. A square shape is the optimal choice for partition geometry due to its mathematical regularity and computational advantages. Square partitions provide complete coverage of the input image space while ensuring that all partitions do not overlap. Therefore, this method initializes the original input image into several consecutive partitions determined by the coordinates of the top-left corner and the side length, and adds a small perturbation to each partition, thereby generating a series of discriminant images. The process is described below: For the input image Square partition set The following formula conditions must be met: , It mainly consists of two parts. Part One The set of all initial partitions represents all pixels of the original input image; Part Two This indicates that any two initial partitions do not overlap. This initialization method not only effectively partitions the original input image but also provides favorable mathematical conditions for subsequent computational evaluation.

[0024] Step 2: Evaluate the discriminant images using a discriminant function. Since each discriminant image is generated by adding a small perturbation to a region of the original input image, this method designs a discriminant function related to the rate of change of the true class prediction probability of the original input image. This function is used to evaluate the influence of different partitions on the model's decision, thereby locating key regions. After calculating the discriminant function values ​​for different regions, this method selects the most influential partitions (…). (This will allow for further analysis.)

[0025] Step 3: Dynamic Adjustment of Partitions. The previous round of selection has preliminarily identified regions containing important features, but these regions cover most of the partitions in the input image. Therefore, this method uses a dynamic adjustment approach to further refine the partitions. Specifically, the partitions are divided into smaller parts according to rules to generate more refined discriminative images, which are then analyzed using a discriminative function. The dynamic adjustment process is described as follows: , in, Indicates the first The region selected in the next iteration. This indicates that the size of each region is one-quarter of the size of the region in the previous iteration.

[0026] Step 4: Repeat steps 2 and 3 until the partition size reaches the predefined threshold.

[0027] Test Case Generation: To address the low testing efficiency in deep learning black-box testing, this invention utilizes region analysis to select regions and combines this with optimization strategies to generate new test inputs. This improves the quality of the test set and increases testing efficiency, thereby further optimizing test performance. This method combines a multi-stage strategy of perturbation generation and iterative optimization, aiming to enhance the sufficiency and effectiveness of testing by generating efficient test input sets. The implementation process of this method mainly includes the following steps: 1) Initialize the mutation vector. The mutation vector (Mutation Vector)... mv The perturbation vector is a random perturbation output according to predetermined rules. Its shape and size are consistent with the selected partition, and the magnitude of the perturbation is within the specified Euclidean norm range, thus ensuring that the perturbed image maintains semantic consistency with the original image. The representation of the perturbation vector is as follows:

[0028] in, Represents the coordinates of a pixel within the mutated vector block; Indicates the center coordinates of the mutated vector block; , In the present invention, two parameters controlling the size of the mutation vector are used. , By controlling the mutation vector function, each mutation vector is a perturbation block whose change value decreases from the center to the surrounding area, and their size is consistent with the partition size.

[0029] 2) Analyze the target model's output after adding the mutation vector to the original test input. After adding the perturbation to the original input, determine whether a test case revealing a specific type of error in the model has been generated based on the value of the objective function. If not, update the value of the mutation vector; otherwise, reinitialize the mutation vector. During iteration, this method only retains mutation vectors that can optimize the objective function and continues iteration; otherwise, it does not retain them. Repeat this process until the query count for the current category is exhausted.

[0030] 3) Switch target category. Once the query count for the current category is exhausted, switch target category, and the objective function changes accordingly. Begin exploring test cases that reveal errors in the next category.

[0031] 4) Repeat the above steps until all categories have been explored.

[0032] Through the aforementioned iterative process, the generated test inputs can more effectively expose model defects, improving testing efficiency and sufficiency. The test input generation method based on region analysis and feedback-guided optimization achieves comprehensive optimization of the test input generation process through innovative region analysis strategies, objective function optimization, and iterative generation methods. This method demonstrates significant advantages in improving test effectiveness, enhancing the ability of test cases to explore target model defects, and improving test efficiency, providing an efficient and practical solution for testing deep learning models. In the future, this method is expected to be further extended to complex scenarios, providing technical support for the testing of intelligent software and promoting the development of deep learning model quality assurance technology.

[0033] The specific implementation steps of this invention are illustrated below through concrete examples: The CIFAR-10 dataset was chosen for evaluation. CIFAR-10 contains 60,000 color images, each 32×32 pixels in size. This includes 50,000 training images and 10,000 test images. The VGG16 and Wide ResNet w34-10 (WRN) models were trained using the CIFAR-10 dataset, and the test accuracies are shown in Table 1. Acc before These two models are widely used for testing deep learning.

[0034] 1. Region analysis is performed on the original input image. This process first initializes partitions according to predefined parameters; for the CIFAR-10 dataset, the initial partition size is 8×8. Then, an iterative process is executed, evaluating, selecting, and dynamically adjusting partitions using a discriminant function, until the partition size reaches a predefined minimum. After several selections, 20% of the original image is ultimately selected as the perturbation target region. In the original image coordinate system, this process records the selected regions determined by the coordinates of the top-left corner and the final region side length, so that perturbations can be added to these regions in subsequent steps. The input to this process is randomly selected correctly classified images from the CIFAR-10 dataset, and the output is the location and size of the selected regions in these images, along with their discriminant function values.

[0035] 2. Generate test cases for the partitions selected in the steps above. For each partition, the process first randomly initializes a mutation vector with the same size as the selected region, ensuring that the Euclidean distance between the mutated image and the original image is less than 5. Then, it determines whether an error-inducing image has been generated based on the objective function value. During iteration, only mutation vectors that can optimize the objective function are retained and iteration continues; otherwise, they are not retained. This process is repeated until the 1000 query budget allocated to each category is exhausted. Then, the target category is switched until all categories have been explored. The input to this process is the original input image and its selected regions; the output is test samples that reveal different types of errors in the model.

[0036] 3. Fine-tune the model using the generated test cases. The ultimate goal of testing is to discover potential model defects through systematic testing and to enhance the model's robustness using these targeted error-inducing samples. Specifically, the test cases generated in the previous steps are added to the training set as augmentation data to fine-tune and optimize the original model. This process involves 10 independent runs for each fine-tuning experiment. Each run randomly selects 1000 test samples generated in the above steps and 5000 original training samples to form the fine-tuning dataset. Each model is fine-tuned for 5 rounds, and finally, the accuracy measurement results for all runs are collected and recorded.

[0037] This invention performs quantitative statistics on the test samples generated by the above process, including the number of samples generated under a given query budget ( ErrorNum The percentage of samples that can reveal model errors () SR ), generating an average number of queries required to generate a sample ( Avg.Q And the improved accuracy of the target model after fine-tuning (optimization efficiency) OE By applying the data to the CIFAR-10 dataset, the following statistics were obtained: Table 1. Application data of this invention on the CIFAR-10 dataset.

[0038] Based on the application results on the CIFAR-10 dataset, this invention demonstrates efficient testing capabilities and model defect detection capabilities. Specifically, on two typical models, VGG-16 and WRN, this method can generate 100% valid test samples under limited conditions, indicating its ability to stably generate valid test inputs. Regarding query efficiency, the average number of queries ( Avg.Q The number of iterations was only 1.14 and 1.12 respectively, significantly outperforming traditional black-box testing methods, demonstrating the advantages of the targeted perturbation strategy based on key region analysis in improving testing efficiency. This invention, by accurately locating key regions and optimizing perturbation strategies, can effectively reveal potential defects in deep learning models with limited query costs, providing an efficient and reliable solution for model robustness evaluation in black-box environments.

Claims

1. A black-box testing method for deep neural networks based on key region analysis, characterized in that, include: (1) Formulate a partitioning strategy to partition the original input image and generate a discriminant map for model testing; (2) Construct a discriminant function to perform region importance analysis on the input image. The discriminant function is defined as follows: ， in, This indicates the original image After inputting the target model, the target model outputs the value of the true label category; Indicates in the original image A certain partition Add small perturbations Then, the target model outputs the value of the true label category; The discriminant function is used to evaluate the degree of influence of different regions on the target model decision in step (1), thereby identifying and automatically locating the key regions on which the target model decision depends; (3) The initial region is dynamically adjusted based on the output of the discriminant function, including fine-grained analysis of the original image and iteration until the region is subdivided into a predefined size; (4) Based on the regional importance score, semantically preserved perturbations are applied to key regions, and targeted test samples are generated through iterative queries and feedback.

2. The deep neural network black-box testing method according to claim 1, characterized in that, The specific steps in step (4) to generate targeted test samples through iterative querying and feedback are as follows: (41) Initialize the mutation vector, which is a random perturbation output according to a predetermined rule. The magnitude of the perturbation is within the specified Euclidean norm range, and the perturbed image maintains semantic consistency with the original image. (42) After adding the mutation vector to the original test input, analyze the output of the target model and determine whether a test case revealing a specific type of error in the model has been generated by the value of the objective function. If not, update the value of the mutation vector; if so, reinitialize the mutation vector. (43) After the number of queries for the current category is exhausted, the target category is switched, the objective function is changed accordingly, and the test cases that reveal errors in the next category are explored. (44) Repeat the above steps until all categories have been explored.

3. The deep neural network black-box testing method according to claim 2, characterized in that, Step (41) involves controlling the mutation vector function. Each mutation vector is a perturbation block whose change value decreases from the center to the surrounding area. Their size is consistent with the partition size. The mutation vectors are represented as follows: ， in, Represents the coordinates of a pixel within the mutated vector block; Indicates the center coordinates of the mutated vector block; , These represent two parameters that control the size of the mutation vector.

4. The deep neural network black-box testing method according to claim 2, characterized in that, The objective function constructed in step (42) is defined as follows: ， in, Indicates the target model's response to the perturbated image. The output probability distribution is a vector of predicted probabilities for each class; the operator This indicates that the predicted probability value corresponding to a specific category is indexed from the probability distribution vector; Indicates the input image The original tag, Indicates the target label. The objective function represents the difference between the original label prediction value and the target label prediction value after adding perturbation. When the objective function value is less than 0, it indicates that an error-induced test case has been generated.

5. The deep neural network black-box testing method according to claim 1, characterized in that, The partitioning strategy described in step (1) is to divide the original image into partitions of size [size missing] based on its geometric characteristics. The non-overlapping regions are square partitions, which initialize the original input image into several consecutive partitions determined by the coordinates of the top left corner and the side length.

6. The deep neural network black-box testing method according to claim 5, characterized in that, The square partitions are as follows: For height, width, and number of color channels respectively , , Input image Square partition set The following formula conditions must be met: ， in, This indicates the number of partitions in the original input image. Represents arbitrary partitioning; the first part of the above formula The set of all initial partitions represents all pixels of the original input image; Part Two This indicates that any two initial partitions do not overlap.

7. The deep neural network black-box testing method according to claim 1, characterized in that, The dynamic adjustment process described in step (3) involves dividing the partition into smaller parts according to rules, generating a more refined discriminative image, and then using a discriminative function for analysis. The dynamic adjustment process is described as follows: ， in, Indicates the first The region selected in the next iteration. This indicates that the size of each region is one-quarter of the size of the region in the previous iteration.

Images