A deep neural network test input generation method based on reinforcement learning and dynamic variation selection

Abstract

The application discloses a kind of deep neural network test input generation methods based on reinforcement learning and dynamic variation selection, comprising the following steps: according to the prediction confidence distribution, the sample image classified correctly is reserved, and the seed test input set balancedly covering each category is obtained;State space is constructed, and gradient area and predefined variation operation are combined into discrete action space, and each seed image in seed test input set is cyclically executed: action is selected by policy network according to current state, variation is executed on seed image in specified gradient area, immediate reward is calculated and state is updated, while each step interaction experience is stored in replay pool for continuous updating of policy network;If current seed image triggers model misclassification, record the variation image and reset to the next seed image, if disturbance threshold or maximum number of steps is reached, directly reset to the next seed image;All variation images that trigger model misclassification are summarized to generate the final test input set.

Description

Technical Field

[0001] This invention belongs to the field of deep neural network testing and verification technology, specifically relating to a method for generating deep neural network test inputs based on reinforcement learning and dynamic mutation selection. Background Technology

[0002] Deep neural networks (DNNs) are widely used in safety-critical systems such as image recognition, autonomous driving, and avionics. However, the "black box" nature of DNNs causes them to exhibit unpredictable errors when subjected to small input perturbations. To verify the robustness of DNNs, a large number of test inputs that can trigger model defects are needed. Existing methods are mainly divided into two categories: coverage-guided fuzz testing (such as DeepXplore and DLFuzz) and test generation based on adversarial attacks (such as FGSM and PGD). These methods have the following shortcomings:

[0003] (1) The mutation strategy relies on static neuron selection or fixed gradient information, lacks the ability to adapt to the dynamic response of the model, resulting in perturbations being concentrated in low-value areas such as the background.

[0004] (2) It is impossible to evaluate the merits of strategies in long sequence variations, making it difficult to achieve true sequence decision optimization;

[0005] (3) The perturbation amplitude of the generated test input is uncontrollable and easily deviates from the semantics of the real scene.

[0006] Therefore, there is an urgent need for a test input generation method that can adaptively select variant regions and operations, and balance defect detection efficiency with visual rationality. Summary of the Invention

[0007] Purpose of the invention: In order to overcome the problems of blind mutation, insufficient generalization of defect detection and uncontrollable visual perturbation in existing deep neural network test input generation methods, this invention proposes a deep neural network test input generation method based on reinforcement learning and dynamic mutation selection.

[0008] Technical Solution: This invention proposes a method for generating test inputs using deep neural networks based on reinforcement learning and dynamic mutation selection, comprising the following steps:

[0009] The original test image set is input into the deep neural network. The correctly classified sample images are retained according to the predicted confidence distribution. The confidence screening interval is dynamically determined according to the category. A fixed number of sample images are selected from each category as seed images to obtain a seed test input set that evenly covers all categories.

[0010] Based on the seed image or the current mutated image in the iteration, the activation state of neurons inside the deep neural network, the mutation state of the image itself, the model decision state, and gradient region features are obtained to construct a state space. The gradient region is a pixel importance region based on gradient. At the same time, the gradient region is combined with predefined mutation operations to form a discrete action space, so that each action of the agent specifies both the mutation region and the mutation operation type.

[0011] Design a multi-objective reward and punishment function that integrates four sub-objectives: coverage improvement, defect triggering, perturbation control, and generation efficiency. The weight coefficients are dynamically configured according to the different gradient regions selected by the mutation action.

[0012] Using the state space as input and the action space as output, a policy network is constructed, and iterative training is performed using a deep reinforcement learning algorithm combined with an experience replay mechanism, so that the agent can gradually learn the optimal mutation policy that maximizes the cumulative reward in the interaction with the environment.

[0013] Within a preset time budget, the following steps are performed iteratively for each seed image in the seed test input set: the policy network selects an action based on the current state, performs mutation on the seed image in a specified gradient region, calculates the immediate reward and updates the state, and stores the interaction experience of each step in the replay pool for continuous updating of the policy network; if the current seed image triggers model misclassification, the mutated image is recorded and reset to the next seed image; if the perturbation threshold or the maximum number of steps is reached, the image is directly reset to the next seed image.

[0014] All mutated images that triggered misclassification by the model are aggregated to generate the final test input set.

[0015] Furthermore, the process involves retaining correctly classified sample images based on the predicted confidence distribution, dynamically determining the confidence screening interval according to the category, and selecting a fixed number of sample images from each category as seed images to obtain a seed test input set that evenly covers all categories, including:

[0016] The entire original test dataset is input into the deep neural network. For each sample image, the prediction confidence of the deep neural network is calculated, and only the sample images that are correctly classified are retained.

[0017] For the retained sample images, grouping is performed by category, and the screening interval [L, U] is dynamically determined according to the standard deviation of the confidence of each category. If the standard deviation is less than 0.15, L = max(0.5, μ-3σ) and U = min(0.95, μ+3σ), where μ represents the mean value of the confidence of each category, and σ represents the standard deviation of the confidence of each category; otherwise, L = max(0.5, Q1+0.1) and U = min(0.95, Q3), where Q1 represents the lower quartile of the confidence of each category, and Q3 represents the upper quartile of the confidence of each category;

[0018] Within the screening interval, a fixed number of sample images are selected from each category as seed images, so as to obtain a seed test input set that evenly covers all categories.

[0019] Further, the neuron activation state includes: the proportion of inactivated neurons and a high-potential neuron mask; the high-potential neurons are neurons whose activation values are 0.9 to 1.2 times the average value of the original test dataset; the proportion of inactivated neurons is obtained through classified statistics by gradient regions;

[0020] The image variation state includes: the L∞ perturbation amplitude of the whole image and each gradient region, normalized values of brightness, contrast and saturation, and one-hot codes of the latest 3 actions;

[0021] The model decision state includes: original category confidence p_org, Top-1 error category confidence p_err, original category confidence variation Δp_org, Top-1 error category confidence variation Δp_err, error flag and perturbation out-of-bound flag;

[0022] The gradient region features include: average gradient importance of three types of gradient regions, gradient weight of the current gradient region, and gradient variation trend; the three types of gradient regions are divided according to the pixel importance heat map, including a core region where G(m,n)≥0.25, a secondary region where 0.05<G(m,n)<0.25, and an irrelevant region where G(m,n)≤0.05.

[0023] Further, a multi-objective reward and penalty function integrating four sub-objectives of coverage improvement, defect triggering, perturbation control and generation efficiency is designed, which is expressed as:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] In the formula, α=0.6, β=1.2, γ=1.0, δ=0.9, This represents the gradient weights of the current gradient region, obtained based on the type of gradient region actually bound to the current mutation action. This indicates the step count penalty. This indicates the number of newly activated uncovered neurons. This indicates the number of newly activated high-potential neurons. This represents the confidence level of the model's prediction for the original category at step t. Represents global pixels Disturbance amplitude Indicates the gradient region affected by the current action. Disturbance amplitude This indicates the number of mutation iterations that have been performed on the current seed image. This indicates the cumulative number of times irrelevant regions have been selected.

[0030] Beneficial Effects: This invention obtains seed test inputs that evenly cover all categories by screening and classifying test set samples based on confidence distribution; it constructs a multi-dimensional fused state space based on the activation of neurons within the deep neural network, image mutation states, model decision feedback, and gradient region features; it combines gradient regions and mutation operations into a discrete action space, and uses a reinforcement learning agent to select the optimal mutation strategy; it designs a multi-objective reward and punishment function that integrates coverage improvement, defect triggering, perturbation control, and generation efficiency; it uses DDQN or Dueling DQN policy networks for iterative training and experience replay; and it iteratively executes action selection, image mutation, reward calculation, and state update within the time budget to generate a test input set that can efficiently trigger model defects while maintaining visual plausibility. Compared with existing technologies, this invention has the following advantages:

[0031] (1) This invention uses reinforcement learning to drive dynamic mutation selection, transforming test input generation into interactive decision-making between the agent and the environment, adaptively selecting the optimal mutation operator and the region of action, thus overcoming the shortcomings of static and blind mutation strategies in traditional methods.

[0032] (2) The multi-dimensional fusion state space and multi-objective trade-off reward function constructed in this invention integrate neuron activation, model decision-making, and gradient features, while balancing defect triggering, coverage improvement, reasonable perturbation and efficient generation, thus achieving a unity of accuracy and efficiency in test input generation;

[0033] (3) This invention adopts DDQN and Dueling DQN architectures respectively and combines them with priority experience replay to solve the problem of policy learning stability in high-dimensional state space. The generated test input has a higher defect trigger rate and a smaller perturbation amplitude (experiments show that the L∞ norm is reduced by more than 65% compared with the baseline method), and has good generalization on different datasets and models. Attached Figure Description

[0034] Figure 1 This is a framework diagram of a deep neural network test input generation method based on reinforcement learning and dynamic mutation selection proposed in this invention;

[0035] Figure 2 A comparison chart showing the neuron coverage of different methods under different dataset model combinations. Detailed Implementation

[0036] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] This invention proposes a method for generating test inputs using a deep neural network based on reinforcement learning and dynamic mutation selection, which mainly includes the following steps:

[0038] Step 1: Input the original test image set into the deep neural network, retain the correctly classified sample images according to the predicted confidence distribution, dynamically determine the confidence screening interval according to the category, select a fixed number of sample images from each category as seed images, and obtain a seed test input set that evenly covers all categories, providing initial samples for subsequent iterative mutation.

[0039] One implementation method includes:

[0040] The entire original test dataset is input into a deep neural network (DNN). Based on each test set sample image, the prediction confidence of the deep neural network (DNN) is calculated, and only the test set sample images that are correctly classified are retained.

[0041] For the retained test set samples, group them by category. Dynamically determine the screening interval [L, U] based on the standard deviation of the confidence score for each category. If the standard deviation is <0.15, it is determined to be an easy-to-classify category, and L = max(0.5, μ-3σ) and U = min(0.95, μ+3σ), where μ represents the mean confidence score of each category and σ represents the standard deviation of the confidence score of each category. Otherwise, it is determined to be a difficult-to-classify category, and L = max(0.5, Q1+0.1) and U = min(0.95, Q3), where Q1 represents the lower quartile and Q3 represents the upper quartile. When there are insufficient candidates, the interval is expanded by 0.05 at both ends. A fixed number of seeds are taken for each category to achieve category balance.

[0042] Step 2: Under the Markov decision process framework, the test input generation is modeled as an interactive decision-making process between an agent and the environment. Based on the seed image or the current mutated image in the iteration, the neuron activation state inside the deep neural network, the mutation state of the image itself, the model decision state, and the gradient region features are acquired to construct a state space, where the gradient region is a gradient-based pixel importance region. The neuron activation state includes the proportion of inactivated neurons (classified and counted by gradient region) and a high-potential neuron mask (with activation values at 0.9~1.2 times the average value of the training set);

[0043] The image mutation state includes the global and per-region L∞ perturbation amplitude, the normalized values of brightness / contrast / saturation, and the one-hot encoding of the latest 3 actions;

[0044] The model decision state includes the original class confidence p_org, the Top-1 error class confidence p_err, the confidence changes Δp_org and Δp_err, as well as an error flag and a perturbation out-of-bounds flag;

[0045] The gradient region feature state includes the average gradient importance G_core, G_sec, G_bg of the three types of regions, the gradient weight w_grad of the currently selected region (2.0 for the core region, 0.9 for the secondary region, 0.4 for the irrelevant region), and the gradient change trends ΔG_core, ΔG_sec, ΔG_bg. The three gradient regions are divided according to the pixel importance heat map G: the core region satisfies G(m,n)≥0.25, the secondary region satisfies 0.05<G(m,n)<0.25, and the irrelevant region satisfies G(m,n)≤0.05.

[0046] Step 3: At the same time, the gradient regions and predefined mutation operations are combined into a discrete action space, so that each action of the agent specifies both the mutation region and the type of mutation operation; this provides a perception and action basis for reinforcement learning decision-making.

[0047] As an implementation, the three types of gradient regions are combined with 16 mutation operations to form 48 discrete actions. The 16 mutation operations include basic adjustments (6 types in total: ±5% for each of brightness, contrast and saturation), color adjustments (6 types in total: ±5% for each of the three RGB channels), and image correction (4 types in total: Gamma increase / decrease, histogram equalization, and white balance).

[0048] The specific implementation manner for the agent to select the optimal mutation strategy is as follows:

[0049] (1) Policy network architecture

[0050] The agent employs either DDQN or Dueling DQN architectures as its policy network. The policy network takes the current state vector as input, which integrates multi-dimensional information such as neuron activation states, image mutation states, model decision states, and gradient region features. The network maps the input features layer by layer through a three-layer fully connected structure (128-64-48). The network output layer contains 48 neurons, each corresponding to a Q-value estimate for an action (in the DDQN architecture, the Q-value is directly output; in the Dueling DQN architecture, the 48-dimensional Q-value is obtained by fusing the state value branch and the action advantage branch). Each dimension corresponds to an expected cumulative reward estimate for a discrete action (i.e., a combination of a gradient region and a mutation operation).

[0051] (ii) Action selection mechanism

[0052] The agent employs an ε-greedy exploration strategy for action selection. At each decision step, the agent randomly selects an action with probability ε (exploration) and selects the action with the highest current Q value with probability 1-ε (exploitation). The initial exploration rate is set to 0.95, which linearly decays to 0.05 as the number of training steps increases, thereby achieving a dynamic balance between extensively exploring various variant strategies in the early stage of training and focusing on the optimal strategy in the later stage of training.

[0053] (III) Strategy Optimization and Training

[0054] The agent accumulates experience through continuous interaction with the environment (i.e., the DNN model under test and the current test image). After each action, the environment returns a reward value (calculated by a multi-objective reward function), and the agent stores the sequence "current state - selected action - reward value - next state" as an experience in the experience replay pool. During training, small batches of experience are randomly sampled from the experience replay pool, and the temporal difference (TD) error is minimized using gradient descent to update the parameters of the policy network. The target network synchronizes its parameters every fixed number of steps to stabilize the training process. Through repeated iterations, the agent gradually learns the optimal mutation strategy that maximizes long-term cumulative rewards, i.e., selecting the mutation regions and mutation operations that are most likely to trigger model defects, improve coverage, and have reasonable perturbations under different states.

[0055] (iv) Dynamic adaptation and generalization

[0056] Because the input state of the policy network includes gradient region features (such as the average gradient importance of the region, gradient change trends, etc.), the agent can dynamically adjust its action selection based on the gradient distribution of the current image, without needing to redesign mutation strategies for different datasets or models. This mechanism enables the agent to have good generalization ability and adapt to deep neural network models of different structures and sizes.

[0057] Step 4: Design a multi-objective reward and punishment function that integrates four sub-objectives: coverage improvement, defect triggering, perturbation control, and generation efficiency. The weight coefficients are dynamically configured based on the gradient region selected by the mutation action, ensuring the agent receives stronger positive incentives in core regions and stricter perturbation constraints in irrelevant regions, thereby guiding the policy towards high-value regions. This is represented as:

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] In the formula, α=0.6, β=1.2, γ=1.0, δ=0.9, and the meanings of other symbols are shown in Table 2.

[0064] Step 5: Using the state space as input and the action space as output, construct a policy network using DDQN or Dueling DQN, and use deep reinforcement learning algorithms combined with an experience replay mechanism for iterative training, so that the agent can gradually learn the optimal mutation policy that maximizes the cumulative reward in the interaction with the environment.

[0065] As one implementation, network hyperparameters, a priority experience replay pool, and an ε-greedy exploration strategy are configured. Iterative training is performed using a DDQN or Dueling DQN policy network. The optimal mutation strategy is learned through priority experience replay, and the policy network is iteratively updated based on state-action-reward samples at each step. The network hyperparameters include: 186 input feature dimensions and 48 output action dimensions; a three-layer fully connected network (128-64-48), ReLU activation, and Dropout=0.2; an experience pool capacity of 20000, with weights allocated by region for sample weighting (core region 1.2, secondary region 0.8, irrelevant region 0.4); an initial exploration rate of 0.95 for the ε-greedy exploration strategy, linearly decaying to 0.05 after 20000 steps; a discount factor γ=0.98; parameter synchronization of the target network every 50 steps; and an optimizer learning rate of 5e−5.

[0066] In this embodiment of the invention, DDQN adopts a dual-network architecture (action selection network + target network). DuelingDQN splits the network into a state value branch and an action advantage branch, as shown below:

[0067] )

[0068] Please refer to Table 3 for the meaning of each parameter in the formula.

[0069] The section on "Iterative training and experience playback using policy networks" needs to be explained in conjunction with the state space in step 2, the action space in step 3, and the reward / penalty function in step 4. Specifically, it includes the following four core components:

[0070] (I) Policy Network Architecture

[0071] The policy network employs either DDQN or Dueling DQN architecture. The network input dimension is 186 dimensions (defined by the state space in step 2: 176 dimensions for neuron activation + 3 dimensions for image mutation + 3 dimensions for model decision + 7 dimensions for gradient regions), and the output dimension is 48 dimensions (defined by the action space in step 3: 3 types of gradient regions × 16 mutation operations). The network's hidden layers are a three-layer fully connected structure (128-64-48), using the ReLU activation function, and Dropout (with a dropout rate of 0.2) is introduced to suppress overfitting.

[0072] (II) The Balance Mechanism Between Exploration and Utilization

[0073] The agent employs an ε-greedy strategy for action selection: it randomly selects actions with probability ε (exploring unknown strategies) and selects the action with the highest current Q value with probability 1-ε (utilizing learned knowledge). The initial exploration rate is set to 0.95, which linearly decays to 0.05 within 20,000 steps, achieving a dynamic balance between extensive exploration in the early stages of training and focusing on the optimal strategy in the later stages.

[0074] (III) Experience storage and priority playback

[0075] After each action is performed, the environment returns an immediate reward Rtotal calculated by the reward / penalty function in step 4, and the agent internalizes the interaction experience. Store the data in the experience replay pool (capacity 20,000). During training, a priority experience replay mechanism is adopted, and sampling weights are allocated according to the gradient region corresponding to the action (core region sample weight 1.2, secondary region 0.8, irrelevant region 0.4), so that effective experience in high-value regions can get more training opportunities and improve learning efficiency.

[0076] (iv) Iterative update of network parameters

[0077] Mini-batch experiences (batch size = 128) are sampled from the experience replay pool, and the temporal difference (TD) error is minimized using gradient descent to update the policy network parameters. Specifically, the target Q value is... (Under DDQN, action selection and value assessment are separated), where the discount factor γ = 0.98. The loss function is mean squared error (MSE), the optimizer is Adam, and the learning rate is set to... The target network synchronizes the policy network parameters every 50 steps to stabilize the training process. Through these iterations, the agent gradually learns the optimal mutation policy that maximizes long-term cumulative rewards.

[0078] Step 6: Within a preset time budget, iteratively execute the following steps for each seed image in the seed test input set: select an action based on the current state using the policy network, perform mutation on the seed image in a specified gradient region, calculate the immediate reward and update the state, and simultaneously store the interaction experience of each step in the replay pool for continuous updating of the policy network; if the current seed image triggers model misclassification, record the mutated image and reset to the next seed image; if the perturbation threshold or maximum number of steps is reached, directly reset to the next seed image; summarize all mutated images that trigger model misclassification to generate the final test input set.

[0079] Example 1: MNIST dataset + LeNet-5 model

[0080] Experimental environment configuration: This embodiment was executed on the Ubuntu 20.04.1 LTS operating system, with hardware configuration of NVIDIA RTX A6000 GPU (48GB VRAM), Intel Xeon W-2255 10-core CPU @ 3.70GHz, and 64GB of physical memory. The software environment was Python 3.9.25, and the deep learning frameworks were TensorFlow 2.20.0 and Keras 3.10.0.

[0081] Dataset and Model: The MNIST handwritten digit dataset was used, containing 70,000 grayscale images (28×28 pixels) across 10 categories (digits 0-9). 60,000 images were used for training, and 10,000 were used for testing. The tested model was a LeNet-5 convolutional neural network with 11 trainable layers and 236 neurons, achieving a classification accuracy of 99.94% on the test set.

[0082] Seed Selection: 10,000 test images are input into the LeNet-5 model to obtain the prediction confidence score for each image in each digit category. Only samples correctly classified by the model (accuracy 99.94%) are retained, grouped by digit category 0-9. For each group, the mean μ, standard deviation σ, lower quartile Q1, and upper quartile Q3 of the confidence score are calculated. If σ < 0.15 (easy-to-classify category), the confidence score selection interval is set to L = max(0.5, μ - 3σ), U = min(0.95, μ + 3σ); if σ ≥ 0.15 (difficult-to-classify category), the interval is set to L = max(0.5, Q1 + 0.1), U = min(0.95, Q3). From each group of candidate samples, the top 20 samples are selected in ascending order of confidence score as seeds, resulting in 200 seeds for input across 10 categories, achieving balanced coverage among categories.

[0083] Algorithm Selection and Network Configuration: Dueling DQN is adopted as the policy network architecture. The network input dimension is 186 (176-dimensional neuron activation + 3-dimensional image mutation + 3-dimensional model decision + 7-dimensional gradient region), and the output dimension is 48 (3 types of gradient regions × 16 mutation operations). The hidden layers are three fully connected layers with dimensions of 128 and 64 respectively. ReLU is used as the activation function, and the dropout rate is set to 0.2 to suppress overfitting. The experience replay pool capacity is set to 20,000, and a priority sampling mechanism is adopted, allocating sampling weights according to the gradient region type corresponding to the action: 1.2 for core region actions, 0.8 for secondary regions, and 0.4 for irrelevant regions. The exploration strategy adopts ε-greedy, with an initial exploration rate ε = 0.95, which linearly decays to 0.05 over 20,000 training steps. The discount factor γ = 0.98, and the policy network parameters are synchronized every 50 steps in the target network. The optimizer is Adam, and the learning rate is set to 5 × 10⁻⁻⁻⁶. 5 The maximum number of iterations per seed is set to 50.

[0084] Constraints: The total runtime is set to 30 minutes, and the maximum number of iterations per seed is 50. Termination conditions are: if the mutated image triggers model misclassification (the predicted label is inconsistent with the true label), or the L∞ perturbation amplitude in any gradient region exceeds 0.06, or the global L∞ perturbation amplitude exceeds 0.06, then the mutation iteration of the current seed is terminated, adversarial examples are recorded, and the image is reset to the original seed image.

[0085] Experimental results: The average L∞ norm between the generated test input and the original image is 0.1893 (under the NC criterion), which is significantly lower than 0.7518 for the random mutation method and 0.3047 for DLFuzz.

[0086] The NC neuron coverage rate reached 43%, meaning that approximately 101 out of 236 neurons were successfully activated and covered.

[0087] Under the SNAC coverage criterion, the error diversity reached 48 types, meaning that 48 different misclassification types were successfully triggered (such as "7→1", "5→3" and other incorrect prediction pairs), which is higher than DeepXplore's 31 types and ADAPT's 42 types.

[0088] Within a 30-minute time budget, a total of 402 valid adversarial examples were generated. The comparison results of different seed selection strategies under this embodiment's model are detailed in the MNIST-LeNet5 section of Table 1.

[0089] Example 2: CIFAR-10 dataset + VGG16 model

[0090] Dataset and Model: The CIFAR-10 color image dataset was used, containing 60,000 RGB images (32×32 pixels) across 10 categories. 50,000 images were used for training, and 10,000 for testing. The tested model was a VGG16 convolutional neural network with 23 trainable layers and 5,770 neurons, achieving a classification accuracy of 86.48% on the test set.

[0091] Algorithm selection: This embodiment adopts the DDQN architecture with a dual-network design (separate action selection network and target network), and the remaining hyperparameter configurations are consistent with those in embodiment 1.

[0092] Experimental results:

[0093] The average L∞ norm between the generated test input and the original image is 0.1603 (under the NC criterion), which is about 60% lower on average compared to the baseline method.

[0094] Under the condition of 5% test sample screening (i.e., selecting the top 5% from the sorted test inputs for evaluation), the fault detection rate (FDR) of the method of this invention reaches 0.96, which is 7% to 9% higher than the baseline methods such as DeepGini (0.93), MaxP (0.93) and ATS (0.91).

[0095] The TKNC (Top-k neuron coverage) reached 48%, indicating that the method can accurately trigger core neurons that are strongly correlated with model decisions.

[0096] Example 3: CIFAR-100 dataset + ResNet32 model

[0097] Dataset and Model: The CIFAR-100 fine-grained image classification dataset was used, containing 60,000 RGB images of 32×32 pixels across 100 categories. The tested model was a ResNet32 residual network with 114 trainable layers and 1,332 neurons, achieving a test set accuracy of 69.22%.

[0098] Experimental results:

[0099] Under the TKNC coverage criterion, the error diversity reaches 157 types (the theoretical maximum is 100 original categories × 99 error categories = 9900 types, and 157 different misclassification types are actually triggered), proving that this method can still effectively trigger diverse model defects in fine-grained multi-classification scenarios.

[0100] After retraining the ResNet32 model by adding the top 10% of the sorted test inputs to the original training set, the model's accuracy on the validation set improved by 1.32%, which is higher than the 1.20% of the Random method and the 1.22% of the DeepGini method. This indicates that the test inputs selected by this method have higher practical value for improving model performance.

[0101] Neuron coverage reached 152 under the KMNC (k=1000) criterion, an average improvement of about 15% over the baseline method.

[0102] Example 4: CIFAR-10 dataset + ResNet20 model

[0103] Dataset and Model: The CIFAR-10 dataset was used. The tested model was a ResNet20 residual network with 72 layers and 794 neurons. The test set accuracy was 79.80%.

[0104] Experimental results:

[0105] The Dueling DQN variant of the method of this invention achieves a success rate of 20.00 for the number of seeds successfully attacked (i.e., all 20 seeds successfully trigger misclassification), which is significantly higher than the 13.67 of random selection and the 3.00 of MaxP.

[0106] The average seed category coverage is 10.00, achieving full coverage of all 10 categories, while the Random method only covers 6.67 categories.

[0107] Under the KMNC criterion, Dueling DQN achieves a coverage rate of 48%, which is better than DDQN's 43%, demonstrating that Dueling DQN has superior policy learning stability in high-dimensional state spaces by separating state value and action advantage branches.

[0108] Table 1. Effectiveness Analysis of Seed Selection Strategy

[0109]

[0110] Table 2. Explanation of Parameter Symbols for Multi-Objective Reward and Punishment Functions

[0111]

[0112] Table 3. Explanation of Parameter Symbols for Dueling DQN Q-Value Function

[0113]

Claims

1. A method for generating test inputs using a deep neural network based on reinforcement learning and dynamic mutation selection, characterized in that: Includes the following steps: The original test image set is input into the deep neural network. The correctly classified sample images are retained according to the predicted confidence distribution. The confidence screening interval is dynamically determined according to the category. A fixed number of sample images are selected from each category as seed images to obtain a seed test input set that evenly covers all categories. Based on the seed image or the current mutated image in the iteration, the activation state of neurons inside the deep neural network, the mutation state of the image itself, the model decision state, and gradient region features are obtained to construct a state space. The gradient region is a pixel importance region based on gradient. At the same time, the gradient region is combined with predefined mutation operations to form a discrete action space, so that each action of the agent specifies both the mutation region and the mutation operation type. Design a multi-objective reward and punishment function that integrates four sub-objectives: coverage improvement, defect triggering, perturbation control, and generation efficiency. The weight coefficients are dynamically configured according to the different gradient regions selected by the mutation action. Using the state space as input and the action space as output, a policy network is constructed, and iterative training is performed using a deep reinforcement learning algorithm combined with an experience replay mechanism, so that the agent can gradually learn the optimal mutation policy that maximizes the cumulative reward in the interaction with the environment. Within a preset time budget, the following steps are performed iteratively for each seed image in the seed test input set: selecting an action based on the current state by the policy network, performing mutation on the seed image in a specified gradient region, calculating an immediate reward and updating the state, while storing the interaction experience of each step into the replay pool for continuous updating of the policy network. If the current seed image triggers a misclassification by the model, the mutated image is recorded and the model is reset to the next seed image. If the perturbation threshold or the maximum number of steps is reached, the model is directly reset to the next seed image. All mutated images that triggered misclassification by the model are aggregated to generate the final test input set.

2. The method for generating test inputs based on deep neural networks using reinforcement learning and dynamic mutation selection according to claim 1, characterized in that: The method involves retaining correctly classified sample images based on the predicted confidence distribution, dynamically determining the confidence screening interval according to the category, and selecting a fixed number of sample images from each category as seed images to obtain a seed test input set that evenly covers all categories, including: The entire original test dataset is input into the deep neural network. For each sample image, the prediction confidence of the deep neural network is calculated, and only the sample images that are correctly classified are retained. For the retained sample images, group them by category, and dynamically determine the screening interval [L, U] based on the standard deviation of the confidence score for each category. If the standard deviation is <0.15, take L=max(0.5, μ-3σ) and U=min(0.95, μ+3σ), where μ represents the mean confidence score for each category and σ represents the standard deviation of the confidence score for each category; otherwise, take L=max(0.5, Q1+0.1) and U=min(0.95, Q3), where Q1 represents the lower quartile of the confidence score for each category and Q3 represents the upper quartile of the confidence score for each category. Within the filtering interval, a fixed number of sample images are selected from each category as seed images to obtain a seed test input set that evenly covers all categories.

3. The method for generating test inputs based on deep neural networks using reinforcement learning and dynamic mutation selection according to claim 1, characterized in that: The neuron activation state includes: the proportion of inactivated neurons and the high-potential neuron mask; the high-potential neurons are neurons whose activation values are 0.9 to 1.2 times the average value of the original test dataset; the proportion of inactivated neurons is obtained by classification and statistics according to gradient regions; The image variation state includes: the L∞ perturbation amplitude of the whole and each gradient region, the normalized values of brightness, contrast and saturation, and the one-hot encoding of the latest 3 actions; The model decision state includes: the original category confidence p_org, the Top-1 error category confidence p_err, the original category confidence change Δp_org, the Top-1 error category confidence change Δp_err, an error flag and a perturbation out-of-bound flag; The gradient region features include: the average gradient importance of the three types of gradient regions, the gradient weight of the current gradient region, and the gradient change trend; the three types of gradient regions are divided according to the pixel importance heat map, including the core region where G(m,n)≥0.25, the secondary region where 0.05<G(m,n)<0.25, and the irrelevant region where G(m,n)≤0.

05.

4. The method for generating test inputs for deep neural networks based on reinforcement learning and dynamic mutation selection according to claim 1, characterized in that: A multi-objective reward and penalty function integrating four sub-objectives of coverage improvement, defect triggering, perturbation control and generation efficiency is designed, which is expressed as: In the formula, α=0.6, β=1.2, γ=1.0, δ=0.9, This represents the gradient weights of the current gradient region, obtained based on the type of gradient region actually bound to the current mutation action. This indicates the step count penalty. This indicates the number of newly activated uncovered neurons. This indicates the number of newly activated high-potential neurons. This represents the confidence level of the model's prediction for the original category at step t. Represents global pixels Disturbance amplitude Indicates the gradient region affected by the current action. Disturbance amplitude This indicates the number of mutation iterations that have been performed on the current seed image. This indicates the cumulative number of times irrelevant regions have been selected.

Images