Deep learning test sample generation method using reinforcement learning optimization genetic algorithm

Abstract

The invention relates to the technical field of deep learning testing, in particular to a deep learning test sample generation method utilizing a reinforcement learning optimization genetic algorithm, which comprises the following steps: constructing different types of seed data sets based on an image data set, and determining and initializing an initial population; obtaining a deep learning model, executing a genetic algorithm, and optimizing the initialized initial population; combining reinforcement learning to dynamically optimize genetic algorithm parameters; and performing iterative optimization on the optimized initial population through the optimized genetic algorithm to generate a new population, screening a data sample with a prediction error, and obtaining a test suite. The deep learning model and the genetic algorithm are integrated, the genetic algorithm is executed, reinforcement learning is introduced to dynamically optimize operation parameters of the genetic algorithm, the problems of local optimum and parameter sensitivity are solved, generation of a test suite with smaller distribution difference is promoted, the convergence speed of fitness can be effectively improved, and the robustness of the deep learning model is improved.

Description

Technical Field

[0001] This invention relates to the field of deep learning testing technology, and more specifically to a method for generating deep learning test samples using reinforcement learning to optimize genetic algorithms. Background Technology

[0002] In recent years, with the in-depth research of deep learning, it has been widely used in many fields such as autonomous driving, medical diagnosis, facial recognition and speech recognition. However, deep learning models generally have poor robustness and are easily affected by adversarial attacks. Therefore, in order to ensure the safety of the scenario, it is necessary to test the deep learning model first.

[0003] In existing technologies, test samples generated by adversarial attacks and coverage-guided testing methods have been found to have poor perceptual quality, with most samples being deemed invalid and difficult for the human eye to recognize. Furthermore, existing technologies may suffer from insufficient diversity in test samples due to distribution similarity, making such samples easily detectable by adversarial defense methods and failing to truly assess the robustness of deep learning models. Meanwhile, research shows that the increase in neuron coverage is not directly related to the robustness of deep learning models.

[0004] Due to the limitations of existing technologies, recent research has focused on distributed perception testing, emphasizing the distribution of test samples. This addresses both the perception quality of test samples and the generation of test samples with different distributions, thus simultaneously solving the problems of test sample effectiveness and diversity. Genetic algorithms are typically used to iteratively generate test samples. However, existing genetic algorithms have certain drawbacks. The crossover and mutation probabilities are usually set based on experiments and experience, and fixed settings cannot be adjusted and optimized according to the generation state. Furthermore, this search mechanism limits test cases to local optima, making it difficult to expand to other search areas. Summary of the Invention

[0005] To address the technical problems of low efficiency and susceptibility to local optima in existing genetic algorithms, this invention aims to provide a deep learning test sample generation method that utilizes reinforcement learning to optimize genetic algorithms. The specific technical solution adopted is as follows:

[0006] Obtain the image dataset, construct seed datasets for different categories based on the image dataset, determine the initial population and initialize it;

[0007] Obtain the deep learning model, execute the genetic algorithm, and optimize the initial population after initialization;

[0008] Combine reinforcement learning to dynamically optimize the parameters of the genetic algorithm;

[0009] The optimized initial population is iteratively optimized using the optimized genetic algorithm to generate a new population, and data samples with prediction errors are filtered out to obtain a test suite.

[0010] Preferably, seed datasets of different categories are constructed based on the image dataset to determine and initialize the initial population, including:

[0011] The image dataset is divided into multiple categories, and multiple data samples are selected from each category to construct a seed dataset to form individuals;

[0012] Variants are generated based on individual application mutation strategies, and the combined variants and corresponding individuals form an initial population.

[0013] Preferably, the mutation strategy includes any image feature transformation strategy such as rotation, translation, scaling, or local blurring.

[0014] Preferably, the process includes obtaining a deep learning model, executing a genetic algorithm, and optimizing the initial population after initialization, including:

[0015] The categories include source category and target category; each data sample in the initial population after initialization is input into the deep learning model, the logits layer output is extracted to obtain the logits vector, the corresponding logits set is constructed based on the source category and the target category, and the MMD value is determined to be the fitness;

[0016] The individuals in the initial population after initialization are divided into two groups, and the parents are selected based on fitness.

[0017] Preset the crossover probability, and use a uniform crossover strategy based on the parent generation and the crossover probability to obtain the offspring;

[0018] Analyze the offspring to determine if they have mutated. If so, generate new individuals to replace the corresponding individuals in the initial population after initialization.

[0019] Preferably, the parent generation is obtained by screening based on fitness, including:

[0020] Within each group, fitness is sorted in ascending order, and the individual with the lowest fitness in each group is retained. The remaining individuals are selected through a tournament selection mechanism to generate two sets of fixed-length data sequences, i.e., two sets of serial numbers, and the parent generation is obtained based on the serial numbers.

[0021] From the two sets of data sequences, the parent individuals corresponding to the first sequence number are selected sequentially and crossover mutation is performed to generate new individuals until the number of individuals in each group in the corresponding initial population is satisfied.

[0022] Preferably, a preset crossover probability is used to obtain offspring by employing a uniform crossover strategy based on the parent generation and the crossover probability, including:

[0023] For each data sample in the parent generation, generate a random number and compare the random number with the crossover probability;

[0024] In the two sets of data sequences, the individuals corresponding to the first index are designated as Parent Generation 1 and Parent Generation 2, respectively. When the random number is less than the crossover probability, the data sample at the corresponding position is selected from Parent Generation 1; when the random number is greater than or equal to the crossover probability, the data sample at the corresponding position is selected from Parent Generation 2.

[0025] All selected data samples are combined in the order of the original data samples to form offspring.

[0026] Preferably, the offspring are analyzed to determine if mutation has occurred. If so, a new individual is generated to replace the corresponding individual in the initial population after initialization, including:

[0027] Preset mutation probability, and generate random numbers for each data sample in the offspring;

[0028] When the random number is less than the mutation probability, the mutation mechanism is triggered. The mutation strategy performs feature transformation on the data sample corresponding to the random number to generate a new data sample, and the new data sample replaces the corresponding data sample in the individual.

[0029] Preferably, the genetic algorithm parameters are dynamically optimized using reinforcement learning, including:

[0030] Record the initial population state, and determine the action space by selecting the crossover probability and mutation probability;

[0031] The crossover and mutation probabilities selected in the action space are used to generate a new population and update the initial population state.

[0032] The reward value is obtained by analyzing the updated initial population state in conjunction with the initial population state before the update.

[0033] A Q-table is formed by storing states and actions, and the Q-table is updated based on the executed actions and reward values.

[0034] pass - A greedy strategy balances search and exploitation, selecting an action from the updated Q table based on the current state as the next action to be executed.

[0035] Preferably, by - A greedy strategy balances search and exploitation, selecting an action from the updated Q table based on the current state as the next action to be executed. Specifically:

[0036] Based on exploration rate Generate random numbers when the random number is greater than or equal to the exploration rate. Randomly select an action from the action space; when the random number is less than the exploration rate... Query the Q table corresponding to the current state and select the action corresponding to the maximum reward.

[0037] The present invention has the following beneficial effects:

[0038] This application integrates deep learning models and genetic algorithms. By executing the genetic algorithm and introducing reinforcement learning to dynamically optimize its operational parameters, it overcomes the problems of local optima and parameter sensitivity. This allows the algorithm to dynamically select the most beneficial action for the next step based on the current state during iteration. By dynamically adjusting the crossover and mutation probabilities, it promotes the generation of test suites with smaller distributional differences, i.e., test samples with distributional diversity, which can effectively improve the convergence speed of fitness. At the same time, it generates erroneous samples that are closer to the decision boundary of the target class and are difficult to detect. In practical applications, the deep learning model is retrained using test suites with different distributions, which improves the robustness of the deep learning model. Attached Figure Description

[0039] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the implementation of a method for generating deep learning test samples using reinforcement learning to optimize genetic algorithms, according to an embodiment of the present invention.

[0041] Figure 2 The flowchart illustrates the steps of a method for generating deep learning test samples using reinforcement learning to optimize a genetic algorithm, as provided in an embodiment of the present invention.

[0042] Figure 3 The figure shows a comparison between a deep learning test sample generation method using reinforcement learning to optimize genetic algorithms and a traditional method, as provided in an embodiment of the present invention. Detailed Implementation

[0043] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a deep learning test sample generation method using reinforcement learning to optimize genetic algorithms according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0045] The following description, in conjunction with the accompanying drawings, details a specific scheme for a method of generating deep learning test samples using reinforcement learning to optimize genetic algorithms, provided by this invention.

[0046] To better illustrate this, based on the limitations of existing deep learning models, the main problems they face are: first, the generated test samples have poor perceptual quality, a small effective quantity, are difficult for the human eye to recognize, and are easily detected; second, the generated test samples have a uniform distribution, resulting in redundancy; and third, the parameters in the genetic algorithm are set by the tester based on experience, which cannot expand the search area and easily leads to local optima. Therefore, this application proposes a deep learning test sample generation method that utilizes reinforcement learning to optimize the genetic algorithm, thereby improving the convergence speed of the fitness, achieving a better fitness, and solving the problem of local optima.

[0047] Please combine Figure 1 and Figure 2 This document illustrates an implementation flowchart and step flowchart of a deep learning test sample generation method utilizing reinforcement learning to optimize genetic algorithms, according to an embodiment of the present invention. The method includes:

[0048] Step S1: Obtain the image dataset, construct seed datasets for different categories based on the image dataset, determine the initial population and initialize it;

[0049] Step S2: Obtain the deep learning model, execute the genetic algorithm, and optimize the initial population after initialization;

[0050] Step S3: Dynamically optimize the genetic algorithm parameters by combining reinforcement learning;

[0051] Step S4: Iteratively optimize the optimized initial population using the optimized genetic algorithm to generate a new population, filter out data samples with prediction errors, and obtain the test kit.

[0052] To better illustrate, a genetic algorithm is a highly parallel, randomized search and optimization algorithm that simulates the biological evolution process in nature. It iteratively optimizes the solution to a problem by simulating mechanisms such as natural selection, heredity, and mutation in biological populations, and eventually finds or approximates the global optimum. Its core idea is to encode the potential solution of a problem into chromosomes and continuously generate new offspring through operations such as selection, crossover, and mutation, gradually improving the overall fitness of the population until the termination condition is met.

[0053] Deep learning models simulate the neural network structure of the human brain, constructing multi-layered neural network models to achieve automatic feature extraction and advanced pattern recognition from complex data. The core idea of ​​reinforcement learning is to continuously learn and optimize strategies through continuous interaction with the environment to maximize specific goals. It does not rely on a large amount of labeled data, but rather explores and utilizes environmental information step by step through trial and error, ultimately forming an efficient problem-solving strategy. Therefore, this application combines existing genetic algorithms with deep learning models and introduces reinforcement learning for improvement, solving the problems of low generation efficiency and getting trapped in local optima in existing genetic algorithms.

[0054] Further, step S1 includes:

[0055] Step S11: Divide the image dataset into multiple categories, and select multiple data samples from each category to construct a seed dataset to form individuals.

[0056] As an optional implementation, the image datasets used include public image datasets such as MNIST (Modified National Institute of Standards and Technology database), FMNIST (Fashion-MNIST dataset), SVHN (Street View House Numbers dataset), and CIFAR (Canadian Institute for Advanced Research dataset). For each image dataset, two different deep learning models were used for independent validation experiments. The generality and effectiveness were verified through a combination of multiple image datasets and multiple models.

[0057] Preferably, in this embodiment, the MNIST dataset is used as an example. It mainly consists of handwritten digit images and has ten categories from 0 to 9. A certain number of data samples are selected from each category as seed samples. For example, 50 or 100 data samples are selected to construct the seed dataset for each category. The seed dataset is used as the chromosome in the genetic algorithm. That is, the seed dataset corresponding to each category is integrated to correspond to each individual, thus constructing the initial basic data for the subsequent genetic algorithm.

[0058] Step S12: Generate variants based on individual application mutation strategies, and combine the variants with the corresponding individuals to form an initial population.

[0059] Furthermore, the mutation strategy includes any image feature transformation strategy such as rotation, translation, scaling, and local blurring. The rotation strategy changes the directional features of the image by rotating it at different angles; the translation strategy adjusts the positional features of the image by moving it horizontally and vertically; the scaling strategy changes the size features of the image by enlarging or shrinking it; and the local blurring strategy affects the detailed features of the image by blurring specific regions of it. Based on the mutation strategy, diverse transformations of image features can be achieved, enriching the diversity of image data and improving the adaptability and robustness of the genetic algorithm.

[0060] The explanation is that, for individuals, a mutation strategy is applied to the data samples in the seed dataset to generate several variants. These variants are then combined with the individuals to form the initial population, increasing the diversity of the initial population. This allows for the exploration of more possibilities and improves the search capability of the genetic algorithm.

[0061] Further, step S2 includes:

[0062] Step S21: The categories include the source category and the target category; input each data sample in the initial population after initialization into the deep learning model, extract the logits layer output to obtain the logits vector, construct the corresponding logits set based on the source category and the target category, and determine the MMD value as fitness.

[0063] It can be explained that the ultimate goal of the genetic algorithm is to generate test samples that are close to the distribution of the target class between each pair of classes. Taking the 0-9 classes in the MNIST dataset as an example, it includes the source class and the target class, and the source class and the target class are not equal. That is, it is assumed that the current class is the source class, and the remaining 9 classes are the target classes.

[0064] To clarify, the deep learning model is trained based on various optimized seed datasets. The logits layer refers to the penultimate layer output, which is directly responsible for generating the unnormalized raw output value, i.e., the layer before softmax. Softmax transforms the output of the logits layer into the probability distribution of the corresponding class, so that it is between 0 and 1.

[0065] Specifically, each data sample in the initial population is input into the deep learning model, and the output of the logits layer is extracted to obtain a feature representation, which is used as the logits vector corresponding to the data sample. This yields the logits set of the chromosome constructed from each seed dataset. Then, the MMD (Maximum Mean Discrepancy) value of the source class and target class logits sets is calculated to evaluate the similarity or difference between them, and the MMD value is saved as the fitness. The corresponding calculation formula is as follows:

[0066]

[0067] in, Indicates source category and target category The fitness of; , Representing the source categories respectively and target category The corresponding number of logits sets; , Representing the source categories respectively and target category The Middle The and the first logits vector; Represents the kernel function.

[0068] It can be explained that fitness is directly used as the optimization objective of the genetic algorithm. That is, by minimizing the MMD value, the feature distribution of the generated samples tends to be consistent with the feature distribution of the seed samples of the target class, thereby providing technical guidance for the direction of sample generation and ensuring the generalization ability and performance between different classes. The MMD value maps the data samples to the Reproducing Kernel Hilbert Space (RKHS) through the kernel function. This is used to map the data samples in the low-dimensional space to the high-dimensional space, which can solve nonlinear problems and improve the generalization ability. The mean embeddings of the two distributions corresponding to the source class and the target class in this space are compared to quantify the distribution difference between the source class and the target class in the RKHS, providing a basis for discrimination in subsequent classification or recognition tasks.

[0069] Step S22: Divide the individuals in the initial population after initialization into two groups, and select the parent generation based on fitness.

[0070] The tournament selection mechanism simulates the tournament competition process in nature, randomly selecting a certain number of individuals from the population, sorting and comparing them according to their fitness values, and finally selecting the individual with the highest fitness as the winner to enter the next generation. This mechanism can maintain the diversity of the population, avoid premature convergence, and balance the selection pressure to a certain extent, so that individuals with higher fitness have a greater probability of being selected, which can accelerate the convergence speed of the algorithm.

[0071] Further, in step S22, the parent generation is obtained by filtering based on fitness, including:

[0072] Step S221: Sort the fitness in ascending order within each group, retain the individual with the lowest fitness in each group, and generate two fixed-length data sequences (i.e., two sets of serial numbers) for the remaining individuals through a tournament selection mechanism each time. The parent generation is obtained by filtering according to the serial number.

[0073] Specifically, all individuals in the initial population are randomly divided into two groups, and each group is sorted in order of fitness from low to high. The lower the fitness, the closer the feature distribution of the corresponding individual is to the feature distribution of the individual corresponding to the target category.

[0074] Prioritize retaining the individual with the lowest fitness within each group; that is, the data sample set with the best fitness does not participate in subsequent crossover and mutation processes and directly enters the next generation population. For the remaining individuals within each group, a tournament selection mechanism is used. At least two sets of fixed-length data sequences are randomly generated each time, and the number of groups can be adjusted according to specific experimental data. Each set of data sequences is a set of indices, and each index corresponds to an individual. Individuals are selected from the two sets of data sequences. In each round of selection, the individual with the best fitness is selected from the two individuals pointed to by the two sets of data sequences as a candidate for parent. If the indices of the two selected individuals point to the same individual, the individual corresponding to the next index in the second set of data sequences is automatically selected as a candidate for parent, to ensure that the selected parent individuals are differentiated.

[0075] Step S222: Select the parent individual corresponding to the first index from the two sets of data sequences and perform crossover mutation to generate new individuals until the number of individuals in each group in the corresponding initial population is satisfied.

[0076] Specifically, crossover mutation refers to generating new individuals by exchanging data samples from the parent generation and some individuals from random mutations. Step S221 is repeated until the group size of each group of individuals in step S22 is met for subsequent processing.

[0077] Step S23: Preset the crossover probability, and use a uniform crossover strategy based on the parent generation and the crossover probability to obtain the offspring.

[0078] As an optional implementation, in this embodiment, the range of values ​​for the crossover probability is set to... .

[0079] Furthermore, step S23 includes:

[0080] Step S231: Generate a random number for each data sample in the parent individuals, and compare the random number with the crossover probability;

[0081] Step S232: Define the individuals corresponding to the first index in the two sets of data sequences as Parent Generation 1 and Parent Generation 2, respectively. When the random number is less than the crossover probability, select the data sample at the corresponding position from Parent Generation 1; when the random number is greater than or equal to the crossover probability, select the data sample at the corresponding position from Parent Generation 2.

[0082] Step S233: Combine all the selected data samples in the order of the original data samples to form offspring.

[0083] Specifically, each data sample in each individual is taken as a gene position, and a uniformly distributed random number in the 0-1 interval is generated for the data sample corresponding to each gene position. In this embodiment, it is assumed that the individuals corresponding to the first sequence number selected from the two sets of data sequences are parent generation 1 and parent generation 2, respectively. If the random number is less than the crossover probability, the data sample corresponding to the gene position of the random number is selected from parent generation 1; otherwise, the data sample corresponding to the gene position of the random number is selected from parent generation 2. All the selected data samples are combined in the order of the original gene positions to form offspring. This ensures that the data samples of the offspring inherit the characteristics of the parents while introducing a certain degree of randomness, which is beneficial to optimizing the search performance of the genetic algorithm.

[0084] Step S24: Analyze the offspring to determine if there is a mutation. If so, generate a new individual to replace the corresponding individual in the initial population after initialization.

[0085] Further, step S24 includes:

[0086] Step S241: Preset the mutation probability, and generate random numbers for each data sample in the offspring;

[0087] Step S242: When the random number is less than the mutation probability, the mutation mechanism is triggered. The mutation strategy is used to perform feature transformation on the data sample corresponding to the random number to generate a new data sample. The new data sample is used to replace the corresponding data sample in the individual.

[0088] As an optional implementation, in this embodiment, the range of the mutation probability is set to be... .

[0089] Specifically, for each data sample in the offspring, a uniformly distributed random number in the range of 0-1 is generated. When the random number is less than the mutation probability, the mutation mechanism is triggered, and a strategy is randomly selected from the mutation strategies. A feature transformation operation is performed on the data sample corresponding to the position to generate a new data sample. This new data sample replaces the data sample corresponding to the original gene position in the individual. That is, the mutation update of the data samples in the individual is completed by judging and transforming the gene position one by one. Conversely, when the random number is greater than or equal to the mutation probability, the mutation mechanism is not triggered, and the data sample at the current gene position is maintained. This ensures the randomness and controllability of the mutation operation, which helps to maintain the diversity and evolutionary efficiency of the population.

[0090] Furthermore, step S3 includes:

[0091] Step S31: Record the initial population state and select the crossover probability and mutation probability to determine the action space.

[0092] It can be explained that before the genetic algorithm iterates, reinforcement learning is introduced to dynamically optimize the parameters of the genetic algorithm. That is, by utilizing the adaptive adjustment capability of reinforcement learning, the running status of the genetic algorithm is monitored in real time, and the parameters of the genetic algorithm are intelligently adjusted according to the current status and feedback information. This can effectively avoid the premature convergence problem that occurs in the iteration process of the genetic algorithm, and can also significantly improve the convergence speed and search efficiency of the genetic algorithm, ensuring that the genetic algorithm can more stably and efficiently approach the global optimum when solving complex optimization problems.

[0093] Specifically, the initial population state is recorded, i.e., the initial population state is quantitatively represented, including: the fitness of all chromosomes in the initial population, i.e., the arithmetic mean of the fitness of all classes, to determine the average fitness; and the optimal fitness in the initial population, i.e., the chromosome with the lowest fitness. Then, the action space is defined as a set of parameters combining crossover probability and mutation probability. In this embodiment, the crossover probability ranges from [value missing]. The range of values ​​for the mutation probability is: By dividing the two probability intervals of crossover probability and mutation probability into equal intervals, the discrete combinations of different crossover probabilities and mutation probabilities are determined as specific actions, forming an enumerable action space, which provides a clear range of parameter selection for subsequent strategy adjustments.

[0094] Step S32: Execute the crossover and mutation probabilities selected in the action space to generate a new population and update the initial population state.

[0095] Specifically, based on the requirements of each task, the selected crossover and mutation operations are performed in the corresponding number of iterations to generate a new population. The updated state is calculated and used to update the initial population state. The calculation formula for the state is as follows:

[0096]

[0097] in, This indicates the updated initial population state; This represents the average fitness of chromosomes in the current iteration population; This represents the average fitness of chromosomes in the initial population; This represents the optimal fitness of a chromosome in the current iteration population; This indicates the optimal fitness of chromosomes in the initial population; , This represents the corresponding weight, and .

[0098] It can be explained that the updated state is a weighted sum of the ratios of the average fitness and best fitness of the current iteration population to the corresponding average fitness and best fitness in the initial population, in order to preserve the new fitness and state.

[0099] Step S33: Based on the updated initial population state, analyze the initial population state before the update to obtain the reward value.

[0100] Specifically, based on the updated initial population state, the corresponding crossover and mutation probabilities are obtained, and the corresponding optimal fitness is re-acquired. When the adjusted crossover probability makes the optimal fitness of the current iteration population better than the optimal fitness before the adjacent previous adjustment (i.e., the adjusted optimal fitness value is less than the optimal fitness value before the adjacent previous adjustment), it is determined to be a valid search, and the reward value is calculated based on the difference between the two. Similarly, the reward value corresponding to the adjusted mutation probability is obtained; the corresponding calculation formula is:

[0101]

[0102]

[0103]

[0104] in, This represents the reward value for adjusting the crossover probability; This represents the reward value used to adjust the mutation probability. This represents the total reward after adjusting the crossover and mutation probabilities.

[0105] Step S34: Form a Q table by storing states and actions, and update the Q table according to the executed actions and reward values.

[0106] The explanation is that a Q-table is formed by storing states and actions. The Q-table is used to record and store the expected reward value obtained by a data sample in a specific state and when taking a specific action. That is, the Q-table is stored in a two-dimensional array structure, where the row dimension corresponds to the discretized state space and the column dimension corresponds to the specific actions in the action space. The size of the Q-table is determined by the granularity of state partitioning and the number of actions, that is, the number of rows equals the total number of states and the number of columns equals the total number of actions. It is initialized as an all-zero matrix to provide a unified starting point for subsequent Q-value updates and learning processes.

[0107] Specifically, the Q-table records the expected cumulative reward value for each "state-action" pair. By querying the Q-table, the expected reward of each action in the current state can be directly obtained, thus predicting the optimal action choice for the next state. Initially, the SARSA (State-Action-Reward-State-Action) algorithm is used to update the Q-table. Conversely, when the reward is greater than this, indicating a relatively mature stage, the Q-learning algorithm is used to update the Q-table. Combining the two allows for avoiding excessive oscillations in the early exploration phase thanks to the stability of SARSA, while accelerating the approach to the optimal policy in the later convergence phase through Q-learning. The SARSA algorithm considers the current state, the chosen action, and the joint probability of the next state and the next action at each update step. The Q-learning algorithm, however, updates the Q-table independently of the current policy, directly based on the expected reward of the current state and the optimal action.

[0108] The update formula corresponding to the SARSA algorithm is:

[0109]

[0110] Q-learning algorithm update formula:

[0111]

[0112] in, This represents the value corresponding to the current state and action. Indicates state, Indicates an action; This indicates the Q-table value corresponding to the state and action in the next iteration; Indicates the learning rate; Indicates the reward value; This represents the discount factor.

[0113] To clarify, Q-learning is used to estimate the Q-value under the optimal policy, meaning it assumes the next action will be the optimal one (greedy selection) regardless of the current policy being implemented, and calculates the target Q-value accordingly. SARSA, on the other hand, is used to estimate the Q-value under the currently implemented policy, where the target Q-value depends on the actual next action the current policy will take, including exploratory behavior.

[0114] Step S35: Through - A greedy strategy balances search and exploitation, selecting an action from the updated Q table based on the current state as the next action to be executed.

[0115] Provide an explanation. - The greedy strategy introduces a certain degree of randomness into each decision, allowing the genetic algorithm to not only make full use of the current optimal solution, but also to explore other potential possibilities, thus avoiding getting trapped in local optima. Its core idea is to find a suitable balance between exploration and utilization, so as to ensure that the genetic algorithm can achieve better overall performance in the long run.

[0116] Furthermore, in step S35, specifically:

[0117] Based on exploration rate Generate random numbers when the random number is greater than or equal to the exploration rate. Randomly select an action from the action space; when the random number is less than the exploration rate... Query the Q table corresponding to the current state and select the action corresponding to the maximum reward.

[0118] Specifically, adopt - A greedy strategy is used to select actions, generating random numbers in the range of 0-1. Actions are selected when the random number is greater than or equal to the exploration rate. Actions are randomly selected from the action space, i.e., a combination of crossover and mutation probabilities is used to maintain exploratory nature; conversely, when the random number is less than the exploration rate... The algorithm queries the current state Q-table and selects the optimal action with the highest expected reward. The selected action is then parsed into specific crossover and mutation probability parameters, which serve as the basis for the next step of the genetic algorithm operation. This is achieved through dynamic adjustments, such as decreasing the probability as it iterates. This allows for a shift from an exploration-based strategy to a utilization-based strategy, balancing exploration and utilization to improve the efficiency of parameter adjustment.

[0119] It can be explained that in step S4, the iteration is performed and a test suite is generated; specifically, the genetic algorithm optimized in the aforementioned steps is executed to generate a new population, and the action policy is updated using reinforcement learning based on the new population. The fitness value and state parameters of each iteration are recorded; a convergence condition is preset to determine whether to terminate the iteration, ensuring that the genetic algorithm can stop in time when the expected effect is achieved, avoiding unnecessary waste of computational resources.

[0120] During the iteration process, samples with incorrect predictions are filtered out and used to evaluate the robustness of the deep learning model. In each iteration, the optimal individual fitness is recorded, and then the distribution difference range corresponding to the optimal individual fitness is divided into 10 intervals. The test suite with the best fitness in each interval is taken out to represent coverage of different distributions. The selected test suite is used to retrain the deep learning model to better improve the robustness of the deep learning model.

[0121] Understandably, this application integrates deep learning models and genetic algorithms. By executing the genetic algorithm and introducing reinforcement learning to dynamically optimize the operating parameters of the genetic algorithm, it overcomes the problems of local optima and parameter sensitivity. This allows the most beneficial action to be dynamically selected for the next step based on the current state during the iteration process. By dynamically adjusting the crossover probability and mutation probability, it promotes the generation of test suites with smaller distributional differences, i.e., test samples with distributional diversity, which can effectively improve the convergence speed of fitness. At the same time, it is closer to the decision boundary of the target class, generating erroneous samples that are difficult to detect. In practical applications, the deep learning model is retrained using test suites with different distributions, which improves the robustness of the deep learning model.

[0122] To better illustrate this, the test sample generation method proposed in this application is compared with the prior art, where the prior art is a deep learning test method GA (Genetic Algorithm) that only uses a genetic algorithm, and the genetic algorithm in this application is denoted as PLGA (Proposed Genetic Algorithm).

[0123] Specifically, for a dataset with 10 categories, test samples were generated from each source category to a randomly selected target category, completing a total of 10 cross-category sample generation experiments; using... The performance of the generated samples was evaluated using two adversarial sample detection methods, PLGA and GA. The lower the value of the quantitative index, the better. That is, the lower the value of the generated sample, the better the quality of the generated sample. Table 1 shows the experimental results of the comparison between PLGA and GA.

[0124] Table 1. Comparison of experimental results between PLGA and GA

[0125]

[0126] It can be shown that, in most scenarios, the detection metrics of the generated samples, i.e., the test suites, of this application outperform the traditional GA method; at the same time, the number of erroneous samples generated by this application is 2-3 times higher than that of the traditional GA method, which verifies the generation efficiency and quality advantages of the test sample generation method of this application.

[0127] Please see Figure 3 Under the condition that the dataset and model are completely consistent, the convergence process of the test sample generation method of this application is always faster than that of ordinary GA, which fully demonstrates its effectiveness in accelerating convergence.

[0128] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0129] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A deep learning test sample generation method that utilizes reinforcement learning to optimize genetic algorithms, characterized in that, The method includes: Obtain the image dataset, construct seed datasets for different categories based on the image dataset, determine the initial population and initialize it; Obtain the deep learning model, execute the genetic algorithm, and optimize the initial population after initialization; Combine reinforcement learning to dynamically optimize the parameters of the genetic algorithm; The optimized initial population is iteratively optimized using the optimized genetic algorithm to generate a new population, and data samples with prediction errors are filtered out to obtain a test suite.

2. The method for generating deep learning test samples using reinforcement learning to optimize genetic algorithms according to claim 1, characterized in that, Based on the image dataset, seed datasets of different categories are constructed, and the initial population is determined and initialized, including: The image dataset is divided into multiple categories, and multiple data samples are selected from each category to construct a seed dataset to form individuals; Variants are generated based on individual application mutation strategies, and the combined variants and corresponding individuals form an initial population.

3. The method for generating deep learning test samples using reinforcement learning to optimize genetic algorithms according to claim 2, characterized in that, The mutation strategy includes any image feature transformation strategy such as rotation, translation, scaling, and local blurring.

4. The method for generating deep learning test samples using reinforcement learning to optimize genetic algorithms according to claim 2, characterized in that, Obtain the deep learning model, execute the genetic algorithm, and optimize the initial population after initialization, including: The categories include source category and target category; each data sample in the initial population after initialization is input into the deep learning model, the logits layer output is extracted to obtain the logits vector, the corresponding logits set is constructed based on the source category and the target category, and the MMD value is determined to be the fitness; The individuals in the initial population after initialization are divided into two groups, and the parents are selected based on fitness. Preset the crossover probability, and use a uniform crossover strategy based on the parent generation and the crossover probability to obtain the offspring; Analyze the offspring to determine if they have mutated. If so, generate new individuals to replace the corresponding individuals in the initial population after initialization.

5. The method for generating deep learning test samples using reinforcement learning to optimize genetic algorithms according to claim 4, characterized in that, The parent generation is obtained based on fitness screening, including: Within each group, fitness is sorted in ascending order, and the individual with the lowest fitness in each group is retained. The remaining individuals are selected through a tournament selection mechanism to generate two sets of fixed-length data sequences, i.e., two sets of serial numbers, and the parent generation is obtained based on the serial numbers. From the two sets of data sequences, the parent individuals corresponding to the first sequence number are selected sequentially and crossover mutation is performed to generate new individuals until the number of individuals in each group in the corresponding initial population is satisfied.

6. The method for generating deep learning test samples using reinforcement learning to optimize genetic algorithms according to claim 5, characterized in that, Preset crossover probabilities, and use a uniform crossover strategy based on the parent generation and crossover probabilities to obtain offspring, including: For each data sample in the parent generation, generate a random number and compare the random number with the crossover probability; In the two sets of data sequences, the individuals corresponding to the first index are designated as Parent Generation 1 and Parent Generation 2, respectively. When the random number is less than the crossover probability, the data sample at the corresponding position is selected from Parent Generation 1; when the random number is greater than or equal to the crossover probability, the data sample at the corresponding position is selected from Parent Generation 2. All selected data samples are combined in the order of the original data samples to form offspring.

7. The method for generating deep learning test samples using reinforcement learning to optimize genetic algorithms according to claim 6, characterized in that, Analyze the offspring to determine if they have mutated. If so, generate new individuals to replace the corresponding individuals in the initial population after initialization, including: Preset mutation probability, and generate random numbers for each data sample in the offspring; When the random number is less than the mutation probability, the mutation mechanism is triggered. The mutation strategy performs feature transformation on the data sample corresponding to the random number to generate a new data sample, and the new data sample replaces the corresponding data sample in the individual.

8. The method for generating deep learning test samples using reinforcement learning to optimize genetic algorithms according to claim 7, characterized in that, Combining reinforcement learning with dynamic optimization of genetic algorithm parameters includes: Record the initial population state, and determine the action space by selecting the crossover probability and mutation probability; The crossover and mutation probabilities selected in the action space are used to generate a new population and update the initial population state. The reward value is obtained by analyzing the updated initial population state in conjunction with the initial population state before the update. A Q-table is formed by storing states and actions, and the Q-table is updated based on the executed actions and reward values. pass - A greedy strategy balances search and exploitation, selecting an action from the updated Q table based on the current state as the next action to be executed.

9. The method for generating deep learning test samples using reinforcement learning to optimize genetic algorithms according to claim 7, characterized in that, pass - A greedy strategy balances search and exploitation, selecting an action from the updated Q table based on the current state as the next action to be executed. Specifically: Based on exploration rate Generate random numbers when the random number is greater than or equal to the exploration rate. Randomly select an action from the action space; when the random number is less than the exploration rate... Query the Q table corresponding to the current state and select the action corresponding to the maximum reward.

Images