A test case intelligent generation method based on reinforcement learning

Abstract

This invention relates to the fields of reinforcement learning and automated testing technology, specifically a method for intelligently generating test cases based on reinforcement learning. It involves constructing an improved soft-actor critic policy network containing discrete and continuous action branches, generating legitimate and diverse execution actions by combining a constrained projective action layer, introducing a critic network with a multi-head quantile regression structure to output action value distributions, using quantile difference and variance to measure uncertainty, and combining coverage increment and path novelty to calculate priority, achieving high-value sample priority learning. A curriculum-based reward scheduler dynamically adjusts reward weights according to training stages, including early reinforcement exploration, mid-term balancing exploration and utilization, and late-stage convergence promotion and defect triggering. Finally, the generated test cases are deduplicated, minimized, and the environment configuration is recorded to ensure reproducibility of the results.

Description

Technical Field

[0001] This invention relates to the field of automated testing technology, specifically to a method for intelligent generation of test cases based on reinforcement learning. Background Technology

[0002] As the complexity of software system functions and the dynamism of the operating environment continue to increase, traditional test case design methods are no longer sufficient to meet the requirements of high reliability systems for test coverage and defect detection rate. Especially in scenarios with multiple interface interactions, time-series dependencies and protocol constraints, test cases need to cover multi-dimensional operation space and have strong exploration capabilities in order to effectively expose potential defects. However, traditional automated test case generation technologies still generally have shortcomings.

[0003] First, test case generation methods based on static analysis or random strategies are difficult to fully utilize the runtime state information of the system under test. The generated test cases lack specificity, have limited coverage improvement, and are prone to repeatedly exploring covered paths, resulting in wasted resources. While reinforcement learning-based research introduces intelligent policy generation, it often uses single continuous or discrete action space modeling, which cannot simultaneously handle multi-dimensional decision variables such as interface calls, parameter values, and temporal perturbations, resulting in insufficient diversity and exploration depth of generated test cases.

[0004] Meanwhile, most methods do not impose effective protocol constraints and business rule checks on the actions output by the strategy, which can easily generate unexecutable or invalid actions, reducing training efficiency. Conventional experience replay mechanisms often use uniform sampling or simple priority rules, without combining the uncertainty of action value distribution and path novelty indicators, making it difficult to ensure that high-value samples are learned first, resulting in slow convergence speed and insufficient exploration. Furthermore, traditional reward function designs are usually fixed weights, which do not consider the differences in training stages, and cannot fully incentivize exploration in the early stages or guide strategy convergence in the later stages, which can easily lead to problems such as premature convergence of the strategy, low defect trigger rate, and stagnant coverage growth. Summary of the Invention

[0005] This invention addresses the problems of traditional methods generating test cases that lack specificity, easily lead to repeated exploration of covered paths, resulting in wasted resources and low training efficiency. It proposes an intelligent test case generation method based on reinforcement learning.

[0006] The objective of this invention can be achieved through the following technical solution: a method for intelligently generating test cases based on reinforcement learning, comprising the following steps: Step 1: Establish a data acquisition channel to collect information from the system under test, obtain the current status, and output it. Step 2: Construct a policy network and a critic network using an algorithm, output the original action based on the current state, and perform constraint projection on the original action to obtain the execution action that conforms to the protocol and business rules; Step 3: Execute the actions obtained in Step 2 in the system under test, collect execution feedback information, and determine the next state; Step 4: Calculate the composite reward based on the execution feedback information, and dynamically adjust the weights through the course-based reward scheduler to obtain the weighted reward; Step 5: Store the current state, the action performed, the weighted reward, and the next state into the experience replay buffer. Utilize the commentator network ensemble to calculate the action value distribution and uncertainty, and determine the sample priority. Step 6: Sample from the experience replay buffer according to sample priority, update the policy through the algorithm, and adaptively adjust the policy stochasticity parameter based on the joint policy entropy and coverage growth rate to obtain the updated policy network; Step 7: Generate test cases using the updated policy network, deduplicate and minimize the test cases, record the random seed and environment configuration, and output the final test cases.

[0007] As a preferred embodiment of the present invention, the information of the tested system collected in step one includes statement coverage, branch hit status, execution path identification, abnormal signals, input syntax features and historical statistical information; In step one, after acquiring the information of the system under test, the collected information is deduplicated, synchronized in time, filled with missing values, and truncated with outliers. Next, index encoding, length unification processing, and hash encoding are performed on discrete fields, sequence fields, and path identifiers respectively. The coverage data is expanded into a fixed-length vector, and the processed features are concatenated into a fixed-length state vector in a predetermined order to output the current state.

[0008] As a preferred embodiment of the present invention, the step of obtaining execution feedback information in step three includes: collecting running logs, abnormal signals, return values, status codes, execution time and resource usage, recording changes in coverage, path identifier updates and defect triggering after execution, calculating path novelty index and constraint violation, and forming execution feedback information; In step three, the execution feedback information will be associated with and stored with the current state to confirm the next state.

[0009] In a preferred embodiment of the present invention, the specific process of obtaining the weighted composite reward in step four is as follows: Analyze the execution feedback information, extracting the coverage change data, defect trigger records, path novelty metrics, and constraint violation events contained therein, and then: Calculate the coverage increment value based on coverage change data and generate the coverage increment reward; Based on the defect trigger records, the number of defect triggers and their severity are calculated, and a defect trigger reward is generated. Calculate the path exploration score based on the path novelty index and generate a path novelty reward; Calculate the number of violations or severity based on constraint violation events, and generate constraint penalty values; The coverage increment reward, defect trigger reward, path novelty reward, and constraint penalty value are then input into the course-based reward scheduler. The course-based reward scheduler dynamically adjusts the weights and performs a weighted sum of each individual reward and penalty value based on the dynamic weights to form a weighted composite reward.

[0010] In a preferred embodiment of the present invention, the process of calculating the action value distribution in step five is as follows: The commentator network is called to perform forward computation on the training samples. The commentator network adopts a quantile regression structure with multiple output heads. Each output head corresponds to a preset quantile. The commentator network performs forward propagation through the feature extraction layer and the hidden layer in sequence to obtain the action value prediction value of each quantile. Then, the prediction values ​​of all quantiles are combined to form a complete action value distribution.

[0011] In a preferred embodiment of the present invention, the method for determining the sample priority in step five is as follows: Based on the action value distribution, the mean, quantile difference, variance, and distribution skewness are calculated to obtain the uncertainty measure index of the sample. The uncertainty measure is combined with the corresponding coverage increment and path novelty index of the sample, and the comprehensive priority score is calculated by weighted summation. The sampling weight is written into the buffer index and sorted to form a sample queue arranged by priority, thus determining the sample priority.

[0012] As a preferred embodiment of the present invention, the specific method of step six includes: Batch samples are drawn from the experience replay buffer according to sample priority. The batch samples include the current state, the executed action, the weighted composite reward and the next state to form a training batch. The training batch is input into the critic network to calculate the target quantile distribution and predict the deviation between the quantile and the target quantile, and then the parameters of each output head of the critic network are updated. The policy objective is calculated using the action value distribution output by the updated critic network. A policy loss function containing policy entropy regularization and reward expectation is constructed, and the policy network parameters are optimized through gradient backpropagation. A dual-objective balance index is constructed based on policy entropy and coverage growth rate, and the policy stochasticity parameter is dynamically adjusted according to the training rounds. Write the updated policy network and critic network back into the training framework, replacing the original parameters.

[0013] In a preferred embodiment of the present invention, the deduplication process in step seven is implemented through a two-layer strategy: including hash deduplication and semantic equivalence determination. The minimization process in step seven obtains a minimum use case that does not reduce coverage and defect triggering by removing redundant steps, merging equivalent operations, and reducing input parameters. Calculate the coverage contribution, defect trigger rate, and path novelty score of the minimized test cases. Sort the test cases according to their comprehensive scores, select test cases with comprehensive scores higher than the threshold to form the final test case set, and record the random seed, environment configuration, policy network parameter version, and execution context at the time of generation.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs an improved soft actor critic policy network that includes discrete and continuous action branches. Combined with a constrained projection action layer, it unifies the modeling of multi-dimensional decisions such as interface calls, parameter values, and temporal perturbations. After action generation, it automatically eliminates illegal actions that do not meet the protocol and business rules, thereby realizing the legalization and diversification of the action space. This effectively improves the coverage, path diversity, and defect triggering rate of test case generation, shortens the policy convergence time, reduces invalid sample learning, and achieves efficient, intelligent, and traceable automated test case generation for complex software systems.

[0015] This invention employs a multi-head quantile regression structure to output the action value distribution during the value assessment stage, introducing quantile difference and variance as uncertainty measures. It also combines coverage increment and path novelty to calculate a comprehensive priority score, enabling uncertainty-driven experience replay sampling. This prioritizes high-potential-value and high-exploration-value samples, improving strategy convergence speed. In the reward calculation stage, a curriculum-based reward scheduler is introduced. It dynamically adjusts coverage, defect triggering, novelty, and constraint weights based on training progress, convergence degree, and coverage growth rate. This enhances exploration intensity in the early stages, balances exploration and utilization in the mid-stage, and increases defect triggering and constraint weights in the later stages, guiding continuous strategy optimization. In the test case output stage, generated test cases are deduplicated and minimized, retaining the optimal test cases where coverage and defect triggering do not decrease. Random seeds and environment configurations are recorded to ensure reproducibility of results. Attached Figure Description

[0016] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0017] Figure 1 This is a system flowchart of the present invention; Figure 2This is a schematic diagram of the strategy network and commentator network structure of the present invention; Figure 3 This is a schematic diagram of the test case deduplication and minimization process of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1: Please refer to Figures 1-2 As shown, a method for intelligently generating test cases based on reinforcement learning includes the following steps: Step 1: Perform instrumentation or interface encapsulation on the system under test to establish a runtime data acquisition channel. Collect information about the system under test in the runtime data acquisition channel. The collected information includes statement coverage, branch hit status, execution path identification, exception signals, input syntax features, and historical statistics. After obtaining the information of the system under test, the collected information is deduplicated using a sliding window, synchronized using a unified time base, filled with missing values ​​using linear interpolation, and truncated for obviously abnormal data, thereby completing the preliminary data processing; Subsequently, index encoding is performed on discrete fields, length uniform processing is performed on sequence fields, and hash encoding is performed on path identifiers. Coverage data is expanded into a fixed-length vector, thereby mapping path identifiers into fixed-length integer vectors. The number of hash buckets is dynamically adjusted according to the path space size, and the collision rate is reduced through an open addressing strategy to ensure path distinguishability. The processed features are concatenated into a fixed-length state vector in a predetermined order, and the current state is output.

[0020] Step 2: Construct a policy network and a critic network using an improved soft actor critic algorithm, output the original action based on the current state, and perform constraint projection on the original action to obtain the execution action that conforms to the protocol and business rules.

[0021] Step 3: Input the execution action after constraint projection processing into the interface execution module of the system under test to trigger the corresponding interface call or operation process, thereby executing the execution action in the system under test. At the same time, during the execution process, collect the operation log, abnormal signals, return values, status codes, execution time and resource usage. After execution, record changes in coverage, path identifier updates, and defect triggering, calculate path novelty index and constraint violations, and generate execution feedback information; associate and store the execution feedback information with the current state, and confirm the next state.

[0022] Step 4: Calculate the composite reward based on the execution feedback information, and dynamically adjust the weights using a course-based reward scheduler to obtain a weighted reward. The specific process is as follows: Analyze the execution feedback information, extracting the coverage change data, defect trigger records, path novelty metrics, and constraint violation events contained therein, and then: Calculate the coverage increment value based on coverage change data and generate the coverage increment reward; Based on the defect trigger records, the number of defect triggers and their severity are calculated, and a defect trigger reward is generated. Calculate the path exploration score based on the path novelty index and generate a path novelty reward; Calculate the number of violations or severity based on constraint violation events, and generate constraint penalty values; The coverage increment reward, defect trigger reward, path novelty reward, and constraint penalty value are then input into the curriculum-based reward scheduler. The curriculum-based reward scheduler determines the training stage based on the training progress, convergence degree, and coverage growth rate, and dynamically adjusts the coverage weight, defect weight, novelty weight, and constraint weight. In the early stage, the coverage weight and novelty weight are increased; in the mid-stage, the defect weight is gradually increased; and in the late stage, the exploration weight is reduced while the defect trigger and constraint weights are increased. Finally, the individual rewards and penalties are weighted and summed based on dynamic weights to form a weighted composite reward.

[0023] Step 5: Encapsulate the current state, the action performed, the weighted reward, and the next state into training samples, then store them in the experience replay buffer, and assign timestamps and unique indexes to the training samples. At the same time, manage the capacity of the experience replay buffer during writing, discard the earliest sample according to the first-in-first-out principle, and maintain the index structure to support fast retrieval by priority. The distribution and uncertainty of action value are calculated using a commentator network ensemble, and sample priority is determined. Specifically, the calculation process of the action value distribution is as follows: the commentator network is called to perform forward calculation on the training samples. The commentator network adopts a quantile regression structure with multiple output heads. Each output head corresponds to a preset quantile. The commentator network performs forward propagation through the feature extraction layer and the hidden layer in sequence to obtain the action value prediction value of each quantile. Then, the prediction values ​​of all quantiles are combined to form a complete action value distribution. Specifically, uncertainty is calculated based on the mean, quantile, variance, and skewness of the action value distribution to obtain a measure of the sample's uncertainty. Specifically, the process of determining sample priority is as follows: combining uncertainty measure with the corresponding coverage increment and path novelty index, using weighted summation to calculate comprehensive priority score, writing sampling weights into buffer index and performing sorting to form a sample queue arranged by priority, thus determining sample priority.

[0024] Step 6: Sample from the experience replay buffer according to sample priority, extract batch samples, where the batch samples include the current state, the executed action, the weighted composite reward and the next state, forming a training batch. Input the training batch into the commentator network, calculate the target quantile value distribution, minimize the deviation between the predicted quantile value and the target quantile value based on the quantile regression loss function, update the parameters of each output head of the commentator network, so that the action value distribution is closer to the true distribution. The policy is updated by using an improved soft actor critic algorithm. The policy objective is calculated using the action value distribution output by the updated critic network. A policy loss function containing policy entropy regularization and reward expectation is constructed. The policy network parameters are optimized by gradient backpropagation so that the action distribution output by the policy network maintains appropriate randomness while maximizing the weighted composite reward expectation. During the update process, quantile difference, variance and skewness are calculated to generate sample uncertainty index, and the priority value of the corresponding sample in the experience replay buffer is updated in real time to ensure that sampling is more focused on high-value and high-exploration-potential samples. A dual-objective balance index is constructed based on policy entropy and coverage growth rate. The policy randomness parameter is dynamically adjusted according to the training rounds. In the early stage of training, the policy randomness is increased to enhance the exploration of uncovered paths and potential defects. In the middle and late stages of training, the policy randomness is gradually reduced to promote policy convergence and stably generate high-quality test cases. The updated policy network and critic network are written back into the training framework, replacing the original parameters, and used for action generation and value evaluation of the next round of state input. Step 7: Apply the updated policy network to the current state to generate candidate test cases in batches. The candidate test cases include the interface call sequence, parameter values ​​and execution sequence. Perform deduplication on the candidate test cases, delete identical or semantically equivalent test cases, and retain unique test cases. The deduplication process is implemented through a two-layer strategy: hash deduplication and semantic equivalence determination. Specifically, hash deduplication uses the interface sequence, parameters, and path identifiers of candidate use cases to generate hash values ​​for rapid comparison, eliminating completely duplicate use cases; Semantic equivalence determination is based on abstract syntax trees and path coverage vectors to calculate semantic similarity, merge use cases with equivalent execution effects, and retain only one representative use case; Minimize the deduplicated test cases by removing redundant steps, merging equivalent operations, and reducing input parameters to obtain minimal test cases that do not reduce coverage or defect triggering. Calculate the coverage contribution, defect triggering rate, and path novelty score of the minimized use case, and sort the use cases according to the comprehensive score; Use cases with a comprehensive score higher than the threshold are selected to form the final test case set. The random seed, environment configuration, policy network parameter version and execution context at the time of generation are recorded, and the final test cases are output. The final test case set is provided to the system under test for execution, and the execution results are written back to the state cache as the starting point for the next round of state collection or as the basis for determining the termination condition.

[0025] In steps two and six, the improved soft actor critic algorithm specifically includes: A dual-network structure consisting of a policy network and a commentator network is constructed. The policy network includes discrete action branches and continuous action branches. Specifically, the discrete action branches are used for selecting interface calls, operation types, and execution order, while the continuous action branches are used for generating parameter values, input amplitudes, and temporal perturbations. After the policy network outputs the original action, a constraint projection action layer is set up to perform constraint mapping on actions that do not meet the protocol constraints, interface mode descriptions, or business rules, and to correct actions that exceed the feasible domain into legal actions that meet the constraints. Among them, the commentator network uses a multi-head quantile regression structure to output the action value distribution, calculates the quantile difference and variance of the value distribution as a measure of uncertainty, and uses uncertainty to calculate the priority of experience replay samples, so that samples with high potential value and high exploration value are learned first. During the policy optimization process, the policy entropy and coverage growth rate are used as dual optimization objectives. The policy stochasticity parameters are dynamically and adaptively adjusted to increase the exploration intensity in the early stage of training and enhance the policy convergence in the later stage of training, taking into account both exploring new paths and utilizing known high-value actions.

[0026] Thresholds, preset values, or preset ranges are set for result comparison and analysis to determine whether they are good or bad. The value of these thresholds is determined by a combination of large-scale model analysis of sample data and human experience. They can also be adjusted appropriately based on seasonal or rational factors.

[0027] Obviously, many modifications and variations can be made based on the content of this specification. These embodiments have been selected and specifically described in this specification to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for intelligently generating test cases based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Establish a data acquisition channel to collect information from the system under test, obtain the current status, and output it. Step 2: Construct a policy network and a critic network using an algorithm, output the original action based on the current state, and perform constraint projection on the original action to obtain the execution action that conforms to the protocol and business rules; Step 3: Execute the actions obtained in Step 2 in the system under test, collect execution feedback information, and determine the next state; Step 4: Calculate the composite reward based on the execution feedback information, and dynamically adjust the weights through the course-based reward scheduler to obtain the weighted reward; Step 5: Store the current state, the action performed, the weighted reward, and the next state into the experience replay buffer. Utilize the commentator network ensemble to calculate the action value distribution and uncertainty, and determine the sample priority. Step 6: Sample from the experience replay buffer according to sample priority, update the policy through the algorithm, and adaptively adjust the policy stochasticity parameter based on the joint policy entropy and coverage growth rate to obtain the updated policy network; Step 7: Generate test cases using the updated policy network, deduplicate and minimize the test cases, record the random seed and environment configuration, and output the final test cases.

2. The intelligent test case generation method based on reinforcement learning according to claim 1, characterized in that, The information collected in step one of the tested system includes statement coverage, branch hit status, execution path identification, abnormal signals, input syntax features, and historical statistics. In step one, after acquiring the information of the system under test, the collected information is deduplicated, synchronized in time, filled with missing values, and truncated with outliers. Next, index encoding, length unification processing, and hash encoding are performed on discrete fields, sequence fields, and path identifiers respectively. The coverage data is expanded into a fixed-length vector, and the processed features are concatenated into a fixed-length state vector in a predetermined order to output the current state.

3. The intelligent test case generation method based on reinforcement learning according to claim 1, characterized in that, The steps for obtaining execution feedback information in step three include: collecting runtime logs, exception signals, return values, status codes, execution time and resource usage; recording changes in coverage, path identifier updates and defect triggering after execution; calculating path novelty indices and constraint violations; and generating execution feedback information. In step three, the execution feedback information will be associated with and stored with the current state to confirm the next state.

4. The intelligent test case generation method based on reinforcement learning according to claim 1, characterized in that, In step four, the specific process for obtaining the weighted composite reward is as follows: Analyze the execution feedback information, extracting the coverage change data, defect trigger records, path novelty metrics, and constraint violation events contained therein, and then: Calculate the coverage increment value based on coverage change data and generate the coverage increment reward; Based on the defect trigger records, the number of defect triggers and their severity are calculated, and a defect trigger reward is generated. Calculate the path exploration score based on the path novelty index and generate a path novelty reward; Calculate the number of violations or severity based on constraint violation events, and generate constraint penalty values; The coverage increment reward, defect trigger reward, path novelty reward, and constraint penalty value are then input into the course-based reward scheduler. The course-based reward scheduler dynamically adjusts the weights and performs a weighted sum of each individual reward and penalty value based on the dynamic weights to form a weighted composite reward.

5. The intelligent test case generation method based on reinforcement learning according to claim 1, characterized in that, The process of calculating the action value distribution in step five is as follows: The commentator network is called to perform forward computation on the training samples. The commentator network adopts a quantile regression structure with multiple output heads. Each output head corresponds to a preset quantile. The commentator network performs forward propagation through the feature extraction layer and the hidden layer in sequence to obtain the action value prediction value of each quantile. Then, the prediction values ​​of all quantiles are combined to form a complete action value distribution.

6. The intelligent test case generation method based on reinforcement learning according to claim 1, characterized in that, The method for determining sample priority in step five is as follows: Based on the action value distribution, the mean, quantile difference, variance, and distribution skewness are calculated to obtain the uncertainty measure index of the sample. The uncertainty measure is combined with the corresponding coverage increment and path novelty index of the sample, and the comprehensive priority score is calculated by weighted summation. The sampling weight is written into the buffer index and sorted to form a sample queue arranged by priority, thus determining the sample priority.

7. The intelligent test case generation method based on reinforcement learning according to claim 1, characterized in that, The specific methods for step six include: Batch samples are drawn from the experience replay buffer according to sample priority. The batch samples include the current state, the executed action, the weighted composite reward and the next state to form a training batch. The training batch is input into the critic network to calculate the target quantile distribution and predict the deviation between the quantile and the target quantile, and then the parameters of each output head of the critic network are updated. The policy objective is calculated using the action value distribution output by the updated critic network. A policy loss function containing policy entropy regularization and reward expectation is constructed, and the policy network parameters are optimized through gradient backpropagation. A dual-objective balance index is constructed based on policy entropy and coverage growth rate, and the policy stochasticity parameter is dynamically adjusted according to the training rounds. Write the updated policy network and critic network back into the training framework, replacing the original parameters.

8. The intelligent test case generation method based on reinforcement learning according to claim 1, characterized in that, The deduplication process in step seven is implemented through a two-layer strategy: including hash deduplication and semantic equivalence determination. The minimization process in step seven obtains a minimum use case that does not reduce coverage and defect triggering by removing redundant steps, merging equivalent operations, and reducing input parameters. Calculate the coverage contribution, defect trigger rate, and path novelty score of the minimized test cases. Sort the test cases according to their comprehensive scores, select test cases with comprehensive scores higher than the threshold to form the final test case set, and record the random seed, environment configuration, policy network parameter version, and execution context at the time of generation.

Images