A test case generation method based on reinforcement learning
By using a reinforcement learning-based method to parse the state machine path and dynamically advance the variable interval, test cases for complex state machines are generated. This solves the problems of reliance on manual derivation and low solver efficiency in existing technologies, and achieves efficient and diverse test case generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-13
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies rely excessively on manual derivation when generating state machine test cases. The solver outputs a single solution and the computation is time-consuming, resulting in low test coverage and making it difficult to meet the requirements for automated and efficient coverage of complex state machines.
By employing a reinforcement learning-based approach, a dynamically advancing reinforcement learning environment is constructed by parsing the state machine path into a queue of operation nodes and combining it with interval arithmetic. The agent progressively cuts the variable value range during multiple rounds of trial and error, generating a complete feasible domain and automatically sampling test cases.
It improves test path coverage and efficiency, generates diverse and boundary coverage capabilities, is suitable for state machine systems with complex constraints and multi-variable updates, and enables intelligent and automated test case generation.
Smart Images
Figure CN122111862A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of software testing and intelligent optimization technology, specifically to a test case generation method based on reinforcement learning. Background Technology
[0002] As modern software systems become increasingly complex, their internal behavior is more frequently modeled using state machines or state diagrams. This modeling approach clearly expresses the various states of the system and the transition relationships between them, but it also presents new challenges for testing. In state machine diagrams, each state transition typically includes condition guards and variable assignment operations. Testers need to generate numerous test cases covering different paths based on these conditions to verify the correctness and robustness of the system under various scenarios. Traditional test case generation methods, when faced with state machines containing complex conditions and variable updates, often rely on manual analysis and fixed rule derivation. This is not only inefficient but also prone to overlooking critical boundary conditions, resulting in insufficient test coverage and failing to adequately guarantee system quality.
[0003] Traditional automated test case generation methods also have significant limitations. While static analysis-based or rule-driven methods can automatically enumerate paths and generate input data to some extent, they often struggle to obtain feasible input ranges when faced with multi-layered assignment chains and complex Boolean combinations. Solver-based constraint-solving methods can find a feasible solution that satisfies a certain path, but they typically only output a small number of points and cannot automatically provide the entire feasible region, let alone automatically cover various condition boundaries or generate diverse samples. Evolutionary algorithms or search-based software testing also suffer from shortcomings in objective function construction, convergence speed, and boundary coverage stability. These problems mean that although various tools exist in the industry, complex state machine scenarios still heavily rely on manual derivation and repeated trial and error. Therefore, these issues urgently need to be addressed. Summary of the Invention
[0004] This invention aims to address the problems of excessive reliance on manual derivation, solvers only outputting a single solution, computational delays, and low test coverage in existing state machine test case generation processes. To this end, this invention proposes a reinforcement learning-based test path generation method. This method abandons the traditional approach of constructing global static constraints. By parsing the state machine path into a sequentially arranged queue of operation nodes and combining interval arithmetic, a dynamically progressive reinforcement learning environment is constructed. Utilizing the reinforcement learning algorithm, the variable value range is progressively narrowed and refined through multiple rounds of trial and error during node advancement. Finally, a complete feasible domain spanning the entire path is output and automatically sampled from it, achieving efficient and automated test case generation for complex state machine paths.
[0005] The beneficial effects of this invention are that by replacing the manual solution process with a reinforcement learning agent, it can quickly generate variable value ranges and representative test cases that meet the test path transition conditions, improving path coverage and testing efficiency. This is particularly suitable for state machine systems with complex constraints and multi-variable updates. Simultaneously, the trained agent can generate test cases randomly and in diverse ways, reducing the rigidity of manual rules. While ensuring coverage and generation efficiency, it significantly improves the diversity and boundary coverage of test cases, providing a novel solution for the intelligent and automated generation of state machine test cases.
[0006] This invention provides a test path generation method based on reinforcement learning, comprising the following steps:
[0007] S1. Model parsing and serialization: Obtain the state machine diagram of the system under test, extract the state, transition conditions and assignment operations, and parse the path under test into a queue of operation nodes arranged alternately by transition condition nodes and assignment operation nodes according to the control flow timing.
[0008] S2. Variable range initialization: Allocate a global wide range of values for the initial input variables of the test path. The system maintains the variable state in memory in the form of a range, which serves as the initial carrier of the dynamic forward execution sequence.
[0009] S3. Construction of Reinforcement Learning Environment: Construct a Markov decision process environment model based on dynamic advancement reinforcement learning, and encode the current node index, the upper and lower bounds of the real-time intervals of each variable, and the local constraints of the current node into a state space using vectorized representations; wherein, the vectorized representation of the local constraints includes: variable coefficients, constant terms, comparison signs extracted after parsing the current edge constraint conditions into a linear expression about the root variable, and the satisfaction characteristics of the current variable interval with the local constraints; the boundary tightening operation for the variable interval is defined as the action space;
[0010] Furthermore, the action space includes: tightening the lower bound of a specified variable upwards, tightening the upper bound of a specified variable downwards, and fixing the value of a specified variable to the boundary or median of the current interval; wherein each action is accompanied by a variable index and a tightening magnitude parameter;
[0011] Furthermore, the specific extraction method of the satisfaction feature mentioned in step S3 is as follows: the environment calculates the geometric distance between the variable interval boundary and the constraint boundary in real time according to the constraint logic of the current operation node, and normalizes it into a continuous value in the interval [-1, 1], which serves as the feature dimension in the state space to guide the agent to perceive the urgency of the constraint.
[0012] S4. Agent Policy Training: The PPO algorithm is used to train the agent in multiple rounds. The agent moves forward along the queue of operation nodes and outputs interval tightening actions based on the current state. The environment verifies the legality of actions in real time. If tightening causes the upper and lower bounds of the interval to be reversed or violates the current constraints, the environment determines that there is no solution, imposes a high termination penalty, and forcibly ends the current round. If the interval satisfies the constraints after tightening, the environment provides a stage reward and advances the node. At the same time, the environment triggers reverse constraint propagation, shrinking the effective interval of the current intermediate variable and mapping it back to the envelope of the initial input variable in real time through the inverse operation of the assignment chain. The environment automatically triggers forward interval arithmetic, automatically calculates the new interval of the assigned variable according to the mathematical relationship, and automatically advances to the next node. The reward mechanism guides the agent to obtain the optimal interval tightening strategy that runs through the entire path.
[0013] Furthermore, the reward mechanism employs a composite reward function, including:
[0014] Unsolvable termination penalty: When a tightening action causes the upper and lower bounds of the interval to be reversed or local constraints are violated, a negative reward is given and the turn is terminated;
[0015] Phased completion reward: A positive reward is given when the tightened interval falls completely into the current legal domain and successfully advances to the next node;
[0016] Tightness reward: Positive rewards are given based on the degree of increase in the contraction rate of the variable interval, and excessive tightening behavior is punished;
[0017] Step cost: A small negative reward is deducted for each tightening step to encourage algorithm convergence;
[0018] Furthermore, the processing logic for the transition condition node is as follows: after the agent outputs the tightening action, the environment verifies the intersection of the current interval and the transition condition in real time. If the intersection is empty, it is determined to be illegal and terminated. If the intersection is not empty, the interval is updated and the process is advanced to the next node.
[0019] Furthermore, the processing logic of the assignment operation node is as follows: automatically calculate the new interval of the variable after assignment using forward interval arithmetic, and automatically advance to the next node in the operation node queue;
[0020] S5. Test Case Generation and Reverse Mapping: The trained agent performs forward inference on the target path. During the inference process, the reverse constraint propagation mechanism is used to map the effective contraction of each node back to the initial input variables in real time. When the end point of the path is reached, the final feasible domain of the initial input variables is extracted, and a test case set is generated by sampling from the final feasible domain. The sampling strategy includes: extracting the extreme value boundary of the final feasible domain as boundary value test cases; performing uniform random sampling or taking the mathematical median as internal value test cases within the final feasible domain; and performing orthogonal combination of the extracted multi-dimensional variable values to output a representative test case set.
[0021] Furthermore, the reverse constraint propagation mechanism is as follows: when processing the tightening action of deep intermediate variables, the interval contraction of the variable is simultaneously mapped back to the corresponding initial input variable envelope in real time through the inverse operation of the assignment chain relationship;
[0022] Preferably, the reinforcement learning-based test path generation method is applied to the scenario of automated test case generation for an embedded system state machine model with numerical variables, including multi-level assignment chains and Boolean combination conditions. Attached Figure Description
[0023] Figure 1 This is a flowchart of the reinforcement learning-based test case generation method described in this invention;
[0024] Figure 2 To enhance the internal environment structure diagram of learning. Detailed Implementation
[0025] This implementation method is illustrated with reference to the figures.
[0026] This invention provides a test path generation method based on reinforcement learning, which is described below in conjunction with... Figure 1 The implementation steps of this invention will be described in detail. This embodiment uses a state machine model of an "intelligent greenhouse environment linkage control system" that includes multivariable coupling and a deep assignment chain as the object under test.
[0027] S1, Model parsing and serialization.
[0028] First, the system acquires the state machine diagram of the greenhouse system under test, extracting the states, transition conditions, and assignment operations. For the target test path, the system parses it into a queue of operation nodes arranged alternately according to the control flow timing.
[0029] Node 1 (Assignment): Turn on the heating rod, the temperature rises, mapped to... ;
[0030] Node 2 (Condition): High-temperature safety protection limitations, requirements ;
[0031] Node 3 (Assignment): Turning on the humidifier increases humidity, mapped to... ;
[0032] Node 4 (Condition): Multivariate coupling constraint (comfort index), requirements ;
[0033] Node 5 (Condition): Humidity limit and mold prevention restriction, requirements ;
[0034] in, The initial temperature setting. This is the initial humidity setting. and A controlled intermediate state variable derived from a root variable. This refers to the temperature displayed by the heating element as it changes temperature after being turned on. This shows the humidity level displayed by the humidifier as the humidity changes after it is turned on.
[0035] S2. Variable Range Initialization: The system allocates a global maximum physical value envelope, such as a real number range, for the initial input variables of the test path in memory. In this embodiment, a two-dimensional initial envelope is allocated: the initial temperature setpoint. Initial humidity setting value
[0036] S3, Reinforcement Learning Environment Construction: Combining Figure 2 This describes the construction of a dynamically evolving environment model. At each time step, the environment model encodes the state space in real-time, including the index of the currently processed operation node, the upper and lower bounds of the real-time intervals of each variable, and the vectorized features of the local constraints of the current node. The index of the currently processed node, i.e., the time step... The vectorized representations of node 2, the upper and lower bounds of the real-time intervals of each variable, and the local constraints of the current node are encoded into a state space. This step transforms the abstract test path into a Markov decision process perceptible to the agent by constructing a dynamically driven environment model.
[0037] The state space includes: the analytical representation of the current edge constraints on the root variables, such as variable coefficients, constant terms, and comparison symbols; and the satisfaction characteristics of the current variable interval with local constraints. For example, at node 2, if... Since the upper bound exceeds the constraint boundary and takes the value 40, the satisfaction feature is recorded as "violation".
[0038] The operation of tightening the boundaries of the variable range is defined as the action space.
[0039] Furthermore, the action space includes: tightening the lower bound of a specified variable upwards, tightening the upper bound of a specified variable downwards, and fixing the variable value to the current boundary or median. Each action includes a variable index and a tightening magnitude parameter. For example, in node 2, observing the current... If the upper bound is exceeded, the operation will be executed. The upper limit is reduced to 40.
[0040] Furthermore, the specific extraction method of the satisfaction feature mentioned in step S3 is as follows: The environment calculates the geometric distance between the variable interval boundary and the constraint boundary in real time based on the constraint logic of the current operation node, and normalizes it to a continuous value in the interval [-1, 1], which serves as the feature dimension guiding the agent to perceive the urgency of the constraint in the state space. The system maps the distance value to the interval [-1, 1]. If the distance is positive, it is determined to be out of bounds, and the feature value approaches -1, representing that the constraint is urgent and currently violated; if the distance is negative, it is determined to be safe, and the feature value approaches 1.
[0041] S4. Agent policy training: Multi-round training using the PPO algorithm.
[0042] The agent outputs a range tightening action based on the current state. The environment verifies the legality of the action in real time: if tightening causes the upper and lower bounds of the range to be inverted or violates the current constraints, the environment determines that there is no solution, imposes a high termination penalty, and forcibly ends the current round; if the range satisfies the constraints after tightening, the environment provides a stage reward and advances the node. At the same time, the environment triggers reverse constraint propagation, shrinking the effective range of the current intermediate variable and mapping it back to the envelope of the initial input variable in real time through the inverse operation of the assignment chain.
[0043] Furthermore, the reward mechanism employs a composite reward function: when an action causes interval inversion or violates constraints, a no-solution termination penalty is imposed; when successfully advancing to the next node, a stage completion reward is given, which adds 10 to the agent's current policy score, i.e., the stage reward, to strengthen the probability distribution of the interval contraction policy; a compactness reward is given based on the increase in the contraction rate to encourage the retention of the maximum legal space; and a small reward is deducted as a step cost for each step.
[0044] Furthermore, the transition condition node processing logic is as follows: After the agent outputs the tightening action, the environment verifies the intersection of the current interval and the transition condition in real time. If the intersection is not empty, that is, at node 2, the current interval is observed to be... If the upper bound is exceeded, a contraction operation will be performed. The upper bound shrinks to 40. If the range is updated to [10,40], satisfying the constraints, and the environment background triggers reverse constraint propagation and advances to the next node, then the interval is updated and the process advances; if it is empty, the process terminates.
[0045] Furthermore, the processing logic for the assignment operation node is as follows: the environment automatically triggers forward interval arithmetic, automatically calculates the new interval of the assigned variable using forward interval arithmetic, and automatically advances to the next node in the operation node queue, without requiring the agent to output a tightening action. At node 3, it is observed that the current environment... Automatically calculate And proceed to node 4.
[0046] S5. Test Case Generation and Reverse Mapping: Use the trained agent to perform forward deduction of the target test path.
[0047] Furthermore, the reverse constraint propagation mechanism is as follows: when dealing with deep intermediate variables (such as...) , During the tightening action of ), the environment utilizes "reverse constraint propagation" to map the effective contraction of the foreground back to the initial input variables in real time through the inverse operation of the assignment chain. For example, when When node 2 is tightened to 40, the reverse calculation is performed. The initial temperature envelope is tightened to [0, 30]. Upon reaching the end of the path, the system directly outputs the final feasible region of the initial variables: , .
[0048] Furthermore, the specific process of multidimensional orthogonal combination sampling described in step S5 is as follows:
[0049] (1) Feature point extraction: The system extracts the lower bound extreme points, mathematical median points, and upper bound extreme points for temperature and humidity respectively, forming two independent sets of discrete feature points:
[0050] Set of temperature feature points: ;
[0051] Set of humidity feature points: ;
[0052] (2) Cartesian product operation: The system performs the above feature point set operation. and Perform Cartesian product operation This involves exhaustively combining every element in the temperature set with every element in the humidity set in a fully permuted manner. Through this mathematical operation, the system ultimately calculates 3×3=9 two-dimensional vector tuples, each tuple representing a multi-dimensional variable interaction state;
[0053] (3) Test Case Output: The generated set of vector tuples will be output as a representative test case set, including boundary interaction test cases and one-sided extreme value test cases covering multiple variable interaction states. Possible test cases include:
[0054] Boundary interaction use cases (4): (0, 45), (0, 75), (30, 45), (30, 75). These use cases cover extreme interaction scenarios where two dimensional variables are simultaneously at the edge of their extreme values.
[0055] One-sided extreme value use cases (4): (0, 60), (30, 60), (15, 45), (15, 75). These use cases cover the system performance when one dimension is at an extreme value and the other dimension is at a normal value;
[0056] Typical internal use case (1): (15, 60). This use case is used to verify the basic system functionality when all variables are in the center of the feasible domain.
[0057] Through the Cartesian product sampling logic described above, this embodiment can automatically construct test inputs sufficient to cover complex Boolean combination conditions and key nodes of multi-level assignment chains, significantly improving test coverage efficiency for high-dimensional continuous variable spaces.
[0058] Preferably, this method is applied to embedded system state machine models with strong numerical computation, clear logical paths, and multi-layer assignment chains, which can significantly improve test path coverage and generation efficiency.
Claims
1. A test case generation method based on reinforcement learning, characterized in that, Includes the following steps: S1. Model parsing and serialization: Obtain the state machine diagram of the system under test, extract the state, transition conditions and assignment operations, and parse the path under test into a queue of operation nodes arranged alternately by transition condition nodes and assignment operation nodes according to the control flow timing. S2. Variable range initialization: Allocate a global wide range of values for the initial input variables of the test path. The system maintains the variable state in memory in the form of a range, which serves as the initial carrier of the dynamic forward execution sequence. S3. Construction of Reinforcement Learning Environment: Construct a Markov decision process environment model based on dynamic advancement reinforcement learning, and encode the current node index, the upper and lower bounds of the real-time intervals of each variable, and the local constraints of the current node into a state space using vectorized representations; wherein, the vectorized representation of the local constraints includes: variable coefficients, constant terms, comparison signs extracted after parsing the current edge constraint conditions into a linear expression about the root variable, and the satisfaction characteristics of the current variable interval with the local constraints; the boundary tightening operation for the variable interval is defined as the action space; S4. Agent Policy Training: The PPO algorithm is used to train the agent in multiple rounds. The agent moves forward along the queue of operation nodes and outputs interval tightening actions based on the current state. The environment verifies the legality of actions in real time. If tightening causes the upper and lower bounds of the interval to be reversed or violates the current constraints, the environment determines that there is no solution, imposes a high termination penalty, and forcibly ends the current round. If the interval satisfies the constraints after tightening, the environment provides a stage reward and advances the node. At the same time, the environment triggers reverse constraint propagation, shrinking the effective interval of the current intermediate variable and mapping it back to the envelope of the initial input variable in real time through the inverse operation of the assignment chain. The environment automatically triggers forward interval arithmetic, automatically calculates the new interval of the assigned variable according to the mathematical relationship, and automatically advances to the next node. The reward mechanism guides the agent to obtain the optimal interval tightening strategy that runs through the entire path. S5. Test Case Generation and Reverse Mapping: The trained agent performs forward inference on the target path. During the inference process, the reverse constraint propagation mechanism is used to map the effective contraction of each node back to the initial input variables in real time. When the end point of the path is reached, the final feasible domain of the initial input variables is extracted, and a test case set is generated by sampling from the final feasible domain. The sampling strategy includes: extracting the extreme value boundary of the final feasible domain as boundary value test cases; performing uniform random sampling or taking the mathematical median as internal value test cases within the final feasible domain; and performing orthogonal combination of the extracted multi-dimensional variable values to output a representative test case set.
2. The test case generation method based on reinforcement learning according to claim 1, characterized in that, The action space described in step S3 includes: tightening the lower bound of a specified variable upwards, tightening the upper bound of a specified variable downwards, and fixing the value of a specified variable to the boundary or median of the current interval; wherein each action is accompanied by a variable index and a tightening magnitude parameter.
3. The test case generation method based on reinforcement learning according to claim 1, characterized in that, The specific extraction method of the satisfaction feature mentioned in step S3 is as follows: the environment calculates the geometric distance between the variable interval boundary and the constraint boundary in real time according to the constraint logic of the current operation node, and normalizes it into a continuous value in the interval [-1, 1], which serves as the feature dimension in the state space to guide the agent to perceive the urgency of the constraint.
4. The test case generation method based on reinforcement learning according to claim 1, characterized in that, The reward mechanism described in step S4 employs a composite reward function, including: Unsolvable termination penalty: When a tightening action causes the upper and lower bounds of the interval to be reversed or local constraints are violated, a negative reward is given and the turn is terminated; Phased completion reward: A positive reward is given when the tightened interval falls completely into the current legal domain and successfully advances to the next node; Tightness reward: Positive rewards are given based on the degree of increase in the contraction rate of the variable interval, and excessive tightening behavior is punished; Step cost: A small negative reward is deducted for each tightening step to encourage algorithm convergence.
5. The test case generation method based on reinforcement learning according to claim 1, characterized in that, The transition condition node processing logic described in step S4 is as follows: After the agent outputs the tightening action, the environment verifies the intersection of the current interval and the transition condition in real time. If the intersection is empty, it is determined to be illegal and terminated. If the intersection is not empty, the interval is updated and the process is advanced to the next node.
6. The test case generation method based on reinforcement learning according to claim 1, characterized in that, The assignment operation node processing logic described in step S4 is as follows: automatically calculate the new interval of the variable after assignment using forward interval arithmetic, and automatically advance it to the next node in the operation node queue.
7. The test case generation method based on reinforcement learning according to claim 1, characterized in that, The reverse constraint propagation mechanism described in step S5 is as follows: when processing the tightening action of deep intermediate variables, the interval contraction of the variable is simultaneously mapped back to the corresponding initial input variable envelope in real time through the inverse operation of the assignment chain relationship.
8. The test case generation method based on reinforcement learning according to claim 1, characterized in that, The specific process of multidimensional orthogonal combination sampling described in step S5 is as follows: 1) Feature point extraction: targeting the final feasible region The initial input variable is extracted independently. Set of discrete feature points of each variable , recorded as , It contains 3 discrete feature points, among which This is the lower bound extreme point of the final feasible region. The mathematical mean point of the final feasible region. This is the extreme point of the upper bound of the final feasible region; (2) Cartesian product operation: for the set of discrete feature points of all variables Perform Cartesian product operation From each set Select one element at a time and perform exhaustive permutations and cross combinations to generate a multidimensional vector tuple set. (3) Test case output: The generated set of vector tuples is output as a representative test case set, including boundary interaction test cases and one-sided extreme value test cases covering multiple variable interaction states.
9. The test case generation method based on reinforcement learning according to claim 1, characterized in that, During training, the PPO algorithm controls the policy update magnitude within a fixed range by pruning the probability ratio of the new and old policies in the objective function to upper and lower limits.
10. The test case generation method based on reinforcement learning according to claim 1, characterized in that, The method is applied to the scenario of automatically generating test cases for embedded system state machine models with numerical variables, multi-level assignment chains, and Boolean combination conditions.