A Test Case Selection Method for Deep Reinforcement Learning Software Based on State Level Coverage

Abstract

This invention discloses a test case selection method for deep reinforcement learning software based on state level coverage. This method is the first to propose a coverage criterion measurement design for deep reinforcement learning software and to select test cases based on state coverage criteria. By defining a primary exploration space, discretizing the continuous state space, designing state coverage criteria based on state boundaries and k-multiple regions, and using these coverage criteria to guide test case selection, this method improves testing efficiency and more effectively exposes erroneous behaviors in deep reinforcement learning software. This method can serve as a metric for measuring the adequacy of deep reinforcement learning software testing and can also guide test case selection, thus contributing to improved reliability and security of deep reinforcement learning software.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence software testing technology in software engineering applications, and particularly relates to a deep reinforcement learning software test case selection method based on state level coverage. Background Technology

[0002] Coverage criteria are a measure of test adequacy and a method for evaluating software quality. These criteria indicate what program structure should be executed to constitute a comprehensive test, provide termination conditions for testing (e.g., coverage has been achieved), and quantify the adequacy of test inputs. In the field of software testing, coverage criteria are proposed to approximate the internal behavior of software. Different test cases are selected based on different partitions of the software structure, thus increasing the likelihood of covering different parts of the software structure. For example, in traditional software, statement coverage ensures that every statement in the program is executed at least once; branch coverage attempts to cover all branches in the software. These two coverage criteria are derived from the statements and branches of the software structure.

[0003] In deep learning testing, many coverage criteria have been proposed based on neuron coverage, the idea of ​​which is to maximize the number of activated neurons. For example, hierarchical coverage represents the proportion of the total number of the k neurons with the highest activation values ​​in each layer to the total number of neurons, and t-way combination coverage is the ratio of the t-way of activated neuron combinations to the total number of combinations. Existing deep learning coverage criteria rely heavily on neuron coverage but do not take into account any characteristics of reinforcement learning, and therefore cannot measure the adequacy of testing deep reinforcement learning systems.

[0004] Deep reinforcement learning is a sequential decision-making problem where an agent needs to make consecutive decisions. Generally, this problem is modeled as a Markov Decision Process (MDP). An MDP is a 4-tuple (S; A; P; R), where S is the state space, A is the action space, P is the transition probability, and R is the reward function. A state is an observation of the environment, and an action is a possible decision the agent can make. Transition probabilities are the rules governing transitions between states. The reward is the feedback that measures the success or failure of an action. Given a state, the agent chooses an action at a certain time and receives a reward. The environment transitions to a new state, and the agent chooses the next action. The environment and the agent continue to interact until the agent reaches a terminal state. States and actions define the interaction process between the agent and the environment. Therefore, it is natural to design state coverage criteria according to the different granularities of deep reinforcement learning systems.

[0005] In recent years, software testing for machine learning has become a research hotspot, with a series of testing methods being proposed. However, deep reinforcement learning software differs significantly from other machine learning software. For example, deep reinforcement learning models need to interact with complex environments during training, leading to more diverse reasons for failure. Furthermore, the training data and samples are generated during training, and the training data are highly correlated, resulting in longer runtimes and greater training difficulty. Additionally, the expected output is typically a decision trajectory, making test case generation more challenging. These differences limit the direct application of existing machine learning testing methods to deep reinforcement learning software testing. Deep reinforcement learning is based on Markov decision processes, where each round is a time series. Each sequence starting from the initial state can be viewed as a transition path, including a series of states and state transitions. By comparing the path coverage differences between two test cases during execution, their different error detection capabilities can be reflected. Designing appropriate test sufficiency metrics and test case selection methods for deep reinforcement learning software is a key focus and challenge in reinforcement learning software testing. Summary of the Invention

[0006] The purpose of this invention is to address the structural differences between deep reinforcement learning software and traditional software. Deep reinforcement learning software is a type of software that makes sequential decisions based on Markov decision processes. Coverage criteria are evaluation standards for software test adequacy, used in software testing to approximate the internal behavior of software. Existing deep learning software coverage criteria are mainly based on neuron coverage, failing to reflect the characteristics of deep reinforcement learning software. Therefore, existing coverage criteria cannot measure the test adequacy of deep reinforcement learning software. Therefore, this invention, based on Markov decision processes, designs a coverage criterion for deep reinforcement learning software at the state level to measure the test adequacy of deep reinforcement learning software.

[0007] Another objective of this invention is to guide test case selection for deep reinforcement learning software. Test cases (initial states) are selected by measuring the similarity between paths in two rounds. Path coverage includes the state sequence and state pairs corresponding to state transitions in each round of the algorithm. During deep reinforcement learning software testing, we aim to identify test cases that cover as many paths as possible during software execution. This invention utilizes two similarity metrics in deep reinforcement learning software to select independent paths—paths containing at least one edge that has never appeared in other independent paths, with a similarity of 0 to previously encountered paths. When the maximum path coverage of the software is reached, the most defects are covered, thus selecting test cases and improving testing efficiency.

[0008] The technical solution of this invention is: a method for selecting test cases for deep reinforcement learning software based on state level coverage. This method mainly includes two parts: the design of the state coverage criterion metric and the selection of test cases based on the coverage criterion, comprising the following steps:

[0009] Step 1) For the application scenario of air combat system, select a deep reinforcement learning software to train an agent to control our aircraft in the air combat environment of both sides. The agent is required to complete the air combat confrontation mission with the enemy aircraft. We use it as the benchmark model.

[0010] Step 2) Design a coverage criterion based on state boundaries. Train the agent and extract the values ​​of the state variables: the coordinates of our aircraft (p1.pos.x, p1.pos.z, p1.pos.y), the coordinates of the enemy aircraft (p2.pos.x, p2.pos.z, p2.pos.y), the speed of our aircraft (p1.linear_speed), the speed of the enemy aircraft (p2.linear_speed), our health level (p1.health_level), the health level of the enemy aircraft (p2.health_level), the linear velocities of our aircraft along the three axes (p1.v_move.x, p1.v_move.z, p1.v_move.y), and the linear velocities of the enemy aircraft along the three axes (p2.v_move.x, p2.v_move.z, p2.v_move.y). Compare the values ​​of each state dimension to obtain the maximum and minimum values ​​of each state dimension. Obtain the maximum and minimum values ​​of each state dimension during the testing phase of the agent in the same way.

[0011] Step 3) Calculate the proportion of the state dimension boundary in the testing phase of our aircraft intelligent agent to the state dimension boundary in the training phase, and obtain the state boundary coverage criterion for each dimension of the state quantity. Sum and average or multiply the state boundary coverage rates of each state dimension to obtain the state boundary coverage criterion.

[0012] Step 4) Design the k-multi-region state coverage criterion. First, divide the maximum and minimum boundaries of each dimension of the state variables extracted by the training agent into k parts, and statistically analyze the coverage of the state dimensions of our aircraft agent during the testing phase to the training phase, thus obtaining the k-multi-region state coverage criterion.

[0013] Step 5) Record the states of the fighter jet and adjacent state pairs in the combat rounds to obtain discretized states. The initial state of each round during software operation is used as a test case set. Calculate the similarity based on state occurrence and state transition, and compare the current initial state with the state coverage of each previous round.

[0014] Step 6) Calculate the initial state with a similarity of 0 between the two states. Until the specified number of test rounds is completed, select all initial states with a similarity of 0 to the previous rounds during the software operation. That is, select the test cases with the greatest state difference and the greatest improvement in state coverage.

[0015] This invention proposes a test case selection method for deep reinforcement learning software based on state level coverage. Compared with existing deep reinforcement learning software testing methods, its advantages are as follows: This invention specifically addresses the sufficiency metrics proposed for deep reinforcement learning software, utilizing state coverage information to guide test case selection. This significantly improves testing efficiency, selecting more failure test cases compared to traditional random generation methods. Furthermore, the selected failure test cases exhibit greater diversity than those selected randomly. Therefore, this invention can select more and more diverse failure test cases, thereby improving the efficiency of testing deep reinforcement learning software. Attached Figure Description

[0016] Figure 1 This is an example of the state level coverage criterion for deep reinforcement learning software.

[0017] Figure 2 It is a deep reinforcement learning software test case selection method based on state level coverage. Detailed Implementation

[0018] The invention will now be further described with reference to the accompanying drawings. Figure 1 This paper provides an example of a state level coverage criterion for agents in an air combat system. Figure 1 It is a simplified relationship between states and actions. Assume that there are 4 discrete states in this environment. The dimensions of each state include coordinate position, velocity, health value and linear velocity. The agent has 8 selectable discrete actions, including throttle (T), flaps (F), ailerons (A), elevator (E), rudder (R), aircraft machine gun switch control (N), air-to-air missile launch control (M), and friendly aircraft health value setting (H).

[0019] For combat round 1: States s0 and s3 are covered, and the state level coverage is 1 / 2.

[0020] For combat round 2: States s1, s2, and s3 are covered, resulting in a state level coverage of 3 / 4.

[0021] Combined with appendix Figure 2 This paper provides a detailed explanation of the state-level-based sufficiency evaluation criterion for deep reinforcement learning software testing proposed in this invention.

[0022] Design a state level coverage criterion. Train the agent and extract the values ​​of the state variables: the coordinates of our aircraft (p1.pos.x, p1.pos.z, p1.pos.y), the coordinates of the enemy aircraft (p2.pos.x, p2.pos.z, p2.pos.y), the speed of our aircraft (p1.linear_speed), the speed of the enemy aircraft (p2.linear_speed), our health level (p1.health_level), the health level of the enemy aircraft (p2.health_level), the linear velocities of our aircraft along the three axes (p1.v_move.x, p1.v_move.z, p1.v_move.y), and the linear velocities of the enemy aircraft along the three axes (p2.v_move.x, p2.v_move.z, p2.v_move.y). Each state includes these 16 dimensions, represented by a vector s = (x1, x2, ..., x...). n ) represents the observed value of the state.

[0023] Specifically, make low xi and high xi As the state dimension x i Minimum range value and maximum range value (that is, x) i ∈X i =[low xi high xi [, i∈I={1,2,...,n}). Minimum and maximum range values ​​X i The Cartesian product of i∈I is denoted as П. i∈I X i =X1×X2×…×X n Let be the primary exploration space of states, where is given a state s if and only if s = (x1, x2, ..., xn). n )∈Π i∈I X i We say that the intelligent agent falls into the main exploration space.

[0024] For any state dimension x i In this context, state boundary coverage measures the degree to which the continuous state space (including upper and lower boundary values) explored by our fighter jet agent during the training phase is covered by the main space of the test state. We use the average of the boundary values ​​from N trials, denoted as:

[0025] and

[0026] (x i ∈AvgX i =[AvgLow xi ,AvgHighxi ], i∈I={1,2,...,n}), state dimension x i The average range is The state boundary coverage of the state dimension is defined as follows:

[0027]

[0028] For each dimension, the state boundary coverage can be calculated, and the average value is taken. As a state boundary cover in deep reinforcement learning software, it is defined as:

[0029]

[0030] Further define the state boundary cover of the Cartesian product space of the deep reinforcement learning software as:

[0031]

[0032] State boundary coverage describes state-level coverage from the perspective of state boundaries. This criterion measures the state coverage rate as a whole, but weakens the effect of distribution within the state.

[0033] For example, when our fighter jet agent completes an air combat mission against an enemy fighter jet, taking the coordinates of our aircraft {p1.pos.x, p1.pos.z, p1.pos.y} as an example, there are three states: s1 = {3, 0.8, 32}, s2 = {1, 0.1, 64}, and s3 = {4, 0.9, 20}. The main exploration space of our fighter jet agent is [1, 4] × [0.1, 0.9] × [20, 64]. During the testing phase, the spaces explored by the states are [1, 2], [0.4, 0.6], and [30, 50].

[0034] The state boundary coverage rate for each dimension is... and The state boundary coverage of the agent is:

[0035]

[0036] The state boundary coverage of the agent's Cartesian product space is:

[0037]

[0038] To comprehensively cover the main exploration space of our fighter jet's intelligent agent, we will define each state dimension X i The range i ∈ I is divided into k parts, and each part is required to be covered by a given state. We call this coverage k-multi-region state coverage.

[0039] Given a state k-multiregion state dimension coverage measures one dimension of the state, state dimension X. i The comprehensiveness of the coverage of i∈I. To quantify this, we will define the range of the state dimension X. i If i∈I, divide it into k (k>0) equal parts. Let D be the integer part of I. i =(d i1 ,d i2 ,...,d ik ), i∈I. If the state dimension value x i Belongs to d ij Let i ∈ I, j ∈ J = {1, 2, ..., k}; we call the j-th part the state. cover.

[0040] Multi-region D i Cartesian product П of i∈I i∈I D i =D1×D2×…×D n The state is defined as the discrete primary exploration space. Further, the k-multi-region state coverage of the agent's dimensional product space is defined as:

[0041]

[0042] Among them, (d) 1j ,d 2j ,...,d nj )∈Π i∈I D i .

[0043] Taking the three states mentioned above as examples, the range of the three dimensions of the state is divided into four regions: D1 = (1, 1.75, 2.5, 3.25, 4), D2 = (0.1, 0.3, 0.5, 0.7, 0.9), and D3 = (20, 31, 42, 53, 64). During the agent's testing phase, there are two states: s1 = {2.8, 0.2, 24} and s2 = {3.5, 0.8, 50}. The regions that each state dimension falls into are counted. The first dimension of the state, 2.8 ∈ (2.5, 3.25) and 3.5 ∈ (3.25, 4), belong to two regions. The second dimension of the state, 0.2 ∈ (0.1, 0.3) and 0.8 ∈ (0.7, 0.9), belong to two regions. The third dimension of the state, 55 ∈ (53, 64) and 50 ∈ (53, 64), belong to the same region.

[0044] The multi-region state coverage of the agent is:

[0045] State-level coverage criteria measure the extent to which an agent explores its environment. A state-level coverage criterion is the percentage of states covered during the testing phase, aiming to cover the state transition processes of the environment and explore a sufficient state space. These coverage criteria monitor the internal behavior of deep reinforcement learning software at the state level.

[0046] State coverage information guides test case selection. Coverage criteria can guide the selection of test cases for deep reinforcement learning software. To compare the error detection capabilities of two test cases, we propose two path coverage similarity metrics based on the software execution process, further demonstrating the effectiveness of the coverage criteria proposed from the perspective of the agent's state. Deep reinforcement learning algorithms are based on Markov decision processes, where each round is a time series. Each sequence starting from the initial state can be viewed as a transition path, including a series of states and state transitions. By comparing the differences in path coverage between two test cases during execution, their different error detection capabilities are reflected. The two path similarity metrics are defined as follows:

[0047] State-based path similarity is used. Continuous states are discretized, the state sequence for each round is statistically analyzed, and the Jaccard index between the state sequences of two rounds is calculated to represent the difference in path coverage between two test cases.

[0048]

[0049] Based on the path similarity of transitions, and due to the Markov property, the current state depends only on the previous time step and is independent of any states prior to that time step. A transition from one state to another with a certain probability is called a step, represented by a state pair consisting of these two states. We use multiple state pairs to represent the transition path. We record the number of state pairs in each round and calculate the Jaccard indices for the number of state pairs between two rounds. This index considers the order of state transitions based on the state sequence.

[0050]

[0051] Specifically, let's illustrate how to use the above two similarity metrics to measure the similarity between the execution paths of two test cases of our fighter jet's intelligent agent. Assume two sequences: sequence x: s0, s1, s2, s3, where s0 is the initial state; sequence y: s1, s2, s3, s1, s4, s5, where s1 is the initial state.

[0052] The union of the states has 6 elements: s0, s1, s2, s3, s4, s5; the intersection of the states has 3 elements: s1, s2, s3. The calculation yields...

[0053] Sequence 1 state pairs: (s0,s1), (s1,s2), (s2,s3); Sequence 2 state pairs: (s1,s2), (s2,s3), (s3,s1), (s1,s4), (s4,s5). The union of the state pairs has 6 elements: (s0,s1), (s1,s2), (s2,s3), (s3,s1), (s1,s4), (s4,s5). The intersection of the state pairs has 2 elements: (s1,s2), (s2,s3). The calculation yields... This indicates that test cases s0 and s1 have similar path coverage.

[0054] Furthermore, test cases (initial states) are selected by measuring the similarity between the paths of enemy and friendly aircraft during combat, considering the path coverage during software operation. This path coverage includes the state sequence and state pairs corresponding to state transitions in each round. During testing, we aim to find test cases that cover as many paths as possible. Independent paths are selected using two similarity metrics; these are paths containing at least one edge that has never appeared in other independent paths, and their similarity to previously encountered paths is 0. If maximum path coverage is achieved, the most defects are covered, proving the effectiveness of the path coverage. This allows us to select test cases and improve testing efficiency.

[0055] The above description provides a detailed explanation of the deep reinforcement learning-based software test case selection method of this invention. However, it is clear that the specific implementation of this invention is not limited thereto. For those skilled in the art, various obvious modifications made to this invention without departing from the spirit and scope of the claims are within the protection scope of this invention.

Claims

1. A method for selecting test cases for deep reinforcement learning software based on state level coverage, characterized in that, The steps include the following: Step 1) For the application scenario of air combat system, select a deep reinforcement learning software to train an agent to control our aircraft in the air combat environment of both sides. The agent is required to complete the air combat confrontation mission with the enemy aircraft and use it as the benchmark model. Step 2) Design a coverage criterion based on state boundaries; train the agent and extract the values ​​of state variables: including the coordinate positions of our aircraft p1.pos.x, p1.pos.z, p1.pos.y, the coordinate positions of the enemy aircraft p2.pos.x, p2.pos.z, p2.pos.y, the speed of our aircraft p1.linear_speed, the speed of the enemy aircraft p2.linear_speed, the health value of our aircraft p1.health_level, the health value of the enemy aircraft p2.health_level, the linear velocities of our aircraft along the three axes p1.v_move.x, p1.v_move.z, p1.v_move.y, and the linear velocities of the enemy aircraft along the three axes p2.v_move.x, p2.v_move.z, p2.v_move.y. Compare the values ​​of each state dimension to obtain the maximum and minimum values ​​of each state dimension; obtain the maximum and minimum values ​​of each state dimension of our aircraft agent during the testing phase in the same way. Step 3) Calculate the proportion of the state dimension boundary in the test phase of our aircraft intelligent agent in the state dimension boundary in the training phase to obtain the state boundary coverage criterion for each state dimension; sum and average or multiply the state boundary coverage rates of each state dimension to obtain the state boundary coverage criterion. Step 4) Design k-multi-region state coverage criteria; First, divide the maximum and minimum boundaries of each state dimension extracted by the training agent into k parts, and statistically analyze the coverage of the state dimension of our aircraft agent during the test phase to the training phase, so as to obtain the state multi-region state coverage criteria. Step 5) Record the states of the fighter jet and adjacent state pairs in the combat rounds to obtain discretized states. The initial state of each round during software operation is used as a test case set. Calculate the similarity based on state occurrence and state transition, and compare the current initial state with the state coverage of each previous round. Step 6) Calculate the initial state with a similarity of 0 between the two states. Continue this process until the specified number of test rounds is completed. Select all initial states with a similarity of 0 to the previous rounds during the software operation. In other words, select the test cases with the greatest state difference and the greatest improvement in state coverage.

2. The method for selecting test cases for deep reinforcement learning software based on state level coverage according to claim 1, characterized in that: Using vectors The observed value represents the state; specifically, let and They are respectively state dimensions The minimum and maximum range values, the numerical range of the state dimension range are denoted as... ; Cartesian product notation , represents the primary exploration space of the state, given a state s If and only if It is said that the intelligent agent falls into the main exploration space.

3. The method for selecting test cases for deep reinforcement learning software based on state level coverage according to claim 2, characterized in that: For any state dimension In this context, state boundary coverage measures the degree to which the continuous state space explored by our fighter jet agent during the training phase is covered by the space of the test state; it is denoted as the average of the boundary values ​​from N trials: and ; in, , State Dimension The average range is State dimension State boundary coverage is defined as: ； For each dimension, calculate the state boundary coverage range and take the average value. As a state boundary cover in deep reinforcement learning software, it is defined as: ； Further define the state boundary cover of the Cartesian product space of the deep reinforcement learning software as: 。 4. The method for selecting test cases for deep reinforcement learning software based on state level coverage according to claim 1, characterized in that: To fully cover the exploration space of our fighter jet's intelligent agent, the numerical range of each state dimension will be defined. The scope is divided into k Each part is required to be covered by a given state; this coverage is called... k - Multi-region status coverage.

5. The method for selecting test cases for deep reinforcement learning software based on state level coverage according to claim 4, characterized in that: Given a state , k - Multi-region state dimension coverage measures the numerical range of one of the dimensions of a state. The comprehensiveness of the coverage; for quantification, the numerical range of the state dimension. Divided into k ( k > 0) divided into equal parts; denoted as ; If the value of the state dimension belongs to The first j Partially in a state Coverage; Multiple regions Cartesian product It is defined as a discrete state exploration space; further defined as the dimensional product space of the agent. k -Multi-region status coverage is as follows: , in, .

6. The method for selecting test cases for deep reinforcement learning software based on state level coverage according to claim 1, characterized in that: Deep reinforcement learning algorithms are based on Markov decision processes, where each round is a time series. Each sequence starting from the initial state is regarded as a transition path, including a series of states and state transitions. By comparing the differences in path coverage between two test cases during execution, their different error detection capabilities are reflected. There are two types of path similarity: state-based path similarity and transition-based path similarity.

7. The method for selecting test cases for deep reinforcement learning software based on state level coverage according to claim 6, characterized in that: State-based path similarity; Discretize the continuous states, count the state sequence for each round, and calculate the Jaccard index for the state sequences of two rounds to represent the difference between the path coverage of two test cases. 。 8. The method for selecting test cases for deep reinforcement learning software based on state level coverage according to claim 6, characterized in that: Based on the path similarity of the transfer; The current state is only related to the previous time step and has nothing to do with any state before the previous time step; a jump from one state to another with a certain probability is called a step transition and is represented by a state pair consisting of these two states; multiple state pairs are used to represent the transition path; record the number of state pairs in each round and calculate the Jaccard indices of the number of state pairs in two rounds. This metric takes into account the state transition order based on the state sequence; 。 9. The method for selecting test cases for deep reinforcement learning software based on state level coverage according to claim 1, characterized in that: Independent paths are selected using two similarity metrics. These paths contain at least one edge that has never appeared in any other independent path, and their similarity to previously appearing paths is 0. The maximum path coverage is achieved when the most defects are covered, proving the effectiveness of the path coverage. This allows us to select test cases and improve testing efficiency.

Images