A software testing method based on model-free reinforcement learning

Abstract

The present application relates to the field of aviation machine learning, and is a software testing method for model-free reinforcement learning, including constructing a test condition set C according to the function set to be tested, constructing an observation space O of the tested software, constructing a test action space A, and forming a test simulation environment; constructing a reward function for rewarding the execution effect of the action sequence; constructing an action execution drive interface that can act on the software itself; constructing a test strategy network that can self-learn; constructing a model-free reinforcement learning loss function; driving the test strategy network to generate software operation actions in real time and act on the software through the action execution drive interface, calculating and recording data tuples, and storing them in a data set; the reinforcement learning loss function uses the data set to optimize the test strategy network parameters. It has the technical effects of high test efficiency, short test time, and high accuracy.

Description

Technical Field

[0001] The present application belongs to the field of aviation machine learning, and in particular, relates to a software testing method for model-free reinforcement learning. Background Art

[0002] Black box integration testing of software refers to the use of relevant test cases to generate software operation action sequences after the software development is basically completed, to identify whether there are errors in the actual output of the software or whether the error between the actual output and the expected output meets the specified tolerance range. Therefore, the test cases used to generate software operation action sequences need to fully cover the full life cycle of the software, which requires manual design of a large number of test cases.

[0003] Existing software testing is a sampling test. Although the test cases are designed to be as comprehensive as possible, it is still impossible to traverse the usage form of the software throughout its life cycle. Software errors or failures often show a temporal correlation, that is, there are no problems when several independent test cases are tested separately, but these test cases may have problems when tested in a certain combination order. If the problem needs to be fully tested, the required traversal range needs to increase exponentially. For example, 10 steps of software operation actions for 2 operation actions require traversal of 2^10, which has low test efficiency and inaccurate test results. Therefore, how to conduct software black box integration testing more efficiently and accurately is a problem that needs to be solved. Summary of the invention

[0004] The purpose of this application is to provide a software testing method based on model-free reinforcement learning to solve the problems of low efficiency and insufficient accuracy in software black box integration testing in the prior art.

[0005] The technical solution of the present application is: a software testing method of model-free reinforcement learning, comprising constructing a test condition set C according to a function set to be tested, constructing an observation space O of the tested software, constructing a test action space A, and forming a test simulation environment; constructing a reward function for rewarding the execution effect of an action sequence; constructing an action execution drive interface that can act on the software itself; constructing a test strategy network that can self-learn; constructing a model-free reinforcement learning loss function; driving the test strategy network to generate software operation actions in real time and act on the software through the action execution drive interface, calculating and recording data tuples, and storing them in a data set; the reinforcement learning loss function uses the data set to optimize the test strategy network parameters so that the reward of the reinforcement learning loss function is maximized and the loss is minimized, optimizing iter cycles, generating a new test strategy network, and using the new test strategy network to drive the software until the reinforcement learning loss function converges; recording all errors and mistakes generated in the process, and generating a standard test document.

[0006] Preferably, the reward function is constructed by obtaining software fault criteria and forming a crash reward factor, obtaining a deviation between an actual output and an expected output, obtaining a longest execution step length, and a negative reward for a single-step time consumption.

[0007] Preferably, the test strategy network is:

[0008] a t =f(O 1：t , C, a 1：t-1 )

[0009] Among them, O 1:t is the historical state of the software under test; C is the embedded test condition; a 1：t-1 is the operation from step 1 to step t-1; a t is the operation action of step t.

[0010] Preferably, the reward function is:

[0011] r t =R(d fault , d defect , d time )

[0012] Among them, d fault A reward for software errors or even crashes;d defect The reward for the deviation between the actual output and the expected output of the software; d time It is a time consumption reward, which is a constant negative reward.

[0013] Preferably, when the software runs normally, the crash reward obtains a negative reward; when the software has an error or even crashes, the crash reward obtains a positive reward; when no deviation occurs, the deviation reward obtains a negative reward; when there is a deviation between the actual output of the software and the expected output, the deviation reward obtains a positive reward; when there is no problem with the current action sequence, the time reward value is small; when there is a problem with the current action sequence, the time reward value is large.

[0014] Preferably, the deviation reward between the actual output of the software and the expected output is

[0015] .

[0016] Preferably, after a batch of data is collected, the reward value function of each collected moment is calculated according to a certain discount rate using the reward at each moment, which is

[0017] ,

[0018] Then reinforcement learning is performed, and the reinforcement learning loss function is

[0019]

[0020] in, Used to depict the differences between the new and old test strategy networks;

[0021] Aπ old (s,a) is the advantage function of the network based on the old test strategy;

[0022] ε is used to limit The range of changes;

[0023] Clip is used to ensure that the difference between the new and old test strategy networks is within the set range. If less than 1-ε, then take 1-ε; if If it is greater than 1+ε, then take 1+ε; if it is between the two, then take

[0024] Preferably, the reinforcement learning training method includes: according to the current scene state, driving the test condition set C through the test strategy network through the action execution drive interface; obtaining the test action output value of the next scene state of the test condition set C through the observation space O and the test action space A; determining the deviation between the actual output of the software and the expected output, and obtaining the deviation reward through the reward function; determining whether an error or crash occurs when executing the step, and obtaining the crash reward through the reward function; determining the time consumption of executing the step, and obtaining the time consumption reward; determining whether a single test is completed, if not, continuing to run according to the current test strategy network, if it is completed, initializing the test strategy network; obtaining the observation space O data, test action output value, reward value and completion status of the current step of the current state, storing them in data tuples, and storing them in a data set after recording a certain amount of data tuples; learning the data in the collected data set through the reinforcement learning loss function, obtaining the cumulative reward for each action sequence, and optimizing the test strategy network through the maximum reward value obtained; using the new test strategy network to drive the test condition set C again.

[0025] Preferably, when learning a new test strategy network, the new test strategy network parameters are first copied to the old test strategy network, and then the new test strategy network is learned.

[0026] As a specific implementation method, a model-free reinforcement learning software testing system includes: a test environment establishment module, which is used to form a test simulation environment by constructing a test condition set C, an observation space O, and a test workspace A; a reward module, which is used to construct a reward function by obtaining software fault criteria and forming a crash reward factor, obtaining the deviation between the actual output and the expected output, obtaining the longest execution step length and the negative reward for the single-step time consumption; a test strategy module, which is used to construct a test strategy network capable of self-learning; a loss learning module, which is used to construct a model-free reinforcement learning loss function; an automated testing module, which is used to obtain the running data of each step and form a data set by inputting the test strategy network into the test simulation environment, and the reinforcement learning loss function uses the data set to optimize the test strategy network parameters to form a new test strategy network for cyclic testing.

[0027] The present application discloses a software testing method of model-free reinforcement learning, which sets up a test simulation environment, and sets up a test strategy network and reward function that can self-learn. The reinforcement learning loss function uses the model-free reinforcement learning technology to automatically optimize the test strategy network, continuously outputs test action sequences, and evaluates the cumulative reward of each action sequence in real time, updates the existing test strategy, and realizes accurate and efficient testing of the tested program.

[0028] Preferably, by setting the fault criterion to evaluate the crash reward factor, the deviation between the actual output and the expected output, the longest execution step length and the negative reward of the single-step time consumption, the execution degree of each action sequence can be accurately obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solution provided by the present application, the following is a brief introduction to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present application.

[0030] Figure 1 This is a schematic diagram of the overall process of this application;

[0031] Figure 2 This is a schematic diagram of the test strategy network and valuation network structure of one of the applications;

[0032] Figure 3 This is the iterative test effect diagram for this application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the implementation of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below in conjunction with the drawings in the embodiments of this application.

[0034] A software testing method based on model-free reinforcement learning, such as Figure 1 As shown, including:

[0035] Step S100, obtaining a function set to be tested, constructing a test condition set C, constructing an observation space O of the software under test, constructing a test action space A, and forming a test simulation environment;

[0036] Step S200, constructing a reward function for rewarding the execution effect of the action sequence;

[0037] The reward function obtains software fault criteria and forms a crash reward factor, obtains the deviation between the actual output and the expected output, obtains the longest execution step length, and the negative reward of the single-step time consumption.

[0038] The software fault judgment criteria are obtained through the existing running data of the tested software. By setting the crash reward factor, the deviation between the actual output and the expected output, and the negative reward for the single-step time consumption, the test situation of the tested software can be accurately evaluated.

[0039] Preferably, the reward function is:

[0040] r t =R(d faul t,d defect , d time )

[0041] Among them, d fault A reward for software errors or even crashes;d defect The reward for the deviation between the actual output and the expected output of the software; d time It is a time consumption reward, which is a constant negative reward.

[0042] When the software runs normally, the crash reward gets a negative reward; when the software has an error or even crashes, the crash reward gets a positive reward; when there is no deviation, the deviation reward gets a negative reward; when there is a deviation between the actual output of the software and the expected output, the deviation reward gets a positive reward; when the current step action sequence takes less time, the time reward value is smaller; when the current step action sequence takes more time, the time reward value is larger. By using this reward function for calculation, the larger the reward value obtained, the more errors are found; the smaller the reward value obtained, the fewer errors are found. When learning through the reinforcement learning function, the test strategy network is optimized based on the data with the largest reward value.

[0043] Preferably, the deviation reward between the actual output of the software and the expected output is

[0044] .

[0045] The test condition coefficient γ varies according to different test condition systems. Among multiple input conditions, the greater the correlation with the corresponding input, the greater the test condition coefficient γ, and vice versa.

[0046] Step S300, constructing an action execution driver interface that can act on the software itself;

[0047] The software operation actions and their parameters generated by the test strategy can act on the software itself. This interface can be a background API, or it can be a screen coordinate click and keyboard io operation or external data stimulus. In short, this step realizes that the actions generated by the test strategy neural network can actually operate the software.

[0048] Step S400, constructing a test strategy network capable of self-learning;

[0049] Preferably, the test strategy network is:

[0050] a t =f(O 1：t , C, a 1：t-1 )

[0051] Among them, O 1:t is the historical state of the software under test; C is the embedded test condition; a 1：t-1 is the operation from step 1 to step t-1; a t is the operation action of the tth step; f can be represented by any form of RNN deep neural network.

[0052] The network can choose LSTM, GRU and other recurrent neural networks, O 1:t The input is the historical state of the software under test. C is the embedded test condition. 1：t-1 is the operation output from step 1 to step t-1, a t For the operation action in step t, the network weights are randomly initialized. If actor-critic model-free reinforcement learning is used, it is also necessary to build a critic valuation neural network, which can be isomorphic to the test network, but the output needs to be replaced with the advantage evaluation value advantage or reward evaluation value V.

[0053] By acquiring the vertical state of the software under test and the operation actions from step 1 to step t-1, the software under test can be controlled accurately and stably.

[0054] Step S500, constructing a model-free reinforcement learning loss function;

[0055] Preferably, after a batch of data is collected, the reward value function of each collected moment is calculated according to a certain discount rate using the reward at each moment, which is

[0056] ,

[0057] Then reinforcement learning is performed, and the reinforcement learning loss function is

[0058]

[0059] in, Used to depict the differences between the new and old test strategy networks;

[0060] Aπ old (s,a) is the advantage function of the network based on the old test strategy;

[0061] ε is a small hyperparameter used to limit The range of changes;

[0062] Clip is used to ensure that the difference between the new and old test strategy networks is within the set range. If less than 1-ε, then take 1-ε; if If it is greater than 1+ε, then take 1+ε; if it is between the two, then take

[0063] Step S600, the driving test strategy network generates software operation actions in real time and acts on the software through the action execution driving structure, calculates and records data tuples, and stores them in the database;

[0064] Step S700, the reinforcement learning loss function uses the data set to optimize the test strategy network parameters so that the reward of the reinforcement learning loss function is maximized and the loss is minimized, iter cycles are optimized, a new test strategy network is generated, and the new test strategy network is used to drive the software until the reinforcement learning loss function converges;

[0065] The test strategy network is used to identify the correctness of the software. By executing the action sequence generated by the test strategy network, the errors or defects of the software can be quickly exposed, which means that the test strategy has received a positive reward. Conversely, as time goes by, each step will have a negative reward. In other words, the more errors are found, or the greater the error between the actual output value and the expected output value of the software, the faster enough problems are found, the greater the reward. After cyclic learning, the test strategy network can find more and more errors until it reaches the limit value, which means that all errors are found as much as possible, achieving the purpose of testing.

[0066] Step S800, record all errors and mistakes generated in the process and generate a standard test document.

[0067] By constructing a self-learning test strategy network and a reward function that can reward the execution effect of the action sequence, a model-free reinforcement learning loss function can be established. The reward for the execution effect of the action sequence is evaluated through the reward function, and the test strategy network parameters are optimized using a set of data with the maximum reward and the minimum loss through the reinforcement learning loss function, thereby generating a new test strategy network, and then continuing the next round of testing to generate a new data set. The reinforcement learning loss function can optimize the test strategy network again. Through repeated optimization, the reinforcement learning loss function reaches the maximum, and the test strategy network also achieves the best learning effect.

[0068] Preferably, the reinforcement learning training method comprises:

[0069] Step S710, according to the current scenario state, driving the test condition set C through the test strategy network through the action execution drive interface;

[0070] Step S720, obtaining the test action output value of the next scene state of the test condition set C through the observation space O and the test action space A;

[0071] Step S730, determining the deviation between the actual output and the expected output of the software, and obtaining a deviation reward through a reward function; determining whether an error or crash occurs when executing the step, and obtaining a crash reward through a reward function; determining the time consumption of executing the step, and obtaining a time consumption reward;

[0072] Step S740, determining whether the single test is finished, if not finished, continuing to run according to the current test strategy network, if finished, initializing the test strategy network;

[0073] Step S750, obtaining the observation space O data of the current state, the test action output value, the reward value and the completion status of the current step, storing them in a data tuple, and after recording a certain amount of data tuples, storing them in a data set;

[0074] Step S760, learning the data in the collected data set through the reinforcement learning loss function, obtaining the cumulative reward of each action sequence, and optimizing the test strategy network through the obtained maximum reward value;

[0075] Step S770: Use the new test strategy network to drive the test condition set C again.

[0076] By accurately calculating the reward value for each step in the action sequence, the reinforcement learning loss function can accurately obtain a set of data with the minimum loss to optimize the test strategy network.

[0077] Preferably, when learning a new test strategy network, the new test strategy network parameters are first copied to the old test strategy network, and then the new test strategy network is learned, so as to perform fast and stable cyclic optimization on the test strategy network.

[0078] The present invention has the following advantages:

[0079] 1. With this method, there is no need to manually design use cases. The entire process runs automatically and is cognitive and learnable. The test strategy will spontaneously explore the action sequences that are most likely to cause bugs.

[0080] 2. Compared with the existing technology, it avoids the tediousness and incompleteness of manual use case design, and due to its learnability, it can effectively mine occasional failures with high accuracy.

[0081] 3. Short testing time. For the test of a simulation program, using the traditional method, 6 engineers wrote 300 test cases and ran the tests one by one, and found X number of software problems, which took 20 days. Using this method, it only takes X hours.

[0082] As a specific implementation method, a model-free reinforcement learning software testing method is applied to the agile black-box testing of the core engine software of an air combat game to verify the effectiveness of this method in software black-box testing.

[0083] The game core engine software is used to provide a certain air combat game with functions such as a six-degree-of-freedom aircraft dynamics engine, a radar and electronic warfare engine, and a weapon operation engine. By providing real-time control instructions for the aircraft, the aircraft's air combat situation information at the next moment is returned, which is displayed by another air combat visual software. This embodiment only tests the game core engine.

[0084] The steps include:

[0085] A) Test function set construction. The tests of the three core functions of the engine include aircraft dynamics engine test; radar and electronic warfare engine test; firepower weapon operation engine test. Therefore, test condition C is encoded into onhot vectors of 00, 01, and 10. Test condition 00 is for aircraft dynamics engine test, and the output instructions related to radar electronic warfare and weapon operation are set to the default values. Test condition 01 is for radar and electronic warfare engine testing. Except for the output instructions related to fog operation set to the default value, all other output instruction test strategies can be output. Test condition 10 is for firepower weapon operation engine test, and all output instruction test strategies can be output.

[0086] B) Definition of the observation space of the software under test. The complete variable set used to describe the software state has about 250 dimensions, including the machine's kinematic information, weapon and sensor status, target information, tactical implementation distance, etc. All discrete quantities are encoded in onehot form, and upper and lower bounds are set for continuous quantities, constructing an observation space of about 300 dimensions.

[0087] C) Action space definition: In this embodiment, the action space can be directly determined by the output instruction set of the software, including continuous and discrete quantities with more than 50 dimensions.

[0088] D) Reward function definition. Software crash failures include 15 software failure criteria, including software stopping responding for more than a specified time, software continuing to run without stopping normally after the air battle, and computing resource consumption exceeding a specified value. Crash reward factor d fault Set to 1.0, the range of 300 dimensions of the software observation space is set as the expected output, and the actual output is the expected output The deviation of varies according to different test condition coefficients. For example, under test condition 00, the coefficient γ of the observation space dimension that has little to do with aircraft dynamics will be set relatively small, the longest execution step and the negative reward for single-step time consumption are -0.01, and the longest execution step is 10,000 steps.

[0089] E) The programming is developed and compiled in C++. In order to facilitate the call of the TensorFlow deep learning framework, the software is encapsulated in a docker container. The container contains two parts: one is the software under test itself, and the other is a python script, which is responsible for socket communication with the outside of the container, capturing the test instructions given by the external test strategy, and sending the captured test strategy instructions to the software under test and capturing the response results of the software under test.

[0090] F) Select the actor-critic model-free reinforcement learning network and design the test strategy network and valuation network, such as Figure 2 shown.

[0091] G) The reinforcement learning algorithm adopts the proximal policy optimization algorithm PPO. The PPO algorithm is a new type of PolicyGradient algorithm. It proposes a new objective function that can achieve small batch updates in multiple training steps, solving the problem of difficult to determine step size in the Policy Gradient algorithm.

[0092] Usually, the implementation of the PPO algorithm has three networks, a critic network, and two actor networks (old_actor and new_actor). In an episode, the agent first uses the existing strategy to interact with the environment to obtain a batch of data. In this process, the actor and critic networks will not be optimized. After obtaining a complete batch of data, the actor network and the critic network begin to learn this batch of data n times.

[0093] Among them, after collecting this batch of data, the critic network is used to obtain the valuation function. Then, according to the critic network valuation function and the reward reward at each moment stored in the batch data, the value function of each moment in the collected batch data is calculated according to a certain discount rate, as shown in the following formula:

[0094]

[0095] When learning the collected data, the old_actor network is used. First, the parameters of the new_actor network obtained from the data are copied to the old_actor network, and then the learning of the new_actor network and the critic network begins.

[0096] Then the loss function is used for iterative optimization, and the loss function is:

[0097]

[0098] H) Iteration test results are as follows Figure 3 As shown,

[0099] I) Test results,

[0100] Overall accuracy of bug testing: Before using this test software, the accuracy of bug testing was 70%. After using this test software, the accuracy increased to 87%.

[0101] Test time: The test time is four hours after using the software, and it takes twenty days before using the software. The test results are detailed in Table 1.

[0102]

[0103]

[0104] As a specific implementation method, it also includes a model-enhancing-free software testing system, including a test environment establishment module, a reward module, a test strategy module, a loss learning module, and an automated testing module.

[0105] The test environment establishment module is used to form a test simulation environment by constructing a test condition set C, an observation space O, and a test workspace A; the reward module is used to construct a reward function by obtaining software fault judgment criteria and forming a crash reward factor, obtaining the deviation between the actual output and the expected output, obtaining the longest execution step length, and the negative reward for a single-step time consumption; the test strategy module is used to construct a test strategy network; the loss learning module is used to construct a model-free reinforcement learning loss function; the automated testing module is used to obtain the running data of each step and form a data set by inputting the test strategy network into the test simulation environment, and the reinforcement learning loss function uses the data set to optimize the test strategy network parameters to form a new test strategy network for cyclic testing.

[0106] By setting up the reward module and the test strategy module, the software under test can be automatically tested in a cycle, with high test efficiency, short test time and high accuracy.

[0107] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A software testing method based on model-free reinforcement learning, characterized in that: include According to the function set to be tested, construct the test condition set C, construct the observation space O of the tested software, construct the test action space A, and form a test simulation environment; Construct a reward function that rewards the performance of action sequences; Build an action execution driver interface that can act on the software itself; Build a network of test strategies that can learn by themselves; Constructing model-free reinforcement learning loss functions; The drive test strategy network generates software operation actions in real time and acts on the software through the action execution drive interface, calculates and records data tuples, and stores them in the data set; The reinforcement learning loss function uses the data set to optimize the test strategy network parameters so that the reward of the reinforcement learning loss function is maximized and the loss is minimized. It optimizes iter cycles to generate a new test strategy network, and uses the new test strategy network to drive the software until the reinforcement learning loss function converges; Record all errors and mistakes in the process and generate standardized test documents; The test strategy network is, a t =f(O 1:t ,C,a 1:t-1 ) Among them, O 1:t is the historical state of the software under test; C is the embedded test condition; a 1：t-1 is the operation from step 1 to step t-1; a t is the operation action of step t; The reward function is, r t =R(d fault ,d defect ,d time ) Among them, d fault A reward for software errors or even crashes;d defect The reward for the deviation between the actual output and the expected output of the software; d time It is a time consumption reward, which is a constant negative reward; The deviation reward between the actual output of the software and the expected output is After collecting a batch of data, we first calculate the reward value function of each moment collected according to a certain discount rate through the reward at each moment, which is , Then reinforcement learning is performed, and the reinforcement learning loss function is in, Used to depict the differences between the new and old test strategy networks; Aπ old (s,a) is the advantage function of the network based on the old test strategy; ε is used to limit The range of changes; Clip is used to ensure that the difference between the new and old test strategy networks is within the set range. If less than 1-ε, then take 1-ε; if If it is greater than 1+ε, then take 1+ε; if it is between the two, then take The reinforcement learning training method comprises: According to the current scenario status, the test condition set C is driven by the action execution driver interface through the test strategy network; Obtain the test action output value of the next scene state of the test condition set C through the observation space O and the test action space A; Determine the deviation between the actual output and the expected output of the software, and obtain the deviation reward through the reward function; determine whether an error or crash occurs when executing the step, and obtain the crash reward through the reward function; determine the time consumption of executing the step, and obtain the time consumption reward; Determine whether a single test is finished. If not, continue to run according to the current test strategy network. If finished, initialize the test strategy network. Obtain the current state of the observation space O data, test action output value, reward value and the completion status of the current step, store them in data tuples, and after recording a certain amount of data tuples, store them in the data set; The data in the collected data set is learned through the reinforcement learning loss function to obtain the cumulative reward for each action sequence, and the test strategy network is optimized through the maximum reward value obtained; The test condition set C is driven again using the new test strategy network.

2. The software testing method for model-free reinforcement learning according to claim 1, characterized in that: The reward function is constructed by obtaining software fault criteria and forming a crash reward factor, obtaining a deviation between an actual output and an expected output, obtaining a maximum execution step length, and a negative reward for a single-step time consumption.

3. The software testing method for model-free reinforcement learning according to claim 1, characterized in that: When the software runs normally, the crash reward gets a negative reward; when the software has an error or even crashes, the crash reward gets a positive reward; when there is no deviation, the deviation reward gets a negative reward; when there is a deviation between the actual output of the software and the expected output, the deviation reward gets a positive reward; when there is no problem with the current action sequence, the time reward value is small; the longer the time consumed for each execution step of the action sequence, the greater the time reward value; the shorter the time consumed for each execution step of the action sequence, the smaller the time reward value.

4. The software testing method for model-free reinforcement learning according to claim 1, characterized in that: When learning a new test strategy network, first copy the new test strategy network parameters to the old test strategy network, and then learn the new test strategy network.

5. A software testing system for model-free reinforcement learning, using the software testing method according to any one of claims 1 to 4, characterized in that: include, The test environment establishment module is used to form a test simulation environment by constructing a test condition set C, an observation space O, and a test workspace A; A reward module is used to construct a reward function by obtaining software fault criteria and forming a crash reward factor, obtaining the deviation between the actual output and the expected output, obtaining the longest execution step length, and the negative reward of the single-step time consumption; Test strategy module, used to build a test strategy network capable of self-learning; Loss learning module, used to construct model-free reinforcement learning loss functions; The automated testing module is used to obtain the running data of each step and form a data set by inputting the test strategy network into the test simulation environment. The reinforcement learning loss function uses the data set to optimize the test strategy network parameters, form a new test strategy network, and perform cyclic testing.

Images