A reinforcement learning framework security detection method based on multi-level operator mutation

Abstract

A security detection method for reinforcement learning frameworks based on multi-level operator mutation includes: 1) Reinforcement learning source model training: using the interaction between the reinforcement learning agent and the environment to perform goal-oriented learning, obtaining an initial reinforcement learning agent model M. o ;2) For the original reinforcement learning agent model M o Targeting potential defects; designing mutation operators, and inserting mutation operators of different levels into the original reinforcement learning agent model M. o S3) Modify the test environment design in the initial reinforcement learning environment, and design a mutation score for the reinforcement learning system. The mutation score is the ratio of the number of eliminated mutants to the total number of mutants; S4) Modify the original reinforcement learning agent model M o And the mutation reinforcement learning agent model M M Game tests are conducted in different testing environments, and the generated mutation test scores are compared to determine whether the framework is a vulnerability of this type.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, more particularly, to a reinforcement learning framework security detection method for an automatic driving system. BACKGROUND

[0002] Deep reinforcement learning is one of the directions of artificial intelligence that has attracted much attention in recent years. With the rapid development and application of reinforcement learning, reinforcement learning has been widely used in robot control, game playing, computer vision, unmanned driving and other fields. In order to ensure the safe application of deep reinforcement learning in safety-critical fields, the key lies in analyzing and discovering vulnerabilities in deep reinforcement learning algorithms and models to prevent some people from taking advantage of these vulnerabilities for illegal profiteering. Unlike the single-step prediction task of traditional machine learning, deep reinforcement learning system needs to make multi-step decisions to complete a task, and there is a high degree of correlation between consecutive decisions.

[0003] Reinforcement learning is usually a continuous decision-making process, and its basic idea is to maximize the cumulative reward obtained by the agent from the environment, so as to learn the optimal strategy to achieve the learning purpose. Deep reinforcement learning makes full use of neural networks as parameter structures, combining the perception ability of deep learning and the decision-making ability of reinforcement learning to optimize the deep reinforcement learning strategy, so that the agent can continuously learn from its environment over time. However, the strategy trained by deep reinforcement learning can also be stolen through imitation learning or behavior cloning, and in addition, the thief can also attack the deep reinforcement learning model according to the stolen strategy, which in turn disturbs the strategy trained by reinforcement learning itself, so that the model strategy has security vulnerabilities and cannot respond well to the current environment. This not only leaks the model's own strategy data, but also brings great security risks to the model, especially in privacy-protected safety-critical fields, which may cause great losses.

[0004] Under the great success of traditional software system testing techniques, people have tried to test supervised learning systems many times, especially for classification problems. However, current artificial intelligence framework testing methods rarely consider reinforcement learning systems. As one of the most important testing techniques for traditional software, mutation testing can be applied at different levels. Mutation operators based on simulating potential errors of software systems can detect potential errors of original software systems by detecting the ability of behavior differences between mutants and corresponding original systems.

[0005] Reinforcement learning systems are different from supervised learning systems. Supervised learning learns from labeled training data to obtain a function that maps input data to corresponding output labels. Therefore, the existing deep learning system mutation technology cannot be naturally applied to reinforcement learning systems. SUMMARY

[0006] The present application overcomes the above-mentioned shortcomings of the prior art and provides a multi-level operator mutation-based reinforcement learning framework security detection method.

[0007] The present application designs element-level and agent-level mutation operators for a reinforcement learning system to introduce various potential faults as comprehensively as possible. To further cover problems that the reinforcement learning system may encounter, more comprehensive mutation operators are also proposed.

[0008] The present application is directed to an offline-trained reinforcement learning system and provides a new idea for reinforcement learning framework security testing. Instead of being limited to traditional differential testing algorithms, a multi-level operator mutation-based reinforcement learning framework security detection method is proposed, which uses different levels of mutation operators to introduce various potential faults as comprehensively as possible. A mutation score for the reinforcement learning system is designed for the Markov process specific to reinforcement learning, and the mutation score is used to discover potential vulnerabilities in the system.

[0009] The multi-level operator mutation-based reinforcement learning framework security detection method of the present application includes the following steps:

[0010] S1: Reinforcement learning source model training: target-oriented learning is performed using the interaction between a reinforcement learning agent and an environment, which can be expressed as (S, A, P, r, γ), where S is a finite state space, A is a finite action space, P is the transition probability of the environment, r is the action reward value, and γ is the discount rate parameter. An initial reinforcement learning agent model M is obtained. o ;

[0011] S2: Potential defects are designed for the original reinforcement learning agent model M o . Mutation operators are designed, and different levels of mutation operators are inserted into the original reinforcement learning agent model M o .

[0012] S3: The initial reinforcement learning environment is modified to design a test environment, and a mutation score specific to the reinforcement learning system is designed. The mutation score is the ratio of the number of eliminated mutants to the total number of mutants.

[0013] S4: The original reinforcement learning agent model M o and the mutated reinforcement learning agent model M M are tested according to different test environments, and the generated mutation test scores are compared to determine whether the framework has this type of vulnerability.

[0014] Preferably, the step S1 specifically includes:

[0015] S1.1 Build a deep reinforcement learning car automatic driving simulation environment; use the deep Q network algorithm (DQN) based on reinforcement learning to train the car Car, the goal of Car is to safely reach the destination as quickly as possible, DQN combines Q learning with convolutional neural network to build a reinforcement learning training model M o .

[0016] S1.2 Based on the deep Q network algorithm (DQN) in reinforcement learning to train the unmanned car agent M o , the goal of M o is to safely reach the destination in a short time. DQN combines deep neural network with Q learning algorithm of reinforcement learning, not only solves the problem of large state space difficult to maintain, but also has great potential due to the powerful feature extraction ability of neural network. Q learning in reinforcement learning uses the way of time difference to update state-action value function Q through Bellman equation:

[0017] Q i+1 (s t ,a t )=Q i (s t ,a t )+α(y i -Q i (s t ,a t )) (1)

[0018] According to the theory of Bellman optimal equation, as long as the above formula is updated by iteration, the Q function can be approximated to the true value Q^*, so that the optimal strategy is finally obtained:

[0019]

[0020] DQN and target network mechanism, that is, on the basis of the current Q θ network structure, a target network with the same structure is built to form the overall model framework of DQN. In the training process, the predicted Q value output by the current Q θ network is used to select action a, and the other target network is used to calculate the target Q value. The loss function is defined by calculating the mean square error of the predicted Q value and the target Q value:

[0021]

[0022] S1.3 In the training process, the state conversion process (state s t , action A, reward r, next state s t+1) stored in the experience replay buffer buff as a training dataset of the network model, and batch learning in the form of random sampling;

[0023] S1.4 Sample N training datasets from Buff, update the current Q θ network parameters of the network, for the target network, whose network parameters do not need to be updated iteratively, but copied from the current Q θ network parameters in the network, and then proceed to the next round of learning.

[0024] Preferably, the step S2 specifically comprises:

[0025] S2.1: Element-level mutation operator setting: This scheme designs two types of element-level mutation operators, namely state mutation operator and reward mutation operator. At time step t, the learning agent observes the environment state s t , and selects action a t on this basis as its action a t . As a result, the agent receives the corresponding reward r t , and updates the new state s t+1 of the next time step. The proposed element-level mutation operator is designed based on the potential risks that the agent may suffer between the observed state and the obtained reward value. Among them, the reward mutation operator includes reward reduction operator, reward increase operator and reward instability operator; the state mutation operator includes state record crash, state delay and state error.

[0026] S2.2 Agent-level mutation operator setting: In order to cover more possible failures, the goal of the reinforcement learning system is to train the agent to complete the task in an uncertain environment. The trained agent contains a policy to guide the action of the agent. Therefore, this scheme directly mutates the trained agent to cover more potential risks. The agent-level mutation operator includes input layer neuron removal, output layer neuron removal and output layer neuron addition.

[0027] Preferably, the step S3 specifically comprises:

[0028] S3.1 At each time step, the learning agent will observe the state in the environment and select a corresponding action. And the action will be applied to the environment in time to update the action of the agent and obtain the corresponding reward value.

[0029] S3.2 Apply the mutation operator in S2 to the reinforcement learning system, the interaction record will be mutated, that is, some agent behaviors will be irrelevant to the state, for example, some state-action pairs can be discarded, which corresponds to the state loss mutation operator. Then consider the interaction of the mutation operator in the training process to produce the mutated reinforcement learning agent M M .

[0030] S3.3 Design a mutation score specific to the reinforcement learning system to quantitatively evaluate the quality of the test environment. The mutation score is the ratio of the number of mutants killed to the total number of mutants produced. Due to the randomness in the reinforcement learning environment, the results may be different each time the agent runs in the same environment.

[0031] The present application proposes a reinforcement learning framework test technology based on multi-level operator mutation, including a series of mutation operators, mutation scores and test environments. The reinforcement learning system contains an agent seeking to achieve a goal and its environment. The agent interacts with its environment, and the agent's operation may affect the future state of the environment, thereby affecting the options available to the agent later. At the same time, in the long run, the agent always aims to maximize the reward value sent by the environment after the corresponding action affects the environment.

[0032] According to the characteristics of the reinforcement learning system, the mutation operator is designed by injecting potential faults at the state level and the reward value. For the original agent trained by the original reinforcement learning system, the corresponding mutant agent is obtained by constructing a mutant reinforcement learning system based on the mutation operator. By comparing the mutation scores of the two agents in the same environment, the security of the reinforcement learning system framework can be evaluated.

[0033] The advantages of the present application are that, compared with the prior art, the beneficial effects of the technical solution of the present application are: design element-level and agent-level mutation operators for reinforcement learning systems to introduce various potential faults as comprehensively as possible. In order to further cover the problems that the reinforcement learning system may encounter, a set of test environments are provided, which can discover possible faults in the reinforcement learning system according to the changes in the mutation score. It can better complete tasks such as autonomous driving. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0035] Figure 1 The framework diagram of the present application;

[0036] Figure 2A framework diagram of element-level framework variation test of the present application;

[0037] Figure 3 A framework diagram of agent-level framework variation test of the present application. DETAILED DESCRIPTION

[0038] Various exemplary embodiments of the present application will now be described in detail, which should be considered to be illustrative of certain aspects, features and embodiments of the present application, but not a limitation of the present application.

[0039] It should be understood that the terms used in the present application merely describe particular embodiments, and are not intended to limit the present application. In addition, for numerical ranges in the present application, it should be understood that each intermediate value between the upper limit and the lower limit of the range is also specifically disclosed. Each smaller range within any stated value or stated range, as well as within any other stated value or stated range, is also included in the present application. The upper and lower limits of these smaller ranges can be independently included or excluded from the range.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present application, preferred methods and materials are described. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods and / or materials in connection with which the documents are cited. In case of conflict between the content of the specification and that of any document incorporated by reference, the content of the specification prevails.

[0041] Various modifications and changes can be made to the specific embodiments of the present application described herein without departing from the scope or spirit of the application. Other embodiments of the application will be apparent to those of ordinary skill in the art from the description and examples presented herein. The specification and examples given should be considered exemplary only.

[0042] As used herein, the terms "comprises", "comprising", "includes", "including", "has", "having", and the like are open-ended terms that are intended to denote the inclusion of elements or steps, but not to exclude the presence of other elements or steps.

[0043] The "parts" described in the present application are measured by mass unless otherwise specified.

[0044] The present application provides a reinforcement learning framework security detection method based on multi-level operator variation, as shown in Figure 1 The steps include:

[0045] Step S1: The target agent is pre-trained to obtain a target policy π t: Based on deep Q network algorithm (DQN) in reinforcement learning training car, the goal of car is to reach the destination as fast as possible, DQN combines Q learning with convolutional neural network, and constructs a reinforcement learning training model, the algorithm steps are as follows:

[0046] S1.1 DQN combines deep neural network with Q learning algorithm of reinforcement learning, not only solves the problem of too large state space difficult to maintain, but also has great potential than artificial feature representation due to the powerful feature extraction ability of neural network. Q learning in reinforcement learning updates state-action value function Q through Bellman equation in a way of time difference iteration:

[0047] Q i+1 (s t ,a t )=Q i (s t ,a t )+α(y i -Q i (s t ,a t )) (1)

[0048] Wherein, is the target Q value, s t+1 is the next state as action a t occurs, a t+1 is the possible action under s t+1 state. Alpha is the learning rate, gamma is the discount factor. According to Bellman optimal equation theory, as long as the above formula is updated by iteration, Q function can be approximated to the real value Q * , so as to finally obtain the optimal strategy:

[0049]

[0050] S1.2 DQN also uses the target network mechanism, that is, on the basis of current Q θ network structure, a target network with the same structure is built. The overall model framework of DQN is composed of DQN, in the training process, the predicted Q value output by the current Q θ network is used to select action a, and the other target network is used to calculate the target Q value. The loss function is defined by calculating the mean square error of the predicted Q value and the target Q value:

[0051]

[0052] Wherein, is the target Q value, and the parameters θ of the current Q θ network are updated by the backward gradient propagation of neural network.

[0053] S1.3 During the training process, the DQN adopts an experience replay mechanism, and the state conversion process (state s i , action a i , reward r i , next state s i ') is stored in the experience replay buffer Buff as a training data set for the network model, and is learned in a batch form with random sampling.

[0054] S1.4 Sample N training data sets from Buff, update the network parameters of the current Q θ network by minimizing the loss function. For the target network, its network parameters do not need to be iteratively updated, but are copied from the current Q θ network at regular intervals for the next round of learning.

[0055] Step S2: Design potential defects for the original reinforcement learning agent model M o . Design a mutation operator and insert different levels of mutation operators in the original reinforcement learning agent model M o for retraining to obtain a mutated test model M M .

[0056] S2.1 Element-level mutation operator construction

[0057] First, the reward-level mutation operator is constructed. The reward-level operator mutation should deal with the potential risks that attacks or source code errors may bring to the reinforcement learning system. Specifically, attacks and source code errors may bring abnormal rewards. Considering the characteristics of the reinforcement learning training process, abnormal rewards will lead the learning agent to the wrong direction. Based on this implementation, the following reward mutation operators are designed:

[0058] 1) Reward reduction operator: Reward is the main basis for changing the agent's policy, which defines the benefits or disadvantages to the agent. The reward reduction operator modifies part of the reward to reduce the original higher reward. Intuitively, reward reduction will make the agent run in the opposite direction.

[0059] 2) Reward increase: In contrast to the reward reduction operator, the reward increase operator modifies part of the reward to increase the original smaller reward.

[0060] 3) Reward instability: The reward instability operator mixes the reward reduction operator and the reward increase operator to simulate the case of reward instability.

[0061] Secondly, a state mutation operator is constructed, which simulates real-world potential risks. Reinforcement learning systems are widely deployed in fields such as autonomous driving. In autonomous driving scenarios, the learning agent uses sensors to observe the environment. However, sensors may experience mechanical failures during this observation process. The state mutation operator is designed based on the potential existence of mechanical failures. Specifically, it is shown below:

[0062] 1) State recording crash: The learning agent should observe the state s of the environment. t0 And select operation a at time step t0. t0 Because sensors may have robustness issues, some state-action pairs are not uncommon (s t ,a t Records can be lost or duplicated. State record crash operators hide or duplicate some state-action pairs to simulate such failures.

[0063] 2) State delay: At each time step t, the state-action pair (s) t ,a t The state-action pairs and their corresponding rewards (rt) should be recorded promptly. If these are recorded correctly and in a timely manner, the reinforcement learning system can acquire the correct agent as expected. If the transmission of sensor observations is delayed, the state-action pairs and their corresponding rewards will be recorded inaccurately. Therefore, the state delay mutation operator simulates this type of failure.

[0064] 3) State Errors: State error mutation operators construct erroneous connections between states and operations. While it may be possible to associate every state with erroneous behavior, constructing just a few erroneous associations is sufficient to inject this potential risk and cause significant damage.

[0065] S2.2 Intelligent Agent-Level Mutation Operator Settings

[0066] To cover more potential failures, this scheme proposes and implements a set of agent-level mutation operators. Training agents to complete tasks in uncertain environments is the goal of reinforcement learning systems. A trained agent contains a policy that guides its action selection. Typically, the policy can be a neural network. Therefore, this scheme directly mutates the trained agent to cover more potential risks.

[0067] 1) Input-Layer Neuron Removal (INR): In reinforcement learning, neural networks take the environmental state as input and output a vector containing the Q-values ​​of each possible action. The INR operator removes some neurons from the input layer, causing the trained agent to ignore some environmental information.

[0068] 2) Output-Layer Neuron Disappearance (OND): The OND operator tries to simulate the situation where the agent is unable to perform certain actions. This is achieved by removing selected neurons in the output layer.

[0069] 3) Output-Layer Neuron Addition (ONA): Unlike the OND operator, the ONA operator simulates the opposite situation, where the agent will do some unexpected behavior. The method to achieve this goal is to add selected neurons in the output layer.

[0070] Step S3: Modify the initial reinforcement learning environment to design the test environment, and design the mutation score for the reinforcement learning system, which is the ratio of the number of mutants killed and the total number of mutants in the population.

[0071] S3.1 At each time step, the learning agent observes the state s∈S, selects an action a∈A and applies it to the environment in time, modifies its state and obtains the corresponding reward r. Based on the correct results of these interactions, the reinforcement learning agent M o .

[0072] S3.2 Start the element-level mutation test process according to the proposed framework. First, apply the mutation operator to the reinforcement learning system, and the learned policy will be mutated, which means that some behaviors will be irrelevant to the state. For example, some state-action pairs can be discarded, which corresponds to the state loss mutation operator. Then, the mutated interactions will be considered in the training process to produce the mutated reinforcement learning agent M M .

[0073] S3.3 Design a mutation score specific to the reinforcement learning system to quantitatively evaluate the quality of the test environment. The mutation score is the ratio of the number of mutants killed to the total number of mutants produced. Due to the randomness in the reinforcement learning environment, the results may be different each time the agent runs in the same environment.

[0074] S4: Play the original reinforcement learning agent model M o and the mutated reinforcement learning agent model M M according to different test environments, compare the generated mutation test scores, and determine whether the framework has this type of vulnerability.

[0075] Case analysis

[0076] With the progress of science and technology, especially the rapid development of Internet technology, communication technology and artificial intelligence technology in recent years, the era of intelligent vehicle technology has come. However, the actual use of intelligent vehicles needs a long transition period. Although intelligent vehicle technology has made great progress, there are still some problems to be solved. The driving environment has complexity and variability, which brings great challenges to driving decision-making. This seriously hinders the development and practicality of intelligent vehicle driving technology, therefore, carrying out human driving research can effectively solve the problem of intelligent vehicle driving decision-making.

[0077] The automatic driving system includes three steps: perception, decision and control. The system architecture is set up in line with human driving behavior. The intelligent vehicle first obtains sensor information through various sensors, fuses different sensor information, forms multi-environment information, and transmits it to the planning and decision-making layer. Then, according to the constraints and rules set by the system, a decision is made to generate corresponding driving operation information. Finally, the control module executes the command to realize automatic driving.

[0078] The present scheme detects the possible vulnerabilities in the environment perception and control decision-making links of the automatic driving system. First, in the environment perception link, due to possible sensor faults, multi-environment information fusion errors and other faults of the automatic driving vehicle, the present scheme uses the generated original agent model and the mutated agent model to perform path planning and action determination in the collected environment. If the two agent models are in the same environment, the mutation score of the two is used to detect the collected environment information, thereby detecting the sensors and information fusion strategy in the automatic driving system.

[0079] Secondly, due to the development of Internet of Things system, many existing automatic driving systems are connected with Internet of Things system, so they may face problems such as hacker attacks and modification of decision-making strategy. Therefore, the present scheme realizes the change of agent strategy through the mutated agent model. The mutation score of the mutated agent and the original agent is used to detect whether the decision-making model has vulnerabilities, thereby detecting the decision-making strategy of the agent decision-making model.

[0080] The above-described embodiments are only preferred modes of the present application and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A method for security detection of reinforcement learning frameworks based on multi-level operator mutation, comprising the following steps: S1: Reinforcement learning source model training: the interaction between the reinforcement learning agent and the environment is used for goal-oriented learning, and the process is expressed as , wherein is a finite state space, is a finite action space, is the transition probability of the environment, is the action reward value, is a discount rate parameter; a deep reinforcement learning car automatic driving simulation environment is built; a car is trained by using a deep Q network algorithm DQN based on reinforcement learning, the goal of the car is to safely and quickly reach the destination, the DQN combines Q learning and a convolutional neural network to construct an original reinforcement learning agent model ; S2: For the original reinforcement learning agent model , the potential defects are designed specifically; the mutation operator is designed, and different levels of mutation operators are inserted into the original reinforcement learning agent model ; specifically including: S2.1: Element-level mutation operator setting: This scheme designs two types of element-level mutation operators, namely state mutation operators and reward mutation operators; at time step , the learning agent observes the environment state , and selects an action on this basis as the result of its action , the agent receives the corresponding reward , and updates the new state of the next time step ; the proposed element-level mutation operator is designed based on the potential risk of the agent between the observed state and the obtained reward value; among them, the reward mutation operator includes reward reduction operator, reward increase operator and reward instability operator; the state mutation operator includes state record crash, state delay and state error; S2.2 Agent-level mutation operator settings: To cover faults, training agents to complete tasks in uncertain environments is the goal of reinforcement learning systems; the trained agent contains a policy that guides the agent's actions; therefore, this scheme directly mutates the trained agent to cover more potential risks; agent-level mutation operators include input layer neuron removal, output layer neuron removal, and output layer neuron attachment. S3: Modify the design of the test environment in the initial reinforcement learning environment, and design a mutation score for the reinforcement learning system. The mutation score is the ratio of the number of mutants eliminated to the total number of mutants. S4: the original reinforcement learning agent model and the mutated reinforcement learning agent model According to different test environments, game testing is performed, the generated mutation test scores are compared, and it is determined whether the framework has this type of vulnerability.

2. The method for security detection of reinforcement learning frameworks based on multi-level operator mutation as described in claim 1, characterized in that: Step S1 also includes: S1.1 Training the autonomous vehicle agent based on the Deep Q-Network algorithm (DQN) in reinforcement learning. , The goal is to safely reach the destination in a short time; DQN combines deep neural networks with reinforcement learning. Learning algorithms not only solve the problem of excessively large state spaces that are difficult to maintain, but also, due to the powerful feature extraction capabilities of neural networks, their potential far exceeds that of manual feature representations; reinforcement learning... Learn how to iteratively update the state-action value function using the Bellman equation and a time-difference approach. : （1） According to Bellman's optimal equation theory, by iteratively updating the above equation, the Q function can be made to approximate the true value Q^. Thus, the optimal strategy is ultimately obtained: （2） DQN and the target network mechanism, that is, in the current Based on the network structure, a target with an identical structure was built. The network forms the overall model framework of DQN. During training, the current... Network output prediction The value is used to select the action. Another goal Networks are used to compute targets Value; predicted through calculation Values ​​and Targets The loss function is defined using the mean squared error: （3） During the S1.2 training process, the state transition process (state) ,action ,award Next state Stored in the experience playback buffer In this context, the dataset serves as the training dataset for the network model and is used for batch learning in the form of random sampling. S1.3 from Sampling N training datasets, updating the current dataset by minimizing the loss function. The network parameters of the network, for the target The network parameters do not need to be updated iteratively, but are changed periodically from the current value. The network parameters are copied from the network and then used for the next round of learning.

3. The method for security detection of reinforcement learning frameworks based on multi-level operator mutation as described in claim 1, characterized in that: Step S3 specifically includes: S3.1 At each time step, the learned agent observes the state of the environment and selects a corresponding action; this action is then applied to the environment in a timely manner, updating the agent's actions and obtaining the corresponding reward value. S3.2 applies the mutation operator from S2 to the reinforcement learning system, causing the interaction records to mutate, meaning that some agent behaviors will become unrelated to their states; then, during the training process, the interaction of the mutation operator is considered to generate mutated reinforcement learning agents. ; S3.3 Design a mutation score for the reinforcement learning system to quantitatively evaluate the quality of the test environment; the mutation score is the ratio of the number of mutants killed to the total number of mutants generated; due to the randomness of the reinforcement learning environment, the results will be different each time the agent runs in the same environment.

Images