Deep reinforcement learning model training method based on formal verification and explanation

Abstract

The invention discloses a deep reinforcement learning model training method based on formalized verification and explanation, and the method comprises the steps: judging whether a current deep reinforcement learning model meets an extra training termination condition or not through a single verification method in a formalized verification process, and terminating the training if the current deep reinforcement learning model meets the extra training termination condition; otherwise, measuring a state gap between the current deep reinforcement learning model and the extra training termination condition through a gap measurement method; in the explanation process, explaining and measuring the density of the state characteristics in the constraint space defined in the extra training termination condition through a density measurement method based on a sensitivity analysis method; in the training process, the current deep reinforcement learning model interacts with the environment to generate a track, and the track is stored in a buffer area; and updating the current reward value in the trajectory through a reward remodeling method of the state feature density in the state gap and the constraint space, and updating the current deep reinforcement learning model according to the updated trajectory. According to the method, the safety and activity of the deep reinforcement learning model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep reinforcement learning, and in particular to a deep reinforcement learning model training method based on formal verification and explanation. Background Art

[0002] In recent years, deep reinforcement learning has played an outstanding role in the development of intelligent systems, covering multiple fields such as natural language processing, computer networks, and autonomous driving. However, the inherent opacity of the deep neural network decision-making process that deep reinforcement learning relies on makes it crucial to ensure verifiable and explainable safety and liveness before actual model deployment.

[0003] In the related art, formal verification is combined with model training, but the verification results are only used as adversarial counterexamples or only indicate whether the training process should be extended, resulting in too many training steps and long training time, and the prior art methods are poor in terms of property satisfaction and performance of the final model. In addition, interpretability, a widely concerned factor in the development of trustworthy deep reinforcement learning systems, is not considered, and existing verifiers and interpreters lack reusability and are not easy to use in the design of deep reinforcement learning systems, and lack comprehensiveness in terms of property types and interpretability issues. Summary of the invention

[0004] In order to overcome the problems existing in the prior art, the present invention provides a deep reinforcement learning model training method based on formal verification and explanation. By training, formally verifying and explaining the deep reinforcement learning model, the reward value is reshaped, thereby improving the security and activity of the deep reinforcement learning model.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution: a deep reinforcement learning model training method based on formal verification and explanation, comprising the following steps:

[0006] Step S1: Given an initial deep reinforcement learning model, and setting additional training termination conditions for the deep reinforcement learning model;

[0007] Step S2: During the formal verification process, a single verification method is used to determine whether the current deep reinforcement learning model meets the additional training termination condition. If so, the training is terminated; otherwise, a gap measurement method is used to measure the state gap between the current deep reinforcement learning model and the additional training termination condition.

[0008] Step S3, in the explanation process, explaining and measuring the density of state features in the constraint space defined in the additional training termination condition by a density measurement method based on a sensitivity analysis method;

[0009] Step S4: During the training process, the current deep reinforcement learning model interacts with the environment to generate trajectories, which are stored in a buffer;

[0010] Step S5: Update the current reward value in the trajectory through the reward reshaping method of the state gap and the state feature density in the constraint space, and update the current deep reinforcement learning model according to the updated trajectory, and repeat steps S2-S5.

[0011] Furthermore, the specific process of determining whether the current deep reinforcement learning model meets the additional training termination condition by the single verification method in step S2 is:

[0012] Step S201: Convert the constraint script of the additional training termination condition and the current deep reinforcement learning model into a form recognizable by the deep neural network verifier through a property translation method and a model adaptation method;

[0013] Step S202: connect the single verification method with the deep neural network verifier through the general call method as an intermediary to obtain the binary deep reinforcement learning verification result;

[0014] Step S203: If the deep reinforcement learning verification result shows that the property is established, it means that the additional training termination condition is met; if the deep reinforcement learning verification result shows that the property is not established, it means that the additional training termination condition is not met.

[0015] Furthermore, the constraint script includes: input and output structures matching the deep reinforcement learning model, constraint conditions consisting of comparison expressions, the relationship between any two constraint conditions, and loop conditions for the interaction between the deep reinforcement learning model and the environment.

[0016] Furthermore, the measurement process of the state gap g(φ,N) between the current deep reinforcement learning model and the additional training termination condition in step S2 is specifically as follows:

[0017]

[0018] Where N represents the current deep reinforcement learning model, bps represents the set of mutation point properties within the range of the additional training termination condition for identifying relaxation, and φ[z] represents the specific value of the parameter z in the additional training termination condition of the deep reinforcement learning model. Represents the relaxed value of the parameter z in the additional training termination condition of the deep reinforcement learning model.

[0019] Furthermore, step S3 includes the following sub-steps:

[0020] Step S301, converting the density measurement problem into a sensitivity analysis problem on different state characteristics in the constraint space by using a density measurement method;

[0021] Step S302: Apply a sensitivity analysis method on each state feature, and verify the possible outputs obtained by using the single verification method multiple times on the state feature boundary of the additional training termination condition of the current deep reinforcement learning model input being disturbed;

[0022] Step S303: The deep reinforcement learning verification results obtained through multiple verifications may change. The possible output corresponding to the change and the output obtained by inputting the state feature boundary in the additional training termination condition into the current deep reinforcement learning model are used as the density of the corresponding state feature.

[0023] Furthermore, the density of the state feature in step S303 is divided into the lower limit feature density of the state feature and the upper bound feature density

[0024]

[0025]

[0026] Where φ represents the additional training termination condition, s i represents the state of the i-th time slice defined in the additional training termination condition, Indicates i The j-th state feature in n represents the number of state features, φ [s i ] represents the constraint space defined in the additional training termination condition for s i The lower bound of the constraint is represents the constraint space defined in the additional training termination condition for s i The upper bound of the constraint, represents the disturbance s of the small disturbance value ε i , N represents the model of deep reinforcement learning, N(s i ) represents the input state s of the deep reinforcement learning model i Output when .

[0027] Furthermore, step S4 includes the following sub-steps:

[0028] Step S401, using a constraint translation method to generate a real-time detection program code from the constraint script of the additional training termination condition;

[0029] Step S402: Input the collected data set into the real-time detection program code, and the deep reinforcement learning model interacts with the environment to generate a trajectory consisting of the current state, current action, and current reward value, which is stored in a buffer;

[0030] Step S403: Detect whether the trajectory violates the additional training termination condition by a real-time detection method, and calculate the difference between the current state and the current action and the boundary of the additional training termination condition.

[0031] Furthermore, the specific process of calculating the difference Diff(φ,s,a) between the current state and the current action and the training termination condition boundary in step S403 is:

[0032] Diff(φ,s,a)=±dist(φ[s],s)×dist(φ[a],a)

[0033] Where φ represents the training termination condition boundary, s represents the current state, a represents the current action, dist(φ)[s], s) represents the distance from the current state s to the state-related training termination condition boundary, dist(φ[a], a) represents the distance from the current action to the action-related training termination condition boundary, + represents that the training termination condition is met, and - represents that the training termination condition is violated.

[0034] Furthermore, step 5 includes the following sub-steps:

[0035] Step S501: Determine the state feature density switching point according to the density of the state feature in the constraint space defined in the training termination condition. Get the state features of each time point in the constraint space in the additional training termination condition Value and The corresponding observation value in the trajectory experience Distance

[0036] Step S502: Calculate the difference diff(φ,s,a) from the observed values ​​of all state features in the constraint space to the boundary of the additional training termination condition;

[0037] Step S503: Adjust the learning rate of the deep reinforcement learning model based on the state gap g(φ,N) between the current deep reinforcement learning model and the additional training termination condition, and calculate the difference between all state features in the constraint space and the boundary of the additional training termination condition after adjusting the learning rate. Where Lr() represents the learning rate scheduling function;

[0038] Step S504: Adjust by scaling factor β As an intermediate reward, and trace the intermediate reward through the final track in the buffer area Among them, w represents the weight density, λ represents the discount factor;

[0039] Step S505: trace the intermediate rewards With the current reward value r tReshape the current reward value together

[0040] Furthermore, in step S501 The acquisition process is:

[0041]

[0042] Among them, p1 represents the penalty intensity coefficient, express The lower bound value of express The upper bound value of .

[0043] Compared with the prior art, the present invention has the following beneficial effects: the deep reinforcement learning model training method based on formal verification and explanation of the present invention considers the contribution of different output state characteristics of the deep reinforcement learning model to the model decision through a feature density measurement method based on model interpretation, making the reward reshaping method more accurate and explainable, and has a more accurate grasp of the property learning status of the current deep reinforcement learning model through the state gap, and can dynamically adjust the amplitude of reward reshaping, balance the learning of the deepening strength learning model performance and the additional training termination condition, and in addition, through the verification method, it can formally ensure that the deepening strength learning model ultimately meets the predetermined additional training termination condition, and ultimately the deepening strength learning model still maintains excellent performance while improving the security and activity of the deepening strength learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the deep reinforcement learning model training method based on formal verification and explanation of the present invention;

[0045] Figure 2 is a flow chart of the formal verification method in the present invention;

[0046] Figure 3 is a flow chart of an explanation method in the present invention;

[0047] Figure 4 It is a block diagram of a deep reinforcement learning model training device based on formal verification and explanation of the present invention;

[0048] Figure 5 The present invention provides a block diagram of an electronic device. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described implementation methods should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the present invention are only for the purpose of describing the present invention and are not intended to limit the present invention.

[0051] The deep reinforcement learning model training method based on formal verification and explanation of the present invention is executed by an electronic device, which is a terminal or a server. Optionally, the terminal is a laptop computer, a desktop computer, etc. The terminal can also be called a user device, a portable terminal, a laptop terminal, a desktop terminal, etc. Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0052] In order to specifically illustrate the details of the present invention, the classic control task Mountain Car task will be used as a scenario example. This example is only for the purpose of specifically describing the details of the present invention and is not intended to limit the present invention. The task consists of a car randomly placed at the bottom of a valley. The goal is to accelerate the car to the top of the right mountain. The deep reinforcement learning model can take one of three discrete actions: accelerate to the left, accelerate to the right, or not accelerate; the environment state consists of two eigenvalues: the position of the car in the horizontal direction and the speed of the car.

[0053] like Figure 1 The present invention is a deep reinforcement learning model training method based on formal verification and explanation. The execution subject of the present invention is an electronic device. The training method includes the following steps:

[0054] Step S1: Given an initial deep reinforcement learning model, and setting additional training termination conditions for the deep reinforcement learning model.

[0055] Step S2: During the formal verification process, a single verification method is used to determine whether the current deep reinforcement learning model meets the additional training termination condition. If so, the training is terminated; otherwise, a gap measurement method is used to measure the state gap between the current deep reinforcement learning model and the additional training termination condition.

[0056] like Figure 2 The specific process of judging whether the current deep reinforcement learning model meets the additional training termination condition by the single verification method in the present invention is as follows:

[0057] Step S201, converting the constraint script of the additional training termination condition and the current deep reinforcement learning model into a form recognizable by the deep neural network verifier through a property translation method and a model adaptation method; the constraint script in the present invention includes: input and output structures matching the deep reinforcement learning model, constraint conditions composed of comparison expressions, the relationship between any two constraint conditions, and loop conditions for the interaction between the deep reinforcement learning model and the environment. The constraint script can write the constraint content in a non-coupled manner with simple and intuitive syntax, thereby avoiding the problems of non-reusability and non-intuitiveness when using real-time monitoring program codes and single verification codes with hard-coded constraints.

[0058] The model adaptation method of the present invention takes a neural network model and an expansion multiple k as input, and takes a neural network model that expands the input and output of the neural network model and the hidden layer to a given multiple as output. This is due to the multiple interactions between the agent and the environment, and all constraints from the properties, deep reinforcement learning environment, and deep reinforcement learning model need to be encoded to be consistent with the verification depth k of the model checking algorithm.

[0059] Step S202: connect the single verification method with the deep neural network verifier through the general call method as an intermediary to obtain the binary deep reinforcement learning verification result;

[0060] The general calling method in the present invention takes a model and a deep reinforcement property script as input, and outputs the operation result of the external verifier, that is, whether the model satisfies the given property script and the time-consuming of this verification. For example, given a model and the property of a mountain car task, that is, when the car moves to the right at a high speed (speed not less than 0.03) at the bottom of the valley (coordinates between -0.6 and -0.4), it should not choose to accelerate to the left. The external verifier will check whether the model input is limited to -0.6 to -0.4 and above 0.03, and whether the output must be not to choose to accelerate to the left. If there is no such counterexample, the property of verification is established, otherwise it is not established. The method includes the following steps: saving the property script and model to a local file or to a container run by an external verifier; providing the path of the saved model and script file, calling the operation interface provided by the external verifier; constantly detecting whether the operation of the external verifier is finished; string interpretation and extraction of the operation results in text form returned by the external verifier, which will include the results of this verification and the time spent on verification.

[0061] The single verification method in the present invention includes three methods: bounded model checking method, k-induction method and reachability analysis method. All three methods take a model and a deep reinforcement learning property script as input, and output whether the given model satisfies the deep reinforcement learning property. For example, given a model of a mountain bike task and the properties of neural network verification, that is, when the car is at the bottom of the valley (coordinates between -0.6 and -0.4) and moves to the right at high speed (speed not less than 0.03) from the initial state, it should not choose to accelerate to the left, and the single verification method will return whether the given model satisfies the properties defined in the given deep reinforcement learning property script. Note that the difference between the properties here and the properties in the general call method is that it requires "starting from the initial state", which leads to the existence of a state where the car will not enter the valley and move to the right at high speed from the initial state. If it will not enter this state itself, then the required deep reinforcement learning property is necessarily established; therefore, the example property given by the general call method is a special case of a verification depth of 1 for the property here.

[0062] Step S203: If the deep reinforcement learning verification result shows that the property is established, it means that the additional training termination condition is met; if the deep reinforcement learning verification result shows that the property is not established, it means that the additional training termination condition is not met.

[0063] In general, the additional training termination condition is a property constraint for evaluating the safety and activity of the deep reinforcement learning model. The gap measurement method in the present invention takes a given deep reinforcement learning model and a predefined property constraint script as input, and takes the gap between the model currently being learned and the target property, that is, the gap between the model and the ideal model that meets the target property as output. For example, the constraint encodes a constraint requiring that in a mountain car task, when the car is at the bottom of the valley (coordinates between -0.6 and -0.4) and moves to the right at high speed (speed not less than 0.03), it should not choose to accelerate to the left, but through formal verification, it is found that the given model does not satisfy the given property constraint; but by relaxing the parameter values ​​in the constraint, for example, relaxing the not less than 0.03 in the high-speed movement to the right to not less than 0.04, it is found that the given model satisfies the relaxed property constraint, then the calculated gap is the difference between 0.03 and 0.04, 0.01. Formally speaking, the gap is defined in the form of the deviation between the expected parameter value in the predefined property and the actual parameter value extracted from the identified mutation point. Since the input usually contains multiple input features, the computational complexity of finding the mutation points of all input features is relatively high. Therefore, the gap measurement is limited to a single input feature or a parameter z in the action. Given the property of precondition P and postcondition Q The relaxation of φ is expressed as in, Relaxing the range of properties on the deep neural network N of the deep reinforcement learning model The process of measuring the state gap g(φ,N) between the current deep reinforcement learning model and the additional training termination condition is as follows:

[0064]

[0065] Where N represents the current deep reinforcement learning model, bps represents the set of mutation point properties within the range of the additional training termination condition for identifying relaxation, and φ[z] represents the specific value of the parameter z in the additional training termination condition of the deep reinforcement learning model. Represents the relaxed value of the parameter z in the additional training termination condition of the deep reinforcement learning model.

[0066] By measuring the state gap g(φ,N) between the current deep reinforcement learning model and the additional training termination condition, the progress of the deep reinforcement learning model in learning the additional training termination condition can be more accurately described in the form of numerical results, so that the learning rate scheduling function can be combined to better balance the performance and the learning of the additional training termination condition.

[0067] The identified mutation points correspond to the most stringent properties that the model can satisfy predefined safety or activity. For example, in the mountain bike example above, the model is expected to satisfy the safety property However, the identified mutation point shows that the most stringent property satisfied by the initial model is Accelerate to the left, and then you can reach By extracting numerical verification results from these properties, namely 0.05 and 0.04, it can be shown that the model is gradually becoming safer, which shows the effectiveness of the previous training strategy; on the contrary, if the value is getting farther and farther away from 0.03, it means that the model is gradually becoming less safe, which indicates that there is a problem with the previous training strategy.

[0068] Step S3: During the explanation process, the density of state features in the constraint space defined in the additional training termination condition is explained and measured by a density measurement method based on a sensitivity analysis method. The density of state features is a measure of the contribution of different input features to the output, which eliminates the influence of the order of magnitude differences of different features, making the calculation of subsequent reshaping rewards interpretable.

[0069] like Figure 3 , specifically including the following sub-steps:

[0070] Step S301, converting the density measurement problem into a sensitivity analysis problem on different state characteristics in the constraint space by using a density measurement method;

[0071] The density measurement method of the present invention measures the density of the input and output state characteristics of the property in the training constraint space, that is, the coefficient of the influence of different characteristics on the property verification result. For example, in the model of the mountain bike task, when the position coordinate is -0.5 and the speed is 0.03, the action taken by the model is to accelerate to the right. If the coordinate position is -0.51 and the speed is 0.03, the action taken by the model is still to accelerate to the right, but when the coordinate position is -0.5 and the speed is 0.02, the action taken by the model is to accelerate to the left. This shows that although both input feature values ​​have only changed by 0.01, around the position coordinate of -0.5 and the speed of 0.03, the change in speed has a greater impact on the output, that is, the density of the two is different. The density measurement method includes the following steps: determining whether the constraint space of the density measurement is an upper bound or a lower bound; using the sensitivity analysis interpretation method in the interpretation method to measure the sensitivity at the boundary of the constraint space. The disturbance subjected to the sensitivity measurement analysis here is a one-way disturbance, that is, when measuring the one-way sensitivity of the lower bound of the constraint space, the disturbance value is positive, otherwise it is negative. This is slightly different from the standard sensitivity analysis interpretation method in which the disturbance value has no restrictions on positive and negative.

[0072] Step S302: Apply a sensitivity analysis method on each state feature, and verify the possible outputs obtained by using the single verification method multiple times on the state feature boundary of the additional training termination condition of the current deep reinforcement learning model input being disturbed;

[0073] The sensitivity analysis interpretation method of the present invention inputs a given model and a model input, and outputs the change in the model output when different input features are disturbed at the model input. and perturbation ε, the sensitivity analysis problem aims to evaluate the relationship between N(x0) and , where x0 represents the input feature under discussion that is perturbed by ε. The sensitivity analysis of all input features reveals which input features with higher sensitivity results have the potential to cause larger fluctuations in the output. For example, given a mountain bike model, the action taken when the input coordinate is -0.6 and the speed is 0.03 is to accelerate to the left. Then, a perturbation of 0.01 is applied to the input coordinate and the output coordinate respectively. As a result, when the input coordinate is -0.61 or -0.59 and the speed is 0.03, the output taken by the model is still to accelerate to the left, while the output taken by the model when the input coordinate is -0.6 and the speed is 0.04 is to accelerate to the right; then this means that when the two input features coordinate and speed are at the given model input (-0.6, 0.03), and a perturbation of 0.01 is applied at the same time, the difference in perturbation caused by the coordinate feature to the output is 0 (remains unchanged), while the difference in perturbation caused by the speed feature to the output is 1 (from accelerating to the left to accelerating to the right), which means that the speed input here is more sensitive. The sensitivity analysis interpretation method includes the following steps: through the general interpretation method, set the discussion range of possible y0 (let y0 take the value around N(x0) and gradually expand its gap), so as to find the mutation point in the following property template:

[0074]

[0075] Where D represents the index set of the input feature in question, and D represents the index set of other input features; Calculate y0 and The minimum distance between them gives the sensitivity.

[0076] Step S303: The deep reinforcement learning verification results obtained through multiple verifications may change. The possible output corresponding to the change and the output obtained by inputting the state feature boundary in the additional training termination condition into the current deep reinforcement learning model are used as the density of the corresponding state feature.

[0077] The density of the state feature in the present invention is divided into the lower bound feature density of the state feature and the upper bound feature density

[0078]

[0079]

[0080] Where φ represents the additional training termination condition, s i represents the state of the i-th time slice defined in the additional training termination condition, Indicates i The j-th state feature in n represents the number of state features, φ[s i ] represents the constraint space defined in the additional training termination condition for s i The lower bound of the constraint is represents the constraint space defined in the additional training termination condition for s i The upper bound of the constraint, represents the disturbance s of the small disturbance value ε i ,for for N represents the model of deep reinforcement learning, N(s i ) represents the input state s of the deep reinforcement learning model i Output when .

[0081] Step S4: During the training process, the current deep reinforcement learning model interacts with the environment to generate trajectories, which are stored in a buffer. This step specifically includes the following sub-steps:

[0082] Step S401, generating a real-time detection program code from the constraint script written by the constraint translation method for the additional training termination condition;

[0083] The constraint translation method of the present invention takes the training constraint script encoded in the above language as input, and outputs a real-time program code for whether the model violates the training constraint. For example, based on the above language encoding, a constraint requirement is that in a mountain bike task, when the car is at the bottom of the valley (coordinates between -0.6 and -0.4) and moving to the right at high speed (speed not less than 0.03), it should not choose to accelerate to the left, because this will cause the car to reach the destination of the top of the mountain on the right. The constraint translation method will translate and output a program written in a general programming language, and the program can be run to determine whether the given constraint is satisfied. The method comprises the following steps: treating the script as a string, and using the string to split into the above three parts according to the separator; parsing each part into an abstract syntax tree using a public parsing tool; traversing each syntax node in the abstract syntax tree, determining whether the statement contains input and output variables and obtaining the size of the constraint constant in the statement when it is an input and output variable, thereby obtaining the input and output structure, replacing keywords, detecting a calculable mathematical expression and replacing it with its result, reading the start and end information of the loop statement, respectively bringing in the loop variable to expand the loop structure, and processing the relational definition domain; replacing the variable in the script with the given parameter value; processing the access table of the array and processing the branch judgment; converting the list array into a vector array; removing useless nested structures, such as if there is only one expression for the relational function parameter, then only retaining the expression without using the relational function; converting the property script statement into a program written in a general programming language, the generated program inputs the current environment state and the action taken by the model, and outputs whether the property is satisfied and the difference between the current state and the action and the boundary of the constraint space.

[0084] Step S402: Input the collected data set into the real-time detection program code, and the deep reinforcement learning model interacts with the environment to generate a trajectory consisting of the current state, current action, and current reward value, which is stored in a buffer;

[0085] Step S403: Detect whether the trajectory violates the additional training termination condition by a real-time detection method, and calculate the difference between the current state and the current action to the boundary of the additional training termination condition:

[0086] Diff(φ,s,a)=±dist(φ[s],s)×dist(φ[a],a)

[0087] Among them, Diff(φ,s,a) represents the difference between the current state and the current action to the training termination condition boundary, φ represents the training termination condition boundary, s represents the current state, a represents the current action, dist(φ[s],s) represents the distance from the current state s to the state-related training termination condition boundary, dist(φ[a],a) represents the distance from the current action to the action-related training termination condition boundary, + represents that the training termination condition is met, and - represents that the training termination condition is violated.

[0088] In the present invention, the real-time detection method takes the program code generated by the above-mentioned constraint translation method and an experience composed of the environmental state and the actions taken during the reinforcement learning training process as inputs, outputs whether the experience violates or complies with the given constraints, and calculates the difference between the current state and action and the boundary of the constraint space. This method reads the program code obtained by the above-mentioned constraint translation method into the memory, takes a given experience as a parameter input when given, and calls the above program. For example, when the coordinates are -0.45, the speed is 0.04, and the action taken is to accelerate to the left, the program will return a signal that the property is violated, and calculate the distances of 0.05 and 0.15 between the coordinate -0.45 and the constraint space, that is, between -0.6 and -0.4.

[0089] Step S5: Update the current reward value in the trajectory through the reward reshaping method of the state gap and the state feature density in the constraint space, and update the current deep reinforcement learning model according to the updated trajectory. Repeat steps S2 - S5. The reward reshaping method of the present invention enables the deep reinforcement learning model to consider the original performance while learning additional training termination conditions, improving the safety and activity of the deep reinforcement learning model. Specifically, it includes the following sub-steps:

[0090] Step S501: Determine the state feature density switching point according to the density of the state features in the constraint space defined in the training termination conditions Obtain the state features at each time point in the constraint space in the additional training termination conditions The value is related to The corresponding observed value in the trajectory experience The distance of

[0091]

[0092] Among them, p1 represents the penalty intensity coefficient. When the density p1 > 1 density, even a slight violation will measure a larger distance; on the contrary, when the density 0 < p1 < 1 density, only a serious violation will be measured with a larger distance. Usually, the density p1 is set to 1; Represents The lower bound value of Represents The upper bound value of

[0093] Step S502: According to Calculate the difference Diff(φ, s, a) between the observed values of all state features in the constraint space and the boundary of the additional training termination conditions; the distance dist(φ[s], s) from the current state s to the boundary of the state-related training termination conditions, which is the parameter for each dimension in the state The value of the The distance between the state observation value corresponding to the trajectory experience The weighted sum Among them hour on the contrary When an action is near the boundary of the property constraint space, it indicates that the degree of violation is relatively minor and the corresponding punishment should also be relatively minor. The farther away from the boundary, the more severe the punishment should be. Therefore, the punishment can also be proportional to the distance from the action to the boundary of the property constraint space. For simplicity, for the case of taking discrete actions, the distance dist(φ[a],a) from the current action to the boundary of the constraint space can be set to a constant.

[0094] Step S503: Adjust the learning rate of the deep reinforcement learning model based on the state gap g(φ,N) between the current deep reinforcement learning model and the additional training termination condition, and calculate the difference between all state features in the constraint space and the boundary of the additional training termination condition after adjusting the learning rate. Where Lr() represents the learning rate scheduling function;

[0095] Step S504: Adjust by scaling factor β As an intermediate reward, and trace the intermediate reward through the final track in the buffer area Among them, w represents the weight density, λ represents the discount factor;

[0096] Step S505: trace the intermediate rewards With the current reward value r t Reshape the current reward value together

[0097] For example, in the mountain bike task, the gap measurement obtains that the gap of the current model distance is 0.01, the density measurement obtains that the densities of the two input eigenvalues ​​are 0.1 and 1 respectively, the real-time monitoring method obtains that the current property constraint is violated, and the final reward reshaping method outputs an interactive reward of -1.5, which indicates that the action taken by the model in a given state violates the predetermined property, and the degree of violation of the predetermined property is specifically measured by the reward reshaping method. The reward reshaping method in the present invention believes that the penalty is proportional to the degree of property violation, and the degree of property violation is the deviation from the current state (i.e., the state where the violation occurs) to the state that satisfies the property, and this deviation can be equivalent to the distance between the boundary of the property constraint space and the state where the property violation occurs.

[0098] Based on the training data, the results of the method of the present invention and the training methods in the related art are compared, as shown in Table 1: the deep reinforcement learning model obtained by the training method of the present invention has better performance in the tasks of Mountain Car, Cartpole, Pendulum, B1 (a nonlinear system), B2 (another nonlinear system), and Tora (oscillator) than Trainify#1 (a counterexample-driven property constraint learning method, and trained with public parameters), Trainify#2 (the same method as Trainify#1, using the same parameters as the method of the present application), and the basic training method (i.e., a basic deep reinforcement learning training method that does not consider property constraints); in addition, in the additional training termination conditions consisting of safety or activity properties such as φ1 to φ6, the training method of the present invention can fully guarantee that the final deep reinforcement learning model satisfies such properties, while the model obtained by training in the related art does not have such a guarantee. Therefore, the deep reinforcement learning model obtained by the training method of the present invention has improved safety and activity compared with the related art.

[0099] Table 1: Comparison of the training results of the present invention and the existing training methods

[0100]

[0101] The deep reinforcement learning model training method based on formal verification and explanation of the present invention obtains the gap between the current deep reinforcement learning model and the given property through a formal verification method, and uses the contribution of different input features to the output of the model obtained by the explanation method as an indicator to iteratively train the deep reinforcement learning model so that the final trained model meets the required properties and maintains excellent performance.

[0102] like Figure 4 The present invention also provides a device for a deep reinforcement learning model based on formal verification and explanation, comprising:

[0103] The constraint translation module 501 is used to translate the training constraint script into a program code that determines in real time during the training process whether the deep reinforcement learning model violates the training constraint;

[0104] A real-time detection module 502, used to detect whether the deep reinforcement learning model violates or complies with a given property during the reinforcement learning process;

[0105] A gap measurement module 503 is used to measure the gap between the model currently being learned and the target property;

[0106] A density measurement module 504, for measuring the density of a selected input-output feature of a given deep reinforcement learning model in a property constraint space;

[0107] A reward reshaping module 505 is used to generate and reshape the interactive reward strategy in deep reinforcement learning training;

[0108] Interaction training module 506, used to control the deep reinforcement learning algorithm and reshape the reward through the reward strategy generated by the reward reshaping module to complete the deep reinforcement learning model update and environment interaction, and regularly test the performance of the model;

[0109] A learning module 507 combining training, verification, and explanation, which is used to automatically control the operation of all the above modules to return a deep reinforcement learning model that meets the given properties, and regularly report the current time consumption, performance of the deep reinforcement learning model, and property verification;

[0110] A single verification module 604 is used to formally verify whether the deep reinforcement learning model satisfies the properties corresponding to the given deep reinforcement learning property script when a deep reinforcement learning property script and a deep reinforcement learning model are given, and return a unified form of a single deep reinforcement learning formal verification result, that is, whether the property is satisfied or not satisfied and the accompanying counterexample. This module controls and parses the input and output of the property translation module, the model adaptation module and the general call method;

[0111] Mutation point identification module 606, used to quickly identify the properties of the given property template and the properties of the deep reinforcement learning model that have changed in the verification results within the specified parameter range;

[0112] The sensitivity analysis explanation module 705 is used to analyze and measure the perturbation value generated on the output of a given deep reinforcement learning model when the original input is given and the input features are perturbed.

[0113] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0114] An embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute the above-mentioned deep reinforcement learning model training method based on formal verification and explanation.

[0115] An embodiment of the present application also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the above-mentioned deep reinforcement learning model training method based on formal verification and explanation.

[0116] Figure 5It is a block diagram of an electronic device provided by the present invention. The electronic device 800 may include one or more of the following components: a processing component 801, a memory 802, a power component 803, a multimedia component 804, an audio component 805, an input / output (I / O) interface 806, a sensor component 807, and a communication component 808.

[0117] The processing component 801 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 801 may include one or more processors 810 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 801 may include one or more modules to facilitate the interaction between the processing component 801 and other components. For example, the processing component 801 may include a multimedia module to facilitate the interaction between the multimedia component 804 and the processing component 801.

[0118] The memory 802 is configured to store various types of data to support the operation of the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The power supply component 803 provides power to various components of the electronic device 800.

[0119] The power supply component 803 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800. The multimedia component 804 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments,

[0120] The multimedia component 804 includes a front camera and / or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0121] The audio component 805 is configured to output and / or input audio signals. For example, the audio component 805 includes a microphone (MIC), and when the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 802 or sent via the communication component 808. In some embodiments, the audio component 805 also includes a speaker for outputting audio signals.

[0122] The I / O interface 806 provides an interface between the processing component 801 and the peripheral interface module, which may be a keyboard, a click wheel, a button, etc. These buttons may include, but are not limited to, a home button, a volume button, a start button, and a lock button. The sensor component 807 includes one or more sensors for providing various aspects of status evaluation for the electronic device 800. For example, the sensor component 807 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor component 807 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800.

[0123] The sensor assembly 807 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 807 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 807 may also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0124] The communication component 808 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 808 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel.

[0125] In an exemplary embodiment, the communication component 808 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies. In an exemplary embodiment, the electronic device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method. In an exemplary embodiment, a non-temporary computer-readable storage medium including instructions is also provided, such as a memory 802 including instructions, and the above instructions can be executed by a processor 810 of the electronic device 800 to complete the above method. For example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0126] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0127] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A deep reinforcement learning model training method based on formal verification and explanation, characterized in that: The steps include: Step S1: Given an initial deep reinforcement learning model, and setting additional training termination conditions for the deep reinforcement learning model; Step S2: During the formal verification process, a single verification method is used to determine whether the current deep reinforcement learning model meets the additional training termination condition. If so, the training is terminated. Otherwise, the state gap between the current deep reinforcement learning model and the additional training termination condition is measured by the gap measurement method; Step S3, in the explanation process, explaining and measuring the density of state features in the constraint space defined in the additional training termination condition by a density measurement method based on a sensitivity analysis method; Step S4: During the training process, the current deep reinforcement learning model interacts with the environment to generate trajectories, which are stored in a buffer; Step S5: Update the current reward value in the trajectory through the reward reshaping method of the state gap and the state feature density in the constraint space, and update the current deep reinforcement learning model according to the updated trajectory, and repeat steps S2-S5.

2. A deep reinforcement learning model training method based on formal verification and explanation according to claim 1, characterized in that: The specific process of determining whether the current deep reinforcement learning model meets the additional training termination condition by the single verification method in step S2 is: Step S201: Convert the constraint script of the additional training termination condition and the current deep reinforcement learning model into a form recognizable by the deep neural network verifier through a property translation method and a model adaptation method; Step S202: connect the single verification method with the deep neural network verifier through the general call method as an intermediary to obtain the binary deep reinforcement learning verification result; Step S203: If the deep reinforcement learning verification result is that the property is established, it means that the additional training termination condition is met; If the deep reinforcement learning verification result is that the property does not hold, it means that the additional training termination condition is not met.

3. A deep reinforcement learning model training method based on formal verification and explanation according to claim 2, characterized in that: The constraint script includes: input and output structures that match the deep reinforcement learning model, constraint conditions composed of comparison expressions, the relationship between any two constraint conditions, and loop conditions for the interaction between the deep reinforcement learning model and the environment.

4. The deep reinforcement learning model training method based on formal verification and explanation according to claim 2 is characterized in that: The measurement process of the state gap g(φ,N) between the current deep reinforcement learning model and the additional training termination condition in step S2 is as follows: Where N represents the current deep reinforcement learning model, bps represents the set of mutation point properties within the range of the additional training termination condition for identifying relaxation, and φ[z] represents the specific value of the parameter z in the additional training termination condition of the deep reinforcement learning model. Represents the relaxed value of the parameter z in the additional training termination condition of the deep reinforcement learning model.

5. A deep reinforcement learning model training method based on formal verification and explanation according to claim 4, characterized in that: Step S3 includes the following sub-steps: Step S301, converting the density measurement problem into a sensitivity analysis problem on different state characteristics in the constraint space by using a density measurement method; Step S302: Apply a sensitivity analysis method on each state feature, and verify the possible outputs obtained by using the single verification method multiple times on the state feature boundary of the additional training termination condition of the current deep reinforcement learning model input being disturbed; Step S303: The deep reinforcement learning verification results obtained through multiple verifications may change. The possible output corresponding to the change and the output obtained by inputting the state feature boundary in the additional training termination condition into the current deep reinforcement learning model are used as the density of the corresponding state feature.

6. A deep reinforcement learning model training method based on formal verification and explanation according to claim 5, characterized in that: The density of the state feature in step S303 is divided into the lower limit feature density of the state feature and the upper bound feature density Where φ represents the additional training termination condition, s i represents the state of the i-th time slice defined in the additional training termination condition, Indicates i The j-th state feature in n represents the number of state features, φ [s i ] represents the constraint space defined in the additional training termination condition for s i The lower bound of the constraint is represents the constraint space defined in the additional training termination condition for s i The upper bound of the constraint, represents the disturbance s of the small disturbance value ε i , N represents the model of deep reinforcement learning, N(s i ) represents the input state s of the deep reinforcement learning model i Output when .

7. A deep reinforcement learning model training method based on formal verification and explanation according to claim 6, characterized in that: Step S4 includes the following sub-steps: Step S401, using a constraint translation method to generate a real-time detection program code from the constraint script of the additional training termination condition; Step S402: Input the collected data set into the real-time detection program code, and the deep reinforcement learning model interacts with the environment to generate a trajectory consisting of the current state, current action, and current reward value, which is stored in a buffer; Step S403: Detect whether the trajectory violates the additional training termination condition by a real-time detection method, and calculate the difference between the current state and the current action and the boundary of the additional training termination condition.

8. A deep reinforcement learning model training method based on formal verification and explanation according to claim 7, characterized in that: The specific process of calculating the difference Diff(φ,s,a) between the current state and the current action and the training termination condition boundary in step S403 is: Diff(φ,s,a)=±dist(φ[s],s)×dist(φ[a],a) Among them, φ represents the training termination condition boundary, s represents the current state, a represents the current action, dist(φ[s], s) represents the distance from the current state s to the training termination condition boundary related to the state, dist(φ[a], a) represents the distance from the current action to the training termination condition boundary related to the action, + represents that the training termination condition is met, and - represents that the training termination condition is violated.

9. A deep reinforcement learning model training method based on formal verification and explanation according to claim 8, characterized in that: Step 5 includes the following sub-steps: Step S501: Determine the state feature density switching point according to the density of the state feature in the constraint space defined in the training termination condition. Get the state features of each time point in the constraint space in the additional training termination condition Value and The corresponding observation value in the trajectory experience Distance Step S502: Calculate the difference Diff(φ,s,a) from the observed values ​​of all state features in the constraint space to the boundary of the additional training termination condition; Step S503: Adjust the learning rate of the deep reinforcement learning model based on the state gap g(φ,N) between the current deep reinforcement learning model and the additional training termination condition, and calculate the difference between all state features in the constraint space and the boundary of the additional training termination condition after adjusting the learning rate. Where Lr( ) represents the learning rate scheduling function; Step S504: Adjust by scaling factor β As an intermediate reward, and trace the intermediate reward through the final track in the buffer area Among them, w represents the weight density, λ represents the discount factor; Step S505: trace the intermediate rewards With the current reward value r t Reshape the current reward value together 10. A deep reinforcement learning model training method based on formal verification and explanation according to claim 9, characterized in that: In step S501 The acquisition process is: Among them, p1 represents the penalty intensity coefficient, express The lower bound value of express The upper bound value of .

Images