Secure reinforcement learning methods and devices, as well as intelligent agents and storage media

Abstract

This disclosure proposes a secure reinforcement learning method, a secure reinforcement learning device, an intelligent agent, and a storage medium, relating to the field of computer science. Embodiments of this disclosure decouple the secure reinforcement learning task into a baseline policy learning task and a secure policy correction task; based on the baseline policy learning task, a baseline policy is obtained; based on the secure policy correction task, a secure policy that satisfies security constraints and is closest to the baseline policy is learned by behavioral cloning and imitating the baseline policy; the secure policy is then output. Thus, by decoupling secure reinforcement learning into a dual-agent learning of baseline policy / secure policy, a secure policy can be obtained by fine-tuning the security aspect of the baseline policy, reducing the difficulty of finding a near-optimal secure policy, thereby improving sample efficiency and benefiting the learning of complex tasks.

Description

Technical Field

[0001] This disclosure relates to the field of computer science, particularly to the field of robot control, and especially to a secure reinforcement learning method, a secure reinforcement learning device, an intelligent agent, and a storage medium. Background Technology

[0002] Reinforcement learning (RL) is a type of machine learning in which an agent learns a policy to take the best action in an environment by performing actions and observing feedback. The goal of reinforcement learning is for the agent to automatically find the optimal policy over a continuous time series, typically maximizing cumulative reward. However, applying a policy learned based on this goal can potentially cause unintended side effects on the surrounding environment.

[0003] Safe reinforcement learning can improve upon the above problems. The goal of safe reinforcement learning is to maximize the cumulative reward while satisfying all given safety constraints. That is, it defines safety constraints based on task requirements and models and solves the reinforcement learning problem under certain limitations.

[0004] However, the difficulty of security reinforcement learning lies in the fact that the learning task itself is difficult, requiring a lot of exploration and other skills. But the optimization problem must also consider security constraints, which limits the optimization direction to a small range and is not conducive to learning complex tasks. Summary of the Invention

[0005] This disclosure decouples security reinforcement learning into dual-agent learning of baseline policy and security policy. The security policy can be obtained by fine-tuning the security policy based on the baseline policy, which reduces the learning difficulty of finding a near-optimal security policy and thus improves sample efficiency, which is beneficial for learning complex tasks.

[0006] This disclosure provides some embodiments of a secure reinforcement learning method, including:

[0007] The security reinforcement learning task is decoupled into a baseline policy learning task and a security policy correction task.

[0008] Based on the baseline policy learning task, a baseline policy is obtained;

[0009] Based on the security policy correction task, a security policy is obtained by performing security correction on the baseline policy, including: learning a security policy that satisfies security constraints and is closest to the baseline policy by imitating the baseline policy through behavioral cloning.

[0010] Output the security policy.

[0011] In some embodiments, learning a security policy that satisfies the security constraints and is closest to the baseline policy includes:

[0012] A dual objective function is constructed by associating the original objective function and the security constraint function using Lagrange multipliers. The original objective function is the function that minimizes the distance information between the security policy and the baseline policy, and the security constraint function is the function that makes the expected cumulative security index value greater than or equal to the security threshold.

[0013] For the dual objective function, the parameters of the security policy are updated using a gradient algorithm.

[0014] In some embodiments, the security policy that learns the security constraints and is closest to the baseline policy further includes:

[0015] For the dual objective function, the Lagrange multipliers are updated using the gradient algorithm.

[0016] In some embodiments, the method for calculating the distance information between the security policy and the baseline policy includes:

[0017] Based on the state of the training samples at each time step, and using a security strategy, estimate the actions of the training samples at each time step.

[0018] Based on the state of the training samples at each time step, the action of the training samples at each time step is estimated using the baseline strategy;

[0019] Calculate the expected distance information between the actions of the training samples estimated by the security policy at each time step and the actions of the training samples estimated by the baseline policy at each time step.

[0020] In some embodiments, the expected cumulative security indicator value is calculated using the following methods:

[0021] Based on the state of the training samples at each time step, and using a security strategy, estimate the actions of the training samples at each time step.

[0022] Based on the state and action of the training samples at each moment, the expected value of the cumulative safety index is calculated using the state and action value function of the cumulative safety index value.

[0023] In some embodiments, the security policy that learns the security constraints and is closest to the baseline policy further includes:

[0024] Set up one or more security value networks for evaluating the security policy, and a target network for the security policy and a target network for each security value network;

[0025] Based on the current security index value and the state at the next moment of the training samples, the first security policy evaluation value is calculated using the target network of the security policy and the target network of each security value network.

[0026] Based on the current state and actions of the training samples, the second security policy evaluation value is calculated using each security value network.

[0027] The parameters of the security value network are updated based on the distance information between the first security policy evaluation value and the second security policy evaluation value.

[0028] In some embodiments, calculating the first security policy evaluation value includes:

[0029] Based on the state of the training samples at the next moment, the target network of the security policy is used to estimate the action of the training samples at the next moment.

[0030] Based on the state and actions of the training samples at the next moment, the target security policy evaluation value is calculated using the target network of each security value network.

[0031] The first security policy evaluation value is calculated based on the target security policy evaluation value and the security index value of the training samples at the current moment.

[0032] In some embodiments, the security policy that learns the security constraints and is closest to the baseline policy further includes:

[0033] Update the parameters of the target network of the security policy in the current iteration round based on the parameters of the security policy in the previous iteration round; or,

[0034] Based on the parameters of the security value network in the current iteration and the parameters of the target network in the previous iteration, update the parameters of the target network in the current iteration.

[0035] In some embodiments, obtaining a baseline policy based on the baseline policy learning task includes:

[0036] Load a fixed baseline policy, which is either a pre-learned baseline policy or a non-learning-based control policy; or,

[0037] Baseline strategies are obtained through learning.

[0038] In some embodiments, the baseline strategy obtained through learning includes:

[0039] Based on the state of the training samples at each time step, the action of the training samples at each time step is estimated using the baseline strategy;

[0040] Based on the state and action of the training samples at each time step, the maximum expected value of the cumulative reward is calculated using the state-action value function of the cumulative reward.

[0041] To maximize the expected cumulative reward, the parameters of the baseline policy are updated using a gradient algorithm.

[0042] In some embodiments, the baseline strategy obtained through learning further includes:

[0043] Set up one or more reward value networks for evaluating the baseline policy, and a target network for the baseline policy and a target network for each reward value network;

[0044] Based on the current reward and the state at the next moment of the training samples, the first baseline policy evaluation value is calculated using the target network of the baseline policy and the target network of each reward value network.

[0045] Based on the current state and actions of the training samples, the second baseline policy evaluation value is calculated using each reward value network;

[0046] The parameters of the reward value network are updated based on the distance information between the first baseline policy evaluation value and the second baseline policy evaluation value.

[0047] In some embodiments, calculating the first baseline policy evaluation value includes:

[0048] Based on the state of the training sample at the next moment, the target network of the baseline policy is used to estimate the action of the training sample at the next moment.

[0049] Based on the state and action of the training samples at the next moment, the target baseline policy evaluation value is calculated using the target network of each reward value network.

[0050] The first baseline policy evaluation value is calculated based on the target baseline policy evaluation value and the reward of the training sample at the current moment.

[0051] In some embodiments, the baseline strategy based on learning further includes:

[0052] Update the parameters of the target network of the baseline policy in the current iteration round based on the parameters of the baseline policy in the current iteration round and the parameters of the target network of the baseline policy in the previous iteration round; or,

[0053] Update the parameters of the target network of the reward value network in the current iteration based on the parameters of the reward value network in the current iteration and the parameters of the target network of the reward value network in the previous iteration.

[0054] In some embodiments, the method for generating the training samples required by the security reinforcement learning method includes:

[0055] Randomly select a security policy or baseline policy to generate the corresponding action for the current state and receive the reward;

[0056] Calculate safety index values ​​based on the cost function of actions violating safety requirements;

[0057] The current state, action, reward, safety indicator value, and the state at the next moment are used as a training sample.

[0058] In some embodiments, the security reinforcement learning task is an agent security control learning task, and in the training samples of the agent security control learning task: the state is the state of the environment in which the agent is located, the action is the action of the agent, and the reward is the reward of environmental feedback.

[0059] In some embodiments, the cumulative security index value is a cumulative discount security index value calculated based on the security index value and its discount factor.

[0060] In some embodiments, the cumulative reward is a cumulative discount reward calculated based on the reward and its discount factor.

[0061] In some embodiments, the security policy and each security value network are neural networks, the target network of the security policy and the target network of each security value network are neural networks, the security policy and each security value network are initialized by copying the network parameters of the security policy to initialize the target network of the security policy, and by copying the network parameters of each security value network to initialize the target network of each security value network.

[0062] In some embodiments, the baseline policy and each reward value network are neural networks, the target network of the baseline policy and the target network of each reward value network are neural networks, the baseline policy and each reward value network are initialized, the target network of the baseline policy is initialized by copying the network parameters of the baseline policy, and the target network of each reward value network is initialized by copying the network parameters of each reward value network.

[0063] In some embodiments, the method further includes: the agent performing an action based on the current state of the environment using the security policy, causing the environment to respond and transition to the next state, obtaining a reward from the environment feedback, performing a new action based on the next state of the environment using the security policy, and maximizing the accumulated reward during each action selection process.

[0064] This disclosure provides some embodiments of a security reinforcement learning device, including:

[0065] The decoupling module is configured to decouple the security reinforcement learning task into a baseline policy learning task and a security policy correction task.

[0066] The baseline policy learning module is configured to obtain a baseline policy based on the baseline policy learning task.

[0067] The security policy correction module is configured to perform security correction on the basis of the baseline policy to obtain a security policy based on the security policy correction task, including: learning a security policy that satisfies security constraints and is closest to the baseline policy by imitating the baseline policy through behavior cloning.

[0068] The output module is configured to output the security policy.

[0069] Some embodiments of this disclosure provide a security reinforcement learning apparatus, including: a memory; and a processor coupled to the memory, the processor being configured to execute a security reinforcement learning method based on instructions stored in the memory.

[0070] Some embodiments of this disclosure propose an intelligent agent, including a security reinforcement learning device.

[0071] In some embodiments, the intelligent agent includes a robot.

[0072] Some embodiments of this disclosure propose a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of a secure reinforcement learning method. Attached Figure Description

[0073] The accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. This disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings.

[0074] Obviously, the accompanying drawings described below are merely some embodiments of this disclosure. Those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0075] Figure 1 This diagram illustrates the differences between the baseline policy trajectory and the security policy trajectory in some embodiments of this disclosure.

[0076] Figure 2 The diagram illustrates a flowchart of a dual-agent-based security reinforcement learning method according to some embodiments of this disclosure.

[0077] Figure 3 The diagram illustrates a learning-based baseline strategy based on some embodiments of this disclosure.

[0078] Figure 4This diagram illustrates the learning of security policies according to some embodiments of the present disclosure.

[0079] Figure 5 A schematic diagram illustrating parameters of an updated security policy according to some embodiments of this disclosure is shown.

[0080] Figure 6 The diagram illustrates a dual-agent-based security reinforcement learning method according to some embodiments of this disclosure.

[0081] Figure 7 A schematic diagram of the structure of a security reinforcement learning apparatus according to some embodiments of the present disclosure is shown.

[0082] Figure 8 A schematic diagram of the structure of a security reinforcement learning apparatus according to some embodiments of the present disclosure is shown.

[0083] Figure 9 A schematic diagram of the structure of an intelligent agent according to some embodiments of this disclosure is shown. Detailed Implementation

[0084] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0085] Unless otherwise stated, the terms "first," "second," etc., used in this disclosure are used to distinguish different objects and are not used to indicate size or sequence.

[0086] Reinforcement learning primarily consists of an agent, an environment, states, actions, and rewards. The agent interacts with the environment through states, actions, and rewards. The agent, the intelligent entity, is the ontology of reinforcement learning, acting as a learner or decision-maker. Any independent entity capable of thinking and interacting with its environment can be abstracted as an agent. Examples of agents include, but are not limited to, robots. The environment encompasses everything except the agent, primarily composed of a set of states. States represent environmental data, and the state space (or state set) contains all possible states in the environment. Actions are the actions the agent can perform, and the action space (or action set) contains all possible actions the agent can take. Rewards are the negative or positive feedback information the agent receives from the environment after performing an action; the reward set contains all feedback information the agent can obtain. Reinforcement learning learns the mapping from the states of the environment to the actions of the agent; this mapping relationship is the policy. The interaction process between an agent and its environment includes: after the agent performs an action based on the current state of the environment, the environment transitions from its current state to a new state and provides a reward (positive or negative) to the agent. Subsequently, the agent executes a new action according to a certain strategy based on the new state of the environment, maximizing the accumulated reward during each action selection and execution. Thus, the agent learns the optimal decision sequence during its interaction with the environment.

[0087] For a safety reinforcement learning task, consider risk-aware policy optimization within a Constrained Markov Decision Process (CMDP). In CMDP, the agent not only implements policies and receives rewards but also ensures that certain indicators of the environmental state do not exceed safety limits. The reward is a cumulative reward over time steps; with the introduction of the concept of a trajectory, the reward is also the sum of all rewards along that trajectory. This process can be represented by tuples. It indicates. Among them, and Let represent the state space of the environment and the action space of the agent, respectively, and let s and a represent a state in the state space and an action in the action space, respectively. It is a reward function. Indicates from s t ,a t to s t+1 That is, from the state and action at time step t to the state at time step t+1, therefore... This represents the reward function that generates a reward from the state and action at time step t to the state at time step t+1. This indicates the range of rewards, where r represents a specific reward value. It is a transition probability function describing environmental dynamics, that is, the transition probability function from the state and action at time step t to the state at time step t+1. It is a cost function that represents the cost of violating safety requirements, that is, the cost of changing from the state and actions at time step t to the state at time step t+1.

[0088] Random Policy It maps a given state s to the action space. The probability distribution on the x-axis can be simply denoted as π. For example, deterministic strategies can be characterized using distributions such as the Dirac-delta.

[0089] The goal of reinforcement learning is to maximize cumulative reward, which could be, for example, a cumulative discount reward J. R (π).

[0090]

[0091] Specifically, the behavioral trajectory τ is obtained by sampling the agent's behavior based on policy π. s t ,a t ,r t Let represent the state, action, and reward at time step t, respectively. The first discount factor γ∈(0,1] guarantees that the geometric series converges as the time range T approaches infinity. It represents mathematical expectation, or simply expectation.

[0092] J R The state value function of (π) is defined as:

[0093]

[0094] in, Indicates that J is based on policy π. R (π) is a value function of π in state s.

[0095] J R The state and action value function of (π) is defined as:

[0096]

[0097] in, Indicates that J is based on policy π. R (π) is a value function under state s and action a.

[0098] The optimization problem of reinforcement learning can be formulated as:

[0099]

[0100] Formula (1) represents the calculation based on policy π for each state s of the behavior trajectory τ. The maximum value of the mathematical expectation.

[0101] As previously noted, the reinforcement learning method of Equation (1) often leads to unsafe policies, posing potential risks to the environment or the agent itself. Therefore, a safety metric value I is defined at each time step:

[0102]

[0103] in, It is the cost function mentioned earlier. Represents the state s from time step t. t and action a t The state s at time step t+1 t+1 If the cost of violating safety requirements is greater than 0, the system is considered unsafe, and the corresponding safety index value I is 0. If the cost is equal to 0, the system is considered safe, and the corresponding safety index value I is 1. The specific value of the safety index value I is not limited to this example.

[0104] State safety is defined as follows: policy π is (1-δ) safe, where δ is a preset value, and for any time t, we can obtain:

[0105]

[0106] That is, the expected value of the safety index I is ≥1-δ.

[0107] Considering the impact of sequential decision-making, refer to the value function of the state based on the cumulative discount reward of policy π. The definition of is then the value function of the state based on the cumulative discount security index value of strategy π. Among them, I k The second discount factor represents the safety index value at time step k. To ensure that the geometric series converges as the time interval T approaches infinity, a relatively small threshold can be set. This makes it relatively short-sighted in terms of future security. This represents the mathematical expectation. Correspondingly, it's a value function of the state and action based on the cumulative discount security index value of strategy π.

[0108] based on By definition, formula (3) can be restated as:

[0109]

[0110] Set a safety threshold

[0111] Combining formulas (1) and (4), the optimization of formula (1) needs to satisfy the security constraints of formula (4), that is, the security reinforcement learning task is modeled as:

[0112]

[0113] While directly solving formula (5) is theoretically feasible, it is practically difficult to solve in the task of learning safe control for intelligent agents. This is because it often falls into the exploration-exploitation dilemma, where learning a complex task requires extensive exploration in an unknown environment, but hard constraints on safety limit the output action to a small, safety-verified region. Furthermore, conservative policy updates limit data efficiency, making it impractical for learning in complex and sample-intensive scenarios.

[0114] The inventors studied a large number of robot learning tasks and found that the safety policy (a policy that takes safety constraints into account) and the baseline policy (a policy that ignores safety constraints) are similar in general trend, but the safety policy modifies some of the outputs relative to the baseline policy to avoid danger.

[0115] Taking a robot learning task as an example, suppose the robot is required to move an object from a preset starting point to a preset destination. Figure 1 As shown, according to the baseline strategy (a strategy that ignores safety constraints), the robot will move the object along the shortest path from the starting point to the destination, regardless of whether there are obstacles along the way; according to the safety strategy (a strategy that takes safety constraints into account), the robot will correct the movement path to a curve to avoid obstacles.

[0116] Based on the above findings, this disclosure proposes a dual-agent secure reinforcement learning method, which decouples secure reinforcement learning into dual-agent learning of a baseline policy and a secure policy. The baseline agent explores the environment and learns a baseline policy aimed at maximizing cumulative rewards, while the secure agent quickly acquires basic skills by imitating the baseline policy through online behavior cloning. At the same time, the secure policy can be obtained by fine-tuning the security aspect of the baseline policy. This significantly reduces the learning difficulty of finding a near-optimal secure policy, thereby improving sample efficiency and benefiting the learning of complex tasks such as robot learning tasks.

[0117] Figure 2 The diagram illustrates a flowchart of a dual-agent-based security reinforcement learning method according to some embodiments of this disclosure.

[0118] like Figure 2 As shown, the security reinforcement learning method based on dual agents in this embodiment includes steps 210 to 260.

[0119] In step 210, the security reinforcement learning task is decoupled into a baseline policy learning task and a security policy correction task.

[0120] Security reinforcement learning tasks include, but are not limited to, agent security control learning tasks. Agents include, but are not limited to, intelligent robots.

[0121] The decoupled baseline policy learning task is an independent and unconstrained learning task, thus all reinforcement learning training techniques and exploration strategies are applicable. For example, the baseline policy learning task can be performed by a baseline agent exploring the environment and learning a baseline policy aimed at maximizing cumulative rewards. The safety policy correction task can be performed by a safety agent quickly acquiring basic skills by imitating the baseline policy through online behavioral cloning, while simultaneously fine-tuning the safety policy based on the baseline policy to obtain a safety policy. This significantly reduces the learning difficulty of finding a near-optimal safety policy, thereby improving sample efficiency and benefiting the learning of complex tasks such as robot learning.

[0122] The baseline agent aims to learn a baseline policy π that maximizes cumulative rewards. B , represented as:

[0123]

[0124] Formula (6) represents the formula for each state s of the behavioral trajectory τ, based on the baseline policy π. B calculate The maximum value of the mathematical expectation, where, For policy π B The cumulative reward function in state s Behavioral trajectory τ is a policy-based behavior trajectory of an agent. B The behavior was sampled.

[0125] Security agents aim to learn and be able to perceive security strategies. S At the same time, it is as close as possible to the baseline strategy π B , represented as:

[0126]

[0127] Formula (7) means that when the following conditions are met... Under the premise of safety constraints, calculate the distance closest to π. B π S .in, Indicates the distance closest to π B π S , Indicates based on security policy π S The cumulative discount safety index value is a state-value function, and its calculation formula is referenced. The difference between the two lies in the fact that the behavioral trajectory τ is based on π. S Generate or generate based on π, Let represent the states s for each behavior trajectory τ. The mathematical expectation of Γ is the safety threshold, which is a hyperparameter.

[0128] In step 220, the network is initialized.

[0129] Network initialization includes: initializing security policy π φ and one or more security value networks used to evaluate the security policy. The security strategy and each security value network are, for example, neural networks. π φ The security policy π represents the network parameter φ. S It can also be written as π. S (φ). The network parameter is ξ. i The i-th security value network, where i ≥ 1. For example, suppose there are 2 security value networks when i = 2. and When evaluating security policies using multiple security value networks, improve the stability of the evaluation network itself to avoid overestimation of security policies due to the instability of the evaluation network itself.

[0130] Network initialization also includes: initializing the target network π for the security policy. φ′ and the target network of each security value network The target network of the security strategy and the target network of each security value network are, for example, neural networks. φ′ This represents the target network for a security policy with network parameters φ′. The network parameter is ξ′ i The target network of the i-th security value network, i≥1. Wherein, the target network of the security policy is initialized by copying the network parameters of the security policy, i.e., φ′←φ, and the target network of each security value network is initialized by copying the network parameters of each security value network, i.e., ξ′. i ←ξ i Assume that when i = 2, ξ′1←ξ1, ξ′2←ξ2.

[0131] Network initialization also includes initializing the experience replay pool used to store training samples. After initializing the Lagrange multiplier λ = 0, the safety threshold Γ can also be calculated according to formula (5).

[0132] If the baseline policy is obtained through learning, the network initialization also includes: initializing the baseline policy π. θ and one or more reward value networks used to evaluate the baseline strategy. The baseline policy and each reward value network are, for example, neural networks. πθ The baseline policy π represents the network parameter θ. B It can also be written as π. B (θ). Indicates the network parameter as ω i The i-th reward value network, where i ≥ 1. For example, suppose there are 2 reward value networks when i = 2. and When evaluating a baseline policy using multiple reward value networks, the stability of the evaluation network itself is improved to avoid overestimation of the baseline policy due to the instability of the evaluation network itself.

[0133] If the baseline policy is obtained through learning, the network initialization also includes: the target network π for initializing the baseline policy. θ′ and the target network of each reward value network The target network of the baseline policy and the target network of each reward value network are, for example, neural networks. θ′ This represents the target network of the baseline strategy with network parameters θ′. The network parameter is ω′ i The target network of the i-th reward value network, i≥1. Wherein, the target network of the baseline policy is initialized by copying the network parameters of the baseline policy, i.e., θ′←θ, and the target network of each reward value network is initialized by copying the network parameters of each reward value network, i.e., ω′. i ←ω i Assuming that when i = 2, ω′1 ← ω1, ω′2 ← ω2.

[0134] In step 230, training samples are obtained.

[0135] A training sample for a safety reinforcement learning task may include information such as the current state, action, reward, safety metric value, and the state at the next moment. When the safety reinforcement learning task is an agent safety control learning task, a training sample for the agent safety control learning task may include information such as the current state of the environment, the agent's action, the reward from the environment, the safety metric value, and the state of the environment at the next moment.

[0136] A batch of training samples containing N interaction data points is sampled from the experience replay pool D. Each interaction data point (s, a, r, I, s') is a training sample. For a security reinforcement learning task, s, a, r, I, s' represent the current state, action, reward, security metric value, and next state, respectively. For an agent-based security control learning task, s, a, r, I, s' represent the current state of the environment, the agent's action, the reward from the environment, the security metric value, and the next state of the environment, respectively.

[0137] In some embodiments, the method for generating training samples in the experience replay pool D includes:

[0138] First, randomly select security policy π φ Or baseline strategy π θ To generate the action a corresponding to the current state s, that is, to obtain the corresponding action a from the probability distribution π(s) that maps the given state s to the action space, denoted as a~π(s), where π is the value of π. φ or π θ .

[0139] Then, noise ∈ can be added to action a to obtain a+∈, where ∈ is from a normal distribution. Mid-sampling, denoted as That is, ∈ follows a normal distribution with a mathematical expectation of 0 and a standard deviation of σ.

[0140] Next, according to formula (2), the safety index value I is calculated based on the cost function of the action violating safety requirements.

[0141] Next, the reward for the action is obtained, which is a reward based on environmental feedback.

[0142] Next, the current state, action, environmental feedback reward, safety index value, and the state at the next moment are used as a training sample and stored in the experience replay pool D.

[0143] Repeat the above process, alternating between the safety policy and the baseline policy to generate the training samples needed for learning. Utilizing different policies to obtain novel state-action pairs creates diverse training samples, thereby improving the training effectiveness of the policy.

[0144] In step 240, a baseline policy is obtained based on the baseline policy learning task.

[0145] Based on the baseline policy learning task, there are several methods to obtain the baseline policy: one method is to perform baseline policy learning based on the baseline policy learning task and obtain the baseline policy based on the learning; another method is to load a fixed baseline policy, which is either a baseline policy obtained in advance based on learning or a control policy that is not based on learning. The first method is mainly described below.

[0146] like Figure 3 As shown, the baseline strategy obtained through learning includes steps 240-1 to 240-3.

[0147] In step 240-1, update the parameters of each reward value network.

[0148] First, based on the current reward and the state at the next moment of the training samples, the first baseline policy evaluation value is calculated using the target network of the baseline policy and the target network of each reward value network.

[0149] Specifically, based on the state s′ of the training samples at the next time step, the target network π is used with the baseline policy. θ′ Estimate the action π of the training sample at the next moment. θ′ (s′); based on the state s′ and action π of the training sample at the next moment. θ′ (s′), utilizing the target network of each reward value network Calculate the target baseline policy evaluation value; for example, take the minimum evaluation value of the target network against the baseline policy across all reward value networks. As the target baseline policy evaluation value; based on the target baseline policy evaluation value and the reward r of the training samples at the current time step, calculate the first baseline policy evaluation value y, for example, Where γ is the first discount factor, γ∈(0,1). In all embodiments of this disclosure, ← represents assignment, with the right side being assigned to the left side.

[0150] Next, based on the current state s and action a of the training samples, the reward value network is used for each sample. Calculate the second baseline strategy evaluation value

[0151] Then, the parameters of the reward value network are updated based on the distance information between the first baseline policy evaluation value and the second baseline policy evaluation value.

[0152] For example, calculate the first baseline policy evaluation value y and the second baseline policy evaluation value y. The parameters of the reward value network that minimizes the distance between them And update the parameter ω of the reward value network. i ,Right now, N represents the number of training samples in this training round.

[0153] In step 240-2, update the parameters of the baseline strategy.

[0154] First, based on the state s of the training samples at each time step, the baseline strategy π with parameter determinism is used. B (θ), to estimate the action π of the training sample at each time step. B (s;θ;Based on the state s and action π of the training samples at each time step B (s; θ), utilizing the value function of state and action with cumulative reward. Calculate the maximum expected cumulative reward, see formula (8) for details. Baseline policy π based on parameter determinism. B (θ), Formula (6) can be expressed as Formula (8).

[0155]

[0156] Next, for the maximum expected cumulative reward in Equation (8), the parameters of the baseline policy are updated using a gradient algorithm, such as the gradient ascent algorithm based on θ. See Equation (9) for details.

[0157]

[0158] Where, η θ The table shows the learning rate for θ. This represents the gradient operator based on θ. This represents the gradient operator based on 'a', and ← indicates assignment, where the value on the right is assigned to the value on the left.

[0159] In step 240-3, update the parameters of the target network of the baseline policy and the parameters of the target network of each reward value network.

[0160] For example, the Polyak averaging method can be used to update the parameters of the target network of the baseline policy in the current iteration based on the parameters of the baseline policy in the current iteration and the parameters of the target network of the baseline policy in the previous iteration.

[0161] For example, the sliding coefficient is multiplied by the parameters of the target network of the baseline policy in the previous iteration, and the difference between 1 and the sliding coefficient is multiplied by the parameters of the baseline policy in the current iteration. The sum of these two products is used as the parameters of the target network of the baseline policy in the current iteration. The sliding coefficient ranges from [0,1), for example, sliding coefficient = 0.9.

[0162] For example, the Polyak averaging method can be used to update the parameters of the target network of the reward value network in the current iteration based on the parameters of the reward value network in the current iteration and the parameters of the target network of the reward value network in the previous iteration.

[0163] For example, the sliding coefficient is multiplied by the parameter of the target network of the reward value network in the previous iteration, and the difference between 1 and the sliding coefficient is multiplied by the parameter of the reward value network in the current iteration. The sum of these two products is used as the parameter of the target network of the reward value network in the current iteration. The sliding coefficient ranges from [0,1), for example, sliding coefficient = 0.9.

[0164] In step 250, based on the security policy correction task, security policy learning is performed, that is, security policy is obtained by performing security correction on the basis of the baseline policy.

[0165] The security policy obtained by performing security correction based on the baseline policy includes: learning a security policy that satisfies the security constraints and is closest to the baseline policy by imitating the baseline policy through behavioral cloning, based on the baseline policy and security constraints.

[0166] like Figure 4 As shown, security policy learning includes steps 250-1 to 250-3.

[0167] In step 250-1, update the parameters of each security value network.

[0168] First, based on the current security index value and the state at the next moment of the training samples, the first security policy evaluation value is calculated using the target network of the security policy and the target network of each security value network.

[0169] For example, based on the state s′ of the training samples at the next time step, the target network π is utilized with a security policy. φ′ Estimate the action π of the training sample at the next moment. φ′ (s′); based on the state s′ and action π of the training sample at the next moment. φ′ (s′), utilizing the target network of each security value network Calculate the target security policy evaluation value; for example, take the minimum evaluation value of the target network for the security policy across all security value networks. The target security policy evaluation value is used as the target security policy evaluation value; based on the target security policy evaluation value and the current security index value I of the training samples, the first security policy evaluation value is calculated. For example, in, As the second discount factor,

[0170] Next, based on the current state s and action a of the training samples, each security value network is used. Calculate the evaluation value of the second security strategy

[0171] Finally, the parameters of the security value network are updated based on the distance information between the first security policy evaluation value and the second security policy evaluation value.

[0172] For example, the calculation makes the first security policy evaluation value Compared with the second security strategy evaluation value The parameters of the security value network with the minimum distance between them And update the parameters ξ of the security value network. i ,Right now, N represents the number of training samples in this training round.

[0173] In step 250-2, update the parameters of the security policy, such as... Figure 5 As shown.

[0174] In some embodiments, updating the parameters of the security policy includes steps 250-2A to 250-2E.

[0175] In step 250-2A, the distance information between the security policy and the baseline policy is calculated.

[0176] For example, based on the state s of the training samples at each time step, the security policy π is used. S (φ), estimating the action π of the training sample at each time step. S (s;φ);Based on the state s of the training samples at each time step, the baseline policy π is used. B (θ), to estimate the action π of the training sample at each time step. B (s;θ;);Calculate the expectation of the distance information between the actions of the training samples for security policy estimation at each time step and the actions of the training samples for baseline policy estimation at each time step.

[0177] In step 250-2B, the expected value of the cumulative safety index is calculated.

[0178] For example, based on the state s of the training samples at each time step, the security policy π is used. S (φ), estimating the action π of the training sample at each time step. S (s;φ);Based on the state s and action π of the training samples at each time step S (s;φ), utilizing the value function of the state and action based on the accumulated safety index value. Calculate the expected value of the cumulative safety index

[0179] In step 250-2C, the original objective function and the security constraint function are associated through the Lagrange multiplier λ to construct the dual objective function. The original objective function (the first expression in Formula 10) is the function that minimizes the distance information between the security policy and the baseline policy, and the security constraint function (the second expression in Formula 10) is the function that the expected cumulative security index value is greater than or equal to the security threshold.

[0180] Baseline strategy π based on parameter determinism B (θ) and parameter deterministic security strategy π S (φ), Formula (7) can be expressed as Formula (10):

[0181]

[0182] in, L represents 0 - Norm distance, st indicates that it satisfies the norm, and the meanings of other symbols are as described above.

[0183] The dual objective function of formula (10) is:

[0184]

[0185] In step 250-2D, for the dual objective function, the parameters of the security policy are updated using a gradient algorithm, such as a gradient descent algorithm based on φ.

[0186]

[0187] Where, η φ φ represents the learning rate. This represents the gradient operator based on φ, and ← indicates assignment, where the value on the right is assigned to the value on the left.

[0188] In step 250-2E, for the dual objective function, the Lagrange multipliers are updated using a gradient algorithm, such as the gradient ascent algorithm based on λ.

[0189]

[0190] Where, η λ λ represents the learning rate, and ← indicates assignment, where the value on the right is assigned to the value on the left.

[0191] At the start of the learning process, set λ = 0 and set a small η. λ For example, 0.001, which enables the agent to quickly learn skills from the baseline policy and gradually correct dangerous behaviors through interaction.

[0192] Furthermore, in a single gradient ascent / descent step, the timescale of the original variable update is faster than that of the Lagrange multipliers, allowing the learning rate of φ to be set larger than that of λ, i.e., η. φ 》η λ .

[0193] In step 250-3, update the parameters of the target network of the security policy and the parameters of the target network of each security value network.

[0194] For example, the Polyak averaging method can be used to update the parameters of the target network of the security policy in the current iteration round based on the parameters of the security policy in the current iteration round and the parameters of the target network of the security policy in the previous iteration round.

[0195] For example, the sliding coefficient is multiplied by the parameters of the target network of the security policy in the previous iteration, and the difference between 1 and the sliding coefficient is multiplied by the parameters of the security policy in the current iteration. The sum of these two products is used as the parameters of the target network of the security policy in the current iteration. The sliding coefficient ranges from [0,1), for example, sliding coefficient = 0.9.

[0196] For example, the Polyak averaging method can be used to update the parameters of the target network of the security value network in the current iteration based on the parameters of the security value network in the current iteration and the parameters of the target network of the security value network in the previous iteration.

[0197] For example, the sliding coefficient is multiplied by the parameter of the target network of the security value network in the previous iteration, and the difference between 1 and the sliding coefficient is multiplied by the parameter of the security value network in the current iteration. The sum of these two products is used as the parameter of the target network of the security value network in the current iteration. The sliding coefficient ranges from [0,1), for example, sliding coefficient = 0.9.

[0198] In step 260, after the learning is completed, the security policy is output.

[0199] In addition, baseline strategies can be retained for migration to similar tasks.

[0200] Figure 6 A schematic diagram of security reinforcement learning based on dual agents is shown. Figure 6 As shown, the security reinforcement learning task is decoupled into a baseline policy learning task and a security policy correction task. The baseline policy learning task involves a baseline agent exploring the environment and learning a baseline policy π aimed at maximizing cumulative reward. B The security policy calibration task can be accomplished by a security agent quickly acquiring basic skills by cloning and mimicking a baseline policy online. Simultaneously, fine-tuning the security constraints based on the baseline policy yields the resulting security policy π. S Baseline strategy π for parameter determinism B , π B Based on the input environment state s, output the corresponding action a. B Security strategies π for parameter determinism S , π S Based on the input environment state s, output the corresponding action a. S Furthermore, after an action, the environment transitions to a new state s', and the environment can provide a corresponding reward r. It can also calculate a safety indicator value I based on the cost function of the action violating safety requirements. These interactive data (s, a(a)) B / a S ), r, I, s') can be stored in the experience replay pool D and can be used as training samples.

[0201] The above embodiments decouple security reinforcement learning into dual-agent learning of a baseline policy and a security policy. By fine-tuning the security policy based on the baseline policy, the security policy can be obtained, reducing the difficulty of finding a near-optimal security policy and thus improving sample efficiency, which is beneficial for learning complex tasks. Furthermore, when transferring to a new risky environment, only the security policy can be modified while the baseline policy is frozen, significantly improving sample efficiency compared to training from scratch.

[0202] After obtaining the security policy, the agent can use it to perform actions. That is, based on the current state of the environment, the agent uses the security policy to perform actions, causing the environment to respond and transition to the next state, obtaining rewards from the environment's feedback, and then using the security policy to perform new actions based on the next state of the environment, maximizing the accumulated rewards during each action selection process.

[0203] Since a safety policy is a policy learned by satisfying safety constraints based on a baseline policy, an agent can perform actions more safely by using a safety policy.

[0204] Figure 7 A schematic diagram of the structure of a security reinforcement learning apparatus according to some embodiments of the present disclosure is shown.

[0205] like Figure 7 As shown, the security reinforcement learning device 700 of this embodiment includes:

[0206] Decoupling module 710 is configured to decouple the security reinforcement learning task into a baseline policy learning task and a security policy correction task.

[0207] The baseline policy learning module 720 is configured to obtain a baseline policy based on the baseline policy learning task.

[0208] The security policy correction module 730 is configured to perform security correction on the baseline policy based on the security policy correction task to obtain a security policy.

[0209] Output module 740 is configured to output the security policy.

[0210] In some embodiments, the security policy correction module 730 is configured to learn a security policy that satisfies the security constraints and is closest to the baseline policy by imitating the baseline policy through behavior cloning, based on a baseline policy and security constraints.

[0211] In some embodiments, the security policy correction module 730 is configured to learn a security policy that satisfies the security constraints and is closest to the baseline policy, including:

[0212] A dual objective function is constructed by associating the original objective function and the security constraint function using Lagrange multipliers. The original objective function is the function that minimizes the distance information between the security policy and the baseline policy, and the security constraint function is the function that makes the expected cumulative security index value greater than or equal to the security threshold.

[0213] For the dual objective function, the parameters of the security policy are updated using a gradient algorithm;

[0214] For the dual objective function, the Lagrange multipliers are updated using the gradient algorithm.

[0215] In some embodiments, the security policy correction module 730 is configured to calculate distance information between the security policy and the baseline policy, including:

[0216] Based on the state of the training samples at each time step, and using a security strategy, estimate the actions of the training samples at each time step.

[0217] Based on the state of the training samples at each time step, the action of the training samples at each time step is estimated using the baseline strategy;

[0218] Calculate the expected distance information between the actions of the training samples estimated by the security policy at each time step and the actions of the training samples estimated by the baseline policy at each time step.

[0219] In some embodiments, the security policy correction module 730 is configured to calculate the expected cumulative security metric value, including:

[0220] Based on the state of the training samples at each time step, and using a security strategy, estimate the actions of the training samples at each time step.

[0221] Based on the state and action of the training samples at each moment, the expected value of the cumulative safety index is calculated using the state and action value function of the cumulative safety index value.

[0222] In some embodiments, the security policy correction module 730, configured to learn a security policy that satisfies the security constraints and is closest to the baseline policy, further includes:

[0223] Set up one or more security value networks for evaluating the security policy, and a target network for the security policy and a target network for each security value network;

[0224] Based on the current security index value and the state at the next moment of the training samples, the first security policy evaluation value is calculated using the target network of the security policy and the target network of each security value network.

[0225] Based on the current state and actions of the training samples, the second security policy evaluation value is calculated using each security value network.

[0226] The parameters of the security value network are updated based on the distance information between the first security policy evaluation value and the second security policy evaluation value.

[0227] In some embodiments, the security policy correction module 730 is configured to calculate the first security policy evaluation value including:

[0228] Based on the state of the training samples at the next moment, the target network of the security policy is used to estimate the action of the training samples at the next moment.

[0229] Based on the state and actions of the training samples at the next moment, the target security policy evaluation value is calculated using the target network of each security value network.

[0230] The first security policy evaluation value is calculated based on the target security policy evaluation value and the security index value of the training samples at the current moment.

[0231] In some embodiments, the security policy correction module 730, configured to learn a security policy that satisfies the security constraints and is closest to the baseline policy, further includes:

[0232] Update the parameters of the target network of the security policy in the current iteration round based on the parameters of the security policy in the previous iteration round; or,

[0233] Based on the parameters of the security value network in the current iteration and the parameters of the target network in the previous iteration, update the parameters of the target network in the current iteration.

[0234] In some embodiments, the baseline policy learning module 720 is configured to obtain a baseline policy based on the baseline policy learning task, including: loading a fixed baseline policy, wherein the fixed baseline policy is a baseline policy obtained in advance based on learning or a control policy not based on learning; or, obtaining a baseline policy based on learning.

[0235] In some embodiments, the baseline policy learning module 720, based on learning, obtains a baseline policy including:

[0236] Based on the state of the training samples at each time step, the action of the training samples at each time step is estimated using the baseline strategy;

[0237] Based on the state and action of the training samples at each time step, the maximum expected value of the cumulative reward is calculated using the state-action value function of the cumulative reward.

[0238] To maximize the expected cumulative reward, the parameters of the baseline policy are updated using a gradient algorithm.

[0239] In some embodiments, the baseline policy learning module 720, which obtains a baseline policy based on learning, further includes:

[0240] Set up one or more reward value networks for evaluating the baseline policy, and a target network for the baseline policy and a target network for each reward value network;

[0241] Based on the current reward and the state at the next moment of the training samples, the first baseline policy evaluation value is calculated using the target network of the baseline policy and the target network of each reward value network.

[0242] Based on the current state and actions of the training samples, the second baseline policy evaluation value is calculated using each reward value network;

[0243] The parameters of the reward value network are updated based on the distance information between the first baseline policy evaluation value and the second baseline policy evaluation value.

[0244] In some embodiments, the baseline policy learning module 720 calculates the first baseline policy evaluation value by including:

[0245] Based on the state of the training sample at the next moment, the target network of the baseline policy is used to estimate the action of the training sample at the next moment.

[0246] Based on the state and action of the training samples at the next moment, the target baseline policy evaluation value is calculated using the target network of each reward value network.

[0247] The first baseline policy evaluation value is calculated based on the target baseline policy evaluation value and the reward of the training sample at the current moment.

[0248] In some embodiments, the baseline policy learning module 720, which obtains a baseline policy based on learning, further includes:

[0249] Update the parameters of the target network of the baseline policy in the current iteration round based on the parameters of the baseline policy in the current iteration round and the parameters of the target network of the baseline policy in the previous iteration round; or,

[0250] Update the parameters of the target network of the reward value network in the current iteration based on the parameters of the reward value network in the current iteration and the parameters of the target network of the reward value network in the previous iteration.

[0251] In some embodiments, the security reinforcement learning device 700 further includes: a training sample generation module 750, comprising:

[0252] Randomly select a security policy or baseline policy to generate the corresponding action for the current state and receive the reward;

[0253] Calculate safety index values ​​based on the cost function of actions violating safety requirements;

[0254] The current state, action, reward, safety indicator value, and the state at the next moment are used as a training sample.

[0255] In some embodiments, the security reinforcement learning task is an agent security control learning task, and in the training samples of the agent security control learning task: the state is the state of the environment in which the agent is located, the action is the action of the agent, and the reward is the reward of environmental feedback.

[0256] In some embodiments, the cumulative security index value is a cumulative discount security index value calculated based on the security index value and its discount factor, or the cumulative reward is a cumulative discount reward calculated based on the reward and its discount factor.

[0257] In some embodiments, the security reinforcement learning device 700 further includes: an initialization module 760 configured to:

[0258] When both the security policy and each security value network are neural networks, and both the target network of the security policy and the target network of each security value network are neural networks, the security policy and each security value network are initialized. The target network of the security policy is initialized by copying the network parameters of the security policy, and the target network of each security value network is initialized by copying the network parameters of each security value network.

[0259] When the baseline policy and each reward value network are neural networks, and the target network of the baseline policy and each reward value network are neural networks, the baseline policy and each reward value network are initialized. The target network of the baseline policy is initialized by copying the network parameters of the baseline policy, and the target network of each reward value network is initialized by copying the network parameters of each reward value network.

[0260] Figure 8 A schematic diagram of the structure of a security reinforcement learning apparatus according to some embodiments of this disclosure is shown. For example... Figure 8 As shown, the security reinforcement learning apparatus 800 of this embodiment includes a memory 810 and a processor 820 coupled to the memory 810. The processor 820 is configured to execute the security reinforcement learning method of any of the foregoing embodiments based on instructions stored in the memory 810.

[0261] The memory 810 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, and other programs.

[0262] The processor 820 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, or transistors, or other discrete hardware components.

[0263] Device 800 may also include input / output interfaces 830, network interfaces 840, and storage interfaces 850. These interfaces 830, 840, and 850, as well as the memory 810 and processor 820, can be connected, for example, via a bus 860. Specifically, input / output interface 830 provides a connection interface for input / output devices such as displays, mice, keyboards, and touchscreens. Network interface 840 provides a connection interface for various networked devices. Storage interface 850 provides a connection interface for external storage devices such as SD cards and USB flash drives. Bus 860 can use any bus architecture from various bus structures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0264] Figure 9 The diagram illustrates the structure of an intelligent agent according to some embodiments of this disclosure. For example... Figure 9 As shown, the intelligent agent 900 in this embodiment includes: a security reinforcement learning device 700, 800. Furthermore, the intelligent agent 900 may also include, for example, a support element for supporting the intelligent agent body, an execution element for performing actions, a control element for controlling the execution of actions based on a policy, etc., but is not limited to the examples given.

[0265] The intelligent agent 900 includes, but is not limited to, robots.

[0266] Some embodiments of this disclosure propose a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of a secure reinforcement learning method.

[0267] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more non-transitory computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0268] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0269] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0270] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0271] The above description is only a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A secure reinforcement learning method, characterized in that, include: The agent safety control learning task is decoupled into a baseline policy learning task and a safety policy correction task, with the agent being a robot. Based on the baseline policy learning task, a baseline policy that maximizes cumulative reward is obtained. The baseline policy outputs a corresponding first action based on the state of the input environment. The reward is the negative or positive feedback information obtained by the agent from the environment after performing an action. The state of the environment includes the various states of the robot's behavior trajectory. Based on the security policy correction task, a security policy that satisfies security constraints and is closest to the baseline policy is learned by imitating the baseline policy through behavioral cloning. This includes: constructing a dual objective function by associating the original objective function and the security constraint function using Lagrange multipliers, wherein the original objective function is the function that minimizes the distance information between the security policy and the baseline policy, the distance information including the distance between the actions estimated by the security policy and the actions estimated by the baseline policy, and the security constraint function is the function that the expected value of the cumulative security index is greater than or equal to a security threshold; and updating the parameters of the security policy for the dual objective function, wherein the security constraints include the constraint of avoiding obstacles. The safety policy is output, and the safety policy outputs a corresponding second action based on the state of the input environment, so that the robot can execute the second action output by the safety policy.

2. The method according to claim 1, characterized in that, The parameters for updating security policies include: For the dual objective function, the parameters of the security policy are updated using a gradient algorithm.

3. The method according to claim 2, characterized in that, Learning a security policy that satisfies the security constraints and is closest to the baseline policy also includes: For the dual objective function, the Lagrange multipliers are updated using the gradient algorithm.

4. The method according to claim 2, characterized in that, The methods for calculating the distance information between the security policy and the baseline policy include: Based on the state of the training samples at each time step, and using a security strategy, estimate the actions of the training samples at each time step. Based on the state of the training samples at each time step, the action of the training samples at each time step is estimated using the baseline strategy; Calculate the expected distance information between the actions of the training samples estimated by the security policy at each time step and the actions of the training samples estimated by the baseline policy at each time step.

5. The method according to claim 2, characterized in that, The methods for calculating the expected cumulative safety index value include: Based on the state of the training samples at each time step, and using a security strategy, estimate the actions of the training samples at each time step. Based on the state and action of the training samples at each moment, the expected value of the cumulative safety index is calculated using the state and action value function of the cumulative safety index value.

6. The method according to claim 2, characterized in that, Learning a security policy that satisfies the security constraints and is closest to the baseline policy also includes: Set up one or more security value networks for evaluating the security policy, and a target network for the security policy and a target network for each security value network; Based on the current security index value and the state at the next moment of the training samples, the first security policy evaluation value is calculated using the target network of the security policy and the target network of each security value network. Based on the current state and actions of the training samples, the second security policy evaluation value is calculated using each security value network. The parameters of the security value network are updated based on the distance information between the first security policy evaluation value and the second security policy evaluation value.

7. The method according to claim 6, characterized in that, The calculation of the first security policy evaluation value includes: Based on the state of the training samples at the next moment, the target network of the security policy is used to estimate the action of the training samples at the next moment. Based on the state and actions of the training samples at the next moment, the target security policy evaluation value is calculated using the target network of each security value network. The first security policy evaluation value is calculated based on the target security policy evaluation value and the security index value of the training samples at the current moment.

8. The method according to claim 6, characterized in that, Learning a security policy that satisfies the security constraints and is closest to the baseline policy also includes: Update the parameters of the target network of the security policy in the current iteration round based on the parameters of the security policy in the previous iteration round; or, Based on the parameters of the security value network in the current iteration and the parameters of the target network in the previous iteration, update the parameters of the target network in the current iteration.

9. The method according to claim 1, characterized in that, Based on the baseline policy learning task, the baseline policy is obtained as follows: Load a fixed baseline policy, which is either a pre-learned baseline policy or a non-learning-based control policy; or, Baseline strategies are obtained through learning.

10. The method according to claim 9, characterized in that, Baseline-based learning strategies include: Based on the state of the training samples at each time step, the action of the training samples at each time step is estimated using the baseline strategy; Based on the state and action of the training samples at each time step, the maximum expected value of the cumulative reward is calculated using the state-action value function of the cumulative reward. To maximize the expected cumulative reward, the parameters of the baseline policy are updated using a gradient algorithm.

11. The method according to claim 10, characterized in that, Learning-based baseline acquisition strategies also include: Set up one or more reward value networks for evaluating the baseline policy, and a target network for the baseline policy and a target network for each reward value network; Based on the current reward and the state at the next moment of the training samples, the first baseline policy evaluation value is calculated using the target network of the baseline policy and the target network of each reward value network. Based on the current state and actions of the training samples, the second baseline policy evaluation value is calculated using each reward value network; The parameters of the reward value network are updated based on the distance information between the first baseline policy evaluation value and the second baseline policy evaluation value.

12. The method according to claim 11, characterized in that, The calculation of the first baseline strategy evaluation value includes: Based on the state of the training sample at the next moment, the target network of the baseline policy is used to estimate the action of the training sample at the next moment. Based on the state and action of the training samples at the next moment, the target baseline policy evaluation value is calculated using the target network of each reward value network. The first baseline policy evaluation value is calculated based on the target baseline policy evaluation value and the reward of the training sample at the current moment.

13. The method according to claim 11, characterized in that, Learning-based baseline acquisition strategies also include: Update the parameters of the target network of the baseline policy in the current iteration round based on the parameters of the baseline policy in the current iteration round and the parameters of the target network of the baseline policy in the previous iteration round; or, Update the parameters of the target network of the reward value network in the current iteration based on the parameters of the reward value network in the current iteration and the parameters of the target network of the reward value network in the previous iteration.

14. The method according to claim 1, characterized in that, The method for generating the training samples required by the security reinforcement learning method includes: Randomly select a security policy or baseline policy to generate the corresponding action for the current state and receive the reward; Calculate safety index values ​​based on the cost function of actions violating safety requirements; The current state, action, reward, safety indicator value, and the state at the next moment are used as a training sample.

15. The method according to claim 14, characterized in that, In the training samples of the agent safety control learning task: the state is the state of the environment in which the agent is located, the action is the action of the agent, and the reward is the reward of environmental feedback.

16. The method according to claim 2 or 10, characterized in that, The cumulative security index value is a cumulative discount security index value calculated based on the security index value and its discount factor, or... The cumulative reward is a cumulative discount reward calculated based on the reward and its discount factor.

17. The method according to claim 6 or 11, characterized in that, The security policy and each security value network are both neural networks. The target network of the security policy and the target network of each security value network are both neural networks. The security policy and each security value network are initialized by copying the network parameters of the security policy to initialize the target network of the security policy, and by copying the network parameters of each security value network to initialize the target network of each security value network. or, The baseline policy and each reward value network are both neural networks. The target network of the baseline policy and each reward value network are both neural networks. The baseline policy and each reward value network are initialized by copying the network parameters of the baseline policy to initialize the target network of the baseline policy, and by copying the network parameters of each reward value network to initialize the target network of each reward value network.

18. The method according to claim 1, characterized in that, Also includes: The agent performs actions based on the current state of the environment using the security policy, causing the environment to respond and transition to the next state, obtaining rewards from the environment feedback, and performing new actions based on the next state of the environment using the security policy, maximizing the accumulated rewards during each action selection process.

19. A security reinforcement learning device, characterized in that, include: The decoupling module is configured to decouple the agent's safety control learning task into a baseline policy learning task and a safety policy correction task, with the agent being a robot. The baseline policy learning module is configured to obtain a baseline policy that maximizes cumulative reward based on the baseline policy learning task. The baseline policy outputs a corresponding first action based on the state of the input environment. The reward is negative or positive feedback information obtained by the agent from the environment after performing an action. The state of the environment includes various states of the robot's behavior trajectory. A safety policy correction module is configured to learn a safety policy that satisfies safety constraints and is closest to the baseline policy by mimicking the baseline policy through behavioral cloning based on the safety policy correction task. This includes: constructing a dual objective function by associating the original objective function and the safety constraint function using Lagrange multipliers; wherein the original objective function is the function that minimizes the distance information between the safety policy and the baseline policy, the distance information including the distance between the actions estimated by the safety policy and the actions estimated by the baseline policy; and the safety constraint function is a function whose expected cumulative safety index value is greater than or equal to a safety threshold; and updating the parameters of the safety policy for the dual objective function, wherein the safety constraints include obstacle avoidance constraints. The output module is configured to output the safety policy, which outputs a corresponding second action based on the state of the input environment, so that the robot can execute the second action output by the safety policy.

20. A security reinforcement learning device, comprising: Memory; And a processor coupled to the memory, characterized in that the processor is configured to execute the secure reinforcement learning method of any one of claims 1-18 based on instructions stored in the memory.

21. An intelligent agent system, characterized in that, Includes the security reinforcement learning device as described in claim 19 or 20.

22. The intelligent agent system according to claim 21, characterized in that, The intelligent agent system includes robots.

23. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the secure reinforcement learning method according to any one of claims 1-18.

Images