An offline reinforcement learning method based on proximal adversarial constraint strategy search

Abstract

The present invention discloses an offline reinforcement learning method based on proximal adversarial constraint strategy search. First, a high-level behavior strategy is trained to provide an out-of-sample high-level behavior action that obeys the distribution of a static offline data set but is superior to the action in the static offline data set, as a reference action for the strategy constraint, thereby improving the efficiency of the strategy constraint; then, a Taylor expansion approximation is performed on the constraint strategy search target at the action in the static offline data set and the high-level behavior action, and an adversarial compensation point-neighborhood sample efficient search of the learning strategy at the proximal end of two sufficiently small actions is constructed, allowing the learning strategy to actively select out-of-distribution actions that deviate from the static offline data set to a limited extent. By providing reliable reference actions and low-variance gradient estimates, the present invention can effectively select out-of-distribution actions, break through the decision performance limitation of the point-point strategy constraint that prohibits the selection of out-of-distribution actions, and narrow the performance difference between the learning strategy and the optimal strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of offline reinforcement learning, and in particular to an offline reinforcement learning method based on proximal adversarial constraint strategy search. Background Art

[0002] Reinforcement learning has achieved remarkable results in simulation environments such as video games and nonlinear systems by optimizing decisions through continuous trial-and-error feedback between agents and the environment. However, in real-world scenarios such as autonomous driving, robotics, and resource scheduling, it is impractical to use suboptimal reinforcement learning strategies for on-the-spot trial and error due to cost and safety considerations. To overcome the limitations of applying reinforcement learning in real-world scenarios, offline reinforcement learning has emerged, which aims to transform pre-collected static offline datasets into powerful task-solving strategies. As a promising sample-efficient learning method that does not require any actual interaction with the environment, offline reinforcement learning is particularly suitable for real-world scenarios that are costly or risky to explore from scratch, and has received widespread attention in recent years.

[0003] Although offline reinforcement learning has advantages, directly using online reinforcement learning to learn policies from static offline datasets faces the challenge of distribution shift between the learning policy and the behavior policy (data distribution) due to insufficient coverage of the static offline dataset. The most obvious manifestation of distribution shift in offline reinforcement learning is the Q-value extrapolation error: the Q-value of actions outside the distribution of the unseen static offline dataset is easily overestimated. Moreover, this error will continue to accumulate during the bootstrapping process, causing serious distortion of the Q-value estimation, resulting in poor performance of policies that rely on Q-value learning.

[0004] To address the above challenges, policy constraints are usually enforced to learn a conservative exploration strategy, that is, to avoid excessive generation of out-of-distribution actions while optimizing policy performance. The most typical type of policy constraint method is parameterized behavioral strategy, the core of which is to constrain the learning strategy through the support set provided by the accurate generative model of the behavioral strategy or the density distribution of the behavioral strategy. The parameterized behavioral strategy method can ensure that in most cases, the actions output by the learning strategy can appear in the static offline data set with a high probability. However, when the estimation accuracy of the parameterized behavioral strategy is low or the distribution of the behavioral strategy is close to the uniform distribution, strictly restricting the learning strategy to be close to the support set or distribution of the parameterized behavioral strategy will form a negative constraint, resulting in the failure to correctly avoid out-of-distribution actions, thereby failing to guarantee the performance of the learning strategy. Moreover, behavioral strategies are mostly derived from empirical data provided by humans, manually designed controllers, or multiple baseline strategies, accompanied by situations that are difficult to accurately estimate.

[0005] To avoid parameterized estimation of behavioral policies and still be able to enforce policy constraints, another alternative policy constraint method is dataset regularization, the core of which is to directly use the state-action pairs in the static offline dataset to constrain the search of the learning policy. That is, the behavior cloning equidistance function is added to the policy search objective. At each step of the policy search process, the same or different degree of point-to-point dataset distance constraints are imposed on the actions output by the learning policy and the actions in the corresponding state in the static offline dataset. In the point-to-point dataset distance constraint, whether the regularization factor plays a role of global consistency or local adaptive constraint to encourage the learning policy to sample actions in similar datasets, this will limit the potential of policy search, especially when the action in a given state in the static offline dataset is not the optimal action. This is attributed to the fact that the strength of the point-to-point dataset distance constraint is usually too conservative, prohibiting the selection of out-of-distribution actions, which inevitably damages the generalization performance out of the static offline dataset distribution. Summary of the invention

[0006] Purpose of the invention: In view of the problems and shortcomings existing in the prior art, the present invention provides an offline reinforcement learning method based on proximal adversarial constraint strategy search, which allows the learning strategy to actively select better out-of-distribution actions that are sufficiently conservative but have never appeared in static offline datasets, thereby promoting the application of offline reinforcement learning in real robotics industrial scenarios.

[0007] Technical solution: To achieve the purpose of the present invention, the technical solution adopted by the present invention is an offline reinforcement learning method based on proximal adversarial constraint strategy search, comprising the following steps:

[0008] Step 1: construct a high-level behavior strategy network, Q function, and proximal adversarial constraint strategy network;

[0009] Step 2: Use behavioral strategy π β A static offline dataset that collects experience transformation data (s,a,r,s′) by interacting with the robot environment

[0010] An experience conversion data (s, a, r, s′) represents the current state s of the robot, through the strategy π β Execute action a, obtain immediate reward r, and transition to the next state s′;

[0011] Step 3: From the static offline dataset Sampling small batches of empirical transformation data;

[0012] Step 4: Take the sampled static offline data set action a=π β (s) Constraint strategy search constructed as a reference action to update and obtain a better behavior strategy π βThe high-level behavior strategy μ provides out-of-sample high-level behavior actions μ(s) for proximal adversarial constraint strategy search;

[0013] At the same time, update the Q function to obtain a stable estimate

[0014] Step 5: Use the function High-level behavior action μ(s) and action a in static offline dataset, for proximal adversarial constraint strategy π ρ Make updates;

[0015] Step 6: Repeat steps 3-5 to update the network parameters of the advanced behavior strategy network, Q function, and proximal adversarial constraint strategy network to learn the optimal proximal adversarial constraint strategy.

[0016] Furthermore, step 1 specifically includes: using the neural network parameters θ1, θ2 and ρ parameterize a high-level behavior policy network Two independent Q-function networks and a proximal adversarial constraint policy network π ρ .

[0017] Furthermore, in step 4, the update obtains a behavior strategy π that is better than β The high-level behavior strategy μ has a loss function of:

[0018]

[0019] Among them, Q θ express or a represents the balance coefficient of the high-level behavior strategy network, M represents The size, Used to normalize the Q function;

[0020] The maximum loss function is multiplied by negative one to convert it to the minimum loss function, and the high-level behavior strategy network is updated at each proximal adversarial constraint strategy search step through the stochastic gradient descent method. The parameters of , the gradient update amplitude is:

[0021]

[0022] Furthermore, in step 4, the Q function is updated to obtain a stable estimate Specifically:

[0023] Use the pruned double Q-learning method to The loss function is designed as:

[0024]

[0025] in, Indicates small batches, is the action Gaussian exploration noise with variance σ, clip(·) is the clipping function with clipping boundary c;

[0026] Minimize the Q network loss function by stochastic gradient descent and update the optimization at each strategy evaluation step The parameters of , the gradient update amplitude is:

[0027]

[0028] Furthermore, in step 5, the function And the high-level behavior action μ(s) is used as the reference action to construct the strategy constraint item and the strategy update rule, which is expressed as follows:

[0029]

[0030] Among them, β represents the balance coefficient of the proximal adversarial constraint strategy network;

[0031] And use Taylor expansion to approximate the optimization strategy update rule and construct a proximal constraint strategy search at the high-level action μ(s):

[0032]

[0033] Furthermore, in order to avoid deviating from the limited support range of the behavior strategy, the Taylor expansion is used again to approximate the optimization strategy update rule and construct the proximal constraint strategy search at action a in the static offline dataset:

[0034]

[0035] Furthermore, the two proximal constraint strategy searches are combined in an adversarial weight compensation manner to construct a k+1 The proximal adversarial constraint strategy search rule for the two proximal neighborhoods (s) and a, and the loss function of the proximal adversarial constraint strategy network is finally designed as:

[0036]

[0037] in, λ represents the adversarial weight of the proximal adversarial constraint, and They are used to normalize the derivatives of the Q function.

[0038] Furthermore, the loss function is multiplied by negative one, and the maximization loss function is converted into the minimization loss function after multiplying by negative one. The network π is updated at each proximal adversarial constraint strategy search step by stochastic gradient descent. ρThe parameters of , the gradient update amplitude is:

[0039]

[0040] Furthermore, in step 2, the current state s of the robot includes the position, speed, angles and angular velocity of each joint; and the action a represents the force or torque applied to each joint of the robot.

[0041] Furthermore, the target networks corresponding to the advanced behavior strategy network, the two Q functions and the proximal adversarial constraint strategy network are the advanced behavior strategy target network, the two Q function target networks and the proximal adversarial constraint strategy target network respectively; in step 6, the network parameters of the advanced behavior strategy network, the Q function and the proximal adversarial constraint strategy network are updated, specifically by using the Polyak average method to update the calculated Assign values ​​to Where, υ represents the target network update rate, They are the network parameters of the high-level behavior strategy target network, the two Q-function target networks, and the proximal adversarial constraint strategy target network.

[0042] The proximal adversarial constraint strategy search mechanism constructed in steps 1 to 6 is a safety strategy improvement that can narrow the performance gap between the robot's final learning strategy and the optimal strategy. In addition, the proximal adversarial constraint strategy search mechanism with out-of-sample reference samples and out-of-distribution neighborhood search is a sample-efficient strategy update rule with high decision quality and efficiency and low computational consumption.

[0043] Beneficial effects of the technical solution of the present invention: In view of the fact that the existing technical methods prohibit the selection of actions outside the static offline data set distribution when alleviating the overestimation extrapolation error caused by the action distribution offset, which limits the decision-making performance of the learning strategy, the present invention proposes an offline reinforcement learning method based on proximal adversarial constraint strategy search. (1) Different from directly using the high-variance gradient estimation of sparse samples in the static offline data set, this method uses Taylor expansion approximation for the Q function and the constraint signal, which increases the sensitivity of the strategy search process to the action. Its sufficient uniform samples in the proximal neighborhood can provide low-variance gradient estimation and stable and smooth strategy search. (2) The strategy search of this method no longer relies solely on blind conservative search in the proximal neighborhood of the action in the static offline data set, but guides the strategy search to perform efficient and relaxed search in the neighborhood of the high-level behavior action by introducing the constraint method of adversarial weight compensation, thereby allowing and regulating the strategy search to select more optimal out-of-distribution actions and improve the efficiency of policy constraints. In particular, when the suboptimal static offline data set cannot provide the optimal action, this method can rely on the high-level behavior strategy to dynamically generate more reliable out-of-sample actions as new reference actions for policy constraints to avoid negative policy constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a structural diagram of the offline reinforcement learning method based on proximal adversarial constraint strategy search. DETAILED DESCRIPTION

[0045] The present invention will be further explained below in conjunction with the accompanying drawings.

[0046] like Figure 1 As shown, an offline reinforcement learning method based on proximal adversarial constraint strategy search of the present invention first provides an out-of-sample high-level behavior action that obeys the distribution of a static offline data set but is superior to the action within the static offline data set through a trained high-level behavior strategy as a reference action for strategy constraints, thereby improving the efficiency of strategy constraints; then, Taylor expansion approximation is performed on the constraint strategy search target at the actions within the static offline data set and the high-level behavior actions, and an adversarial compensation point-neighborhood sample efficient search of the learning strategy at the proximal ends of two sufficiently small actions is constructed, allowing the learning strategy to actively select out-of-distribution actions that deviate from the static offline data set by a limited amount.

[0047] The present invention provides an offline reinforcement learning method based on proximal adversarial constraint strategy search, comprising the following specific steps:

[0048] S1: Establish a proximal adversarial constraint strategy search mechanism suitable for offline reinforcement learning, which mainly includes high-level behavior strategies, Q functions and proximal adversarial constraint strategies.

[0049] S101. Using Neural Network Parameters θ1, θ2 and ρ parameterize a high-level behavior policy network Two independent Q-function networks and a proximal adversarial constraint policy network π ρ ,The network is initialized in the following ways: random initialization;

[0050] S102. Use neural network parameters and Parameterize a high-level behavior policy target network accordingly Two Q-function target networks and a proximal adversarial constraint policy target network To stabilize the network training, the target network is initialized randomly: the initial parameters of the original network are Directly assign values ​​to the parameters of the corresponding target network

[0051] S2: Utilizing behavioral strategy π β Interact with the simulated / actual robot environment to collect a static offline dataset containing N experience transformation data (s, a, r, s′)

[0052] An experience conversion data (s, a, r, s′) in the static offline data set means: the agent observes the current state s, immediately executes action a through the behavior strategy, obtains the immediate reward r and converts the state through the state transition probability distribution in the environment Transition to the next state s′;

[0053] The behavior strategy is the strategy for the agent to interact with the environment, which can be a mixture of multiple strategies or a single strategy.

[0054] In the scenario where the agent is a robot, the static offline dataset consists of a quaternion (s, a, r, s′), where: the state s includes all key physical information such as the robot’s position, speed, angles and angular velocity of each joint at the current moment; the action a is expressed as a control signal such as force or torque applied to each joint of the robot; the reward r is related to the robot’s movement, navigation, dexterous operation speed, energy consumption and smoothness of the action, and is used to encourage the robot to learn quickly while avoiding excessive energy consumption and violent unsafe actions; the state s′ describes the robot’s new position, speed, angles and angular velocity of each joint at the next moment;

[0055] S3: From static offline datasets We sample mini-batches of experience transformation data as input to train the proximal adversarial constraint policy search mechanism.

[0056] S4, using the static offline data set action a = π β (s) is used as a reference action to construct a constraint policy search to update the obtained behavior policy π β The high-level behavior strategy μ is used to provide better out-of-sample high-level behavior actions μ(s) that approximately follow the distribution of the static offline dataset for the proximal adversarial constraint strategy search, while ensuring stable policy evaluation to update the Q function to obtain a stable estimate

[0057] S401, different from the behavior strategy π that is directly recovered from the static offline dataset using behavior cloning β The proximal adversarial constraint strategy search mechanism uses a constraint strategy search mechanism that includes reinforcement learning and behavior cloning signals to continuously learn strategies that outperform behavior strategies from static offline datasets. As an advanced behavior strategy, its loss function is designed as:

[0058]

[0059] in or Can be used as Q θ , a represents the balance coefficient of the high-level behavior strategy network, M represents The size, Used to normalize the Q function;

[0060] S402: For ease of practice, the maximum loss function in step S403 is multiplied by negative one to convert it into a minimum loss function, and the high-level behavior strategy network is updated at each proximal adversarial constraint strategy search step by stochastic gradient descent. The parameters of , the gradient update amplitude is:

[0061]

[0062] S403, in order to alleviate the overestimation problem associated with training the Q function, the network is trimmed using the double Q-learning method. The loss function is designed as:

[0063]

[0064] in Indicates small batches, is the action Gaussian exploration noise with variance σ, and clip(·) is the clipping function with clipping margin c.

[0065] S404, minimize the Q network loss function in step S404 by stochastic gradient descent, and update the optimization at each strategy evaluation step The parameters of , the gradient update amplitude is:

[0066]

[0067] S5. Not participating in advanced behavior strategies μ and Q functions The proximal adversarial constraint strategy π is coupled to update, and its update process depends on the function and the policy constraint term constructed with the high-level action μ(s) as the reference action, using Taylor expansion to approximate the proximal constraint policy search at the high-level action μ(s), and combining it with a=π β The proximal constraint strategy search constructed at (s) forms two proximal neighborhood adversarial action constraint strategy searches, thereby establishing a proximal adversarial constraint strategy search mechanism to guide strategy π to perform sample-efficient search: active search of uniformly distributed outer neighborhood actions.

[0068] S501, introduce a non-participation high-level behavior strategy network and Q function Q θ Updated learning strategy π ρ ;

[0069] S502. Using Advanced Behavioral Strategy Network Provided out-of-sample actions As a proximal adversarial constraint policy network π ρ The new reference action constructs the following constraint efficiency more efficient strategy update rule for the proximal adversarial constraint strategy network π ρ To update:

[0070]

[0071] Among them, β represents the balance coefficient of the proximal adversarial constraint strategy network;

[0072] S503, using Taylor expansion approximation to clone the maximized Q function and minimized behavior in step S502, respectively The following first-order linear approximation and second-order quadratic approximation are performed:

[0073]

[0074] Through the first-order Taylor expansion, Can provide Q θ (s,π(s)) acts out of sample Linear approximation in a small enough neighborhood, The zero-order and first-order difference results on the right side that are independent of the variable π(s) are recorded as a constant C and compared with the behavior constraint term The second-order difference result on the right can be ignored; through the second-order Taylor expansion, Can provide exist The quadratic approximation in a small enough neighborhood, The zero-order and first-order difference calculation results on the right are 0, that is, The second-order quadratic approximation error is 0; through the linear approximation of the Q function and the quadratic approximation of the behavior cloning signal, the strategy search can be realized in the query The Q value of the action within a sufficiently small neighborhood is not penalized, and only the Q value of the action outside the neighborhood is penalized; thus, the update rule of step S502 is reconstructed as Perform a constrained strategy search within a sufficiently small proximal neighborhood:

[0075]

[0076] This proximal constraint policy search rule allows the learned policy to act out of sample Actively conduct a small-scale uniformly distributed external action search in the proximal neighborhood of Constraint relaxation is achieved within a sufficiently small proximal neighborhood, and sufficient conservatism is still achieved outside the proximal neighborhood. However, this is easily overlooked: relying entirely on out-of-sample reference actions may cause the learning strategy to deviate from the limited deviation support range of the behavioral strategy.

[0077] S504, again using Taylor expansion approximation to construct a proximal constraint strategy search at action a in the static offline data set similar to step S503 for the update rule of step S502:

[0078]

[0079] in,

[0080]

[0081] The proximal constrained policy search rule allows the learning policy to actively search for a small range of uniformly distributed out-of-band actions in the proximal neighborhood of action a in a static offline dataset.

[0082] S505, combining the update rules of step 503 and step 504 in an adversarial weight compensation manner, constructing k+1 The proximal adversarial constraint strategy search rules for the two proximal neighborhoods of (s) and a are designed, and the loss function of the proximal adversarial constraint strategy network is designed as:

[0083]

[0084] in, λ represents the adversarial weight of the proximal adversarial constraint, and They are used to normalize the derivatives of the Q function.

[0085] S506. Also for the convenience of practice, the maximization loss function in step S505 is multiplied by negative one to convert it into a minimization loss function, and the network π is updated at each proximal adversarial constraint strategy search step by the stochastic gradient descent method. ρ The parameters of , the gradient update amplitude is:

[0086]

[0087] S6. Repeat steps S3 to S5 until the static offline dataset is transformed into a powerful task-solving strategy through the proximal adversarial constraint strategy search mechanism, and then deployed and applied in decision-making scenarios such as robot movement, navigation, and dexterous operation.

[0088] S601, repeating steps S402, S404 and S505 for at least 1 million times, wherein the target network parameters corresponding to the high-level behavior strategy network, the two learning Q function networks and the proximal adversarial constraint strategy network after each iteration are updated using the Polyak average method, and the calculated Assign values ​​to Among them, υ represents the target network update rate;

[0089] S602, executing step S601 until the optimal network model parameters are obtained, that is, the optimal proximal adversarial constraint sequence task solution strategy is learned.

[0090] The present invention relates to the technical field of offline reinforcement learning, and in particular to an offline reinforcement learning method based on proximal adversarial constraint strategy search. The proximal adversarial constraint strategy search mechanism constructed by the method is a safety strategy improvement, which can narrow the performance difference between the final learning strategy and the optimal strategy. In addition, the proximal adversarial constraint strategy search mechanism with out-of-sample reference samples and out-of-distribution neighborhood search is a sample-efficient strategy update rule, which has high decision-making quality and efficiency, low computational consumption, and is suitable for expansion to realistic industrial robot application scenarios with limited computing power. Specifically, firstly, a high-level behavioral action out of sample that obeys the distribution of a static offline data set but is superior to an action in a static offline data set is provided through a trained high-level behavioral strategy as a reference action for strategy constraints, thereby improving the efficiency of strategy constraints; then, a Taylor expansion approximation is performed on the constraint strategy search target at the action in the static offline data set and the high-level behavioral action, and an adversarial compensation point-neighborhood sample efficient search of the learning strategy at the proximal end of two sufficiently small actions is constructed, allowing the learning strategy to actively select an out-of-distribution action that deviates from the static offline data set by a limited amount.

[0091] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An offline reinforcement learning method based on proximal adversarial constraint strategy search, characterized in that: The steps include: Step 1: construct a high-level behavior strategy network, Q function, and proximal adversarial constraint strategy network; Step 2: Use behavioral strategy π β A static offline dataset that collects experience transformation data (s,a,r,s′) by interacting with the robot environment An experience conversion data (s, a, r, s′) represents the current state s of the robot, through the strategy π β Execute action a, obtain immediate reward r, and transition to the next state s′; The current state s of the robot includes position, speed, angles and angular speeds of each joint; action a represents the force or torque applied to each joint of the robot; Step 3: From the static offline dataset Sampling small batches of empirical transformation data; Step 4: Take the sampled static offline data set action a=π β (s) Constraint strategy search constructed as a reference action to update and obtain a better behavior strategy π β The high-level behavior strategy μ provides out-of-sample high-level behavior actions μ(s) for proximal adversarial constraint strategy search; At the same time, update the Q function to obtain a stable estimate Step 5: Use the function High-level behavior action μ(s) and action a in static offline dataset, for proximal adversarial constraint strategy π ρ Make updates; Step 6: Repeat steps 3-5 to update the network parameters of the advanced behavior strategy network, Q function, and proximal adversarial constraint strategy network to learn the optimal proximal adversarial constraint strategy.

2. According to claim 1, an offline reinforcement learning method based on proximal adversarial constraint strategy search is characterized in that: Step 1 specifically includes: Using neural network parameters θ1, θ2 and ρ parameterize a high-level behavior policy network Two independent Q-function networks and a proximal adversarial constraint policy network π ρ .

3. The offline reinforcement learning method based on proximal adversarial constraint strategy search according to claim 2, characterized in that: Update acquisition is better than behavior policy π β The high-level behavior strategy μ has a loss function of: Among them, Q θ express or α represents the balance coefficient of the high-level behavior strategy network, M represents The size, Used to normalize the Q function; Multiply the loss function by negative one, convert the maximized loss function into the minimized loss function after multiplying by negative one, and update the high-level behavior strategy network at each proximal adversarial constraint strategy search step through the stochastic gradient descent method. The parameters of , the gradient update amplitude is:

4. According to claim 2, an offline reinforcement learning method based on proximal adversarial constraint strategy search is characterized in that: In step 4, update the Q function to obtain a stable estimate Specifically: The network is transformed using the pruned double Q-learning method. The loss function is designed as: in, Indicates small batches, is the action Gaussian exploration noise with variance σ, clip(·) is the clipping function with clipping boundary c; Minimize the Q network loss function by stochastic gradient descent and update the optimization at each strategy evaluation step The parameters of , the gradient update amplitude is:

5. The offline reinforcement learning method based on proximal adversarial constraint strategy search according to claim 2, characterized in that: By function And the high-level behavior action μ(s) is used as the reference action to construct the strategy constraint item and the strategy update rule, which is expressed as follows: Among them, β represents the balance coefficient of the proximal adversarial constraint strategy network; And use Taylor expansion to approximate the optimization strategy update rule and construct a proximal constraint strategy search at the high-level action μ(s):

6. The offline reinforcement learning method based on proximal adversarial constraint strategy search according to claim 5, characterized in that: Using Taylor expansion to approximate the optimization policy update rule, we construct a proximal constraint policy search at action a in a static offline dataset:

7. The offline reinforcement learning method based on proximal adversarial constraint strategy search according to claim 6, characterized in that: Combining the two proximal constraint strategy searches in an adversarial weight compensation manner, we construct k+1 The proximal adversarial constraint strategy search rule for the two proximal neighborhoods (s) and a, and the loss function of the proximal adversarial constraint strategy network is finally designed as: in, λ represents the adversarial weight of the proximal adversarial constraint, and They are used to normalize the derivatives of the Q function.

8. The offline reinforcement learning method based on proximal adversarial constraint strategy search according to claim 7, characterized in that: Multiply the loss function by negative one, convert the maximized loss function into the minimized loss function after multiplying by negative one, and update the network π at each proximal adversarial constraint strategy search step through the stochastic gradient descent method. ρ The parameters of , the gradient update amplitude is:

9. The offline reinforcement learning method based on proximal adversarial constraint strategy search according to claim 2, characterized in that: The high-level behavior strategy network, two Q functions and proximal adversarial constraint strategy network, the corresponding target networks are high-level behavior strategy target network, two Q function target networks and proximal adversarial constraint strategy target network; In step 6, the network parameters of the advanced behavior strategy network, Q function and proximal adversarial constraint strategy network are updated by using the Polyak average method. Assign values ​​to Where, υ represents the target network update rate, They are the network parameters of the high-level behavior strategy target network, the two Q-function target networks, and the proximal adversarial constraint strategy target network.

Images