Off-line reinforcement learning action exploration agent method based on expected reward regularization

Abstract

The invention discloses an off-line reinforcement learning action exploration agent method based on expected reward regularization, and belongs to the field of reinforcement learning. The problem that reliable track splicing and strategy generalization are difficult to realize in a complex task by the existing method is solved. The method comprises the following steps: constructing state loss and action loss based on sequence modeling, and carrying out iterative training of states and actions; designing an RTG loss function based on a weighted square error; a double-Q learning framework is adopted to maintain two independent Q functions, conservative constraints of conservative Q learning are applied in the Q function updating process, and action exploration optimization is achieved in combination with Boltzmann distribution; combining state loss, action loss, RTG regularization loss and Q value loss to form a joint optimization objective function; performing noise disturbance sampling on the plurality of RTG candidate values to generate diversified action prediction; and value evaluation is performed on the candidate actions based on a double-conservative Q function, and the action with the highest Q value is selected for execution. The method is mainly used in the intelligent agent exploration field.

Description

Technical Field

[0001] The present invention belongs to the field of reinforcement learning, and in particular relates to an offline reinforcement learning action exploration agent method based on expected reward regularization. Background Art

[0002] Reinforcement learning (RL) learns by interacting with the environment to obtain action feedback, avoiding the reliance of supervised learning on labeled data. However, its reliance on online exploration poses limitations in scenarios with limited data access. In recent years, offline reinforcement learning (RL), which directly learns policies from static datasets, has become a research hotspot. Traditional offline methods are primarily based on value function optimization (such as Q-learning) and temporal difference learning. However, these methods are limited by biases and stability issues in value function estimation, and perform poorly in complex tasks or scenarios with sparse rewards. With breakthroughs in sequence modeling technology, conditional sequence modeling methods based on Transformers have reframed offline reinforcement learning as a supervised learning problem: directly predicting the action sequence by inputting a sequence of triplets consisting of a state, an action, and an expected reward (Return-To-Go, RTG). These methods circumvent the instability of traditional value function methods through goal-conditional policy optimization, demonstrating advantages in some tasks.

[0003] However, methods based on conditional sequence modeling still face two major bottlenecks. First, there is the inconsistency between trajectory-level rewards (RTG) and action-level values. Existing methods cannot guarantee global optimality when sampling RTG from a single trajectory. In particular, when the trajectory endpoint is non-optimal, erroneous RTG values can lead to cumulative deviations in the spliced trajectory. For example, in sparse reward tasks, locally high-reward actions may be irrelevant to the global objective, and existing methods lack a dynamic mechanism to adjust the correlation between RTG and actions. Second, supervised trajectory prediction limits action exploration capabilities. These methods, which aim to maximize rewards, overly rely on the distribution of offline data, resulting in an inability to effectively explore potential high-value actions in suboptimal datasets. For example, in the Maze2D task, which contains a large number of low-quality trajectories, existing methods cannot even outperform imitation learning. Furthermore, some improved methods (such as Q-value regularization) attempt to alleviate the overestimation problem by constraining action selection, but they do not address the theoretical connection between RTG and action selection and still face the challenge of high-value actions being disconnected from the objective in sparse reward scenarios. The above problems make it difficult for existing methods to achieve reliable trajectory splicing and policy generalization in complex tasks (such as AntMaze navigation and multi-stage Kitchen tasks). There is an urgent need for a new framework that takes into account both reward consistency and action exploration. Summary of the Invention

[0004] In view of this, the present invention aims to propose an offline reinforcement learning action exploration agent method based on expected reward regularization to solve the problem that existing methods are difficult to achieve reliable trajectory splicing and strategy generalization in complex tasks (such as AntMaze navigation and multi-stage Kitchen tasks).

[0005] To achieve the above objectives, the present invention adopts the following technical solutions: an offline reinforcement learning action exploration agent method based on expected reward regularization, the method comprising an agent training phase and an inference phase;

[0006] The agent training phase includes:

[0007] Construct state loss and action loss based on sequence modeling, and perform iterative training of state and action through supervised learning;

[0008] Introducing expected reward RTG regularization prediction loss, designing an RTG loss function based on weighted square error, and introducing a learnable parameter α for dynamic regularization;

[0009] A dual Q-learning framework is used to maintain two independent Q-functions. The conservative constraints of conservative Q-learning are applied during the Q-function update process, and the Boltzmann distribution is combined to achieve action exploration optimization.

[0010] Combine state loss, action loss, RTG regularization loss and Q-value loss to form a joint optimization objective function;

[0011] The inference phase includes:

[0012] Perform noise perturbation sampling on multiple RTG candidate values to generate diverse action predictions;

[0013] The candidate actions are evaluated based on the double conservative Q function, and the action with the highest Q value is selected for execution.

[0014] Furthermore, a preferred method is proposed, where the expected reward RTG regularization prediction loss is:

[0015]

[0016] in, is a learnable regularization parameter, π θ is the agent strategy function, g i ,s i ,a i are the expected reward, state, and action at the corresponding time step, s t+K is the state of the t+Kth time step, m∈(0,1) is the temperature parameter of the expected regression, g t For residual income.

[0017] Furthermore, a preferred method is proposed, where the RTG loss function is:

[0018]

[0019] Among them, K is the time step length of one iteration, Represents the dataset Find the corresponding expectation, is the expected reward RTG regularized prediction loss, t is the time step, is the expected reward value predicted by the model, λ1 is the regularization parameter of the first term, λ2 is the regularization parameter of the second term, and α is the regularization variable.

[0020] Furthermore, a preferred method is proposed, wherein the conservative constraint method is specifically as follows: adding a conservative constraint term to the Q function optimization objective, and updating the Q function by the following formula:

[0021]

[0022] in, For the critic target network, For critic target strategy, s t is the state at time step t, a t-1 is the action at time step t-1, is the expected reward value predicted by the model at time step t.

[0023] Furthermore, a preferred method is proposed, in which the Q value loss is calculated as follows:

[0024]

[0025] Where U is a uniformly distributed random variable, is the action selected according to the corresponding strategy, For critics network strategy 1, Network strategy for critics2.

[0026] Furthermore, a preferred embodiment is proposed, wherein the method further comprises: using data enhancement technology to add noise or random perturbations to the state variables to generate diversified training samples.

[0027] Furthermore, a preferred embodiment is proposed, wherein the action selection process in the reasoning stage specifically includes:

[0028] Add Gaussian noise to the candidate RTG values to generate multiple sets of predicted actions;

[0029] Calculate the Q value of the state-action pair;

[0030] The optimal action is selected based on the Boltzmann distribution of Q values.

[0031] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes an offline reinforcement learning action exploration intelligent agent method based on expected reward regularization as described in any one of the above items.

[0032] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of an offline reinforcement learning action exploration agent method based on expected reward regularization as described above are executed.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The proposed method introduces learnable regularization and temperature parameters to dynamically adjust the prediction error weights of expected rewards, significantly alleviating the inconsistency between trajectory-level rewards and action-level values. In Maze2D and AntMaze tasks, the trajectory splicing error rate is reduced by over 30%, demonstrating that RTG regularization can effectively correct the accumulated errors at the endpoints of non-optimal trajectories. Furthermore, for the first time, a mathematical correlation between RTG and action selection is established within a conditional sequence modeling framework. Through a weighted loss function, the stability of long-term reward prediction is achieved, overcoming the limitations of traditional methods that rely solely on experimental verification.

[0035] 2. Our proposed method utilizes dual Q functions and conservative Q learning (CQL) constraints to effectively address the problem of misjudgment of high-value actions caused by suboptimal data. In sparse reward scenarios (e.g., the hopper-medium-replay task), our method achieves a score of 101.4, a 6.8% improvement over the traditional CQL score of 95.0, demonstrating that the dual Q mechanism can more accurately identify potentially high-reward actions. The conservative constraint forces the Q function to model the difference in action distribution between the behavioral and target policies, preventing overly optimistic estimates of the Q values for new actions.

[0036] 3. The method proposed in this paper introduces random Gaussian noise and Boltzmann distribution sampling during the inference phase, significantly improving the diversity of action exploration, especially in low-quality data scenarios, to achieve policy optimization. In the Kitchen-mixed task, the proposed method achieved a score of 67.1, a 21.8% improvement over the 55.0 of the Q-value regularized Transformer (QT) algorithm, demonstrating that noise injection and probabilistic sampling mechanisms can overcome the limitations of suboptimal data distribution. Data augmentation (such as state perturbations) expands the diversity of training samples and reduces dependence on expert trajectories.

[0037] 4. Our proposed method achieves the best average scores across the four D4RL benchmark domains of Gym, Maze2D, AntMaze, and Kitchen, demonstrating its adaptability to both dense and sparse reward tasks, as well as multi-stage tasks. By jointly optimizing RTG regularization and action exploration, we achieve the first-ever balance between trajectory concatenation and policy exploration within a conditional sequence modeling framework.

[0038] 5. This paper proposes a phased optimization of “supervised reinforcement learning loss” and “action exploration loss” to solve the problem of unstable end-to-end training in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0040] Figure 1 This is a flow chart of the offline reinforcement learning action exploration agent method based on expected reward regularization described in the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely explain the technical solutions in the embodiments of the present invention. It should be noted that the embodiments of the present invention and the features therein can be combined with each other in the absence of conflict, and the embodiments described are only part of the embodiments of the present invention, not all of the embodiments.

[0042] Implementation method 1, see Figure 1 This embodiment describes an offline reinforcement learning action exploration agent method based on expected reward regularization, which includes an agent training phase and an inference phase;

[0043] The agent training phase includes:

[0044] Construct state loss and action loss based on sequence modeling, and perform iterative training of state and action through supervised learning;

[0045] Introducing expected reward RTG regularization prediction loss, designing an RTG loss function based on weighted square error, and introducing a learnable parameter α for dynamic regularization;

[0046] A dual Q-learning framework is used to maintain two independent Q-functions. The conservative constraints of conservative Q-learning are applied during the Q-function update process, and the Boltzmann distribution is combined to achieve action exploration optimization.

[0047] Combine state loss, action loss, RTG regularization loss and Q-value loss to form a joint optimization objective function;

[0048] The inference phase includes:

[0049] Perform noise perturbation sampling on multiple RTG candidate values to generate diverse action predictions;

[0050] The candidate actions are evaluated based on the double conservative Q function, and the action with the highest Q value is selected for execution.

[0051] Implementation 2: This implementation further limits the offline reinforcement learning action exploration agent method based on expected reward regularization described in Implementation 1. The expected reward RTG regularization prediction loss is:

[0052]

[0053] in, is a learnable regularization parameter, π θ is the agent strategy function, g i ,s i ,a i are the expected reward, state, and action at the corresponding time step, s t+K is the state of the t+Kth time step, m∈(0,1) is the temperature parameter of the expected regression, g t For residual income.

[0054] Implementation method 3: This implementation method further limits the offline reinforcement learning action exploration agent method based on expected reward regularization described in implementation method 2. The RTG loss function is:

[0055]

[0056] Among them, K is the time step length of one iteration, Represents the dataset Find the corresponding expectation, is the expected reward RTG regularized prediction loss, t is the time step, is the expected reward value predicted by the model, λ1 is the regularization parameter of the first term, λ2 is the regularization parameter of the second term, and α is the regularization variable.

[0057] Implementation 4: This implementation further limits the offline reinforcement learning action exploration agent method based on expected reward regularization described in Implementation 1. The conservative constraint method is specifically: adding a conservative constraint term to the Q function optimization objective and updating the Q function by the following formula:

[0058]

[0059] in, For the critic target network, For critic target strategy, s t is the state at time step t, a t-1 is the action at time step t-1, is the expected reward value predicted by the model at time step t.

[0060] Implementation 5: This implementation further limits the offline reinforcement learning action exploration agent method based on expected reward regularization described in Implementation 1. The Q-value loss is calculated as follows:

[0061]

[0062] Where U is a uniformly distributed random variable, is the action selected according to the corresponding strategy, For critics network strategy 1, Network strategy for critics2.

[0063] Implementation method six. This implementation method is a further limitation of the offline reinforcement learning action exploration agent method based on expected reward regularization described in implementation method one. The method also includes: using data enhancement technology to add noise or random perturbations to state variables to generate diversified training samples.

[0064] Implementation 7: This implementation further limits the offline reinforcement learning action exploration agent method based on expected reward regularization described in Implementation 1. The action selection process in the reasoning stage specifically includes:

[0065] Add Gaussian noise to the candidate RTG values to generate multiple sets of predicted actions;

[0066] Calculate the Q value of the state-action pair;

[0067] The optimal action is selected based on the Boltzmann distribution of Q values.

[0068] Implementation 8. A computer device described in this implementation includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes an offline reinforcement learning action exploration agent method based on expected reward regularization as described in any one of Implementation 1 to Implementation 7.

[0069] Implementation method nine: A computer-readable storage medium described in this implementation method stores a computer program, and when the computer program is executed by a processor, the steps of an offline reinforcement learning action exploration agent method based on expected reward regularization as described in any one of implementation methods one to seven are executed.

[0070] Implementation 10: This implementation provides a specific example of the offline reinforcement learning action exploration agent method based on expected reward regularization described in Implementation 1, and is also used to explain Implementation 2 to Implementation 7. Specifically:

[0071] The overall process of this implementation is divided into two main parts: the agent training phase and the reasoning phase.

[0072] 1. Agent training phase

[0073] This phase consists of two core components: RTG regularization and action exploration. First, we explain the concept of RTG regularization proposed in this implementation, and theoretically demonstrate its ability to guide the agent's decision-making process to maximize reward during conditional sequence modeling optimization. Next, we explain the concept of action exploration under the constraints of a critic network. Here, we employ the principles of dual Q-learning to maintain two independent Q-functions and apply conservative constraints from conservative Q-learning (CQL) to the updates of these Q-functions to mitigate overestimation of the Q-values of new actions.

[0074] Step 1: Before performing RTG regularization, we must first build state and action losses based on sequence modeling reinforcement learning, and complete the iteration of state (State) and action (Action) in the form of supervised learning. Due to its similarity to data-driven supervised learning, offline reinforcement learning has been replaced by supervised sequence modeling, which is different from the classic Markov decision process formula. Offline dataset It can be defined as a sequence of segments in policy optimization without relying on transition probabilities, where g t is RTG, defined as It is the sum of future rewards at the current time step t, otherwise The Decision Transformer (DT) (Chen et al., 2021) is a representative work of this paradigm, which is based on the offline trajectory τ K Constructing the loss function and through the residual income g t Predict actions.

[0075]

[0076] Among them, τ K =(s t-K+1 ,at-K+1 ,g t-K+1 …,s t ,a t ,g t ) is the trajectory of the first K time steps, and the residual income is g t In DT, the strategy π DT Implemented by the causal transformer. For each time step t, three different tokens (s t ,a t ,g t ) is input into the model. The next action a t+1 Forecasting is done using an autoregressive model.

[0077] Step 2: Build state and action losses based on sequence modeling reinforcement learning, and further add RTG prediction loss. In the process of learning the optimal trajectory from the suboptimal trajectory, the goal of the agent is to select the best action based on the current state by maximizing the expected reward. Expected regression has been widely recognized in the fields of statistics and economics and has been widely used in offline reinforcement learning. Based on previous research results, this implementation integrates the inherent parallel estimation advantages of the Transformer architecture to calculate the RTG loss:

[0078]

[0079] in is a learnable regularization parameter, Where m∈(0,1) is the temperature parameter of the expected regression, and the setting method is the same as Algorithm 1. Based on this, the present invention proposes a regularized RTG loss function:

[0080]

[0081] Step 3: After constructing the state, action, and RTG prediction-related losses, design an optimization strategy for the action exploration portion. While predicting the RTG can alleviate the problem of inconsistency between expected and optimal returns, this solution is still not perfect. To address this issue, this implementation utilizes RTG regularization and employs the principles of dual Q-learning to maintain two independent Q functions. Conservative Q-learning (CQL) applies conservative constraints during the update process to mitigate overestimation of the Q value of new actions.

[0082] First, dual Q learning is used to maintain two independent Q functions and This reduces the estimation bias caused by using a single Q function. Taking into account the characteristics of trajectory prediction, the present invention uses the n-step Bellman equation to estimate the two Q functions. Specifically, the target strategy update of the Q value network is as follows:

[0083]

[0084] Secondly, during the update of these two Q-functions, we employ a conservative constraint approach from CQL to mitigate the overestimation of the Q-values associated with new actions. This is achieved by incorporating a conservative constraint term into the optimization objective of each Q-function. Specifically, a conservative lower bound Q-function is learned by minimizing the Q-value and the standard Bellman error objective.

[0085]

[0086] In formula (5), the first half represents the conservative penalty term, which captures the difference between the two expectations; and γ is a weight term that controls the trade-off between conservatism and the standard Q-learning loss. This implementation assumes that from the policy π β The sampled actions are expected to produce higher Q-values, while the target policy The sampled actions should better reflect the actual data distribution. By minimizing the formula, the value function estimate of the actions triggered by the new policy will not be overly optimistic, thereby enhancing the stability and reliability of the policy when executed in the real environment.

[0087] Finally, to further enhance action exploration capabilities and prevent the behavioral policy from falling prey to offline data errors, this implementation utilizes the Boltzmann distribution to select actions. This method calculates the probability of each action being selected based on its Q-value, effectively implementing a softmax action selection method. Furthermore, this implementation employs data augmentation techniques to transform the raw data, such as introducing noise or random perturbations into the state variables, to generate diverse training samples.

[0088] Step 4: After completing the loss function and Q value optimization strategy construction in steps 1 to 3, this implementation integrates the above steps to form a unified loss function for training. Since AE-RTG contains two core components, the loss function can be divided into supervised reinforcement learning loss and motion exploration loss As shown in Formula 7. For reinforcement learning through supervised learning loss, the state loss based on the DT model is introduced. and RTG regularization terms, as well as the Q-valued stochastic loss from double Q-learning, which facilitate the joint optimization of the supervisory components:

[0089]

[0090] in, and This is achieved through L2 norm loss, is a uniformly distributed random variable between (0,1).

[0091]

[0092] 2. Agent Reasoning Stage

[0093] Step 5: After completing the agent training based on step 4, the agent can be inferred. Since this embodiment targets an offline reinforcement learning method, no parameters are updated during the inference phase. During the inference phase, this embodiment does not rely on a single RTG estimate; instead, it samples multiple RTG prediction value candidates and adds random Gaussian noise to provide an evaluation basis for the action exploration phase. Subsequently, this embodiment generates multiple sets of action predictions based on these high-value reward estimates and uses the doubly conservative Q function learned in the action exploration phase to sample actions with higher Q values. The detailed steps of this training and inference process are outlined in Algorithm 1.

[0094]

[0095]

[0096] In order to verify the advantages of the method proposed in this embodiment, the performance of the method is compared with some state-of-the-art methods from different task fields and multiple perspectives, so as to analyze the advantages of the method proposed in this embodiment.

[0097] (1) Evaluation criteria

[0098] Four different task domains are considered in the D4RL benchmark: Gym, Maze2D, AntMaze, and Kitchen. The Gym task is a motion control challenge implemented in the MuJoCo engine, which is often used as a standardized benchmark in D4RL. This task exhibits a variety of data distribution configurations and dense and smooth reward configurations, providing a strong foundation for evaluating agent performance. The Maze2D task evaluates the agent's ability to effectively splice subtrajectories and identify the shortest path to the goal location, as the dataset for this task consists of trajectory fragments rather than complete optimal trajectories. While AntMaze and Maze2D show similarities in sparse reward performance, AntMaze presents a more challenging scenario; here, the agent must navigate a quadruped robot for path planning, unlike the simpler point-target navigation in Maze2D. The Kitchen task is based on the Adept environment and presents a multi-task challenge that evaluates the agent's ability to generalize from unseen observations to unstructured task completion by requiring the agent to complete four subtasks.

[0099] (2) Comparison method

[0100] During the validation process, this implementation evaluated several baseline methods from multiple domains and categorized them into two main groups: traditional reinforcement learning (RL) methods and conditional sequence modeling (CSM) methods. Each group of methods demonstrated excellent performance on tasks within specific domains. For traditional RL methods, this implementation used Algorithm 1: Behavior Cloning (BC), Algorithm 2: TD3+BC, Algorithm 3: Conservative Q-Learning (CQL), and Algorithm 4: Implicit Q-Learning (IQL), as these methods demonstrated excellent performance within their respective paradigms and could serve as foundational heuristic prototypes for this implementation. In the CSM domain, representative methods were compared, including Algorithm 5: Decision Transformer (DT), Algorithm 6: Online Decision Transformer (ODT), Algorithm 7: Elastic Decision Transformer (EDT), Algorithm 8: Graph Decision Transformer (GDT), and Algorithm 9: Q-learning Decision Transformer (QDT), as well as state-of-the-art techniques such as Algorithm 10: Reinformer and Algorithm 11: Q-value Regularized Transformer (QT). In addition, this embodiment also evaluates diffusion-based methods, including Algorithm 12 Diffuser, Algorithm 13 DD, and Algorithm 14 BG+DT. The task scores of these baseline methods are either derived from the best results reported in their corresponding academic papers or from repeated experiments in this paper to ensure fair principled comparison.

[0101] (3) Main results

[0102] The specific results of the method proposed in this embodiment and Algorithms 1-14 are shown in Table 1. To ensure fairness and standardization in the comparison, the scores are normalized according to the D4RL benchmark, where 100 and 0 correspond to the expert strategy and random strategy, respectively. The results for different task domains are further analyzed here.

[0103] Table 1 Comparison results of the present invention and Algorithms 1-14 on the D4RL benchmark

[0104]

[0105]

[0106] Gym task domain results. The results in Table 1 show that the traditional RL methods used in the baseline model are comparable to most conditional SM methods. It is worth noting that Reinformer and QT perform well in the Gym series of tasks, and the method proposed in this embodiment further improves their effectiveness. AE-RTG performs significantly better than other methods in the "medium" series of tasks. It is worth noting that the "medium" datasets are derived from one-third of the performance level of expert agents, and these datasets lack a sufficient number of high-reward trajectories. Specifically, in the "hopper-medium" and "walker2d-medium" tasks, the lack of RTG predictions in the SM method causes its performance to be inferior to traditional RL methods such as IQL. The method proposed in this embodiment adopts dual conservative Q learning and performs exploratory policy optimization based on RTG predictions, thereby achieving excellent performance.

[0107] Results in the Maze2D and AntMaze task domains. Maze-related tasks are often a good environment for evaluating an agent's ability to effectively stitch together trajectories, as their datasets contain a large number of suboptimal trajectories and sparse rewards. Combining RTG prediction with a conditional sequence modeling approach based on doubly conservative Q-learning is a means of effectively evolving existing suboptimal trajectories. While agents with the added diffusion model perform well on some datasets, the method proposed in this embodiment is more adaptable to data of varying quality and achieves better average performance.

[0108] Kitchen domain results. The Kitchen task involves numerous subtasks and delayed rewards, which can affect the performance of the RTG agent and lead to ineffective predictions. While reinforcement learning methods using diffusion modeling techniques can alleviate these challenges to some extent, our method performs well on both "complete" and "mixed" datasets, outperforming other methods on average.

[0109] The specific embodiments of the present invention disclosed above are intended only to help illustrate the present invention. The specific embodiments do not describe all details in detail, nor do they limit the invention to the specific embodiments described. Numerous modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention.

Claims

1. An offline reinforcement learning action exploration agent method based on expected reward regularization, characterized by: The method includes an agent training phase and an inference phase; The agent training phase includes: Construct state loss and action loss based on sequence modeling, and perform iterative training of state and action through supervised learning; Introducing expected reward RTG regularization prediction loss, designing an RTG loss function based on weighted square error, and introducing a learnable parameter α for dynamic regularization; A dual Q-learning framework is used to maintain two independent Q-functions. The conservative constraints of conservative Q-learning are applied during the Q-function update process, and the Boltzmann distribution is combined to achieve action exploration optimization. Combine state loss, action loss, RTG regularization loss and Q-value loss to form a joint optimization objective function; The inference phase includes: Perform noise perturbation sampling on multiple RTG candidate values to generate diverse action predictions; The candidate actions are evaluated based on the double conservative Q function, and the action with the highest Q value is selected for execution.

2. The offline reinforcement learning action exploration agent method based on expected reward regularization according to claim 1, characterized in that: The expected reward RTG regularized prediction loss is: in, is a learnable regularization parameter, π θ is the agent strategy function, g i ,s i ,a i are the expected reward, state, and action at the corresponding time step, s t+K is the state of the t+Kth time step, m∈(0,1) is the temperature parameter of the expected regression, g t For residual income.

3. The offline reinforcement learning action exploration agent method based on expected reward regularization according to claim 2, characterized in that: The RTG loss function is: Among them, K is the time step length of one iteration, Represents the dataset Find the corresponding expectation, is the expected reward RTG regularized prediction loss, t is the time step, is the expected reward value predicted by the model, λ1 is the regularization parameter of the first term, λ2 is the regularization parameter of the second term, and α is the regularization variable.

4. The offline reinforcement learning action exploration agent method based on expected reward regularization according to claim 1, characterized in that: The conservative constraint method is specifically as follows: adding a conservative constraint term to the Q function optimization objective and updating the Q function by the following formula: in, For the critic target network, For critic target strategy, s t is the state at time step t, a t-1 is the action at time step t-1, is the expected reward value predicted by the model at time step t.

5. The offline reinforcement learning action exploration agent method based on expected reward regularization according to claim 1, characterized in that: The Q-value loss is calculated as follows: Where U is a uniformly distributed random variable, is the action selected according to the corresponding strategy, For critics network strategy 1, Network strategy for critics2.

6. The offline reinforcement learning action exploration agent method based on expected reward regularization according to claim 1, characterized in that: The method further includes: using data enhancement technology to add noise or random disturbance to the state variables to generate diversified training samples.

7. The offline reinforcement learning action exploration agent method based on expected reward regularization according to claim 1, characterized in that: The action selection process in the reasoning stage specifically includes: Add Gaussian noise to the candidate RTG values to generate multiple sets of predicted actions; Calculate the Q value of the state-action pair; The optimal action is selected based on the Boltzmann distribution of Q values.

8. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes an offline reinforcement learning action exploration agent method based on expected reward regularization according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of an offline reinforcement learning action exploration agent method based on expected reward regularization according to any one of claims 1 to 7.

Images