Method and device for state adaptive reinforcement learning

Abstract

A method for a state adaptive offline to online reinforcement learning module is disclosed. The reinforcement learning module comprises a balance assembly and a universal assembly. The method includes: receiving a state of an environment of a reinforcement learning module, the state based on at least one of an image, a video, an audio, and text representing the environment; generating, by a balancing component, a balancing coefficient based on the received state, the balancing coefficient controlling a balance between a policy improvement and a policy constraint of a generic component for the output action; and outputting, by the generic component, an action based on the received state and the generated balance coefficient.

Description

Technical Field

[0001] This disclosure generally relates to the field of artificial intelligence technology, and more specifically, to offline-to-online reinforcement learning with state-adaptive policy constraints. Background Technology

[0002] Reinforcement learning (RL) is an important field of machine learning that aims to solve the problem of how an agent should take actions under different states of an environment in order to maximize some kind of cumulative reward. Offline RL is a practical approach that learns policies based on previously collected datasets without any interaction with the environment. Therefore, in offline RL, the quality of the dataset often limits the performance of the agent's policy for taking actions. Offline-to-online RL alleviates this problem by combining offline RL policies pre-trained on previously collected datasets and fine-tuning the pre-trained policies in an online environment.

[0003] Furthermore, since offline RL learns policies on previously collected datasets, it often suffers from distribution bias due to the gap between the state-action distribution of the offline dataset and the interaction between the current policy and the test environment. The involvement of online fine-tuning can exacerbate this distribution bias problem. Existing solutions to this problem involve imposing policy constraints on the policy improvement objective in both offline and online learning. Typically, a uniform balance is struck between policy improvement and policy constraints for the diverse data in the collected dataset. However, this generalized solution may not optimally utilize each collected sample.

[0004] Therefore, there is a need to provide an improved solution for offline-to-online reinforcement learning. Summary of the Invention

[0005] The following is a simplified overview of one or more aspects of this disclosure in order to provide a basic understanding of these aspects. This overview is not a comprehensive overview of all anticipated aspects, and is neither intended to identify key or essential elements of all aspects, nor to depict the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that follows.

[0006] In one aspect of this disclosure, a method for outputting an action by a reinforcement learning module is disclosed. The reinforcement learning module includes a balancing component and a general component. The method includes: receiving a state of an environment of the reinforcement learning module, the state being based on at least one of an image, video, audio, and text representing the environment; generating balancing coefficients by the balancing component based on the received state, the balancing coefficients controlling a balance between policy improvement and policy constraints of the general component used to output the action; and outputting an action by the general component based on the received state and the generated balancing coefficients.

[0007] In another aspect of this disclosure, an apparatus for outputting an action by a reinforcement learning module is disclosed. The apparatus may include a memory and at least one processor coupled to the memory. The reinforcement learning module includes a balancing component and a general component, which may be stored in the memory and executed by the processor. The at least one processor may be configured to: receive a state of the environment of the reinforcement learning module, the state being based on at least one of an image, video, audio, and text representing the environment; generate balancing coefficients by the balancing component based on the received state, the balancing coefficients controlling the balance between policy improvement and policy constraints of the general component used to output the action; and output an action by the general component based on the received state and the generated balancing coefficients.

[0008] In another aspect of this disclosure, a computer program product for outputting an action by a reinforcement learning module is disclosed. The reinforcement learning module includes a balancing component and a general component. The computer program product may include processor-executable computer code for: receiving a state of the environment of the reinforcement learning module, the state being based on at least one of an image, video, audio, and text representing the environment; generating balancing coefficients by the balancing component based on the received state, the balancing coefficients controlling the balance between policy improvement and policy constraints of the general component used to output the action; and outputting an action by the general component based on the received state and the generated balancing coefficients.

[0009] In another aspect of this disclosure, a method for training a reinforcement learning module is disclosed. The reinforcement learning module includes a balancing component and a general component. The method includes: training the balancing component to learn a mapping from each state of the environment of the reinforcement learning module to balancing coefficients, the balancing coefficients controlling the balance between policy improvement and policy constraints of the general component for outputting actions, and the state being based on at least one of an image, video, audio, and text representing the environment; and training a family of policies with different balancing coefficients for the general component to output actions based on the state and the state-associated balancing coefficients.

[0010] In another aspect of this disclosure, an apparatus for training a reinforcement learning module is disclosed. The apparatus may include a memory and at least one processor coupled to the memory. The reinforcement learning module includes a balancing component and a general component, which may be stored in the memory and executed by the processor. The at least one processor may be configured to: train the balancing component to learn a mapping from each state of the environment of the reinforcement learning module to balancing coefficients, the balancing coefficients controlling the balance between policy improvement and policy constraints of the general component for outputting actions, and the states being based on at least one of an image, video, audio, and text representing the environment; and train a family of policies with different balancing coefficients for the general component to output actions based on the states and the state-related balancing coefficients.

[0011] In another aspect of this disclosure, a computer program product for training a reinforcement learning module is disclosed. The reinforcement learning module includes a balancing component and a general component. The computer program product may include processor-executable computer code for: training the balancing component to learn a mapping from each state of the environment of the reinforcement learning module to balancing coefficients, the balancing coefficients controlling the balance between policy improvement and policy constraints of the general component for outputting actions, and the states being based on at least one of an image, video, audio, and text representing the environment; and training a family of policies with different balancing coefficients for the general component to output actions based on the states and the state-related balancing coefficients.

[0012] In another aspect of this disclosure, a computer-readable medium is disclosed for storing computer code for reinforcement learning. When executed by a processor, the computer code can cause the processor to perform the methods described in this disclosure.

[0013] Other aspects or variations of this disclosure will become apparent from the following detailed description and accompanying drawings. Attached Figure Description

[0014] The following figures depict various embodiments of the present disclosure for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the methods and structures disclosed herein can be implemented without departing from the spirit and principles of the present disclosure described herein.

[0015] Figure 1 The basic structure of a reinforcement learning model according to one aspect of this disclosure is shown.

[0016] Figure 2 A tSNE visualization of the state from an existing dataset is shown according to one aspect of this disclosure.

[0017] Figure 3 A block diagram of a state-adaptive offline-to-online reinforcement learning module according to one aspect of this disclosure is shown.

[0018] Figure 4 A flowchart is shown for a method for outputting actions by a reinforcement learning module according to one aspect of this disclosure.

[0019] Figure 5 A flowchart is shown for a method for training a state-adaptive offline-to-online reinforcement learning module according to one aspect of this disclosure.

[0020] Figure 6 A block diagram of an apparatus for a state-adaptive offline-to-online reinforcement learning module according to one aspect of this disclosure is shown.

[0021] Figure 7 A comparison between existing reinforcement learning schemes and state-adaptive reinforcement learning schemes according to one aspect of this disclosure is shown.

[0022] Figure 8 A comparison is shown between a fixed equilibrium coefficient scheme and a state-adaptive equilibrium coefficient scheme according to one aspect of this disclosure. Detailed Implementation

[0023] Before explaining any embodiment of this disclosure in detail, it should be understood that this disclosure is not limited in its application to the arrangement of the details and features of the construction set forth in the following description. This disclosure can have other embodiments and can be implemented or performed in various ways.

[0024] Figure 1 The basic structure of a reinforcement learning model according to one aspect of this disclosure is shown. Reinforcement learning mainly includes an agent, environment, state, action, and reward. Figure 1 As shown, the current state of environment 120 s t The intelligent agent 110 can perform actions on the environment 120 based on policies. a t Then, environment 120 can transition to the new state. s t+1 And as a reward for feedback from Environment 120 r An agent 110 is given a reward. The agent 110 can then iteratively perform new actions based on the reward and the new state of the environment 120. The policy of the agent 110 aims to maximize the cumulative reward. This process illustrates how the agent and the environment interact through states, actions, and rewards. Through reinforcement learning, the agent can determine what actions it should take to obtain the maximum reward when in different states of the environment.

[0025] Typically, reinforcement learning can be represented as tuples labeled with tuples. The Markov Decision Process (MDP), in which It is a state space. It is the action space. This represents the probability of state transitions in the environment. and It belongs to the state space. Within the action space, the agent's policy can be based on such probabilities. Represents the initial state , Define the reward function, and It is a discount factor.

[0026] Reinforcement learning models can be implemented as reinforcement learning modules that include software modules, hardware modules, or a combination of both. For simplicity, the terms RL model and RL module are used interchangeably unless otherwise specified. RL modules can be used for various applications suitable for reinforcement learning.

[0027] In one specific implementation, a reinforcement learning module can be used for a robot. For example, the robot could be an AI-powered robotic arm used in an automated production workshop. In a robot control implementation, the agent can be the robot itself or, more specifically, the robot's brain. The environment can be the robot's working environment, such as the environment where the robot works. The environment can have various states, such as obstacles, ramps, pits, etc. Information about the environment can be obtained from various sensors equipped on the robot, such as cameras, LiDAR, millimeter-wave radar, ultrasonic radar, microphones, temperature sensors, pressure sensors, etc. The state of the environment can be determined based on at least one of images, videos, audio, and text captured and representing the environment by various sensors. Typically, the state can be a vector describing the current observation of the environment. Actions can be the robot's maneuvers. For example, the robot can take different actions in different states based on a learned strategy in its brain, such as walking on a road. If the robot takes the correct action and navigates around obstacles, it may receive a higher total reward.

[0028] In another specific implementation, the reinforcement learning module can be used in autonomous vehicles. In such an implementation, the agent can be a vehicle or a central control unit of a vehicle. The environment can be the traffic environment of the autonomous vehicle, including other vehicles and pedestrians on the road, traffic lights, traffic signs, etc. Actions can be driving actions of the autonomous vehicle, such as turning left, turning right, accelerating, braking, etc. Rewards can be related to various factors, such as safety, economy (fuel consumption), speed (time spent reaching the destination), etc., or combinations thereof.

[0029] In another specific implementation, the reinforcement learning module can be used for natural language processing (NLP), particularly for large language models (LLM). For example, the agent can be a chatbot, and the environment can be a dialogue context. The state of the environment can be determined based on language segments in text or speech form, and actions can include responding to or answering the input language segments. The RL scheme disclosed herein is not limited to a specific application and can improve the performance of RL modules used in a variety of applications due to the novel structure employed in this RL scheme.

[0030] Offline-to-online (OTO) RL is a widely used approach for training RL models. OTO RL consists of two phases: (i) offline pre-training: pre-training the policy based on an offline dataset; and (ii) online fine-tuning: fine-tuning the pre-trained policy by interacting with the environment. During the offline pre-training phase, the policy cannot interact with the environment, but during the online fine-tuning phase, the policy can access the offline dataset.

[0031] One of the most concerning issues in offline RL is distribution shift, where agents may learn inaccurate estimates of out-of-distribution (OOD) state-action pairs. Existing solutions to this problem include policy constraints, importance sampling, regularization, uncertainty estimation, and imitation learning. Offline-to-online RL adds online fine-tuning to offline RL to enhance policy performance. While online fine-tuning significantly improves policy performance, its involvement can exacerbate the distribution shift problem, where policy behavior deviates from the distribution of the collected dataset. The distribution shift problem is often triggered by overly aggressive policy improvements. Furthermore, distribution shift worsens due to changes in the state distribution when switching from offline learning to online fine-tuning. To address this, policy constraints can be imposed on policy improvements to prevent the policy from overexploring the unknown policy space. However, insufficient exploration can be counterproductive to policy improvements and make online fine-tuning overly conservative. In other words, offline-to-online RL requires a proper trade-off between policy improvement and policy constraints in the policy optimization objective. Existing offline-to-online RL solutions typically employ a one-size-fits-all balance between improvements and constraints across the entire collected dataset or all data in a small batch.

[0032] Specifically, the training objective of an offline-to-online RL solution can consist of two parts, explicitly or implicitly: (i) policy improvement, which aims to optimize the policy based on the current value function; and (ii) policy constraint, which keeps the policy around the distribution of the offline dataset or the current replay buffer. For example, training objectives similar to those of implicit Q-learning (IQL) and advantage-weighted actor-commentator (AWAC) RL schemes are shown in equation (1). in, It is to maximize the goal. It's a playback buffer. It is a state-action function, and It is a value function. The policy constraint term and policy improvement term are defined and shown in equation (1). Balance coefficients. These are predefined hyperparameters that weigh policy improvement terms against policy constraints, and for All data in the simulation remain the same. Simulation weights are determined for simulating state-action pairs. The strength of the strategy. At the start of the offline pre-training phase, using a pre-collected offline dataset... offline ={( s k , a k , s' k , r k )| k =1, 2, ..., N} to the beginning Furthermore, during the online fine-tuning phase, samples collected from online interaction can be continuously added to the replay buffer. .

[0033] Figure 2 A tSNE (t-distributed random neighbor embedding) visualization of states from an existing dataset, according to one aspect of this disclosure, is shown, illustrating the correlation between the quality of the collected data and the states. Plot (a) is generated based on states randomly sampled from the HalfCheah intermediate expert dataset in D4RL, and plot (b) is generated based on states randomly sampled from the Hopper intermediate expert dataset in D4RL. Each point in the plot represents a state, and the grayscale of each point indicates the return of the trajectory to which each state belongs; the darker the color, the higher the return. Figure 2 It can be seen that the quality of the collected data varies greatly with different states. In other words, the quality of the collected data is highly correlated with the state. It can be recognized that for states where the collected data produces high trajectory rewards, the RL scheme needs to place more emphasis on policy constraints and take "conservative" actions; otherwise, for states where the collected data leads to poor trajectory rewards, the algorithm should focus more on policy improvement and take "aggressive" actions. A general balance between policy constraints and policy improvement may not optimally utilize every sample.

[0034] This disclosure provides a state-adaptive offline-to-online RL scheme that determines an appropriate improvement-constraint balance for each state. For example, in an autonomous driving implementation, the state may be associated with a complex or simple traffic environment, a highway traffic environment, or an urban traffic environment. In another natural language processing implementation, the state may be associated with language from speech or text from individuals with different educational backgrounds. It can be demonstrated that a state-related balance can contribute to higher performance of the RL module.

[0035] The state-adaptive offline-to-online RL scheme in this disclosure aims to solve policy optimization problems with pointwise KL (Kullback-Leibler) constraints as defined in the following equations (2)-(4): in, It is the first Strategy in the next iteration This indicates a replay buffer. Behavioral strategies for action selection in the context of action selection. yes The state distribution, and Is with state The relevant constants. The optimal strategy found in equations (2) to (4) is used as... .

[0036] For comparison, this paper also provides an optimization problem with distributed KL constraints, which is widely used in existing schemes. The definition of the policy optimization problem with distributed KL constraints is the same as the definition of the policy optimization problem with pointwise KL constraints provided above, except that equation (3) is replaced by equation (5) below, where It is a constant: (5) The update rule in equation (1) is derived under the optimization problem with distributed KL constraints.

[0037] If the optimal value found in a policy optimization problem with pointwise KL constraints is labeled as And the optimal value found in the policy optimization problem with distributed KL constraints will be labeled as Then it can be proven that these optimal values ​​satisfy the following: (6) This indicates that by utilizing appropriate point-by-point constraints The optimal value under pointwise KL constraints is no less than the optimal value under distributed KL constraints. Therefore, the method under pointwise constraints can achieve better performance than the previous method under distributed KL constraints.

[0038] For policy optimization problems with pointwise KL constraints, assume the state space... ( l (is the state dimension), and the feasible space constrained by equations (3) to (4) for each If it is not empty, then it is represented as of The optimal solution satisfies this , (7) in, It is a state-adaptive or state-dependent balance coefficient, and It is a normalization term. When using a parameterization strategy... To approximate the optimal strategy At that time, the training objective can be formulated as: (8) This indicates that the design of state-dependent equilibrium coefficients is necessary for solving optimization problems with pointwise KL constraints.

[0039] The most important difference between the scheme in this disclosure and existing schemes is that the balance coefficients in equations (7) and (8) are state-adaptive, while the balance coefficients in equation (1) are predefined hyperparameters. Since the state-adaptive offline-to-online RL scheme deals with problems with finer-grained constraints, requiring the balance coefficients to depend on the state, it is beneficial to achieve higher performance. It should be understood that the above equations are provided to illustrate the basic ideas and technical effects of this disclosure, and not to limit the scope of this disclosure.

[0040] Figure 3 A block diagram of a state-adaptive offline to online reinforcement learning module according to one aspect of this disclosure is shown. Figure 3 As shown, the state-adaptive offline-to-online RL module 310 includes a balancing component 312 and a general component 314. The general component 314 can be based on the state of the environment 320 determined by the balancing component 312. s Determined balance coefficient To adjust. Based on the balance coefficient. and state s The general component 314 can decide whether to take a conservative or aggressive action. The balancing component 312 can learn the mapping from the state to the balance coefficient and appropriately assist the general component 314 in execution at each state.

[0041] Specifically, the balancing component 312 can be based on a balancing model. Furthermore, the general component 314 can be based on a general model. Balance coefficient The space can be represented as For each state Balance model To obtain a suitable balance coefficient State-based State-related equilibrium coefficient General Model Output Action a It is worth noting that, and Both are stochastic models. (The symbol "...") "Used solely for simplicity and as a function. Balance coefficient" Used to control the general model In processing status The degree of conservatism / radicalism at that time. Environment 320 can be determined by... It means that among them It is relevant to the previous state s Input actions a The next state of the environment. Taking autonomous driving applications as an example, the next state of the vehicle in the traffic environment. s After making a left turn, the vehicle will face a new traffic situation. .

[0042] This state-adaptive RL scheme is a simple and effective plug-in framework that adaptively determines appropriate balancing coefficients for each state. It can be easily implemented on top of existing offline-to-online schemes with advantage-weighted regression (which can be referred to as the "base scheme"), such as IQL and AWAC.

[0043] Figure 4 A flowchart is shown for a method 400 for outputting actions by a reinforcement learning module according to one aspect of this disclosure. The reinforcement learning module may be as follows: Figure 3 The state-adaptive offline-to-online RL module shown includes balanced and general components. The reinforcement learning module can be based on the following references... Figure 5 The method for training the RL model is described in detail. For example, the balancing component can be trained based on a convolutional neural network or a multilayer perceptron neural network. In one embodiment, the balancing component and the general component can be trained based on the same set of state-action Q functions and value V functions, for example, in equations (7) and (8).

[0044] like Figure 4As shown, method 400 may include receiving the state of the environment of the reinforcement learning module 410. The state may be a vector describing the current observation of the environment. Specifically, the state is based on at least one of images, video, audio, and text representing the environment. For example, in an embodiment where the reinforcement learning module is used for an autonomous vehicle, the environment is the traffic environment of the autonomous vehicle, and the state of the traffic environment may be based on video and / or images captured by cameras and / or various radar sensors on the vehicle. The state of the traffic environment may also be based on text on traffic signs captured by cameras and processed by existing computer vision techniques. These images, videos, and / or text may represent the state of the traffic environment, such as the location or movement of pedestrians, the color of traffic lights, or speed limits of traffic signs. Data associated with these images, videos, and / or text may also be received from other vehicles or networks. Similarly, in another embodiment where the reinforcement learning module is used for a robot, the environment is the robot's working environment; and in another embodiment where the reinforcement learning module is used for a large language model, the environment may be a dialogue context represented by audio and / or text.

[0045] Given the current state of the environment, method 400 can generate 420 balance coefficients based on the received state by a balancing component (such as balancing component 312). The balance coefficients can control the balance between policy improvement and policy constraints for general components (such as general component 314) used for output actions. Taking autonomous driving as an example, the balance coefficients can control whether the agent (i.e., in this example, the vehicle's control unit) takes actions conservatively (i.e., drives the vehicle) in a way more similar to samples in the dataset, or takes actions aggressively (i.e., drives the vehicle) in a way that deviates from samples but obtains higher rewards. The balance coefficients can be passed to general components of the RL module.

[0046] Then, method 400 can output action 430 by a general component based on the received state and the generated balance coefficient. In various embodiments, the action can be driving actions of an autonomous vehicle (such as turning, accelerating, braking, etc.), maneuvering actions of a robot (such as walking, raising an arm, etc.), or speaking / writing actions of a chatbot (such as answering questions). Since the balance coefficient is input into the general component along with the high-dimensional state, in a preferred embodiment, the dimension of the balance coefficient can be extended by positional encoding, and method 400 can output the action based on the received state and the balance coefficient with extended dimensions to avoid the balance coefficient being ignored by the general component.

[0047] Figure 5A flowchart of a method 500 for training a state-adaptive offline-to-online reinforcement learning module according to one aspect of this disclosure is shown. Method 500 can be implemented by one or more computers. The computers may include a server computer, a client computer, or both a server computer and a client computer. Figure 3 As shown, the state-adaptive offline-to-online RL module can include balancing components and general-purpose components. Therefore, as Figure 5 As shown, in box 510, method 500 includes training a balancing component to learn a mapping from each state of the RL module's environment to balancing coefficients. The balancing coefficients can control the balance between policy improvements and policy constraints for the general component used to output actions. The state can be based on at least one of an image, video, audio, and text representing the environment. And in box 520, method 500 includes training a family of policies with different balancing coefficients for the general component to output actions based on the state and the state-associated balancing coefficients. It should be recognized that, although shown in two separate boxes 510 and 520 and described in further detail below, the balancing component and the general component are trained together in combination during the training process.

[0048] Typically, state-adaptive offline-to-online RL training methods can include an offline pre-training phase and an online fine-tuning phase. In the offline pre-training phase, based on the extensive environmental information contained in the collected dataset, Method 500 can train a family of policies from conservative to aggressive, where each policy is optimized with different balance coefficients. In the online fine-tuning phase, guided by environmental feedback, Method 500 can learn to select policies from the policy family corresponding to appropriate balance coefficients for each state.

[0049] Specifically, to train a family of policies with different balance coefficients for a general component, in the offline pre-training phase, method 500 may include: sampling a first mini-batch from an offline dataset, assigning random balance coefficients to each sample in the first mini-batch, and updating the general component with the first mini-batch and a set of assigned random balance coefficients based on the optimization objective; and in the online fine-tuning phase, method 500 may include: sampling a second mini-batch from an online dataset, generating balance coefficients for each sample in the second mini-batch using the balance component, and updating the general component with the second mini-batch and a set of generated balance coefficients based on the optimization objective. The online dataset may include samples from the offline dataset and samples collected by the RL module interacting with the environment during the online training phase.

[0050] In one embodiment, a general model of a general component The optimization objective is shown as follows: (9) in, and It is the first In the next iteration and Function. Equation (9) is similar to equation (8), except that the strategy in equation (9) additionally receives a balance coefficient. As input. During the offline pre-training phase, the balanced coefficient space can be used. In the random sampling equation (9) To enforce a universal model Learn a family of strategies. During the online fine-tuning phase, the data is input into the general model. Previously, the balance model was based on the balance components. Sure This will enhance the collaboration between the two models.

[0051] To train the balancing component, method 500 may include updating the balancing component by translating balancing coefficients into actions using a general component, thereby maximizing the value of the state-action Q-function of the reinforcement learning module. In one embodiment, the balancing model of the balancing component... The update targets are as follows: (10) in, This is the updated general model in equation (9). Equation (10) can be used... The search can maximize the ability to convert balance coefficients into actions. Value balance model Under the update rule in equation (10), the balancing component can successfully learn to determine the appropriate balancing coefficient for each state based on the data quality associated with the current state.

[0052] As shown in equations (9) to (10), a single set of Q and V functions are used to evaluate the general model. π u and equilibrium model π b This is because of the equilibrium model. With the general model They can be considered together as standard RL strategies Therefore, the update of the value function can be the same as the update in the basic scheme for implementing the state-adaptive RL scheme of this disclosure, only using... Replace the original strategy.

[0053] The following provides an example of pseudocode for a state-adaptive offline-to-online RL model training process according to a specific embodiment of this disclosure.

[0054] In this embodiment, the state-adaptive offline-to-online RL scheme can be implemented on the code of existing basic RL schemes such as IQL and AWAC. Key modifications may include (i) adding a balancing component, and (ii) conditioned the policy on balancing coefficients, which is implemented by a generic component. For the first modification, the balancing model for the balancing component can be constructed using a stochastic model employed by the soft actor-critic (SAC) in rlkit (a collection of algorithms for reinforcement learning). For the second modification, the policy dimension of the generic component can be changed from the state dimension. l Change to l + l b ,in l b This refers to the dimension of the balance coefficient encoding. The balance coefficients can be encoded in the same way as the positional encoding proposed by Vaswani, A et al. in "Attention is all you need" at "Advances in neural information processing systems" 2017. This encoding expands the dimension of the balance coefficients and prevents them from being ignored by general components.

[0055] The reasonable range of the balancing coefficients can depend on the base scheme and the environment in which the state-adaptive offline-to-online RL scheme is implemented. However, the state-adaptive RL scheme is insensitive to the range of balancing coefficients for a specific implementation within a reasonable range. For example, when implementing a state-adaptive offline-to-online RL scheme on IQL, the adaptive balancing coefficients can be no less than 1.0 and no greater than 5.0 for the Locomotion dataset in D4RL, and no less than 6.0 and no greater than 14.0 for the AntMaze dataset in D4RL. When implementing a state-adaptive offline-to-online RL scheme on AWAC, the adaptive balancing coefficients can be no less than 1.0 and no greater than 3.0 for the Locomotion dataset, and no less than 9.0 and no greater than 11.0 for the AntMaze dataset.

[0056] Regarding training steps N1 and N2, for the implementation on IQL, 10 can be used in both the offline pre-training phase and the online fine-tuning phase. 6 Gradient step size. For the AWAC implementation, 2.5x10 can be used during the offline pre-training phase. 4 Gradient step size, and the use of 2x10 during the online fine-tuning phase. 5 (Locomotion) / 1x10 6(AntMaze) Gradient step size. Tables 1 and 2 below show more details of the implementation on IQL and AWAC, respectively, where “Adam” was proposed by Kingma, DP, et al. in “A method forstochastic optimization” at the “International Conference on Learning Representations” 2015, and “ReLU” was proposed by Agaurp, AF in “Deep learning using rectified linear units (reLU)”. arXiv preprint arXiv It was proposed in 2018, 1803.08375. Table 1 Implementations on IQL Table 2 Implementation on AWAC

[0057] Figure 6 A block diagram of an apparatus for a state-adaptive offline-to-online reinforcement learning module according to one aspect of the present disclosure is shown. In one embodiment, apparatus 600 can be used to output actions from a reinforcement learning module. Apparatus 600 may include memory 610 and at least one processor 620. Processor 620 may be coupled to memory 610 and configured to perform the actions described above. Figure 4 Method 400 is described. In another embodiment, apparatus 600 can be used to train a reinforcement learning module. Processor 620 can be configured to perform the above-described method. Figure 5 The method 500 is described. Processor 620 can be a general-purpose processor, or it can be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Memory 610 can store input data, output data, data generated by processor 620, and / or instructions executed by processor 620. Memory 610 can store state-adaptive offline to online reinforcement learning modules, where they are implemented in software, as well as their included balancing and general-purpose components.

[0058] The various operations, modules, components, and networks described herein can be implemented by hardware, software executed by a processor, firmware, or any combination thereof. According to embodiments of this disclosure, a computer program product for a state-adaptive RL scheme may include methods for performing the above-described references. Figure 4-5The described methods 400 and 500 contain processor-executable computer code. According to another embodiment of this disclosure, a computer-readable medium can store computer code for a state-adaptive RL scheme. This computer code, when executed by a processor, enables the processor to perform the above-described... Figure 4-5 Methods 400 and 500 are described. A computer-readable medium includes both non-transitory computer storage media and communication media, wherein the communication medium includes any medium that facilitates the transfer of a computer program from one place to another. Any connection can be suitably referred to as a computer-readable medium. Other embodiments and implementations are also within the scope of this disclosure.

[0059] Figure 7 A comparison between existing reinforcement learning schemes and state-adaptive reinforcement learning schemes according to one aspect of this disclosure is shown. Figure 7 The graphs (a), (b), (c), and (d) in the figure show the performance of various RL schemes in four metrics (i.e., IQM, intermediate, optimal gap, and mean), respectively. The horizontal axis represents the D4RL normalized score. These RL schemes, from top to bottom, implement the state-adaptive RL scheme of this disclosure on IQL, CQL, AWAC, TD3+BC, SAC, and BC. It can be seen that, particularly in IQM, optimal gap, and mean, the 95% CI of the state-adaptive scheme does not overlap with existing schemes. For intermediate, the expectations of all baselines are below the lower bound of the 95% CI of the state-adaptive scheme. This indicates that the state-adaptive policy-constrained scheme can achieve better performance.

[0060] Figure 8 A comparison is shown between a fixed equilibrium coefficient scheme and a state-adaptive equilibrium coefficient scheme according to one aspect of this disclosure. Figure 8 Graphs (a) and (b) in the figure illustrate the implementations on AWAC and IQL, respectively. In these graphs, "---" represents the adaptive balancing coefficient, "—" represents the fixed balancing coefficient, the horizontal axis shows the values ​​of the fixed balancing coefficient from low to high, the vertical axis shows the normalized D4RL score, and the shaded area represents the 95% CI. It can be seen that the 95% CIs of the adaptive and fixed balancing coefficients do not overlap. This indicates that the adaptive balancing coefficient scheme is superior to all fixed balancing coefficient schemes on both AWAC and IQL.

[0061] The prior description of embodiments of this disclosure is provided to enable those skilled in the art to make or use various embodiments. Various modifications to these embodiments will readily be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the scope of the various embodiments. Therefore, the claims are not intended to limit them to the embodiments shown herein, but are accorded the widest scope consistent with the appended claims and the principles and novel features disclosed herein.

Claims

1. A method for outputting actions from a reinforcement learning module, wherein, The reinforcement learning module includes a balanced component and a general component, and the method includes: The reinforcement learning module receives the state of the environment, the state being based on at least one of an image, video, audio, and text representing the environment; The balancing component generates a balance coefficient based on the received state, the balance coefficient controlling the balance between policy improvement and policy constraints of the general component used to output the action; and The general component outputs the action based on the received state and the generated balance coefficient.

2. The method according to claim 1, wherein, The balancing component is trained based on a convolutional neural network or a multilayer perceptron neural network.

3. The method according to claim 1, wherein, The balanced component and the general component are trained based on the same set of state-action Q functions and value V functions.

4. The method according to claim 1, further comprising: The dimensionality of the balance coefficients is extended by positional encoding, and Specifically, outputting the action includes: outputting the action based on the received state and the balance coefficient having an extended dimension.

5. The method according to claim 1, wherein, The reinforcement learning module is used for autonomous vehicles, the environment is the traffic environment of the autonomous vehicle, and the action is the driving action of the autonomous vehicle.

6. The method according to claim 1, wherein, The reinforcement learning module is used for the robot, the environment is the robot's working environment, and the action is the robot's operating action.

7. An apparatus for outputting actions from a reinforcement learning module, comprising: Memory; as well as At least one processor coupled to the memory and configured to perform the method according to any one of claims 1-6.

8. A computer program product for outputting actions from a reinforcement learning module, comprising: Processor-executable computer code for performing the method according to any one of claims 1-6.

9. A computer-implemented method for training a reinforcement learning module, wherein, The reinforcement learning module includes a balanced component and a general component, and the method includes: The balancing component is trained to learn a mapping from each state of the environment of the reinforcement learning module to balancing coefficients, which control the balance between policy improvement and policy constraints of the general component used for output actions, and the states are based on at least one of images, videos, audio, and text representing the environment; and A family of policies with different balance coefficients are trained for the general component to output actions based on the state and the balance coefficients associated with the state.

10. The method of claim 9, comprising: The offline pre-training phase and the online fine-tuning phase, and In the offline pre-training phase, training a family of policies with different balance coefficients for use in the general component includes: The first small batch is sampled from the offline dataset. Assign random balance coefficients to each sample in the first mini-batch. The general component is updated based on the optimization objective using the first mini-batch and a set of randomly assigned balance coefficients; and In the online fine-tuning phase, training a family of policies with different balance coefficients for use in the general component includes: The second mini-batch was sampled from the online dataset. The balancing component generates a balance coefficient for each sample in the second mini-batch. The general component is updated using the second mini-batch and a set of generated balance coefficients based on the optimization objective.

11. The method according to claim 9, wherein, Training the balancing component includes: The balance component is updated to maximize the value of the state-action Q-function of the reinforcement learning module by utilizing the general component to transform the balance coefficients into actions.

12. The method according to claim 9, wherein, The balancing component is trained based on a convolutional neural network or a multilayer perceptron neural network.

13. The method according to claim 9, wherein, The balanced component and the general component are trained based on the same set of state-action Q functions and value V functions.

14. An apparatus for training a reinforcement learning module, comprising: Memory; as well as At least one processor coupled to the memory and configured to perform the method according to any one of claims 9-13.

15. A computer program product for training a reinforcement learning module, comprising: Processor-executable computer code for performing the method according to any one of claims 9-13.

Images