Online adaptive methods, systems, apparatus, and media for offline reinforcement learning

Abstract

The application discloses an online adaptive method, system and device for offline reinforcement learning and a medium, and belongs to the field of offline reinforcement learning strategy deployment. The method comprises the following steps: obtaining online state data s and storing the online state data s in a buffer; calculating the action entropy of the online state data s after offline strategy and screening out confidence samples; fine-tuning preset parameters in the offline strategy by using the confidence samples to adapt to the change of online environment distribution; using the minimum action entropy as a loss function to update the offline strategy parameters in a reverse propagation manner, so that the offline strategy can adapt to the offset of various online environment distributions; and using KL divergence to constrain the offline strategy to avoid that the parameter update amplitude is too large, so as to prevent model collapse. In the test stage, the offline reinforcement learning strategy is further fine-tuned by using the confidence state data s to adapt to the change of online environment distribution. In addition, the offline strategy is constrained by using the KL divergence to avoid that the parameter update amplitude is too large.

Description

Technical Field

[0001] This invention relates to the field of offline reinforcement learning strategy deployment, and in particular to an online adaptive method, system, device, and medium for offline reinforcement learning. Background Technology

[0002] In reality, due to difficulties in sample collection or high costs of interaction simulation, enabling robots (agents) to interact with the environment and acquire data in real time is extremely challenging. Therefore, offline reinforcement learning methods utilize pre-collected offline data for policy learning and then deploy the trained policy to real-world application scenarios. For example, in scenarios with high safety requirements, such as autonomous driving, a batch of data is typically collected offline to train a better autonomous driving policy before deploying this policy in an online environment.

[0003] The offline policy may not adapt well to the online environment due to potential distribution shifts between the offline dataset and the online environment. This distribution shift occurs because the offline dataset is collected using policies with unknown behaviors, and it cannot guarantee that the collected state-action pairs (s, a) will cover all possibilities in the state-action space. During testing, the offline policy may give poor evaluations when faced with unseen state-action pairs (s, a).

[0004] To address this issue, existing methods are mainly divided into two categories: conservative estimation and uncertainty assessment methods. The core idea of ​​conservative estimation methods is to add KL divergence constraints or policy value penalties to make the trained policy biased towards selecting (s,a) that already exist in the dataset, thus avoiding overestimation of unseen (s,a). Examples of conservative estimation methods include Conservative Q-Learning (CQL), TD3+BC, and Mildly Conservative Q-Learning (MCQL). ​​The core idea of ​​uncertainty assessment methods is to measure the uncertainty of (s,a) using a specific metric, quantifying the uncertainty of the assessment result to weigh the risks and benefits brought by unseen (s,a). Examples of uncertainty assessment methods include Uncertaintyweighted Actor Critic (UWAC) and Ensemble-Diversified Actor Critic (EDAC).

[0005] However, regardless of whether it is a conservative estimation method or an uncertain estimation method, they only use training data to adjust the model policy to become conservative or to impose uncertain penalties on the policy, without directly using the test state data s to further fine-tune the offline policy. Summary of the Invention

[0006] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide an online adaptive method, system, device and medium for offline reinforcement learning.

[0007] The technical solution adopted in this invention is:

[0008] An online adaptive method for offline reinforcement learning includes the following steps:

[0009] Acquire online status data (such as RGB images) and store it in a replay buffer;

[0010] Calculate the action entropy of online state data s after passing through offline policies (such as autonomous driving policies), and determine whether the action entropy is a confidence sample;

[0011] Confidence samples are used to fine-tune the preset parameters in the offline strategy to adapt to changes in the distribution of the online environment, such as the parameters of the LayerNorm layer;

[0012] The offline policy parameters are updated by backpropagating by minimizing the action entropy as the loss function, so that the offline policy can adapt to the offset of various online environment distributions.

[0013] KL divergence is used to constrain the offline strategy to avoid excessive parameter updates and prevent model collapse.

[0014] Furthermore, the step of acquiring online status data s and storing it in a buffer includes:

[0015] The robot (agent) interacts with the online environment, acquires state data s, and stores the state data s in a buffer (replay buffer);

[0016] For autonomous driving and gaming scenarios, the state data s is image data; for Sim-to-Real and offline energy strategy migration scenarios, the state data s is vector data.

[0017] Furthermore, the calculation of the online state data s involves processing the action entropy of the offline strategy and filtering out confidence samples, including:

[0018] Calculate the action entropy of online state data s after applying offline strategies;

[0019] For discrete control tasks, state data s less than a preset threshold are considered confidence samples; for continuous control tasks, samples with the smallest threshold in the buffer (topk) are considered confidence samples.

[0020] Furthermore, for discrete control tasks, the following formula is used to filter the state data s:

[0021]

[0022] In the formula, Θ represents the offline strategy parameter to be fine-tuned. It is an indicator function, which is 1 when the condition is met and 0 otherwise. E0=α*lnA is the confidence sample screening threshold. Samples less than the threshold are confidence samples, and samples greater than the threshold are discarded.

[0023] For continuous control tasks, the following formula is used to filter the state data s:

[0024]

[0025] In the formula, E′0 represents the k-th smallest entropy value in the buffer, and the offline strategy is updated by filtering out the top k smallest state data in the buffer.

[0026] Furthermore, the preset parameters are either affine parameters in the LayerNorm layer or parameters of the last layer.

[0027] Furthermore, the step of fine-tuning the preset parameters in the offline strategy using confidence samples includes:

[0028] For discrete control tasks, the following optimization objective is used to fine-tune the offline strategy:

[0029]

[0030] In the formula, a represents the action, A represents the action space, and π Θ This indicates the offline strategy that needs to be fine-tuned;

[0031] Based on the selected confidence samples, the following optimization objectives are formed:

[0032]

[0033] For continuous control tasks, the following optimization objectives are used to fine-tune the offline strategy:

[0034]

[0035] In the formula, σ represents the variance of continuous actions;

[0036] Based on the selected confidence samples, the following optimization objectives are formed:

[0037]

[0038] By minimizing the unsupervised optimization objective, the affine parameters in the LayerNorm layer or the parameters of the last layer in the offline strategy are fine-tuned; the expression for the unsupervised optimization objective is:

[0039]

[0040] Furthermore, the use of KL divergence to constrain the offline strategy includes:

[0041] Add a parameter to freeze the strategy π base Let the fine-tuned strategy π off and π base Add a KL-Divergence loss function between them, as shown in the following formula:

[0042]

[0043] The overall loss function is as follows:

[0044]

[0045] In the formula, λ is the weight.

[0046] Another technical solution adopted in this invention is:

[0047] An online adaptive system for offline reinforcement learning includes:

[0048] The data acquisition module is used to acquire online status data s and store it in a buffer;

[0049] The entropy calculation module is used to calculate the action entropy of online state data s after passing through offline strategies and to filter out confidence samples.

[0050] The parameter fine-tuning module is used to fine-tune the preset parameters in the offline strategy using confidence samples to adapt to changes in the distribution of the online environment;

[0051] The parameter update module is used to backpropagate and update the offline policy parameters using the minimized action entropy as the loss function, so that the offline policy can adapt to the offset of various online environment distributions.

[0052] The amplitude limiting module is used to constrain offline strategies using KL divergence to prevent excessively large parameter update amplitudes and thus prevent model collapse.

[0053] Another technical solution adopted in this invention is:

[0054] An online adaptive device for offline reinforcement learning, comprising:

[0055] At least one processor;

[0056] At least one memory for storing at least one program;

[0057] When the at least one program is executed by the at least one processor, the at least one processor performs the method as described above.

[0058] Another technical solution adopted in this invention is:

[0059] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0060] The beneficial effects of this invention are: during the testing phase, the confidence state data s is used to further fine-tune the offline reinforcement learning strategy to adapt to changes in the online environment distribution. Furthermore, KL divergence is used to constrain the offline strategy, preventing excessively large parameter update amplitudes. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart of the steps of an online adaptive method for offline reinforcement learning distribution shift in an embodiment of the present invention;

[0063] Figure 2 This is a framework diagram of the online adaptive algorithm for offline reinforcement learning strategy in an embodiment of the present invention;

[0064] Figure 3 This is a comparison diagram of action distribution in an embodiment of the present invention. Detailed Implementation

[0065] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0066] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0067] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0068] Furthermore, in the description of this invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0069] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0070] To address the existing technical problems, this invention proposes an online adaptive reinforcement learning algorithm (TARL). During the testing phase, the algorithm uses the test state s to further fine-tune the LayerNorm (LN) layer parameters in the offline strategy to adapt to out-of-distribution data, thereby solving the distribution shift problem.

[0071] like Figure 1 and Figure 2 As shown, this embodiment provides an online adaptive method for distribution shift in offline reinforcement learning, specifically including the following steps:

[0072] S1. Obtain online status data s and store it in the buffer.

[0073] Offline strategies (such as autonomous driving strategies) are deployed in the online environment. The robot (agent) interacts with the online environment to obtain state data s, which is then placed in a replay buffer. This state data s is typically an image for autonomous driving and gaming scenarios; however, it is a vector for Sim-to-Real and offline energy strategy migration scenarios.

[0074] S2. Calculate the action entropy of the online state data s after passing through the offline strategy, and filter out the confidence samples.

[0075] Calculate the output entropy of the state data s after passing through the offline policy. For discrete control tasks, state data s below a certain threshold are considered confidence samples; for continuous control tasks, samples with the top k smallest threshold values ​​in the buffer are considered confidence samples.

[0076] For discrete control tasks, the following formula is used to filter the state data s:

[0077]

[0078] In formula (1), Θ represents the offline strategy parameter to be fine-tuned. E0 is an indicator function, which is 1 when the condition is met and 0 otherwise. E0 = α * lnA is the confidence sample screening threshold. Samples less than the threshold are confidence samples, and samples greater than the threshold are discarded. A is the number of action categories.

[0079] For continuous control tasks, the following formula is used to filter the state data s:

[0080]

[0081] In formula (2), E′0 represents the kth smallest entropy value in the buffer. The offline strategy is updated by filtering out the top k smallest state data in the buffer.

[0082] S3. Use confidence samples to fine-tune the preset parameters in the offline strategy to adapt to changes in the distribution of the online environment.

[0083] The selected confidence samples are used to further fine-tune the model strategy to adapt to changes in the online environment distribution. To adjust the offline strategy more efficiently, typically only the parameters of the LayerNorm layer or the last layer in the offline strategy are adjusted.

[0084] For discrete control tasks, the following optimization objective is used to fine-tune the offline strategy:

[0085] min Θ E discrete (s;Θ)= -∑ a∈A π Θ (a|s)logπ Θ (a|s) (3)

[0086] Here, 'a' represents the action, 'A' represents the action space, and 'π' represents the action space. Θ This indicates the offline strategy to be fine-tuned. By minimizing the unsupervised optimization objective, the affine parameters in the LayerNorm layer or the parameters of the last layer in the offline strategy can be fine-tuned.

[0087] Based on the selected confidence samples, the following optimization objectives are formed:

[0088]

[0089] For continuous control tasks, the following optimization objectives are used to fine-tune the offline strategy:

[0090]

[0091] In the formula, σ represents the variance of continuous actions;

[0092] Based on the selected confidence samples, the following optimization objectives are formed:

[0093]

[0094] In summary, unsupervised optimization objectives can be summarized as follows:

[0095]

[0096] S4. Use the minimization of action entropy as the loss function to backpropagate and update the offline policy parameters, so that the offline policy can adapt to the offset of various online environment distributions.

[0097] Minimizing action entropy is used as the loss function to backpropagate and update the offline policy parameters. Since environmental feedback information (reward) is not available during testing, minimizing action entropy as the loss function is used to backpropagate and update the offline policy, making the policy more confident in the output of the state and further increasing the probability of confident actions.

[0098] Experiments revealed that simply minimizing action entropy leads to model policy collapse, meaning the output will no longer be a specific category, such as... Figure 3 As shown, Figure 3 (a) is a schematic diagram of the action output of the model policy without fine-tuning, which shows that the action output is relatively uniform. Figure 3 (b) is the action output distribution diagram after minimizing the action entropy of the model. It can be seen that the action output only outputs category 3.

[0099] S5. Use KL divergence to constrain the offline strategy to avoid excessive parameter updates and prevent model collapse.

[0100] To limit the magnitude of offline policy updates, KL divergence was added to constrain parameter changes from becoming too drastic; that is, a policy π with frozen parameters was added. base Let the fine-tuned strategy π off and π base Add a KL-Divergence loss function between them, as shown in the following formula:

[0101]

[0102] The overall loss function is as follows:

[0103]

[0104] Here, λ represents a trade-off between policy updates and policy constraints, resulting in better offline policy fine-tuning. λ should be chosen to a suitable value; too large a value will cause the offline policy to be biased towards not fine-tuning, while too small a value will lead to excessively large policy update magnitudes.

[0105] In summary, compared with the prior art, the present invention has at least the following advantages and beneficial effects:

[0106] (1) The present invention can fine-tune the offline strategy without the need for environmental feedback signals (reward).

[0107] (2) This invention only requires a small amount of online status data to update the offline strategy, unlike the domainadaptation method which requires a large number of test samples to fine-tune the strategy.

[0108] (3) The present invention has a low fine-tuning cost, and only some layer parameters in the offline strategy need to be updated to adapt to the offset of the online environment distribution.

[0109] This embodiment also provides an online adaptive system for offline reinforcement learning, including:

[0110] The data acquisition module is used to acquire online status data s and store it in a buffer;

[0111] The entropy calculation module is used to calculate the action entropy of online state data s after passing through offline strategies and to filter out confidence samples.

[0112] The parameter fine-tuning module is used to fine-tune the preset parameters in the offline strategy using confidence samples to adapt to changes in the distribution of the online environment;

[0113] The parameter update module is used to backpropagate and update the offline policy parameters using the minimized action entropy as the loss function, so that the offline policy can adapt to the offset of various online environment distributions.

[0114] The amplitude limiting module is used to constrain offline strategies using KL divergence to prevent excessively large parameter update amplitudes and thus prevent model collapse.

[0115] This embodiment provides an online adaptive system for offline reinforcement learning, which can execute the online adaptive method for offline reinforcement learning provided in the method embodiment of the present invention. It can execute any combination of implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0116] This embodiment also provides an online adaptive device for offline reinforcement learning, comprising:

[0117] At least one processor;

[0118] At least one memory for storing at least one program;

[0119] When the at least one program is executed by the at least one processor, the at least one processor performs the following: Figure 1 The method shown.

[0120] This embodiment provides an online adaptive device for offline reinforcement learning, which can execute an online adaptive method for offline reinforcement learning provided in the method embodiment of the present invention. It can execute any combination of implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0121] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0122] This embodiment also provides a storage medium storing instructions or programs that can execute the online adaptive method for offline reinforcement learning provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0123] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0124] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0125] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0127] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0128] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0129] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0130] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0131] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An online adaptive method for offline reinforcement learning, characterized in that, Includes the following steps: Acquire online status data s and store it in a buffer; For autonomous driving and gaming scenarios, the state data s is image data; Calculate the action entropy of online state data s after applying the offline strategy, and then filter out confidence samples; The preset parameters in the offline strategy are fine-tuned using confidence samples to adapt to changes in the distribution of the online environment; the preset parameters are affine parameters in the LayerNorm layer or parameters of the last layer. The offline policy parameters are updated by backpropagating by minimizing the action entropy as the loss function, so that the offline policy can adapt to the offset of various online environment distributions. KL divergence is used to constrain the offline strategy to avoid excessive parameter updates and prevent model collapse. For discrete control tasks, the following formula is used to filter the state data s: In the formula, This indicates the offline strategy parameters that need to be fine-tuned. It is an indicator function; it returns 1 when the condition is met and 0 otherwise. This is the confidence sample selection threshold. Samples below this threshold are considered confidence samples, while samples above this threshold are discarded. For continuous control tasks, the following formula is used to filter the state data s: In the formula, This represents the k-th smallest entropy value in the buffer. The offline strategy is updated by filtering out the top k smallest state data from the buffer.

2. The online adaptive method for offline reinforcement learning according to claim 1, characterized in that, The method of fine-tuning preset parameters in the offline strategy using confidence samples includes: For discrete control tasks, the following optimization objective is used to fine-tune the offline strategy: In the formula, Indicates an action, Represents the action space. This indicates the offline strategy that needs to be fine-tuned; Based on the selected confidence samples, the following optimization objectives are formed: For continuous control tasks, the following optimization objectives are used to fine-tune the offline strategy: In the formula, Represents the variance of continuous actions; Based on the selected confidence samples, the following optimization objectives are formed: By minimizing the unsupervised optimization objective, the affine parameters in the LayerNorm layer or the parameters of the last layer in the offline strategy are fine-tuned; the expression for the unsupervised optimization objective is:

3. The online adaptive method for offline reinforcement learning according to claim 1, characterized in that, The use of KL divergence to constrain offline strategies includes: Add a parameter to the freeze strategy Let the fine-tuned strategy and Add a KL-Divergence loss function between them, as shown in the following formula: The overall loss function is as follows: In the formula, As weight.

4. An online adaptive system for offline reinforcement learning, characterized in that, include: The data acquisition module is used to acquire online status data s and store it in a buffer; For autonomous driving and gaming scenarios, the state data s is image data; The entropy calculation module is used to calculate the action entropy of online state data s after passing through offline strategies and to filter out confidence samples. The parameter fine-tuning module is used to fine-tune the preset parameters in the offline strategy using confidence samples to adapt to changes in the distribution of the online environment; the preset parameters are the affine parameters in the LayerNorm layer or the parameters of the last layer; The parameter update module is used to backpropagate and update the offline policy parameters using the minimized action entropy as the loss function, so that the offline policy can adapt to the offset of various online environment distributions. The amplitude limiting module is used to constrain offline strategies using KL divergence to avoid excessively large parameter update amplitudes and prevent model collapse. For discrete control tasks, the following formula is used to filter the state data s: In the formula, This indicates the offline strategy parameters that need to be fine-tuned. It is an indicator function; it returns 1 when the condition is met and 0 otherwise. This is the confidence sample selection threshold. Samples below this threshold are considered confidence samples, while samples above this threshold are discarded. For continuous control tasks, the following formula is used to filter the state data s: In the formula, This represents the k-th smallest entropy value in the buffer. The offline strategy is updated by filtering out the top k smallest state data from the buffer.

5. An online adaptive device for offline reinforcement learning, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-3.

6. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-3.

Images