Dual slight generalization method and device in offline reinforcement learning, equipment and medium

Abstract

The invention relates to the technical field of off-line reinforcement learning, in particular to a dual slight generalization method and device in off-line reinforcement learning, equipment and a medium, and the method comprises the steps: obtaining a target action of an adjacent region of an off-line data set based on a slight generalization strategy; determining the maximum value of slight generalization by using the target action, and fusing the maximum value of slight generalization and the maximum value in the target sample into the Bellman target to obtain an improved Bellman target; and adjusting a value function overestimation amplification effect in off-line reinforcement learning by using an improved Bellman target so as to obtain a dual slight generalization result of the off-line reinforcement learning. Therefore, the problems that the utilization rate of offline data is low and the performance of a reinforcement learning algorithm is reduced due to the fact that the generalization ability of a neural network cannot be utilized in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the technical field of offline reinforcement learning, and in particular to a method, device, equipment and medium for double slight generalization in offline reinforcement learning. Background Art

[0002] Reinforcement learning aims to solve sequential decision-making problems and has received extensive attention in recent years. However, its practical application faces many challenges, such as risky exploration attempts and time-consuming data collection phase. Offline reinforcement learning, as a promising paradigm, alleviates these challenges by learning without interacting with the environment. It eliminates the need for unsafe exploration and facilitates the utilization of existing large-scale datasets.

[0003] Offline reinforcement learning faces the out-of-distribution problem and extrapolation error. From a generalization perspective, this well-known challenge can be viewed as the result of overgeneralization of the value function or policy to out-of-distribution actions. Specifically, due to the complexity of generalization, the overestimation of the potential value function of out-of-distribution actions is often improperly captured by the maximization operation. This overestimation propagates to the value function of in-distribution samples through Bellman updates, and further extends to the value function of out-of-distribution samples through generalization. In this way, the overestimation of the value function will propagate, accumulate, and intensify.

[0004] Therefore, in order to solve the overestimation caused by out-of-distribution actions, a lot of efforts have been invested in the field, and recent progress in in-sample learning has successfully constructed the Bellman objective using only actions present in the dataset and extracted strategies through weighted behavior cloning. However, the inability to utilize the generalization ability of neural networks has led to low offline data utilization, which has reduced the performance of reinforcement learning algorithms and needs to be improved urgently. Summary of the invention

[0005] The present application provides a dual slight generalization method, device, equipment and medium in offline reinforcement learning to solve the problem that the generalization ability of neural networks cannot be utilized in related technologies, resulting in low offline data utilization and reduced performance of reinforcement learning algorithms.

[0006] The first aspect of the present application provides a double slight generalization method in offline reinforcement learning, comprising the following steps: based on a slight generalization strategy, obtaining a target action in a neighboring area of ​​an offline data set; using the target action to determine a slight generalization maximum value, and fusing the slight generalization maximum value and a maximum value in a target sample into a Bellman objective to obtain an improved Bellman objective; using the improved Bellman objective to adjust the overestimation amplification effect of the value function in offline reinforcement learning to obtain a double slight generalization result of the offline reinforcement learning.

[0007] Optionally, in one embodiment of the present application, determining the maximum value of slight generalization using the target action includes: using an actor-critic learning framework to regularize the actions in high-value samples to obtain the maximum value of slight generalization.

[0008] Optionally, in one embodiment of the present application, the slightly generalized maximum value and the maximum value in the target sample are fused into the Bellman target to obtain the improved Bellman target, including: determining the maximum value in the target sample according to the target sample in-learning technology; fusing the slightly generalized maximum value and the maximum value in the target sample into the Bellman target to obtain the improved Bellman target.

[0009] Optionally, in one embodiment of the present application, the slight generalization strategy is expressed as:

[0010]

[0011] and,

[0012] in, is the empirical behavior strategy observed in the offline dataset, s is the state, a is the action, ∈ a is the preset threshold.

[0013] A second aspect of the present application provides a dual slight generalization device in offline reinforcement learning, including: an acquisition module, used to acquire a target action in a neighboring area of ​​an offline data set based on a slight generalization strategy; a determination module, used to determine a slight generalization maximum value using the target action, and fuse the slight generalization maximum value and the maximum value in the target sample into a Bellman target to obtain an improved Bellman target; a control module, used to use the improved Bellman target to adjust the overestimation amplification effect of the value function in the offline reinforcement learning to obtain a dual slight generalization result of the offline reinforcement learning.

[0014] Optionally, in one embodiment of the present application, the determination module includes: an acquisition unit, configured to perform regularization processing on actions in high-value samples using an actor-critic learning framework to obtain the maximum value of the slight generalization.

[0015] Optionally, in one embodiment of the present application, the control module includes: a determination unit, used to determine the maximum value within the target sample according to the target sample learning technology; a control unit, used to fuse the slightly generalized maximum value and the maximum value within the target sample into the Bellman objective to obtain the improved Bellman objective.

[0016] Optionally, in one embodiment of the present application, the slight generalization strategy is expressed as:

[0017]

[0018] and,

[0019] in, is the empirical behavior strategy observed in the offline dataset, s is the state, a is the action, ∈ a is the preset threshold.

[0020] A third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the double slight generalization method in offline reinforcement learning as described in the above embodiment.

[0021] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the above-mentioned double slight generalization method in offline reinforcement learning.

[0022] The fifth aspect embodiment of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above double slight generalization method in offline reinforcement learning.

[0023] The embodiment of the present application can obtain the target action in the vicinity of the offline data set based on the slight generalization strategy, use the target action to determine the maximum value of slight generalization, and merge the maximum value of slight generalization with the maximum value in the target sample into the Bellman target, so as to use the improved Bellman target to adjust the overestimation amplification effect of the value function in offline reinforcement learning, so as to obtain the double slight generalization result of offline reinforcement learning, effectively improve the utilization rate of offline data, and enhance the performance of reinforcement learning algorithm. Thus, the problem that the generalization ability of neural network cannot be utilized in the related technology, resulting in low utilization rate of offline data and reduced performance of reinforcement learning algorithm is solved.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 A flowchart of a double slight generalization method in offline reinforcement learning provided according to an embodiment of the present application;

[0027] Figure 2It is a structural schematic diagram of a double slight generalization device in offline reinforcement learning provided according to an embodiment of the present application;

[0028] Figure 3 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] The following describes the double slight generalization method, device, equipment and medium in offline reinforcement learning of the embodiment of the present application with reference to the accompanying drawings. In view of the problem that the generalization ability of the neural network cannot be utilized in the related technology mentioned in the background technology center, resulting in low utilization of offline data and reduced performance of the reinforcement learning algorithm, the present application provides a double slight generalization method in offline reinforcement learning, in which the target action in the adjacent area of ​​the offline data set can be obtained based on the slight generalization strategy, the maximum value of slight generalization can be determined by using the target action, and the maximum value of slight generalization and the maximum value in the target sample can be fused into the Bellman target, so as to adjust the overestimation amplification effect of the value function in offline reinforcement learning by using the improved Bellman target, so as to obtain the double slight generalization result of offline reinforcement learning, effectively improve the utilization of offline data and enhance the performance of the reinforcement learning algorithm. Thus, the problem that the generalization ability of the neural network cannot be utilized in the related technology, resulting in low utilization of offline data and reduced performance of the reinforcement learning algorithm is solved.

[0031] Specifically, Figure 1 A schematic diagram of a flow chart of a double slight generalization method in offline reinforcement learning provided in an embodiment of the present application.

[0032] like Figure 1 As shown in FIG. 1 , the dual slight generalization method in offline reinforcement learning includes the following steps:

[0033] In step S101, based on a slight generalization strategy, a target action in a neighborhood of an offline dataset is obtained.

[0034] It is understandable that the embodiment of the present application can obtain the target action in the neighborhood of the offline data set based on the slight generalization strategy. For example, because under certain continuity conditions, the Q function can generalize well and approximate the true Bellman update in the neighborhood of the sample in the offline data set, this means that slight generalization outside the offline data set can be used to potentially pursue better performance. Therefore, the embodiment of the present application defines a slight generalization strategy If the slightly generalized strategy A strategy is called a slightly generalized strategy if the following conditions are met:

[0035]

[0036] and,

[0037] in, is the empirical behavior strategy observed in the offline dataset, s is the state, a is the action, ∈ a is the preset threshold.

[0038] Therefore, the embodiment of the present application generalizes the action slightly, that is, based on the slight generalization strategy exist In addition to generalization, The support set range ratio is much wider, and when measured in terms of action space distance, Generalization outside the dataset is ∈ a Limit, the Q function is in this slightly generalized region There is a high chance of generalizing correctly, which effectively improves the feasibility of properly utilizing generalization in offline reinforcement learning.

[0039] In step S102, the slightly generalized maximum value is determined using the target action, and the slightly generalized maximum value and the maximum value in the target sample are fused into the Bellman objective to obtain an improved Bellman objective.

[0040] It can be understood that the embodiments of the present application can use the target action to determine the maximum value of slight generalization. For example, the embodiments of the present application can select actions in the neighboring area of ​​the data set to maximize the Q value in the first part of the Bellman objective to obtain the maximum value of slight generalization, and fuse the maximum value of slight generalization with the maximum value in the target sample into the Bellman objective to obtain an improved Bellman objective. That is, the embodiments of the present application reduce the propagation of generalization while retaining the propagation of reinforcement learning signals by combining the maximum value of slight generalization with the maximum value in the sample without hindering the propagation of RL learning signals.

[0041] In one embodiment of the present application, determining the maximum value of slight generalization using the target action includes: using an actor-critic learning framework to regularize the actions in high-value samples to obtain the maximum value of slight generalization.

[0042] In the actual implementation process, the embodiments of the present application can use the actor-critic learning framework to regularize actions in high-value samples. For example, in an offline data set, actions with higher value, such as actions that have performed well in past experience or are likely to bring greater benefits under the current strategy, are regularized to obtain the maximum value of slight generalization, thereby significantly improving the performance of the offline reinforcement learning algorithm.

[0043] In step S103, the improved Bellman objective is used to adjust the overestimation amplification effect of the value function in the offline reinforcement learning to obtain a double slight generalization result of the offline reinforcement learning.

[0044] It can be understood that the embodiments of the present application can use the improved Bellman objective to adjust the overestimation amplification effect of the value function in offline reinforcement learning, reduce the generalization propagation, and reduce the discount factor of the generalization propagation, thereby reducing the overestimation amplification effect of the value function caused by bootstrap in reinforcement learning, so as to obtain a double slight generalization result of offline reinforcement learning and improve the performance of the offline reinforcement learning algorithm.

[0045] For example, since the learned value function will produce a certain degree of generalization error in this slightly generalized region, the possible erroneous generalization will still be propagated and aggravated through the Bellman update. For this reason, the embodiment of the present application can introduce an additional slightly generalized level, called slightly generalized propagation, and propose a novel DMG (Doubly MildGeneralization) operator As shown below:

[0046]

[0047] in, is the empirical behavior strategy observed in the offline dataset, is a slightly generalized strategy, Q is the Q function, s is the state, a is the action, R is the reward function, γ is the discount factor, P is the state transfer function, s ′ is the next state, and λ∈(0,1) is the parameter to balance the two terms.

[0048] In a typical offline reinforcement learning algorithm, the extrapolation error and value overestimation caused by incorrect generalization are propagated through Bellman update, and the discount factor of this process is γ. The embodiment of the present application can reduce this discount factor to λγ, thereby reducing the amplification effect of the overestimation of the value function. On the other hand, compared with the in-sample method, the double slight generalization in the embodiment of the present application allows slight generalization, and uses the generalization ability of the neural network to pursue better performance.

[0049] For another example, the offline reinforcement learning algorithm in the embodiment of the present application includes the following network: strategy network πφ 、Target strategy network π φ′ ,Q NetworkQ θ 、Target Q Network Q θ′ and V Network ψ .

[0050] Among them, the embodiment of the present application can perform strategy learning. In fact, the embodiment of the present application expects that DMG will show a slight generalization tendency for good behavior actions in the data set. To this end, first consider the empirical behavior strategy Reshape it to favor actions with high advantage function values, that is:

[0051]

[0052] Then, the trained strategy is kept close to the reshaped behavior strategy to limit the generalization area. The embodiment of the present application maximizes the following objective function:

[0053]

[0054] in, is the data set, s is the state, a is the action, and v is the penalty coefficient.

[0055] Through simple derivation, the above formula is equivalent to the following strategy training objective:

[0056]

[0057] Where α is the inverse temperature coefficient, Q θ′ (s,a)-V ψ (s) Calculate the advantage function A(s,a).

[0058] Furthermore, the embodiment of the present application can perform value function learning. The embodiment of the present application introduces the above strategy to In Replace with for In Any in-sample learning technique can be used to calculate the in-sample maximum. In particular, based on the reinforcement learning algorithm, the embodiment of the present application performs expected quantile regression:

[0059]

[0060] in, and τ∈(0,1). For τ≈1, V ψ It can capture the maximum Q value in the sample. Finally, the following value training loss function is obtained:

[0061]

[0062] In some embodiments, embodiments of the present application provide a comprehensive analysis of DMG in predicted and worst-case generalization scenarios, with a particular focus on value function estimation and performance. Under predicted generalization, DMG is guaranteed to have better performance than the sample optimal strategy in the dataset. Even in the worst-case generalization, DMG can still limit the overestimation of the value function and guarantee the output of a safe strategy with a performance lower bound.

[0063] Optionally, in one embodiment of the present application, the slightly generalized maximum value and the maximum value in the target sample are fused into the Bellman objective to obtain an improved Bellman objective, including: determining the maximum value in the target sample based on the target sample in-sample learning technology; fusing the slightly generalized maximum value and the maximum value in the target sample into the Bellman objective to obtain an improved Bellman objective.

[0064] As a possible implementation method, the embodiment of the present application can determine the maximum value within the target sample based on the target sample learning technology, such as expected quantile regression to determine the maximum value within the sample, and merge the slightly generalized maximum value with the maximum value within the target sample into the Bellman objective to obtain an improved Bellman objective, which takes into account both the advantages brought by the slightly generalized strategy and the information within the offline data set, thereby obtaining a more accurate strategy estimate and reducing the risk of overestimation of the value function.

[0065] It should be noted that the embodiment of the present application can use double slight generalization for offline reinforcement learning, which includes slight action generalization and slight generalization propagation. In theory: under the predicted generalization, double slight generalization guarantees better performance than the optimal in-sample policy in the data set. Even in the worst-case generalization, double slight generalization can still limit the overestimation of the value function and ensure that the output is a safe policy with a performance lower bound.

[0066] Experimentally: Double Slight Generalization achieves state-of-the-art performance on standard offline reinforcement learning benchmarks, including the Gym-MuJoCo locomotion task and the challenging AntMaze task. In addition, thanks to its flexibility in generalization, Double Slight Generalization can seamlessly transition from offline learning to online learning and obtain superior online fine-tuning performance.

[0067] According to the double slight generalization method in offline reinforcement learning proposed in the embodiment of the present application, the target action in the adjacent area of ​​the offline data set can be obtained based on the slight generalization strategy, the maximum value of slight generalization can be determined by using the target action, and the maximum value of slight generalization and the maximum value in the target sample can be merged into the Bellman target, so as to use the improved Bellman target to adjust the overestimation amplification effect of the value function in offline reinforcement learning, so as to obtain the double slight generalization result of offline reinforcement learning, effectively improve the utilization rate of offline data, and enhance the performance of reinforcement learning algorithm. Thus, the problem that the generalization ability of neural network cannot be utilized in the related technology, resulting in low utilization rate of offline data and reduced performance of reinforcement learning algorithm is solved.

[0068] Next, a dual slight generalization device in offline reinforcement learning proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0069] Figure 2 is a block diagram of a double slight generalization device in offline reinforcement learning according to an embodiment of the present application.

[0070] like Figure 2 As shown, the dual slight generalization device 10 in offline reinforcement learning includes: an acquisition module 100 , a determination module 200 and a control module 300 .

[0071] Specifically, the acquisition module 100 is used to acquire the target action in the neighborhood of the offline data set based on a slight generalization strategy.

[0072] The determination module 200 is used to determine the maximum value of slight generalization by using the target action, and fuse the maximum value of slight generalization and the maximum value in the target sample into the Bellman objective to obtain the improved Bellman objective.

[0073] The control module 300 is used to adjust the overestimation amplification effect of the value function in offline reinforcement learning by using the improved Bellman objective to obtain a double slight generalization result of offline reinforcement learning.

[0074] Optionally, in one embodiment of the present application, the determination module 200 includes: an acquisition unit.

[0075] Among them, the acquisition unit is used to regularize high-value in-sample actions using the actor-critic learning framework to obtain the maximum value of slight generalization.

[0076] Optionally, in one embodiment of the present application, the control module 300 includes: a determination unit and a control unit.

[0077] The determination unit is used to determine the maximum value within the target sample according to the target sample learning technology.

[0078] A control unit is used to fuse the slightly generalized maximum value and the maximum value within the target sample into the Bellman objective to obtain an improved Bellman objective.

[0079] Optionally, in one embodiment of the present application, the slight generalization strategy is expressed as:

[0080]

[0081] and,

[0082] in, is the empirical behavior strategy observed in the offline dataset, s is the state, a is the action, ∈ a is the preset threshold.

[0083] It should be noted that the above explanation of the embodiment of the double slight generalization method in offline reinforcement learning is also applicable to the double slight generalization device in offline reinforcement learning in this embodiment, which will not be repeated here.

[0084] According to the double slight generalization device in offline reinforcement learning proposed in the embodiment of the present application, the target action in the vicinity of the offline data set can be obtained based on the slight generalization strategy, the maximum value of slight generalization can be determined by using the target action, and the maximum value of slight generalization and the maximum value in the target sample can be merged into the Bellman target, so as to use the improved Bellman target to adjust the overestimation amplification effect of the value function in offline reinforcement learning, so as to obtain the double slight generalization result of offline reinforcement learning, effectively improve the utilization rate of offline data, and enhance the performance of reinforcement learning algorithm. Thus, the problem that the generalization ability of neural network cannot be utilized in the related technology, resulting in low utilization rate of offline data and reduced performance of reinforcement learning algorithm is solved.

[0085] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0086] A memory 301 , a processor 302 , and a computer program stored in the memory 301 and executable on the processor 302 .

[0087] When the processor 302 executes the program, the double slight generalization method in offline reinforcement learning provided in the above embodiment is implemented.

[0088] Furthermore, the electronic device further comprises:

[0089] The communication interface 303 is used for communication between the memory 301 and the processor 302 .

[0090] The memory 301 is used to store computer programs that can be run on the processor 302 .

[0091] The memory 301 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0092] If the memory 301, the processor 302 and the communication interface 303 are implemented independently, the communication interface 303, the memory 301 and the processor 302 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0093] Optionally, in a specific implementation, if the memory 301, the processor 302 and the communication interface 303 are integrated on a chip, the memory 301, the processor 302 and the communication interface 303 can communicate with each other through an internal interface.

[0094] The processor 302 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0095] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the double slight generalization method in offline reinforcement learning as described above is implemented.

[0096] This embodiment also provides a computer program product, including a computer program, which is used to implement the above double slight generalization method in offline reinforcement learning when the computer program is executed.

[0097] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0098] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0099] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0100] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0101] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0102] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0103] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0104] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A double slight generalization method in offline reinforcement learning, characterized in that The following steps are involved: Based on the slight generalization strategy, the target action in the neighboring area of ​​the offline dataset is obtained; Determine a slightly generalized maximum value by using the target action, and fuse the slightly generalized maximum value with the maximum value in the target sample into a Bellman objective to obtain an improved Bellman objective; The improved Bellman objective is used to adjust the overestimation amplification effect of the value function in offline reinforcement learning to obtain a double slight generalization result of the offline reinforcement learning.

2. The method according to claim 1, characterized in that The determining a maximum value of slight generalization using the target action includes: An actor-critic learning framework is used to regularize high-value in-sample actions to maximize the slight generalization.

3. The method according to claim 1, characterized in that The step of fusing the slightly generalized maximum value and the maximum value in the target sample into the Bellman objective to obtain an improved Bellman objective includes: Determine the maximum value within the target sample according to the target sample learning technique; The slightly generalized maximum value and the target sample maximum value are fused into the Bellman objective to obtain the improved Bellman objective.

4. The method according to claim 1, characterized in that: The slight generalization strategy is expressed as: and, in, is the empirical behavior strategy observed in the offline dataset, s is the state, a is the action, ∈ a is the preset threshold.

5. A double slight generalization device in offline reinforcement learning, characterized in that include: The acquisition module is used to obtain the target action in the neighboring area of ​​the offline dataset based on a slight generalization strategy; A determination module, configured to determine a slightly generalized maximum value using the target action, and fuse the slightly generalized maximum value with a maximum value within the target sample into a Bellman objective to obtain an improved Bellman objective; A control module is used to adjust the overestimation amplification effect of the value function in offline reinforcement learning by using the improved Bellman objective to obtain a double slight generalization result of the offline reinforcement learning.

6. The device according to claim 5, characterized in that The determination module comprises: An acquisition unit is used to regularize high-value in-sample actions using an actor-critic learning framework to maximize the slight generalization.

7. The device according to claim 5, characterized in that The control module comprises: A determination unit, configured to determine the maximum value within the target sample according to the target sample learning technique; A control unit is used to fuse the slightly generalized maximum value and the maximum value in the target sample into the Bellman objective to obtain the improved Bellman objective.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the double slight generalization method in offline reinforcement learning according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the double slight generalization method in offline reinforcement learning as claimed in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed by a processor to implement the double slight generalization method in offline reinforcement learning as claimed in any one of claims 1 to 4.

Images