Reinforcement learning method, device, and system using offline data

Abstract

One embodiment of the present disclosure provides a reinforcement learning system comprising one or more processors, and one or more memories collectively storing instructions that, when executed by the one or more processors, cause the system to perform operations for performing offline reinforcement learning, and performing online reinforcement learning, wherein the operation for performing the offline reinforcement learning includes the operations for identifying a data retention area and a data non-retention area, and reducing a Q-value estimated in the data non-retention area.

Description

Reinforcement learning methods, devices, and systems using offline data

[0001] One embodiment of the present disclosure aims to provide a reinforcement learning method, apparatus, and system using offline data. More specifically, one embodiment of the present disclosure aims to provide a method for appropriately adjusting Q-values ​​without overestimating them for out-of-distribution spaces where data is not yet available when performing reinforcement learning using offline data.

[0002] Reinforcement learning is an artificial intelligence learning method in which an agent learns how to make decisions by interacting with its environment, primarily used in applications such as robot control and autonomous driving. Reinforcement learning can encompass both online and offline reinforcement learning. Online reinforcement learning is a learning method where the agent directly interacts with the environment to collect data. In contrast, offline reinforcement learning is a method where the agent does not directly interact with the environment; instead, a separate behavioral algorithm exists to learn policies based on pre-collected fixed data without interaction with the environment. While offline reinforcement learning offers the advantage of enabling learning without risk in real-world environments, such as in robots and autonomous driving, it suffers from a decline in reasoning ability in situations outside of the pre-collected fixed data. Consequently, research is currently being conducted on methods that utilize both online and offline reinforcement learning for training.

[0003] One embodiment of the present disclosure aims to provide a method for preventing Q-value overestimation in offline reinforcement learning and online reinforcement learning.

[0004] A reinforcement learning system according to one embodiment of the present disclosure may include one or more processors; and one or more memories that collectively store instructions that cause the system to perform operations when executed by the one or more processors. The operations include: an operation of performing offline reinforcement learning; and an operation of performing online reinforcement learning, wherein the operation of performing offline reinforcement learning may include an operation of identifying a data-possessing region and a data-non-possessing region; and an operation of decreasing an estimated Q-value for the data-non-possessing region.

[0005] In one embodiment, the operation of reducing the estimated Q-value for the data-unavailable region comprises a constant c greater than 1 in the reward function used in the offline reinforcement learning. reward It may include an operation to reduce the above Q-value using compensation scaling that multiplies by .

[0006] In one embodiment, the operation of reducing the estimated Q-value for the data-unavailable region comprises a constant c greater than 1 in the reward function used in the offline reinforcement learning. reward It may include an operation to perform reward scaling by multiplying; an operation to perform hierarchical normalization using the reward scaled reward as input; and an operation to learn a critique ensemble composed of a plurality of critique networks that have undergone hierarchical normalization.

[0007] In one embodiment, the operation of performing the online reinforcement learning comprises the constant c in the reward function of the replay buffer used in the online reinforcement learning. reward It may include an operation to perform online fine-tuning using compensation scaling that multiplies by .

[0008] In one embodiment, the constant c reward It can be set to a value of 10 or more.

[0009] In one embodiment, the operation of reducing the estimated Q-value for the data-non-possessed area may include a penalty-imposing operation of setting the Q-value for the data-non-possessed area to be less than or equal to a preset value.

[0010] In one embodiment, the penalty imposition operation includes: an operation to calculate a penalty loss; an operation to calculate a Temporal-Difference (TD) loss; and an operation to determine a first loss based on the penalty loss and the TD loss, wherein the operations may include an operation to perform at least one of the offline reinforcement learning and the online reinforcement learning based on the first loss.

[0011] In one embodiment, the first loss may be determined by adding a value obtained by multiplying the penalty loss by a weight to the TD loss.

[0012] A reinforcement learning method performed by at least one processor according to one embodiment of the present disclosure comprises: an operation of performing offline reinforcement learning; and an operation of performing online reinforcement learning, wherein the operation of performing offline reinforcement learning may include: an operation of identifying a data-possessing region and a data-non-possessing region; and an operation of reducing a Q-value estimated in the data-non-possessing region.

[0013] In one embodiment, the operation of reducing the Q-value estimated in the data-free region comprises a constant c greater than 1 in the reward function used in the offline reinforcement learning. reward It may include an operation to reduce the above Q-value using compensation scaling that multiplies by .

[0014] In one embodiment, the operation of reducing the Q-value estimated in the data-free region comprises a constant c greater than 1 in the reward function used in the offline reinforcement learning. rewardIt may include an operation to perform reward scaling by multiplying; an operation to perform hierarchical normalization using the reward scaled reward as input; and an operation to learn a critique ensemble composed of a plurality of critique networks that have undergone hierarchical normalization.

[0015] In one embodiment, the operation of performing the online reinforcement learning comprises the constant c in the reward function of the replay buffer used in the online reinforcement learning. reward It may include an operation to perform online fine-tuning using compensation scaling that multiplies by .

[0016] In one embodiment, the operation of reducing the estimated Q-value in the data-unheld region may include a penalty-imposing operation of setting the Q-value for the data-unheld region to be less than or equal to a preset value.

[0017] In one embodiment, the method may include: an operation of calculating a penalty loss; an operation of calculating a Temporal-Difference (TD) loss; an operation of determining a first loss based on the penalty loss and the TD loss; and an operation of performing at least one of the offline reinforcement learning and the online reinforcement learning based on the first loss.

[0018] One embodiment of the present disclosure includes a program stored on a recording medium to execute a method according to one embodiment of the present disclosure on a computer.

[0019] One embodiment of the present disclosure includes a computer-readable recording medium having a program for executing a method according to one embodiment of the present disclosure on a computer.

[0020] One embodiment of the present disclosure includes a computer-readable recording medium that records a database used in one embodiment of the present disclosure.

[0021] According to one embodiment of the present disclosure, in offline reinforcement learning, overestimation of Q-values ​​in areas where there is no data can be suppressed.

[0022] FIG. 1a is a drawing showing a Q-value corresponding to the correct answer according to one embodiment of the present disclosure, and FIG. 1b is a drawing showing a Q-value estimated by linear extrapolation according to one embodiment of the present disclosure.

[0023] FIG. 2 is a diagram showing a target Q-value function according to one embodiment of the present disclosure.

[0024] FIG. 3 is a drawing illustrating a compensation scaling method according to one embodiment of the present disclosure.

[0025] FIG. 4 is a diagram showing the results of performance improvement by compensation scaling according to one embodiment of the present disclosure.

[0026] FIG. 5 is a diagram illustrating a method of penalizing an impossible action according to one embodiment of the present disclosure.

[0027] FIG. 6 is a drawing showing the performance of a method according to one embodiment of the present disclosure.

[0028] FIG. 7 is a flowchart illustrating a reinforcement learning method according to one embodiment of the present disclosure.

[0029] FIG. 8 is a block diagram showing a reinforcement learning system according to one embodiment of the present disclosure.

[0030] To clarify the technical concept of the present disclosure, embodiments of the present disclosure will be described in detail with reference to the attached drawings. In describing the present disclosure, detailed descriptions of related known functions or components will be omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the present disclosure. Components having substantially the same functional configuration among the drawings have been assigned the same reference numerals and symbols as much as possible, even if they are shown in different drawings. For convenience of explanation, devices and methods will be described together where necessary. Each operation of the present disclosure does not necessarily have to be performed in the order described and may be performed in parallel, selectively, or individually.

[0031] The terms used in the embodiments of this disclosure have been selected to be as widely used and general as possible, taking into account the functions of this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, terms used in this specification should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0032] Throughout this disclosure, singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms such as “comprising” or “having” are intended to specify the presence of features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, throughout this disclosure, when a part is described as “comprising” a certain component, it means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0033] Expressions such as "at least one" modify the entire list of components and do not modify the components of the list individually. For example, "at least one of A, B, and C" and "at least one of A, B, or C" refer to only A, only B, only C, both A and B, both B and C, both A and C, all of A, B, and C, or any combination thereof.

[0034] Additionally, terms such as “...part,” “...module,” etc., as described in this disclosure refer to a unit that processes at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software.

[0035] Throughout the entire disclosure, when a part is described as being “connected” to another part, this includes not only cases where they are “directly connected” but also cases where they are “electrically connected” with other elements interposed between them. Furthermore, when a part is described as “comprising” a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0036] As used throughout this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “system configured to” may mean that the system is “capable of” together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in memory.

[0037] In one embodiment, agents performing online reinforcement learning can learn strategies for making optimal decisions by interacting in real-time with the environment. However, since such interaction with the environment involves learning from experience data collected by the agent, the cost of data collection may be substantial or the agent may be exposed to significant risk. To mitigate these drawbacks, offline reinforcement learning, which derives optimal policies from pre-collected data, is being researched.

[0038] In addition, agents trained using offline reinforcement learning can be deployed in real-world environments to learn more knowledge for making optimal decisions. However, due to the limited range of offline data, offline reinforcement learning can generate extrapolation errors that degrade overall performance by overestimating the Q-values ​​of out-of-distribution (OOD) behaviors.

[0039] Throughout this disclosure, extrapolation may refer to the process of estimating the value of a variable based on its relationship with other variables beyond the original range of observation. Accordingly, extrapolation error may refer to an error that occurs during the extrapolation process. Additionally, Q-value may refer to an estimated value of the cumulative reward expected when a certain action is taken in a specific state. While reward is an immediate benefit received by an agent, Q-value may represent a long-term benefit.

[0040] In one embodiment, when performing reinforcement learning using offline data, a Q-value extrapolation error may occur when the agent encounters a situation that is not in the training data distribution during the offline state, in which the Q-value estimated by the Q-network differs significantly from the actual value. This is because in offline reinforcement learning, the Q-network is trained using data collected from past policies or other policies, rather than data collected from the current policy. In other words, when the agent encounters a situation for the first time in the offline state, the Q-value may be inflated or underestimated as the Q-network makes inaccurate estimations, such as by performing linear extrapolation beyond the data range. Consequently, the agent may trust the incorrectly estimated Q-value and choose irrational behavior, which may lead to unstable learning or compromise of the policy.

[0041] One embodiment of the present disclosure aims to solve these problems by providing a method for making stable decisions by progressively reducing Q-values ​​outside the data range through reward scaling with layer normalization (RS-LN) and a penalty mechanism for infeasible actions.

[0042] FIG. 1a is a drawing showing a Q-value corresponding to the correct answer according to one embodiment of the present disclosure, and FIG. 1b is a drawing showing a Q-value estimated by linear extrapolation according to one embodiment of the present disclosure.

[0043] In describing an embodiment of the present disclosure, the method by which an agent performs learning when the correct answer is as shown in the graph of FIG. 1a will be explained by way of example. In one embodiment, given that data (110, 120) in an action region within a distribution (ID) as shown in FIG. 1b is provided, the agent [describes] an internal region A existing between ID action regions by linear extrapolation. ODD-in (150) and external area A existing outside the ID behavior area ODD-out When inferring (130, 140), the agent can infer the trend based on linear extrapolation, such as the form of 130, 140, and 150 in Fig. 1b. However, if the agent simply performs linear extrapolation, a discrepancy with the actual correct answer, as shown in Fig. 1a, may occur.

[0044] In one embodiment, one of the factors of extrapolation error in offline reinforcement learning is the tendency for linear extrapolation beyond the range of collected data. ReLU (Rectified Linear Unit)-based MLPs (Multi-Layer Perceptrons) often [experiment with] the observed data outside region A ODD-out In inferring (130, 140), A ODD-out(140) or estimating a tendency that continues to increase as it goes out of the boundary, or A ODD-out Estimation of a continuously decreasing trend can be performed, such as in the outer region of (130). For example, given data (110, 120) in an in-distribution (ID) behavioral region, ReLU-based MLPs infer the outer region of the data (130, 140) and the outer region A adjacent to the decreasing data region (110). ODD-out (130) is estimated to be a linearly continuously decreasing and increasing data area (120) and an adjacent external area A ODD-out (140) tends to estimate that it increases linearly and continuously. Due to this tendency, Q overestimation can occur for OOD behaviors. Accordingly, reinforcement learning methods using offline data require a method to effectively limit Q-values ​​outside the data range.

[0045] FIG. 2 is a diagram showing a target Q-value function according to one embodiment of the present disclosure.

[0046] Referring to FIG. 2, in a situation where data (220, 240) in an in-distribution (ID) action domain is provided, the agent [includes] an internal domain A existing between the ID action domains. ODD-in (230) and external area A existing outside the ID behavior area ODD-out (210, 250) can be inferred. In particular, the outer region A ODD-outIn inferring (210, 250), the agent may perform inference that reduces the Q-value, such as in the form of 210, 250 of FIG. 2. One embodiment of the present disclosure provides a method to solve the above-mentioned problems by using at least one of a reward scaling method through hierarchical normalization and a method of penalizing unfeasible behaviors, so that the agent according to one embodiment aims to estimate the Q-value as in FIG. 2.

[0047] In one embodiment, the reinforcement learning problem is a Markov Decision Process (MDP). It can be formalized as. Here, is the initial state distribution, S is the state space, and A is the action space, is a state transition function, is a reward function, represents the coefficient applied when converting future rewards to present value, that is, the discount factor.

[0048] In one embodiment, action space A is a set of actions, and in-distribution action region A D A, which is a feasible action region for the action space agent in F (eg, [-1, 1] n A, an infeasible action region consisting of the action space in ) and actions that the agent cannot perform in any state. I It can include the action space in.

[0049] In one embodiment, in offline reinforcement learning, a set of actions A in a specific state s. s A action space A F = [-1, 1] nSince it is determined internally, due to this, OOD behavior In other words, behaviors that do not exist in the data may occur, and if a policy selects such behaviors that do not exist in the data, the Q-network must perform extrapolation. Such extrapolation can be a source of error.

[0050] In one embodiment, among the OOD behaviors, behaviors within the convex hull (e.g., region 230 in FIG. 2) and behaviors outside of it (e.g., regions 210 and 250 in FIG. 2) need to be distinguished because they have different properties. First, the convex hull is a safe extrapolable region that can be inferred from given data, A s It refers to the set of all points that can be formed by linearly combining various behaviors existing in, and the region within the convex hull can be defined as follows [Equation 1]. For example, the region 150 in Fig. 1b may be the region within the convex hull.

[0051] [Mathematical Formula 1]

[0052]

[0053] Here, a i is A s It is the i-th action among the actions belonging to, and is the i-th non-negative weight.

[0054] The region (150) inside the convex shell of FIG. 1b is inferred to be similar to FIG. 1a, which corresponds to the correct answer, whereas the region (130, 140) outside the convex shell of FIG. 1b can be inferred to be dissimilar to FIG. 1b. That is, A ODD-in Since (s) is inside the convex hull from the data, extrapolation is performed relatively safely, and A ODD-out Since (s) is located outside the observed data, dangerous extrapolation with reduced prediction reliability may be performed. In one embodiment, ODD behavior a can be classified as follows [Equation 2].

[0055] [Mathematical Formula 2]

[0056]

[0057] That is, A ODD-in In (s), it does not pose a significant problem for the agent in reasoning, but A ODD-out In (s), because the ReLU-based MLP tends to behave linearly outside the data range, increasing or decreasing trends at the boundaries of the convex hull are extrapolated, which can cause the inference result to deviate from the correct answer. That is, A ODD-out Since training data for (s) does not exist, A ODD-out Q-value inference in (s) can lead to an increase in the possibility of uncontrollable errors. Accordingly, one embodiment of the present disclosure proposes a method to reduce Q-value inference errors.

[0058] In fact, beyond the given range of data, the agent in a neural network cannot grasp the trends of the actual data. Therefore, in offline reinforcement learning, to select the optimal action within the given range of data, A ODD-out The Q-value within (s) must be smaller than the maximum Q-value within the data range. Accordingly, according to one embodiment of the present disclosure, A ODD-out To keep the curve of (s) below the maximum value within the data, convex hull Conv(A s Extrapolation that flattens or decreases at the boundaries of ) can be performed.

[0059] According to one embodiment, the agent is in region A inside the convex shell. ODD-in In (230), the Q-value is estimated similarly to the region 150 in FIG. 1b, but the region outside the convex shell (210, 250) can be estimated to have a lower Q-value, unlike the regions 130 and 140 in FIG. 1b. Accordingly, the error rate due to linear extrapolation can be reduced.

[0060] FIG. 3 is a drawing illustrating a compensation scaling method according to one embodiment of the present disclosure.

[0061] In one embodiment, TD (Temporal Difference) reinforcement learning may refer to reinforcement learning that performs learning using actual rewards and future estimated values ​​for the next step. Additionally, in one embodiment, the TD target is a target value used in TD learning, which is the currently estimated value. or behavioral value As a target value calculated by bootstrapping with the experience one step (or n steps) ahead, TD target = It can be defined as follows. Here, r(s, a) is the immediate reward received after performing action a in the current state s, γ∈[0,1] is the discount factor, and G(s') is the estimated present value for the next state s', which can be defined as [Equation 3] below.

[0062] [Mathematical Formula 3]

[0063]

[0064] Here, is a policy function representing the probability distribution of selecting the next action a' from a specific state s', and is a parameter of the neural network representing the state value function V, used to predict the total reward expected in state s. The TD target value may be bootstrapped by including multiple levels of cumulative rewards.

[0065] In one embodiment, the Q-function When training with a positive (+) Temporal Difference (TD) target, since the network is initially initialized with weights near 0, The output starts with a small value but can gradually increase as training progresses to match the target value. During the training process, the learning effect obtained from a single input can be propagated to other inputs that the network recognizes as similar. For example, A behavior outside the data distribution (OOD) OOD-out Action A within distribution (s) D If it is determined that it is less similar to, the gradient update that increases the Q-value is A OOD-out It may act weakly in the (s) region. As a result, A OOD-out An increase in the Q-value for (s) is behavior A within the distribution D It can be naturally suppressed compared to.

[0066] Accordingly, one embodiment of the present disclosure is A D and A OOD-out In order to clearly distinguish between (s) We intend to introduce a method to enhance the effect of through compensation scaling.

[0067] 100% of the 5

[0068] To reduce such errors, a finer division of the input space may be necessary. One embodiment of the present disclosure intends to apply this to a neural network.

[0069] In one embodiment, as the output scale increases, small differences in the input lead to large differences in the output, so the neural network can learn more detailed and expressive features. However, if the input range is also reduced (e.g., reduced from [0, 1] to [0, 0.2]), the resolution requirement may disappear. To prevent this, Layer Normalization (LN) can be used. Since LN always normalizes the hidden layer output to a unit sphere to maintain the input volume, it can reliably derive the effect of increased resolution from the reward scale expansion.

[0070] According to one embodiment of the present disclosure, increasing reward scaling using hierarchical normalization (LN) may reduce the perceived similarity between actions within the data range and actions outside the data range. Additionally, in one embodiment, gradient updates for in-distribution (ID) actions may have a weak effect on out-of-distribution (OOD) Q-value predictions. This may lead to a reduction in out-of-distribution (OOD) Q-values ​​beyond the data range. Furthermore, one embodiment of the present disclosure may penalize the Q-values ​​of unexecutable actions that are far from the agent's executable action range.

[0071] FIG. 4 is a diagram showing the results of performance improvement by compensation scaling according to one embodiment of the present disclosure.

[0072] In one embodiment, the Toy dataset may consist of 2D inputs = (x1, x2) in the shape of an inverted cone with an obliquely cut opening. That is, c reward is the compensation scaling factor, and the Q-function Defined as, and the feasible input area We will explain by giving an example of an embodiment set as such.

[0073] In one embodiment, It is an in-distribution region where data is collected only within the area satisfying , and the rest When the region is outside the data distribution (OOD), the ReLU MLP is applied (a) without LN or PA (Penalizing Infeasible Actions), (b) after applying LN, and (c) after applying both LN and PA. reward The results of training with 1, 10, and 100 may be as shown in Fig. 4.

[0074] Referring to Figure 4, when a toy dataset is fitted without LN or PA applied, Q-values ​​can be explosively overestimated in the out-of-distribution region, as shown in the first column (None column) of Figure 4. In contrast, when the dataset is fitted with a Multi-Layer Perceptron (MLP) network using LN, it is confirmed that rapid overestimation is prevented to some extent in the second column (LN column) of Figure 4. In other words, linear extrapolation of Q-values ​​can be mitigated when LN is used.

[0075] In one embodiment, as the scale value of the compensation scaling increases, overestimation can be suppressed more strongly.

[0076] However, even if LN is applied, A OOD-out The Q-value of region (s) does not become lower than the Q-value of the region within the distribution. To address this, one embodiment of the present disclosure provides a region x1 or x2 ∈ (-2000, -1000) that is further realizable in LN and is far from the region. A penalty can be applied so that the Q-value approaches zero at (1000, 2000). Accordingly, as shown in the third column (LN and PA columns) of Fig. 4, it is confirmed that the Q-value decreases smoothly as it moves away from the region within the distribution. In other words, when a high compensation scale and the LN method are combined, A OOD-outSince the Q-value approaches 0 in the (s) region, according to one embodiment of the present disclosure, by using LN and PA together, A OOD-out In the (s) region, the Q-value can be effectively reduced.

[0077] FIG. 5 is a diagram illustrating a method of penalizing an impossible action according to one embodiment of the present disclosure.

[0078] Referring to Fig. 5, a feasible action domain A in a one-dimensional action space (n = 1). F and impossible behavioral domain A I The relationship between them is illustrated. In one embodiment, A I The Q-value at the minimum threshold Q min To converge to , the PA (Penalizing Infeasible Actions) loss may be considered. However, A F The internal Q-function is sufficiently learned using only the data and A I A guard interval may exist between the two regions to avoid being significantly affected by constraints.

[0079] In one embodiment, a subset of the unexecutable behavior domain can be defined as follows [Equation 4].

[0080] [Mathematical Formula 4]

[0081]

[0082] Here, n is the action dimension, L is the lower bound of the impossible action domain in each action dimension, U is the upper bound of the impossible action domain in each action dimension, and the feasible domain A F Is It is defined as. In order to secure a buffer interval, and must be satisfied.

[0083] In one embodiment, the PA loss function that minimizes the Q-value in the impossible behavior region can be defined as follows [Equation 5].

[0084] [Mathematical Formula 5]

[0085]

[0086] At this time, E s∼D represents the expected value for state s sampled from dataset D, and Q min eu c reward ·r min / ( It can be calculated as ). The minimum reward r of the task min If it is unknown, it can be replaced with the minimum reward observed in the data.

[0087] Accordingly, the total modified TD loss is the addition of PA loss to the existing TD loss, so the final TD loss = existing TD loss + α x PA loss, which can be expressed as [Equation 6] below.

[0088] [Mathematical Formula 6]

[0089]

[0090] Here is a policy function that selects the next action a from a specific state s, It can be defined as.

[0091] That is, according to one embodiment of the present disclosure, A F The lower limit of Q-value in the impossibility region sufficiently separated from Q min By reducing it to , the Q-value can naturally decrease outside the boundary. Through this, overestimation of the Q-value in the out-of-distribution region can be suppressed more effectively.

[0092] FIG. 6 is a drawing showing the performance of a method according to one embodiment of the present disclosure.

[0093] PARS in FIG. 6 stands for Penalizing infeasible Actions and Reward Scaling and may refer to a method according to one embodiment of the present disclosure. Referring to FIG. 6, it is confirmed that it exhibits superior performance compared to other algorithms in both offline reinforcement learning and online fine-tuning reinforcement learning.

[0094] A method according to one embodiment may be provided with a critique ensemble composed of up to 10 critique networks. The critique ensemble may be operated by applying different objective functions to the offline learning stage and the online fine-tuning stage during the policy improvement process, as shown in [Equation 7] below.

[0095] [Mathematical Formula 7]

[0096] offline training:

[0097] online finetuning:

[0098] FIG. 7 is a flowchart illustrating a reinforcement learning method according to one embodiment of the present disclosure.

[0099] Referring to FIG. 7, in operation 710, the processor can identify a data-holding region and a data-non-holding region during offline reinforcement learning. In one embodiment, the data-holding region may include an in-distribution region, and the data-non-holding region may include an out-of-distribution (OOD) region. For example, the data-holding region may include regions 110 and 120 of FIG. 1b, and the data-non-holding region may include regions 130, 140, and 150 of FIG. 1b.

[0100] In operation 730, the processor may perform an operation to reduce the estimated Q-value for the data-unretained region. In one embodiment, the operation to reduce the estimated Q-value for the data-unretained region may include a compensation scaling operation and a penalty application operation.

[0101] In one embodiment, the compensation scaling operation is a constant c greater than 1 in the compensation function. reward It may include an operation to generate a new reward function by multiplying. This reward scaling operation can be used in both offline reinforcement learning and online reinforcement learning.

[0102] For example, the processor uses a constant c greater than 1 in the reward function used in offline reinforcement learning. reward Perform reward scaling by multiplying, and the same constant c for the reward function of the replay buffer used in online reinforcement learning. reward Online fine-tuning can be performed using compensation scaling that multiplies by .

[0103] In one embodiment, the processor has a constant c greater than 1 in the reward function used in offline reinforcement learning. reward Reward scaling is performed by multiplying, and hierarchical normalization is performed using the reward scaled reward as input, and a critique ensemble composed of multiple critique networks that have undergone hierarchical normalization can be trained.

[0104] In one embodiment, the constant c reward is a constant greater than 1, for example, it can be from 10 to 100. As shown in the example in Fig. 4, the constant c reward It is confirmed that the Q-value is effectively reduced when α is 10 or 100.

[0105] In one embodiment, the processor may perform a penalty-applying operation to set the Q-value for a data-free region to be less than or equal to a preset value. For example, the processor may apply a penalty to cause the Q-value for a data-free region to converge to a preset lower limit or less.

[0106] In one embodiment, the processor may calculate a penalty loss and a Temporal-Difference (TD) loss, and determine a first loss based on the penalty loss and the TD loss. Additionally, the processor may perform at least one of the offline reinforcement learning and the online reinforcement learning based on the first loss. Here, the first loss may be determined by adding a value obtained by multiplying the penalty loss by a weight to the TD loss. For example, the first loss is (TD loss) + It can be determined as.

[0107] In operation 750, the processor can perform online reinforcement learning. In one embodiment, the online reinforcement learning may include an online fine-tuning step.

[0108] In one embodiment, the processor has the same constant c for the transitions collected in real time. reward You can apply [this] to save to the replay buffer, update the critique ensemble by applying reward scaling, hierarchy normalization, and penalty loss equally, and update the agent using the average Q-value of a randomly selected subset of the critique ensemble when improving the policy.

[0109] FIG. 8 is a block diagram showing a reinforcement learning system according to one embodiment of the present disclosure.

[0110] Referring to FIG. 8, the reinforcement learning system (800) may include a transceiver (810), memory (820), and a processor (830). However, not all components shown in FIG. 8 are essential components of the reinforcement learning system (800). The reinforcement learning system (800) may be implemented with more components than those shown in FIG. 8, or with fewer components than those shown in FIG. 8. Furthermore, the transceiver (810), processor (830), and memory (820) may be implemented in the form of a single chip.

[0111] In one embodiment, the transceiver (810) may communicate with a terminal or other electronic device connected to the reinforcement learning system (800) via wired or wireless connection. Various types of data, such as programs and files, such as applications, may be installed and stored in the memory (820). The processor (830) may access and use the data stored in the memory (820), or may store new data in the memory (820). In one embodiment, the memory (820) may include a database (not shown).

[0112] The processor (830) controls the overall operation of the reinforcement learning system (800) and may include at least one processor, such as a CPU, GPU, etc. The processor (830) may control other components included in the reinforcement learning system (800) to perform operations for operating the reinforcement learning system (800). For example, the processor (830) may execute a program stored in memory (820), read a stored file, or save a new file. In one embodiment, the processor (830) may perform operations for operating the reinforcement learning system (800) by executing a program stored in memory (820).

[0113] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0114] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0115] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0116] One embodiment of the present disclosure may also be implemented in the form of a recording medium comprising computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include both computer storage media and communication media. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. A communication medium typically includes computer-readable instructions, data structures, or program modules and includes any information transmission medium.

[0117] The foregoing description of the present disclosure is for illustrative purposes only, and those skilled in the art will understand that modifications can be easily made to other specific forms without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0118] The scope of the present disclosure is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present disclosure.

Claims

1. In reinforcement learning systems, One or more processors; and The system includes one or more memories that collectively store instructions that cause the system to perform operations when executed by the above-mentioned one or more processors, and the operations are: Actions to perform offline reinforcement learning; and Includes an action to perform online reinforcement learning, The operation of performing the above offline reinforcement learning is, An operation to identify data-holding areas and data-non-holding areas; and A system comprising an operation to reduce the estimated Q-value for the above-mentioned data-unpossible region.

2. In paragraph 1, the operation of reducing the estimated Q-value for the data-non-possessed area is, A constant c greater than 1 in the reward function used in the above offline reinforcement learning. reward A system comprising an operation to reduce the Q-value using compensation scaling that multiplies by 3. In paragraph 1, the operation of reducing the estimated Q-value for the data-non-possessed area is, A constant c greater than 1 in the reward function used in the above offline reinforcement learning. reward An operation that performs compensation scaling by multiplying; An operation to perform hierarchical normalization using the above-mentioned reward scaled as input; and A system comprising the operation of learning a critique ensemble composed of a plurality of critique networks on which the above-mentioned hierarchical normalization has been performed.

4. In paragraph 2, the operation of performing the online reinforcement learning is, The constant c in the reward function of the replay buffer used in the above online reinforcement learning reward A system comprising an operation to perform online fine-tuning using compensation scaling that multiplies by 5. In paragraph 2, the above constant c reward Is, A system set to a value of 10 or more.

6. In paragraph 1, the operation of reducing the estimated Q-value for the data-non-possessed area is, A system comprising a penalty-imposing operation that sets the Q-value for the above-mentioned data non-possessed area to be less than or equal to a preset value.

7. In Paragraph 6, the penalty imposition action above is, Action to calculate penalty loss; Operation to calculate TD (Temporal-Difference) loss; and Based on the above penalty loss and the above TD loss, the operation of determining a first loss is included, The above operations are, A system comprising an operation to perform at least one of the offline reinforcement learning and the online reinforcement learning based on the first loss.

8. In Paragraph 7, the above-mentioned first loss is, A system determined by adding a weighted value to the above TD loss.

9. A reinforcement learning method performed by at least one processor, Actions to perform offline reinforcement learning; and Includes an action to perform online reinforcement learning, The operation of performing the above offline reinforcement learning is, An operation to identify data-holding areas and data-non-holding areas; and A method comprising an operation to reduce the estimated Q-value in the above-mentioned data-unpossible region.

10. In paragraph 9, the operation of reducing the estimated Q-value in the data-unpossible region is, A constant c greater than 1 in the reward function used in the above offline reinforcement learning. reward A method comprising an operation to reduce the Q-value using compensation scaling that multiplies by 11. In paragraph 9, the operation of reducing the estimated Q-value in the data-unpossible region is, A constant c greater than 1 in the reward function used in the above offline reinforcement learning. reward An operation that performs compensation scaling by multiplying; An operation to perform hierarchical normalization using the above-mentioned reward scaled as input; and A method comprising the operation of learning a critique ensemble composed of a plurality of critique networks on which the above-mentioned layer normalization has been performed.

12. In Paragraph 10, the operation of performing the above-mentioned online reinforcement learning is, The constant c in the reward function of the replay buffer used in the above online reinforcement learning reward A method comprising an operation to perform online fine-tuning using compensation scaling that multiplies by 13. In paragraph 9, the operation of reducing the estimated Q-value in the data-unpossible region is, A method comprising a penalty-imposing operation that sets the Q-value for the above-mentioned data non-possessed area to be less than or equal to a preset value.

14. In Paragraph 13, the above method is, Action to calculate penalty loss; Operation to calculate TD (Temporal-Difference) loss; An operation to determine a first loss based on the above penalty loss and the above TD loss; and A method comprising the operation of performing at least one of the offline reinforcement learning and the online reinforcement learning based on the first loss.

15. A computer-readable recording medium having a program recorded thereon for performing the method of any one of paragraphs 9 through 14 on a computer.

Images