Offline priority-based reinforcement learning device and method

Abstract

An offline priority-based reinforcement learning apparatus and method are provided that train a compensation model using a route ranking list in which priority levels are assigned to all route segments based on priority feedback. [Solution] The reinforcement learning device 100 includes a memory and one or more processors that store a program and a dataset for performing reinforcement learning, and a control unit that operates by executing the program stored in the memory, extracts path segments, and repeats the steps of adding the path segments to a path ranking list sorted by priority level based on priority feedback for path pairs including the path segments multiple times to build a path ranking list, and learns a compensation model based on a priority pair including two path segments extracted from the path ranking list and priority labels assigned to the two path segments.

Description

[Technical Field]

[0001] The embodiments disclosed herein relate to an offline priority-based reinforcement learning apparatus and method.

[0002] This research was conducted as a result of the "Research into new continuous learning algorithms for handling data and environments with practical constraints" project (NRF-2021R1A2C2007884) of the Ministry of Science and ICT and the National Research Foundation of Korea (NRF)'s Individual Basic Research Project. This research was conducted as part of the "Support for Graduate School of Artificial Intelligence (Seoul National University)" project (IITP-2021-0-01343) under the ICT Broadcasting Innovation Talent Development Project of the Ministry of Science and ICT and the Institute for Information and Communications Technology Planning (IITP), and the "Development of a Sustainable Collaborative Multimodal Lifelong Learning Framework" project (IITP-2022-0-00113) under the Human-Centered Artificial Intelligence Core Source Technology Development Project. [Background technology]

[0003] In general, reinforcement learning is a method of learning through trial and error, in which an agent recognizes its current state in the environment and learns an action or policy that maximizes compensation among available actions. Reinforcement learning can be used to train agents such as autonomous robots, as disclosed in Korean Patent Publication No. 10-2021-0048969.

[0004] On the other hand, offline reinforcement learning performs reinforcement learning using a fixed offline dataset, and unlike general reinforcement learning, which performs reinforcement learning through interaction with the environment, offline reinforcement learning performs learning without interaction with the environment.

[0005] In both conventional and offline reinforcement learning, the design of the compensation function is the most important factor. To overcome the difficulty of designing an effective compensation function, offline priority-based reinforcement learning has been recently proposed, which learns a compensation model based on priority feedback obtained from humans and applies the learned compensation model to reinforcement learning.

[0006] On the other hand, the above-mentioned background art is technical information that the inventor possessed in order to derive the present invention or that he acquired in the process of deriving the present invention, and it cannot necessarily be said to be publicly known art that was disclosed to the general public prior to the filing of the present invention. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Korean Patent Publication No. 10-2021-0048969 Summary of the Invention [Problem to be solved by the invention]

[0008] The embodiments disclosed herein aim to provide an offline priority-based reinforcement learning apparatus and method for learning a compensation model using a route ranking list in which priority levels are assigned to entire route segments based on priority feedback. [Means for solving the problem]

[0009] As a technical means for achieving the above-mentioned technical object, according to one embodiment, a reinforcement learning device is disclosed, which includes: a memory for storing a program and a dataset for performing reinforcement learning; and a control unit including one or more processors, which operates by executing the program stored in the memory, extracting path segments, and repeating the steps of adding the path segments to a path ranking list sorted by priority level based on priority feedback for path pairs including the path segments a plurality of times to build the path ranking list, and learning a compensation model based on a priority pair including two path segments extracted from the path ranking list and priority labels assigned to the two path segments.

[0010] According to another embodiment, a reinforcement learning method is disclosed, which is executed by a reinforcement learning device and includes: extracting a path segment and repeating a plurality of times an operation of adding the path segment to a path order list sorted by priority level based on priority feedback for a path pair including the path segment to construct the path order list; and training a compensation model based on a priority pair including two path segments extracted from the path order list and priority labels assigned to the two path segments.

[0011] According to yet another embodiment, a recording medium storing a program executed by a reinforcement learning device to perform a reinforcement learning method is disclosed, the reinforcement learning method including: extracting a path segment and repeating an operation of adding the path segment to a path ranking list sorted by priority level based on priority feedback for a path pair including the path segment a plurality of times to construct the path ranking list; and training a compensation model based on a priority pair including two path segments extracted from the path ranking list and priority labels assigned to the two path segments.

[0012] According to yet another embodiment, a computer program is disclosed that is executed by a reinforcement learning device and is stored on a computer-readable recording medium for performing a reinforcement learning method, the reinforcement learning method including: extracting a path segment and repeating the steps of adding the path segment to a path ranking list sorted by priority level based on priority feedback for a path pair including the path segment a plurality of times to construct the path ranking list; and training a compensation model based on a priority pair including two path segments extracted from the path ranking list and priority labels assigned to the two path segments. [Effects of the Invention]

[0013] According to any one of the above-mentioned problem solving means, a route ranking list is constructed in which the extracted entire route segments are sorted by priority level, and priority pairs are generated using route pairs extracted from the route ranking list. This makes it possible to generate a larger number of priority pairs even with a small number of route segments, thereby enabling effective compensation model learning to be performed even within a fixed feedback collection limit.

[0014] In addition, according to any one of the above-mentioned problem solving means, priority pairs are generated by extracting route segments from a route ranking list sorted based on priority levels, so that the compensation model can learn the relative relationship between the generated priority pairs, i.e., secondary priorities, thereby improving the prediction accuracy of the compensation model.

[0015] The effects obtained from the disclosed embodiments are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the embodiments disclosed below pertain. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a configuration of a reinforcement learning device according to an embodiment; [Figure 2]FIG. 1 is a diagram illustrating a reinforcement learning process according to an embodiment. [Figure 3] FIG. 1 illustrates a framework for performing reinforcement learning according to one embodiment. [Figure 4] 1 is a flowchart illustrating a reinforcement learning method according to one embodiment. [Figure 5] FIG. 1 is a diagram for explaining the performance of a reinforcement learning method according to an embodiment. [Figure 6] FIG. 1 is a diagram for explaining the performance of a reinforcement learning method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Various embodiments will be described in detail below with reference to the accompanying drawings. The embodiments described below may be implemented in various modified forms. In order to more clearly describe the features of the embodiments, detailed descriptions of matters that are well known to those skilled in the art to which the following embodiments pertain will be omitted. In addition, parts of the drawings that are not relevant to the description of the embodiments will be omitted, and similar parts will be designated by similar reference numerals throughout the specification.

[0018] Throughout the specification, when a certain component is said to be "connected" to another component, this includes not only "directly connected" but also "connected via another component in between." Furthermore, when a certain component is said to "include" another component, this does not exclude the other component, but means that the other component may also be included, unless otherwise specified.

[0019] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0020] Before describing the embodiments, the meanings of the terms used below will be defined.

[0021] "Ternary Feedback" can mean feedback defined in three forms. For example, ternary feedback can respond with only one of three types: Bad, Equal, and Good for a specific target.

[0022] Alternatively, ternary feedback can be for the relative magnitude or priority between two different targets. For example, ternary feedback can respond for both targets A and B when A is greater than B (i.e., A > B), when A and B are equal (i.e., A = B), or when A is less than B (i.e., A < B).

[0023] "Offline Reinforcement Learning" means reinforcement learning performed based on a dataset collected from an unknown policy without the interaction of an agent with its environment.

[0024] Here, reinforcement learning can be executed based on preference feedbacks for an offline dataset and pairs of trajectories generated from the offline dataset. In this specification, reinforcement learning based on preference feedbacks for an offline dataset and pairs of trajectories generated from the offline dataset is called reinforcement learning based on offline priorities.

[0025] Preference Feedbacks are feedbacks obtained from a Human with respect to a specific topic. For example, by asking a Human which of two presented paths is advantageous for achieving a Goal, the response obtained from the Human to the question can be Preference Feedback. Preference Feedback can be ternary feedback. In other words, it can be something that indicates the preference for two options by selecting one of three fixed types of responses. As an example, for Path A and Path B, the ternary feedback can be one selected from the cases where A is preferred over B (i.e., A>B), where A and B are equal (i.e., A=B), or where A is not preferred over B (i.e., A<B).

[0026] Terms that require explanation other than those defined above will be explained separately below.

[0027] A reinforcement learning device according to an embodiment is a device that performs reinforcement learning based on offline priorities, extracts path segments based on a limited dataset, collects preference feedback for the generated path segments, learns a compensation model based on the collected preference feedback, and uses the learned compensation model to perform general reinforcement learning, that is, reinforcement learning that searches for an optimal policy that maximizes cumulative discounted compensation considering a Markov decision process (MDP).

[0028] Here, the reinforcement learning device can generate a path ranking list based on the preference feedback collected for a path pair including the extracted path segments, which are extracted based on an offline dataset.

[0029] For example, the reinforcement learning device can sequentially extract path segments one by one, determine a priority level for the extracted path segment based on priority feedback for a path pair including the extracted path segment, and add the path segment to a path ranking list based on the priority level.

[0030] In addition, the route segments in the route order list may be sorted based on priority levels, and the route order list may include route segment groups in which route segments having the same priority are grouped. The route segment groups may be sorted based on the priority levels corresponding to the route segment groups, and may be numbered according to the priority levels. More details about the route order list will be described later.

[0031] Meanwhile, the reinforcement learning device can extract any two path segments from the path ranking list, assign priority labels to the extracted path segments, and generate priority pairs including the path segments and the priority labels. The reinforcement learning device can extract all possible combinations of priority pairs from the path ranking list, and determine the value of the priority label to be assigned to the extracted priority pairs. The priority pairs can be included in a priority dataset.

[0032] Then, when training a compensation model based on the generated priority pairs, the reinforcement learning device can train parameters of the compensation model so that a loss function, which is an objective function of the compensation model, is minimized. Then, general reinforcement learning can be performed using the trained compensation model.

[0033] The reinforcement learning device may be implemented as an electronic terminal or a server-client system. When the reinforcement learning device is implemented as a server-client system, it may include a user's electronic terminal for interaction with the user.

[0034] Here, the electronic terminal may be connected to a remote server via a network, or may be implemented as a computer, a portable terminal, a wearable device, or the like that can be connected to other electronic terminals and servers. Here, the computer may include, for example, a notebook PC, a desktop PC, a laptop PC, etc. equipped with a web browser, and the portable terminal may be, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as PCS (Personal Communication System), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), GSM (Global System for Mobile communications), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet), smartphones, mobile WiMAX (Mobile Worldwide Interoperability for Microwave Access), etc. A wearable device is a type of information processing device that can be worn directly on the human body, such as a watch, glasses, accessory, clothing, or footwear, and can be connected to a remote server or other terminal via a network, either directly or through another information processing device.

[0035] The server may be implemented as a computing device capable of communicating with the electronic terminal via a network or as a cloud computing server, so that the reinforcement learning device may be implemented as a server-client system.

[0036] FIG. 1 is a block diagram showing a reinforcement learning device according to an embodiment.

[0037] Referring to FIG. 1, a reinforcement learning device 100 according to an embodiment may include a memory 110, a control unit 120, a communication unit 130, and an input / output unit 140.

[0038] The memory 110 may be configured with various types of memory such as DRAM, SSD, etc., and may incorporate and store a program for reinforcement learning and data therefor. For example, a reinforcement learning method may be incorporated and stored in the memory 110 in the form of a program. The memory 110 may also store a data set collected according to an unknown policy, i.e., an offline data set, and may store priority feedback for a path pair, which is a combination of any path segments, and a path ranking list generated by the control unit 120.

[0039] The control unit 120 includes at least one processor such as a CPU, a GPU, or the like, and can perform the reinforcement learning method described below by executing a program stored in the memory 110. For example, the control unit 120 can perform reinforcement learning by executing a program stored in the memory 110 using the processor.

[0040] In addition, the control unit 120 may control other components included in the reinforcement learning apparatus 100 to perform an operation corresponding to an input received via the input / output unit 140. For example, the control unit 120 may read a file stored in the memory 110 or store a new file in the memory 110, and may receive a pre-collected offline dataset from another server or device via the communication unit 130 (described later). The process by which the control unit 120 performs reinforcement learning based on offline priority will be described in detail below with reference to other figures.

[0041] Meanwhile, the communication unit 130 may perform wired or wireless communication with other devices or networks. For example, if the reinforcement learning device is implemented as a server-client system, the communication unit 130 may communicate with a user's electronic terminal connected to a server to receive a reinforcement learning request or priority feedback, or may transmit a reinforcement learning result to the user's electronic terminal. Alternatively, the communication unit 130 may receive priority feedback for any combination of path segments from another device or a server.

[0042] For this purpose, the communication unit 130 may include a communication module supporting at least one of various wired and wireless communication methods, and the communication module may be implemented in the form of a chipset. The wireless communication supported by the communication unit 130 may be, for example, Wireless Fidelity (WiFi), Wi-Fi Direct, Bluetooth, Ultra Wide Band (UWB), or Near Field Communication (NFC).

[0043] The input / output unit 140 may include an output device such as a display panel or a wearable display device for displaying the results or process of reinforcement learning, or a speaker for outputting the results of reinforcement learning or sounds required when performing reinforcement learning, and may also include various types of input devices (e.g., keyboard, touch screen, camera, etc.) for receiving priority feedback from a user.

[0044] The following describes in detail a reinforcement learning method performed by the reinforcement learning device according to one embodiment as a result of the control unit 120 executing a program stored in the memory 110. Unless otherwise specified, the process described below is performed as a result of the control unit 120 executing a program stored in the memory 110.

[0045] 2 is a diagram illustrating a reinforcement learning process according to an embodiment, specifically, a diagram illustrating a process in which a reinforcement learning device according to an embodiment learns a compensation model based on offline priority.

[0046] 2, the control unit 120 generates a set of route segments including route segments generated from an offline dataset 10 previously collected using an unknown policy μ, extracts new route segments from the set of generated route segments (20), collects priority feedback based on the extracted route segments (30), and constructs a route ranking list in which the route segments are sorted based on priority levels based on the collected priority feedback (40). The control unit 120 also generates priority pairs based on the route ranking list (50) and trains a compensation model based on the generated priority pairs. Thus, the control unit 120 enables the compensation model to perform list-based compensation prediction (60), which learns priority relationships between all route segments included in the route ranking list.

[0047] The offline dataset 10 is collected by an unknown policy μ and is a set of tuples consisting of a current state S, an action α, and a next state S′.

[0048]

number

[0049] The route segments are generated by combining tuples included in the offline dataset 10 to a predetermined length. S teeth

[0050]

number

[0051] The ranked list of trajectories (RLT) 40 corresponds to a preference level as follows, and can be defined as a list of path segment groups sorted based on the preference level.

[0052]

number

[0053]

number

[0054] For example, the route segment groups included in the route order list 40 may be sorted in ascending order. That is, for m and n where m>n, the route segment group g m σ is an element of i

[0055]

number

[0056]

number

[0057] Meanwhile, the control unit 120 can collect priority feedback for the route pair including the new route segment 20 (30), and determine a route segment group into which the extracted route segment will be inserted based on the collected priority feedback. The control unit 120 can add the extracted new route segment 20 to the route ranking list by adding the new route segment 20 to the determined route segment group.

[0058] Here, the collected priority feedback is a type of ternary feedback, which is a ternary priority feedback, and priority feedback for only one path pair can be collected at a time, and priority feedback for all segments included in the path ranking list cannot be collected at once.

[0059] Therefore, the control unit 120 can construct a route ranking list by repeating the steps of extracting route segments one by one, collecting priority feedback for route pairs that include the extracted route segments, determining the route segment group to which the extracted route segments belong based on the collected priority feedback, and adding the extracted route segments to the route segment group (40).

[0060] Here, since the route priority list is initially empty, the control unit 120 does not collect priority feedback for any segment σ1 that is extracted first, but adds it to the route priority list and initializes the route priority list as [{σ1}].

[0061] The control unit 120 may generate a priority pair based on the route order list (50). The priority pair may include two route segments and priority labels for the two route segments.

[0062] The control unit 120 may extract two route segments from the route ranking list and generate all possible route segment combination pairs. The control unit 120 may also assign priority labels based on the difference in priority between the route segment combinations included in the route segment combination pairs. For example, the control unit 120 may assign priority labels by comparing the route segment group numbers to which each route segment included in the route segment combinations belongs, and may generate priority pairs by combining the route segment combination pairs and the priority labels.

[0063] The control unit 120 can train a compensation model based on the priority pairs and predict compensation using the trained compensation model. As described above, the priority pairs include combinations of route segments included in the route ranking list and priority labels based on the difference in priority between the combined route segments. Therefore, the compensation model can learn the priority relationships between all route segments included in the route ranking list, thereby performing list-based compensation prediction 60.

[0064] 3 is a diagram illustrating a framework for performing a reinforcement learning method according to an embodiment. Referring to FIG. 3, the framework 300 may include a path generation module 310, a priority feedback collection module 320, a path order list generation module 330, a priority pair generation module 340, and a compensation model learning module 350. The control unit 120 executes a program stored in the memory 110 to implement and operate the modules included in the framework 300.

[0065] The trajectory generation module 310 is a module that generates trajectory segments from an offline dataset, and the control unit 120 can execute the trajectory generation module 310 to generate trajectory segments from the offline dataset.

[0066] The control unit 120 can generate route segments from the offline dataset according to a predetermined condition. For example, the control unit 120 can generate route segments for all cases from the offline dataset and include the generated route segments in the route segment set. Alternatively, the control unit 120 can generate only a predetermined number of route segments at a time and include the generated route segments in the route segment set.

[0067] The control unit 120 may extract one route segment from the set of route segments and transmit the extracted route segment to the priority feedback collection module 320 .

[0068] Meanwhile, the control unit 120 may collect priority feedback for a path pair defined by two path segments by executing the priority feedback collection module 320. As described above, the priority feedback is ternary feedback, which is one response selected from three fixed types, and may be obtained from input by a user or any respondent. Also, the priority feedback collection module 320 may be a module that acquires priority feedback for two received paths.

[0069] Here, one of the two routes included in the route pair is a route segment received from the route generation module 310 and sampled from a route segment set, and the other may be one of the route segments included in the existing route order list. The other route segment included in the route pair is a route segment group g m element σ k

[0070]

number

[0071] The control unit 120 can obtain necessary priority feedback using the priority feedback collection and extraction module 320 until the extracted route segment is added to the route ranking list.

[0072] The route order list generation module 330 is a module that generates a route order list based on priority feedback for route pairs including extracted route segments, and the control unit 120 can execute the route order list generation module 330 to obtain the route order list.

[0073] The control unit 120 calculates the newly extracted route segment σ iand an arbitrary route segment group g, which is a route segment already included in the route order list. m Path segment σ, which is an element of k

[0074]

number

[0075] The control unit 120 calculates the extracted route segment σ i Priority of and any route segment group g m element σ k If the priority of and is the same, the route segment group g m , and suppose the priority of the path segment σi and the element σ k If the priorities of the pair of paths are not the same, the control unit 120 may select elements of another path segment group and collect priority feedback for the new path pair via the priority feedback collection module 320 .

[0076] for example,

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081] The control unit 120 may search for a path segment group to which the extracted path segment is to be added by recursively using a binary search algorithm based on binary insertion sorting, as shown in Table 1. In some embodiments, a path order list may be constructed after collecting a large number of segments using merge sorting or quick sorting. However, if there is already a partially constructed path order list, the feedback efficiency of binary insertion sorting may be higher.

[0082] [Table 1] Referring to Table 1, if the route order list L is empty, the control unit 120 may add a route segment to the route order list to initialize the route order list. If the route order list is not empty, the control unit 120 may add a route segment to the intermediate group of the route segment group. g mid

[0083]

number

[0084] however,

[0085]

number

[0086]

number

[0087] On the other hand, if there is no route segment group to which the route segment σ is to be added (low=high), the control unit 120 low +1, and transfer the path segment σ to the new path segment group g low You can add +1.

[0088] On the other hand, one path segment σ i Because multiple priority feedback is required to add a route segment to the route order list, within a limited feedback collection budget, only a small number of route segments can be included in the route order list.

[0089] In particular, when using the binary search algorithm shown in Table 1, the longer the length of the path order list, the greater the number of priority feedbacks required to add a path segment to the path order list.

[0090] Therefore, the control unit 120 can also construct a route ranking list including multiple sub-ranked lists by setting a total feedback collection limit and a sub-feedback limit that is a sub-feedback limit of the total feedback collection limit, and generating multiple sub-ranked lists within the total feedback collection limit for a sub-ranked list including multiple route segment groups aligned to priority levels according to the set sub-feedback limits.

[0091] Here, the control unit 120 can generate a sub-order list by repeating the process of extracting a path segment as shown in Table 1 within the sub-feedback limit, determining a path segment group to which the extracted path segment will be added based on priority feedback for a path pair including the extracted path segment and an element of any path segment group included in the sub-order list, and adding the extracted path segment to the determined path segment group.

[0092] As mentioned above, for the same feedback collection limit, adding more path segments to a path order list than would be possible when generating a single path order list can increase the diversity of the samples.

[0093] Meanwhile, the control unit 120 may generate a priority dataset based on the route ranking list through the priority dataset generation module 340. The control unit 120 may generate a priority dataset by extracting priority pairs from the route ranking list, rather than independently extracting route segment pairs as in existing offline priority-based reinforcement learning methods. l ) includes multiple priority pairs as follows, and each priority pair (σ i1 , σ i2 , l i ) is divided into two path segments (σ i1 , σ i2 ) and the priority labels (l i ).

[0094]

number

[0095] The priority label may be a ternary label in which a predetermined value is assigned to each of three types, similar to the priority feedback. For example, a priority pair (σ i1 , σ i2 , l i ), the path segment σ i1 is the path segment σ i2 If it is preferred to i can be assigned 0, and the path segment σ i2 is the path segment σ il If it is preferred over i can be assigned a value of 1, and the path segment σ i2 and path segment σ i1 If and have the same priority, the priority label l i can be given a 0.5.

[0096] The control unit 120 may assign priority labels to the route segments included in the extracted route segment combination pair based on the priority levels corresponding to the route segment groups to which the route segments belong. For example, i1 , σ i2 , l i ), the path segment σ i1 is a route segment group g m The elements, i.e.,

[0097]

number

[0098]

Number

[0102]

number

[0103]

number

[0104]

number

[0105]

number

[0106] On the other hand, the compensation model θ can be defined as in Equation 25.

[0107]

number

[0108] As described above, the reinforcement learning device 100 according to one embodiment constructs a path ranking list in which the extracted entire path segments are sorted by priority level, and generates priority pairs using path pairs extracted from the path ranking list. This allows a larger number of priority pairs to be generated even with a small number of path segments, thereby enabling effective compensation model learning to be performed even within a fixed feedback collection limit.

[0109] In addition, the reinforcement learning device according to one embodiment generates priority pairs by extracting route segments from a route ranking list in which route segments are already sorted based on priority levels, and therefore can make the compensation model learn the relative relationship between the generated priority pairs, i.e., secondary priority, thereby improving the prediction accuracy of the compensation model.

[0110] For example, σ a <σ b <σ c Three route segments σ with priority a , σ b and σ c If we extract the priority pair using the route order list for (σ a , σ b , 1), (σ b , σ c , 1), (σ a , σ c , 1) can be obtained. By using Equation 25 and the obtained three priority pairs, the compensation model is c σ a The priority for σ c σb It can learn that the priority for the .times. ...

[0111] The term "module" used in the above embodiments refers to software or hardware components such as FPGAs (field programmable gate arrays) or ASICs, and the "module" performs a certain function. However, the term "module" is not limited to software or hardware. A "module" may be configured to reside on an addressable storage medium or to execute one or more processors. Thus, by way of example, "module" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0112] The functionality provided within components and units may be combined into fewer components and units or separated into additional components and units.

[0113] Furthermore, the components and "units" may be implemented to implement one or more CPUs within a device or a secure multimedia card.

[0114] FIG. 4 is a flowchart illustrating a reinforcement learning method according to an embodiment.

[0115] The reinforcement learning method shown in Fig. 4 includes a step of performing time-series processing by the reinforcement learning device 100 shown in Figs. 1 to 3. Therefore, even if the content is omitted below, the content described above regarding the reinforcement learning device 100 shown in Figs. 1 to 3 can also be used in the reinforcement learning method according to the embodiment shown in Fig. 4.

[0116] 4, the reinforcement learning device 100 can collect an offline dataset necessary for compensation model learning. The offline dataset is collected by an unknown policy and can include multiple tuples defined by a current state, an action, and a next state.

[0117] Next, the reinforcement learning device 100 may extract a path segment and repeat the process of adding the path segment to a path ranking list sorted by priority level based on priority feedback for the path pair including the path segment (S410). The path ranking list may include path segment groups to which group numbers are assigned corresponding to priority levels, and a higher priority level or a higher number for the path segment group may indicate a higher priority.

[0118] For example, the reinforcement learning device 100 may generate a path ranking list using a binary search algorithm. To this end, the path ranking list may be constructed by repeatedly extracting one path segment and adding the extracted path segment to the path ranking list based on priority feedback for the path pair including the path segment. Meanwhile, initially, that is, when the path ranking list is empty, the reinforcement learning device 100 may immediately add the extracted path segment to the path ranking list without collecting priority feedback.

[0119] Specifically, the reinforcement learning device 100 may generate a path segment set including a plurality of path segments generated based on an offline dataset, and may collect priority feedback for a path pair including one path segment extracted from the path segment set and a path segment already added to a path ranking list. Here, the path segment included in the path pair among the path segments in the existing path ranking list may be determined by a binary search algorithm, and may be one of the elements of a path segment group corresponding to a priority level of an intermediate value.

[0120] The reinforcement learning device 100 may add the extracted path segment to a path ranking list based on the collected priority feedback for the path pair. The priority feedback is a ternary priority feedback, and if the priority of the extracted path segment is the same as the priority of the path segment already added to the path ranking list, the extracted path segment may be added to the path segment group to which the path segment already added to the path ranking list belongs. If not, the reinforcement learning device 100 may also perform a process of collecting and comparing priority feedback for a path pair composed of elements of other path segment groups and the extracted path segment.

[0121] The reinforcement learning device 100 can generate priority pairs based on the route ranking list and train a compensation model based on the generated priority pairs (S420). Specifically, the reinforcement learning device 100 extracts all route segment combination pairs that can be combined with two route segments extracted from the route ranking list, and assigns priority labels based on priority levels corresponding to route segment groups to which the route segments included in the extracted route segment combination pairs belong, thereby generating a priority pair including two route segments and the priority labels assigned to the two route segments.

[0122] For example, the priority pair (σ i1 , σ i2 , l i ), the path segment σ i1 is a route segment group g m The elements, i.e.,

[0123]

number

[0124]

Number

[0125] Next, based on the generated priority pairs, the reinforcement learning device 100 can learn a compensation model according to mathematical formulas 19, 24, and 25, and execute reinforcement learning using the learned compensation model. Specifically, the reinforcement learning device 100 can update the parameters of the compensation model in the direction that minimizes the loss of the compensation model calculated by the loss function of mathematical formula 19 based on the priority pairs. Then, the reinforcement learning device 100 can execute reinforcement learning to search for an optimal policy that maximizes the cumulative discounted compensation in consideration of the Markov decision process (MDP) using the learned compensation model.

[0126] The reinforcement learning method according to an embodiment can construct a path ranking list in which the extracted overall path segments are sorted by priority levels, and generate more priority pairs using the path pairs extracted from the path ranking list, so that more priority pairs can be generated even with a small number of path segments, and effective learning of the compensation model can be executed within the fixed feedback collection limit.

[0127] In addition, the reinforcement learning method according to one embodiment generates priority pairs by extracting route segments from a route ranking list in which route segments are sorted based on priority levels, and thus allows the compensation model to learn the relative relationship between the generated priority pairs, i.e., secondary priority, thereby improving the prediction accuracy of the compensation model.

[0128] 5 and 6 are diagrams illustrating the performance of the reinforcement learning method according to one embodiment.

[0129] Figure 5 shows the correlation between the estimated reward and the actual reward using a compensation model. Figure 5(a) shows a compensation model trained using a conventional priority-based reinforcement learning method, i.e., without using a route ranking list, by extracting two route segments from a set of route segments and then using a priority pair including a priority label assigned based on priority feedback for the extracted route segments. Figure 5(b) shows a compensation model trained using a route ranking list generated using a reinforcement learning method according to an embodiment.

[0130] Referring to FIG. 5, it can be seen that the correlation coefficient of the reinforcement learning method according to the embodiment (b) has a higher value than the correlation coefficient of the existing priority-based reinforcement learning method (a).

[0131] Meanwhile, Figure 6 shows the performance of the reinforcement learning method according to one embodiment (the present invention) and existing reinforcement learning methods (MR, IPL, SeqRank) for specific tasks (Button-Press-Topdown, Box-Close, Dial-Turn). To evaluate the performance, we used the Meta World medium-replay dataset.

[0132] In Figure 6, MR stands for Markovian Reward, which refers to a base model trained in a multi-layer perceptron layer using a Markovian compensation process. IPL stands for Inverse Preference Learning (Hejna, J. et al. “Contrastive Preference Learning: Learning from Human Feedback without RL.” in arXiv preprint arXiv:2310.13639, 2023), which refers to a reinforcement learning method that learns a policy without a compensation model. SeqRank (Sequential Preference Ranking) (Hwang et al. “Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback.” in Advances in Neural Information Processing Systems (NeurIPS), 2023) refers to a reinforcement learning method that sequentially collects priority feedback between newly observed segments and previously collected segments and learns a compensation model based on this.

[0133] 6, the feedback number indicates the total feedback collection limit. For example, 500 indicates that the total feedback collection limit is 500. In one embodiment of the reinforcement learning method, the sub-feedback limit is set to 100. Therefore, the path order list can include 5 sub-order lists when the total feedback collection limit is 500, and 10 sub-order lists when the total feedback collection limit is 1000.

[0134] 6, it can be seen that the reinforcement learning method according to one embodiment exhibits superior performance compared to existing reinforcement learning methods in three tasks. In particular, when the number of feedbacks is small, the reinforcement learning method according to one embodiment exhibits improved performance compared to other reinforcement learning methods. This may mean that the reinforcement learning method according to one embodiment can effectively learn a compensation model even with a small number of feedbacks.

[0135] The reinforcement learning method according to the embodiment described with reference to FIG. 4 may also be embodied in the form of a computer-readable medium storing computer-executable instructions and data. Here, the instructions and data may be stored in the form of program code, which, when executed by a processor, may generate a predetermined program module and perform a predetermined operation. The computer-readable medium may be any available medium accessible by a computer, including both volatile and nonvolatile media, and both separable and non-separable media. The computer-readable medium may also be a computer recording medium. The computer recording medium may include both volatile and non-volatile, separable and non-separable media embodied by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer recording medium may be a magnetic storage medium such as a hard disk drive (HDD) or solid-state drive (SSD), an optical storage medium such as a CD, DVD, or Blu-ray disc, or a memory included in a server accessible via a network.

[0136] The reinforcement learning method according to the embodiment described with reference to Fig. 4 may also be embodied as a computer program (or a computer program product) including instructions executable by a computer. The computer program includes programmable machine instructions processed by a processor, and may be embodied in a high-level programming language, an object-oriented programming language, an assembly language, a machine language, or the like. The computer program may also be recorded on various types of computer-readable recording media (e.g., memory, a hard disk, a magnetic / optical medium, or a solid-state drive (SSD)).

[0137] Therefore, the reinforcement learning method according to the embodiment described with reference to Figure 4 can be implemented by executing the above-mentioned computer program on a computing device. The computing device can include at least a processor, a memory, a storage device, a high-speed interface connecting to the memory and a high-speed expansion port, and a low-speed interface connecting to a low-speed bus and the storage device. Each of these components is connected to each other using various buses and can be mounted on a common motherboard or in other suitable manners.

[0138] Here, the processor may process instructions within a computing device. Such instructions may include instructions stored in a memory or storage device for displaying graphical information to provide a GUI (Graphical User Interface) on an external input and output device, such as a display connected to a high-speed interface. In other embodiments, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as appropriate. Also, the processor may be implemented as a chipset consisting of chips containing multiple independent analog and / or digital processors.

[0139] Also, memory stores information within a computing device. As an example, memory may be comprised of a volatile memory unit or collection thereof. As another example, memory may be comprised of a non-volatile memory unit or collection thereof. Memory may also be in other forms of computer-readable media, such as, for example, a magnetic or optical disk.

[0140] Memory can provide a large amount of storage space for a computing device. Memory may be a computer-readable medium or a configuration that includes such a medium, such as a device in a Storage Area Network (SAN) or other configuration, such as a floppy disk drive, hard disk drive, optical disk drive, tape drive, flash memory, or other similar semiconductor memory device or device array.

[0141] The above-described embodiments are merely illustrative, and those skilled in the art will understand that the above-described embodiments may be easily modified into other specific forms without changing the technical ideas or essential features of the above-described embodiments. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. For example, each component described as a single component may be implemented in a distributed form, and similarly, each component described as a distributed component may be implemented in a combined form.

[0142] The scope of protection sought by this specification is determined by the claims below rather than the above detailed description, and should be construed to include all modifications or variations derived from the meaning and scope of the claims and their equivalents. [Explanation of symbols]

[0143] 100 Reinforcement Learning Device 110 memory 120 control section 130 Communications Department 140 Input / output section

Claims

1. A reinforcement learning device that performs offline priority-based reinforcement learning, A memory for storing a program and a dataset for performing reinforcement learning; a control unit including one or more processors, which operates by executing a program stored in a memory, extracting a path segment, and repeating the steps of adding the path segment to a path ranking list sorted by priority level based on priority feedback for a path pair including the path segment a plurality of times to construct the path ranking list, and learning a compensation model based on a priority pair including two path segments extracted from the path ranking list and priority labels assigned to the two path segments.

2. The reinforcement learning device according to claim 1 , wherein the control unit collects priority feedback for the plurality of path segments in the form of ternary feedback.

3. The reinforcement learning device according to claim 1 , wherein the control unit assigns the priority label based on a difference in priority between the path segments included in the priority pair, and the priority label is in the form of ternary feedback.

4. 2. The reinforcement learning device of claim 1, wherein the control unit generates the route ranking list by adding the extracted route segment to the route ranking list based on priority feedback for the newly extracted route segment from the dataset and the route segments already included in the route ranking list.

5. 2. The reinforcement learning device of claim 1, wherein the control unit generates the sub-ordered list by repeating a process of adding the path segments to the sub-ordered list based on priority feedback for the path pair within the sub-feedback limit based on a total feedback collection limit required to generate one path order list and a sub-feedback limit set by dividing the total feedback collection limit, and constructs the path order list by generating a plurality of the sub-ordered lists within the total feedback collection limit.

6. A reinforcement learning method executed by a reinforcement learning device, extracting a route segment and adding the route segment to a route order list sorted by priority level based on a priority feedback for a route pair including the route segment, by repeating the process a plurality of times to construct the route order list; and training a compensation model based on a priority pair including two route segments extracted from the route ranking list and priority labels assigned to the two route segments.

7. The reinforcement learning method of claim 6 , wherein the constructing step includes collecting priority feedback for the plurality of path segments in the form of ternary feedback.

8. the training step includes assigning the priority labels based on a difference in priority between the path segments included in the priority pairs; The reinforcement learning method of claim 6 , wherein the priority labels are in the form of ternary feedback.

9. 7. The reinforcement learning method of claim 6, wherein the constructing step includes determining a priority level based on priority feedback for a newly extracted path segment from the dataset and a path segment already included in the path ranking list, and adding the extracted path segment to the path ranking list based on the priority level.

10. 7. The reinforcement learning method of claim 6, wherein the constructing step includes: generating the sub-ordered list by repeating a process of adding the path segments to the sub-ordered list based on priority feedback for the path pair within the sub-feedback limit based on a total feedback collection limit required to generate one path order list and a sub-feedback limit set by dividing the total feedback collection limit, and generating a plurality of the sub-ordered lists within the total feedback collection limit to construct the path order list.

11. A computer program stored on a computer-readable recording medium for being executed by a reinforcement learning device and for carrying out the method of claim 6.

12. A computer-readable recording medium having a computer program recorded thereon for executing the method of claim 6.

Images