Reinforcement learning apparatus and method based on offline priority

Abstract

An offline priority-based reinforcement learning apparatus and method are provided that train a compensation model using a route ranking list in which priority levels are assigned to all route segments based on priority feedback. [Solution] The reinforcement learning device 100 includes a memory and one or more processors that store a program and a dataset for performing reinforcement learning, and a control unit that operates by executing the program stored in the memory, extracts path segments, and repeats the steps of adding the path segments to a path ranking list sorted by priority level based on priority feedback for path pairs including the path segments multiple times to build a path ranking list, and learns a compensation model based on a priority pair including two path segments extracted from the path ranking list and priority labels assigned to the two path segments.

Description

[Technical Field]

[0001] The embodiments disclosed herein relate to an offline priority-based reinforcement learning apparatus and method.

[0002] This research was conducted as a result of the project "Research on a New Continuous Learning Algorithm for Handling Data and Environments with Practical Constraints" (NRF-2021R1A2C2007884) of the Individual Basic Research Program of the Ministry of Science and ICT and the Korea Research Foundation (NRF). This research was conducted as a result of the "Support for Graduate School of Artificial Intelligence (Seoul National University)" project (IITP-2021-0-01343) of the Information and Communications Broadcasting Innovation Human Resource Development Project of the Ministry of Science and ICT and the Institute of Information and Communications Planning and Evaluation (IITP), and the "Development of a Sustainable Collaborative Multimodal Lifelong Learning Framework" project (IITP-2022-0-00113) of the Human-Centered Artificial Intelligence Core Source Technology Development Project. [Background technology]

[0003] Generally, reinforcement learning is a method of learning through trial and error, in which an agent recognizes its current state in its environment and learns the action or policy that maximizes compensation among the available actions. Reinforcement learning can be used to train agents such as autonomous robots, as described in Korean Published Patent No. 10-2021-0048969.

[0004] On the other hand, Offline Reinforcement Learning uses a fixed offline dataset to perform reinforcement learning. Unlike general reinforcement learning, which performs reinforcement learning through interaction with the environment, Offline Reinforcement Learning performs learning without interaction with the environment.

[0005] In both general reinforcement learning and offline reinforcement learning, the design of the compensation function is paramount. To overcome the difficulty of designing effective compensation functions, offline priority-based reinforcement learning has recently been proposed, which learns a compensation model based on priority feedback acquired from humans and applies the learned compensation model to reinforcement learning.

[0006] On the other hand, the aforementioned background technology is technical information that the inventor possessed for the purpose of deriving the present invention, or acquired during the process of deriving the present invention, and is not necessarily publicly known technology that was made public before the filing of the present invention. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Korean Published Patent No. 10-2021-0048969 [Overview of the project] [Problems that the invention aims to solve]

[0008] The embodiments disclosed herein aim to present an offline priority-based reinforcement learning apparatus and method for training a compensation model using a route ranking list to which priority levels are assigned to the overall route segments based on priority feedback. [Means for solving the problem]

[0009] As a technical means for achieving the technical challenges described above, according to one embodiment, a reinforcement learning device is disclosed that includes a memory for storing a program and a dataset for performing reinforcement learning, and a control unit which includes one or more processors and operates by executing the program stored in the memory, extracts path segments, constructs a path ranking list by repeatedly adding the path segments to a path ranking list sorted by priority level based on priority feedback for path pairs containing the path segments, and trains a compensation model based on priority pairs including two path segments extracted from the path ranking list and priority labels assigned to the two path segments.

[0010] Another embodiment discloses a reinforcement learning method which is performed by a reinforcement learning device and includes the steps of: constructing a path ranking list by repeatedly extracting path segments and adding the path segments to a path ranking list sorted by priority level based on priority feedback for path pairs containing the path segments; and training a compensation model based on priority pairs including two path segments extracted from the path ranking list and priority labels assigned to the two path segments.

[0011] Further embodiments disclose a recording medium that is executed by a reinforcement learning device and stores a program for executing a reinforcement learning method, the reinforcement learning method comprising: a step of constructing a path ranking list by repeatedly extracting path segments and adding the path segments to a path ranking list sorted by priority level based on priority feedback for path pairs containing the path segments; and a step of training a compensation model based on priority pairs including two path segments extracted from the path ranking list and priority labels assigned to the two path segments.

[0012] Further embodiments disclose a computer program, which is executed by a reinforcement learning device and stored on a computer-readable recording medium for executing a reinforcement learning method, the reinforcement learning method comprising: a step of constructing a path ranking list by repeatedly extracting path segments and adding the path segments to a path ranking list sorted by priority level based on priority feedback for path pairs containing the path segments; and a step of training a compensation model based on priority pairs including two path segments extracted from the path ranking list and priority labels assigned to the two path segments. [Effects of the Invention]

[0013] According to any one of the aforementioned problem-solving methods, a route ranking list is constructed in which the extracted overall route segments are sorted by priority level, and priority pairs are generated using route pairs extracted from the route ranking list. This allows for the generation of more priority pairs even with a small number of route segments, thus enabling the training of an effective compensation model even within a fixed feedback collection limit.

[0014] Furthermore, using any one of the aforementioned problem-solving methods, route segments are extracted from a route ranking list sorted based on priority level to generate priority pairs. This allows the compensation model to learn the relative relationships between the generated priority pairs, i.e., secondary priorities, thereby improving the prediction accuracy of the compensation model.

[0015] The effects obtained from the disclosed embodiments are not limited to those mentioned above, and other effects not mentioned below will be readily apparent to those with ordinary skill in the art to which the disclosed embodiments belong. [Brief explanation of the drawing]

[0016] [Figure 1] This block diagram shows the configuration of a reinforcement learning device according to one embodiment. [Figure 2]This figure schematically illustrates the reinforcement learning process according to one embodiment. [Figure 3] This figure shows a framework for performing reinforcement learning according to one embodiment. [Figure 4] This flowchart shows a reinforcement learning method according to one embodiment. [Figure 5] This figure illustrates the performance of a reinforcement learning method according to one embodiment. [Figure 6] This figure illustrates the performance of a reinforcement learning method according to one embodiment. [Modes for carrying out the invention]

[0017] The following describes various embodiments in detail based on the attached drawings. The embodiments described below can be modified and implemented in various forms. In order to more clearly describe the characteristics of the embodiments, detailed explanations of matters that are commonly known to those who have ordinary skill in the art to which the embodiments belong are omitted. Furthermore, parts of the drawings that are not relevant to the description of the embodiments are omitted, and similar parts are denoted by similar reference numerals throughout the specification.

[0018] Throughout the specification, when we say that one configuration is "linked" to another, this includes not only cases where they are "directly linked," but also cases where they are "linked with another configuration in between." Furthermore, when we say that one configuration "includes" another, unless otherwise stated, this does not mean that it excludes other configurations, but rather that it may include even more configurations.

[0019] The following describes an embodiment in detail based on the attached drawings.

[0020] Before describing the examples, we define the meanings of the terms used below.

[0021] "Ternary Feedback" can mean feedback defined in three forms. For example, ternary feedback can respond to a specific target with only one of three types: Bad, Equal, and Good in terms of priority.

[0022] Alternatively, ternary feedback can be for the relative magnitude or priority between two different targets. For example, ternary feedback can respond to both targets A and B when A is greater than B (i.e., A > B), when A and B are equal (i.e., A = B), or when A is less than B (i.e., A < B).

[0023] "Offline Reinforcement Learning" means reinforcement learning performed based on a dataset collected from an unknown policy without the interaction between an agent and its environment.

[0024] Here, reinforcement learning can be executed based on preference feedbacks for an offline dataset and pairs of trajectories generated from the offline dataset. In this specification, reinforcement learning based on preference feedbacks for an offline dataset and pairs of trajectories generated from the offline dataset is referred to as reinforcement learning based on offline priorities.

[0025] Preference Feedbacks are feedbacks obtained from a Human for a specific topic. For example, asking a Human which of two presented paths is advantageous for achieving a Goal, and the response obtained from the Human to the question can be Preference Feedback. Preference Feedback can be ternary feedback. In other words, it can be something that indicates the preference for two options by selecting one of three fixed types of replies. As an example, for Path A and Path B, the ternary feedback can be one of the following selected: when A is preferred over B (i.e., A > B), when A and B are equivalent (i.e., A = B), or when A is not preferred over B (i.e., A < B).

[0026] Terms that require explanation other than those defined above will be explained separately below.

[0027] A reinforcement learning device according to an embodiment is a device that performs reinforcement learning based on offline priority, extracts path segments based on a limited dataset, collects preference feedback for the generated path segments, learns a compensation model based on the collected preference feedback, and uses the learned compensation model to perform general reinforcement learning, that is, reinforcement learning that searches for an optimal policy that maximizes cumulative discounted compensation considering a Markov decision process (MDP).

[0028] Here, the reinforcement learning device can generate a path ranking list based on the preference feedback collected for a path pair including the extracted path segments, which are extracted based on an offline dataset.

[0029] For example, a reinforcement learning device can sequentially extract path segments one by one, determine a priority level for the extracted path segments based on priority feedback for the path pairs containing the extracted path segments, and add the path segments to a path ranking list based on the priority level.

[0030] Furthermore, route segments within a route ranking list can be sorted based on their priority level, and the route ranking list can include route segment groups, which are groups of route segments with the same priority. Route segment groups can be sorted based on the priority level corresponding to the route segment group, and can also be numbered according to the priority level. More detailed information about route ranking lists will be provided later.

[0031] On the other hand, a reinforcement learning device can extract any two path segments from a path ranking list, assign priority labels to the extracted path segments, and generate priority pairs containing the path segments and priority labels. The reinforcement learning device can extract all combinatable priority pairs from the path ranking list and determine the value of the priority label to be assigned to the extracted priority pairs. Priority pairs can be included in a priority dataset.

[0032] The reinforcement learning device can then train a compensation model based on the generated priority pairs, training the parameters of the compensation model so that the loss function, which is the objective function of the compensation model, is minimized. The trained compensation model can then be used to perform general reinforcement learning.

[0033] The reinforcement learning device described above can be implemented using an electronic terminal or a server-client system. When the reinforcement learning device is implemented using a server-client system, it may include the user's electronic terminal for interaction with the user.

[0034] Here, the electronic terminal can be connected to a remote server via a network, or it can be embodied as a computer, portable terminal, or wearable device that can connect to other electronic terminals and servers. Here, "computer" includes, for example, notebook PCs, desktop PCs, and laptop PCs equipped with a web browser, and "portable terminal" includes, for example, wireless communication devices that guarantee portability and mobility, and can include all types of handheld wireless communication devices such as PCS (Personal Communication System), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), GSM (Global System for Mobile communications), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet), smartphones, and mobile WiMAX (Mobile Worldwide Interoperability for Microwave Access). Furthermore, wearable devices are information processing devices that can be worn directly on the human body, such as watches, glasses, accessories, clothing, and footwear, and can connect to remote servers or other terminals via a network, either directly or through other information processing devices.

[0035] Furthermore, since the server can be implemented by a computing device that can communicate with electronic terminals via a network, or by a cloud computing server, a reinforcement learning device can be implemented by a server-client system.

[0036] Figure 1 is a block diagram showing a reinforcement learning device according to one embodiment.

[0037] Referring to Figure 1, one embodiment of the reinforcement learning device 100 may include a memory 110, a control unit 120, a communication unit 130, and an input / output unit 140.

[0038] Memory 110 can be composed of various types of memory such as DRAM and SSD, and can store programs and data for reinforcement learning. For example, reinforcement learning methods can be stored in memory 110 in the form of a program. Memory 110 can also store datasets collected by unknown policies, i.e., offline datasets, and can store priority feedback for path pairs, which are combinations of arbitrary path segments, and path ranking lists generated by the control unit 120.

[0039] The control unit 120 is configured to include at least one processor, such as a CPU or GPU, and can execute the reinforcement learning method described below by running a program stored in the memory 110. For example, the control unit 120 can perform reinforcement learning by having the processor execute a program stored in the memory 110.

[0040] Furthermore, the control unit 120 can control other components included in the reinforcement learning device 100 to perform actions corresponding to inputs received via the input / output unit 140. For example, the control unit 120 can read files stored in the memory 110 or save new files to the memory 110, and can receive offline datasets previously collected from other servers or devices via the communication unit 130, which will be described later. The process by which the control unit 120 performs reinforcement learning based on offline priority will be described in detail below with reference to other figures.

[0041] On the other hand, the communication unit 130 can perform wireless communication with other devices or networks. For example, if the reinforcement learning device is embodied by a server-client system, the communication unit 130 can communicate with the user's electronic terminal connected to the server to receive reinforcement learning requests or priority feedback, or transmit the results of reinforcement learning to the user's electronic terminal. Alternatively, the communication unit 130 can receive priority feedback from other devices or servers for any combination of route segments.

[0042] For this purpose, the communication unit 130 may include a communication module that supports at least one of a variety of wireless communication methods, and the communication module may be embodied in the form of a chipset. The wireless communication supported by the communication unit 130 may be, for example, WiFi (Wireless Fidelity), Wi-Fi Direct, Bluetooth®, UWB (Ultra Wide Band), or NFC (Near Field Communication).

[0043] The input / output unit 140 may include a display panel or wearable display device for displaying the results or process of reinforcement learning, or an output device such as a speaker for outputting the results of reinforcement learning or sounds necessary during reinforcement learning execution. In addition, it may include various types of input devices (e.g., keyboard, touchscreen, camera, etc.) for receiving priority feedback from the user.

[0044] The following describes in detail a reinforcement learning method performed by a reinforcement learning device according to one embodiment, which is carried out by the control unit 120 executing a program stored in the memory 110. Unless otherwise specified, the process described below is performed by the control unit 120 executing a program stored in the memory 110.

[0045] Figure 2 is a schematic diagram illustrating the reinforcement learning process according to one embodiment. Specifically, it is a schematic diagram illustrating the process by which a reinforcement learning device according to one embodiment learns a compensation model based on offline priority.

[0046] Referring to Figure 2, the control unit 120 generates a set of route segments including route segments generated from an offline dataset 10 previously collected by an unknown policy μ, extracts new route segments from the generated set of route segments (20), collects priority feedback based on the extracted route segments (30), and constructs a route ranking list in which route segments are sorted based on priority levels based on the collected priority feedback (40). The control unit 120 also generates priority pairs based on the route ranking list (50) and allows a compensation model to be trained based on the generated priority pairs. Thus, the control unit 120 enables the compensation model to perform list-based compensation prediction (60) which learns the priority relationships between all route segments included in the route ranking list.

[0047] The offline dataset 10 is collected by an unknown policy μ and is a set of tuples consisting of the current state S, action α, and next state S'.

[0048]

number

[0049] Route segments are generated by combining tuples included in the offline dataset 10 to a predetermined length, and the route segment set D S teeth

[0050]

number

[0051] The Ranked List of Trajectories (RLT) 40 corresponds to a Preference Level and can be defined as a list of route segment groups sorted according to their Preference Level.

[0052]

number

[0053]

number

[0054] For example, the route segment groups included in route ranking list 40 may be sorted in ascending order. That is, for m and n where m > n, the route segment group g m The element σ i

[0055]

number

[0056]

number

[0057] On the other hand, the control unit 120 can collect priority feedback for the route pair that includes the new route segment 20 (30), determine the route segment group into which the extracted route segment will be inserted based on the collected priority feedback, and add the extracted new route segment 20 to the route ranking list by adding the new route segment 20 to the determined route segment group.

[0058] Here, the collected priority feedback is a type of ternary priority feedback, meaning that priority feedback can only be collected for one route pair at a time, and priority feedback for the entire segment included in the route ranking list cannot be collected at once.

[0059] Therefore, the control unit 120 can construct a route ranking list by repeatedly extracting route segments one by one, collecting priority feedback for route pairs including the extracted route segments, determining the route segment group to which the extracted route segments belong based on the collected priority feedback, and adding the extracted route segments to the said route segment group (40).

[0060] Here, since the route priority list is initially empty, the control unit 120 does not collect priority feedback for any segment σ1 that is extracted first, but can add it to the route priority list and initialize the route priority list as [{σ1}].

[0061] The control unit 120 can generate priority pairs based on the route ranking list (50). A priority pair may include two route segments and priority labels for those two route segments.

[0062] The control unit 120 can extract two route segments from the route ranking list and generate all possible route segment combination pairs. The control unit 120 can also assign priority labels based on the priority differences between the route segment combinations included in the route segment combination pairs. For example, the control unit 120 can compare the route segment group numbers to which each route segment in a route segment combination belongs, assign priority labels, and then combine the route segment combination pairs and priority labels to generate priority pairs.

[0063] The control unit 120 can train a compensation model based on priority pairs and predict compensation using the trained compensation model. As mentioned above, priority pairs include combinations of route segments included in the route ranking list and priority labels based on the difference in priority between the combined route segments. Therefore, the compensation model can learn the priority relationships between all route segments included in the route ranking list and can perform list-based compensation prediction 60.

[0064] Figure 3 illustrates a framework for implementing a reinforcement learning method according to one embodiment. Referring to Figure 3, the framework 300 may include a path generation module 310, a priority feedback collection module 320, a path ranking list generation module 330, a priority pair generation module 340, and a compensation model learning module 350. The control unit 120 can execute a program stored in memory 110 to realize and operate the modules included in the framework 300.

[0065] The route generation module 310 is a module that generates route segments from an offline dataset, and the control unit 120 can execute the route generation module 310 to generate route segments from the offline dataset.

[0066] The control unit 120 can generate route segments from an offline dataset under predetermined conditions. For example, the control unit 120 can generate route segments for all cases from the offline dataset and include the generated route segments in a route segment set. Alternatively, the control unit 120 can generate a predetermined number of route segments at once and include the generated route segments in a route segment set.

[0067] The control unit 120 can extract one route segment from the route segment set and transmit the extracted route segment to the priority feedback collection module 320.

[0068] On the one hand, the control unit 120 can execute the priority feedback collection module 320 to collect priority feedback for a path pair defined by two path segments. As described above, the priority feedback is a ternary feedback that is one response selected from three fixed types, and can be obtained from the input of the user or any responder. Also, the priority feedback collection module 320 can be a module that obtains priority feedback for two received paths.

[0069] Here, one of the two paths included in the path pair is a path segment received from the path generation module 310, which is a path segment extracted (sampled) from the set of path segments, and the other can be one of the path segments included in the existing path ranking list. The other path segment included in the path pair can be an element σ of an arbitrary path segment group g m of k

[0070]

Number

[0071] The control unit 120 can obtain the necessary priority feedback until the path segment extracted using the priority feedback collection extraction module 320 is added to the path ranking list.

[0072] The path ranking list generation module 330 is a module that generates a path ranking list based on the priority feedback for the path pair including the extracted path segment, and the control unit 120 can execute the path ranking list generation module 330 to obtain the path ranking list.

[0073] The control unit 120 newly extracts the path segment σ iand route segments that are already included in the route ranking list, and any route segment group g m The element of the path segment σ k

[0074]

number

[0075] The control unit 120 processes the extracted path segment σ i Priority and arbitrary route segment group g m Element σ k If the priority is the same as that of route segment group g m It can be added, and if the priority and element σ of the path segment σi are... k If the priorities are not the same, the control unit 120 can select elements from other route segment groups and collect priority feedback for the new route pair via the priority feedback collection module 320.

[0076] for example,

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081] As shown in Table 1, the control unit 120 can recursively use a binary search algorithm based on binary insertion sort to find a group of route segments to which the extracted route segments will be added. In some embodiments, a route ranking list can also be constructed after collecting a large number of segments using merge sort or quick sort. However, if a route ranking list has already been partially constructed, the feedback efficiency of binary insertion sort can be improved.

[0082] [Table 1] Referring to Table 1, the control unit 120 can initialize the route ranking list by adding a route segment to the route ranking list L if the route ranking list L is empty. If the route ranking list is full, the control unit 120 will add an intermediate group of route segment groups. g mid

[0083]

number

[0084] however,

[0085]

number

[0086]

number

[0087] On the other hand, the control unit 120 determines if there is no route segment group to which route segment σ is added (low=high), g low A new route segment group called +1 is created, and route segment σ is added to the new route segment group g low It can be added to +1.

[0088] On the other hand, one path segment σ i Since multiple priority feedbacks are required to add a route to the route ranking list, only a small number of route segments can be included in the route ranking list within a limited feedback budget.

[0089] In particular, when using the binary search algorithm shown in Table 1, the longer the route ranking list, the more priority feedbacks required to add a route segment to the route ranking list can also be increased.

[0090] Therefore, the control unit 120 can also set a total feedback collection limit and a total feedback collection limit, and construct a route ranking list that includes multiple sub-ranking lists by generating multiple sub-ranking lists within the total feedback collection limit for a sub-ranking list that includes multiple route segment groups aligned to priority levels by the set sub-feedback limits.

[0091] Here, the control unit 120 can generate a sub-ranking list by extracting route segments within the sub-feedback limit, as shown in Table 1, determining the route segment group to which the extracted route segments will be added based on priority feedback for route pairs containing the elements of any route segment group included in the sub-ranking list, and repeating the process of adding the extracted route segments to the determined route segment group.

[0092] As mentioned above, sample diversity can be increased by adding more route segments to the route ranking list than when generating a single route ranking list, compared to the same feedback collection limit.

[0093] On the other hand, the control unit 120 can generate a priority dataset based on the route ranking list via the priority dataset generation module 340. The control unit 120 does not independently extract route segment pairs like existing offline priority-based reinforcement learning methods, but can generate a priority dataset by extracting priority pairs from the route ranking list. (D) l ) includes multiple priority pairs as shown below, and each priority pair (σ i1 , σ i2 , l i ) consists of two path segments (σ i1 , σ i2 ) and priority labels for the two route segments (l i ) can include

[0094]

number

[0095] Priority labels, like priority feedback, can be ternary labels to which predetermined values ​​are assigned for three types. For example, priority pairs (σ i1 , σ i2 , l i ) for the path segment σ i1 is the path segment σ i2 If preferred, the control unit 120 will set the priority label (label) l i It is possible to assign 0 to the path segment σ i2 is the path segment σ il If preferred, the priority label (label) i It is possible to assign 1 to the route segment σ i2 and path segment σ i1 If they have the same priority, the priority label l i A value of 0.5 can be assigned to it.

[0096] The control unit 120 can assign priority labels to the route segments included in the extracted route segment combination pair based on the priority level corresponding to the route segment group to which each segment belongs. For example, the priority pair (σ i1 , σ i2 , l i ) for the path segment σ i1 Route segment group g m The element, namely,

[0097]

number

[0098] [Number] if it is said that σ i1 and the path segment σ i2 belong to the same path segment group (that is, m = n), the control unit 120 can assign 0.5 to the priority label l i . On the other hand, when the priority level corresponding to the path segment group g i1 to which the path segment σ m belongs is higher than the priority level corresponding to the path segment group g i2 to which the path segment σ n belongs (that is, m > n), the control unit 120 can assign 0 to the priority label l i , and in the opposite case (that is, m < n), the control unit 120 can assign 1 to the priority label l i .

[0099] On the other hand, the control unit 120 can execute the compensation model learning module 350 to perform learning of the compensation model based on the priority pairs included in the priority dataset. The control unit 120 can update the parameters of the compensation model so that the value of the loss function is minimized using the priority pairs included in the priority dataset. The loss function can be a cross-entropy loss function as shown in Mathematical Formula 19.

[0100] [Number] In Mathematical Formula 19, (σ i1 , σ i2 , l i ) is a priority pair, D pref is a priority dataset,

[0101] [Number] σ is one path segment i1 other route segments σ i2 This is a better probability,

[0102]

number

[0103]

number

[0104]

number

[0105]

number

[0106] On the other hand, compensation model r θ It can be defined as shown in mathematical formula 25.

[0107]

Number

[0108] As described above, the reinforcement learning device 100 according to an embodiment constructs a path ranking list in which the extracted overall path segments are sorted by priority level, and generates priority pairs using the path pairs extracted from the path ranking list, so that more priority pairs can be generated even with a small number of path segments. Therefore, effective learning of the compensation model can be executed within the fixed feedback collection limit.

[0109] In addition, the reinforcement learning device according to an embodiment extracts path segments from a path ranking list in which path segments are sorted based on the priority level, and generates priority pairs, so that the relative relationship between the priority pairs generated in the compensation model, that is, the secondary priority, can be learned, and the prediction accuracy of the compensation model can be improved.

[0110] For example, for three path segments σ a <σ b <σ c having the priority of σ a σ b and σ c , if priority pairs are extracted using the path ranking list, three priority pairs (σ a σ b , 1), (σ b σ c , 1), and (σ a σ c , 1) can be obtained. With Mathematical formula 25 and the three obtained priority pairs, the compensation model can determine that the priority of σ c with respect to σ a is σ c with respect to σb It can learn that it has a higher priority than the priority given to [another element].

[0111] As used in the above embodiments, the term "~part" refers to software or hardware components such as FPGAs (field programmable gate arrays) or ASICs, and "~part" performs a certain role. However, "~part" is not limited to software or hardware. "~part" can be configured to reside in an addressable storage medium, or to regenerate one or more processors. Thus, as an example, "~part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of programmatic code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0112] Components and the functions provided within a "~part" can be combined with a smaller number of components and "~parts" or separated from additional components and "~parts".

[0113] Furthermore, the components and "~parts" can also be embodied to regenerate one or more CPUs within the device or security multimedia card.

[0114] Figure 4 is a flowchart illustrating a reinforcement learning method according to one embodiment.

[0115] The reinforcement learning method shown in Figure 4 includes a step of processing in a time series using the reinforcement learning device 100 shown in Figures 1 to 3. Therefore, even if some details are omitted below, the information described above regarding the reinforcement learning device 100 shown in Figures 1 to 3 can also be used in the reinforcement learning method according to the embodiment shown in Figure 4.

[0116] Referring to Figure 4, the reinforcement learning device 100 can collect offline datasets necessary for training the compensation model. The offline datasets are collected by unknown policies and may include multiple tuples defined by the current state, the action, and the next state.

[0117] Next, the reinforcement learning device 100 can construct the route ranking list by repeatedly extracting route segments and adding them to a route ranking list sorted by priority level, based on priority feedback for the route pairs containing the route segments (S410). The route ranking list may include route segment groups to which group numbers are assigned corresponding to priority levels, and a higher priority level or route segment group number may indicate a higher priority.

[0118] For example, the reinforcement learning device 100 can generate a path ranking list using a binary search algorithm. To do this, it can repeatedly extract a path segment and add the extracted path segment to the path ranking list based on priority feedback for the path pair containing the path segment, thereby constructing the path ranking list. On the other hand, initially, i.e., when the path ranking list is empty, the reinforcement learning device 100 can immediately add the extracted path segment to the path ranking list without collecting priority feedback.

[0119] Specifically, the reinforcement learning device 100 can generate a set of path segments containing multiple path segments generated based on an offline dataset, and can collect priority feedback for a path pair containing one path segment extracted from the set of path segments and a path segment that has already been added to the path ranking list. Here, the path segments included in the path pair from among the path segments in the existing path ranking list can be determined by a binary search algorithm and may be one of the elements of the path segment group corresponding to the intermediate priority level.

[0120] The reinforcement learning device 100 can add a path segment extracted based on the collected priority feedback to the path ranking list for a path pair. The priority feedback is ternary priority feedback. If the priority of the extracted path segment is the same as the priority of the path segment already added to the path ranking list, the extracted path segment can be added to the path segment group to which the path segment already added to the path ranking list belongs. Otherwise, the reinforcement learning device 100 can also execute the process of collecting and comparing the priority feedback for the path pair composed of the elements of other path segment groups and the extracted path segment.

[0121] The reinforcement learning device 100 can generate a priority pair based on the path ranking list and learn a compensation model based on the generated priority pair (S420). Specifically, the reinforcement learning device 100 extracts all path segment combination pairs that can be combined with two path segments extracted from the path ranking list, and assigns a priority label based on the priority level corresponding to the path segment group to which the path segments included in the extracted path segment combination pair belong, thereby generating a priority pair including two path segments and the priority labels assigned to the two path segments.

[0122] For example, for the priority pair (σ i1 , σ i2 , l i ), the path segment σ i1 is an element of the path segment group g m , that is,

[0123]

Number

[0124]

Number

[0125] Next, the reinforcement learning device 100 can learn a compensation model by mathematical formulas 19, 24, and 25 based on the generated priority pairs, and execute reinforcement learning using the learned compensation model. Specifically, the reinforcement learning device 100 can update the parameters of the compensation model in the direction that minimizes the loss of the compensation model calculated by the loss function of mathematical formula 19 based on the priority pairs. Then, the reinforcement learning device 100 can execute reinforcement learning to search for an optimal policy that maximizes the cumulative discounted compensation in consideration of the Markov decision process (MDP) using the learned compensation model.

[0126] The reinforcement learning method according to an embodiment can construct a path ranking list in which the extracted overall path segments are sorted by priority levels, and generate more priority pairs even with a small number of path segments by generating priority pairs using the path pairs extracted from the path ranking list, and execute effective learning of the compensation model within the fixed feedback collection limit.

[0127] ​Furthermore, in one embodiment of the reinforcement learning method, since path segments are extracted from a path ranking list in which path segments are sorted according to priority level and priority level, the compensation model can learn the relative relationships between the generated priority pairs, i.e., secondary priorities, thereby improving the prediction accuracy of the compensation model.

[0128] Figures 5 and 6 illustrate the performance of a reinforcement learning method according to one embodiment.

[0129] Figure 5 shows the correlation between the estimated reward and the actual reward using a compensation model. Figure 5(a) relates to an existing priority-based reinforcement learning method, i.e., a compensation model learned based on priority pairs that include priority labels assigned based on priority feedback to two path segments extracted from a set of path segments, without using a path rank list, and without using a path rank list, while Figure 5(b) relates to a compensation model learned based on priority pairs generated using a path rank list generated by a reinforcement learning method according to one embodiment.

[0130] Referring to Figure 5, it can be seen that the correlation coefficient of the reinforcement learning method (b) according to one embodiment is higher than the correlation coefficient of the existing priority-based reinforcement learning method (a).

[0131] On the other hand, Figure 6 shows the performance of a reinforcement learning method according to one embodiment (the present invention) and existing reinforcement learning methods (MR, IPL, SeqRank) for specific tasks (Button-Press-Topdown, Box-Close, Dial-Turn). The Meta World medium-replay dataset was used to evaluate the performance.

[0132] In Figure 6, MR stands for Markovian Reward, which refers to a basic model trained on a multi-layer perceptron using a Markovian compensation process. IPL stands for Inverse Preference Learning (Hejna, J. et al. “Contrastive Preference Learning: Learning from Human Feedback without RL.” in arXiv preprint arXiv:2310.13639, 2023), which refers to a reinforcement learning method that learns policies without a compensation model. SeqRank (Sequential Preference Ranking) (Hwang et al. “Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback.” In Advances in Neural Information Processing Systems (NeurIPS), 2023) refers to a reinforcement learning method that sequentially collects priority feedback between newly observed segments and previously collected segments, and learns a compensation model based on this feedback.

[0133] In Figure 6, the number of feedbacks represents the total feedback collection limit. For example, 500 means that the total feedback collection limit is 500. In one embodiment of the reinforcement learning method, the sub-feedback limit is set to 100, so the path ranking list can contain 5 sub-ranking lists when the total feedback collection limit is 500, and can contain 10 sub-ranking lists when the total feedback collection limit is 1000.

[0134] Referring to Figure 6, it can be seen that the reinforcement learning method according to one embodiment shows superior performance compared to existing reinforcement learning methods in three tasks. In particular, it showed improved performance compared to other reinforcement learning methods when the number of feedbacks was small. This may mean that the reinforcement learning method according to one embodiment can effectively train a compensation model even with a small number of feedbacks.

[0135] The reinforcement learning method according to the embodiment described with reference to Figure 4 can also be embodied in the form of a computer-readable medium that stores computer-executable instructions and data. Here, the instructions and data can be stored in the form of program code, which, when executed by a processor, can generate a predetermined program module and perform a predetermined operation. The computer-readable medium may be any available medium accessible by a computer, and includes volatile and non-volatile media, and isolated and non-isolated media. The computer-readable medium may also be a computer recording medium. The computer recording medium may include volatile and non-volatile, isolated and non-isolated media embodied by any method or technique for storing computer-readable information such as instructions, data structures, program modules, or other data. For example, the computer recording medium may be a magnetic storage medium such as an HDD and an SSD, an optical recording medium such as a CD, DVD, and a Blu-ray disc, or memory contained in a server accessible via a network.

[0136] Furthermore, the reinforcement learning method described in the embodiment based on Figure 4 can also be embodied in a computer program (or computer program product) that includes programmable machine instructions processed by a processor, and can be embodied in a high-level programming language, object-oriented programming language, assembly language, or machine language. In addition, the computer program can be recorded on a type of computer-readable recording medium (e.g., memory, hard disk, magnetic / optical media, or SSD (Solid-State Drive)).

[0137] Therefore, the reinforcement learning method according to the embodiment described with reference to Figure 4 can be realized by executing the computer program described above on a computing device. The computing device may include a processor, memory, storage device, a high-speed interface connected to the memory and high-speed expansion port, and at least a portion of a low-speed interface connected to a low-speed bus and storage device. Each of these components is connected to one another using various buses and can be mounted on a common motherboard or in other suitable manner.

[0138] Here, the processor can process instructions within the computing device. Such instructions may include, for example, instructions stored in memory or storage devices for displaying graph information to provide a GUI (Graphical User Interface) on an external input and output device, such as a display connected to a high-speed interface. In other embodiments, a number of processors and / or a number of buses can be appropriately used together with a number of memories and memory forms. The processor can also be embodied in a chipset consisting of chips containing a number of independent analog and / or digital processors.

[0139] Memory also stores information within a computing device. For example, memory can consist of volatile memory units or a collection thereof. Alternatively, memory can consist of non-volatile memory units or a collection thereof. Furthermore, memory may be other forms of computer-readable media, such as magnetic or optical disks.

[0140] Furthermore, memory can provide computing devices with a large storage space. Memory may be a computer-readable medium or a configuration including such a medium, and may include, for example, devices in a SAN (Storage Area Network) or other configurations, and may be floppy disk drives, hard disk drives, optical disk drives, or tape drives, flash memory, or other similar semiconductor memory devices or device arrays.

[0141] The embodiments described above are illustrative, and those with ordinary skill in the art to which the embodiments belong will understand that they can be easily modified into other specific forms without altering the technical idea or essential features of the embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type can be implemented in a distributed manner, and similarly, components described as distributed can be implemented in a combined manner.

[0142] The scope of protection sought by this specification shall be determined by the claims set forth below rather than by the detailed description above, and shall be interpreted to include all forms of modification or variation derived from the meaning and scope of the claims and their equivalent concepts. [Explanation of symbols]

[0143] 100 Reinforcement Learning Devices 110 memory 120 Control Unit 130 Communications Department 140 Input / output section

Claims

1. A reinforcement learning device that performs reinforcement learning based on offline priority, Memory for storing the program and dataset for performing reinforcement learning, A reinforcement learning device comprising: one or more processors, which operates by executing a program stored in memory, which extracts path segments, constructs a path ranking list by repeatedly adding the path segments to a path ranking list sorted by priority level based on priority feedback for path pairs containing the path segments, and which causes a compensation model to learn based on priority pairs including two path segments extracted from the path ranking list and priority labels assigned to the two path segments.

2. The reinforcement learning apparatus according to claim 1, wherein the control unit collects priority feedback for the plurality of path segments in the form of ternary feedback.

3. The reinforcement learning apparatus according to claim 1, wherein the control unit assigns the priority label based on the difference in priority between the path segments included in the priority pair, and the priority label is in the form of ternary feedback.

4. The reinforcement learning apparatus according to claim 1, wherein the control unit generates the route ranking list by adding the extracted route segments to the route ranking list based on priority feedback for newly extracted route segments from the dataset and route segments already included in the route ranking list.

5. The reinforcement learning apparatus according to claim 1, wherein the control unit generates the sub-ranking list by repeatedly adding the path segments to the sub-ranking list based on priority feedback for the path pairs within the sub-feedback limits, based on a total feedback collection limit required to generate one path ranking list and sub-feedback limits set by dividing the total feedback collection limit, and generates a plurality of the sub-ranking lists within the total feedback collection limit to construct the path ranking list.

6. A reinforcement learning method performed by a reinforcement learning device, The process involves extracting route segments and constructing a route ranking list by repeatedly adding the route segments to a route ranking list sorted by priority level based on priority feedback for route pairs containing the route segments, and A reinforcement learning method comprising the step of training a compensation model based on priority pairs including two path segments extracted from the path ranking list and priority labels assigned to the two path segments.

7. The reinforcement learning method according to claim 6, wherein the construction step includes a step of collecting priority feedback for the plurality of path segments in the form of ternary feedback.

8. The learning step includes assigning priority labels based on the difference in priority between the path segments included in the priority pair, The reinforcement learning method according to claim 6, wherein the priority label is in the form of ternary feedback.

9. The reinforcement learning method according to claim 6, wherein the construction step includes determining a priority level based on priority feedback for newly extracted path segments from the dataset and path segments already included in the path ranking list, and adding the extracted path segments to the path ranking list based on the priority level.

10. The reinforcement learning method according to claim 6, wherein the construction step includes generating the sub-ranking list by repeatedly adding the path segments to the sub-ranking list based on priority feedback for the path pair within the sub-feedback limits, based on a total feedback collection limit required to generate one path ranking list and sub-feedback limits set by dividing the total feedback collection limit, and generating a plurality of the sub-ranking lists within the total feedback collection limit to construct the path ranking list.

11. A computer program executed by a reinforcement learning device and stored on a computer-readable recording medium for performing the method according to claim 6.

12. A computer-readable recording medium on which a computer program for performing the method described in claim 6 is recorded.

Images