Machine learning program, machine learning device, and machine learning system

The described system enhances reinforcement learning efficiency by generating dense rewards from sparse data, improving agent performance through automated reward design and correlation-based optimization, addressing interpretability and customization challenges.

JP2024162914APending Publication Date: 2024-11-21FUJITSU LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023078900
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-11
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing reinforcement learning methods face challenges in automating reward design, leading to inefficient learning due to difficulties in designing appropriate rewards, reduced interpretability, and performance issues when using deep neural networks, especially with semi-optimal expert data.

Method used

A machine learning system that collects expert behavior data in a sparse reward environment, calculates correlation coefficients between event types and revenue, normalizes and clips these coefficients to generate a dense reward function, and applies it in a learning environment to optimize agent behavior.

Benefits of technology

This approach improves learning efficiency by automating reward design, allowing non-experts to create interpretable and customizable rewards, enhancing the performance of reinforcement learning agents without relying on deep neural networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024162914000001_ABST
    Figure 2024162914000001_ABST
Patent Text Reader

Abstract

To provide a machine learning program, a machine learning device, and a machine learning system for improving learning efficiency of machine learning.SOLUTION: A machine learning program causes a computer to execute processing of acquiring a behavior of an expert between a predetermined number of episodes in a predetermined environment to which a sparse reward function is given, collecting, for every episode, information on types of events occurring on the basis of the behavior in the predetermined environment and an amount of the event for every type, and on a profit that is finally gained by the expert on the basis of the sparse reward function, calculating a correlation coefficient between the amount of the event and the profit for every type, and executing normalization and clipping for the correlation coefficient to generate a dense reward function to define a reward at occurrence of the event for every type.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a machine learning program, a machine learning device, and a machine learning system. [Background technology]

[0002] In recent years, machine learning called reinforcement learning has been widely adopted in various fields such as AI (Artificial Intelligence) in electronic games, autonomous vehicle control, and autonomous robot control. In reinforcement learning, learning is carried out to maximize the reward given to the actions of an agent, which is an operating subject, in a given environment.

[0003] In reinforcement learning, rewards are usually designed manually by experts with knowledge of reinforcement learning or by people familiar with the nature of the target task. Rewards are designed not only to determine the behavior that the agent is desired to acquire, but also to include rewards for indirect events that promote the acquisition of that behavior, and conversely, rewards that suppress undesirable behavior.

[0004] However, manual reward design is essentially a series of trial and error, and it can take a very long time to arrive at an optimal design. In addition, in the case of deep reinforcement learning, it is necessary to design it based on the characteristics of Deep Neural Networks (DNNs), and efficient learning is often difficult to achieve by simply rewarding desired behaviors, so it is generally difficult for people without knowledge of reinforcement learning to design rewards.

[0005] Therefore, research is being conducted into automating reward design in order to quickly design appropriate rewards. As an attempt to automate reward design, there is a technical field called reward shaping, which is one of the techniques that automatically converts sparse rewards into dense rewards. One method of realizing reward shaping is to automatically shape the sparse rewards obtained from the action sequence when the expert acts in the task into a dense reward based on the relationship between the action sequence and the sparse reward obtained there, and most of these are realized by learning using a deep neural network.

[0006] Here, sparse rewards refer to a small number of situations where rewards can be obtained, while dense rewards refer to a large number of situations where rewards can be obtained. For example, in a particular game, if you receive a reward for winning the game, the reward is sparse, whereas if you receive rewards for various moves during the game in addition to winning the game, the reward is dense.

[0007] In addition, a conventional technique for reinforcement learning has been proposed in which a set of state-action pairs leading up to the state-action pair immediately before the reward is obtained is classified into multiple categories and stored, and the expected reward value of each category of state-action pair is updated based on information after a certain time has passed to train a machine learning model. Another technique has been proposed to optimize a manufacturing process to maximize reward using a function that maps one or more additive manufacturing processes to the reward associated with a given manufacturing event. Another technique has been proposed to determine what action should be performed at a particular time based on past events and to reassess the importance of past events based on their contribution to future rewards. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2010-73200 A [Patent Document 2] Patent Publication No. 2021-8113 [Patent Document 3] Special Publication No. 2022-504739 Summary of the Invention [Problem to be solved by the invention]

[0009] However, in reality, it is difficult to obtain a solution that is as close to optimal as expected with reward shaping, and in many cases, only agents with performance significantly inferior to that of manually designed rewards are obtained. In addition, in reward shaping, the reward function is often expressed by a deep neural network, which has issues such as reduced human interpretability and difficulty in customization. Furthermore, some reward shaping methods assume that the given expert data is optimal, so performance when suboptimal expert data is given may not be guaranteed. Thus, even if reward shaping is used, it is difficult to automate the design of appropriate rewards. Therefore, it is difficult to improve the learning efficiency of machine learning.

[0010] In addition, in a technique that trains a machine learning model by updating the reward expectation value of each classification state-action pair based on information after a certain time has passed, the reward is only updated, and it is difficult to design the reward appropriately. In a technique that performs optimization using a function that maps one or more additive manufacturing processes to the reward, the reward is not designed, and it is difficult to set an appropriate reward. In a technique that reevaluates the importance of past events based on the contribution of past events to future rewards, the reward is not designed, and it is difficult to set an appropriate reward. Therefore, in neither technique is the appropriate reward automatically set, and it is difficult to improve the learning efficiency of machine learning.

[0011] The disclosed technology has been made in consideration of the above, and aims to provide a machine learning program, a machine learning device, and a machine learning system that improve the learning efficiency of machine learning. [Means for solving the problem]

[0012] In one aspect of the machine learning program, machine learning device, and machine learning system disclosed in the present application, the actions of an expert during a predetermined number of episodes in a predetermined environment given a sparse reward function are obtained, and for each episode, information is collected on the type of event that occurred in the specified environment based on the actions and the amount of each type of event, as well as the revenue ultimately obtained by the expert based on the sparse reward function, a correlation coefficient between the amount of the events and the revenue for each type is calculated, and normalization and clipping are performed on the correlation coefficient to generate a dense reward function that specifies the reward when the event for each type occurs. Effect of the Invention

[0013] In one aspect, the present invention can improve the learning efficiency of machine learning. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a system configuration diagram of a machine learning system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram of a machine learning device according to an embodiment. [Diagram 3] FIG. 3 is a flowchart illustrating an overview of a reward function generation process performed by the machine learning device according to the embodiment. [Figure 4] FIG. 4 is a flowchart of the data collection process. [Diagram 5] FIG. 5 is a flowchart of the synchronous distributed reinforcement learning process. [Figure 6] FIG. 6 is a flowchart of a search process in a synchronous distributed reinforcement learning process. [Figure 7] FIG. 7 is a flowchart of the learning process in the synchronous distributed reinforcement learning process. [Figure 8] FIG. 8 is a flowchart of the asynchronous distributed reinforcement learning process. [Figure 9] FIG. 9 is a diagram showing the hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, examples of the machine learning program, the machine learning device, and the machine learning system disclosed in the present application will be described in detail with reference to the drawings. Note that the machine learning program, the machine learning device, and the machine learning system disclosed in the present application are not limited to the following examples. EXAMPLES

[0016] Fig. 1 is a system configuration diagram of a machine learning system according to an embodiment. As shown in Fig. 1, the machine learning system 1 according to the embodiment includes, for example, a data collection device 2, a reward calculation device 3, an environment providing device 4, and a learning device 5. However, the data collection device 2, the reward calculation device 3, the environment providing device 4, and the learning device 5 can also be integrated into one device.

[0017] The data collection device 2 has a data acquisition environment 11. The data acquisition environment 11 provides an environment for an agent, which is a decision-making entity, to act. The data collection device 2 acquires a sparse reward function and sets it in the data acquisition environment 11. Then, the expert 20 observes the data acquisition environment 11, acquires an action determined from the observation result, and performs the action in the data acquisition environment 11. Then, the data collection device 2 acquires a profit obtained based on the sparse reward function from the execution result of the action and an event amount generated by the action. The event amount is various amounts of change included in an event, which is an event that occurs based on an action in the data acquisition environment 11. There are multiple types of events. The data collection device 2 accumulates rewards and event amounts for each type of event for the observations and actions sequentially performed by the expert 20 until the end of the episode.

[0018] Thereafter, the data collection device 2 calculates, from the rewards and event volume, a revenue, which is the total of rewards for each type of event for one episode, and an event sum, which is the total of the event volume.The data collection device 2 then holds the calculated revenue and event sum for each type of event for one episode.The data collection device 2 repeats the above data collection process for each of a predetermined number of episodes, and collects and stores the revenue and event sum for each of the predetermined number of episodes.

[0019] The reward calculation device 3 executes the following process by operating the reward calculation algorithm 13. The reward calculation device 3 acquires the revenue and event total for each predetermined number of episodes from the data collection device 2. Next, the reward calculation device 3 calculates the correlation coefficient with the revenue for each type of event. Next, the reward calculation device 3 normalizes the calculated correlation coefficient. After that, the reward calculation device 3 applies clipping to the correlation coefficient to generate a reward function for the type of event. After that, the reward calculation device 3 outputs the reward function for the type of event generated by the reward calculation algorithm 13 to the environment providing device 4. This reward function is a dense reward function.

[0020] The environment providing device 4 holds a learning environment 12 that provides an environment equivalent to the data acquisition environment 11. The environment providing device 4 receives an input of a reward function for each event from the reward calculation device 3. Then, the environment providing device 4 sets the reward function for each acquired event in the learning environment 12.

[0021] Here, the learning environment 12 and the data acquisition environment 11 may be the same, or may not be completely the same environments as long as the contents provided are the same, that is, the contents of the observation space that represents the state of the environment and the action space that represents the actions taken in the observation space are the same. For example, the learning environment 12 and the data acquisition environment 11 can each be an environment for playing shogi.

[0022] The learning device 5 operates the reinforcement learning algorithm 14 to train an agent that performs an action in the learning environment 12. For example, the learning device 5 observes the learning environment 12. Then, the learning device 5 determines an action to be performed based on the observation. Next, the learning device 5 inputs the determined action to the learning environment 12. Then, the learning device 5 obtains a reward calculated for the action result using a dense reward function from the learning environment 12. The learning device 5 adjusts the parameters 15 that define the agent's action so that the reward for the action selected by the agent is as large as possible. In this way, the learning device 5 can generate an agent that obtains a solution close to the optimum.

[0023] Fig. 2 is a block diagram of a machine learning device according to an embodiment. Fig. 2 illustrates a case in which the data collection device 2, the reward calculation device 3, the environment providing device 4, and the learning device 5 in Fig. 1 are integrated into one machine learning device 100.

[0024] As shown in Fig. 2, the machine learning device 100 has a data collection unit 101 corresponding to the data collection device 2, a reward calculation unit 102 corresponding to the reward calculation device 3, and a learning unit 103 corresponding to the environment providing device 4 and the learning device 5. The operation of the machine learning device 100 will be described in detail below with reference to Fig. 2. Below, a configuration in which the devices of the machine learning system 1 are integrated into one will be described using the machine learning device 100 as an example, but the same applies to the devices being different from each other.

[0025] The data collection unit 101 has a data acquisition environment 11. The data collection unit 101 is also connected to an expert 20. In this embodiment, the expert 20 is a rule-based agent, which is an AI that determines behavior based on pre-registered rules that determine behavior from observation. Alternatively, the expert 20 may be a human being who has expert knowledge about behavior in the data acquisition environment 11. The data collection unit 101 uses the data acquisition environment 11 to perform the following data collection process.

[0026] The data collection unit 101 acquires a sparse reward function and sets it in the data acquisition environment 11. For example, the data acquisition environment 11 can be an environment simulating a shogi game. In this case, the sparse reward function is, for example, a function that awards 1 point for a win and -1 point for a loss. In addition, one episode in this case is one game from the start of the game to checkmate.

[0027] The data collection unit 101 initializes the data acquisition environment 11. Next, the data collection unit 101 collects the observation data from the data acquisition environment 11. t Obtain the observation t More specifically, the state of the environment obtained as a result of observing the data acquisition environment 11. Here, t is an episode time that indicates a specific time in an episode. When the episode length that indicates the time length of an episode is T, the episode time includes times t=1 to T. Also, the observation t is the observation taken at time t.

[0028] Next, the data collection unit 101 sends the observation t Enter the observations from Expert 20 t Actions against a t Get Action a t is the action generated by the expert 20 at time t.

[0029] Next, the data collection unit 101 inputs the action a to the data acquisition environment 11. t Then, the data collector 101 inputs the reward r obtained from the sparse reward function. t =r sparse (o t ,a t ) where the reward r t is the reward obtained at time t. Also, r sparse represents the given sparse reward function.

[0030] The data collection unit 101 also collects K types of event quantities e t k(k=1, , K). The event amount is the amount of change contained in the event that occurred in the data acquisition environment 11. The event amount e t k is the k-th event amount obtained at time t. In this case, there are K types of events. In the following, each of the K types of events will be called the k-th event. An event may be discrete information in which one occurrence is counted as 1, or it may be continuous information in which a magnitude is assigned to each occurrence.

[0031] For example, when the data acquisition environment 11 is a machine learning model of a shogi move, the observation space in the data acquisition environment 11 is the state of the board. Also, the action space in the data acquisition environment 11 is the next move. In this case, the data collection unit 101 can collect information on the type of event that occurred and the amount of events in each event by having the rule-based agents play against each other. Here, the type of event includes the movement of each type of piece such as "advancing a pawn", the acquisition of an opponent's piece such as "acquiring an opponent's bishop", the use of a hand piece such as "using a knight that is a hand piece", and the like. Also, the amount of events is the number of times or the number of times each type of event occurred, for example, information such as "advancing a rook" of 5 squares and "acquiring an opponent's gold" of 2 squares.

[0032] Next, the data collection unit 101 calculates the agent's earnings by the end of one episode using the reward calculated for the following formula (1). The earnings is the cumulative reward obtained by adding up all the rewards acquired by the agent. Here, the earnings R i is the cumulative reward in the i-th episode.

[0033]

number

[0034] In addition, the data collecting unit 101 calculates the sum of events that occurred in one episode using the amount of each type of event calculated using the following formula (2). The sum of events is the sum of the event amounts for each event. Here, the sum of events E t k is the sum of the amount of the kth event in the ith episode. Hereinafter, i may be called the episode number.

[0035]

number

[0036] Then, the data collection unit 101 stores the calculated earnings and event sums for one episode. The data collection unit 101 repeats the calculation and storage of the earnings and event sums a predetermined number of times, which is the number of episodes that has been determined in advance. In this way, the data collection unit 101 collects earnings and event sums resulting from the actions of the expert 20 in the data acquisition environment 11 for the determined number of episodes.

[0037] As described above, the data collection unit 101 collects, for each episode, information on the type of events that occurred based on the actions in the predetermined environment, the amount of each type of event, and the profit finally acquired by the expert 20 based on the sparse reward function. Here, the data acquisition environment 11 is an example of the predetermined environment. Also, the data acquisition environment 11 is an example of the "first environment."

[0038] The reward calculation unit 102 generates a dense reward function using the profit and the event sum collected by the data collection unit 101. The reward calculation unit 102 includes a correlation coefficient calculation unit 121, a normalization unit 122, and a reward function generation unit 123.

[0039] The correlation coefficient calculation unit 121 acquires the revenue and event total for each of a predetermined number of episodes from the data collection unit 101. Next, the correlation coefficient calculation unit 121 calculates the correlation coefficient c(k) between the event total for each event type and the corresponding revenue. The correlation coefficient c(k) is generally expressed by the following formula (3). Here,<a,b> represents the correlation coefficient between a and b.

[0040]

number

[0041] For example, the correlation coefficient calculation unit 121 can calculate the correlation coefficient c(k) using the Pearson product-moment correlation coefficient shown in the following formula (4). Here, c(k) represents the reward coefficient for the k-th event. In this case, (X, Y)=(R i ,E t k ) where σ represents the standard deviation and cov represents the covariance.

[0042]

number

[0043] The normalization unit 122 obtains the correlation coefficient for each event from the correlation coefficient calculation unit 121. Then, the normalization unit 122 normalizes the correlation coefficient using the following formula (5). In this way, the normalization unit 122 adjusts the reward so that it does not exceed the original sparse reward, thereby preventing the cobra effect. Here, c'(k) represents the normalized reward coefficient for the kth event.

[0044]

number

[0045] The reward function generator 123 obtains the normalized reward coefficient from the normalizer 122. Then, the reward function generator 123 generates a reward function using the following formula (6): dense(k, a) is a shaped dense reward function that represents the reward to be given when the kth event occurs with an event amount of a.

[0046]

number

[0047] Here, the reward given by the formula (5) includes a small reward. Although a small reward also has the effect of promoting learning, if the reward value is too small, it may be rounded off when a simple correlation function is used. Therefore, the reward function generation unit 123 generates a dense correlation function by clipping the reward so that its absolute value is equal to or greater than ε, as shown in the formula (6). The reward function generated by the reward function generation unit 123 can utilize small but useful information without rounding it off. After that, the reward function generation unit 123 sets the generated reward function in the learning environment 12.

[0048] Equation (6) is designed based on the idea that events that occurred particularly frequently in episodes in which the expert 20 achieved high results are likely to be events that are beneficial for learning. In other words, in equation (6), a large reward is assigned to an event that the expert 20 experienced a lot when the expert 20 achieved a large profit in the data acquisition environment 11 in which only sparse rewards are defined. Conversely, in equation (6), a small reward is assigned to an event that the expert 20 experienced a lot when the expert 20 achieved a small profit.

[0049] The learning unit 103 has a learning environment 12. The learning environment 12 is an environment for learning an agent. A dense reward function generated by the reward function generation unit 123 of the reward calculation unit 102 is set in the learning environment 12. This learning environment 12 is an example of a "second environment." In addition, the learning unit 103 has a reinforcement learning execution unit 131 and a parameter storage unit 132.

[0050] The reinforcement learning execution unit 131 observes the learning environment 12 and gives the agent information on the state of the learning environment 12, which is the observation result, to determine an action. The reinforcement learning execution unit 131 then inputs the action determined by the agent to the learning environment 12 and obtains a reward for the action from the learning environment 12. The reinforcement learning execution unit 131 repeats observation, action determination, and reward acquisition. The reinforcement learning execution unit 131 then adjusts the parameters 15 so that the reward from the action selected by the agent is as large as possible. Then, when learning is completed, the reinforcement learning execution unit 131 stores the parameters 15 that maximize the reward obtained by learning in the parameter storage unit 132. The agent can select an action close to an optimal action by determining an action using the parameters 15 stored in the parameter storage unit 132.

[0051] 3 is a flowchart showing an overview of the reward function generation process performed by the machine learning device 100 according to the embodiment. Next, the flow of the reward function generation process performed by the machine learning device 100 according to the embodiment will be described with reference to FIG.

[0052] The data collection unit 101 executes a data collection process using the data acquisition environment 11, and collects the earnings and event totals due to the actions of the expert 20 for a determined number of episodes (step S1).

[0053] The correlation coefficient calculation unit 121 acquires the revenue and event sum for a determined number of episodes from the data collection unit 101. Next, the correlation coefficient calculation unit 121 calculates the correlation coefficient c(k) between the event sum for each type of event shown in Equation (3) and the revenue obtained by the corresponding sparse reward function (step S2).

[0054] The normalization unit 122 obtains the correlation coefficient for each event type from the correlation coefficient calculation unit 121. Then, the normalization unit 122 normalizes the correlation coefficient using Equation (5) (step S3).

[0055] The reward function generation unit 123 obtains the normalized reward coefficient from the normalization unit 122. Then, the reward function generation unit 123 generates a dense reward function using Equation (6) (step S4). After that, the reward function generation unit 123 sets the generated dense reward function in the learning environment 12.

[0056] The reinforcement learning execution unit 131 executes learning of the agent using the learning environment 12 in which a dense reward function is set (step S5). After that, the reinforcement learning execution unit 131 stores the parameters adjusted by the learning in the parameter storage unit 132.

[0057] Fig. 4 is a flowchart of the data collection process. Each process shown in the flow of Fig. 4 corresponds to an example of the process executed in step S1 of Fig. 3.

[0058] The data collection unit 101 sets a sparse reward function in the data acquisition environment 11 (step S101).

[0059] Next, the data collection unit 101 initializes the episode number i to 1 (step S102). Here, the total number of episodes, which is a predetermined number, is I.

[0060] Next, the data collection unit 101 initializes the data acquisition environment 11 (step S103).

[0061] Next, the data collecting unit 101 initializes the time t to 1 (step S104). Here, the predetermined episode length is T.

[0062] Next, the data collection unit 101 collects the observed data from the data acquisition environment 11. t (Step S105).

[0063] Next, the data collection unit 101 sends the observation t Enter the observations from Expert 20 t Actions against a t is obtained (step S106).

[0064] Next, the data collection unit 101 inputs the action a to the data acquisition environment 11. t (Step S107).

[0065] Then, the data collection unit 101 calculates the action a obtained from the sparse reward function. t Reward for r t =r sparse (o t ,a t ) from the data acquisition environment 11. The data collection unit 101 also acquires the amount of K types of events e t k (k=1, . . . , K) is acquired from the data acquisition environment 11 (step S108).

[0066] Next, the data collection unit 101 sums up the rewards calculated for formula (1) to calculate the agent's earnings until the end of the episode with episode number i. The data collection unit 101 also sums up the event amounts for each type of event calculated for formula (2) to calculate the total event amount for each type of event that occurred in the episode with episode number i (step S109).

[0067] Then, the data collection unit 101 stores the calculated profit and event total for one episode (step S110).

[0068] Next, the data collection unit 101 determines whether t is smaller than T (step S111). If t is smaller than T (step S111: Yes), the data collection unit 101 increments t by 1 (step S112). After that, the data collection unit 101 returns to step S105.

[0069] On the other hand, if t has reached T (step S111: No), the data collection unit 101 determines whether i is smaller than I (step S113). If i is smaller than I (step S113: Yes), the data collection unit 101 increments i by 1 (step S114). Thereafter, the data collection unit 101 returns to step S103. On the other hand, if i has reached I (step S113: No), the data collection unit 101 ends the data collection process.

[0070] Next, an example of reinforcement machine learning processing executed by the reinforcement learning execution unit 131 using the learning environment 12 in which the dense reward function generated by the reward function generation unit 123 is set will be described. FIG. 5 is a flowchart of the synchronous distributed reinforcement learning processing. For example, the reinforcement learning execution unit 131 learns an agent by executing the synchronous distributed reinforcement learning processing shown in FIG. 5. Next, the flow of the synchronous distributed reinforcement learning processing by the reinforcement learning execution unit 131 will be described with reference to FIG. 5.

[0071] The reinforcement learning execution unit 131 starts a learning process of synchronous distributed reinforcement learning, and initializes parameters of a machine learning model that determines the behavior of an agent (step S201).

[0072] Next, the reinforcement learning execution unit 131 creates a predetermined number of environments based on the learning environment 12 and starts a predetermined number of search processes (step S202). For example, the learning environment 12 is replicated and assigned to each search process.

[0073] Next, the reinforcement learning execution unit 131 executes a distributed search process (step S203). In the execution of the distributed search process in step S203, each search process executes a search process to generate experience data (step S231).

[0074] Next, the reinforcement learning execution unit 131 executes a learning process of synchronous distributed reinforcement learning using the experience data acquired from each of the predetermined number of search processes (step S204).

[0075] The reinforcement learning execution unit 131 determines whether a predetermined synchronous learning end condition, such as whether learning for k epochs has been completed, has been satisfied (step S205). If the synchronous learning end condition has not been satisfied (step S205: No), the reinforcement learning execution unit 131 returns to step S203. On the other hand, if the synchronous learning end condition has been satisfied (step S205: Yes), the reinforcement learning execution unit 131 ends the learning process.

[0076] Fig. 6 is a flowchart of the search process in the synchronous distributed reinforcement learning process. Each process executed in the flow of Fig. 6 corresponds to an example of the process executed in step S231 of Fig. 5. Next, the flow of the search process by each search process operated by the reinforcement learning execution unit 131 in the synchronous distributed reinforcement learning process will be described with reference to Fig. 6.

[0077] Next, the search process updates the parameters held by the machine learning model that determines the behavior of the agent with the latest parameters (step S301).

[0078] Next, the search process resets the state of the allocated learning environment 12 (step S302).

[0079] Next, the search process acquires information on the observation of the learning environment 12 (step S303).

[0080] Next, the search process uses the held machine learning model to select an action to be taken by the agent in the learning environment 12, inputs the action to the learning environment 12, and causes the agent to execute the selected action (step S304).

[0081] Next, the exploration process obtains a reward calculated using a dense reward function for the selected action from the learning environment 12 (step S305).

[0082] Then, the exploration process adds experience data including the environmental data of the learning environment 12, the observation data before the action, the action data, and the reward data to a replay buffer of the reinforcement learning execution unit 131 (step S306).

[0083] Next, the search process determines whether the environment has reached the end state (step S307). If the environment has not reached the end state (step S307: No), the search process returns to step S303.

[0084] On the other hand, if the environment has reached the end state (step S307: Yes), the search process determines whether or not the search has been completed a predetermined number of times (step S308). If the search has not been completed a predetermined number of times (step S308: No), the search process returns to step S302.

[0085] On the other hand, if the predetermined number of searches have been completed (step S308: Yes), the search process ends the search process.

[0086] Fig. 7 is a flowchart of the learning process in the synchronous distributed reinforcement learning process. Each process executed in the flow of Fig. 7 corresponds to an example of the process executed in step S204 in Fig. 5. Next, the flow of the learning process by the reinforcement learning execution unit 131 in the synchronous distributed reinforcement learning process will be described with reference to Fig. 7.

[0087] The reinforcement learning execution unit 131 receives a notification of the end of the search process from each search process. Then, the reinforcement learning execution unit 131 acquires the experience data generated by all the data generation units 200 in which the search process has been started from the replay buffer (step S401).

[0088] Next, the reinforcement learning execution unit 131 calculates a value function for maximizing the total reward in the machine learning model that determines the behavior of the agent (step S402).

[0089] Next, the reinforcement learning execution unit 131 uses the accumulated value of rewards obtained by the calculation of the value function to calculate a policy gradient for obtaining an optimal action policy in the machine learning model (step S403).

[0090] Then, the reinforcement learning execution unit 131 updates the parameters of the machine learning model according to the calculated policy gradient. After that, the reinforcement learning execution unit 131 updates the parameters stored in the parameter buffer with the updated parameters (step S404).

[0091] Fig. 8 is a flowchart of an asynchronous distributed reinforcement learning process. The process executed in the flow shown in Fig. 8 corresponds to another example of the reinforcement machine learning process executed by the reinforcement learning execution unit 131. Next, the flow of the asynchronous distributed reinforcement learning process by the reinforcement learning execution unit 131 according to this embodiment will be described with reference to Fig. 8.

[0092] The reinforcement learning execution unit 131 starts a learning process of asynchronous distributed reinforcement learning, and initializes parameters of a machine learning model that determines the behavior of an agent (step S501).

[0093] Next, the reinforcement learning execution unit 131 creates a predetermined number of environments based on the learning environment 12 and starts a predetermined number of search processing processes (step S502).

[0094] Next, the reinforcement learning execution unit 131 executes an asynchronous learning process (step S503). In the execution of the asynchronous learning process in step S503, a learning process is executed by a learning process (step S531) and a search process is executed by a predetermined number of search processes (step S532).

[0095] After that, the reinforcement learning execution unit 131 determines whether the learning satisfies a final end condition (step S504). If the final end condition is not satisfied (step S504: No), the reinforcement learning execution unit 131 returns to step S503. On the other hand, if the final end condition is satisfied (step S504: Yes), the reinforcement learning execution unit 131 ends the learning process of the asynchronous distributed reinforcement learning.

[0096] In the learning process in step S531 and the search process in step S532, processes equivalent to the search process and learning process shown in each flow of Figures 6 and 7 are executed. However, in asynchronous learning, the objective function is calculated using the experience data updated by the learning process and stored in the replay buffer, and learning proceeds.

[0097] The reinforcement learning execution unit 131 can use the following objective function in learning the machine learning model for determining the behavior of the agent. For example, the reinforcement learning execution unit 131 can train the machine learning model by a value iteration method using the following formula (7).

[0098]

number

[0099] where D is the set of experiences used for learning. t is the observation at time t. t is the action actually taken at time t. t is the reward given as a result of taking action at at time t. (t+1) is the action a at time t. t This is the next observation that transitioned as a result of taking a (t+1) is the optimal action to be taken at time t+1. γ is the discount rate. Specifically, when γ=1, long-term profits are most important, and when γ=1, short-term profits are most important. Q θ is the action value function expressed in terms of the parameter θ. "| |2" is the square norm.

[0100] In this case, the objective function L TD represents an error. That is, the reinforcement learning execution unit 131 TD The machine learning model is trained to minimize this.

[0101] Alternatively, the reinforcement learning execution unit 131 can cause the machine learning model to learn using the Actor-Critic type policy gradient method by using the following formula (8).

[0102]

number

[0103] where D is the set of experiences used for learning. t is the observation at time t. t is the action actually taken at time t. θ is a policy function expressed by the parameter θ. Also, β is a weighting coefficient. In this case, L AC is the objective function.

[0104] In this case, L PG represents the expected value of the profit. Also, the objective function L TD represents the error. That is, the reinforcement learning execution unit 131 PG Make as large as possible and L TD The machine learning model is trained to minimize this.

[0105] Here, the parameter θ is often common to the policy function π and the action value function Q, but the reinforcement learning execution unit 131 may completely separate the policy function π and the action value function Q and execute learning as different deep neural networks. In that case, the reinforcement learning execution unit 131 uses the policy function π as L PG The action value function Q is updated as L TD to update.

[0106] As described above, the machine learning system and the machine learning device according to the present embodiment repeat a predetermined number of episodes in which an expert selects an action in a data acquisition environment and obtains a reward based on a sparse reward function. Then, the machine learning system and the machine learning device collect information on events that occurred in the data acquisition environment in each episode and the profit finally acquired by the expert using the sparse reward function, and calculate a correlation coefficient between the amount of events and the profit. Then, the machine learning system and the machine learning device normalize and clip the calculated correlation coefficient to generate a dense reward function. Then, the machine learning system and the machine learning device set the generated dense reward function in the learning environment to train a machine learning model that determines the behavior of an agent.

[0107] This makes it possible to automate reward design. In addition, even those without specialized knowledge of reinforcement learning can design suboptimal rewards. In addition, since the reward function is generated without using DNN learning, the generated reward function is highly interpretable by humans and easy to customize. Furthermore, there is no need to train an additional machine learning model as in reward shaping, and the cost of application can be kept low compared to reward shaping.

[0108] (Modification) In the above-described embodiment, the correlation coefficient calculation unit 121 calculates the correlation coefficient between the sum of the amounts of each event and the corresponding profit using Pearson's product-moment correlation coefficient, which is a so-called ordinary correlation coefficient, but it is also possible to use other correlation coefficients depending on the characteristics of the target environment.

[0109] For example, the correlation coefficient calculation unit 121 can calculate the correlation coefficient c(k) using the Spearman's rank correlation coefficient shown in the following equation (9).

[0110]

number

[0111] Here, D represents the difference in rank between the corresponding values ​​of X and Y. In this case, (X, Y) = (R i ,E t k ), where N represents the number of pairs of values. For example, when the magnitude relationship represented by the given sparse reward function is an ordinal scale, it is preferable that the correlation coefficient calculation unit 121 calculates the correlation coefficient c(k) using this Spearman's rank correlation coefficient. The ordinal scale refers to a scale in which the order has meaning but the magnification has no meaning. For example, in the ordinal scale, when the first condition with a reward of "+1" and the second condition with a reward of "+2" are compared, the second condition is ranked higher than the first condition, but it cannot be said that the second condition is twice as good as the first condition.

[0112] Alternatively, the correlation coefficient calculation unit 121 can calculate the correlation coefficient c(k) using a partial correlation coefficient shown in the following formula (10). by-a represents the correlation between b and y, excluding the effect of factor a.

[0113]

number

[0114] Alternatively, the correlation coefficient calculation unit 121 can calculate the correlation coefficient c(k) using a coefficient of linear regression. In this case, the correlation coefficient calculation unit 121 can use various calculation methods such as the least squares method, Lasso regression, and Ridge regression. The correlation coefficient calculation unit 121 can also use an advanced type such as logistic regression.

[0115] As explained above, the correlation coefficient can be calculated by using a correlation coefficient according to the target environment to calculate the correlation coefficient between the sum of the amount of each event and the corresponding revenue. In this way, by using a correlation coefficient according to the target environment, a more appropriate correlation coefficient can be calculated, making it possible to more appropriately design rewards.

[0116] (Hardware configuration) Fig. 9 is a hardware configuration diagram of a computer. Next, an example of a hardware configuration for realizing each function of the machine learning device 100 will be described with reference to Fig. 9. The functions of each unit of the machine learning device 100 can be realized by a computer 90 shown in Fig. 9.

[0117] 7, a computer 90 includes, for example, a CPU (Central Processing Unit) 91, a memory 92, a hard disk 93, and a network interface 94. The CPU 91 is connected to the memory 92, the hard disk 93, and the network interface 94 via a bus.

[0118] The network interface 94 is an interface for communication between the computer 90 and an external device. The network interface 94 relays communication between the machine learning device 100 and the expert 20, which is a rule-based agent, for example.

[0119] The hard disk 93 is an auxiliary storage device. The hard disk 93 may store the data acquisition environment 11 and the learning environment 12 illustrated in Fig. 2. The hard disk 93 also stores a nuclide program including a program for realizing the functions of the data collection unit 101, the remuneration calculation unit 102, and the learning unit 103 illustrated in Fig. 2.

[0120] The memory 92 is a main storage device, and may be, for example, a dynamic random access memory (DRAM).

[0121] The CPU 91 reads out various programs from the hard disk 93, and deploys and executes them in the memory 92. In this way, the CPU 91 realizes the functions of the data collection unit 101, the reward calculation unit 102, and the learning unit 103 illustrated in FIG.

[0122] In the case of the machine learning system 1 shown in Fig. 1, the data collection device 2, the reward calculation device 3, the environment providing device 4, and the learning device 5 are each realized by a computer 90 shown in Fig. 9. For example, the reward calculation algorithm 13 and the reinforcement learning algorithm 14 are operated by a CPU 91. [Explanation of symbols]

[0123] 1. Machine Learning Systems 2. Data collection equipment 3 Reward calculation device 4 Environment provision device 5 Learning Device 11 Data Acquisition Environment 12 Learning Environment 13 Reward calculation algorithm 14 Reinforcement Learning Algorithms 15 Parameters 20 Expert 100 Machine Learning Device 101 Data Collection Department 102 Remuneration Calculation Department 103 Learning Department 121 Correlation coefficient calculation section 122 Normalization section 123 Reward function generation unit 131 Reinforcement Learning Execution Unit 132 Parameter storage section

Claims

1. Obtain the expert's actions for a given number of episodes in a given environment given a sparse reward function; For each episode, information is collected regarding the type of events and the amount of each type of event that occurred in the specified environment based on the actions, and the profit that the expert ultimately obtained based on the sparse reward function; Calculating a correlation coefficient between the volume of the event and the revenue for each type; The correlation coefficients are normalized and clipped to generate a dense reward function that defines the reward for each type of event. A machine learning program that causes a computer to execute processing.

2. generating the dense reward function that calculates a reward according to the amount of each type of event when the event occurs; Generate the dense reward function that calculates a reward to be given when a specific event occurs, depending on the type of the specific event and the amount of the specific event. The machine learning program according to claim 1, characterized in that the program causes the computer to execute processing.

3. A machine learning model is caused to determine an action according to a state of a learning environment in which the dense reward function is set; Calculating a reward based on the dense reward function according to a type of event and an amount of the event that occurred based on the determined action in the learning environment; The machine learning model is trained based on the determined action and the reward for the determined action. The machine learning program according to claim 1, further comprising causing the computer to execute a process.

4. The machine learning program according to claim 1, characterized in that the computer is caused to execute a process of obtaining the expert's actions in the specified environment during the specified number of episodes based on pre-registered rules indicating the relationship between the state and actions of the specified environment given the sparse reward function.

5. a data collection unit that acquires the actions of an expert during a predetermined number of episodes in a predetermined environment to which a sparse reward function is given, and collects, for each episode, information on the type of events that occurred in the predetermined environment based on the actions and the amount of each type of event, as well as the profit that the expert ultimately earned based on the sparse reward function; a reward calculation unit that calculates a correlation coefficient between the amount of the event and the profit for each type, and performs normalization and clipping on the correlation coefficient to generate a dense reward function that specifies a reward for each type when the event occurs; A machine learning device comprising:

6. A machine learning system having a data collection device and a reward calculation device, A data collection device acquires actions of an expert during a predetermined number of episodes in a first environment to which a sparse reward function is given, and collects, for each episode, information on the type of events and the amount of each type of event that occurred in the first environment based on the actions, and the profit that the expert ultimately earned based on the sparse reward function; The reward calculation device calculates a correlation coefficient between the amount of the event and the profit for each type, normalizes and clips the correlation coefficient to generate a dense reward function that specifies a reward at the time of occurrence of the event for each type, and sets the dense reward function to a second environment that has the same observation space and action space as the first environment and is used for learning a machine learning model that determines actions within the second environment. A machine learning system characterized by:

Citation Information

Patent Citations

  • Learning system and learning method

    JP2010073200A

  • Real-time adaptive control of additive manufacturing processes using machine learning

    JP2021008113A

  • Controlling agents over long timescales using temporal value transfer

    JP2022504739A