Reinforcement learning system, reinforcement learning device, reinforcement learning method and program

Abstract

In order to select a more suitable action, this reinforced learning system (1) comprises an acquisition unit (11) for acquiring a first state in an environment that is the object of reinforced learning, a generation unit (12) for generating a second state by adding noise to the first state, a calculation unit (13) for calculating a first action value function in accordance with the second state, and a selection unit (14) for selecting an action in accordance with the first action value function.

Description

[Technical field]

[0001] The present invention relates to a reinforcement learning system, a reinforcement learning device, and a reinforcement learning method. [Background technology]

[0002] Research is being conducted into reinforcement learning, which learns actions that maximize rewards obtained when the next action is performed in a certain state. Patent Document 1 describes a technology that uses reinforcement learning to learn images of concave and convex parts and control amounts for assembling parts in assembly work using a robot arm. Patent Document 2 describes a technology that uses reinforcement learning to learn accelerator operation amounts and select actions consisting of throttle opening command values ​​and retardation amounts according to states. Patent Document 2 also describes that a function approximator may be used for the action value function Q. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 146770 [Patent Document 2] Japanese Patent Publication No. 2021-67193 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the techniques described in Patent Documents 1 and 2 have room for improvement in terms of selecting a more suitable action. If the action value function can be accurately estimated in reinforcement learning, an appropriate action can be selected, but the action value function estimated in the techniques described in Patent Documents 1 and 2 contains errors. It is difficult to accurately estimate the action value function, especially when the state-action space is huge.

[0005] One aspect of the present invention has been made in consideration of the above problems, and one example of a purpose of the present invention is to provide a technique that enables a more suitable action to be selected. [Means for solving the problem]

[0006] A reinforcement learning system according to one aspect of the present invention includes an acquisition means for acquiring a first state in an environment that is the subject of reinforcement learning, a generation means for generating a second state by adding noise to the first state, a calculation means for calculating a first action value function according to the second state, and a selection means for selecting an action according to the first action value function.

[0007] A reinforcement learning device according to one aspect of the present invention includes an acquisition means for acquiring a first state in an environment that is the subject of reinforcement learning, a generation means for generating a second state by adding noise to the first state, a calculation means for calculating a first action value function according to the second state, and a selection means for selecting an action according to the first action value function.

[0008] A reinforcement learning method according to one aspect of the present invention includes generating a second state by adding noise to the first state in an environment that is the subject of reinforcement learning, calculating a first action value function according to the second state, and selecting an action according to the first action value function. A program according to one aspect of the present invention is a program for causing a computer to function as a reinforcement learning device, causing the computer to function as: acquisition means for acquiring a first state in an environment that is the subject of reinforcement learning; generation means for generating a second state by adding noise to the first state; calculation means for calculating a first action value function according to the second state; and selection means for selecting an action according to the first action value function. Effect of the Invention

[0009] According to one aspect of the present invention, a more suitable action can be selected. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a configuration of a reinforcement learning system according to a first exemplary embodiment of the present invention. [Diagram 2] 1 is a flow diagram showing a flow of a reinforcement learning method according to an exemplary embodiment 1 of the present invention. [Diagram 3]FIG. 1 is a block diagram illustrating a configuration of a reinforcement learning system according to a first exemplary embodiment of the present invention. [Figure 4] FIG. 1 is a block diagram showing an example of a device configuration for implementing the first exemplary embodiment of the present invention. [Diagram 5] FIG. 11 is a block diagram showing a configuration of a reinforcement learning system according to an exemplary embodiment 2 of the present invention. [Figure 6] FIG. 11 is a flow chart showing the flow of a reinforcement learning method according to an exemplary embodiment 2 of the present invention. [Figure 7] FIG. 11 is a diagram showing an example of a game screen according to an exemplary embodiment 3 of the present invention. [Figure 8] FIG. 11 is a diagram illustrating a first state according to an exemplary embodiment 3 of the present invention. [Figure 9] FIG. 13 is a diagram showing an example of an evaluation result according to an application example of the exemplary embodiment 3 of the present invention. [Figure 10] FIG. 13 is a diagram showing an example of an evaluation result according to an application example of the exemplary embodiment 3 of the present invention. [Figure 11] FIG. 13 is a diagram showing an example of an evaluation result according to an application example of the exemplary embodiment 3 of the present invention. [Figure 12] FIG. 13 is a diagram showing an example of an evaluation result according to an application example of the exemplary embodiment 3 of the present invention. [Figure 13] FIG. 1 is a block diagram showing the configuration of a reinforcement learning device, a terminal 20, and a computer functioning as a server 30 according to first to seventh exemplary embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Example Embodiment 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0012] <Structure of Reinforcement Learning System> The configuration of a reinforcement learning system 1 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the reinforcement learning system 1. The reinforcement learning system 1 is a system that selects an action by reinforcement learning. The reinforcement learning system 1 is, for example, a system that controls the construction operation of a construction machine such as an excavator, a system that controls transportation by a transportation device, or a system for autonomous play of a computer game. However, the reinforcement learning of the reinforcement learning system 1 is not limited to the above examples, and the reinforcement learning performed by the reinforcement learning system 1 is applicable to various systems. The action is the action of an agent in reinforcement learning, and for example, the excavation operation control of an excavator, the transportation operation control of a transportation device, or the autonomous play control of a computer game. However, the action is not limited to these examples, and may be other than the above.

[0013] As shown in Fig. 1, the reinforcement learning system 1 includes an acquisition unit 11, a generation unit 12, a calculation unit 13, and a selection unit 14. The acquisition unit 11 is configured to realize the acquisition means in this exemplary embodiment. The generation unit 12 is configured to realize the generation means in this exemplary embodiment. The calculation unit 13 is configured to realize the calculation means in this exemplary embodiment. The selection unit 14 is configured to realize the selection means in this exemplary embodiment.

[0014] The acquisition unit 11 acquires a first state. The first state is a state in an environment that is a target of reinforcement learning. For example, when the reinforcement learning system 1 is a system for selecting an excavation operation of an excavator, the first state includes, as an example, a part or all of the posture and position of the excavator excavating the soil, the shape of the soil to be excavated, and the amount of soil in the bucket of the excavator. When the reinforcement learning system 1 is a system for selecting a transport operation of a transport device, the first state includes, as an example, a part or all of the position, movement direction, speed, and each angle of the transport device, the position of the passage, and the position and speed of a static obstacle or a dynamic obstacle. When the reinforcement learning system 1 is a system for autonomous play of a computer game, the first state includes, as an example, a state of an object that affects the progress of the game in the computer game. However, the first state is not limited to the above-mentioned ones, and may be other states. The first state may include, for example, an environmental state such as temperature or weather.

[0015] The generating unit 12 generates the second state by adding noise to the first state. An example of the noise is a random number such as a normal random number or a uniform random number. However, the noise added to the first state by the generating unit 12 is not limited to these, and may be noise other than the above. The generating unit 12 may add noise to all of the elements included in the first state, or may add noise to some of the elements included in the first state.

[0016] The calculation unit 13 calculates the first action value function according to the second state. As an example, the calculation unit 13 calculates the first action value function using a state sequence including a plurality of second states. The calculation unit 13 may also calculate the first action value function using a state sequence including the first state and one or more second states. In other words, the state sequence used by the calculation unit 13 for calculating the first action value function includes one or more second states, and the states included in the state sequence are the first state or the second state. In the following description, when it is not necessary to distinguish between the first state and the second state, they are also simply referred to as "states".

[0017] The first action value function is a function for evaluating an action in a state. The first action value function is, for example, an action value function used in Q-learning, and is updated, for example, by the following formula (1). However, the first action value function is not limited to that given by formula (1), and may be another function.

[0018]

number

[0019] In formula (1), s t (i) (1≦i≦n; i and n are natural numbers) is a state included in the state sequence (i.e., the first state or the second state), a is an action, and Q(s t (i) ,a) is the first action value function. α is the learning rate, s t+1 (i) is the state after the transition, r t+1 is the agent in state s t+1 (i) The reward obtained when transitioning to , γ(0≦γ≦1) is the discount rate. Also, a′∈A, set A is state s t (i) is the set of possible actions for an agent in

[0020] The reward is a reward that the agent receives from the environment by taking action. For example, the reward is a value that is added or subtracted depending on the amount of excavation by the excavator, the time required for excavation, the time required for transportation, whether or not an obstacle is encountered during transportation, the outcome of a game, or the score of the game. However, the reward is not limited to these examples and may be something other than the above.

[0021] When formula (1) is used, the calculation unit 13 calculates the first action-value function for each state included in the state sequence. In other words, the calculation unit 13 calculates the first action-value function for the number of states included in the state sequence.

[0022] The selection unit 14 selects an action according to the first action value function. As an example, the selection unit 14 selects an action that maximizes the first action value function. The selection unit 14 may select an action by the ε-greedy method, roulette selection used in genetic algorithms, softmax method using the Boltzmann distribution, or the like.

[0023] In addition, when a plurality of first action value functions are used, the selection unit 14 may select an action using any of the plurality of first action value functions, or may calculate a second action value function using the plurality of first action value functions calculated by the calculation unit 13, and select an action using the calculated second action value function. The second action value function is a function for evaluating an action in a state. The second action value function may be, for example, an expected value of the plurality of first action value functions, or may be, for example, a function that becomes smaller than the expected value as the variation of the plurality of first action value functions increases. The second action value function is, for example, given by the following formula (2) or formula (3). However, the second action value function is not limited to that given by formula (2) or (3), and may be a function other than these.

[0024]

number

number

[0025] In equations (2) and (3), J(s t ,a) is the second action value function, s t is the first state, a is the action, θ is the hyperparameter, Q(s t (i) ,a) is the first action value function, s t (i) is a state included in the state sequence, and E is the expected value. Note that equation (3) is obtained by Taylor expansion of equation (2), adopting the first second-order terms and discarding the third-order and higher terms.

[0026] When the selection unit 14 calculates the second action value function, the selection unit 14 selects an action that maximizes the second action value function by using a measure given by, for example, formula (4). Note that the measure for selecting an action is not limited to the measure given by formula (4) and may be another measure. For example, the selection unit 14 may select an action by the ε-greedy method, roulette selection used in genetic algorithms, or a softmax method using the Boltzmann distribution. When the ε-greedy method is used, the measure is given by the following formula (5), for example.

[0027]

number

number

[0028] <Effects of reinforcement learning system> According to the reinforcement learning system 1 of this exemplary embodiment, the first action value function that takes into account the variability of the states can be calculated by calculating the action value function using the second state obtained by adding noise to the first state. By selecting an action using this first action value function, the reinforcement learning system 1 can select a more suitable action.

[0029] <Reinforcement learning method flow> 2 is a flow diagram showing the flow of the reinforcement learning method S1 executed by the reinforcement learning system 1. The reinforcement learning system 1 repeatedly selects an action by repeating the reinforcement learning method S1. Note that the contents already explained will not be explained again.

[0030] The reinforcement learning method S1 includes steps S11 to S14. In step S11, the acquisition unit 11 acquires a first state. In step S12, the generation unit 12 generates a second state by adding noise to the first state.

[0031] In step S13, the calculation unit 13 calculates a first action-value function according to the second state. Here, in the nth (n is a natural number) repetition, as data to be referred to by the calculation unit 13 for calculating the first action-value function, the states, actions, and rewards accumulated up to the (n-1)th repetition are used, for example. In step S14, the selection unit 14 selects an action according to the first action-value function.

[0032] <Effects of reinforcement learning methods> According to the reinforcement learning method S1 of this exemplary embodiment, an action value function that takes into account the variability of states can be calculated by calculating an action value function using a second state obtained by adding noise to a first state. By selecting an action using this action value function, a more suitable action can be selected.

[0033] <Example of device configuration for reinforcement learning system> Next, an example of the device configuration of the reinforcement learning system 1 according to this exemplary embodiment will be described with reference to the drawings. FIG. 3 is a block diagram showing an example of the configuration of the reinforcement learning system 1. In the example of FIG. 3, the reinforcement learning system 1 includes a reinforcement learning device 10. The reinforcement learning device 10 includes an acquisition unit 11, a generation unit 12, a calculation unit 13, and a selection unit 14. The reinforcement learning device 10 is, for example, a server device, a personal computer, or a game device, but is not limited to these, and may be a device other than the above. For example, the reinforcement learning device 10 may acquire the first state by receiving the first state via a communication interface.

[0034] Fig. 4 is a block diagram showing another example of the configuration of the reinforcement learning system 1. In the example of Fig. 4, the reinforcement learning system 1 includes a terminal 20 and a server 30. The terminal 20 is, for example, a personal computer or a game device, but is not limited to these and may be a device other than the above. The terminal 20 includes an acquisition unit 11. The server 30 includes a generation unit 12, a calculation unit 13, and a selection unit 14. The terminal 20 acquires a first state and supplies the acquired first state to the server 30.

[0035] In this exemplary embodiment, FIGS. 3 and 4 are shown as configuration examples of the reinforcement learning system 1, but the configuration of the reinforcement learning system 1 is not limited to those shown in FIGS. 3 and 4, and various other configurations are applicable.

[0036] Exemplary embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals and will not be described repeatedly.

[0037] <Structure of Reinforcement Learning System> Fig. 5 is a block diagram showing a configuration of the reinforcement learning system 2. As shown in Fig. 5, the reinforcement learning system 2 includes a terminal 40 and a reinforcement learning device 50. The terminal 40 and the reinforcement learning device 50 are configured to be able to communicate with each other via a communication line N. Although the specific configuration of the communication line N does not limit the present exemplary embodiment, as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0038] The terminal 40 is, for example, a general-purpose computer, and more specifically, for example, a control device for controlling a construction machine such as an excavator, a management device for managing transportation by a transportation device, or a game device for playing a computer game. Note that the terminal 40 is not limited to these, and may be a device other than the above. The reinforcement learning device 50 is, for example, a server device.

[0039] <Device configuration> The terminal 40 includes a communication unit 41, a control unit 42, and an input receiving unit 43. The communication unit 41 transmits and receives information to and from the reinforcement learning device 50 via a communication line N under the control of the control unit 42. Hereinafter, the transmission and reception of information between the control unit 42 and the reinforcement learning device 50 via the communication unit 41 will also be simply referred to as the control unit 42 transmitting and receiving information to and from the reinforcement learning device 50.

[0040] The control unit 42 includes a state providing unit 421, an action executing unit 422, and a reward providing unit 423. The state providing unit 421 acquires a first state and provides the acquired first state to the reinforcement learning device 50. In this exemplary embodiment, the first state acquired by the state providing unit 421 includes a plurality of elements to which attributes are attached. The attribute is information indicating the characteristics and / or type of the element, and includes information indicating, for example, whether the element is a dynamic element that moves in the environment or a static element that does not move in the environment. The attribute may also be information indicating the type of element, such as a person, a car, a bicycle, or a building. However, the attribute is not limited to the above examples, and may be other information other than the above.

[0041] As an example, the status providing unit 421 may obtain sensor information output by a sensor that detects the operation of a construction machine, a transport device, or the like, as the first status. Also, as an example, the status providing unit 421 may obtain the first status of an object that affects the progress of a computer game. However, the first status obtained by the status providing unit 421 is not limited to the above examples, and may be a status other than the above.

[0042] As an example, the state providing unit 421 accepts an input of the first state via the input accepting unit 43, and provides the accepted first state to the reinforcement learning device 50. In addition, as an example, the state providing unit 421 may receive the first state from another device connected via the communication unit 41, and provide the received first state to the reinforcement learning device 50.

[0043] The action execution unit 422 executes the action determined by the reinforcement learning device 50. As an example, the action execution unit 422 outputs control information for making a construction machine, a transport device, or the like execute the action determined by the reinforcement learning device 50. As another example, the action execution unit 422 controls the action of an object that is the target of a user operation in a computer game. However, the action executed by the reinforcement learning device 50 is not limited to the above example, and may be an action other than the above.

[0044] The reward providing unit 423 provides the reinforcement learning device 50 with a reward obtained when the agent executes the action determined by the reinforcement learning device 50. As an example, the reward providing unit 423 provides the reinforcement learning device 50 with information indicating the amount of excavation by the excavator, the time required for excavation, the time required for transportation by the transportation device, whether or not an obstacle was contacted during transportation, the outcome of a game, or the score of a game as a reward. However, the reward provided by the reward providing unit 423 is not limited to the above example, and may be a reward other than the above.

[0045] As an example, the reward providing unit 423 provides the reinforcement learning device 50 with a reward acquired via the input receiving unit 43. In addition, the reward providing unit 423 may receive a reward from another device connected via the communication unit 41, and provide the received reward to the reinforcement learning device 50.

[0046] The input receiving unit 43 receives various inputs to the terminal 40. Although the specific configuration of the input receiving unit 43 is not limited to this exemplary embodiment, as an example, the input receiving unit 43 may be configured to include input devices such as a keyboard and a touchpad. The input receiving unit 43 may also be configured to include a data scanner that reads data via electromagnetic waves such as infrared rays and radio waves, and a sensor that senses the state of the environment. As an example, the reward providing unit 423 measures the time required for transportation by the transportation device based on the sensing result acquired by the input receiving unit 43, and provides a reward indicating the measurement result to the reinforcement learning device 50.

[0047] The input receiving unit 43 supplies the received input information to the control unit 42 via the above-mentioned input device, data scanner, sensor, etc. The input receiving unit 43, as one example, acquires the above-mentioned state and the above-mentioned reward, and supplies the acquired state and reward to the control unit 42.

[0048] <Configuration of reinforcement learning device> The reinforcement learning device 50 includes a communication unit 51, a control unit 52, and a storage unit 53. The communication unit 51 transmits and receives information to and from the reinforcement learning device 50 via a communication line N under the control of the control unit 52. Hereinafter, the transmission and reception of information between the control unit 52 and the terminal 40 via the communication unit 51 will also be simply described as the control unit 52 transmitting and receiving information to and from the terminal 40.

[0049] The control unit 52 includes a reward acquisition unit 521, a state observation unit 522, a state randomization unit 523, a learning unit 524, an estimation unit 525, and a selection unit 526. The state observation unit 522 is configured to realize an acquisition means in this exemplary embodiment. The state randomization unit 523 is configured to realize a generation means in this exemplary embodiment. The estimation unit 525 is configured to realize a calculation means in this exemplary embodiment. The selection unit 526 is configured to realize a selection means in this exemplary embodiment.

[0050] The reward acquisition unit 521 acquires a reward provided by the terminal 40 via the communication unit 51. The state observation unit 522 acquires a first state provided by the terminal 40 via the communication unit 51. The state randomization unit 523 generates one or more second states by adding noise to the first state acquired by the state observation unit 522. The learning unit 524 trains an action value function model 531 for updating the first action value function. The action value function model 531 is used to estimate the first action value function.

[0051] The estimation unit 525 calculates a first action-value function according to a state sequence including a first state and one or more second states, or a state sequence including more than one second state. The estimation unit 525 also calculates a second action-value function using the first action-value function.

[0052] The selection unit 526 selects an action using the second action-value function, and stores information indicating the selected action in the storage unit 53, and transmits the information indicating the selected action to the terminal 40.

[0053] The storage unit 53 stores various data referred to by the control unit 52. As an example, the storage unit 53 stores an action value function model 531 and learning data 532. The action value function model 531 is a learning model for updating the first action value function. The learning data 532 is data used in the reinforcement learning performed by the reinforcement learning device 50. As an example, the learning data 532 includes a first state, a second state, an action, and a reward.

[0054] <Reinforcement learning method flow> 6 is a flow diagram showing the flow of the reinforcement learning method S2 executed by the reinforcement learning system 2. The reinforcement learning system 2 repeatedly selects an action by repeating steps S21 to S29. Note that some steps may be executed in parallel or in a different order.

[0055] In step S21, the state providing unit 421 receives the first state s t and obtain the first state s tIn step S22, the state observer 522 receives the first state s t Get the.

[0056] In step S23, the state randomizer 523 randomizes the first state s t The state randomizer 523 generates one or more second states by adding noise to the first state s t The noise to be added to the first state s is, for example, a normal random number or a uniform random number. t The noise to be added to is not limited to these, and may be other noises. The second state to which noise has been added is the first state s t This indicates a state in which there is some blurring.

[0057] In this operation example, the state randomizer 523 randomizes the first state s t The second state is generated by selectively adding noise to a plurality of elements included in the state randomization unit 523. As an example, the state randomization unit 523 adds noise to an element associated with an attribute that satisfies a predetermined condition. The predetermined condition is, for example, an attribute indicating a dynamic element, or an attribute indicating a static element. However, the predetermined condition is not limited to the above example, and may be another condition.

[0058] In addition, the state randomization unit 523 generates a state sequence {s t (i)} (1≦i≦n; i is a natural number, n is a natural number equal to or greater than 2). t (i)} is the first state s t and one or more second states, or a state sequence including a plurality of second states. t (i)} includes at least a second state and also includes a first state s t It may or may not include.

[0059] In step S24, the estimation unit 525 estimates the state sequence {s t (i)}, the first action value function Q(s t (i) , a). For example, the estimation unit 525 calculates the state sequence {s t (i)} contains multiple states s t (i) For each of these, the first action value function Q(s t (i) More specifically, the estimation unit 525 calculates the state s t (i) The first action value function Q(s t (i) , a) is updated. In this operation example, the first action value function (s t (i) ,a) is an m-dimensional vector (m is an integer greater than or equal to 2), where m is the number of elements in set A (i.e., the number of types of actions a).

[0060] In step S25, the estimation unit 525 estimates the calculated first action-value functions Q(s t (i) , a) based on which the second action value function J(s t ,a) is calculated. The second action value function J(s t , a) is given by the above formula (2) or formula (3), for example. In other words, the estimation unit 525 calculates the second action value function given by the above formula (2) or formula (3). The second action value function given by the above formula (2) or formula (3) is obtained by multiplying a plurality of first action value functions Q(s t (i) , a) the greater the variation in the first action value function Q(s t (i) ,a) is a function whose value is lower than the expected value.

[0061] In step S26, the selection unit 526 selects the first action-value function Q(s t (i) , a) is calculated based on the second action value function J(s t, a), the selection unit 526 selects the action a according to the strategy given by the above formula (4), as an example. Note that the strategy for selecting the action a is not limited to the strategy given by the above formula (4), and other strategies such as the ε-greedy strategy and the softmax method may be used. The selection unit 526 notifies the terminal 40 of the selected action a.

[0062] In step S27, the action execution unit 422 executes the action a notified by the reinforcement learning device 50. In step S28, the reward provision unit 423 provides the reward r obtained by executing the action selected by the reinforcement learning device 50. t to the reinforcement learning device 50. In step S29, the reward acquisition unit 521 t (i)}, and reward r t Accumulate learning data including:

[0063] <Effects of reinforcement learning system> In reinforcement learning, even a slight difference in state may result in a large difference in the value of the action value function. In other words, a slight difference in state may have a large effect on the value of the action value function. In this exemplary embodiment, the first state s t By calculating the first action-value function Q using the second state in which some noise is intentionally added to, it is possible to calculate the first action-value function Q taking into consideration the variability of the state. By selecting the action a using this first action-value function Q, according to this exemplary embodiment, the action a can be selected more appropriately.

[0064] In addition, in the reinforcement learning system 2 according to this exemplary embodiment, a plurality of states s including a second state to which noise is added are generated. t (i) Therefore, according to the reinforcement learning system 2 according to the present exemplary embodiment, in addition to the effects of the reinforcement learning system 1 according to the exemplary embodiment 1, an effect of being able to select a more appropriate action a can be obtained.

[0065] In addition, in this exemplary embodiment, when the reinforcement learning system 2 calculates the second action value function J using the above-mentioned formula (2), the second action value function J becomes an index sensitive to risk (variability) including higher-order influences. By the reinforcement learning system 2 selecting an action a using the second action value function J, it is possible to select an action a that is more sensitive to risk.

[0066] In addition, in this exemplary embodiment, when the reinforcement learning system 2 calculates the second action-value function J using the above-mentioned formula (3), no exponential operation is included in formula (3), so that no overflow occurs in the calculation process. By the reinforcement learning system 2 selecting an action a using the second action-value function J, the action a can be more suitably selected, and the processing load related to the selection of the action a is reduced.

[0067] Exemplary embodiment 3 Exemplary embodiment 3 of the present invention will be described with reference to the drawings. Note that components having the same functions as those described in exemplary embodiments 1 and 2 are denoted by the same reference numerals, and the description thereof will not be repeated.

[0068] <Structure of Reinforcement Learning System> A reinforcement learning system according to this exemplary embodiment (hereinafter referred to as "reinforcement learning system 3") is obtained by applying the reinforcement learning system 2 according to the above-mentioned exemplary embodiment 2 to autonomous play of a computer game. The reinforcement learning system 3 has a similar configuration to the reinforcement learning system 2 shown in FIG. 5 in the above-mentioned exemplary embodiment 2. The components of the reinforcement learning system 3 are similar to the components of the reinforcement learning system 2, and the description thereof will not be repeated here.

[0069] In this exemplary embodiment, the first state s t For example, the state of an object in a computer game that affects the progress of the game is included. For example, the action a includes the movement of an object operated by a player of the computer game. The reward r t Examples of rewards include rewards related to winning or losing a game, or game scores.

[0070] 7 is a diagram showing a screen SC1 which is an example of a game screen of a computer game related to the reinforcement learning system 3. The screen SC1 includes a first dynamic object C11, second dynamic objects C21-C23, first static objects C31-C34, and a second static object C4. The first dynamic object C11, the second dynamic objects C21-C23, the first static objects C31-C34, and the second static object C4 are examples of objects that affect the progress of the game.

[0071] The computer game of Figure 7 is a game in which the player specifies the movement direction of a first dynamic object C11 moving within a maze, and clears a round by recovering first static objects C31 to C34 placed within the maze while evading the pursuit of second dynamic objects C21 to C23.

[0072] The first dynamic object C11 and the second dynamic objects C21-C23 are objects that move on the screen during the game, and are an example of a dynamic element that moves in the environment. On the other hand, the first static objects C31-C34 and the second static object C4 are objects that do not move on the screen during the game, and are an example of a static element that does not move in the environment. The first dynamic object C11 is an object that is operated by the player. The first dynamic object C11 moves in the maze at a constant speed during the game, and changes the moving direction according to the player's operation. The second dynamic objects C21-C23 are objects that move following the first dynamic object C11 during the game. Although three second dynamic objects C21-C23 are illustrated in FIG. 7, the number of second dynamic objects is not limited to three, and may be more or less than three.

[0073] The first static objects C31-C34 are objects that are placed in the maze and are collected by the first dynamic object C11. When the first dynamic object C11 collides with the first static objects C31-C34, the first static objects C31-C34 are collected by the first dynamic object C11. Although four first static objects C31-C34 are illustrated in FIG. 7, the number of first static objects is not limited to four and may be more or less than four. The second static object C4 is a wall that constitutes the maze.

[0074] In the example of FIG. 7, the first state s t The first state includes states related to the first dynamic object C11, the second dynamic objects C21-C23, the first static objects C31-C34, and the second static object C4. In other words, the first state includes states related to dynamic elements that move within the environment, and states related to static elements that do not move within the environment. More specifically, the first state s t includes the position of the first dynamic object C11, the positions of the second dynamic objects C21 to C23, the positions of the first static objects C31 to C34, and the position of the second static object C4.

[0075] In this exemplary embodiment, the first state s t is an image showing the game play screen. FIG. 8 shows the first state s t 1 is a diagram showing an image Img11, which is an example of the image Img11. Image Img11 is a grayscale image in which elements included in the game screen are represented by pixel values ​​of 0 to 255. Image Img11 is divided into a predetermined number of squares, and each square is represented by a pixel value according to the attribute of the element located in that square. As an example, the position of the first dynamic object C11 is represented by a pixel value of 255, the positions of the second dynamic objects C21 to C23 by a pixel value of 160, the positions of the first static objects C31 to C34 by a pixel value of 128, the position of the passage formed by the second static object C4 by a pixel value of 64, and unmovable locations by a pixel value of 0.

[0076] In this exemplary embodiment, the action a is the movement of the first dynamic object C11, and there are four types: move up, move down, move right, and move left. t For example, the reward r is a predetermined additional value (e.g., +1) obtained when the score is increased, and a predetermined subtraction value (e.g., -10) obtained when the player is captured by the second dynamic objects C21 to C23. Regardless of the degree of increase in the score in one action, when the score is increased by an action, the predetermined additional value (e.g., +1) is added to the reward r. t It may be obtained as:

[0077] <Reinforcement learning method flow> The reinforcement learning system 3 executes the reinforcement learning method S2 of Fig. 6 according to the above-mentioned exemplary embodiment 2. In the following, characteristic operations in this exemplary embodiment will be mainly described, and the contents described in the above-mentioned exemplary embodiment 2 will not be described repeatedly.

[0078] In this exemplary embodiment, in step S23, the state randomizer 523 randomizes the first state s t The state randomization unit 523 generates the second state by adding noise to the states of the dynamic elements included in the first dynamic object C11. As an example, the state randomization unit 523 generates the second state by randomizing the position of the first dynamic object C11 and the positions of the second dynamic objects C21 to C23 by random walk.

[0079] More specifically, as an example, the state randomization unit 523 divides the game screen into a predetermined number of squares (for example, 33 × 33 squares), and selects equal probability for moving forward or backward, left or right, or one square in a direction in which the object can move (a direction in which there is a path). The state randomization unit 523 determines σ 2 Random walks are performed σ times (σ is an integer greater than or equal to 1). 2 By performing random walks, the dynamic element moves by σ squares on average.

[0080] In step S23, the state sequence {s t (i)} is the first state s t , and the first state s t In addition, since the action a has four types, ie, moving up, moving down, moving right, and moving left, the first action value function (s t (i) ,a) is a four-dimensional vector.

[0081] In step S26, the selection unit 526 selects one of four types of movement direction of the first dynamic object C11, up, down, left, or right, as the action a at an intersection or corner (i.e., a point where the movement direction can be changed). However, the selection unit 526 excludes directions in which the first dynamic object C11 cannot move.

[0082] <Evaluation of this exemplary embodiment> 9 to 12 are diagrams showing an example of an evaluation result of an autonomous play of a computer game related to the reinforcement learning system 3. In the computer game according to this exemplary embodiment, the first dynamic object has one life, and the game is over when the first dynamic object is captured by the second dynamic object. In addition, there is one stage, and the game ends when the game is cleared, that is, when all the first static objects are collected.

[0083] In the examples of Figures 9 to 12, the results of the autonomous play of a computer game by the reinforcement learning system 3 under multiple conditions in which the values ​​of σ and θ in the reinforcement learning of the reinforcement learning system 3 were changed were evaluated. In addition, the results of the autonomous play using a conventional reinforcement learning method other than the reinforcement learning system 3 were also compared. As the conventional reinforcement learning method, an improved action selection policy in the DQN (deep Q-network) method was used.

[0084] FIG. 9 is a graph showing scores from autonomous play when σ=2. As described above, σ is the average number of moves in a random walk. In FIG. 9, the vertical axis shows scores. Graph g91 shows the average scores from autonomous play using conventional reinforcement learning. Graphs g11 to g14 show the average scores from autonomous play using the reinforcement learning system 3. Graphs g11 to g14 have different values ​​for the hyperparameter θ in the equation (the above equation (2) or equation (3)) representing the second action value function J. Graphs g11 to g14 show the average scores when the hyperparameter θ is set to "0", "0.001", "0.01", and "0.1", respectively.

[0085] Comparing graph g91 with graphs g11 to g14, the score of the reinforcement learning system 3 according to this exemplary embodiment is higher than the score obtained by conventional reinforcement learning, and the score is particularly high when the value of the hyperparameter θ is set to "0.01."

[0086] FIG. 10 is a graph showing the recovery rate of the first static object through autonomous play when σ=2. In FIG. 10, the vertical axis shows the recovery rate. Graph g92 shows the average value of the recovery rate of autonomous play through conventional reinforcement learning. Graphs g21 to g24 show the average value of the recovery rate through autonomous play in the reinforcement learning system 3. Graphs g21 to g24 have different values ​​of the hyperparameter θ in the equation (the above equation (2) or equation (3)) showing the second action value function J. Graphs g21 to g24 show the average value of the recovery rate when the hyperparameter θ is set to "0", "0.001", "0.01", and "0.1", respectively.

[0087] Comparing graph g92 with graphs g21 to g24, the recovery rate of the reinforcement learning system 3 according to this exemplary embodiment tends to be higher than the recovery rate of conventional reinforcement learning, and the score is particularly high when the value of the hyperparameter θ is set to “0.01.”

[0088] FIG. 11 is a graph showing the relationship between the score in autonomous play and σ. In FIG. 11, the horizontal axis shows σ, and the vertical axis shows the score. Graphs g31 to g34 show the average score when σ is 1 to 5, when the hyperparameter θ is "0", "0.001", "0.01", and "0.1", respectively. The average score in autonomous play using conventional reinforcement learning is "2009".

[0089] In the example of Figure 11, the scores for σ values ​​of 1 to 3 are often higher than those obtained by conventional reinforcement learning. In particular, the scores for θ = 0.01 and σ = 2 are higher than the others.

[0090] FIG. 12 is a graph showing the relationship between the recovery rate and σ in autonomous play. In FIG. 12, the horizontal axis shows σ, and the vertical axis shows the recovery rate. Graphs g41 to g44 show the average recovery rate for each value of σ when the hyperparameter θ is "0", "0.001", "0.01", and "0.1", respectively. The average recovery rate in autonomous play using conventional reinforcement learning is 67.5%.

[0091] In the example of Figure 12, the recovery rate when σ is 1 to 3 is often higher than the recovery rate by conventional reinforcement learning. In particular, the recovery rate when θ=0.01 and σ=2 is higher than the others.

[0092] As described above, according to this exemplary embodiment, the reinforcement learning system 3 calculates a first action-value function using a second state in which noise is added to a first state, thereby more suitably selecting an action in autonomous play of a computer game.

[0093] Exemplary embodiment 4 A fourth exemplary embodiment of the present invention will be described. Note that components having the same functions as those described in the first to third exemplary embodiments will be designated by the same reference numerals and will not be described repeatedly.

[0094] A reinforcement learning system according to this exemplary embodiment (hereinafter referred to as "reinforcement learning system 4") is obtained by applying the reinforcement learning system 2 according to the above-described exemplary embodiment 2 to the control of a construction machine such as an excavator that excavates earth and sand. The reinforcement learning system 3 has a similar configuration to the reinforcement learning system 2 shown in FIG. 5 in the above-described exemplary embodiment 2. The components of the reinforcement learning system 4 are similar to the components of the reinforcement learning system 2, and the description thereof will not be repeated here.

[0095] The reinforcement learning system 4 selects the operation of the construction machine, such as the excavation operation of a hydraulic excavator when excavating earth and sand, by using reinforcement learning. As an example, the objective of the action in the reinforcement learning is to excavate a bucket full of earth and prevent the vehicle body from tilting or being dragged during excavation.

[0096] In this exemplary embodiment, the first state s t For example, the information includes some or all of the following: the attitude and position of a construction machine such as a hydraulic excavator, the shape (3D data, etc.) of the soil to be excavated, and the amount of soil in the excavator bucket. The attitude of the construction machine, for example, includes the angles of the bucket, arm, boom, and rotating body of the construction machine. The position of the construction machine, for example, includes the position and direction of the crawler of the construction machine.

[0097] An example of the behavior a is posture control of a construction machine (angle control of a bucket, arm, boom, rotating body, etc.). t As an example, the reward may include some or all of a positive reward whose absolute value increases the greater the amount of excavation, and a negative reward whose absolute value increases the greater the degree of inclination of the body of the construction machine, the degree of dragging, or the time taken for excavation.

[0098] The state randomizer 523 randomizes the first state s t Noise may be added to all of the elements included in the image, or noise may be added to some of the elements. When noise is added to some of the elements, the elements to which noise is added may include, for example, the attitude of the hydraulic excavator and 3D data of the observed soil and sand.

[0099] According to this exemplary embodiment, the reinforcement learning system 4 is t By calculating the first action-value function using the second state obtained by adding noise to the first action-value function, it is possible to more appropriately select the operation of the construction machine.

[0100] Exemplary embodiment 5 A fifth exemplary embodiment of the present invention will be described. Note that components having the same functions as those described in the first to fourth exemplary embodiments will be designated by the same reference numerals and will not be described repeatedly.

[0101] A reinforcement learning system according to this exemplary embodiment (hereinafter referred to as "reinforcement learning system 5") applies the reinforcement learning system 2 according to the above-mentioned exemplary embodiment 2 to the control of a transport device that transports luggage. The transport device is, for example, an automated guided vehicle (AGV) that runs automatically. The reinforcement learning system 5 has a similar configuration to the reinforcement learning system 2 shown in FIG. 5 in the above-mentioned exemplary embodiment 2. The components of the reinforcement learning system 5 are similar to the components of the reinforcement learning system 2, and the description thereof will not be repeated here.

[0102] When transporting luggage from a given position to another, the reinforcement learning system 5 selects actions that will shorten the transport time (increase the transport speed) as much as possible and avoid contact with static obstacles (shelves, luggage, etc.) and dynamic obstacles (people, other robots, etc.) along the way.

[0103] In this exemplary embodiment, the first state s t For example, the reward r includes some or all of the following: the position, moving direction, speed, and angular speed of the conveying device that conveys the object, the position of the passage, the position of the static obstacle, and the position and moving speed of the dynamic obstacle. For example, the action a includes the speed control and angular speed control of the conveying device. t For example, the reward may include some or all of a positive reward obtained when the transport is completed, a negative reward obtained when an obstacle is encountered, or a negative reward whose absolute value increases the longer the transport time.

[0104] The state randomizer 523 randomizes the first state by t Noise may be added to all of the elements included in the state randomization unit 523, or noise may be added to some of the elements. When noise is added to some of the elements, the elements to which noise is added may include, for example, the position, direction, speed, and angular velocity of the transport device, and may also include the position of a static obstacle, or the position and speed of a dynamic obstacle. In addition, the state randomization unit 523 may add noise to obstacles located in the traveling direction or on the traveling path of the transport device, and not add noise to obstacles located outside the traveling direction or outside the traveling path.

[0105] According to this exemplary embodiment, the reinforcement learning system 5 is t By calculating the first action-value function using the second state in which noise has been added to the first action-value function, it is possible to more appropriately control the transport of the transport device.

[0106] Exemplary embodiment 6 A sixth exemplary embodiment of the present invention will be described. Note that components having the same functions as those described in the first to fifth exemplary embodiments will be designated by the same reference numerals and will not be described repeatedly.

[0107] A reinforcement learning system according to this exemplary embodiment (hereinafter referred to as "reinforcement learning system 6") applies the reinforcement learning system 2 according to the above-described exemplary embodiment 2 to the control of a forklift. The reinforcement learning system 6 has a similar configuration to the reinforcement learning system 2 shown in FIG. 5 in the above-described exemplary embodiment 2. The components of the reinforcement learning system 6 are similar to the components of the reinforcement learning system 2, and the description thereof will not be repeated here.

[0108] When transporting a pallet from a given position to another position, the reinforcement learning system 6 selects an action that shortens the transport time (increases the transport speed) as much as possible and avoids contact with static obstacles (shelves, luggage, etc.) and dynamic obstacles (people, other rubbish, etc.) along the way.

[0109] In this exemplary embodiment, the first state s t For example, the reward r includes some or all of the following: the position, movement direction, speed, and angular speed of the forklift, the position of the passage, the position of the static obstacle, and the position and speed of the dynamic obstacle. For example, the action a includes the speed control and the angular speed control of the forklift. t For example, the reward may include some or all of a positive reward obtained when the transport is completed, a negative reward obtained when an obstacle is encountered, or a negative reward whose absolute value increases the longer the transport time.

[0110] The state randomizer 523 randomizes the first state by t Noise may be added to all of the elements included in the state randomization unit 523, or noise may be added to some of the elements. When noise is added to some of the elements, the elements to which noise is added may include, for example, the position, direction, speed, and angular velocity of the forklift, and may also include the position of a static obstacle, or the position and speed of a dynamic obstacle. Furthermore, the state randomization unit 523 may add noise to obstacles located in the traveling direction or on the traveling path of the forklift, and not add noise to obstacles located outside the traveling direction or outside the traveling path.

[0111] According to this exemplary embodiment, the reinforcement learning system 5 is t By calculating the first action-value function using the second state obtained by adding noise to the first action-value function, it is possible to more appropriately control the forklift.

[0112] Exemplary Embodiment 7 A seventh exemplary embodiment of the present invention will be described. Note that components having the same functions as those described in the first to sixth exemplary embodiments will be designated by the same reference numerals and will not be described repeatedly.

[0113] A reinforcement learning system according to this exemplary embodiment (hereinafter referred to as "reinforcement learning system 7") has a similar configuration to the reinforcement learning system 2 shown in Fig. 5 in the above-described exemplary embodiment 2. The components of the reinforcement learning system 6 are similar to the components of the reinforcement learning system 2, and the description thereof will not be repeated here.

[0114] In this exemplary embodiment, the first state s t The first state s includes a plurality of elements with attributes. t When adding noise to the environment, the state randomizer 523 weights the noise depending on the attribute. As an example, the state randomizer 523 may weight the dynamic elements that move within the environment more heavily, while weighting the static elements that do not move within the environment less heavily. Also, as an example, the state randomizer 523 may weight the position of a person, among the dynamic elements that move within the environment, more heavily than the weights of other dynamic elements.

[0115] According to this exemplary embodiment, the first action value function is calculated using the second state to which noise is added with weighting according to the attribute of the element, thereby making it possible to calculate the first action value function taking into account the variability of data according to the attribute. By selecting an action using this first action value function, it is possible to select an action taking into account the variability of data according to the attribute.

[0116] In addition, in this exemplary embodiment, the state randomizer 523 may change the weighting of the noise addition during the execution of reinforcement learning. As an example, the state randomizer 523 may perform control such that the weighting is increased when a dynamic element is moving in the environment, while the weighting is decreased for a dynamic element that is not moving in the environment.

[0117] [Software implementation example] Some or all of the functions of the reinforcement learning device 10, the terminal 20, the server 30, the terminal 40, and the reinforcement learning device 50 (hereinafter referred to as "the reinforcement learning device 10, etc.") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0118] In the latter case, the reinforcement learning device 10 etc. is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in FIG. 13. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the reinforcement learning device 10 etc. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the reinforcement learning device 10 etc.

[0119] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

[0120] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0121] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[0122] [Additional Note 1] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiment are also included in the technical scope of the present invention.

[0123] [Additional Note 2] Some or all of the above-described embodiments can be described as follows. However, the present invention is not limited to the following described aspects.

[0124] (Appendix 1) An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning; generating means for generating a second state by adding noise to the first state; A calculation means for calculating a first action value function according to the second state; a selection means for selecting an action in accordance with the first action-value function; A reinforcement learning system comprising:

[0125] According to the above configuration, the first action-value function is calculated using the second state obtained by adding noise to the first state, so that a more suitable action can be selected.

[0126] (Appendix 2) the calculation means calculates the first action-value function according to the first state and the second state; 2. The reinforcement learning system of claim 1.

[0127] According to the above configuration, the first action-value function is calculated using a plurality of states including the second state to which noise has been added, making it possible to select a more suitable action.

[0128] (Appendix 3) The calculation means calculates the first action-value function for each of the first state and the second state; the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions. 4. The reinforcement learning system of claim 2.

[0129] According to the above configuration, by using the second action-value function calculated using a plurality of first action-value functions, it is possible to select a more suitable action.

[0130] (Appendix 4) The first state includes at least one of a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys the conveyed object, a position of a passage, and a position and a speed of a static or dynamic obstacle; 4. The reinforcement learning system of claim 1.

[0131] According to the above configuration, the transport operation of the transport device can be more suitably selected through reinforcement learning.

[0132] (Appendix 5) The first state includes at least one of the following: the attitude and position of the construction machine, the shape of the soil to be excavated, and the amount of soil in the bucket of the excavator. 4. The reinforcement learning system of claim 1.

[0133] According to the above configuration, the construction operation of the construction machine can be more appropriately selected through reinforcement learning.

[0134] (Appendix 6) the first state includes a plurality of elements having attributes associated therewith; the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute. 6. The reinforcement learning system of claim 1.

[0135] According to the above configuration, it is possible to calculate the first action-value function taking into consideration the variability in data regarding elements associated with attributes that satisfy a predetermined condition.

[0136] (Appendix 7) the first state includes a state regarding a dynamic element moving within an environment; the generating means generates the second state by adding noise to a state of the dynamic element included in the first state. 7. The reinforcement learning system of claim 6.

[0137] According to the above configuration, it is possible to calculate the first action value function taking into consideration the variability in data regarding the dynamic element.

[0138] (Appendix 8) An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning; generating means for generating a second state by adding noise to the first state; A calculation means for calculating a first action value function according to the second state; a selection means for selecting an action in accordance with the first action-value function; A reinforcement learning device comprising:

[0139] According to the above configuration, the first action-value function is calculated using the second state obtained by adding noise to the first state, so that a more suitable action can be selected.

[0140] (Appendix 9) the calculation means calculates the first action-value function according to the first state and the second state; 9. The reinforcement learning apparatus of claim 8.

[0141] According to the above configuration, the first action-value function is calculated using a plurality of states including the second state to which noise has been added, making it possible to select a more suitable action.

[0142] (Appendix 10) The calculation means calculates the first action-value function for each of a plurality of states included in the state sequence; the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions. 10. The reinforcement learning apparatus of claim 9.

[0143] According to the above configuration, by using the second action-value function calculated using a plurality of first action-value functions, it is possible to select a more suitable action.

[0144] (Appendix 11) The first state includes at least one of a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys the conveyed object, a position of a passage, and a position and a speed of a static or dynamic obstacle; 11. The reinforcement learning device of claim 8.

[0145] According to the above configuration, the transport operation of the transport device can be more suitably selected through reinforcement learning.

[0146] (Appendix 12) The first state includes at least one of the following: the attitude and position of the construction machine, the shape of the soil to be excavated, and the amount of soil in the bucket of the excavator. 11. The reinforcement learning device of claim 8.

[0147] According to the above configuration, the construction operation of the construction machine can be more appropriately selected through reinforcement learning.

[0148] (Appendix 13) the first state includes a plurality of elements having attributes associated therewith; the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute. 13. The reinforcement learning device of any one of appendix 8 to 12.

[0149] According to the above configuration, it is possible to calculate the first action-value function taking into consideration the variability in data regarding elements associated with attributes that satisfy a predetermined condition.

[0150] (Appendix 14) the first state includes a state regarding a dynamic element moving within an environment; the generating means generates the second state by adding noise to a state of the dynamic element included in the first state. 14. The reinforcement learning apparatus of claim 13.

[0151] According to the above configuration, it is possible to calculate the first action value function taking into consideration the variability in data regarding the dynamic element.

[0152] (Appendix 15) Obtaining a first state of an environment that is the subject of reinforcement learning; generating a second state by adding noise to the first state; calculating a first action-value function according to the second state; selecting an action in response to said first action-value function; Reinforcement learning methods including

[0153] According to the above configuration, the first action-value function is calculated using the second state obtained by adding noise to the first state, so that a more suitable action can be selected.

[0154] (Appendix 16) In calculating the first action value function, calculating the first action-value function according to the first state and the second state; 16. The reinforcement learning method of claim 15.

[0155] According to the above configuration, the first action-value function is calculated using a plurality of states including the second state to which noise has been added, making it possible to select a more suitable action.

[0156] (Appendix 17) In calculating the first action-value function, the first action-value function is calculated for each of a plurality of states included in the state sequence; In selecting the action, the action is selected according to a second action value function calculated based on a plurality of the first action value functions. 17. The reinforcement learning method of claim 16.

[0157] According to the above configuration, by using the second action-value function calculated using a plurality of first action-value functions, it is possible to select a more suitable action.

[0158] (Appendix 18) The first state includes at least one of a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys the conveyed object, a position of a passage, and a position and a speed of a static or dynamic obstacle; 18. The reinforcement learning method of any one of appendix 15 to 17.

[0159] According to the above configuration, the transport operation of the transport device can be more suitably selected through reinforcement learning.

[0160] (Appendix 19) The first state includes at least one of the following: the attitude and position of the construction machine, the shape of the soil to be excavated, and the amount of soil in the bucket of the excavator. 18. The reinforcement learning method of any one of claims 15 to 17.

[0161] According to the above configuration, the construction operation of the construction machine can be more appropriately selected through reinforcement learning.

[0162] (Appendix 20) the first state includes a plurality of elements having attributes associated therewith; generating the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute; 20. The reinforcement learning method of any one of appendix 15 to 19.

[0163] According to the above configuration, it is possible to calculate the first action-value function taking into consideration the variability in data regarding elements associated with attributes that satisfy a predetermined condition.

[0164] (Appendix 21) the first state includes a state regarding a dynamic element moving within an environment; generating the second state by adding noise to the state of the dynamic element included in the first state; 21. The reinforcement learning method of claim 20.

[0165] According to the above configuration, it is possible to calculate the first action value function taking into consideration the variability in data regarding the dynamic element.

[0166] (Appendix 22) the first state includes a plurality of elements having attributes associated therewith; The generating means varies a weight of the noise addition depending on the attribute. 6. The reinforcement learning system of claim 1.

[0167] According to the above configuration, by using the second state in which noise is added with weighting according to the attribute of the element, it is possible to calculate the first action-value function taking into consideration the variance in data according to the attribute.

[0168] (Appendix 23) The first state includes some or all of the following: the attitude and position of the construction machine, the shape of the soil to be excavated, and the amount of soil in the bucket of the excavator; The action includes attitude control of the construction machine. 19. The reinforcement learning system of claim 1.

[0169] According to the above configuration, by calculating the first action-value function using the second state obtained by adding noise to the first state, it is possible to more appropriately select the excavation operation of the excavator through reinforcement learning.

[0170] (Appendix 24) The first state includes some or all of the following: a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys an object, a position of a passage, and a position and a speed of a static or dynamic obstacle; The action includes speed control and angular velocity control of the transport device. 19. The reinforcement learning system of claim 1.

[0171] According to the above configuration, by calculating the first action value function using the second state obtained by adding noise to the first state, it is possible to more appropriately select the transportation operation of the transportation device through reinforcement learning.

[0172] (Appendix 25) the first state includes a state of an object that affects progress of the computer game; the action includes a movement of an object controlled by a player of the computer game; 19. The reinforcement learning system of claim 1.

[0173] According to the above configuration, by calculating the first action value function using the second state in which noise is added to the first state, it is possible to more appropriately select an action of an object in autonomous play of a computer game.

[0174] (Appendix 26) A program for causing a computer to function as a reinforcement learning device, The program causes the computer to An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning; generating means for generating a second state by adding noise to the first state; A calculation means for calculating a first action value function according to the second state; a selection means for selecting an action in accordance with the first action-value function; A program characterized by causing the program to function as a

[0175] (Appendix 27) the calculation means calculates the first action-value function according to the first state and the second state; 27. The program according to claim 26,

[0176] (Appendix 28) The calculation means calculates the first action-value function for each of the first state and the second state; the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions. 28. The program according to claim 27,

[0177] (Appendix 29) The first state includes at least one of a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys the conveyed object, a position of a passage, and a position and a speed of a static or dynamic obstacle; 29. The program of any one of appendices 26 to 28.

[0178] (Appendix 30) The first state includes at least one of the following: the attitude and position of the construction machine, the shape of the soil to be excavated, and the amount of soil in the bucket of the excavator. 29. The program of any one of appendixes 26 to 28.

[0179] (Appendix 31) the first state includes a plurality of elements having attributes associated therewith; the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute. 31. The program according to any one of appendices 26 to 30.

[0180] (Appendix 32) the first state includes a state regarding a dynamic element moving within an environment; the generating means generates the second state by adding noise to a state of the dynamic element included in the first state. 28. The program according to claim 27,

[0181] [Additional Note 3] A part or all of the above-described embodiments can be further expressed as follows.

[0182] At least one processor, the processor comprising: An acquisition process for acquiring a first state in an environment that is a target of reinforcement learning; a generation process for generating a second state by adding noise to the first state; A calculation process of calculating a first action value function according to the second state; a selection process for selecting an action according to the first action-value function; A reinforcement learning device that executes.

[0183] The reinforcement learning device may further include a memory, and the memory may store a program for causing the processor to execute the acquisition process, the generation process, the calculation process, and the selection process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0184] 1, 2, 3, 4, 5, 6, 7 Reinforcement Learning System 10, 50 Reinforcement learning device 11 Acquisition Department 12 Generation part 13 Calculation section 14,526 Selection Section 20, 40 terminals 30 Servers 41, 51 Communications Department 42, 52 Control unit 43 Input reception section 53 Storage section 421 Status Report 422 Action Execution Department 423 Compensation Department 521 Compensation Acquisition Department 522 Status Observation Unit 523 State Randomization Unit 524 Learning Department 525 Estimation Department

Claims

1. An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning; generating means for generating a second state by adding noise to the first state; A calculation means for calculating a first action value function according to the second state; a selection means for selecting an action in accordance with the first action-value function; Equipped with The calculation means calculates the first action-value function for each of the first state and the second state; the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions. A reinforcement learning system characterized by:

2. The first state includes at least one of a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys an object, a position of a passage, and a position and a speed of a static or dynamic obstacle; The reinforcement learning system of claim 1 .

3. The first state includes at least one of the following: an attitude and a position of a construction machine, a shape of soil to be excavated, and an amount of soil in a bucket of an excavator. The reinforcement learning system according to claim 1 or 2.

4. the first state includes a plurality of elements having attributes associated therewith; the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute. The reinforcement learning system according to any one of claims 1 to 3.

5. the first state includes a state regarding a dynamic element moving within an environment; the generating means generates the second state by adding noise to a state of the dynamic element included in the first state. The reinforcement learning system of claim 4 .

6. An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning; generating means for generating a second state by adding noise to the first state; A calculation means for calculating a first action value function according to the second state; a selection means for selecting an action in accordance with the first action-value function; Equipped with The calculation means calculates the first action-value function for each of the first state and the second state; the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions. A reinforcement learning device characterized by:

7. Obtaining a first state of an environment that is the subject of reinforcement learning; generating a second state by adding noise to the first state; calculating a first action-value function in response to the second state; selecting an action in response to said first action-value function; Including, In the calculating step, the first action-value function is calculated for each of the first state and the second state; In the selecting step, the action is selected according to a second action-value function calculated based on a plurality of the first action-value functions. Reinforcement learning methods.

8. A program for causing a computer to function as a reinforcement learning device, the program comprising: An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning; generating means for generating a second state by adding noise to the first state; A calculation means for calculating a first action value function according to the second state; a selection means for selecting an action in accordance with the first action-value function; Functioning as a The calculation means calculates the first action-value function for each of the first state and the second state; the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions. program.

9. An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning; generating means for generating a second state by adding noise to the first state; A calculation means for calculating a first action value function according to the second state; a selection means for selecting an action in accordance with the first action-value function; Equipped with the first state includes a plurality of elements having attributes associated therewith; the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute. A reinforcement learning system characterized by:

10. An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning; generating means for generating a second state by adding noise to the first state; A calculation means for calculating a first action value function according to the second state; a selection means for selecting an action in accordance with the first action-value function; Equipped with the first state includes a plurality of elements having attributes associated therewith; the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute. A reinforcement learning device characterized by:

11. Obtaining a first state of an environment that is the subject of reinforcement learning; generating a second state by adding noise to the first state; calculating a first action-value function in response to the second state; selecting an action in response to said first action-value function; Including, the first state includes a plurality of elements having attributes associated therewith; In the generating step, the second state is generated by selectively adding noise to a plurality of elements included in the first state according to the attribute. Reinforcement learning methods.

12. A program for causing a computer to function as a reinforcement learning device, the program comprising: An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning; generating means for generating a second state by adding noise to the first state; A calculation means for calculating a first action value function according to the second state; a selection means for selecting an action in accordance with the first action-value function; Functioning as a the first state includes a plurality of elements having attributes associated therewith; the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute. program.

Images