Information processing device, information processing method, and information processing program

Abstract

To perform reinforcement learning while being applied to a device.SOLUTION: An information processing device 10A includes an acquisition section 40A, a first action value function identification section 40B, a second action value function identification section 40C, and an action determination section 40D. The acquisition section 40A acquires a current state of a mobile robot 20 as an example of a device. The first action value function identification section 40B has a learning function of learning a first inference model by reinforcement learning, and identifies a first action value function of the mobile robot 20 on the basis of the current state and the first inference model. The second action value function identification section 40C identifies a second action value function of the mobile robot 20 on the basis of the current state and a second inference model that is not a parameter update target. The action determination section 40D determines a first action of the mobile robot 20 on the basis of the first action value function and the second action value function.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Behavioral learning is being carried out using reinforcement learning. In reinforcement learning, optimal behavior of a device is learned by repeating various actions through trial and error. When learning is carried out using actual devices such as robots, the device may fall into an unintended state, causing accidents or breakdowns during learning. For this reason, ensuring safety during learning is required.

[0003] For example, a method has been disclosed in which a rule-based framework for safe operation is determined and behavioral parameters are determined within the determined framework by reinforcement learning. Also, a method has been proposed in which control variables generated by a learning-based algorithm are corrected so as not to deviate from the control variables generated by the rule-based algorithm.

[0004] However, because reinforcement learning within the framework involves random behavior search, conventional techniques have low performance immediately after the start of learning and take time to reach the required performance. For this reason, it has been difficult to learn the behavior of a device through reinforcement learning while applying it to the device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-105891 [Patent Document 2] Patent No. 6663822 Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in view of the above, and aims to provide an information processing device, an information processing method, and an information processing program that are capable of performing reinforcement learning while being applied to a device. [Means for solving the problem]

[0007] An information processing device according to an embodiment includes an acquisition unit, a first action value function identification unit, a second action value function identification unit, and an action decision unit. The acquisition unit acquires a current state of a device. The first action value function identification unit has a learning function for learning a first inference model through reinforcement learning, and identifies a first action value function of the device based on the current state and the first inference model. The second action value function identification unit identifies a second action value function of the device based on the current state and a second inference model that is not subject to parameter updating. The action decision unit determines a first action of the device based on the first action value function and the second action value function. The behavior decision unit includes an action value function correction unit that identifies a fourth action value function obtained by correcting the first action value function based on the second action value function, a third action value function identification unit that identifies the maximum function of the fourth action value function and the second action value function as a third action value function, and a behavior selection unit that selects the first behavior based on the third action value function. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram of an information processing system. [Figure 2] FIG. 1 is a functional block diagram of an information processing system. [Figure 3A] Schematic diagram of the first action-value function in the early stages of learning. [Figure 3B] Schematic diagram of the second action-value function. [Figure 4] FIG. 10 is an explanatory diagram of the transition of the first selection probability and the second selection probability. [Figure 5] Schematic diagram of a display screen. [Figure 6] Schematic diagram of a display screen. [Figure 7] Schematic diagram of a display screen. [Figure 8] 1 is a flowchart of the flow of information processing executed by an information processing device. [Figure 9] 10 is a flowchart of the flow of information processing executed by a mobile robot. [Figure 10] An illustration of the reward line. [Figure 11]FIG. 1 is a functional block diagram of an information processing system. [Figure 12A] Schematic diagram of the first action-value function in the early stages of learning. [Figure 12B] Schematic diagram of the second action-value function in the early stages of learning. [Figure 12C] Schematic diagram of the relationship between the third action value function and the first and second action value functions in the early stages of learning. [Figure 13A] Schematic diagram of the first action-value function in the learning progression stage. [Figure 13B] Schematic diagram of the second action-value function in the learning progression stage. [Figure 13C] A schematic diagram of the relationship between the third action value function and the first and second action value functions during the learning progression stage. [Figure 14] Schematic diagram of a display screen. [Figure 15] Schematic diagram of a display screen. [Figure 16] 1 is a flowchart of the flow of information processing executed by an information processing device. [Figure 17] FIG. 1 is a functional block diagram of an information processing system. [Figure 18] Schematic diagram of a display screen. [Figure 19A] Schematic diagram of the first action-value function in the early stages of learning. [Figure 19B] Schematic diagram of the second action-value function in the early stages of learning. [Figure 19C] FIG. 10 is an explanatory diagram of the relationship between the first action-value function and the fourth action-value function. [Figure 19D] Illustrative illustration of a specific third action-value function. [Figure 20] 1 is a flowchart of the flow of information processing executed by an information processing device. [Figure 21] Hardware configuration diagram. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of an information processing device, an information processing method, and an information processing program will be described in detail with reference to the accompanying drawings.

[0010] (First embodiment) FIG. 1 is an explanatory diagram showing an example of an information processing system 1 according to the present embodiment.

[0011] The information processing system 1 includes an information processing device 10A, a mobile robot 20, and an operation control system 30. The information processing device 10A, the mobile robot 20, and the operation control system 30 are connected to each other wirelessly or via a wire so as to be able to communicate with each other.

[0012] The information processing device 10A is an example of the information processing device 10. The information processing device 10 is an information processing device that learns the behavior of the mobile robot 20 by reinforcement learning.

[0013] The mobile robot 20 is an example of a device. A device is an object that executes behavior learned by the information processing device 10. Behavior is sometimes referred to as movement. A device is a mobile object in which at least some of its mechanisms operate. A device operates by being controlled by a computer. A mobile object may be one in which at least some of its mechanisms operate, and may be one in which the position of the main body of the mobile object moves, or one in which the position of the main body of the mobile object is fixed and does not move. Examples of devices include a work robot with a manipulator, a mobile robot with a position movement mechanism, an automobile, a ship, and an air vehicle. A specific example of an automobile is an autonomously driven car. An air vehicle is, for example, a drone.

[0014] In this embodiment, a mobile robot 20 is used as an example of the device.

[0015] The mobile robot 20 has, for example, a predetermined function. FIG. 1 shows an example in which the mobile robot 20 has a transport function of transporting luggage or the like while autonomously moving within a predetermined area such as a factory or warehouse. For example, the mobile robot 20 moves autonomously from a starting point A toward a destination point B. The starting point A may be a waiting location for the mobile robot 20, and the destination point B may be a storage location for the luggage to be transported.

[0016] The departure point information indicating the departure point A and the destination point information indicating the destination point B are transmitted to the mobile robot 20 from, for example, the operation control system 30. The operation control system 30 is an information processing device that controls the operation of the mobile robot 20.

[0017] Between the starting point A and the destination point B, for example, there are various obstacles C and no-entry areas D. The obstacles C are objects that hinder the movement of the mobile robot 20. The obstacles C are, for example, other mobile robots, walls, shelves, doors, people, etc. The no-entry areas D are areas into which the mobile robot 20 is prohibited. The mobile robot 20 needs to move safely and efficiently through the drivable area E while avoiding collisions with the obstacles C and avoiding entry into the no-entry areas D.

[0018] The information processing device 10A uses reinforcement learning to learn the behavior of the mobile robot 20. The information processing device 10A of this embodiment does not learn in advance, but learns the behavior of the mobile robot 20 by reinforcement learning while using the actual mobile robot 20 in the field.

[0019] 2 is a functional block diagram of an example of an information processing system 1A. The information processing system 1A is an example of the information processing system 1.

[0020] The information processing system 1 includes an information processing device 10A, a mobile robot 20, and an operation control system 30. The information processing device 10A, the mobile robot 20, and the operation control system 30 are communicably connected via a network N or the like.

[0021] The mobile robot 20 includes a control unit 22, a communication unit 24, a sensor 26, and a memory unit 28. The control unit 22, the communication unit 24, the sensor 26, and the memory unit 28 are communicatively connected via a bus 29 or the like.

[0022] The control unit 22 executes information processing in the mobile robot 20. The control unit 22 controls the behavior of the mobile robot 20. The communication unit 24 is a communication interface that communicates with the information processing device 10A and the operation control system 30 via the network N or the like.

[0023] The sensor 26 detects the mobile robot 20 and the surrounding environment of the mobile robot 20. In this embodiment, the sensor 26 detects the position information, vehicle speed information, acceleration information, environmental information, etc. of the mobile robot 20. The environmental information is information that represents the state of the surroundings of the mobile robot 20. The environmental information is information that represents, for example, photographed images of the surroundings of the mobile robot 20, detection results of objects such as obstacle C around the mobile robot 20, etc.

[0024] The sensor 26 may be any sensor that can detect the mobile robot 20 and the surrounding environment of the mobile robot 20. The sensor 26 may be, for example, a camera, a radar device, a laser range finder, a vehicle speed sensor, or an acceleration sensor.

[0025] The sensors 26 may be arranged at any position and in any number as long as they are capable of detecting the mobile robot 20 and the environment surrounding the mobile robot 20. The sensors 26 may also be mounted on the outside of the mobile robot 20. For example, the sensors 26 may be installed in the mobile environment in which the mobile robot 20 moves. The sensors 26 may also be installed in both the mobile robot 20 and the mobile environment in which the mobile robot 20 moves.

[0026] The control unit 22 transmits the detection results of the sensor 26 to the information processing device 10A. In this embodiment, the control unit 22 transmits the detection results of the sensor 26 to the information processing device 10A at predetermined time intervals. That is, the control unit 22 of the mobile robot 20 sequentially transmits successive detection results in time series to the information processing device 10A.

[0027] The storage unit 28 stores various types of data and is, for example, a semiconductor memory element such as a RAM (Random Access Memory), a flash memory, a hard disk, or an optical disk.

[0028] The operation control system 30 is an information processing device that controls the operation of the mobile robot 20. In this embodiment, the operation control system 30 transmits map information to the mobile robot 20. Note that the operation control system 30 may transmit map information to both the mobile robot 20 and the information processing device 10A.

[0029] The map information includes starting point information, destination point information, no-entry area information, drivable area information, obstacle information, etc. The starting point information is information that represents the starting point. The destination point information is information that represents the destination. The no-entry area information is information that represents the no-entry area. The drivable area information is information that represents the drivable area in which the mobile robot 20 can travel. Specifically, the starting point information, destination point information, no-entry area information, and drivable area information are information that represent the starting point A, destination point B, no-entry area D, and drivable area E, respectively, in FIG. 1.

[0030] The obstacle information is information that represents an obstacle C. In this embodiment, the obstacle information included in the map information is information that represents an obstacle C that is unlikely to move or that moves infrequently. Examples of obstacles C that are unlikely to move or that move infrequently include walls, shelves, doors, etc.

[0031] The information processing device 10A includes a control unit 40, a storage unit 42, an input unit 44, a display unit 46, and a communication unit 48. The control unit 40, the storage unit 42, the input unit 44, the display unit 46, and the communication unit 48 are communicatively connected via a bus 49 or the like.

[0032] The storage unit 42 stores various types of data. The storage unit 42 is, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, an optical disk, or the like. The storage unit 42 may be a storage device provided outside the information processing device 10A. The storage unit 42 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN (Local Area Network), the Internet, or the like.

[0033] The input unit 44 receives various operations from the user and is, for example, a keyboard, a pointing device such as a mouse, a microphone, and the like.

[0034] The display unit 46 displays various types of information, such as settings related to the mobile robot 20, the state of the mobile robot 20, and the status of work being performed by the mobile robot 20.

[0035] The display unit 46 and the input unit 44 may be integrally configured as a touch panel. Alternatively, the input unit 44 and the display unit 46 may be configured as separate units from the information processing device 10B. In this case, the input unit 44 and the display unit 46 may be configured to be communicably connected to the information processing device 10B.

[0036] The communication unit 48 is a communication interface that communicates with the mobile robot 20 and the operation control system 30 via a network N or the like.

[0037] At least one of the storage unit 42 and the control unit 40 may be mounted on an external information processing device such as a server device connected via the network N and the communication unit 48. Furthermore, at least one of the functional units included in the control unit 40, which will be described later, may be mounted on an external information processing device such as a server device connected to the control unit 40 via the network N and the communication unit 48.

[0038] The control unit 40 executes various information processes in the information processing device 10A.

[0039] The control unit 40 includes an acquisition unit 40A, a first action value function specification unit 40B, a second action value function specification unit 40C, a behavior determination unit 40D, a display control unit 40E, and a transmission control unit 40F.

[0040] The acquisition unit 40A, the first action-value function identification unit 40B, the second action-value function identification unit 40C, the action decision unit 40D, the display control unit 40E, and the transmission control unit 40F are realized by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC, i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0041] The acquisition unit 40A acquires the current state of the mobile robot 20. The acquisition unit 40A acquires the current state of the mobile robot 20 using the detection results of the sensor 26 received from the mobile robot 20 and map information received from the operation control system 30. The acquisition unit 40A may also receive map information from the mobile robot 20.

[0042] The current state of the mobile robot 20 is the current state information of the mobile robot 20. The state information includes state information of the mobile robot 20 itself, state information of surrounding obstacles, and map information.

[0043] The state information of the mobile robot 20 body is information that indicates the state of the mobile robot 20 body. The state information of the mobile robot 20 body is, for example, position information, speed information, acceleration information, and the like of the mobile robot 20.

[0044] For example, the acquisition unit 40A acquires position information of the mobile robot 20 by analyzing images captured by multiple cameras included in the detection results acquired from the mobile robot 20. For example, the acquisition unit 40A identifies the position coordinates of the mobile robot 20 by analyzing images of the mobile robot 20 captured by multiple sensors 26 installed in the movement environment of the mobile robot 20. Then, the acquisition unit 40A acquires the identified position coordinates as position information of the mobile robot 20.

[0045] Furthermore, for example, the acquisition unit 40A acquires the vehicle speed information and acceleration information of the mobile robot 20 by identifying the vehicle speed information and acceleration information included in the detection results acquired from the mobile robot 20. The acquisition unit 40A may acquire the vehicle speed information and acceleration information of the mobile robot 20 by analyzing time-series photographed images of the mobile robot 20, etc.

[0046] The status information of the surrounding obstacle is information that represents the status of the obstacle C that exists around the mobile robot 20. The status information of the surrounding obstacle includes, for example, position information and speed information of the obstacle C that exists around the mobile robot 20. The acquisition unit 40A acquires the position information and speed information of the obstacle C by analyzing the detection results of the sensor 26, such as a camera or a radar device. The acquisition unit 40A may acquire the relative position and relative speed of the obstacle C with respect to the mobile robot 20 as the position information and speed information of the obstacle C. The acquisition unit 40A may also acquire captured images of the periphery of the mobile robot 20 as the status information of the surrounding obstacle. The acquisition unit 40A may also acquire the status information of the surrounding obstacle by identifying obstacle information included in map information.

[0047] When the acquisition unit 40A acquires the current state of the mobile robot 20, it calculates a reward for the acquired current state. The reward is a reward used in reinforcement learning of the first inference model provided in the first action-value function identification unit 40B, which will be described later.

[0048] The acquisition unit 40A calculates a reward to be used in reinforcement learning of the first inference model based on the desirability of the current state, which is the result of the behavior of the mobile robot 20.

[0049] In detail, the more favorable the current state resulting from the behavior of the mobile robot 20 is, the larger the reward value calculated by the obtaining unit 40A. On the other hand, the more unfavorable the current state resulting from the behavior of the mobile robot 20 is, the smaller the reward value calculated by the obtaining unit 40A.

[0050] The desirable state means a state in which the mobile robot 20 behaves in accordance with a predetermined standard. The standard may be determined in advance. For example, the desirable state may represent avoiding a collision with an obstacle C, avoiding entry into a no-entry area D, reaching a destination point B, reaching a destination point B in a short time, or making multiple round trips from a starting point A to a destination point B.

[0051] The acquisition unit 40A determines whether the behavior of the mobile robot 20 is desirable or not based on the acquired current state of the mobile robot 20, thereby calculating a reward.

[0052] For example, in the following cases, the acquisition unit 40A determines that the behavior of the mobile robot 20 is favorable and calculates a larger reward value. For example, assume that the acquisition unit 40A determines that the mobile robot 20 has reached destination point B based on the acquired current state. In this case, the acquisition unit 40A calculates a larger reward value. Also assume that the acquisition unit 40A determines the time required to reach destination point B from starting point A based on the acquired current state. In this case, the acquisition unit 40A calculates a larger reward value as the time required is shorter. Also assume that the acquisition unit 40A calculates the number of round trips per unit time from starting point A to destination point B based on the acquired current state. In this case, the acquisition unit 40A calculates a larger reward value as the number of round trips is greater.

[0053] Furthermore, for example, in the following cases, the acquisition unit 40A determines that the behavior of the mobile robot 20 is in an undesirable state and calculates a smaller reward value, for example, a negative value. For example, assume that the acquisition unit 40A determines that the mobile robot 20 is in an undesirable state based on the acquired current state. For example, assume that the acquisition unit 40A determines, based on the current state, that the destination point B cannot be reached, that the mobile robot 20 has entered a no-entry area D, or that the mobile robot 20 has collided with an obstacle C. In this case, the acquisition unit 40A calculates a smaller reward value.

[0054] Next, the first action value function specifying unit 40B will be described. The first action value function specifying unit 40B has a learning function that learns the first inference model through reinforcement learning. The first action value function specifying unit 40B specifies the first action value function of the mobile robot 20 based on the current state and the first inference model.

[0055] The first inference model is a machine learning model that outputs a first action-value function based on the current state. That is, the first inference model is a machine learning model that takes the current state as input and outputs the first action-value function. The first inference model is also a machine learning model that is the target of reinforcement learning by the first action-value function identifying unit 40B. The first inference model is configured, for example, by a neural network.

[0056] The first action-value function is an action-value function output from the first inference model.

[0057] The action value function is a known action value function used in reinforcement learning, and is a function that represents the action value when action a is selected in a trial at timing t. Timing t represents the number of trials. Timing t may also be a time of day. In this embodiment, a description will be given assuming that timing t is time t.

[0058] Action value is the expected value of the discounted cumulative reward that can be obtained as a result of selecting action a in the current state. For example, in the reinforcement learning algorithm DQN (Deep Q-Network), action a is selected based on the action value function.

[0059] The first action-value function is expressed as Q1(s, a), for example, where s represents the current state and a represents the action.

[0060] FIG. 3A is a schematic diagram showing an example of a first action-value function in the early learning stage. The early learning stage indicates that the first inference model is in the early learning stage. In FIG. 3A, the horizontal axis represents action and the vertical axis represents action value. Diagram 51 represents the first action-value function.

[0061] Returning to Figure 2, we continue the explanation.

[0062] The first action-value function specifying unit 40B uses the current state and reward of the mobile robot 20 acquired by the acquiring unit 40A to learn the first inference model by reinforcement learning so as to output the first action-value function.

[0063] As described above, the reward is calculated by the acquisition unit 40A based on the acquired current state of the mobile robot 20. The mobile robot 20 behaves in accordance with a first behavior sequentially determined by the behavior determination unit 40D, which will be described later.

[0064] The first action is information representing the next action to be taken by the mobile robot 20. The first action is determined by the action determination unit 40D, which will be described later. That is, the first action is information representing the next action to be taken by the mobile robot 20, which is determined by the information processing device 10A. The first action is, for example, information representing the traveling direction of the mobile robot 20. The traveling direction represents, for example, one of right, left, straight, and stop. Note that the first action may be information representing the traveling direction and target speed of the mobile robot 20. Furthermore, the first action may be information representing the next action to be taken by the mobile robot 20, and is not limited to the traveling direction and target speed. For example, the first action may be determined according to the operating mechanism of a device such as the mobile robot 20.

[0065] The current state of the mobile robot 20 changes depending on the action of the mobile robot 20 according to the first action. Therefore, the reward is a value according to the first action output to the mobile robot 20.

[0066] Therefore, the first action value function identification unit 40B learns the first inference model through reinforcement learning using the current state of the mobile robot 20 acquired by the acquisition unit 40A, the reward, and the first action of the mobile robot 20 used to control the current situation used to calculate the reward.

[0067] In detail, the first action-value function specifying unit 40B determines the current state s of the mobile robot 20 acquired by the acquiring unit 40A at time t.t , reward r t , the reward r t The action a used to control the current situation of the mobile robot 20 used in the calculation of t , and the current state s acquired by the acquisition unit 40A at the next time t+1. t+1 , is stored in the storage unit 42. Then, the first action-value function specifying unit 40B uses the experience data to learn the first inference model by reinforcement learning.

[0068] A known reinforcement learning algorithm may be used, such as Deep Q-Network (DQN), Sarsa, or Actor-Critic.

[0069] Assume that reinforcement learning is performed using DQN. Also, the empirical data x = (s t ,a t ,r t ,s t+1 ) is sampled. In this case, the first action-value function specification unit 40B performs backpropagation using the loss function expressed by the following equation (1) to determine the update amount for each of the multiple parameters included in the first inference model.

[0070] L=(r t +γmax{Q1(s t+1 ,a t+1 )}-Q1(s t ,a t )) 2 Formula (1)

[0071] In equation (1), L represents the loss function. t represents the reward at time t. γ is a value called the discount rate in reinforcement learning. Q1(s t+1 ,a t+1 ) represents the first action value function at the next time t+1. Q1(s t ,a t ) represents the first action-value function at time t. t represents the current state s at time t. t represents the behavior at time t.

[0072] Backpropagation is an algorithm used in machine learning to train neural networks. Backpropagation is also sometimes called backpropagation.

[0073] The first action-value function identification unit 40B inputs the current state acquired by the acquisition unit 40A into the first inference model that has undergone reinforcement learning through the above process, and identifies the first action-value function to be used at time t as an output from the first inference model.

[0074] The first action-value function specifying section 40B outputs the first action-value function specified using the first inference model to the action decision-making section 40D.

[0075] Next, the second action value function specifying unit 40C will be described. The second action value function specifying unit 40C specifies the second action value function of the mobile robot 20 based on the current state and the second inference model.

[0076] The second inference model is a model that outputs a second action-value function based on the current state. That is, the second inference model is a model that takes the current state as input and outputs the second action-value function. The second inference model is also a model that is not subject to parameter update. That is, the second inference model is a model that is not subject to reinforcement learning. For example, the second inference model is a machine learning model, a statistical model, etc.

[0077] In this embodiment, an example will be described in which the second inference model is a machine learning model configured using a neural network or the like. Also, in this embodiment, an example will be described in which the second inference model is a pre-trained, trained machine learning model. A known learning algorithm may be used for the pre-training of the second inference model. For example, an offline reinforcement learning algorithm such as Conservative Q Learning may be used for the pre-training of the second inference model.

[0078] The training data used for pre-training the second inference model may be collected in any manner, such as empirical data collected by humans operating a robot, empirical data collected using a random policy, etc.

[0079] In this embodiment, a case will be described as an example in which the second inference model is learned in advance using data on the behavior of the mobile robot 20 based on the first rule.

[0080] The behavior of the mobile robot 20 based on the first rule is information that represents the next behavior that the mobile robot 20 will take. In other words, the behavior of the mobile robot 20 based on the first rule is information that represents behavior that is determined on a rule basis. The behavior of the mobile robot 20 based on the first rule may be, for example, information that represents the traveling direction of the mobile robot 20, similar to the first behavior. Furthermore, the behavior may be information that represents the traveling direction and target speed of the mobile robot 20, similar to the first behavior.

[0081] The first rule is a rule that satisfies minimum required performance regarding the behavior of the mobile robot 20. The minimum required performance is a minimum standard of required performance for the mobile robot 20 that is determined in advance by the manager of the mobile robot 20 or the like.

[0082] For example, the first rule is to find A from the departure point information, destination point information, and map information. * The first rule is a rule that indicates that the mobile robot 20 will proceed along a path determined by a search algorithm such as the (A-star) method. The first rule is a rule that indicates that, if an obstacle C is present, the mobile robot will stop in front of the obstacle C or will proceed while avoiding the obstacle C. If the rule indicates that the mobile robot 20 will stop in front of the obstacle C, a collision with the obstacle C will be avoided. However, the greater the number of obstacles C that the mobile robot 20 detects before reaching the destination point B, the lower the behavioral efficiency of the mobile robot 20 per unit time. For this reason, it is preferable to set the first rule so that it satisfies both the safety guarantee and the minimum required performance.

[0083] The second action value function is an action value function output from the second inference model. The definition of the action value function has been described above, so a detailed explanation will be omitted here. The second action value function is expressed as Q2(s, a), for example. s represents the current state, and a represents the action.

[0084] 3B is a schematic diagram showing an example of the second action-value function. In FIG. 3B, the horizontal axis represents action, and the vertical axis represents action value. Diagram 52 represents the second action-value function.

[0085] Returning to Figure 2, we continue the explanation.

[0086] The second action-value function identification unit 40C inputs the current state acquired by the acquisition unit 40A into the second inference model, and identifies the second action-value function to be used at time t as an output from the second inference model.

[0087] As described above, in this embodiment, the second action-value function specifying unit 40C specifies the second action-value function using the second inference model that has been trained in advance using data on the actions of the mobile robot 20 based on the first rule. Therefore, the action that maximizes the action value represented by the second action-value function in the current state represents the action determined on a rule-based basis according to the first rule.

[0088] The second action-value function specifying section 40C outputs the second action-value function specified using the second inference model to the action decision-making section 40D.

[0089] Next, the behavior determining unit 40D will be described.

[0090] Based on the first action-value function and the second action-value function, the action decision unit 40D decides the first action of the mobile robot 20. As described above, the first action is information indicating the next action that the mobile robot 20 will take.

[0091] For example, the behavior decision unit 40D decides, as the first behavior, an behavior that maximizes either the behavior value represented by the first behavior value function or the behavior value represented by the second behavior value function. In other words, the behavior decision unit 40D decides, as the first behavior, an behavior that corresponds to the maximum behavior value among the behavior values ​​represented by the first behavior value function or the maximum behavior value among the behavior values ​​represented by the second behavior value function.

[0092] The behavior decision unit 40D may select either the first or second behavior value function as a third behavior value function and determine the first behavior based on the selected third behavior value function. The third behavior value function is expressed as Q3(s, a), for example, where s represents the current state and a represents the behavior.

[0093] For example, the behavior decision unit 40D changes the first selection probability of selecting the first action value function as the third action value function and the second selection probability of selecting the second action value function as the third action value function according to the learning time of the first inference model. The longer the learning time, the more progress has been made in the learning of the first inference model. A longer learning time indicates more progress in the learning. The learning time is represented, for example, by the number of trials or cumulative calculation time of the first inference model.

[0094] The first selection probability represents the selection probability that the action decision unit 40D selects the first action value function as the third action value function. In other words, the first selection probability represents the selection probability that the action corresponding to the maximum action value among the action values ​​represented by the first action value function is selected as the first action.

[0095] The second selection probability represents the selection probability that the action decision unit 40D selects the second action value function as the third action value function. In other words, the second selection probability represents the selection probability that the action corresponding to the maximum action value among the action values ​​represented by the second action value function is selected as the first action.

[0096] In this case, the behavior determining unit 40D lowers the first selection probability and increases the second selection probability as the study time becomes shorter, and increases the first selection probability and decreases the second selection probability as the study time becomes longer.

[0097] That is, the behavior determining unit 40D sets the first selection probability lower than the second selection probability at the start of learning, and then linearly increases the first selection probability as learning time passes.

[0098] Figure 4 is an explanatory diagram of an example of the transition of the first selection probability and the second selection probability. The first selection probability is ε, and the second selection probability is 1-ε. The learning time is t. In Figure 4, the vertical axis represents the first selection probability, and the horizontal axis represents the learning time.

[0099] For example, the first selection probability at the start of learning, i.e., t = 0, is set to ε = ε0. The maximum learning time is t max Let the first choice probability at time t be ε = ε0(t / t max ) In this case, the second selection probability is 1-ε = ε0(1-t / t max )

[0100] The behavior decision-making unit 40D sets the first selection probability low at the start of learning. For example, ε = ε0 = 0.05. Then, as learning time passes, the behavior decision-making unit 40D increases the first selection probability and decreases the second selection probability. Therefore, the first selection probability ε at which the behavior decision-making unit 40D selects the first action-value function as the third action-value function is represented, for example, by the diagram 50 shown in FIG. 4.

[0101] Therefore, in the early stages of learning when the first inference model is not yet fully learned, the probability that the second action value function will be selected as the third action value function increases. As the learning of the first inference model progresses, the probability that the first action value function will be selected as the third action value function increases.

[0102] Then, the action decision unit 40D decides the first action based on the third action value function. For example, the action corresponding to the maximum action value in the third action value function is decided as the first action. Specifically, the first action is expressed by the following formula (A).

[0103] arg max{Q1(s,a)} Equation (A)

[0104] The search method used by the action determination unit 40D is not limited to this method. For example, the action determination unit 40D may perform a Boltzmann search using a third action value function. This search increases the probability that the action with the highest selection probability, i.e., the action corresponding to the largest action value in the third action value function, will be determined as the first action. Alternatively, for example, the action determination unit 40D may determine the first action by an epsilon-ready search using the third action value function.

[0105] When performing a Boltzmann search using the third action value function, the selection probability p(a|s) of selecting action a in the current state s is expressed by the following formula (2).

[0106] p(a|s)=exp(Q3(s,a) / T) / Σexp(Q3(s,b) / T) Formula (2)

[0107] In equation (2), Q3(s, a) represents the third action value function. T represents the Boltzmann temperature. Σ represents calculating the sum of Q3(s, b) for possible actions b∈A. A represents the set of possible actions, i.e., the action space. s represents the current state. a represents an action. b in Q3(s, b) is an element of the action space A, i.e., one of the possible actions.

[0108] Through these processes, the action decision section 40D decides, as the first action, the action corresponding to the maximum action value in the first action value function selected as the third action value function or the second action value function selected as the third action value function.

[0109] Therefore, by the action decision unit 40D changing the first selection probability and the second selection probability according to the learning time of the first inference model, the probability that the action corresponding to the maximum action value in the second action value function will be determined as the first action becomes higher in the early stages of learning. Furthermore, as learning time passes and learning progresses, the probability that the action corresponding to the maximum action value in the first action value function will be determined as the first action becomes higher.

[0110] In this embodiment, the second inference model used to identify the second action-value function is learned in advance using data on the behavior of the mobile robot 20 based on the first rule, which is a rule that satisfies the minimum required performance regarding the behavior of the mobile robot 20.

[0111] Therefore, the behavior decision unit 40D can select an action that has a high potential reward as the first behavior, compared to when the first behavior is decided regardless of the progress of learning. In other words, the behavior decision unit 40D can decide on a first behavior that can learn the first inference model while satisfying the minimum required performance for the behavior of the mobile robot 20, from the early stage of learning of the first inference model to the stage where learning has progressed sufficiently.

[0112] 4 shows an example in which the first selection probability changes linearly with respect to the study time. However, the change in the first selection probability is not limited to a linear change. The change in the first selection probability may be changed as appropriate by a user's operation instruction on the input unit 44.

[0113] Returning to Figure 2, we continue the explanation.

[0114] The display control unit 40E displays various images on the display unit 46.

[0115] In this embodiment, the display control unit 40E displays on the display unit 46 information representing at least one of the progress of learning of the first inference model, the probability of selection by the behavior decision unit 40D of at least one of the first action value function and the second action value function, the number of times that the behavior decision unit 40D selects at least one of the first action value function and the second action value function, and whether the first action is the action that maximizes the action value represented by either the first action value function or the second action value function.

[0116] The progress of the learning of the first inference model displayed by the display control unit 40E is the progress at the current time. For example, the display control unit 40E displays the learning time of the first inference model, the time until the learning of the first inference model is completed, etc. as the progress at the current time. Therefore, the display control unit 40E can provide the user with an easily checkable progress status of the learning of the first inference model at the current time.

[0117] The selection probability displayed by the display control unit 40E is at least one of the first selection probability at the current time that the behavior decision unit 40D selects the first action value function as the third action value function and the second selection probability at the current time that the behavior decision unit 40D selects the second action value function as the third action value function. As described above, the first selection probability and the second selection probability change according to the progress of learning. Therefore, by displaying the selection probabilities, the display control unit 40E can provide the user with at least one of the first selection probability and the second selection probability at the current time in an easily recognizable manner.

[0118] The number of selections displayed by the display control unit 40E is at least one of the number of times the behavior decision unit 40D selected the first action value function as the third action value function and the number of times the behavior decision unit 40D selected the second action value function as the third action value function during the period from the start of learning of the first inference model to the current time. As described above, the first selection probability and the second selection probability change according to the progress of learning. Therefore, the number of times each of the first action value function and the second action value function is selected also changes according to the progress of learning. Therefore, by displaying the number of selections, the display control unit 40E can provide the user with an easily recognizable status of the selection of the first action value function and the second action value function by the behavior decision unit 40D at the current time.

[0119] The information displayed by the display control unit 40E, which indicates whether the first action maximizes the action value represented by the first action value function or the second action value function, is information indicating the first action selected by the action decision unit 40D at the current time. Therefore, the display control unit 40E can provide the user with information that allows them to easily confirm whether the next action that the mobile robot 20 will take at the current time is the action that maximizes the action value represented by the first action value function or the action that maximizes the action value represented by the second action value function.

[0120] The display control unit 40E may display these pieces of information on one display screen or on multiple different display screens. Furthermore, the display control unit 40E may display all of these pieces of information on the display screen at the same time, or may display at least one piece of information on the display screen at a different time from the other pieces of information.

[0121] 5 to 7 are schematic diagrams showing an example of the display screen 70. FIG.

[0122] 5 is a schematic diagram showing an example of a display screen 60. The display screen 60 is an example of a display screen 70.

[0123] The display screen 60 includes an image 60A representing the progress of learning of the first inference model, an image 60B representing a first selection probability, which is the probability of selecting the first action-value function, an image 60C representing the number of times the first action-value function is selected, and an image 60D representing the number of times the second action-value function is selected. Figure 5 shows an example of the image 60A representing the progress of learning of the first inference model, showing the time until the end of learning of the first inference model.

[0124] The display control unit 40E generates and displays the display screen 60 using the first action selected by the action decision unit 40D, the first selection probability ε of the first action value function used when selecting the first action, the first action value function used in selecting the first action, the second action value function, and the progress.

[0125] 6 is a schematic diagram showing an example of a display screen 62. The display screen 62 is an example of a display screen .

[0126] The display screen 62 includes information indicating whether the first action is determined by the first action-value function or the second action-value function. For example, assume that the candidate actions of the mobile robot 20 are "go straight," "right," "left," and "stop," which are directions of travel. Also assume that the first action-value function is selected as the third action-value function by the action-determining unit 40D, and the first action determined from the selected third action-value function indicates the direction of travel of "go straight."

[0127] In this case, for example, the display control unit 40E displays information 62C representing a learning-based action, which is an action determined from the first action-value function, and information 62D representing a rule-based action, which is an action determined from the second action-value function. The display control unit 40E then displays the selection icon for the information 62C in color and the selection icon for the information 62D in colorless. By displaying the selection icon for the information 62C in color, the display control unit 40E indicates that the first action has been determined from the first action-value function.

[0128] Furthermore, for example, the display control unit 40E displays an image 62A representing a candidate behavior of the mobile robot 20 around an icon image 62B representing the mobile robot 20. Then, the display control unit 40E displays the image representing the selected "go straight" in a color or brightness different from that of the other candidate behaviors. For example, the display control unit 40E lights up the image representing "go straight" and does not light up the images representing the other candidate behaviors. By lighting up the image representing "go straight", the display control unit 40E displays that the traveling direction "go straight" has been determined as the first action from the first action-value function.

[0129] 7 is a schematic diagram showing an example of a display screen 64. The display screen 64 is an example of a display screen .

[0130] The display screen 64 includes information indicating whether the first action is determined by the first action-value function or the second action-value function. For example, assume that the candidate actions of the mobile robot 20 are "go straight," "right," "left," and "stop," which are directions of travel. Also assume that the second action-value function is selected as the third action-value function by the action-determining unit 40D, and the first action determined from the selected third action-value function indicates a direction of travel of "left."

[0131] In this case, for example, the display control unit 40E displays information 64C representing a learning-based action, which is an action determined from the first action-value function, and information 64D representing a rule-based action, which is an action determined from the second action-value function. The display control unit 40E then displays the selection icon for the information 64D in color and the selection icon for the information 64C in colorless. By displaying the selection icon for the information 64D in color, the display control unit 40E indicates that the first action has been determined from the second action-value function.

[0132] Furthermore, for example, the display control unit 40E displays an image 64A representing a candidate action of the mobile robot 20 around an icon image 64B representing the mobile robot 20. Then, the display control unit 40E displays the image representing the selected "left" in a color or brightness different from that of the other candidate actions. For example, the display control unit 40E lights up the image representing "left" and does not light up the images representing the other candidate actions. By lighting up the image representing "left," the display control unit 40E displays that the traveling direction "left" has been determined as the first action from the second action-value function.

[0133] The display control unit 40E displays these display screens 70 on the display unit 46, thereby providing the user with various information such as the progress of learning in an easy-to-understand manner. Therefore, by checking the display screen 70, the user can easily understand whether learning of the first inference model regarding the behavior of the mobile robot 20 is in progress or has been completed.

[0134] Returning to Figure 2, we continue the explanation.

[0135] The transmission control unit 40F transmits the first action selected by the action determination unit 40D to the mobile robot 20 via the communication unit .

[0136] The control unit 22 of the mobile robot 20 receives map information from the operation control system 30 via the communication unit 24. The control unit 22 controls the driving unit of the mobile robot 20 so that the mobile robot 20 starts moving from a starting point A represented by starting point information included in the map information toward a destination point B represented by destination point information included in the map information. This process causes the mobile robot 20 to start moving.

[0137] When the control unit 22 of the mobile robot 20 starts an action, it transmits the detection results of the sensor 26 to the information processing device 10A. The control unit 22 sequentially transmits the detection results of the sensor 26 to the information processing device 10A at predetermined intervals. The control unit 22 receives a first action from the information processing device 10A in response to the transmitted detection results. The control unit 22 controls the driving unit of the mobile robot 20 so as to perform an action corresponding to the received first action.

[0138] Therefore, the mobile robot 20 executes the first action determined by the information processing device 10A. That is, the mobile robot 20 can move from the starting point A toward the destination point B by acting in accordance with the first actions sequentially transmitted from the information processing device 10A.

[0139] Next, an example of the flow of information processing executed by the information processing device 10A of this embodiment will be described.

[0140] FIG. 8 is a flowchart showing an example of the flow of information processing executed by the information processing device 10A of this embodiment.

[0141] The acquisition unit 40A acquires the current state of the mobile robot 20 (step S100). The acquisition unit 40A acquires the current state of the mobile robot 20 using the detection results of the sensor 26 received from the mobile robot 20 and the map information received from the operation management system 30.

[0142] Next, the acquisition unit 40A calculates the reward for the current state acquired in step S100 (step S102).

[0143] The first action value function identification unit 40B uses the current state obtained in step S100 and the reward calculated in step S102 to learn the first inference model through reinforcement learning so as to output a first action value function that obtains a current state with a higher reward (step S104).

[0144] The first action value function identification unit 40B identifies the first action value function (step S106). The first action value function identification unit 40B inputs the current state acquired in step S100 into the first inference model that underwent reinforcement learning in step S104, thereby identifying the first action value function as an output from the first inference model.

[0145] The second action value function identification unit 40C identifies the second action value function (step S108). The second action value function identification unit 40C inputs the current state acquired in step S100 into the second inference model, thereby identifying the second action value function as an output from the second inference model.

[0146] Next, the behavior determining unit 40D identifies the learning progress of the first inference model of the first action-value function identifying unit 40B (step S110). For example, the behavior determining unit 40D identifies the learning time of the first inference model as the learning progress.

[0147] The behavior decision unit 40D selects, according to the progress of the learning identified in step S110, either the first behavior value function identified in step S106 or the second behavior value function identified in step S108 as the third behavior value function (step S112).

[0148] The action determining section 40D determines the first action from the third action-value function selected in step S112 (step S114).

[0149] The transmission control unit 40F transmits the first action determined in step S114 to the mobile robot 20 (step S116).

[0150] The display control unit 40E displays the display screen 70 on the display unit 46 (step S118). Note that the timing for displaying the display screen 70 on the display unit 46 is not limited to this timing. For example, the display control unit 40E may start displaying the display screen 70 on the display unit 46 before the start of the processing of step S100, and update the display content of the display screen 70 according to the processing status by the control unit 40.

[0151] The control unit 40 determines whether or not to end the process (step S120). For example, the control unit 40 makes the determination in step S120 by determining whether or not a signal indicating the end of the process has been input by a user operating the input unit 44. If the determination in step S120 is negative (step S120: No), the process returns to step S100. If the determination in step S120 is positive (step S120: Yes), the present routine ends.

[0152] Next, an example of the flow of information processing executed by the mobile robot 20 of this embodiment will be described.

[0153] FIG. 9 is a flowchart showing an example of the flow of information processing executed by the mobile robot 20 of this embodiment.

[0154] The control unit 22 of the mobile robot 20 acquires map information from the operation control system 30 via the communication unit 24 (step S200). The control unit 22 controls the driving unit of the mobile robot 20 so that the mobile robot 20 starts moving from a starting point A represented by starting point information included in the map information toward a destination point B represented by destination point information included in the map information. This process causes the mobile robot 20 to start moving (step S202).

[0155] The control unit 22 transmits the detection result of the sensor 26 to the information processing device 10A (step S204).

[0156] The control unit 22 receives the first action from the information processing device 10A as a response to the detection result transmitted in step S204 (step S206). The control unit 22 controls the driving unit of the mobile robot 20 so as to execute an action corresponding to the received first action. Therefore, the mobile robot 20 executes the first action determined by the information processing device 10A (step S208).

[0157] The control unit 22 determines whether or not to end the process (step S210). For example, the control unit 22 determines whether or not to end the process by determining whether or not a process end signal has been input by an operation instruction from the user (step S210). If the determination in step S210 is negative (step S210: No), the process returns to step S204. On the other hand, if the determination in step S210 is positive (step S210: Yes), the present routine ends.

[0158] As described above, the information processing device 10A of this embodiment includes an acquisition unit 40A, a first action value function identification unit 40B, a second action value function identification unit 40C, and a behavior decision unit 40D. The acquisition unit 40A acquires the current state of the mobile robot 20, which is an example of a device. The first action value function identification unit 40B has a learning function that learns a first inference model through reinforcement learning, and identifies a first action value function of the mobile robot 20 based on the current state and the first inference model. The second action value function identification unit 40C identifies a second action value function of the mobile robot 20 based on the current state and a second inference model that is not subject to parameter updating. The behavior decision unit 40D determines a first behavior of the mobile robot 20 based on the first action value function and the second action value function.

[0159] In the prior art, reinforcement learning was performed by random behavioral search, which made it difficult to apply the prior art to devices such as the mobile robot 20 and learn the behavior of the device.

[0160] On the other hand, in the information processing device 10A of this embodiment, the first action of the mobile robot 20 is determined based on the first action value function identified from the first inference model that is the target of reinforcement learning and the second action value function identified from the second inference model that is not the target of parameter update.

[0161] The learning status of the information processing device 10A changes. Examples of the learning status include the progress of learning of the first inference model, the first action-value function that changes according to the progress of learning, and the fluctuation status of the first action-value function relative to the second action-value function. Therefore, by the action decision unit 40D determining the first action from the first action-value function and the second action-value function according to the learning status, it is possible to determine a first action that can learn the first inference model while satisfying the minimum required performance for the behavior of the mobile robot 20, from the early stage of learning of the first inference model to a stage where learning has progressed sufficiently.

[0162] Therefore, the information processing device 10A of this embodiment can learn the first inference model by reinforcement learning while operating the mobile robot 20, which is an actual device.

[0163] Therefore, the information processing device 10A of this embodiment can perform reinforcement learning while being applied to a device.

[0164] Furthermore, in the information processing device 10A of this embodiment, the behavior decision unit 40D selects either the first action value function or the second action value function as the third action value function, and decides on the first behavior based on the selected third action value function. As described above, the behavior decision unit 40D changes the first selection probability of selecting the first action value function as the third action value function and the second selection probability of selecting the second action value function as the third action value function according to the learning time of the first inference model, lowering the first selection probability and increasing the second selection probability as the learning time becomes shorter, and increasing the first selection probability and decreasing the second selection probability as the learning time becomes longer.

[0165] As described above, the first behavior probability function is an behavior value function determined using the first inference model. The second behavior value function is an behavior value function determined using the second inference model that has been trained in advance using data on the behavior of the mobile robot 20 based on the first rule, for example. The behavior decision unit 40D then selects either the first or second behavior value function as the third behavior value function depending on the learning time of the first inference model, and determines the first behavior using the third behavior value function.

[0166] Therefore, in the early stages of learning when the first inference model is not yet fully learned, the behavior decision unit 40D determines the first behavior using the second behavior value function determined on a rule basis using the first rule, and as the learning of the first inference model progresses, the probability of determining the first behavior using the first behavior value function determined on a learning basis becomes higher.

[0167] Therefore, in the early stage of learning, there is a high probability that the action that maximizes the action value represented by the second action-value function will be determined as the first action. The second inference model used to identify the second action-value function is learned in advance using data on the actions of the mobile robot 20 based on the first rule that satisfies the minimum required performance for the actions of the mobile robot 20. Therefore, the information processing device 10A of this embodiment can satisfy the minimum required performance for the actions of the mobile robot 20 in the early stage of learning.

[0168] Furthermore, as the learning time increases and the learning progresses, the probability that the action that maximizes the action value represented by the first action-value function will be determined as the first action increases. As the learning progresses, the reward based on the first action obtained by the first inference model used to identify the first action function increases. Therefore, the information processing device 10A of this embodiment can satisfy the minimum required performance for the behavior of the mobile robot 20 even at a stage where learning has progressed.

[0169] Therefore, the behavior decision unit 40D can decide on an action that will provide a higher reward as the first behavior than when deciding on the first behavior regardless of the progress of learning. In other words, the behavior decision unit 40D can decide on a first behavior that will enable reinforcement learning of the first inference model while satisfying the minimum required performance for the behavior of the mobile robot 20, from the early stage of learning of the first inference model to a stage where learning has progressed sufficiently.

[0170] FIG. 10 is an explanatory diagram of an example of a reward line.

[0171] In Fig. 10, the vertical axis represents reward, and the horizontal axis represents learning time. In Fig. 10, P1 represents the reward for satisfying the minimum required performance of the mobile robot 20. P2 represents the reward for satisfying the optimal required performance of the mobile robot 20. Diagram 59 represents the reward line according to the reinforcement learning method of this embodiment. Diagram 590 represents the reward line according to a conventional reinforcement learning method.

[0172] As shown in Figure 10, conventional reinforcement learning methods randomly select actions, especially in the early stages of learning. As a result, the mobile robot 20 is unable to reach the destination point B and therefore fails to obtain a reward. Furthermore, conventional reinforcement learning methods require time to reach the reward P2 that satisfies the optimal required performance. Furthermore, in conventional techniques, if the mobile robot 20 executes an action determined based on learning in the early stages of learning when the first inference model is not yet fully learned, the mobile robot 20 may fall into an unintended state, resulting in an accident or breakdown during reinforcement learning.

[0173] For this reason, in the prior art, it was difficult to perform reinforcement learning while applying it to a real device such as the mobile robot 20 from the viewpoint of realizing the required performance and ensuring safety.

[0174] On the other hand, the information processing device 10A of this embodiment can select either the first action value function or the second action value function as the third action value function depending on the learning time of the first inference model. That is, in the early stage of learning when reinforcement learning of the first inference model is insufficient, the information processing device 10A of this embodiment can select the second action value function of the second inference model as the third action value function. Furthermore, as the learning of the first inference model progresses, the information processing device 10A can select the first action value function of the first inference model as the third action value function. Therefore, the information processing device 10A of this embodiment can cause the mobile robot 20 to behave while collecting learning data and satisfying the minimum required performance throughout all stages of learning, from the early stage to the advanced stage.

[0175] Therefore, the information processing device 10A of this embodiment can learn the first inference model while operating the mobile robot 20, which is an actual device.

[0176] (Second embodiment) In this embodiment, an embodiment will be described in which an action value function obtained by combining the first action value function and the second action value function is used as the third action value function.

[0177] In this embodiment, the same functions and configurations as those in the above embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0178] 11 is a functional block diagram of an example of an information processing system 1B according to the present embodiment. The information processing system 1B is an example of the information processing system 1.

[0179] The information processing system 1B includes an information processing device 10B, a mobile robot 20, and an operation control system 30. The information processing device 10B, the mobile robot 20, and the operation control system 30 are communicatively connected via a network N or the like. The mobile robot 20 and the operation control system 30 are the same as those in the above embodiment.

[0180] The information processing device 10B is an example of the information processing device 10. The information processing device 10B includes a control unit 41, a storage unit 42, an input unit 44, a display unit 46, and a communication unit 48. The control unit 41, the storage unit 42, the input unit 44, the display unit 46, and the communication unit 48 are communicatively connected via a bus 49 or the like. The information processing device 10B is similar to the information processing device 10A of the above embodiment, except that the information processing device 10B includes a control unit 41 instead of the control unit 40.

[0181] The control unit 41 executes various information processes in the information processing device 10B.

[0182] The control unit 41 includes an acquisition unit 40A, a first action value function specification unit 40B, a second action value function specification unit 40C, a behavior decision unit 41D, a display control unit 41E, and a transmission control unit 40F. The acquisition unit 40A, the first action value function specification unit 40B, the second action value function specification unit 40C, and the transmission control unit 40F are the same as those in the above embodiment. That is, the control unit 41 is the same as the control unit 40 in the above embodiment, except that it includes a behavior decision unit 41D and a display control unit 41E instead of the behavior decision unit 40D and the display control unit 40E.

[0183] The behavior decision unit 41D decides the first behavior of the mobile robot 20 based on the first action-value function and the second action-value function, similar to the behavior decision unit 40D of the above embodiment.

[0184] In the above embodiment, the behavior determination unit 40D changes the first selection probability of selecting the first action value function as the third action value function and the second selection probability of selecting the second action value function as the third action value function according to the learning time of the first inference model. When the behavior determination unit 40D changes the first selection probability and the second selection probability according to the learning time, the user must manually adjust the change in the first selection probability ε in advance. For example, as described with reference to FIG. 4, when the first selection probability is set to increase linearly with the elapse of the learning time, the user must manually adjust the slope of the increase, which is the rate of change of the first selection probability with respect to the learning time.

[0185] Therefore, the behavior decision unit 41D of this embodiment has a third behavior value function specification unit 41G and a behavior selection unit 41H.

[0186] The third action value function specifying unit 41G specifies a third action value function obtained by combining the first action value function and the second action value function. In this embodiment, the third action value function specifying unit 41G specifies the maximum function of the first action value function and the second action value function as the third action value function obtained by combining the first action value function and the second action value function.

[0187] The maximum function of the first action value function and the second action value function represents the set of maximum action values ​​for each action among the action values ​​for actions represented by the first action value function and the action values ​​for actions represented by the second action value function.

[0188] In detail, the third action value function specifying unit 41G specifies the third action value function using the following equation (4).

[0189] Q3(s,a)=Max(Q1(s,a),Q2(s,a)) Equation (4)

[0190] In equation (4), Q3(s,a) represents the third action-value function. Q1(s,a) represents the first action-value function. Q2(s,a) represents the first action-value function. Max(Q1(s,a),Q2(s,a)) represents the maximum function of the first action-value function and the second action-value function. s represents the current state. a represents an action.

[0191] The action selection unit 41H selects the first action based on the third action value function identified by the third action value function identification unit 41G. The action selection unit 41H may select the first action from the third action value function by Boltzmann search or the like, in the same manner as the action decision unit 40D of the above embodiment.

[0192] An example of the processing by the behavior determining unit 41D will be described below using a specific example.

[0193] Fig. 12A is a schematic diagram showing an example of a first action value function in the early stage of learning. Fig. 12B is a schematic diagram showing an example of a second action value function in the early stage of learning. Fig. 12C is a schematic diagram showing an example of the relationship between a third action value function in the early stage of learning and the first and second action value functions. In Figs. 12A to 12C, line 51 represents the first action value function, line 52 represents the second action value function, and line 53 represents the third action value function.

[0194] For example, assume that the first action-value function in the early stage of learning is the function represented by the line 51 in FIG. 12A, and the second action-value function is the function represented by the line 52 in FIG. 12B.

[0195] As described in the above embodiment, the first behavior probability function is an behavior value function determined using the first inference model. The second behavior value function is an behavior value function determined using the second inference model that has been trained in advance using data on the behavior of the mobile robot 20 based on the first rule, for example. In other words, the behavior a2 that maximizes the behavior value represented by the second behavior value function shown by the line 52 in Fig. 12B coincides with the behavior obtained by controlling the mobile robot 20 based on the rule base using the first rule.

[0196] 12A and 12B, the maximum value of the action value of the first action value function represented by line 51 in the early stage of learning is smaller than the maximum value of the action value of the second action value function represented by line 52. Therefore, when the third action value function identification unit 41G identifies the maximum function of the first action value function and the second action value function as the third action value function, the third action value function represented by line 53 shown in Fig. 12C is identified.

[0197] When the action selection unit 41H selects the first action based on the third action value function represented by the line 53 shown in Figure 12C, the selection probability that the action a2 that maximizes the action value represented by the second action value function represented by the line 52 will be selected as the first action is increased.

[0198] Next, an example of an action value function at a learning progress stage where learning has progressed will be described.

[0199] Fig. 13A is a schematic diagram showing an example of a first action value function in a learning progress stage. Fig. 13B is a schematic diagram showing an example of a second action value function in a learning progress stage. Fig. 13C is a schematic diagram showing an example of the relationship between a third action value function in a learning progress stage and the first and second action value functions. In Figs. 13A to 13C, line 51 represents the first action value function, line 52 represents the second action value function, and line 53 represents the third action value function.

[0200] For example, consider a situation in which the first action-value function in the learning progression stage is the function represented by line 51 in Figure 13A. The second action-value function is an action-value function output from the second inference model that does not update parameters, and therefore becomes an action-value function represented by line 52 similar to line 52 shown in Figure 12B.

[0201] 13A and 13B, the maximum value of the action value of the first action value function represented by line 51 in the learning progress stage is greater than the maximum value of the action value of the second action value function represented by line 52. Therefore, when the third action value function identification unit 41G identifies the maximum function of the first action value function and the second action value function as the third action value function, the third action value function represented by line 53 shown in Fig. 13C is identified.

[0202] When the action selection unit 41H selects the first action based on the third action value function represented by the diagram 53 shown in Figure 13C, the first selection probability, which is the selection probability that the action a1 that maximizes the action value represented by the first action value function represented by the diagram 51 of the learning progress stage will be selected as the first action, becomes high.

[0203] That is, in this embodiment, the second selection probability that an action based on the second inference model will be selected as the first action is high in the early learning stage, and the first selection probability that an action based on the first inference model will be selected as the first action is high in the advanced learning stage. The third action-value function changes according to the progress of learning of the first inference model.

[0204] As described above, in this embodiment, the third action value function specifying unit 41G of the action decision unit 41D specifies a third action value function that combines the first action value function and the second action value function, and the action selection unit 41H selects a first action based on the specified third action value function. Therefore, in this embodiment, the action decision unit 41D can automatically adjust the first selection probability without the user having to manually adjust the slope of the increase in the first selection probability relative to the study time. That is, the action decision unit 41D can automatically adjust the selection probabilities so that the shorter the study time, the lower the first selection probability and the higher the second selection probability, and the longer the study time, the higher the first selection probability and the lower the second selection probability.

[0205] Returning to FIG. 11, the explanation will be continued.

[0206] The display control unit 41E displays various images on the display unit 46, similar to the display control unit 40E of the above embodiment.

[0207] Similar to the display control unit 40E of the above embodiment, the display control unit 41E displays on the display unit 46 information representing at least one of the progress of learning of the first inference model, the probability of selection by the action decision unit 40D of at least one of the first action value function and the second action value function, the number of times that the action decision unit 40D selects at least one of the first action value function and the second action value function, and whether the first action is the action that maximizes the action value represented by either the first action value function or the second action value function.

[0208] In this embodiment, the display control unit 41E further displays on the display unit 46 information representing at least one of the selection probability that the action that maximizes the action value represented by the second action value function is selected as the first action, and the transition of the selection probability.

[0209] The display control unit 41E may display these pieces of information on one display screen or on multiple different display screens. Furthermore, the display control unit 41E may display all of these pieces of information on the display screen at the same time, or may display at least one piece of information on the display screen at a different time from the other pieces of information.

[0210] 14 is a schematic diagram showing an example of a display screen 66. The display screen 66 is an example of a display screen .

[0211] The display screen 66 includes an image 66E representing the selection probability that the action that maximizes the action value represented by the second action-value function is selected as the first action. That is, the display screen 66 displays the image 66E representing the current second selection probability.

[0212] Therefore, the display control unit 41E can provide the second selection probability that the rule-based action is selected as the first action to the user in an easily recognizable manner.

[0213] Furthermore, for example, the display screen 66 includes information indicating whether the first action is determined by the first action-value function or the second action-value function. For example, assume that the candidate actions of the mobile robot 20 are "go straight," "go right," "go left," and "stop," which are directions of travel. Then, assume that the third action-value function specifying unit 41G specifies the first action-value function as the third action-value function, and the first action selected by the action selecting unit 41H from the specified third action-value function indicates the direction of travel "go straight."

[0214] In this case, for example, the display control unit 41E displays information 66C representing a learning-based action, which is an action determined from the first action-value function, and information 66D representing a rule-based action, which is an action determined from the second action-value function. The display control unit 41E then displays the selection icon for the information 66C in color and the selection icon for the information 66D in colorless. By displaying the selection icon for the information 66C in color, the display control unit 41E indicates that the first action has been determined from the first action-value function.

[0215] Furthermore, for example, the display control unit 41E displays images representing candidate actions of the mobile robot 20 around the icon image 66B representing the mobile robot 20, similar to the display control unit 40E in the above embodiment. Then, the display control unit 41E displays the image representing the selected "go straight" in a color or brightness different from that of the other candidate actions. For example, the display control unit 41E lights up the image representing "go straight" and does not light up the images representing the other candidate actions. By lighting up the image representing "go straight", the display control unit 41E displays that the traveling direction "go straight" has been determined as the first action from the first action-value function.

[0216] 15 is a schematic diagram showing an example of a display screen 68. The display screen 68 is an example of a display screen .

[0217] The display screen 68 includes information 68A that indicates the transition of the selection probability at which the action that maximizes the action value represented by the second action-value function is selected as the first action, i.e., the transition of the second selection probability. For example, the display control unit 41E displays the display screen 68, including the information 68A that indicates the transition of the second selection probability from the start of learning to the current time, on the display unit 46.

[0218] Therefore, the display control unit 41E can provide the user with an easily recognizable transition of the second selection probability when the rule-based action is selected as the first action.

[0219] Next, an example of the flow of information processing executed by the information processing device 10B of this embodiment will be described.

[0220] FIG. 16 is a flowchart showing an example of the flow of information processing executed by the information processing device 10B of this embodiment.

[0221] The information processing device 10B executes the processes of steps S300 to S308 in the same manner as steps S100 to S108 (see FIG. 8) in the above embodiment.

[0222] Specifically, the acquisition unit 40A acquires the current state of the mobile robot 20 (step S300). Next, the acquisition unit 40A calculates a reward for the current state acquired in step S300 (step S302). The first action-value function identification unit 40B uses the current state acquired in step S300 and the reward calculated in step S302 to train a first inference model by reinforcement learning so as to output a first action-value function that obtains a current state with a higher reward (step S304). The first action-value function identification unit 40B inputs the current state acquired in step S300 into the first inference model reinforced learned in step S304 to identify a first action-value function (step S306). The second action-value function identification unit 40C inputs the current state acquired in step S300 into the second inference model to identify a second action-value function (step S308).

[0223] Next, the third action value function identifying section 41G of the action decision making section 41D identifies the maximum function of the first action value function identified in step S306 and the second action value function identified in step S308 as the third action value function (step S310).

[0224] The action selecting section 41H selects a first action based on the third action-value function identified in step S310 (step S312).

[0225] The transmission control unit 40F transmits the first action selected in step S312 to the mobile robot 20 (step S314). The display control unit 41E displays the display screen 70 on the display unit 46 (step S316). Note that the timing for displaying the display screen 70 on the display unit 46 is not limited to this timing. For example, the display control unit 41E may start displaying the display screen 70 on the display unit 46 before the start of the processing of step S300, and update the display content of the display screen 70 according to the processing status of the control unit 41.

[0226] The control unit 41 determines whether or not to end the process (step S318). The control unit 41 may determine whether or not to end the process in the same manner as step S120 (see FIG. 8) in the above embodiment. If a negative determination is made in step S318 (step S318: No), the process returns to step S300. If a positive determination is made in step S318 (step S318: Yes), the present routine ends.

[0227] As described above, in the information processing device 10B of this embodiment, the third action value function identification unit 41G of the action decision unit 41D identifies the third action value function by combining the first action value function and the second action value function. The action selection unit 41H selects the first action based on the third action value function.

[0228] The third action value function generated by synthesis changes according to the progress of learning of the first inference model. Therefore, in this embodiment, the behavior determination unit 41D can automatically adjust the first selection probability without the user having to manually adjust the slope of the increase in the first selection probability relative to the learning time. That is, the behavior determination unit 41D can automatically adjust the selection probabilities so that the shorter the learning time, the lower the first selection probability and the higher the second selection probability, and the longer the learning time, the higher the first selection probability and the lower the second selection probability.

[0229] Therefore, in addition to the effects of the above-described embodiments, the information processing device 10B of this embodiment can further easily improve the performance in the early stage of reinforcement learning.

[0230] (Third embodiment) In this embodiment, a form will be described in which the third action value function is specified using a fourth action value function obtained by correcting the first action value function based on the second action value function.

[0231] In this embodiment, the same functions and configurations as those in the above embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0232] 17 is a functional block diagram of an example of an information processing system 1C according to the present embodiment. The information processing system 1C is an example of the information processing system 1.

[0233] The information processing system 1C includes an information processing device 10C, a mobile robot 20, and an operation management system 30. The information processing device 10C, the mobile robot 20, and the operation management system 30 are communicatively connected via a network N or the like. The mobile robot 20 and the operation management system 30 are the same as those in the above embodiment.

[0234] The information processing device 10C is an example of the information processing device 10. The information processing device 10C includes a control unit 43, a storage unit 42, an input unit 44, a display unit 46, and a communication unit 48. The control unit 43, the storage unit 42, the input unit 44, the display unit 46, and the communication unit 48 are communicatively connected via a bus 49 or the like. The information processing device 10C is similar to the information processing device 10B of the above embodiment, except that the information processing device 10C includes a control unit 43 instead of the control unit 41.

[0235] The control unit 43 executes various information processes in the information processing device 10C.

[0236] The control unit 43 includes an acquisition unit 40A, a first action value function specification unit 40B, a second action value function specification unit 40C, a behavior decision unit 43D, a display control unit 43E, and a transmission control unit 40F. The acquisition unit 40A, the first action value function specification unit 40B, the second action value function specification unit 40C, and the transmission control unit 40F are the same as those in the above embodiment. That is, the control unit 43 is the same as the control unit 41 in the above embodiment, except that it includes a behavior decision unit 43D and a display control unit 43E instead of the behavior decision unit 41D and the display control unit 41E.

[0237] The behavior decision unit 43D decides the first behavior of the mobile robot 20 based on the first action-value function and the second action-value function, similar to the behavior decision unit 40D of the above embodiment.

[0238] In this embodiment, the behavior decision unit 43D includes an action value function correcting unit 43I, a third action value function specifying unit 43G, and an action selecting unit 41H. The action selecting unit 41H is similar to the behavior selecting unit 41H in the second embodiment. That is, the behavior decision unit 43D is similar to the behavior decision unit 41D in the above embodiment except that it includes a third action value function specifying unit 43G instead of the third action value function specifying unit 41G, and further includes an action value function correcting unit 43I.

[0239] In the second embodiment, the third action value function specification unit 41G specifies the third action value function by combining the first action value function and the second action value function. The action selection unit 41H automatically adjusts the first selection probability and the second selection probability by determining the first action based on the third action value function. The first selection probability is the selection probability that the action that maximizes the action value represented by the first action value function is selected as the first action. The second selection probability is the selection probability that the action that maximizes the action value represented by the second action value function is selected as the second action.

[0240] When the third action value function identification unit 41G is configured to identify a third action value function that is a combination of the first action value function and the second action value function, it may be necessary for the user to manually adjust the reward setting calculated from the current state by the acquisition unit 40A.

[0241] This problem will be explained in detail below. The value of the action value expressed by the action value function depends on the reward setting.

[0242] For example, assume that, in the early stage of learning, the maximum value of the action value represented by the second action value function is smaller than the minimum value of the action value represented by the first action value function. In this case, the third action value function coincides with the first action value function in the early stage of learning. Therefore, in this case, if the action selection unit 41H selects a first action based on the third action value function, the probability that the action that maximizes the action value represented by the second action value function will be selected as the first action in the early stage of learning is extremely low. In other words, in this case, it is difficult for the information processing device 10 to determine a first action that can learn the first inference model while satisfying the minimum required performance for the mobile robot 20's behavior.

[0243] Also, for example, assume that in the early stage of learning, the maximum value of the action value represented by the first action value function is smaller than the minimum value of the action value represented by the second action value function. In this case, the third action value function coincides with the second action value function in the early stage of learning. Therefore, in this case, if the action selection unit 41H selects the first action based on the third action value function, the probability that the action that maximizes the action value represented by the first action value function will be selected as the first action in the early stage of learning is extremely low. Therefore, in this case, it becomes difficult for the information processing device 10 to proceed with reinforcement learning of the first inference model.

[0244] In order to avoid these situations, in the information processing device 10, the user may need to manually adjust the reward setting so that the action value represented by the first action value function in the early stage of learning is a value between the maximum and minimum values ​​of the action value represented by the second action value function.

[0245] However, in order for the user to manually adjust the reward settings, the actions of the mobile robot 20 need to be performed by trial and error.

[0246] Therefore, in the information processing device 10C of this embodiment, the behavior decision unit 43D has an action value function correction unit 43I, a third action value function identification unit 43G, and a behavior selection unit 41H.

[0247] The action value function correcting unit 43I specifies a fourth action value function by correcting the first action value function based on the second action value function. For example, the third action value function specifying unit 43G specifies the fourth action value function by the following equation (5).

[0248] Q4(s,a)=Q1(s,a)+D(s) Equation (5)

[0249] In equation (5), Q4(s, a) represents the fourth action-value function. Q1(s, a) represents the first action-value function. s represents the current state. a represents the action. D(s) is the correction value of the first action-value function, and is a value calculated based on the second action-value function.

[0250] For example, the action value function correction unit 43I identifies the fourth action value function by correcting the first action value function so that the action value for the action represented by the first action value function becomes a value between the maximum and minimum values ​​of the action value for the action represented by the second action value function.

[0251] Specifically, the action-value function correcting unit 43I calculates D(s), which is the corrected value in the above formula (5), using the following formula (6).

[0252] D(s)=(maxQ2(s,a)+minQ2(s,a)) / 2 Equation (6)

[0253] In equation (6), maxQ2(s, a) represents the maximum function of the second action-value function. minQ2(s, a) represents the minimum function of the second action-value function. s represents the current state. a represents the action.

[0254] Then, the action value function correction unit 43I corrects the first action value function by substituting the correction value D(s) expressed by equation (6) into the above equation (5) so that the action value for the action expressed by the first action value function becomes a value between the maximum and minimum values ​​of the action value for the action expressed by the second action value function.

[0255] Furthermore, for example, the action-value-function correcting unit 43I may specify a fourth action-value function obtained by correcting the first action-value function so that the second selection probability at the start of learning of the first inference model becomes a predetermined selection probability. As described above, the second selection probability represents the selection probability that the action selecting unit 41H selects the second action-value function as the third action-value function, and represents the selection probability that the action corresponding to the maximum action value represented by the second action-value function is selected as the first action.

[0256] In detail, the action-value function correcting unit 43I calculates D(s), which is the corrected value in the above formula (5), by the following formula (7).

[0257] D(s)=log(1-ε) / ((N-1)×ε)+maxQ2(s,a) Equation (7)

[0258] In equation (7), ε represents the predetermined selection probability mentioned above. maxQ2(s, a) represents the maximum function of the second action value function. N represents the number of candidate actions that the mobile robot 20 can take next. For example, N may be four of the actions that the mobile robot 20 can take next: "right," "left," "go straight," and "stop." s represents the current state. a represents an action.

[0259] Then, the action value function correction unit 43I identifies a fourth action value function that is a correction of the first action value function by substituting the correction value D(s) expressed by equation (7) into the above equation (5) so that the second selection probability at the start of learning of the first inference model becomes a predetermined selection probability ε.

[0260] The action-value-function correcting unit 43I may specify a fourth action-value function by correcting the first action-value function so that it becomes the selection probability ε input by the user. In this case, for example, the display control unit 43E displays a display screen 70 on the display unit 46 for receiving input of a selection probability determined as the second selection probability.

[0261] FIG. 18 is a schematic diagram of an example of a display screen 69. The display screen 69 is an example of a display screen 70. The display screen 69 includes an input field 69A for the value of a selection probability determined as the second selection probability. The user operates the input unit 44 while viewing the display screen 69 to input a desired value of the selection probability into the input field 69A. The action-value-function correcting unit 43I uses the selection probability input by the user via the display screen 69 as the selection probability ε used to correct the first action-value function, and corrects the first action-value function so that the second selection probability at the start of learning of the first inference model becomes the selection probability ε.

[0262] Returning to FIG. 17, the explanation will be continued.

[0263] Also, for example, as described above, assume that the fourth action value function is identified by the action value function correction unit 43I correcting the first action value function so that the action value for the action represented by the first action value function becomes a value between the maximum and minimum values ​​of the action value for the action represented by the second action value function.

[0264] In this assumption, the action value function correction unit 43I may identify a fourth action value function obtained by correcting the first action value function so that the action value for the action represented by the first action value function becomes a value input by the user between the maximum and minimum values ​​of the action value for the action represented by the second action value function.

[0265] In this case, for example, the display control unit 43E displays on the display unit 46 an input screen for an adjustment value indicating to which value between the maximum and minimum values ​​of the action value of the second action value function the action value represented by the first action value function should be corrected. The user operates the input unit 44 while viewing the input screen to input the adjustment value. The action value function correction unit 43I may use the adjustment value input by the user to identify a fourth action value function obtained by correcting the action value represented by the first action value function to a value between the maximum and minimum values ​​of the action value of the second action value function.

[0266] Next, the third action-value function specifying unit 43G will be described.

[0267] The third action value function specifying unit 43G specifies the maximum function of the fourth action value function specified by the action value function correcting unit 43I and the second action value function specified by the second action value function specifying unit 40C as the third action value function. The third action value function specifying unit 43G may specify the third action value function in the same manner as the third action value function specifying unit 41G, except that the fourth action value function is used instead of the first action value function.

[0268] Then, the action selecting section 41H selects the first action based on the third action value function identified by the third action value function identifying section 413, as in the above embodiment.

[0269] An example of the processing by the behavior determining unit 43D of this embodiment will be described below using a specific example.

[0270] 19A to 19D are explanatory diagrams of an example of processing by the behavior determining unit 43 D. In Fig. 19A to 19D, the horizontal axis represents behavior, and the vertical axis represents behavior value.

[0271] Figure 19A is a schematic diagram showing an example of a first action value function in the early stage of learning. Figure 19B is a schematic diagram showing an example of a second action value function in the early stage of learning. For example, assume a situation in which the first action value function is the function represented by line 51 in Figure 19A and the second action value function is the function represented by line 52 in Figure 19B.

[0272] As shown in Figures 19A and 19B, assume that the maximum value of the action value of the second action value function represented by line 52 in the early learning stage is smaller than the minimum value of the action value of the first action value function represented by line 51. In this assumption, the third action value function coincides with the first action value function in the early learning stage. Therefore, in this case, if the action selection unit 41H selects a first action based on the third action value function, the probability that the action that maximizes the action value represented by the second action value function will be selected as the first action in the early learning stage is extremely low. In other words, in this case, it is difficult for the information processing device 10 in the early learning stage to determine a first action that can learn the first inference model while satisfying the minimum required performance for the mobile robot 20's behavior.

[0273] Therefore, the action-value-function correcting unit 43I specifies a fourth action-value function by correcting the first action-value function based on the second action-value function. For example, the action-value-function correcting unit 43I corrects the action value for each action of the first action-value function represented by the line 51 so that it becomes a value between the maximum and minimum of the action value for each action of the second action-value function represented by the line 52.

[0274] FIG. 19C is an explanatory diagram of an example of the relationship between the first action-value function before correction and the fourth action-value function, which is the first action-value function after correction.

[0275] Let us consider a situation in which the action-value-function correcting unit 43I corrects the first action-value function shown in Fig. 19A using the second action-value function shown in Fig. 19B. In this case, as shown in Fig. 19C, the first action-value function represented by line 52 is corrected to a fourth action-value function represented by line 54 (see arrow B). That is, the fourth action-value function represented by line 54 is obtained by correcting the action value of the first action-value function represented by line 51 so that it becomes a value between the maximum and minimum values ​​of the action value for each action of the second action-value function represented by line 52.

[0276] FIG. 19D is an explanatory diagram of an example of identification of the third action value function by the third action value function identification unit 43G.

[0277] For example, consider a situation in which the first action value function is the function represented by line 51 in Fig. 19A, and the second action value function is the function represented by line 52 in Fig. 19B. Also consider a situation in which the action value function correction unit 43I identifies a fourth action value function represented by line 54 shown in Fig. 19C.

[0278] In this case, the third action value function identification unit 43G identifies the third action value function represented by diagram 53 by identifying the maximum function of the fourth action value function represented by diagram 54 and the second action value function represented by diagram 52 as the third action value function.

[0279] When the action selection unit 41H selects the first action based on the third action value function represented by the diagram 53 shown in Figure 19D, the selection probability that the action a2 that maximizes the action value represented by the second action value function represented by the diagram 52 will be selected as the first action is increased.

[0280] Therefore, in this embodiment, the reward setting can be automatically adjusted so that the action value represented by the first action value function in the early learning stage is a value between the maximum and minimum values ​​of the action value represented by the second action value function, without the user having to manually adjust the reward setting.

[0281] That is, in the information processing device 10C of this embodiment, the action value function correction unit 43I corrects the first action value function, so that, regardless of the reward setting, a first action that can learn the first inference model while satisfying the minimum required performance regarding the behavior of the mobile robot 20 can be determined for the initial stage of learning of the first inference model.

[0282] Returning to FIG. 17, the explanation will be continued.

[0283] The display control unit 43E is similar to the display control unit 41E of the above embodiment, except that it can also display an input screen for adjustment values ​​and a display screen 69 on the display unit .

[0284] As described in the above embodiment, the first action-value function specifying unit 40B uses the reward of the current state and the first action to learn the first inference model by reinforcement learning so as to maximize the reward. That is, the first action-value function specifying unit 40B learns the first inference model by reinforcement learning using the current state of the mobile robot 20 acquired by the acquiring unit 40A, the reward, and the first action of the mobile robot 20 used to calculate the reward.

[0285] In this embodiment, it is preferable that the first action-value function specifying unit 40B learns the first inference model by reinforcement learning using the first action when the specified fourth action-value function of the third action-value function is used.

[0286] When the first action-value function specifying unit 40B uses DQN as the algorithm used to train the first inference model, the loss function L expressed by the following equation (8) can be used instead of the loss function L expressed by equation (1) described in the above embodiment. Then, the first action-value function specifying unit 40B can determine the update amount for each of the multiple parameters included in the first inference model by performing backpropagation using the loss function L, as in the above embodiment.

[0287] L=(r t +γmax{Q4(s t+1 ,at+1 )}-Q4(s t ,a t )) 2 Formula (8)

[0288] In formula (8), L, r t ,γ,s t , and a t is the same as the above equation (1). Q4(s t+1 ,a t+1 ) represents the fourth action value function at the next time t+1. Q4(s t ,a t ) represents the fourth action-value function at time t. t represents the current state s at time t. t represents the behavior at time t.

[0289] Next, an example of the flow of information processing executed by the information processing device 10C of this embodiment will be described.

[0290] FIG. 20 is a flowchart showing an example of the flow of information processing executed by the information processing device 10C of this embodiment.

[0291] The information processing device 10C executes the processes of steps S400 to S408 in the same manner as steps S300 to S308 (see FIG. 16) in the above embodiment.

[0292] Specifically, the acquisition unit 40A acquires the current state of the mobile robot 20 (step S400). Next, the acquisition unit 40A calculates a reward for the current state acquired in step S400 (step S402). The first action-value function identification unit 40B uses the current state acquired in step S400 and the reward calculated in step S402 to train the first inference model by reinforcement learning so as to output a first action-value function that obtains a current state with a higher reward (step S404). The first action-value function identification unit 40B inputs the current state acquired in step S400 into the first inference model reinforced learned in step S304 to identify the first action-value function (step S406). The second action-value function identification unit 40C inputs the current state acquired in step S400 into the second inference model to identify the second action-value function (step S408).

[0293] Next, the action value function correcting unit 43I of the action decision-making unit 43D identifies a fourth action value function by correcting the first action value function identified in step S406 using the second action value function identified in step S408 (step S410).

[0294] Next, the third action value function identifying section 43G of the action decision section 43D identifies the maximum function of the fourth action value function identified in step S410 and the second action value function identified in step S408 as the third action value function (step S412).

[0295] The action selecting section 41H selects a first action based on the third action-value function identified in step S412 (step S414).

[0296] The transmission control unit 40F transmits the first action selected in step S414 to the mobile robot 20 (step S416). The display control unit 43E displays the display screen 70 on the display unit 46 (step S418).

[0297] The control unit 43 determines whether or not to end the process (step S420). The control unit 43 may determine whether or not to end the process in the same manner as step S120 (see FIG. 8) in the above embodiment. If a negative determination is made in step S420 (step S420: No), the process returns to step S400. If a positive determination is made in step S420 (step S420: Yes), the present routine ends.

[0298] As described above, the behavior decision unit 43D of the information processing device 10C of this embodiment has an action value function correction unit 43I. The action value function correction unit 43I specifies a fourth action value function obtained by correcting the first action value function based on the second action value function. The third action value function identification unit 43G identifies the maximum function of the fourth action value function and the second action value function as the third action value function.

[0299] Therefore, in addition to the effects of the above embodiments, the information processing device 10C of this embodiment can determine a first behavior that can learn the first inference model while satisfying the minimum required performance for the behavior of the mobile robot 20, regardless of the reward setting, in the early stage of learning of the first inference model.

[0300] (Variation) In the above first to third embodiments, an example has been described in which the information processing device 10 includes one second action value function specifying unit 40C. That is, in the above embodiments, an example has been described in which each of the action decision unit 40D, the action decision unit 41D, and the action decision unit 43D specifies a third action value function using one second action value function output from one second inference model.

[0301] However, the information processing device 10 may be configured to include a plurality of second action value function specifying units 40 C. In other words, the information processing device 10 may specify a third action value function using a plurality of second action value functions output from two different inference models.

[0302] In this case, the behavior decision unit 40D, the behavior decision unit 41D, and the behavior decision unit 43D may use any one of the second action value functions output from each of the multiple second inference models to identify the third action value function in the same manner as in the above embodiment.

[0303] Next, an example of the hardware configuration of the information processing device 10 (information processing device 10A, information processing device 10B, information processing device 10C), the mobile robot 20, and the operation control system 30 of the above embodiment will be described.

[0304] FIG. 21 is a hardware configuration diagram of an example of the information processing device 10, the mobile robot 20, and the operation management system 30 of the above embodiment.

[0305] The information processing device 10, mobile robot 20, and operation control system 30 of the above embodiment have a hardware configuration using a normal computer, in which a CPU (Central Processing Unit) 86, a ROM (Read Only Memory) 88, a RAM (Random Access Memory) 90, an I / F 82, etc. are interconnected by a bus 96.

[0306] The CPU 86 is a computing device that controls the information processing device 10, the mobile robot 20, and the operation control system 30 of the above-described embodiment. The ROM 88 stores programs and the like that realize information processing by the CPU 86. The RAM 90 stores data necessary for various processes by the CPU 86. The I / F 82 is an interface that connects to the memory unit 42, the input unit 44, the display unit 46, the communication unit 48, the communication unit 24, the sensor 26, the memory unit 28, and the like, and transmits and receives data.

[0307] In the information processing device 10, the mobile robot 20, and the operation control system 30 of the above embodiment, the CPU 86 reads out a program from the ROM 88 onto the RAM 90 and executes it, thereby realizing each of the above functions on the computer.

[0308] The programs for executing the above processes executed by the information processing device 10, the mobile robot 20, and the operation control system 30 of the above embodiments may be stored in an HDD (hard disk drive). Also, the programs for executing the above processes executed by the information processing device 10, the mobile robot 20, and the operation control system 30 of the above embodiments may be provided by being pre-installed in the ROM 88.

[0309] The programs for executing the above processes executed by the information processing device 10, the mobile robot 20, and the traffic management system 30 of the above embodiments may be stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD) and provided as a computer program product. The programs for executing the above information processes executed by the information processing device 10, the mobile robot 20, and the traffic management system 30 of the above embodiments may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. The programs for executing the above information processes executed by the information processing device 10, the mobile robot 20, and the traffic management system 30 of the above embodiments may be provided or distributed via a network such as the Internet.

[0310] Although the above describes an embodiment, the embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included within the scope and spirit of the invention, and are also included in the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0311] 10, 10A, 10B, 10C Information processing device 20 Mobile Robot 40A Acquisition Department 40B First action value function identification section 40C Second action value function specification section 40D, 41D, 43D Action determining part 40E, 41E, 43E Display control unit 40F Transmission control section 41G, 43G Third action value function specification part 41H Action Selection Section 43I Action Value Function Correction Unit

Claims

1. an acquisition unit that acquires a current state of the device; a first action-value function specifying unit that has a function of learning a first inference model by reinforcement learning and that specifies a first action-value function of the device based on the current state and the first inference model; a second action-value function specifying unit that specifies a second action-value function of the device based on the current state and a second inference model that is not a parameter update target; an action determination unit that determines a first action of the device based on the first action-value function and the second action-value function; Equipped with The behavior determination unit an action-value-function correcting unit that specifies a fourth action-value function obtained by correcting the first action-value function based on the second action-value function; a third action value function specifying unit that specifies a maximum function of the fourth action value function and the second action value function as a third action value function; an action selection unit that selects the first action based on the third action-value function; having Information processing device.

2. The behavior determination unit selecting one of the first action-value function and the second action-value function as a third action-value function, and determining the first action based on the selected third action-value function; The information processing device according to claim 1 .

3. The behavior determination unit changing a first selection probability of selecting the first action value function as the third action value function and a second selection probability of selecting the second action value function as the third action value function according to a learning time of the first inference model; the shorter the learning time, the lower the first selection probability and the higher the second selection probability; The longer the learning time, the higher the first selection probability and the lower the second selection probability. The information processing device according to claim 2 .

4. The action value function correction unit specifying the fourth action-value function obtained by correcting the first action-value function so that the action value for the action represented by the first action-value function is a value between the maximum value and the minimum value of the action value for the action represented by the second action-value function; The information processing device according to claim 1 .

5. The action value function correction unit specifying the fourth action value function obtained by correcting the first action value function so that a second selection probability of selecting the second action value function as the third action value function at the start of learning of the first inference model becomes a predetermined selection probability; The information processing device according to claim 1 .

6. The action value function correction unit identifying the fourth action-value function obtained by correcting the first action-value function so as to achieve the selection probability input by the user; The information processing device according to claim 5 .

7. The action value function correction unit specifying the fourth action-value function obtained by correcting the first action-value function so that the action value for the action represented by the first action-value function is a value input by the user between the maximum and minimum values ​​of the action value for the action represented by the second action-value function; The information processing device according to claim 4 .

8. The first action-value function specifying unit learning the first inference model by reinforcement learning using the current state, the reward for the current state, and the first action; learning the first inference model by reinforcement learning using the first action when the fourth action-value function is used to identify the third action-value function; The information processing device according to claim 1 .

9. The second inference model is The information processing device according to claim 1 , which is pre-learned using data on the current state and the behavior of the device based on a first rule.

10. A plurality of the second action-value function specifying units are provided, The information processing device according to claim 1 .

11. a display control unit that displays on a display unit information indicating at least one of the progress of learning of the first inference model, the probability of selection by the action decision unit of at least one of the first action value function and the second action value function, the number of times that the action decision unit has selected at least one of the first action value function and the second action value function, whether the first action is the action that maximizes the action value represented by the first action value function or the second action value function, the selection probability that the action that maximizes the action value represented by the second action value function is selected as the first action, and the transition of the selection probability. The information processing device according to claim 1 .

12. an acquisition step of acquiring a current state of the device; a first action-value function specifying step having a learning function of learning a first inference model by reinforcement learning, and specifying a first action-value function of the device based on the current state and the first inference model; a second action-value function specifying step of specifying a second action-value function of the device based on the current state and a second inference model that is not a parameter update target; an action determination step of determining a first action of the device based on the first action-value function and the second action-value function; Including, The action determination step includes: an action value function correcting step of correcting the first action value function based on the second action value function to identify a fourth action value function; a third action value function specifying step of specifying a maximum function of the fourth action value function and the second action value function as a third action value function; an action selection step of selecting the first action based on the third action-value function; Including, Information processing methods.

13. An information processing program to be executed by a computer, an acquisition step of acquiring a current state of the device; a first action-value function specifying step having a learning function of learning a first inference model by reinforcement learning, and specifying a first action-value function of the device based on the current state and the first inference model; a second action-value function specifying step of specifying a second action-value function of the device based on the current state and a second inference model that is not a parameter update target; an action determination step of determining a first action of the device based on the first action-value function and the second action-value function; Including, The action determination step includes: an action value function correcting step of correcting the first action value function based on the second action value function to identify a fourth action value function; a third action value function specifying step of specifying a maximum function of the fourth action value function and the second action value function as a third action value function; an action selection step of selecting the first action based on the third action-value function; Including, Information processing program.

Images