Robot control model learning method, robot control model learning device, robot control method, robot control device, recording medium, and robot

Abstract

A robot control model learning device (10) takes state information representing the state of a robot driving autonomously toward a destination in a dynamic environment as input, selects and outputs an action corresponding to the state of the robot from a plurality of actions including intervention actions in the intervention environment, and performs reinforcement learning on the robot control model using the number of intervention actions performed as a negative reward.

Description

Technical Field

[0001] The technology disclosed herein relates to a robot control model learning method, a robot control model learning device, a robot control model learning program, a robot control method, a robot control device, a robot control program, and a robot. Background Art

[0002] In path planning methods such as RRT (Rapidly-exploring Random Tree) and PRM (Probabilistic Road Map), a path from an initial position to a destination position is derived by performing a graph search using sampled points in a space as nodes.

[0003] These methods are based on static and known environments. In dynamic environments, "replanning" is required every time the environment changes.

[0004] Existing "replanning" methods basically update the map according to environmental changes and search for another global path that can be continuously deformed. However, they cannot find a solution in dense and continuously changing environments such as crowd environments, and may frequently cause the robot to stop during replanning.

[0005] In addition, in crowded environments such as crowds, simply continuously removing obstacles in front of you puts a lot of pressure on the environment.

[0006] Non-Patent Document 1 (Decentralized Non-communicating Multiagent CollisionAvoidance with Deep Reinforcement Learning https: / / arxiv.org / pdf / 1609.07845) discloses a technique for obtaining a collision avoidance strategy using deep reinforcement learning. The technique disclosed in Non-Patent Document 1 obtains a strategy that minimizes the time it takes to reach a destination while avoiding collisions with surrounding agents.

[0007] Non-Patent Document 2 (Socially Aware Motion Planning with Deep Reinforcement Learning https: / / arxiv.org / pdf / 1703.08862.pdf) discloses a technique that improves on the technique disclosed in Non-Patent Document 1. The technique disclosed in Non-Patent Document 2 implements socially natural avoidance behavior by adding a social norm (social norm) to a reward function that takes into account the characteristics of human collision avoidance behavior.

[0008] Non-patent document 3 (ZMP https: / / news.mynavi.jp / article / 20180323-604926 / ) discloses a technology that allows autonomous driving without changing the robot's own path plan by intervening to urge obstacles (people) on the path plan to give way. Summary of the Invention

[0009] Problems to be solved by the invention

[0010] However, the technologies disclosed in the above-mentioned non-patent documents 1 and 2 only deal with passive collision avoidance actions against the environment and do not involve intervention actions.

[0011] Furthermore, the techniques disclosed in Non-Patent Documents 1 and 2 assume interaction with a small number of agents and do not assume situations in a crowd environment.

[0012] Furthermore, although intervention based on a simple strategy, as disclosed in Non-Patent Document 3, is easy to implement, if the frequency of intervention is high, it may cause stress on the environmental side and may also deteriorate the transportation efficiency of the surrounding pedestrian groups.

[0013] The technology disclosed herein is completed in view of the above-mentioned points, and its purpose is to provide a robot control model learning method, a robot control model learning device, a robot control model learning program, a robot control method, a robot control device, a robot control program and a robot that can reduce the number of intervention actions of the robot in the surrounding environment when moving the robot to a destination in a dynamic environment.

[0014] Means for solving problems

[0015] The first method disclosed in the present invention is a robot control model learning method, which comprises: a learning step, for a robot control model that takes state information representing the state of a robot that autonomously travels to a destination in a dynamic environment as input, selects and outputs an action corresponding to the state of the robot from multiple actions including intervention actions that intervene in the environment, and reinforces the robot control model with the number of interventions in which the intervention action is performed as a negative reward.

[0016] In the above-mentioned first method, the action may also include the moving direction of the robot, the moving speed of the robot and at least one of the intervention actions, so that the reward is given in a manner that reduces at least one of the arrival time for the robot to reach the destination and the number of interventions.

[0017] In the first aspect, the action may include an avoidance action for avoiding a collision between the robot and another object, and the reward may be given so that the number of avoidance operations for avoiding the collision is reduced.

[0018] In the first aspect described above, the learning step may perform reinforcement learning by updating a state-value function representing the state of the robot.

[0019] The second method disclosed in the present invention is a robot control model learning device, which includes: a learning unit, which takes state information representing the state of a robot that autonomously travels to a destination in a dynamic environment as input, selects and outputs a robot control model of an action corresponding to the state of the robot from multiple actions including intervention actions that intervene in the environment, and performs reinforcement learning on the robot control model using the number of interventions in which the intervention action is performed as a negative reward.

[0020] The third method disclosed in the present invention is a robot control model learning program, which enables a computer to execute processing including the following steps: a learning step, for a robot control model that takes state information representing the state of a robot that autonomously travels to a destination in a dynamic environment as input and selects and outputs an action corresponding to the state of the robot from multiple actions including intervention actions that intervene in the environment, and reinforcement learning is performed on the robot control model using the number of interventions in which the intervention action is performed as a negative reward.

[0021] The fourth method disclosed in the present invention is a robot control method, which enables a computer to perform processing including the following steps: an acquisition step, obtaining status information representing the status of a robot that autonomously travels to a destination in a dynamic environment; and a control step, controlling the robot to move to the destination based on the status information and a robot control model learned by a robot control model learning method.

[0022] The fifth embodiment of the present disclosure is a robot control device, which includes: an acquisition unit, which acquires status information representing the status of a robot that autonomously travels to a destination in a dynamic environment; and a control unit, which controls the robot to move to the destination based on the status information and a robot control model learned by a robot control model learning device.

[0023] The sixth method disclosed in the present invention is a robot control program, which enables a computer to perform processing including the following steps: an acquisition step, obtaining status information representing the status of the robot as it autonomously travels to a destination in a dynamic environment; and a control step, controlling the robot to move to the destination based on the status information and the robot control model learned by the robot control model learning method.

[0024] The seventh embodiment of the present disclosure is a robot, comprising: an acquisition unit that acquires status information indicating the status of the robot as it autonomously travels to a destination in a dynamic environment; an autonomous driving unit that enables the robot to travel autonomously; and a control unit that controls the robot to move to the destination based on the status information and a robot control model learned by a robot control model learning device.

[0025] Effects of the Invention

[0026] According to the technology disclosed herein, when a robot is moved toward a destination in a dynamic environment, the number of intervention actions in which the robot intervenes in the surrounding environment can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a diagram showing the schematic structure of a robot control model learning system.

[0028] Figure 2 This is a block diagram showing the hardware configuration of a robot control model learning device.

[0029] Figure 3 This is a block diagram showing the functional structure of a robot control model learning device.

[0030] Figure 4 This diagram shows a robot moving toward a destination among a crowd.

[0031] Figure 5 This is a flowchart showing the flow of a robot control model learning process performed by a robot control model learning device.

[0032] Figure 6 This is a block diagram showing the functional structure of a robot control device.

[0033] Figure 7 This is a block diagram showing the hardware configuration of a robot control device.

[0034] Figure 8 This is a flowchart showing the flow of robot control processing by the robot control device. DETAILED DESCRIPTION

[0035] An example of an embodiment of the technology disclosed herein is described below with reference to the accompanying drawings. In the drawings, identical or equivalent components and parts are denoted by the same reference numerals. Furthermore, the dimensional ratios in the drawings may be exaggerated for ease of description and may differ from the actual ratios.

[0036] Figure 1 1 is a diagram showing a schematic configuration of the robot control model learning system 1 .

[0037] like Figure 1 As shown, the robot control model learning system 1 includes a robot control model learning device 10 and a simulator 20. The simulator 20 will be described later.

[0038] Next, the robot control model learning device 10 will be described.

[0039] Figure 2 2 is a block diagram showing the hardware configuration of the robot control model learning device 10 .

[0040] like Figure 2 As shown, the robot control model learning device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage device 14, an input unit 15, a monitor 16, an optical disk drive 17, and a communication interface 18. These components are connected via a bus 19 so as to be able to communicate with each other.

[0041] In this embodiment, a robot control model learning program is stored in the storage device 14. The CPU 11 is a central processing unit (CPU) that executes various programs and controls various components. Specifically, the CPU 11 reads programs from the storage device 14 and executes them using the RAM 13 as a work area. The CPU 11 controls the various components and performs various calculations according to the programs stored in the storage device 14.

[0042] The ROM 12 stores various programs and data. The RAM 13 temporarily stores programs and data as a work area. The storage device 14 is composed of an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0043] The input unit 15 includes pointing devices such as a keyboard 151 and a mouse 152 for various inputs. The monitor 16 is, for example, a liquid crystal display, and displays various information. The monitor 16 can also be a touch panel and function as the input unit 15. The optical disk drive 17 reads data stored on various recording media (such as CD-ROMs and Blu-ray discs) and writes data to the recording media.

[0044] The communication interface 18 is an interface for communicating with other devices such as the simulator 20 , and uses a standard such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark).

[0045] Next, the functional configuration of the robot control model learning device 10 will be described.

[0046] Figure 3 3 is a block diagram showing an example of the functional configuration of the robot control model learning device 10 .

[0047] like Figure 3 As shown, the robot control model learning device 10 includes a state value calculation unit 30 and an action selection unit 32 as functional components. Each functional component is implemented by the CPU 11 reading the robot control program stored in the storage device 14, expanding it in the RAM 13, and executing it. The state value calculation unit 30 and the action selection unit 32 are examples of a learning unit.

[0048] In this embodiment, a case where a state-value function described later is learned by value-based deep reinforcement learning is described.

[0049] The state value calculation unit 30 obtains state information from the simulator 20. The simulator 20 has a state value calculation unit 30. Figure 4 As shown in the figure, the autonomous driving robot RB moves towards the destination p g The simulator 20 outputs state information related to the state of the robot RB and the environment around the robot RB to the state value calculation unit 30.

[0050] Here, the state information includes robot information regarding the state of the robot RB, environment information regarding the surrounding environment of the robot RB, and destination information regarding a destination that the robot RB should reach.

[0051] The robot information includes information on the position and velocity of the robot RB. In this embodiment, the velocity v of the robot RB is expressed as a vector in a two-dimensional coordinate system as follows.

[0052] v={v x 、v y}

[0053] In addition, in the present embodiment, the position p of the robot RB is expressed by coordinates in a two-dimensional coordinate system as follows.

[0054] p={p x 、p y}

[0055] In this embodiment, the state s of the robot RB at time t is expressed as follows: t .

[0056] st ={p x 、p y 、v x 、v y 、r b}

[0057] Here, r b represents the influence radius of robot RB. As described later, the influence radius r b Used to determine whether the robot RB collides with an object other than the robot RB.

[0058] Environmental information includes information on the dynamic environment, specifically, information on the position and speed of moving objects such as a person HB existing around the robot RB. In this embodiment, a case where the environmental information is information on the person HB will be described.

[0059] In this embodiment, the following is expressed as follows Figure 4 As shown in the figure, the state of the people HB around the robot RB is ～ s t In addition, in this embodiment, for convenience, in mathematical formulas, etc., the mark " ～ When a symbol such as "(tilde)" or "^(hat mark)" is added to a character above a character, the symbol may be placed before the character.

[0060]

[0061] Here, N is the number of people HB present in the surroundings. ～ s t 1 、 ～ s t 2 、… ～ s t N The state of each person HB at time t, that is, the position and speed are shown.

[0062] In this embodiment, the state s of the robot RB at time t is expressed as follows: t And the state of the human HB around the robot RB ～ s t State of combination

[0063]

[0064] Destination information includes destination p g Location information of the destination. g It is expressed as coordinates in a two-dimensional coordinate system as follows.

[0065] p g ={p g x, p g y}

[0066] The state value calculation unit 30 uses the reward function R(s) based on the acquired state information. jn , a) to calculate the reward r. Here, a represents the action, including the movement direction, movement speed, intervention action and avoidance action of the robot RB. And, the robot RB reaches the destination p g The reward r is given so that the arrival time, the number of interventions which is the number of times the intervention action is performed, and the number of avoidances by the robot RB to avoid collision become smaller.

[0067] Here, an intervention action refers to an action that notifies the surrounding person HB of the robot RB's presence in order to keep the robot RB moving without stopping. Specifically, this includes actions such as outputting a message such as "Make way" or sounding a warning, but intervention actions are not limited to these. Furthermore, an avoidance action refers to an action that the robot RB takes to avoid collision with another object, moving in a direction and at a speed that allows it to avoid the object.

[0068] In this embodiment, the reward function R(s) is set as follows jn , a). In addition, it may be simply referred to as the reward function R below.

[0069] R(s jn , a)=αr e +βr c

[0070] Here, r e is the reward from the environment, r c is the impact reward of intervention. In addition, α is the reward r e The weight of β is the reward r c The weight of is set to an arbitrary value. The reward r is expressed as follows e 、Reward c .

[0071]

[0072]

[0073] Here, d is a distance used to determine whether the robot RB and the human HB collide, and is expressed by the following equation.

[0074] d=D-(r b +r h )

[0075] D represents the distance between the robot RB and the human HB. b is the influence radius of the robot RB mentioned above, r h is the influence radius of person HB. Alternatively, it can be set to r b =r h , can also be set to r b ≠r h When d is less than 0, it indicates the influence radius r of robot RB b The area within and the influence radius r of person HB h In the present embodiment, when d is less than 0, it is considered that the robot RB and the human HB have collided.

[0076] In addition, b t It is an intervention parameter indicating whether the robot RB has intervened in the surrounding environment at time t. t When the intervention parameter is "0", it indicates that no intervention action is performed. On the other hand, when the intervention parameter is a value other than "0", it indicates that the robot RB has performed an intervention action.

[0077] As shown above, the reward r e If the distance d is less than 0, that is, the robot RB and the human HB are considered to have collided, then the value is “ε c ”.

[0078] In addition, the robot RB reaches the destination p at position p g In the case of e Become "ε g Here, the reward ε g The range of possible values ​​is 0≤ε g ≤1. And, the reward ε g To reach the destination g The later the arrival time is, the smaller the value is, that is, the closer to "0". g To reach the destination g The value is given so that the earlier the arrival time is, the larger the value is, that is, the closer to "1".

[0079] In addition, the reward r e In cases other than the above, it is "0".

[0080] In addition, the intervention parameter b t If the value is other than "0", the reward r c For "ε b ”. That is, ε b It can be said that it is a reward related to the intervention action. In addition, in the intervention parameter bt When it is "0", the reward r c is "0".

[0081] Here, ε c , ε b As a negative reward, it is set to a value less than 0, that is, a negative value. c It can be said that it is the reward related to avoiding collision, ε b It can be said that it is a reward related to the intervention action. In addition, ε can also be expressed as a function of the distance d c Alternatively, you can use the intervention parameter b t The function of ε b .

[0082] In addition, the state value calculation unit 30 uses the state value function The following formula is used to calculate the value y of the state at time t t .

[0083]

[0084] Here, r t is the reward at time t calculated by the reward function R. Δt is the time increment in one step. γ is the discount rate of the reward, which is defined as follows.

[0085] γ∈(0,1)

[0086] That is, the discount rate γ can take a value greater than or equal to 0 and less than or equal to 1. The discount rate γ is set to a value that evaluates a reward that is obtained in the distant future at a more discounted value.

[0087] State Value Function It indicates that the robot RB and the surrounding human HB are in state s in the strategy selected by the strategy function π described later. jn The value function of is represented by the following formula. Here, "*" represents the optimal function, V* represents the optimal state value function, and π* represents the optimal policy function. In addition, the state value function V is sometimes referred to as the state value function below.

[0088]

[0089] The above formula (2) represents the state value function The expected reward is the sum of the discounted cumulative rewards obtained by accumulating future rewards discounted by the discount rate γ. In this embodiment, a deep neural network (value-network) is used to approximate the state-value function V. Hereinafter, the deep neural network representing the state-value function V will be referred to as the V-network.

[0090] The state value calculation unit 30 learns the V network. In this embodiment, as an example, the V network is learned by the gradient descent method using the experience reply buffer. That is, the state St is stored in the buffer E. jn , value y t , randomly read out the state St from the buffer E jn , value y t The read pairs are used as teacher data to learn the V network. That is, update the parameters of the V network.

[0091] The action selection unit 32 selects an action a that the robot RB should take based on the value of the state calculated by the state value calculation unit 30. t Action a t Use the policy function shown in the following formula (3) In addition, it is sometimes referred to as the policy function π below.

[0092]

[0093] here, Indicates that action a is selected t The state transition probability under the condition .

[0094] The action selection unit 32 will use the policy function Selected action a t Output to the simulator 20. Thus, the simulator 20 makes the robot RB perform action a by simulation. t For example, in action a t The moving direction m t and movement speed v t In the case of movement, the simulator 20 moves in the direction m during the simulation. t and movement speed v t Make robot RB move. In addition, in action a t In the case of an intervention action, the simulator 20 simulates the avoidance action that the surrounding person HB may take when performing an intervention action such as outputting a message such as "Please make way" or sounding a warning sound. t , the state s of robot RB t and the HB status of the people around you ～ s t Then, for the changed state, the V network is learned by repeating the calculation of reward r, calculation of state value V, selection and execution of action a, and parameter update of the V network in the same manner as described above.

[0095] In this manner, the robot control model learning device 10 can be said to functionally input state information and select and output a robot control model of an action corresponding to the input state information.

[0096] Next, the operation of the robot control model learning device 10 will be described.

[0097] Figure 5 1 is a flowchart showing the flow of the robot control model learning process of the robot control model learning device 10. The CPU 11 reads the robot control model learning program from the storage device 14, expands it in the RAM 13, and executes it, thereby performing the robot control model learning process.

[0098] In step S100, the CPU 11, as the state value calculation unit 30, obtains the destination p from the simulator 20. g location information.

[0099] In step S102 , the CPU 11 , as the state value calculation unit 30 , initializes the state value function V. That is, the parameters of the V network are initialized.

[0100] In step S104, the CPU 11, as the state value calculation unit 30, calculates the state s of the robot RB. t Initialize.

[0101] In step S106, the CPU 11 as the state value calculation unit 30 calculates the state of the surrounding person HB. ～ s t Initialize.

[0102] In step S108 , the CPU 11 , as the action selection unit 32 , sets the action a to be initially taken by the robot RB, and outputs the set action a to the simulator 20 , thereby causing the robot RB to execute the action a. As a result, the simulator 20 executes the action a in the simulation.

[0103] In step S110, the CPU 11, as the state value calculation unit 30, obtains the state s of the robot RB from the simulator 20. t .

[0104] In step S112, the CPU 11 as the state value calculation unit 30 obtains the state of the surrounding person HB from the simulator 20. ～ s t .

[0105] In step S114, the CPU 11 as the state value calculation unit 30 calculates the state value of the robot RB and the surrounding human HB based on the state value of the robot RB and the surrounding human HB obtained from the simulator 20. and action a t, calculate the reward r through the reward function R t .

[0106] In step S116, the CPU 11, as the state value calculation unit 30, calculates the state value y using the above formula (1). t .

[0107] In step S118, the CPU 11, as the action selection unit 32, selects action a according to the above formula (3). t , the selected action a t Output to the simulator 20. Thus, the simulator 20 causes the robot RB to perform action a in the simulation. t .

[0108] In step S120, the CPU 11, as the state value calculation unit 30, calculates the state of the robot RB. and the state value y t Stored in buffer E as a pair.

[0109] In step S122, the CPU 11, as the state value calculation unit 30, updates the parameters of the V network. In other words, the V network is learned. At this time, the past state s stored in the buffer E is randomly selected. jn The state value y is used as the teacher data to update the parameters of the V network. This is done by using the gradient descent method used in reinforcement learning to update the parameters of the V network. Furthermore, the process of step S122 need not be performed every time; it can be performed only once out of multiple times.

[0110] In step S124, the CPU 11 as the state value calculation unit 30 determines whether the robot RB has reached the destination p g That is, determine whether the position p of the robot RB is consistent with the destination p g Then, when it is determined that the robot RB has reached the destination p g On the other hand, if it is determined that the robot RB has not reached the destination p, the process proceeds to step S126. g In the case of a failure, the process proceeds to step S110, and the processes of steps S110 to S124 are repeated until it is determined that the robot RB has reached the destination p. g That is, the V network is learned. Note that the processing of steps S110 to S124 is an example of a learning step.

[0111] In step S126, the CPU 11 as the state value calculation unit 30 determines whether the end condition for ending the learning is satisfied. In this embodiment, the end condition is, for example, that the robot RB reaches the destination p from the starting point. gIf the CPU 11 determines that the termination condition is satisfied, the routine ends. On the other hand, if the termination condition is not satisfied, the routine proceeds to step S100, and the destination p is changed. g , repeat the processing from step S100 to step S126 until the end condition is met.

[0112] Based on the above, in this embodiment, the reward r calculated by the reward function R t Including the reward ε related to collision avoidance c and the compensation associated with the intervention ε b , these rewards r t By using such a reward function R to learn the V network, the number of interventions and collision avoidances by the robot RB can be reduced. This can reduce the pressure on the surrounding environment and shorten the time it takes for the robot RB to reach its destination p. g The time until now.

[0113] Next, the robot RB controlled by the robot control model learned by the robot control model learning device 10 will be described.

[0114] exist Figure 6 The schematic structure of the robot RB is shown in FIG. Figure 6 As shown, the robot RB includes a robot control device 40 , a camera 42 , a robot information acquisition unit 44 , a notification unit 46 , and an autonomous driving unit 48 . The robot control device 40 includes a state information acquisition unit 50 and a control unit 52 .

[0115] The camera 42 moves from the starting point to the destination point p g During the period, the surroundings of the robot RB are imaged at predetermined intervals, and the captured images are output to the state information acquisition unit 50 of the robot control device 40.

[0116] The robot information acquisition unit 44 acquires the status s of the robot RB t Specifically, the position p of the robot RB can be obtained using, for example, GPS (Global Positioning System). t The speed of the robot RB can be obtained using a global positioning system (GPS) device or a known self-position estimation technology such as SLAM (Simultaneous Localization and Mapping). The speed of the robot RB can be obtained using a speed sensor, for example.

[0117] The robot information acquisition unit 44 acquires the status s of the robot RB. t Output to the status information acquisition unit 50.

[0118] The state information acquisition unit 50 acquires the state of the person HB based on the image captured by the camera 42. ～ s t Specifically, the captured image is analyzed using a known method to calculate the position and speed of the person HB existing around the robot RB.

[0119] Furthermore, destination information is input to the state information acquisition unit 50 from an external device through, for example, communication.

[0120] The state information acquisition unit 50 includes the acquired destination information, the state s of the robot RB, t and the status of human HB ～ s t The status information is output to the control unit 52.

[0121] The control unit 52 has the function of the robot control model learned by the robot control model learning device 10. In other words, it has the function of the state value calculation unit 30 and the action selection unit 32 after the V network learning.

[0122] Control unit 52 selects an action corresponding to the input status information, and controls at least one of notification unit 46 and autonomous driving unit 48 based on the selected action.

[0123] The notification unit 46 has a function of notifying the surrounding people HB of the presence of the robot RB by outputting a sound or a warning tone.

[0124] The autonomous driving unit 48 includes tires, a motor for driving the tires, and the like, and has a function of causing the robot RB to autonomously drive.

[0125] When the selected action is an action of moving the robot RB in a designated direction and at a designated speed, the control unit 52 controls the autonomous driving unit 48 to move the robot RB in the designated direction and at the designated speed.

[0126] Furthermore, when the selected action is an intervention action, the control unit 52 controls the notification unit 46 to output a message such as “Please give way” or sound a warning sound.

[0127] Next, the hardware configuration of the robot control device 40 will be described.

[0128] like Figure 7As shown, the robot control device 40 includes a CPU (Central Processing Unit) 61, a ROM (Read Only Memory) 62, a RAM (Random Access Memory) 63, a storage device 64, and a communication interface 65. These components are connected via a bus 66 so as to be able to communicate with each other.

[0129] In this embodiment, a robot control program is stored in storage device 64. CPU 61, a central processing unit, executes various programs and controls various components. Specifically, CPU 61 reads programs from storage device 64 and executes them using RAM 63 as a work area. CPU 61 controls the aforementioned components and performs various calculations according to the programs stored in storage device 64.

[0130] The ROM 62 stores various programs and data. The RAM 63 temporarily stores programs and data as a work area. The storage device 64 is composed of an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including the operating system and various data.

[0131] The communication interface 65 is an interface for communicating with other devices, and uses a standard such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark).

[0132] Next, the operation of the robot control device 40 will be described.

[0133] Figure 8 4 is a flowchart showing the flow of the robot control process of the robot control device 40. The CPU 61 reads the robot control program from the storage device 64, expands the program in the RAM 63, and executes the program, thereby performing the robot control process.

[0134] In step S200, the CPU 61, as the state information acquisition unit 50, acquires the destination p from, for example, an external device (not shown) through communication. g location information.

[0135] In step S202, the CPU 61, as the state information acquisition unit 50, acquires the state s of the robot RB from the robot information acquisition unit 44. t .

[0136] In step S204, the CPU 61 as the state value calculation unit 30 obtains the state of the surrounding person HB based on the image captured by the camera 42. ～ st .

[0137] In step S206, the CPU 61 as the control unit 52 determines the status of the robot RB and the surrounding human HB based on the status information acquired from the status information acquisition unit 50. and action a t , calculate the reward r through the reward function R t .

[0138] In step S208, the CPU 61 as the control unit 52 calculates the value y of the state using the above formula (1). t .

[0139] In step S210, the CPU 61 as the control unit 52 selects action a according to the above formula (3). t , based on the selected action a t To control at least one of the notification unit 46 and the autonomous driving unit 48. As a result, the robot RB performs action a t .

[0140] In step S212, the CPU 61 as the control unit 52 determines whether the robot RB has reached the destination p g That is, determine whether the position p of the robot RB is consistent with the destination p g Then, when it is determined that the robot RB has reached the destination p g On the other hand, if it is determined that the robot RB has not reached the destination p, this routine is terminated. g In the case of a failure, the process proceeds to step S202, and the processing of steps S202 to S212 is repeated until it is determined that the robot RB has reached the destination p. g Note that the processing of steps S202 to S212 is an example of a control step.

[0141] Thus, in this embodiment, the robot RB is controlled based on the robot control model learned by the robot control model learning device 10. This can reduce the number of interventions and collision avoidances required by the robot RB. This can reduce the stress on the surrounding environment and shorten the time it takes for the robot RB to reach its destination p. g The time until now.

[0142] In addition, in this embodiment, the case of learning the state value function V is described, but the learning method is not limited to this. For example, instead of learning the state value function V, the action value function Q(s) for calculating the action value of the robot RB may be learned. jn 、a).

[0143] In this embodiment, the reward output as the reward function R includes the reward ε related to collision avoidance. c , Rewards related to intervention actions ε b The case of , however, it is also possible to exclude the reward ε related to collision avoidance c .

[0144] In addition, in this embodiment, the robot RB is described as being equipped with the camera 42, but the present invention is not limited thereto. For example, the camera 42 may be omitted, and an overhead image of the robot RB may be acquired from an external device, and the status of the person HB around the robot RB may be acquired by analyzing the acquired overhead image. ～ s t .

[0145] In this embodiment, the robot RB is described as having a robot control device 40. However, the functions of the robot control device 40 may be provided to an external server. In this case, the robot RB transmits images captured by the camera 42 and robot information acquired by the robot information acquisition unit 44 to the external server, and executes the actions instructed by the external server.

[0146] In addition, the robot control processing executed by the CPU reading the software (program) in each of the above-mentioned embodiments can also be executed by various processors other than the CPU. Examples of processors in this case include FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices) whose circuit structure can be changed after manufacturing, and ASICs (Application Specific Integrated Circuits), which are processors with circuit structures specifically designed to perform specific processing, i.e., dedicated circuits. In addition, the robot control model learning processing and the robot control processing can be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, etc.). In addition, more specifically, the hardware structure of these various processors is a circuit that combines circuit elements such as semiconductor elements.

[0147] Furthermore, in the above embodiments, the robot control model learning program is pre-stored in the storage device 14, and the robot control program is pre-stored in the storage device 64. However, this is not limiting. The program may also be provided by recording it on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory device. Furthermore, the program may be downloaded from an external device via a network.

[0148] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, and technical standard were specifically and individually described.

[0149] Label Description

[0150] 10: Robot control model learning device; 20: Simulator; 30: State value calculation unit; 32: Action selection unit; 40: Robot control device; 42: Camera; 44: Robot information acquisition unit; 46: Notification unit; 48: Autonomous driving unit; 50: State information acquisition unit; 52: Control unit; HB: Human; RB: Robot.

Claims

1. A robot control model learning method, wherein a computer performs a process comprising the following steps: A learning step of performing reinforcement learning on a robot control model that takes state information representing a state of a robot autonomously traveling to a destination in a dynamic environment as input and selects and outputs an action corresponding to the state of the robot from a plurality of actions including intervention actions that intervene in the environment, using a reward that takes a negative value only when the robot performs the intervention action; The reinforcement learning includes: learning a deep neural network representing a state-value function, the state-value function representing the state of the robot; and The state of the robot is determined based on the reward function, the discount rate, the increase in time in 1 step, and the state-value function, and the parameters of the deep neural network are updated, thereby updating the state-value function.

2. The robot control model learning method according to claim 1, wherein: The action includes at least one of the movement direction of the robot, the movement speed of the robot, and the intervention action. The reward is given so as to reduce at least one of an arrival time for the robot to reach the destination and a number of interventions for which the intervention action is performed.

3. The robot control model learning method according to claim 1 or 2, wherein: The actions include avoidance actions to avoid collision of the robot with other objects. The reward is given so that the number of avoidance times for avoiding the collision becomes smaller.

4. A robot control model learning device, comprising: a learning unit that performs reinforcement learning on a robot control model that takes state information representing the state of a robot autonomously traveling to a destination in a dynamic environment as input and selects and outputs an action corresponding to the state of the robot from a plurality of actions including intervention actions that intervene in the environment, using a reward that takes a negative value only when the robot performs the intervention action; The reinforcement learning includes: learning a deep neural network representing a state-value function, wherein the state-value function represents a state of the robot; as well as The state of the robot is determined based on the reward function, the discount rate, the increase in time in 1 step, and the state-value function, and the parameters of the deep neural network are updated, thereby updating the state-value function.

5. A recording medium having a robot control model learning program recorded thereon, the robot control model learning program causing a computer to execute a process comprising the following steps: A learning step of performing reinforcement learning on a robot control model that takes state information representing a state of a robot autonomously traveling to a destination in a dynamic environment as input and selects and outputs an action corresponding to the state of the robot from a plurality of actions including intervention actions that intervene in the environment, using a reward that takes a negative value only when the robot performs the intervention action; The reinforcement learning includes: learning a deep neural network representing a state-value function, the state-value function representing the state of the robot; and The state of the robot is determined based on the reward function, the discount rate, the increase in time in 1 step, and the state-value function, and the parameters of the deep neural network are updated, thereby updating the state-value function.

6. A robot control method, comprising executing a process comprising the following steps by a computer: an acquiring step of acquiring state information indicating the state of the robot as it autonomously travels to a destination in a dynamic environment; and A control step of controlling the robot to move to the destination based on the state information and the robot control model learned by the robot control model learning method according to any one of claims 1 to 3.

7. A robot control device comprising: an acquisition unit that acquires state information indicating a state of the robot as it autonomously travels to a destination in a dynamic environment; as well as A control unit controls the robot to move to the destination based on the state information and the robot control model learned by the robot control model learning device according to claim 4.

8. A recording medium having a robot control program recorded thereon, the robot control program causing a computer to execute a process comprising the following steps: an acquiring step of acquiring state information indicating the state of the robot as it autonomously travels to a destination in a dynamic environment; and A control step of controlling the robot to move to the destination based on the state information and the robot control model learned by the robot control model learning method according to any one of claims 1 to 3.

9. A robot comprising: an acquisition unit that acquires state information indicating a state of the robot as it autonomously travels to a destination in a dynamic environment; an autonomous driving unit, which enables the robot to drive autonomously; as well as A control unit controls the robot to move to the destination based on the state information and the robot control model learned by the robot control model learning device according to claim 4.

Images