Learning and control device, learning device, control device, learning and control method, learning method, control method, learning and control program, learning program, and control program

The learning and control system efficiently generates and optimizes models for robot actions using multiple dynamics models and reinforcement learning, reducing the number of required trials and enhancing control efficiency.

JP7585999B2Active Publication Date: 2024-11-19OMRON CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2021109158
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-06-30
Publication Date
2024-11-19
Estimated Expiration
2041-06-30

AI Technical Summary

Technical Problem

Existing methods require significant effort to program a series of actions for controlled objects like robots, and training a single model to accurately predict all state transitions necessitates a large number of trials.

Method used

A learning and control system that generates multiple dynamics models from state transition data, learns a switching model to identify the appropriate model for each state and action, and uses reinforcement learning to optimize command sequences for efficient robot operation.

Benefits of technology

Enables learning a model for the entire series of robot actions in a small number of trials, allowing effective control of the robot's operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007585999000001
    Figure 0007585999000001
  • Figure 0007585999000002
    Figure 0007585999000002
  • Figure 0007585999000003
    Figure 0007585999000003
Patent Text Reader

Abstract

To make it possible to learn by a small number of trials, a model that can be applied to the entire sequence of operations performed by a controlled object.SOLUTION: A learning device 50 in a learning and control device 40 generates a plurality of dynamics models f, and learns a switching model g that identifies, among the dynamics models, a dynamics model f corresponding to a state st of a robot 10 and a command operation at. A control device 60 is configured to: acquire the state st of the robot 10; generate a plurality of tentative command sequences for the robot 10; identify a dynamics model f to be applied for each command and state corresponding to the command, by implementing the switching model g by inputting each command included in each tentative command sequence, and a state corresponding to each command; generate, for each of the tentative command sequences, a predicted state sequence using the dynamics model f identified corresponding to each of the commands included in the tentative command sequence; generate a predicted command sequence that is predicted to maximize reward of the predicted state sequence; and output a first command included in the predicted command sequence.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The disclosed technology relates to a learning and control device, a learning device, a control device, a learning and control method, a learning method, a control method, a learning and control program, a learning program, and a control program. [Background technology]

[0002] Non-Patent Document 1 discloses a model-based reinforcement learning method using multiple state transition models.

[0003] Non-Patent Document 2 discloses a method of dividing the entire space into small spaces having different subgoals from a teaching trajectory, and learning a different policy (controller) for each divided small space. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] K.Doya, K.Samejima, K.Katagiri, and M.Kawato, "Multiple model-based reinforcement learning," Neural computation, vol.14, no.6, pp.1347-1369,2002. [Non-Patent Document 2] Paul, Sujoy, Jeroen van Baar, and Amit K. Roy-Chowdhury. "Learning from trajectories via subgoal discovery." arXiv preprint arXiv:1911.07224 (2019). Summary of the Invention [Problem to be solved by the invention]

[0005] It is laborious to program a series of operations to be executed by a controlled object such as a robot. If the controlled object could autonomously learn a series of operations, this labor could be eliminated.

[0006] However, if one tries to train a single model to accurately predict all state transitions in a series of actions, a large number of trials are required.

[0007] The disclosed technology has been made in consideration of the above-mentioned points, and aims to provide a learning and control device, a learning device, a control device, a learning and control method, a learning method, a control method, a learning and control program, a learning program, and a control program that can learn a model that can be applied to the entire series of actions performed by the controlled object in a small number of trials, and can control the entire series of actions performed by the controlled object using the learned model. [Means for solving the problem]

[0008] A first aspect of the disclosure is a learning and control device, comprising: a state transition data acquisition unit that acquires a plurality of state transition data including a state of a control object obtained by making the control object perform a predetermined series of operations, a command operation commanded to the control object in the state, and a next state after the control object has performed the command operation; a dynamics model generation unit that generates a plurality of dynamics models having the state and the command operation as input and the next state as output, each of the dynamics models conforming to a set of state transition data consisting of a portion of the acquired plurality of state transition data, and the plurality of dynamics models conforming to sets of state transition data that are different from one another; an assignment unit that assigns a label to the state transition data included in the set of state transition data that conforms to the generated dynamics model, the label being used to identify the conforming dynamics model; and using the state transition data to which the label has been assigned as learning data, from among the plurality of dynamics models, a previous state corresponding to the input state of the control object and the command operation is selected. the dynamics model to be applied to each command and each state corresponding to the command by inputting each command included in each provisional command sequence and a state corresponding to each command and executing the switching model; a predicted state sequence generation unit to generate a predicted state sequence for each of the provisional command sequences using the dynamics model specified corresponding to each command included in the provisional command sequence; a calculation unit to calculate a reward for each predicted state sequence; a predicted command sequence generation unit to generate a predicted command sequence predicted to maximize the reward;

[0009] In the first aspect, when generating one dynamics model among the plurality of dynamics models, the dynamics model generation unit may generate a provisional dynamics model by using all of the state transition data usable for generating the dynamics model, and thereafter calculate an error between the next state obtained by inputting the state of the control object and the command action into the generated provisional dynamics model, and the next state included in the state transition data including the state and the command action, and generate the provisional dynamics model excluding the state transition data with the maximum error, thereby repeating the process of generating the provisional dynamics model, thereby generating the dynamics model with the calculated error being equal to or less than a predetermined threshold.

[0010] In the first aspect, the dynamics model generation unit may be configured to, each time generating one of the plurality of dynamics models, make the state transition data remaining without being removed in a generation process of the dynamics model unusable in a subsequent generation of the dynamics model, and generate a next dynamics model.

[0011] In the first aspect, the dynamics model generation unit may generate the dynamics model by restoring the state transition data randomly selected from the state transition data with the maximum error removed to the state transition data used to generate the dynamics model at a predetermined frequency.

[0012] In the first aspect, the interim command sequence generation unit generates one interim command sequence, the predicted state sequence generation unit generates the predicted state sequence corresponding to the interim command sequence generated by the interim command sequence generation unit, the calculation unit calculates a reward for the predicted state sequence generated by the predicted state sequence generation unit, and the predicted command sequence generation unit may generate a predicted command sequence predicted to maximize the reward by updating the interim command sequence one or more times so as to increase the reward by executing a series of operations of the interim command sequence generation unit, the identification unit, the predicted state sequence generation unit, and the calculation unit a plurality of times.

[0013] In the first aspect, the tentative command sequence generation unit may generate a plurality of tentative command sequences collectively, the predicted state sequence generation unit may generate the predicted state sequence from each of the plurality of tentative command sequences, the calculation unit may calculate a reward for each predicted state sequence, and the predicted command sequence generation unit may generate a predicted command sequence that is predicted to maximize the reward based on the reward for each predicted state sequence.

[0014] In the above first aspect, the interim command sequence generation unit may repeatedly execute a series of processes from the process of collectively generating the plurality of interim command sequences to the process of calculating the reward, and in the second or subsequent series of processes, the interim command sequence generation unit may select a plurality of interim command sequences corresponding to a predetermined higher reward among the rewards calculated in the previous series of processes, and generate new a plurality of interim command sequences based on a distribution of the selected a plurality of interim command sequences.

[0015] A second aspect of the disclosure is a learning device comprising: a state transition data acquisition unit that acquires a plurality of state transition data including a state of a control object obtained by making the control object perform a predetermined series of operations, a command operation commanded to the control object in the state, and a next state after the control object performs the command operation; a dynamics model generation unit that generates a plurality of dynamics models having the state and the command operation as input and the next state as output, each of the dynamics models conforming to a set of state transition data consisting of a portion of the acquired plurality of state transition data, and the plurality of dynamics models are suitable for sets of state transition data that are different from each other; an assignment unit that assigns a label to the state transition data included in the set of state transition data that conforms to the generated dynamics model, the label identifying the conforming dynamics model; and a learning unit that uses the state transition data to which the label has been assigned as learning data and learns a switching model that identifies the dynamics model corresponding to the input state of the control object and the command operation from among the plurality of dynamics models.

[0016] A third aspect of the disclosure is a control device comprising: a state acquisition unit that acquires a state of a control object; a provisional command sequence generation unit that generates a plurality of provisional command sequences for the control object; an identification unit that inputs each command included in each provisional command sequence and a state corresponding to each command, and executes a switching model learned by the learning device to identify the dynamics model to be applied to each command and a state corresponding to the command, among the dynamics models generated by the learning device; a predicted state sequence generation unit that generates a predicted state sequence for each of the provisional command sequences using the dynamics model identified corresponding to each command included in the provisional command sequence; a calculation unit that calculates a reward for each predicted state sequence; a predicted command sequence generation unit that generates a predicted command sequence that is predicted to maximize the reward; an output unit that outputs a first command included in the generated predicted command sequence; and an execution control unit that controls an operation of the control object by repeating a series of operations of the state acquisition unit, the provisional command sequence generation unit, the identification unit, the predicted state sequence generation unit, the calculation unit, the predicted command sequence generation unit, and the output unit.

[0017] A fourth aspect of the disclosure is a learning and control method, comprising: a state transition data acquisition step of acquiring a plurality of state transition data including a state of the control object, a command action commanded to the control object in the state, and a next state after the control object has performed the command action, the state transition data being obtained by causing the control object to perform a predetermined series of actions; a dynamics model generation step of generating a plurality of dynamics models having the state and the command action as input and the next state as output, each of the dynamics models being adapted to a set of state transition data consisting of a part of the acquired plurality of state transition data, and the plurality of dynamics models being adapted to sets of state transition data that are different from one another; an assignment step of assigning a label that identifies the adapted dynamics model to the state transition data included in the set of state transition data that is adapted to the generated dynamics model; and a switch step of identifying the dynamics model that corresponds to the input state of the control object and the command action from among the plurality of dynamics models, using the state transition data to which the label has been assigned as learning data. a learning step of learning a switching model, a state acquisition step of acquiring a state of the controlled object, a tentative command sequence generation step of generating a plurality of tentative command sequences for the controlled object, a specification step of inputting each command included in each tentative command sequence and a state corresponding to each command, thereby executing the switching model, specifying a dynamics model to be applied to each command and a state corresponding to each command, a predicted state sequence generation step of generating a predicted state sequence for each of the tentative command sequences using the dynamics model specified corresponding to each command included in the tentative command sequence, a calculation step of calculating a reward for each predicted state sequence, a predicted command sequence generation step of generating a predicted command sequence that is predicted to maximize the reward, an output step of outputting a first command included in the generated predicted command sequence, and an execution control step of controlling an operation of the controlled object by repeating a series of operations including the state acquisition step, the tentative command sequence generation step, the specification step, the predicted state sequence generation step, the calculation step, the predicted command sequence generation step, and the output step.

[0018] A fifth aspect of the disclosure is a learning method, the computer executing a process including: a state transition data acquisition step of acquiring a plurality of state transition data, the state of the control object being obtained by making the control object perform a predetermined series of operations, the state transition data including a state of the control object, a command operation commanded to the control object in the state, and a next state after the control object has performed the command operation; a dynamics model generation step of generating a plurality of dynamics models having the state and the command operation as input and the next state as output, each of the dynamics models conforming to a set of state transition data consisting of a part of the acquired plurality of state transition data, and the plurality of dynamics models conforming to different sets of state transition data from each other; an assignment step of assigning a label to the state transition data included in the set of state transition data conforming to the generated dynamics model, the label identifying the conforming dynamics model; and a learning step of learning a switching model, using the state transition data to which the label has been assigned as learning data, from among the plurality of dynamics models, which identifies the dynamics model corresponding to the input state of the control object and the command operation.

[0019] A sixth aspect of the disclosure is a control method, in which a computer executes processes including a state acquisition step of acquiring a state of a controlled object, a tentative command sequence generation step of generating a plurality of tentative command sequences for the controlled object, a specification step of inputting each command included in each tentative command sequence and a state corresponding to each command, and executing a switching model learned by the learning method, thereby specifying the dynamics model to be applied to each command and a state corresponding to the command, among the dynamics models generated by the learning method, a predicted state sequence generation step of generating a predicted state sequence for each of the tentative command sequences using the dynamics model specified corresponding to each command included in the tentative command sequence, a calculation step of calculating a reward for each predicted state sequence, a predicted command sequence generation step of generating a predicted command sequence that is predicted to maximize the reward, an output step of outputting a first command included in the generated predicted command sequence, and an execution control step of controlling an operation of the controlled object by repeating a series of operations of the state acquisition step, the tentative command sequence generation step, the specification step, the predicted state sequence generation step, the calculation step, the predicted command sequence generation step, and the output step.

[0020] A seventh aspect of the disclosure is a learning and control program, comprising: a state transition data acquisition step of acquiring a plurality of state transition data including a state of a control object, a command action commanded to the control object in the state, and a next state after the control object has performed the command action, the state transition data being acquired by causing a computer to perform a predetermined series of actions on the control object; a dynamics model generation step of generating a plurality of dynamics models having the state and the command action as input and the next state as output, each of the dynamics models being compatible with a set of state transition data consisting of a part of the acquired plurality of state transition data, and the plurality of dynamics models being compatible with sets of state transition data different from each other; an assignment step of assigning a label that identifies the compatible dynamics model to the state transition data included in the set of state transition data that is compatible with the generated dynamics model; and a switch for identifying the dynamics model that corresponds to the input state of the control object and the command action from among the plurality of dynamics models, using the state transition data to which the label has been assigned as learning data. a learning step of learning a switching model, a state acquisition step of acquiring a state of the controlled object, a tentative command sequence generation step of generating a plurality of tentative command sequences for the controlled object, a specification step of inputting each command included in each tentative command sequence and a state corresponding to each command, thereby executing the switching model, specifying a dynamics model to be applied to each command and a state corresponding to each command, a predicted state sequence generation step of generating a predicted state sequence for each of the tentative command sequences using the dynamics model specified corresponding to each command included in the tentative command sequence, a calculation step of calculating a reward for each predicted state sequence, a predicted command sequence generation step of generating a predicted command sequence that is predicted to maximize the reward, an output step of outputting a first command included in the generated predicted command sequence, and an execution control step of controlling an operation of the controlled object by repeating a series of operations including the state acquisition step, the tentative command sequence generation step, the specification step, the predicted state sequence generation step, the calculation step, the predicted command sequence generation step, and the output step.

[0021] An eighth aspect of the disclosure is a learning program that causes a computer to execute a process including: a state transition data acquisition step of acquiring a plurality of state transition data including a state of the control object, a command action commanded to the control object in the state, and a next state after the control object has performed the command action, the state transition data being obtained by causing the control object to perform a predetermined series of actions; a dynamics model generation step of generating a plurality of dynamics models having the state and the command action as input and the next state as output, each of the dynamics models conforming to a set of state transition data consisting of a part of the acquired plurality of state transition data, and the plurality of dynamics models conforming to different sets of state transition data from each other; an assignment step of assigning a label that identifies the conforming dynamics model to the state transition data included in the set of state transition data that conforms to the generated dynamics model; and a learning step of learning a switching model that identifies the dynamics model corresponding to the input state of the control object and the command action from among the plurality of dynamics models, using the state transition data to which the label has been assigned as learning data.

[0022] A ninth aspect of the disclosure is a control program that causes a computer to execute processing including: a state acquisition step of acquiring a state of a controlled object; a tentative command sequence generation step of generating a plurality of tentative command sequences for the controlled object; a specification step of inputting each command included in each tentative command sequence and a state corresponding to each command, and executing a switching model learned by the learning program, thereby specifying the dynamics model to be applied to each command and a state corresponding to the command, among the dynamics models generated by the learning program; a predicted state sequence generation step of generating a predicted state sequence for each of the tentative command sequences using the dynamics model specified corresponding to each command included in the tentative command sequence; a calculation step of calculating a reward for each predicted state sequence; a predicted command sequence generation step of generating a predicted command sequence that is predicted to maximize the reward; an output step of outputting a first command included in the generated predicted command sequence; and an execution control step of controlling an operation of the controlled object by repeating a series of operations including the state acquisition step, the tentative command sequence generation step, the specification step, the predicted state sequence generation step, the calculation step, the predicted command sequence generation step, and the output step. Effect of the Invention

[0023] According to the present invention, a model that can be applied to the entire series of operations performed by the controlled object can be learned with a small number of trials, and the entire series of operations performed by the controlled object can be controlled using the learned model. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a configuration diagram of a robot system. [Diagram 2] FIG. 1 is a diagram showing a schematic configuration of a robot. [Diagram 3] FIG. 2 is a diagram showing a series of operations performed by the robot. [Figure 4] FIG. 2 is a block diagram showing a hardware configuration of a learning and control device. [Diagram 5] FIG. 2 is a diagram illustrating a functional configuration of a learning device. [Figure 6] FIG. 2 is a diagram illustrating a functional configuration of a control device. [Figure 7] 13 is a flowchart of a learning process. [Figure 8] 13 is a flowchart of a model generation process. [Figure 9] 2 is a flowchart of a control process 1. [Figure 10] 13 is a flowchart of a control process 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings may be exaggerated for the convenience of explanation and may differ from the actual ratios.

[0026] 1 shows a configuration of a robot system 1. The robot system 1 includes a robot 10 as an example of an object to be controlled, a model 20, a state observation sensor 30, and a learning and control device 40.

[0027] (robot)

[0028] 2 is a diagram showing a schematic configuration of a robot 10 as an example of a control target. The robot 10 in this embodiment is a six-axis vertical articulated robot having an arm 11 with six degrees of freedom. A flat hand 12 is provided at the tip of the arm 11.

[0029] The robot 10 is not limited to a vertical articulated robot, but may be a horizontal articulated robot (SCARA robot). Although a six-axis robot is given as an example, the robot may be a five-axis or seven-axis articulated robot with other degrees of freedom, or may be a parallel link robot.

[0030] In this embodiment, as shown in Fig. 3, a state in which a ball BL is placed on the front surface of the hand 12 is set as an initial state, and a series of juggling actions are performed by the robot 10 in which the ball BL is thrown upward in Fig. 3, the hand 12 is turned over, and the ball BL is placed on the back surface of the hand 12. In other words, when the hand 12 is regarded as a human hand, a series of juggling actions are performed by the robot 10 in which the ball BL is thrown upward from a state in which the ball BL is placed on the horizontal palm of the hand, the hand is turned over, and the ball BL is placed on the back of the horizontal hand.

[0031] (Model)

[0032] The model 20 includes a dynamics model group F, a switching model g, and a model selection unit 21. The dynamics model group F includes a plurality of dynamics models f1, f2, .... When there is no need to distinguish between the individual dynamics models, they will be referred to as dynamics model f.

[0033] The dynamics model f is the state s of the robot 10. t and state s t The command action a given to the robot 10 t The robot 10 performs the command action a t Next state s after t+1 This is a model whose output is

[0034] The switching model g selects the state s of the input robot 10 from among multiple dynamics models f. t and command action a t Identify the dynamics model f corresponding to

[0035] The model selection unit 21 selects a dynamics model f specified by the switching model g, and calculates the next state s output from the selected dynamics model f. t+1 to the learning and control device 60.

[0036] The robot system 1 acquires the switching model g that selects the dynamics model f for controlling the robot 10 as described above, using machine learning (for example, model-based reinforcement learning).

[0037] (Status Observation Sensor)

[0038] The state observation sensor 30 observes the states of the robot 10 and the ball BL, and outputs the observed data as state observation data. The state observation sensor 30 includes, for example, an encoder for the joints of the robot 10. As the state of the robot 10, the position and posture of the hand 12 at the tip of the arm 11 can be identified from the angle of each joint. The state observation sensor 30 also includes, for example, a camera that captures an image of the ball BL. The position of the ball BL can be identified based on the image captured by the camera.

[0039] (Learning and Control Device)

[0040] As shown in FIG. 1, the learning and control device 40 includes a learning device 50 and a control device 60 .

[0041] Fig. 4 is a block diagram showing a hardware configuration of the learning and control device 40 according to this embodiment. As shown in Fig. 4, the learning and control device 40 has the same configuration as a general computer (information processing device), and includes a CPU (Central Processing Unit) 40A, a ROM (Read Only Memory) 40B, a RAM (Random Access Memory) 40C, a storage 40D, a keyboard 40E, a mouse 40F, a monitor 40G, and a communication interface 40H. Each component is connected to each other so as to be able to communicate with each other via a bus 40I.

[0042] In this embodiment, the ROM 40B or the storage 40D stores a learning program for executing a model learning process and a control program for controlling the robot 10. The CPU 40A is a central processing unit, and executes various programs and controls each configuration. That is, the CPU 40A reads a program from the ROM 40B or the storage 40D, and executes the program using the RAM 40C as a working area. The CPU 40A controls each of the above configurations and performs various arithmetic processing according to the program recorded in the ROM 40B or the storage 40D. The ROM 40B stores various programs and various data. The RAM 40C temporarily stores programs or data as a working area. The storage 40D is composed of a HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory, and stores various programs including an operating system and various data. The keyboard 40E and the mouse 40F are examples of input devices, and are used to perform various inputs. The monitor 40G is, for example, a liquid crystal display, and displays a user interface. The monitor 40G may be a touch panel type and function as an input unit. The communication interface 40H is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark).

[0043] Next, the functional configuration of the learning device 50 will be described.

[0044] 5, the learning device 50 includes, as its functional components, a state transition data acquisition unit 51, a dynamics model generation unit 52, an assignment unit 53, and a learning unit 54. Each functional component is realized by the CPU 40A reading out a learning program stored in the ROM 40B or the storage 40D, expanding it in the RAM 40C, and executing it. Note that some or all of the functions may be realized by a dedicated hardware device.

[0045] The state transition data acquisition unit 51 acquires the state s of the robot 10 obtained by making the robot 10 perform a series of predetermined actions. t and state s t The command action a given to the robot 10 t and the next state s after the robot 10 performs the commanded action. t+1 A plurality of tuples including the above are obtained as state transition data.

[0046] The dynamics model generating unit 52 generates a state s t and command action a t The input is the next state s t+1 Each dynamics model f is adapted to a set of tuples consisting of a part of the acquired multiple tuples, and the multiple dynamics models f are adapted to different sets of tuples from each other.

[0047] In addition, when generating one of the dynamics models f among the multiple dynamics models f, the dynamics model generating unit 52 generates a provisional dynamics model f by using all tuples that can be used to generate the dynamics model f, and thereafter, the dynamics model generating unit 52 applies the state s of the robot 10 to the generated provisional dynamics model f. t and command action a t The next state s obtained by inputting t+1 and the state s t and the command action a t The next state s contained in the tuple containing t+1 By repeatedly calculating the error between and , and excluding the tuple with the maximum error to generate a provisional dynamics model f, a dynamics model f with a calculated error equal to or smaller than a predetermined threshold is generated.

[0048] In addition, each time the dynamics model generation unit 52 generates one of the multiple dynamics models f, it makes the tuples that remain in the generation process of the dynamics model f unusable in the generation of the subsequent dynamics model f, and generates the next dynamics model f.

[0049] Furthermore, the dynamics model generating unit 52 returns a tuple randomly selected from the removed tuples with the maximum error to a tuple used for generating the dynamics model f at a predetermined frequency, thereby generating the dynamics model f.

[0050] The assigning unit 53 assigns a label that identifies the matching dynamics model f to a tuple included in a set of tuples that match the generated dynamics model f.

[0051] The learning unit 54 uses the labeled tuples as learning data to learn the state s of the input robot 10 from among a plurality of dynamics models f. t and command action a t We learn a switching model g that specifies the dynamics model f corresponding to

[0052] 6, the control device 60 includes, as its functional configuration, a state acquisition unit 61, a tentative command sequence generation unit 62, a determination unit 63, a predicted state sequence generation unit 64, a calculation unit 65, a predicted command sequence generation unit 66, an output unit 67, and an execution control unit 68. Each functional configuration is realized by the CPU 40A reading out a control program stored in the ROM 40B or the storage 40D, expanding it in the RAM 40C, and executing it. Note that some or all of the functions may be realized by a dedicated hardware device.

[0053] The state acquisition unit 61 acquires the state s of the robot 10. t Get the.

[0054] The tentative command sequence generating unit 62 generates a plurality of tentative command sequences for the robot 10.

[0055] The specifying unit 63 inputs each command included in each tentative command sequence and a state corresponding to each command, and executes the switching model g, thereby specifying a dynamics model f to be applied for each command and a state corresponding to the command.

[0056] The predicted state sequence generating unit 64 generates a predicted state sequence for each tentative command sequence by using a dynamics model f specified corresponding to each command included in the tentative command sequence.

[0057] The calculation unit 65 calculates the reward for each predicted state sequence.

[0058] The predicted command sequence generator 66 generates a predicted command sequence that is predicted to maximize the reward.

[0059] Here, the interim command sequence generation unit 62 generates one interim command sequence, the predicted state sequence generation unit 64 generates a predicted state sequence corresponding to the interim command sequence generated by the interim command sequence generation unit 62, the calculation unit 65 calculates a reward for the predicted state sequence generated by the predicted state sequence generation unit 64, and the predicted command sequence generation unit 66 generates a command sequence predicted to maximize the reward by updating the interim command sequence one or more times to increase the reward by executing a series of operations of the interim command sequence generation unit 62, the identification unit 63, the predicted state sequence generation unit 64, and the calculation unit 65 multiple times.

[0060] In addition, the interim command sequence generation unit 62 may generate a plurality of interim command sequences collectively, the predicted state sequence generation unit 64 may generate a predicted state sequence from each of the plurality of interim command sequences, the calculation unit 65 may calculate a reward for each predicted state sequence, and the predicted command sequence generation unit 66 may generate a command sequence that is predicted to maximize the reward based on the reward for each predicted state sequence.

[0061] In this case, the interim command sequence generation unit 62 may repeatedly execute a series of processes from the process of generating a plurality of interim command sequences collectively to the process of calculating the reward, and in the second or subsequent series of processes, the interim command sequence generation unit 62 may select a plurality of interim command sequences corresponding to a predetermined higher reward from among the rewards calculated in the previous series of processes, and generate a plurality of new interim command sequences based on the distribution of the selected plurality of interim command sequences.

[0062] An output unit 67 outputs the first command included in the generated predicted command sequence.

[0063] The execution control unit 68 controls the operation of the robot 10 by repeating a series of operations of the state acquisition unit 61, the tentative command sequence generation unit 62, the identification unit 63, the predicted state sequence generation unit 64, the calculation unit 65, the predicted command sequence generation unit 66, and the output unit 67.

[0064] (Learning process)

[0065] FIG. 7 is a flowchart showing the flow of the learning process executed by the learning device 50 using machine learning.

[0066] In step S100, the learning device 50 performs preparation settings. Specifically, a target state of the robot 10 is set. In this embodiment, since the series of actions performed by the robot 10 is a juggling action, the target state is a state in which the ball BL is placed on a predetermined central portion of the back surface of the hand 12 when the hand 12 becomes horizontal after the ball BL is thrown upward and the hand 12 is turned over. The target state can be determined by the position and posture of the hand 12 and the relative position between the hand 12 and the ball BL.

[0067] If the series of actions executed by the robot 10 is to grip a peg and insert it into a hole, and the hand 12 is a gripper that grips the peg, the goal state is a state in which the peg is inserted into the hole. In this case, the goal state can be determined by the positions and orientations of the peg and the gripper.

[0068] Alternatively, an intermediate state leading to the goal state may be specified as the goal state. In this case, an intermediate goal that specifies the intermediate state, a part of the goal trajectory, a part of the goal path, a reward calculation method, and the like are set.

[0069] In addition, the structure of the dynamics model f may be given to a certain degree. In this embodiment, the dynamics model f is a model obtained by combining a model of the hand 12 and a model of the ball BL. As an example, the model of the hand 12 is a neural network, and the model of the ball BL is a linear function.

[0070] In step S101, the learning device 50 executes a trial motion and acquires a plurality of tuples. That is, the learning device 50 causes the robot 10 to execute the above-described juggling motion and acquires a plurality of tuples during the juggling motion. Specifically, in the state s t In command action a t The robot 10 is commanded to perform the commanded operation, and the state observation data observed by the state observation sensor 30 after the robot 10 performs the commanded operation is converted into the next state s t+1 Next, the next state s t+1 The state s t Command action a t The robot 10 is commanded to perform the commanded operation, and the state observation data observed by the state observation sensor 30 after the robot 10 performs the commanded operation is converted into the next state s t+1 By repeating this process, a trial juggling motion is performed, and multiple tuples during the juggling motion are obtained.

[0071] In step S102, the learning device 50 judges whether a predetermined learning end condition has been satisfied. Here, the learning end condition is a condition under which it can be determined that the robot 10 has mastered a series of actions, and can be, for example, a case where trial actions have been performed a specified number of times. Alternatively, the learning end condition may be a case where the number of times the target state has been reached, i.e., the number of times the trial actions have been successful, reaches a specified number. Alternatively, the learning end condition may be a case where the time it takes to reach the target state falls within a specified time. Alternatively, the learning end condition may be a case where the success rate of trial actions per certain number of times reaches a specified value or more.

[0072] If the learning end condition is met, this routine ends, and if the learning end condition is not met, the process proceeds to step S103.

[0073] In step S103, the learning device 50 adds the tuples acquired in step S101 to the main database. Note that the main database is a concept representing a storage area for storing the acquired tuples.

[0074] In step S104, the model generation process shown in FIG. 8 is executed.

[0075] As shown in FIG. 8, in step S200, learning device 50 initializes k, which represents the number of generated dynamics models f and serves as a label for identifying dynamics model f, by substituting "1" for k.

[0076] In step S201, the learning device 50 calculates n t is the lower bound of the number of tuples required to create one dynamics model f. low Then, we check whether n t n low If it is equal to or greater than this, the process proceeds to step S202. t n low If it is less than this, this routine ends and the process proceeds to step S105 in FIG.

[0077] In step S202, the learning device 50 moves all tuples in the main database to a work bin. Note that a work bin is a concept representing a storage area for storing tuples used to generate the dynamics model f.

[0078] In step S202, the number of tuples stored in the work bin, n f In t Substitute. Also, n t Initialize by substituting "0" for . Also, initialize counter c by substituting 0 for .

[0079] In step S203, the learning device 50 divides the value of the counter c by the divisor n ext MOD(c,n ext ) is n ext Check whether it is equal to -1. Then, MOD(c,n ext ) is n ext If it is equal to -1, proceed to step S204, and MOD(c,n ext ) is n extIf it is not equal to -1, the process proceeds to step S205. ext It is executed at a predetermined frequency depending on the divisor n ext is set in advance according to the frequency with which the process of step S204 is desired to be executed.

[0080] In step S204, the learning device 50 moves the m-th tuple in the main database to the work bin. Note that m≦n t m is set randomly. That is, the tuples moved from the main database to the work bin are selected randomly. The tuples in the main database are selected based on the maximum prediction error d max This is the tuple that generated the dynamics model f and was removed in the process of generating the dynamics model f. Therefore, in step S209, the maximum prediction error d max The tuple that generates the tuple is used to generate the dynamics model f at a predetermined frequency. This makes it possible to prevent the generated dynamics model f from becoming a locally optimal dynamics model f.

[0081] In step S205, the learning device 50 calculates the number of tuples stored in the work bin, n f is the lower bound of the number of tuples required to create one dynamics model f. low Then, determine whether it is less than n f n low If it is less than n, the dynamics model f cannot be created, so this routine is terminated and the process proceeds to step S105 in FIG. f n low In the above cases, the dynamics model f can be created, and the process proceeds to step S206.

[0082] In step S206, the learning device 50 generates a dynamics model f that matches the set of tuples stored in the work bin. In this embodiment, the dynamics model f is a linear function as an example, and is obtained using, for example, the least squares method. Note that the dynamics model f is not limited to a linear function. For example, the dynamics model f may be generated using other linear or nonlinear approximation methods such as a neural network, Gaussian Mixture Regression (GMR), Gaussian Process Regression (GPR), and support vector regression.

[0083] In step S206, the learning device 50 calculates the maximum prediction error d max First, we calculate the error d i (i=1, 2, ..., n f ) is calculated.

[0084] d i =||s t+1 - f(s t ,a t )|| 2

[0085] And the calculated error d i The largest error d i The maximum error d max Let us assume that.

[0086] In step S207, the learning device 50 calculates the maximum error d max is a predetermined threshold d up Then, determine whether the maximum error d max is the threshold d up If it is less than the maximum error d max is the threshold d up If so, the process proceeds to step S209.

[0087] In step S208, the learning device 50 converts the dynamics model f generated in step S207 into the k-th (k=1, 2, . . . ) dynamics model f k As described above, k is a label for identifying the dynamics model f.

[0088] In step S208, the learning device 50 moves all tuples stored in the work bin to the k-th sub-database. In other words, the label k is assigned to all stored tuples. Here, the sub-database is the sub-database of the generated dynamics model f k This is the concept of storage for storing matching tuples. Since the work bin is now empty, n f is initialized by substituting "0". Also, k is incremented. That is, k←k+1 is set. Then, the process proceeds to step S201.

[0089] In step S209, the learning device 50 calculates the maximum error d max Move the resulting tuple to the main database. This removes one tuple from the work bin, so n f Decrement n. f ←n f -1. Also, the number of tuples in the main database increases by one, so n f Increment n. t ←n t +1. Also, the counter c is incremented. That is, c←c+1. Then, the process proceeds to step S203.

[0090] In this way, a dynamics model f is generated from the tuples that remain in step S209, but the tuples used to generate this dynamics model f are moved to the kth sub-database and therefore cannot be used in the generation of the next (k+1)th dynamics model f.

[0091] When a negative determination is made in step S201 or when an affirmative determination is made in step S205, the process proceeds to step S105 in FIG.

[0092] In step S105 of FIG. 7, the learning device 50 moves all tuples previously acquired in step S101 to the main database.

[0093] In this way, the dynamics model f k Discard all previously acquired tuples and move them to the main database for retraining. This automatically generates multiple dynamics models f until the training termination condition is met.

[0094] Conventionally, if one were to train a single model to accurately predict all state transitions in a series of actions performed by the robot 10, a large number of trials would be required. However, according to this embodiment, a model that can be applied to the entire series of actions performed by the robot 10 can be learned with a small number of trials.

[0095] (Control process 1)

[0096] FIG. 9 is a flowchart showing the flow of control process 1 executed by the control device 60.

[0097] In step S300, a motion end condition for the robot 10 is set. The motion end condition is, for example, the state s t This is the case where the difference between the actual state and the target state is within a specified value.

[0098] The processes of steps S301 to S308 described below are executed at regular time intervals according to a control period, which is set to a period during which the processes of steps S301 to S308 can be executed.

[0099] In step S301, the control device 60 waits until a predetermined time equivalent to the length of the control cycle has elapsed since the start of the previous control cycle.

[0100] In step S302, the control device 60 determines the state s of the robot 10. t That is, state observation data of the robot 10 is obtained from the state observation sensor 30. Specifically, the state s t is, for example, the position of the robot 10 (hand 12) and the operation target object (ball BL). Note that the velocity is calculated from the past position and the current position.

[0101] In step S303, the control device 60 calculates the state s t It is determined whether or not the state s satisfies the operation end condition set in step S300. t If the operation end condition is satisfied, this routine is terminated. t If the operation end condition is not satisfied, the process proceeds to step S304.

[0102] In step S304, the control device 60 generates a tentative command sequence for the robot 10. In this embodiment, the number of time series steps is set to 3 (t, t+1, t+2), and the state s of the robot 10 measured in step S302 is t A tentative command sequence a corresponding to t ,a t+1 ,a t+2 The number of time series steps is not limited to 3 and can be set arbitrarily. In the first loop, a tentative command sequence a is generated randomly. t ,a t+1 ,a t+2 In the second and subsequent loops, a tentative command sequence a is generated using, for example, the Newton method so that the reward becomes larger. t ,a t+1 ,a t+2 Update.

[0103] In step S305, the control device 60 generates a predicted state sequence of the robot 10. That is, the tentative command sequence a generated in step S304 is t ,a t+1 ,a t+2 A predicted state sequence is generated using the dynamics model f specified corresponding to

[0104] Specifically, state s t , directive a t are input to multiple dynamics models f and switching model g, and the state s t , directive a t The dynamics model f k The next state s output from t+1 Note that the state s t , directive a t is input to the switching model g, and the state s t , directive a t The dynamics model f k Only in states t , directive a t Enter the next state s t+1 This also applies to the following processing.

[0105] Next, state s t+1 , directive a t+1 are input to multiple dynamics models f and switching model g, and the state s t+1 , directive a t+1 The dynamics model f k The next state s output from t+2 Get the.

[0106] Next, state s t+2 , directive a t+2 are input to multiple dynamics models f and switching model g, and the state s t+2 , directive a t+2 The dynamics model f k The next state s output from t+3 This gives the predicted state sequence s t+1 , s t+2 , s t+3 is obtained.

[0107] In step S306, the control device 60 calculates the predicted state sequence s t+1 , st+2 , s t+3 The reward corresponding to the amount is calculated using a predetermined formula.

[0108] In step S307, the control device 60 determines whether the reward calculated in step S306 satisfies a prescribed condition. Here, the prescribed condition is satisfied when, for example, the reward exceeds a prescribed value, or when the loop of the processing of steps S304 to S307 is executed a prescribed number of times. The prescribed number of times is set to, for example, 10 times, 100 times, 1000 times, etc.

[0109] If the remuneration satisfies the specified condition, the process proceeds to step S308, and if the remuneration does not satisfy the specified condition, the process proceeds to step S304.

[0110] In step S308, the control device 60 generates a predicted command sequence based on the reward corresponding to the predicted state sequence of the robot 10 calculated in step S307. The predicted command sequence may be the command sequence itself when the reward satisfies a specified condition, or may be a predicted command sequence predicted from a history of changes in reward corresponding to changes in the command sequence, which can maximize the reward. Then, the first command a of the generated predicted command sequence is t is output to the robot 10.

[0111] In this manner, the processes of steps S301 to S308 are repeated for each control period.

[0112] (Control process 2)

[0113] As another example of the control process, Fig. 10 is a flow chart showing the flow of a control process 2 executed by the control device 60. Note that the steps performing the same processes as those in Fig. 9 are given the same reference numerals, and detailed explanations will be omitted.

[0114] As shown in FIG. 10, the processes in steps S304A to S308A are different from those shown in FIG.

[0115] In step S304A, the control device 60 collectively generates a plurality of provisional command sequences for the robot 10. For generating the plurality of provisional command sequences, for example, a cross-entropy method (CEM) can be used, but is not limited to this.

[0116] In the case of CEM, the first loop generates multiple (e.g., 300) temporary command sequences a t ,a t+1 ,a t+2 From the second loop onwards, multiple (e.g. 30) command sequences corresponding to the highest rewards determined in advance are selected from the rewards calculated in the previous series of processes, and multiple new (e.g. 300) command sequences are generated according to the distribution (average, variance) of the selected command sequences.

[0117] In step S305A, the control device 60 generates a predicted state sequence for each command sequence generated in step S304 A. The process of generating a predicted state sequence corresponding to each command sequence is the same as the process in step S305 in FIG.

[0118] In step S306A, the control device 60 calculates a reward for each predicted state sequence generated in step S305A. The process of calculating the reward for each predicted state sequence is the same as the process in step S306.

[0119] In step S307A, the control device 60 determines whether or not the processes in steps S304A to S306A have been executed a predetermined number of times. The predetermined number of times can be, for example, 10 times, but can be set arbitrarily as long as it is equal to or greater than 1 time.

[0120] In step S308A, the control device 60 generates a predicted command sequence that is predicted to maximize the reward based on the reward for each predicted state sequence calculated in step S306A. For example, t ,a t+1 ,a t+2 and the command sequence a t ,a t+1 ,a t+2 The predicted state sequence s obtained fromt+1 , s t+2 , s t+3 A relational expression that represents the correspondence between the reward and the predicted command sequence a corresponding to the maximum reward on the curve represented by the calculated relational expression is calculated. t ,a t+1 ,a t+2 and its first command a t Output.

[0121] By the way, in step S302, state s t After the command is acquired, the command action a t It takes time to execute the process shown in FIG. 9 or FIG. 10 until the state s of the robot 10 is determined and output. t and state s t The command action a given to the robot 10 t The robot 10 performs the command action a t Next state s after t+1 This is a model whose output is s t and command action a t In theory, it is the value at the same time, but in reality, the state s t Use command action a t It is unavoidable that processing time is required to calculate the state s t The command action a given to the robot 10 t The expression "does not exclude the case where this processing time exists. In order to make the actual control operation closer to the theoretical operation, it is preferable that the length of the control period is sufficiently large relative to this processing time (for example, 10 times or more).

[0122] The above-described embodiment merely illustrates an exemplary configuration of the present invention. The present invention is not limited to the above-described specific embodiment, and various modifications are possible within the scope of the technical concept.

[0123] In the above example, the series of movements performed by the robot 10 is juggling movements, but the series of movements performed by the robot 10 may be any movements.

[0124] In addition, the learning process and control process executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. In this case, examples of the processor include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. In addition, the learning process and control process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). In addition, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0125] In addition, in each of the above embodiments, the learning program and the control program are described as being stored (installed) in advance in the storage 40D or the ROM 40B, but the present invention is not limited to this. The programs may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The programs may also be downloaded from an external device via a network. [Explanation of symbols]

[0126] 1. Robot System 10. Robot 11 Arm 12 Hands 20 Models 21 Model Selection Section g Switching model 30 Condition Observation Sensor 40 Learning and control devices 50 Learning Device 51 State transition data acquisition unit 52 Dynamics model generation section 53 Granting Department 54 Learning Department 60 Control device 61 Status acquisition unit 62 Provisional command sequence generation unit 63 Specific part 64 Predicted state sequence generation unit 65 Calculation section 66 Prediction command sequence generation unit 67 Output section 68 Execution Control Unit BL Ball f Dynamics model g Switching model

Claims

1. a state transition data acquisition unit that acquires a plurality of state transition data, the state transition data including a state of the control object, a command action commanded to the control object in the state, and a next state of the control object after the command action is performed, the state transition data being obtained by causing the control object to perform a series of predetermined actions; a dynamics model generation unit that generates a plurality of dynamics models having the state and the command action as input and the next state as output, each of the dynamics models being adapted to a set of state transition data consisting of a part of the acquired plurality of state transition data, and the plurality of dynamics models being adapted to mutually different sets of the state transition data; an assigning unit that assigns a label for identifying the dynamics model to the state transition data included in the set of state transition data that is compatible with the generated dynamics model; a learning unit that learns a switching model that identifies the dynamics model corresponding to the input state of the control object and the command operation from among a plurality of the dynamics models, using the state transition data to which the label is assigned as learning data; and A state acquisition unit that acquires a state of the control target; a provisional command sequence generating unit configured to generate a plurality of provisional command sequences for the control object; a determination unit that determines a dynamics model to be applied to each command and each state corresponding to the command by inputting each command included in each provisional command sequence and a state corresponding to each command and executing the switching model; a predicted state sequence generation unit that generates, for each of the tentative command sequences, a predicted state sequence by using the dynamics model identified in correspondence with each command included in the tentative command sequence; A calculation unit for calculating a reward for each predicted state sequence; a predicted command sequence generator for generating a predicted command sequence that is predicted to maximize the reward; an output unit that outputs a first command included in the generated predicted command sequence; an execution control unit that controls an operation of the control target by repeating a series of operations of the state acquisition unit, the tentative command sequence generation unit, the identification unit, the predicted state sequence generation unit, the calculation unit, the predicted command sequence generation unit, and the output unit; A learning and control device comprising:

2. The dynamics model generation unit, when generating one of the plurality of dynamics models, generates a provisional dynamics model using all of the state transition data usable for generating the dynamics model, and thereafter calculates an error between the next state obtained by inputting the state of the controlled object and the command action into the generated provisional dynamics model, and the next state included in the state transition data including the state and the command action, and generates the provisional dynamics model excluding the state transition data with the maximum error, thereby generating the dynamics model with the calculated error being equal to or less than a predetermined threshold value. The learning and control device according to claim 1 .

3. The dynamics model generating unit generates a next dynamics model by making the state transition data remaining in the process of generating the dynamics model unusable in the generation of the subsequent dynamics model every time the dynamics model generating unit generates one of the plurality of dynamics models. The learning and control device according to claim 2.

4. The dynamics model generating unit generates the dynamics model by restoring the state transition data randomly selected from the state transition data with the maximum error removed to the state transition data used for generating the dynamics model at a predetermined frequency. The learning and control device according to claim 2 or 3.

5. The interim command sequence generation unit generates one interim command sequence, the predicted state sequence generation unit generates the predicted state sequence corresponding to the interim command sequence generated by the interim command sequence generation unit, the calculation unit calculates a reward for the predicted state sequence generated by the predicted state sequence generation unit, and the predicted command sequence generation unit generates a predicted command sequence that is predicted to maximize the reward by updating the interim command sequence one or more times so as to increase the reward by executing a series of operations of the interim command sequence generation unit, the identification unit, the predicted state sequence generation unit, and the calculation unit multiple times. A learning and control device according to any one of claims 1 to 4.

6. The tentative command sequence generation unit collectively generates a plurality of tentative command sequences, the predicted state sequence generation unit generates the predicted state sequence from each of the plurality of tentative command sequences, the calculation unit calculates a reward for each predicted state sequence, and the predicted command sequence generation unit generates a predicted command sequence that is predicted to maximize the reward based on the reward for each predicted state sequence. A learning and control device according to any one of claims 1 to 4.

7. The provisional command sequence generation unit executes a series of processes from the process of collectively generating the plurality of provisional command sequences to the process of calculating the remuneration a plurality of times, and in the second or subsequent series of processes, the provisional command sequence generation unit selects a plurality of provisional command sequences corresponding to a predetermined higher remuneration among the remunerations calculated in the previous series of processes, and generates new provisional command sequences based on a distribution of the selected plurality of provisional command sequences. The learning and control device according to claim 6.

8. a state transition data acquisition unit that acquires a plurality of state transition data, the state transition data including a state of the control object, a command action commanded to the control object in the state, and a next state of the control object after the command action is performed, the state transition data being obtained by causing the control object to perform a series of predetermined actions; a dynamics model generating unit that generates a plurality of dynamics models having the state and the command action as input and the next state as output, each of the dynamics models being adapted to a set of state transition data consisting of a part of the acquired plurality of state transition data, and the plurality of dynamics models being adapted to sets of the state transition data different from each other; an assigning unit that assigns a label for identifying the dynamics model to the state transition data included in the set of state transition data that is compatible with the generated dynamics model; a learning unit that learns a switching model that identifies the dynamics model corresponding to the input state of the control object and the command operation from among a plurality of the dynamics models, using the state transition data to which the label is assigned as learning data; and A learning device equipped with

9. a state acquisition unit for acquiring a state of a control target; a provisional command sequence generating unit configured to generate a plurality of provisional command sequences for the control object; a specifying unit that executes a switching model learned by the learning device according to claim 8 by inputting each command included in each provisional command sequence and a state corresponding to each command, and specifies a dynamics model to be applied to each command and a state corresponding to the command among the dynamics models generated by the learning device; a predicted state sequence generation unit that generates, for each of the tentative command sequences, a predicted state sequence by using the dynamics model identified in correspondence with each command included in the tentative command sequence; A calculation unit for calculating a reward for each predicted state sequence; a predicted command sequence generator for generating a predicted command sequence that is predicted to maximize the reward; an output unit that outputs a first command included in the generated predicted command sequence; an execution control unit that controls an operation of the control target by repeating a series of operations of the state acquisition unit, the tentative command sequence generation unit, the identification unit, the predicted state sequence generation unit, the calculation unit, the predicted command sequence generation unit, and the output unit; A control device comprising:

10. The computer a state transition data acquisition step of acquiring a plurality of state transition data, the state transition data including a state of the control object, a command action commanded to the control object in the state, and a next state of the control object after the command action is performed, the state transition data being acquired by causing the control object to perform a series of predetermined actions; a dynamics model generating step of generating a plurality of dynamics models having the state and the command action as inputs and the next state as an output, each of the dynamics models being adapted to a set of state transition data consisting of a portion of the acquired plurality of state transition data, and the plurality of dynamics models being adapted to mutually different sets of the state transition data; an assignment step of assigning a label for identifying the dynamics model to the state transition data included in the set of state transition data that is compatible with the generated dynamics model; a learning step of learning a switching model that specifies the dynamics model corresponding to the input state of the control object and the command operation from among a plurality of the dynamics models, using the state transition data to which the label has been assigned as learning data; a state acquisition step of acquiring a state of the control object; a provisional command sequence generating step of generating a plurality of provisional command sequences for the control object; a step of inputting each command included in each provisional command sequence and a state corresponding to each command, and executing the switching model to specify a dynamics model to be applied to each command and each state corresponding to the command; a predicted state sequence generating step of generating, for each of the tentative command sequences, a predicted state sequence by using the dynamics model specified corresponding to each command included in the tentative command sequence; A calculation step of calculating a reward for each predicted state sequence; a predicted command sequence generating step of generating a predicted command sequence that is predicted to maximize the reward; an output step of outputting a first command included in the generated predicted command sequence; an execution control step of controlling an operation of the control target by repeating a series of operations including the state acquisition step, the tentative command sequence generation step, the identification step, the predicted state sequence generation step, the calculation step, the predicted command sequence generation step, and the output step; A learning and control method for performing a process including the steps of:

11. The computer a state transition data acquisition step of acquiring a plurality of state transition data, the state transition data including a state of the control object, a command action commanded to the control object in the state, and a next state of the control object after the command action is performed, the state transition data being acquired by causing the control object to perform a series of predetermined actions; a dynamics model generating step of generating a plurality of dynamics models having the state and the command action as inputs and the next state as an output, each of the dynamics models being adapted to a set of state transition data consisting of a portion of the acquired plurality of state transition data, and the plurality of dynamics models being adapted to mutually different sets of the state transition data; an assignment step of assigning a label for identifying the dynamics model to the state transition data included in the set of state transition data that is compatible with the generated dynamics model; a learning step of learning a switching model that specifies the dynamics model corresponding to the input state of the control object and the command operation from among a plurality of the dynamics models, using the state transition data to which the label has been assigned as learning data; A learning method that performs processing involving.

12. The computer A state acquisition step of acquiring a state of a control object; a provisional command sequence generating step of generating a plurality of provisional command sequences for the control object; a step of inputting each command included in each provisional command sequence and a state corresponding to each command, and executing a switching model trained by the learning method according to claim 11, thereby identifying a dynamics model to be applied to each command and each state corresponding to the command among the dynamics models generated by the learning method; a predicted state sequence generating step of generating, for each of the tentative command sequences, a predicted state sequence by using the dynamics model specified corresponding to each command included in the tentative command sequence; A calculation step of calculating a reward for each predicted state sequence; a predicted command sequence generating step of generating a predicted command sequence that is predicted to maximize the reward; an output step of outputting a first command included in the generated predicted command sequence; an execution control step of controlling an operation of the control target by repeating a series of operations including the state acquisition step, the tentative command sequence generation step, the identification step, the predicted state sequence generation step, the calculation step, the predicted command sequence generation step, and the output step; A control method for executing a process including the steps of:

13. On the computer, a state transition data acquisition step of acquiring a plurality of state transition data, the state transition data including a state of the control object, a command action commanded to the control object in the state, and a next state of the control object after the command action is performed, the state transition data being acquired by causing the control object to perform a series of predetermined actions; a dynamics model generating step of generating a plurality of dynamics models having the state and the command action as inputs and the next state as an output, each of the dynamics models being adapted to a set of state transition data consisting of a portion of the acquired plurality of state transition data, and the plurality of dynamics models being adapted to mutually different sets of the state transition data; an assignment step of assigning a label for identifying the dynamics model to the state transition data included in the set of state transition data that is compatible with the generated dynamics model; a learning step of learning a switching model that specifies the dynamics model corresponding to the input state of the control object and the command operation from among a plurality of the dynamics models, using the state transition data to which the label has been assigned as learning data; a state acquisition step of acquiring a state of the control object; a provisional command sequence generating step of generating a plurality of provisional command sequences for the control object; a step of inputting each command included in each provisional command sequence and a state corresponding to each command, and executing the switching model to specify a dynamics model to be applied to each command and each state corresponding to the command; a predicted state sequence generating step of generating, for each of the tentative command sequences, a predicted state sequence by using the dynamics model specified corresponding to each command included in the tentative command sequence; A calculation step of calculating a reward for each predicted state sequence; a predicted command sequence generating step of generating a predicted command sequence that is predicted to maximize the reward; an output step of outputting a first command included in the generated predicted command sequence; an execution control step of controlling an operation of the control target by repeating a series of operations including the state acquisition step, the tentative command sequence generation step, the identification step, the predicted state sequence generation step, the calculation step, the predicted command sequence generation step, and the output step; A learning and control program for executing a process including the steps of:

14. On the computer, a state transition data acquisition step of acquiring a plurality of state transition data, the state transition data including a state of the control object, a command action commanded to the control object in the state, and a next state of the control object after the command action is performed, the state transition data being acquired by causing the control object to perform a series of predetermined actions; a dynamics model generating step of generating a plurality of dynamics models having the state and the command action as inputs and the next state as an output, each of the dynamics models being adapted to a set of state transition data consisting of a portion of the acquired plurality of state transition data, and the plurality of dynamics models being adapted to mutually different sets of the state transition data; an assignment step of assigning a label for identifying the dynamics model to the state transition data included in the set of state transition data that is compatible with the generated dynamics model; a learning step of learning a switching model that specifies the dynamics model corresponding to the input state of the control object and the command operation from among a plurality of the dynamics models, using the state transition data to which the label has been assigned as learning data; A learning program that executes a process including the steps of:

15. On the computer, A state acquisition step of acquiring a state of a control object; a provisional command sequence generating step of generating a plurality of provisional command sequences for the control object; a step of inputting each command included in each provisional command sequence and a state corresponding to each command, and executing a switching model learned by the learning program according to claim 14, thereby identifying a dynamics model to be applied to each command and each state corresponding to the command among the dynamics models generated by the learning program; a predicted state sequence generating step of generating, for each of the tentative command sequences, a predicted state sequence by using the dynamics model specified corresponding to each command included in the tentative command sequence; A calculation step of calculating a reward for each predicted state sequence; a predicted command sequence generating step of generating a predicted command sequence that is predicted to maximize the reward; an output step of outputting a first command included in the generated predicted command sequence; an execution control step of controlling an operation of the control target by repeating a series of operations including the state acquisition step, the tentative command sequence generation step, the identification step, the predicted state sequence generation step, the calculation step, the predicted command sequence generation step, and the output step; A control program that causes the processing to be performed.

Citation Information

Patent Citations

  • Control device and control method

    JP2018124982A

  • Robot control device, robot system and robot control method

    JP2020104215A