Learning device, learning method, and program

Abstract

A learning device 1X comprises mainly an optimization problem calculation means 51X and an executable state set learning means 52X. On the basis of an abstract system model and a detailed system model pertaining to a system in which a robot implements operations, a controller pertaining to the robot, and target parameters pertaining to robot movements, the optimization problem calculation means 51X calculates a function value that becomes a solution to an optimization problem in which an evaluation function for evaluating the attainability of a target state is used. On the basis of the function value, the executable state set learning means 52X learns an executable state set of the robot movements to be executed by the controller.

Description

[Technical field]

[0001] The present disclosure relates to the technical fields of a learning device, a learning method, and a storage medium for learning about the movements of a robot. [Background technology]

[0002] A system has been proposed in which robots are controlled by providing skills that modularize the robot's movements when controlling the robots required to execute a task. For example, Patent Literature 1 discloses a technology in which robot skills that can be selected according to a task are defined as tuples in a system in which an articulated robot executes a given task, and parameters included in the tuples are updated by learning. In addition, Non-Patent Literature 1 discloses Level Set Estimation (LSE), which is an estimation method using Gaussian process regression based on the idea of ​​Bayesian optimization. In addition, Non-Patent Literature 2 discloses truncated variance reduction (TRUVAR), which is another method for estimating a level set function. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication WO2018 / 219943 [Non-patent literature]

[0004] [Non-Patent Document 1] A. Gotovos, N. Casati, G. Hitz, and A. Krause, “Active learning for level set estimation,” in Int. Joint. Conf. Art. Intel., 2013. [Non-Patent Document 2] Ilija Bogunovic, Jonathan Scarlett, Andreas Krause, and Volkan Cevher. Truncated variance reduction: A unified approach to Bayesian optimization and level-set estimation. In Advances in Neural Information Processing Systems (NIPS), pages 1507-1515, 2016. Summary of the Invention [Problem to be solved by the invention]

[0005] When modularizing robot movements as skills and planning the robot's movements using these skills, it is necessary to acquire the skills in advance and store them in a database. In this case, it is necessary to have information on the state in which the system can execute the skill.

[0006] In view of the above-mentioned problems, one object of the present disclosure is to provide a learning device, a learning method, and a storage medium that are capable of suitably executing learning regarding executable states of robot motion. [Means for solving the problem]

[0007] One aspect of the learning device is Abstract system model and detailed system model of a system in which a robot performs a task, a low-level controller that generates a control command for the robot; and a high-level controller that outputs a control parameter α for operating the low-level controller. A controller and an evaluation function for evaluating the reachability of a goal state based on goal parameters related to the robot's operation. g An optimization problem is set using The control parameter α The evaluation function g Function value of and An optimization problem calculation means for calculating a feasible state set learning means for learning a feasible state set of the robot's operation executed by the controller based on the function value; death, The optimization problem is expressed by the following equation for finding the control parameter α that minimizes the evaluation function g: TIFF0007647862000001.tif1996 g indicates an evaluation function that determines that a transition from a specified initial state to a goal state set is feasible if "g≦0" is satisfied, γ denotes a mapping from the states of the detailed system model to the states of the abstract system model, x(t) is the state of the real system x calculated based on the state equation in equation (2). 0 indicates the state x at the time t has elapsed from T indicates the skill execution time included in the target parameter information. It is a learning device.

[0008] One aspect of the learning method includes: The computer Abstract system model and detailed system model of a system in which a robot performs a task, a low-level controller that generates a control command for the robot; and a high-level controller that outputs a control parameter α for operating the low-level controller. A controller and an evaluation function for evaluating the reachability of a goal state based on goal parameters related to the robot's operation. g Set up an optimization problem using This is the solution to the optimization problem. The control parameter α The evaluation function g Function value of and Calculate A set of feasible states of the robot's motion to be executed by the controller is learned based on the function values. death , The optimization problem is expressed by the following equation for finding the control parameter α that minimizes the evaluation function g: TIFF0007647862000002.tif1996 g indicates an evaluation function that determines that a transition from a specified initial state to a goal state set is feasible if "g≦0" is satisfied, γ denotes a mapping from the states of the detailed system model to the states of the abstract system model, x(t) is the state of the real system x calculated based on the state equation in equation (2). 0 indicates the state x at the time t has elapsed since T indicates the skill execution time included in the target parameter information. It is a way of learning.

[0010] One aspect of the program is Abstract system model and detailed system model of a system in which a robot performs a task, a low-level controller that generates a control command for the robot; and a high-level controller that outputs a control parameter α for operating the low-level controller.A controller and an evaluation function for evaluating the reachability of a goal state based on goal parameters related to the robot's operation. g Set up an optimization problem using This is the solution to the optimization problem. The control parameter α The evaluation function g Function value of and Calculate A process for learning a feasible state set of the robot's motion to be executed by the controller based on the function value. 、 The optimization problem is expressed by the following equation for finding the control parameter α that minimizes the evaluation function g: TIFF0007647862000003.tif1996 g indicates an evaluation function that determines that a transition from a specified initial state to a goal state set is feasible if "g≦0" is satisfied, γ denotes a mapping from the states of the detailed system model to the states of the abstract system model, x(t) is the state of the real system x calculated based on the state equation in equation (2). 0 indicates the state x at the time t has elapsed since T indicates the skill execution time included in the target parameter information. It is a program. Effect of the Invention

[0011] This makes it possible to preferably execute learning regarding executable states of robot motion. [Brief description of the drawings]

[0012] [Figure 1] 1 shows a configuration of a robot control system in a first embodiment. [Diagram 2] (A) The hardware configuration of the learning device. (B) The hardware configuration of the robot controller. [Diagram 3] 3A is a diagram showing a robot (manipulator) that grasps an object and the object to be grasped in real space, and (B) is a diagram showing the state shown in FIG. 3A in abstract space. [Figure 4] FIG. 2 is a block diagram showing a control system for skill execution. [Diagram 5] 1 is an example of a functional block diagram of a learning device relating to updating of a skill database. [Figure 6] 4 is an example of a functional block of a skill learning unit. [Figure 7] 13 is an example of a flowchart showing a process of updating a skill database by a learning device. [Figure 8] 13 is an example of a functional block of a skill learning unit in the modified example. [Figure 9] 13 shows a schematic configuration of a learning device in a second embodiment. [Figure 10] 13 is an example of a flowchart executed by the learning device in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of a learning device, a learning method, and a storage medium will be described with reference to the drawings. For the sake of convenience, in this specification, a character with an arbitrary symbol "x" added to an arbitrary letter "A" is referred to as "A x " is expressed as ".

[0014] First Embodiment (1) System Configuration 1 shows the configuration of a robot control system 100 according to the first embodiment. The robot control system 100 mainly includes a learning device 1, a storage device 2, a robot controller 3, a measuring device 4, and a robot 5. The learning device 1 performs data communication with the storage device 2 via a communication network or by wireless or wired direct communication. The robot controller 3 also performs data communication with the storage device 2, the measuring device 4, and the robot 5 via a communication network or by wireless or wired direct communication.

[0015] The learning device 1 determines the motion of the robot 5 for executing a given task by self-supervised learning, and learns a set of states in which the motion can be executed. In this case, the learning device 1 learns skills in which specific motions of the robot 5 are modularized for each motion (including learning a set of states in which the skill can be executed). Then, the learning device 1 registers a tuple of information on the learned skill (also called a "skill tuple") in a skill database 24 stored in the storage device 2. The skill tuple includes various information required to execute the motion to be modularized. In this case, the learning device 1 generates a skill tuple based on detailed system model information 21, low-level controller information 22, and target parameter information 23 stored in the storage device 2.

[0016] The storage device 2 stores information referenced by the learning device 1 and the robot controller 3. The storage device 2 stores at least detailed system model information 21, low-level controller information 22, target parameter information 23, and a skill database 24. The storage device 2 may be an external storage device such as a hard disk connected to or built into the learning device 1 or the robot controller 3, a storage medium such as a flash memory, or a server device that performs data communication with the learning device 1 and the robot controller 3. The storage device 2 may be composed of multiple storage devices and may have the above-mentioned storage units in a distributed manner.

[0017] The detailed system model information 21 is information that represents a detailed model of the robot 5 and the operating environment in the actual system in which the robot 5 operates (also referred to as a "detailed system model"). The detailed system model information 21 may be a differential or difference equation that represents the detailed system model, or may be a physical simulator.

[0018] The low-level controller information 22 is information about a low-level controller that generates an input for controlling the operation of the actual robot 5 based on parameters output by the high-level controller. For example, when the high-level controller generates a trajectory of the robot 5, the low-level controller may generate a control input that follows the operation of the robot 5 according to the trajectory, or may perform servo control by PID (Proportional Integral Differential) based on parameters output by the high-level controller.

[0019] The target parameter information 23 is information that represents parameters related to states or conditions that should be satisfied for each skill to be learned. The target parameter information 23 includes target state information that represents a target state (for example, in the case of a grasping operation, information related to stable grasping conditions such as form closure and force closure is included), execution time information related to a limit on the execution time length (time limit), and general constraint condition information that represents other general constraint conditions (for example, conditions related to limits on the movable range of the robot 5, speed limits, input limits, etc.).

[0020] The skill database 24 is a database of skill tuples prepared for each skill. The skill tuple includes at least information on a high-level controller for executing the target skill, information on a low-level controller for executing the target skill, and information on a set of states (executable state set) in which the target skill can be executed. The executable state set is defined in an abstract space that abstracts an actual space. The executable state set can be expressed, for example, by using Gaussian process regression or an approximation function of a level set function estimated by a level set estimation method. In other words, whether or not the executable state set includes a certain state can be determined by whether or not the value (for example, the average value) of the Gaussian process regression for the certain state or the value of the approximation function for the certain state satisfies a constraint condition for determining feasibility.

[0021] After the learning process by the learning device 1, the robot controller 3 formulates an operation plan for the robot 5 based on the measurement signal supplied by the measuring device 4 and the skill database 24, etc., generates a control command (control input) for making the robot 5 execute the planned operation, and supplies the control command to the robot 5. In this case, the robot controller 3 converts a task to be executed by the robot 5 into a sequence for each time step (time interval) of the task that the robot 5 can accept. Then, the robot controller 3 controls the robot 5 based on the control command equivalent to an execution command of the generated sequence. The control command corresponds to a control input output by a low-level controller.

[0022] The measurement device 4 is one or more sensors, such as a camera, a range sensor, a sonar, or a combination of these, that detect the state in the workspace in which the robot 5 executes a task. The measurement device 4 supplies the generated measurement signal to the robot controller 3. The measurement device 4 may be a self-propelled or flying sensor (including a drone) that moves in the workspace. The measurement device 4 may also include a sensor provided on the robot 5 and a sensor provided on another object in the workspace. The measurement device 4 may also include a sensor that detects sound in the workspace. In this way, the measurement device 4 may include various sensors that detect the state in the workspace and are provided at any location.

[0023] The robot 5 performs work related to a specified task based on a control command supplied from the robot controller 3. The robot 5 is a robot that operates, for example, in various factories such as an assembly factory or a food factory, or at a logistics site. The robot 5 may be a vertical articulated robot, a horizontal articulated robot, or any other type of robot. The robot 5 may supply a status signal indicating the status of the robot 5 to the robot controller 3. This status signal may be an output signal of a sensor that detects the status (position, angle, etc.) of the entire robot 5 or a specific part such as a joint, or may be a signal indicating the progress of the operation of the robot 5.

[0024] 1 is an example, and various modifications may be made to the configuration. For example, the robot controller 3 and the robot 5 may be integrated. In another example, at least two of the learning device 1, the storage device 2, and the robot controller 3 may be integrated.

[0025] (2) Hardware Configuration 2(A) shows the hardware configuration of the learning device 1. The learning device 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 10.

[0026] The processor 11 executes a program stored in the memory 12 to function as a controller (arithmetic device) that performs overall control of the learning device 1. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0027] Memory 12 is composed of various types of volatile and non-volatile memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. Memory 12 also stores programs for executing the processes executed by learning device 1. Note that some of the information stored in memory 12 may be stored in one or more external storage devices (e.g., storage device 2) capable of communicating with learning device 1, or may be stored in a storage medium that is removable from learning device 1.

[0028] The interface 13 is an interface for electrically connecting the study device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like. For example, the interface 13 may perform interface operations with input devices that accept user input (external input) such as a touch panel, button, keyboard, or voice input device, display devices such as a display or projector, and sound output devices such as speakers.

[0029] The hardware configuration of the study device 1 is not limited to the configuration shown in Fig. 2(A). For example, the study device 1 may include at least one of a display device, an input device, and a sound output device. The study device 1 may also include a storage device 2.

[0030] 2(B) shows the hardware configuration of the robot controller 3. The robot controller 3 includes, as hardware, a processor 31, a memory 32, and an interface 33. The processor 31, the memory 32, and the interface 33 are connected via a data bus 30.

[0031] The processor 31 executes a program stored in the memory 32 to function as a controller (arithmetic device) that performs overall control of the robot controller 3. The processor 31 is, for example, a processor such as a CPU, a GPU, or a TPU. The processor 31 may be composed of multiple processors.

[0032] The memory 32 is composed of various types of volatile and non-volatile memories such as RAM, ROM, and flash memory. The memory 32 also stores programs for executing processes executed by the robot controller 3. Note that part of the information stored in the memory 32 may be stored in one or more external storage devices (e.g., the storage device 2) capable of communicating with the robot controller 3, or may be stored in a storage medium that is detachable from the robot controller 3.

[0033] The interface 33 is an interface for electrically connecting the robot controller 3 to other devices. These interfaces may be wireless interfaces such as a network adapter for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0034] The hardware configuration of the robot controller 3 is not limited to the configuration shown in Fig. 2(B). For example, the robot controller 3 may incorporate at least one of a display device, an input device, and a sound output device. The robot controller 3 may also include the storage device 2.

[0035] (3) abstract space The robot controller 3 formulates a motion plan for the robot 5 in an abstract space based on the skill tuple. The abstract space targeted in the motion plan for the robot 5 will now be described.

[0036] Fig. 3(A) is a diagram showing in real space a robot (manipulator) 5 that grasps an object, and an object to be grasped 6. Fig. 3(B) is a diagram showing in abstract space the state shown in Fig. 3(A).

[0037] In general, to formulate an operation plan for the robot 5 having a pick-and-place task, strict calculation is required taking into consideration the shape of the end effector of the robot 5, the geometric shape of the object 6 to be grasped, the grasping position and posture of the robot 5, and the object characteristics of the object 6 to be grasped. On the other hand, in this embodiment, the robot controller 3 formulates an operation plan in an abstract space in which the states of each object such as the robot 5 and the object 6 to be grasped are abstractly (simply) represented. In the example of FIG. 3(B), in the abstract space, an abstract model 5x corresponding to the end effector of the robot 5, an abstract model 6x corresponding to the object 6 to be grasped, and an area in which the robot 5 can grasp the object 6 to be grasped (see the dashed frame 60) are defined. In this way, the state of the robot in the abstract space is abstractly (simply) represented by the state of the end effector, etc. In addition, the state of each object corresponding to the operation object or the environmental object is also abstractly represented in a coordinate system or the like based on a reference object such as a work table.

[0038] The robot controller 3 in this embodiment uses the skills to formulate a motion plan in an abstract space that abstracts the actual system. This effectively reduces the computational cost required for motion planning even in a multi-stage task. In the example of Fig. 3(B), the robot controller 3 formulates a motion plan for executing the skills for performing grasping in a graspable area (dashed frame 60) defined in the abstract space, and generates a control command for the robot 5 based on the formulated motion plan.

[0039] Hereinafter, the state of the system in the real space will be denoted as "x", and the state of the system in the abstract space will be denoted as "x'". The state x' is represented as a vector (abstract state vector), and for example, in the case of a task such as pick-and-place, it includes a vector representing the state of the object to be operated (e.g., position, posture, speed, etc.), a vector representing the state of the end effector of the manipulable robot 5, and a vector representing the state of environmental objects. In this way, the state x' is defined as a state vector that abstractly represents the states of some elements in the real system.

[0040] (4)Skill execution control system FIG. 4 is a block diagram showing a control system for executing a skill. The processor 31 of the robot controller 3 functionally includes a motion planning unit 34, a high-level control unit 35, and a low-level control unit 36. A system 50 corresponds to an actual system. For convenience of explanation, FIG. 4 shows a balloon representing a diagram (see FIG. 3(B)) illustrating an abstract space targeted by the motion planning unit 34 in association with the motion planning unit 34, and a balloon representing a diagram (see FIG. 3(A)) illustrating an actual system corresponding to the system 50 in association with the system 50. Similarly, FIG. 4 shows a balloon representing information about an executable state set of a skill in association with the high-level control unit 35.

[0041] The motion planning unit 34 formulates a motion plan for the robot 5 based on the state x' in the abstract system and the skill database 24. In this case, the motion planning unit 34 expresses the goal state by, for example, a logical formula based on temporal logic. In this case, for example, the motion planning unit 34 may express the logical formula by using any temporal logic such as linear temporal logic, MTL (Metric Temporal Logic), or STL (Signal Temporal Logic). Then, the motion planning unit 34 converts the generated logical formula into a sequence (motion sequence) for each time step. This motion sequence includes, for example, information on the skills to be used in each time step.

[0042] The high-level control unit 35 recognizes the skill to be executed for each time step based on the motion sequence generated by the motion planning unit 34. Then, the high-level control unit 35 calculates the high-level controller “π H", the high-level control unit 35 generates a parameter "α" to be input to the low-level control unit 36. In this case, when the state "x0'" in the abstract space at the start of execution of the skill to be executed (also called the "initial state") belongs to the executable state set "χ0'" of the skill, the high-level control unit 35 generates a control parameter α as shown in the following formula (1). α=π H (x0') (1)

[0043] In addition, if an approximation function of the level set function capable of determining whether or not a skill belongs to the executable state set χ0′ is defined as “ĝ”, the robot controller 3 determines whether or not the state x0′ belongs to the executable state set χ0′ as follows: g^(x0')≦0 is satisfied. In other words, the condition can be said to represent a constraint condition for determining the feasibility of a certain state. Alternatively, the function "g^" can be said to be a model that can evaluate whether a given goal state can be reached from a certain initial state x0'. In this case, the goal state set, which is the set of goal states in the abstract space after the execution of the target skill, is called "χ' d ', the time length required to execute the target skill (execution time length) is 'T', and the state after T time has elapsed from the initial state x0' is 'x'(T)'. By using the low-level controller 36, x'(T)∈χ' d The approximation function g^ can be obtained by the learning device 1 through learning, as will be described later.

[0044] The low-level control unit 36 ​​generates the input "u" based on the control parameter α generated by the high-level control unit 35 and the current state x of the real system obtained from the system 50. In this case, the low-level control unit 36 ​​generates the input "u" based on the low-level controller "π L " Based on this, the input u is generated as a control command as shown in the following equation. u=π L (x, α) In addition, the low-level controller π Lis not limited to the above formula, but may be a controller having various forms.

[0045] The low-level control unit 36 ​​acquires the state of the robot 5 and the environment recognized by using any state recognition technology based on the measurement signal (which may include a signal from the robot 5) output by the measurement device 4 as the state x. In FIG. 4, the system 50 calculates a state equation "x" using a function "f" with an input u to the robot 5 and the state x as arguments. · =f(x,u)". · " represents the differentiation with respect to time or the difference with respect to time.

[0046] (5) Skills database update overview Fig. 5 is an example of a functional block of the learning device 1 related to updating the skill database. The processor 11 of the learning device 1 functionally has an abstract system model setting unit 14, a skill learning unit 15, and a skill tuple generating unit 16. Note that Fig. 5 shows an example of data exchanged between each block, but is not limited to this. The same applies to the diagrams of other functional blocks described later.

[0047] The abstract system model setting unit 14 sets an abstract system model based on the detailed system model information 21. This abstract system model is a simple model for operation planning, having as its state an abstract state vector x' configured based on the state x of the detailed system model (model corresponding to the system 50 in FIG. 4) specified by the detailed system model information 21. In this case, the abstract system model setting unit 14 calculates the abstract system model from the detailed system model based on an algorithm previously stored in the storage device 2 or the like, for example. Note that, when information on the abstract system model is previously stored in the storage device 2 or the like, the abstract system model setting unit 14 may acquire information on the abstract system model from the storage device 2 or the like. The abstract system model setting unit 14 supplies information on the set abstract system model to the skill learning unit 15 and the skill tuple generating unit 16, respectively.

[0048] The skill learning unit 15 performs learning on the skills to be generated based on the abstract system model set by the abstract system model setting unit 14, detailed system model information 21, low-level controller information 22, and target parameter information 23 (including target state information, execution time information, and general constraint condition information). In this case, the skill learning unit 15 performs learning on the skills to be generated based on the abstract system model set by the abstract system model setting unit 14, detailed system model information 21, low-level controller information 22, and target parameter information 23 (including target state information, execution time information, and general constraint condition information). L The feasible state set χ0' of the skill to be executed by the L A high-level controller π outputs the value of the control parameter α (a value that satisfies the criteria for suitability, for example, an optimal value). H The detailed process of the skill learning unit 15 will be described later.

[0049] The skill tuple generation unit 16 generates a set of feasible states χ0′ learned by the skill learning unit 15 and a high-level controller π H The skill tuple generation unit 16 generates a skill tuple including information about the abstract system model set by the abstract system model setting unit 14, low-level controller information 22, and target parameter information 23. The skill tuple generation unit 16 then registers the generated skill tuple in the skill database 24.

[0050] Here, each component of the abstract system model setting unit 14, the skill learning unit 15, and the skill tuple generating unit 16 can be realized, for example, by the processor 11 executing a program. Also, each component may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as necessary. At least a part of these components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Also, at least a part of these components may be realized by using an integrated circuit that is programmable by a user, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program composed of each of the above components. Also, at least a part of each component may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum computer control chip. In this way, each component may be realized by various hardware. The above is the same in other embodiments described later. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0051] (6) Details of the Skills Learning Department Next, the process executed by the skill learning unit 15 shown in FIG. 5 will be described in detail.

[0052] (6-1) Functional block 6 is an example of a functional block of the skill learning unit 15. The skill learning unit 15 functionally includes an optimal control problem calculation unit 51, a level set learning unit 52, a level set approximation unit 53, and a high-level controller learning unit .

[0053] The optimal control problem calculation unit 51 calculates a target state set χ' of a solution based on the detailed system model, the abstract system model, the low-level controller, and the target parameters, with the initial state in the abstract state being x0'. d In this case, the state after T time has elapsed from the initial state x0' is "x'(T)", and the goal state set χ' d is given as a set of states that satisfies g(x')≦0, the control problem calculation unit 51 sets an optimal control problem (optimization problem) that minimizes the evaluation function g(x'(T)). The optimal control problem calculation unit 51 then calculates the value of the control parameter α (a value that satisfies a condition for determining whether it is suitable as a solution, for example, an optimal value, hereinafter referred to as the "optimal control parameter α"), which is a solution obtained according to the optimal control problem. * ) and the function value g * Then, the optimal control problem calculation unit 51 calculates the function value g * "g * ≦0", the initial state x0' to the target state set χ' d It is determined that the transition to is feasible. Here, the initial state x0' is specified by the level set learning unit 52 and the high-level controller learning unit 54, as described later. Details of the processing of the optimal control problem calculation unit 51 will be further explained in the section "(6-2) Details of the optimal control problem calculation unit".

[0054] The level set learning unit 52 learns a level set function that represents a feasible state set χ0' in the abstract space of the low-level controller of the target skill. In this case, the level set learning unit 52 requests the optimal control problem calculation unit 51 to calculate the optimal control problem specifying the initial state x0' of the state x', and receives the specified initial state x0' and the function value g * In this case, the level set learning unit 52 learns the level set function "g GPAs an example of "," an approximation function of the level set function is identified. Details of the processing of the level set training unit 52 will be further explained in the section "(6-3) Details of the Level Set Training Unit."

[0055] The level set approximation unit 53 considers the calculation cost of the level set function in the motion plan and determines an approximation function g^ (also called a "level set approximation function g^") that simplifies the level set function by polynomial approximation or the like. In this case, the level set approximation unit 53 calculates an inner set of the level set approximation function g^(x0') of any form that satisfies the following equation and the level set function g GP The inner set of g GP The level set approximation function g^ is determined so that (x0')≦0" satisfies the following relationship. g GP (x0')≦g^(x0')≦0

[0056] The high-level controller learning unit 54 learns the initial state x0' belonging to the feasible state set χ0' specified by the approximation function g^ and the optimal control parameters α * Based on multiple pairs of α = π H (x0')" (see equation (1)) H The details of the processing by the high-level controller learning unit 54 will be further explained in the section "(6-4) Details of the High-Level Controller Learning Unit".

[0057] (6-2) Details of the optimal control problem calculation part The calculation of the optimal control problem by the optimal control problem calculation unit 51 will be specifically described. The optimal control problem calculation unit 51 calculates a target state set χ' in an abstract state according to the optimal control problem. d In this case, the goal state set χ' d is expressed by the following equation using the evaluation function g.

[0058]

number

[0059]

number

[0060]

number

[0061] In other words, the problem shown in equation (2) can be said to be a model that represents the relationship between the first state (x0'), the second state (X(T)), and the control parameter (α). * The process of calculating the function value g using a model (equation (2)) that includes constraints on state changes, etc. *However, the solution of the model shown in equation (2) does not have to be a mathematically optimal solution, but only needs to satisfy a criterion for determining whether it is a solution.

[0062] Here, for the optimal control problem shown in formula (2), when the function f representing the detailed system model is analytically obtained, the optimal control problem calculation unit 51 can solve the problem using any optimal control algorithm, such as the Direct Collocation method or Differential Dynamic Programming (DDP). When the function f representing the detailed system model is not analytically obtained (for example, when a physical simulator or the like is used), the optimal control problem calculation unit 51 can solve the optimal control problem shown in formula (2) using a model-free optimal control method, such as Path Integral Control. In this case, the optimal control problem calculation unit 51 finds the control parameter α according to the problem of minimizing the evaluation function g based on the function c representing the constraint condition.

[0063] Here, when generating a skill of a gripping operation in the pick-and-place task shown in FIG. 3(A), the target parameter information 23 and the low-level controller π L A specific example of the above will be described.

[0064] In this case, a physical simulator capable of acquiring information on the state x, the input u, and the contact force F that is the force for gripping the object to be gripped 6 is used for the detailed model. In addition, the target state information of the target parameter information 23 is information on stable gripping conditions such as form closure and force closure, and is expressed by the following equation. g(x,F)≦0

[0065] In addition, the execution time information of the target parameter information 23 includes the upper limit of the skill execution time T, "T max " (T≦T maxThe general constraint condition information of the target parameter information 23 includes information expressing the following constraint equation regarding the state x, the input u, and the contact force F. c(x,u,F)≦0 For example, this constraint is the upper limit of the contact force F, "F max " (F≦F max ), the limit of the range of motion (or speed) "x max ” (|x|≦x max ), the upper limit of input u is u max ” (|u|≦u max ) is a comprehensive formula that represents

[0066] In addition, the low-level controller π L is, for example, a servo controller based on PID. Here, the state of the robot 5 is expressed as "x r ” and the target trajectory for the state of robot 5 is “x rd " Then, input u and target trajectory x rd (Here, it is assumed to be a polynomial) is expressed by the following equation.

[0067]

number

[0068]

number

[0069] (6-3) Details of the level set learning part Next, learning by the level set learning unit 52 will be described. * Let g be a function that outputs * If (x0'), then the executable state set χ0' of the target skill is defined as follows:

[0070]

number

[0071] Note that the level set function may be calculated using TRUVAR, which is an estimation method using Gaussian process regression similar to the level set estimation method. TRUVAR is disclosed in Non-Patent Document 2.

[0072] As described above, the level set function may be any model that evaluates the initial state that can be reached for the desired state. The parameters in the model are the initial state x0′ and the function value g * By determining the model, it is possible to evaluate an initial state that can be reached for a certain desired state, and therefore it is possible to determine control parameters that can achieve the desired state for the system.

[0073] Furthermore, it may be possible to determine whether a desired state can be reached from a certain state according to the model, and if so, to control the robot so that it operates according to control parameters that represent the operation performed in the certain state.

[0074] (6-4) Details of high-level controller learning part Next, the high-level controller π H This article explains learning.

[0075] The high-level controller learning unit 54 learns the high-level controller π using any learning model used in machine learning. H In this case, the high-level controller learning unit 54 learns the initial state x0′ and the optimal control parameter α * Multiple pairs of (x 0i ',α * i ) (i=1 to N) as a learning sample to learn the learning model. In this case, the high-level controller learning unit 54 learns x 0i ' is the input data, * i The learning model is trained to output the training data. The learning model in this case may be any machine learning model such as a neural network, a Gaussian process regression, or a support vector regression.

[0076] In this case, the high-level controller learning unit 54 uses the set (x 0i ',α * i) is the initial state x0' that the level set learning unit 52 has specified to the optimal control problem calculation unit 51 in learning the level set function, and the optimal control parameter α * In this case, the high-level controller learning unit 54 uses the approximation function g^ supplied from the level set approximation unit 53 to select the initial state x0' and the optimal control parameter α that satisfy "g^(x0')≦0". * The combination of is selected as a learning sample.

[0077] In addition, in order to increase the number of learning samples, the high-level controller learning unit 54 instructs the optimal control problem calculation unit 51 to calculate the optimal control problem by specifying an initial state x0′ further selected from the feasible state set χ0′, and calculates the optimal control parameter α * may be obtained from the optimal control problem calculation unit 51. In this case, the high-level controller learning unit 54 learns the optimal control parameter α * The set of training samples is added to the high-level controller π H Learn about the following:

[0078] (7) Processing Flow 7 is an example of a flowchart showing a process of updating the skill database 24 by the learning device 1. The learning device 1 executes the process of the flowchart for each skill to be generated.

[0079] First, the abstract system model setting unit 14 of the learning device 1 sets an abstract system model based on the detailed system model information 21 (step S11). Next, the optimal control problem calculation unit 51 of the skill learning unit 15 sets an optimal control problem shown in equation (2) based on the detailed system model indicated by the detailed system model information 21, the abstract system model set in step S11, the low-level controller indicated by the low-level controller information 22, and the target parameters indicated by the target parameter information 23, and calculates a solution to the optimal control problem (step S12). In this case, the optimal control problem calculation unit 51 sets an optimal control problem for each of the initial states x0' specified by the level set learning unit 52 and the level set approximation unit 53, and calculates the function value g * and the optimal control parameter α * Calculate.

[0080] Next, the level set learning unit 52 of the skill learning unit 15 estimates the level set function of the feasible state set χ0' of the low-level controller of the target skill based on the solution of the optimal control problem calculated in step S12 (step S13). In this case, the level set learning unit 52 instructs the optimal control problem calculation unit 51 to calculate the optimal control problem with the initial state x0' specified, and obtains the function value g * and the specified initial state x0', the level set function g GP Calculate.

[0081] Next, the level set approximation unit 53 of the skill learning unit 15 calculates a level set approximation function g^ that approximates the level set function estimated in step S13 (step S14).

[0082] Then, the high-level controller learning unit 54 of the skill learning unit 15 learns the high-level controller π based on the state elements in the level set specified by the level set approximation function. H (Step S15). In this case, the initial state x0′ belonging to the feasible state set χ0′ specified by the approximation function ĝ and the optimal control parameter α * Based on multiple pairs of HLearn.

[0083] Next, the skill tuple generator 16 generates a set of the abstract system model, the high-level controller, the low-level controller, the target parameter information, and the level set approximation function as a skill tuple, and stores the generated skill tuple in the skill database 24 (step S16).

[0084] As described above, the learning device 1 preferably learns the feasible state set χ0′ of the low-level controller of the target skill, and also learns the feasible state set π0′ of the high-level controller π required for executing the target skill. H Thus, the learning device 1 can simultaneously obtain information on the controller for skill execution and the feasible area of ​​the skill, and can preferably build a skill database 24 that can be used for motion planning of the robot 5. The skill database 24 can then be preferably used when performing tasks that include complex movements such as assembly and tool use.

[0085] (8) Variations Next, modifications of the above-described embodiment will be described. The following modifications may be applied to the above-described embodiment in any combination.

[0086] (Variation 1) The level set approximation unit 53 may not be present in the functional block of the skill learning unit 15 shown in FIG.

[0087] Fig. 8 is an example of a functional block of the skill learning unit 15A in the modified example. In the example shown in Fig. 8, the skill learning unit 15A has an optimal control problem calculation unit 51, a level set learning unit 52, and a high-level controller learning unit 54. The level set learning unit 52 calculates a level set function g GP The high-level controller learning unit 54 outputs the level set function g output by the level set learning unit 52 to the skill tuple generation unit 16. GP Using g GPThe initial state x0' that satisfies (x0') ≤ 0 is identified as a learning sample, and the high-level controller π H Learn.

[0088] In this way, the learning device 1 uses the level set function g GP A skill tuple may be generated without calculating the approximation function g^.

[0089] (Variation 2) The learning device 1 calculates a function g representing a processing procedure for determining a feasible state set χ0' based on the processing of the optimal control problem calculation unit 51 and the level set learning unit 52 (and the level set approximation unit 53) when the parameters of the low-level controller are determined. GP (or g^) may be calculated. In this modification, there is no need for a high-level controller that determines the parameters of the low-level controller.

[0090] In this case, the optimal control problem calculation unit 51 sets an optimal control problem (optimization problem) for minimizing the evaluation function g based on information about the system model, the controller corresponding to the low-level controller, the target parameters, and the initial state x0′ specified by the level set learning unit 52, and calculates a function value g, which is a solution to the set optimal control problem. * Then, the level set learning unit 52 calculates the initial state x0′ and the function value g * Based on the set of GP Even in this case, the learning device 1 can preferably generate information about the feasible state set χ0′ and include it in the skill tuple to be registered in the skill database 24.

[0091] <Second embodiment> 9 shows a schematic configuration diagram of a learning device 1X in the second embodiment. The learning device 1X mainly includes an optimization problem calculation means 51X and a feasible state set learning means 52X. The learning device 1X may be composed of multiple devices.

[0092] The optimization problem calculation means 51X sets an optimization problem using an evaluation function that evaluates the reachability of a target state based on an abstract system model and a detailed system model related to the system in which the robot performs work, a controller related to the robot, and target parameters related to the robot's operation, and calculates a function value of the evaluation function that is a solution to the optimization problem. Here, the "controller" is, for example, the low-level controller in the first embodiment (including modified examples, the same applies below). Also, the "evaluation function that evaluates the reachability of a target state" is, for example, the evaluation function g in the first embodiment (including modified examples, the same applies below). The "function value" is, for example, the function value g in the first embodiment. * The optimization problem calculation means 51X can be, for example, the optimal control problem calculation unit 51 in the first embodiment.

[0093] The feasible state set learning means 52X learns a feasible state set of the robot's operation executed by the controller based on the function value. Here, the feasible state set may be learned as a function (for example, the level set function in the first embodiment). The feasible state set learning means 52X may be, for example, the level set learning unit 52 in the first embodiment.

[0094] 10 is an example of a flowchart in the second embodiment. The optimization problem calculation means 51X sets an optimal control problem using an evaluation function that evaluates the reachability of a target state based on an abstract system model and a detailed system model related to a system in which the robot performs work, a controller related to the robot, and target parameters related to the robot's operation (step S21). Then, the optimization problem calculation means 51X calculates a function value of the evaluation function that is a solution to the optimization problem (step S22). The feasible state set learning means 52X learns a feasible state set of the robot's operation executed by the controller based on the function value of the function that is a solution to the optimal control problem (step S23).

[0095] According to the second embodiment, in a case where the operation of a robot executed by a controller is modularized as a skill, the learning device 1X can suitably identify, through learning, a set of executable states of the skill.

[0096] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

[0097] [Appendix 1] an optimization problem calculation means for setting an optimization problem using an evaluation function for evaluating the reachability of a target state based on an abstract system model and a detailed system model related to a system in which a robot performs an operation, a controller related to the robot, and target parameters related to the operation of the robot, and for calculating a function value of the evaluation function that is a solution to the optimization problem; a feasible state set learning means for learning a feasible state set of the robot's operation executed by the controller based on the function value; A learning device having the above configuration. [Appendix 2] The learning device according to claim 1, wherein the feasible state set learning means learns a level set function representing the feasible state set based on a plurality of pairs of the function values ​​and initial states set in the optimization problem. [Appendix 3] 3. The learning device according to claim 2, further comprising a level set approximation means for calculating a level set approximation function that approximates the level set function. [Appendix 4] The learning device according to claim 2 or 3, wherein the feasible state set learning means specifies the initial state by sampling based on Gaussian process regression, and learns the level set function based on the initial state and the function value that is a solution to the optimization problem based on the specified initial state. [Appendix 5] The controller includes a low-level controller that generates a control command for the robot, and a high-level controller that outputs a control parameter for operating the low-level controller; the optimization problem calculation means calculates the control parameters and the function values ​​that are solutions to an optimal control problem set based on the abstract system model, the detailed system model, the low-level controller, and the target parameters; The learning device according to any one of appendix 1 to 4, further comprising a high-level controller learning means for learning the high-level controller based on states included in the learned feasible state set. [Appendix 6] The learning device described in Appendix 5, wherein the high-level controller learning means learns the high-level controller based on a pair of a state included in the feasible state set and the control parameter that becomes a solution to the optimal control problem when the state is set as an initial state in the optimal control problem. [Appendix 7] the evaluation function is a function for evaluating the reachability of a state in an abstract space, 7. The learning device according to claim 1, wherein the feasible state set learning means learns the feasible state set in the abstract space. [Appendix 8] The learning device according to any one of appendices 1 to 7, further comprising a skill tuple generation means for generating a skill tuple for an operation of the robot based on the learned feasible state set. [Appendix 9] The computer An optimization problem is set using an evaluation function that evaluates the reachability of a target state based on an abstract system model and a detailed system model related to a system in which the robot performs a task, a controller for the robot, and target parameters related to the operation of the robot; Calculating a function value of the evaluation function that is a solution to the optimization problem; learning a set of feasible states for a motion of the robot to be executed by the controller based on the function values; How to learn. [Appendix 10] An optimization problem is set using an evaluation function that evaluates the reachability of a target state based on an abstract system model and a detailed system model related to a system in which the robot performs a task, a controller for the robot, and target parameters related to the operation of the robot; Calculating a function value of the evaluation function that is a solution to the optimization problem; A storage medium storing a program for causing a computer to execute a process of learning a feasible state set for the robot's motion to be executed by the controller based on the function value. [Appendix 11] The computer determining, for a system in which a state is changed by a robot operating according to a control parameter, the control parameter from a first state to a second state using a first model that represents a relationship between a plurality of the states and the control parameter; determining a second model that estimates an initial state that can be reached for a desired state of the system based on the first state and the determined control parameters; How to learn.

[0098] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer readable media and can be supplied to a computer, such as a processor. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium can supply the program to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0099] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited documents (including patent documents and non-patent documents) are incorporated by reference into this specification. [Explanation of symbols]

[0100] 1. 1X learning device 2 Storage device 3 Robot Controller 4. Measurement equipment 5. Robot 100 Robot Control System

Claims

1. an abstract system model and a detailed system model relating to a system in which a robot performs an operation, a controller relating to the robot, the controller including a low-level controller for generating control commands for the robot and a high-level controller for outputting a control parameter α for operating the low-level controller, and an optimization problem calculation means for setting an optimization problem using an evaluation function g for evaluating the reachability of a target state based on target parameters relating to the operation of the robot, and calculating a function value of the control parameter α and the evaluation function g that is a solution to the optimization problem; and a feasible state set learning means for learning a feasible state set of the robot's operation executed by the controller based on the function value, The optimization problem is expressed by the following equation for finding the control parameter α that minimizes the evaluation function g: g indicates an evaluation function that determines that a transition from a specified initial state to a goal state set is feasible if "g≦0" is satisfied, γ denotes a mapping from the states of the detailed system model to the states of the abstract system model, x(t) represents the state x of the real system calculated based on the state equation of Equation (2) at the time when t time has elapsed from the state x 0 , T indicates the skill execution time length included in the target parameter information.

2. 2. The learning device according to claim 1, wherein the feasible state set learning means learns a level set function that represents the feasible state set based on a plurality of pairs of the function values ​​and initial states set in the optimization problem.

3. The learning device according to claim 2 , further comprising a level set approximation means for calculating a level set approximation function that approximates the level set function.

4. 4. The learning device according to claim 2 or 3, wherein the feasible state set learning means specifies the initial state by sampling based on Gaussian process regression, and learns the level set function based on the initial state and the function value that is a solution to the optimization problem based on the specified initial state.

5. A learning device as described in any one of claims 1 to 4, further having a high-level controller learning means for learning the high-level controller based on states included in the learned set of feasible states.

6. 6. The learning device according to claim 5, wherein the high-level controller learning means learns the high-level controller based on a pair of a state included in the feasible state set and the control parameter that becomes a solution to the optimization problem when the state is set as an initial state in the optimization problem.

7. the evaluation function is a function for evaluating the reachability of a state in an abstract space, 7. The learning device according to claim 1, wherein the executable state set learning means learns the executable state set in the abstract space.

8. The learning device according to any one of claims 1 to 7, further comprising a skill tuple generation means for generating a skill tuple for an operation of the robot based on the learned set of feasible states.

9. The computer An optimization problem is set using an evaluation function g that evaluates the reachability of a target state based on an abstract system model and a detailed system model related to a system in which a robot performs an operation, a controller related to the robot, the controller including a low-level controller that generates control commands for the robot and a high-level controller that outputs a control parameter α for operating the low-level controller, and target parameters related to the operation of the robot; Calculating the control parameter α and the evaluation function g, which are solutions to the optimization problem; learning a set of feasible states for a motion of the robot to be executed by the controller based on the function values; The optimization problem is expressed by the following equation for finding the control parameter α that minimizes the evaluation function g: g indicates an evaluation function that determines that a transition from a specified initial state to a goal state set is feasible if "g≦0" is satisfied, γ denotes a mapping from the states of the detailed system model to the states of the abstract system model, x(t) represents the state x of the real system calculated based on the state equation of Equation (2) at the time when t time has elapsed from the state x 0 , In this learning method, T indicates the skill execution time length included in the target parameter information.

10. An optimization problem is set using an evaluation function g that evaluates the reachability of a target state based on an abstract system model and a detailed system model related to a system in which a robot performs an operation, a controller related to the robot, the controller including a low-level controller that generates control commands for the robot and a high-level controller that outputs a control parameter α for operating the low-level controller, and target parameters related to the operation of the robot; Calculating the control parameter α and the evaluation function g, which are solutions to the optimization problem; causing a computer to execute a process of learning a set of feasible states for an operation of the robot to be executed by the controller based on the function value; The optimization problem is expressed by the following equation for finding the control parameter α that minimizes the evaluation function g: g indicates an evaluation function that determines that a transition from a specified initial state to a goal state set is feasible if "g≦0" is satisfied, γ denotes a mapping from the states of the detailed system model to the states of the abstract system model, x(t) represents the state x of the real system calculated based on the state equation of Equation (2) at the time when t time has elapsed from the state x 0 , T indicates a skill execution time length included in the target parameter information.

Images