Control apparatus, method, and program

Abstract

To enable a robot to execute task while suppressing the robot from contacting an obstacle in a hard state when the robot cannot acquire information on the environment in advance.SOLUTION: A control apparatus according to the present invention acquires position data of a control target portion provided on a robot. The control apparatus controls the control target portion so as to achieve a first mode in which the control target portion is set to have a degree of softness equal to or greater than a prescribed value if the control target portion has never visited the position indicated by the position data in the past or if the level at which the control target portion makes contact with an obstacle is equal to or greater than a prescribed level in the position data. Meanwhile, the control apparatus controls the control target portion so as to achieve a second mode in which the control target portion is set to have a degree of hardness equal to or greater than a prescribed value if the control target portion has visited the position indicated by the position data in the past and the level at which the control target portion makes contact with an obstacle is less than a prescribed level in the position data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a control device, a method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, robot systems in which the wrist portion of the robot is flexible are known (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Hamaya, M., Lee, R., Tanaka, K., von Drigalski, F., Nakashima, C., Shibata, Y., & Ijiri, Y. (2020, May). "Learning robotic assembly tasks with lower dimensional systems by leveraging physical softness and environmental constraints." In 2020 IEEE International Conference on Robotics and Automation (ICRA) (pp. 7747-7753). IEEE. Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where a robot must execute a task in a situation where the robot cannot obtain information about the environment in advance. For example, when executing a peg-in-hole task in which the robot grasps a peg and inserts the peg into a hole, there may be an obstacle on the path from the current position of the gripper of the robot to the position of the hole. In such a situation, if the robot cannot obtain information about the position of the obstacle in advance, the gripper of the robot may come into contact with the obstacle. In such a case, it is preferable that the gripper part of the robot is flexible. This is because if the gripper part is hard, the impact when the gripper comes into contact with the obstacle is large, and the impact may cause damage. On the other hand, if the gripper part of the robot is flexible throughout the entire process of the peg-in-hole task, a situation may occur in which vibration occurs when the gripper is moved or it becomes difficult to move the gripper accurately to a desired position.

[0005] The technology disclosed in the above-mentioned non-patent document 1 is a technology related to a flexible robot system, but does not take into consideration how the robot should behave when it is unable to obtain information regarding the position of an obstacle in advance.

[0006] The present disclosure has been made in consideration of the above points, and aims to perform a task while preventing the robot from coming into contact with an obstacle in a rigid state when the robot is unable to obtain information about the environment in advance. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the control device of the present disclosure is a control device comprising: an acquisition unit that acquires position data of a control target part equipped in a robot; and a control unit that controls the control target part to be in a first mode in which the degree of softness of the control target part is equal to or greater than a predetermined value if the control target part has not previously visited the position represented by the position data or if the degree of contact of the control target part with an obstacle in the position data is equal to or greater than a predetermined value, and controls the control target part to be in a second mode in which the degree of hardness of the control target part is equal to or greater than a predetermined value if the control target part has previously visited the position represented by the position data and the degree of contact of the control target part with an obstacle in the position data is less than a predetermined value.

[0008] In addition, the control method disclosed herein is a control method in which a computer executes a process to obtain position data of a control target part provided in a robot, and if the control target part has not previously visited the position represented by the position data or if the degree of contact of the control target part with an obstacle in the position data is greater than or equal to a predetermined value, control the control target part to a first mode in which the degree of softness of the control target part is greater than or equal to a predetermined value, and if the control target part has previously visited the position represented by the position data and the degree of contact of the control target part with an obstacle in the position data is less than a predetermined value, control the control target part to a second mode in which the degree of hardness of the control target part is greater than or equal to a predetermined value.

[0009] In addition, the control program disclosed herein is a control program for causing a computer to execute a process of acquiring position data of a control target part provided in a robot, and controlling the control target part to be in a first mode in which the degree of softness of the control target part is equal to or greater than a predetermined value if the control target part has never visited the position represented by the position data in the past or if the degree of contact of the control target part with an obstacle in the position data is equal to or greater than a predetermined value, and controlling the control target part to be in a second mode in which the degree of hardness of the control target part is equal to or greater than a predetermined value if the control target part has visited the position represented by the position data in the past and the degree of contact of the control target part with an obstacle in the position data is less than a predetermined value. Effect of the Invention

[0010] According to the control device, method, and program disclosed herein, when a robot is unable to obtain information about the environment in advance, the robot can perform a task while preventing the robot from coming into contact with an obstacle in a rigid state. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram for explaining a control system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a control target part according to the embodiment. [Diagram 3] 4A to 4C are diagrams for explaining the operation of a control target part according to the embodiment. [Figure 4] FIG. 2 is a diagram for explaining control of a control target part. [Diagram 5] FIG. 2 is a diagram for explaining control of a control target part. [Figure 6] FIG. 1 is a diagram for explaining a Probabilistic Ensemble Model. [Figure 7] FIG. 1 is a block diagram showing a schematic configuration of a control system according to an embodiment. [Figure 8] FIG. 2 is a block diagram showing a hardware configuration of a control device according to the embodiment. [Figure 9] FIG. 1 is a diagram for explaining a dynamics model. [Figure 10] 4 is a flowchart showing a flow of a control process in the embodiment. [Figure 11] 4 is a flowchart showing a flow of a control process in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] An example of an embodiment of the present disclosure will be described below with reference to the drawings. In this embodiment, a control system equipped with a control device according to the present disclosure will be described as an example. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensions and ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0013] Fig. 1 is a diagram for explaining a control system 10 of an embodiment. As shown in Fig. 1, the control system 10 includes a robot 12, a control device 14, and a control target portion 16. A peg T is attached to the control target portion 16. The control device 14 controls the operation of the robot 12 to execute a peg-in-hole task of inserting the peg T into a hole H. Although not shown in Fig. 1, it is assumed that an obstacle exists in the actual environment when the peg-in-hole task is executed.

[0014] 2 is a diagram showing an example of the controlled part 16 of this embodiment. As shown in FIG. 2, the controlled part 16 includes an arm 16A, a gripper 16B, and a spring 16C which is an example of an elastic body connecting the arm 16A and the gripper 16B. In this embodiment, the position data representing the position of the controlled part 16 is the position p of the arm 16A in FIG. arm is adopted as the position of the control target part 16.

[0015] Fig. 3 is a diagram for explaining the operation of the controlled part 16 in this embodiment. In the state shown by X in Fig. 3, the arm 16A and the gripper 16B of the controlled part 16 are connected by the spring 16C, and the degree of flexibility of the controlled part 16 is high, so that even if the controlled part 16 comes into contact with an obstacle, the impact is small. For example, as shown by Z in Fig. 3, when the controlled part 16 is in the first mode, the controlled part 16 is in a soft state, so that even if the peg T comes into contact with the floor surface FL, which is an example of an obstacle, the impact is small.

[0016] On the other hand, as shown in Y of Fig. 3, the controlled part 16 is configured to be able to control the arm 16A and the gripper 16B to be integrated. For example, it is possible to realize the state shown in Y of Fig. 3 based on a known mechanism not shown. The state shown in Y of Fig. 3 is a state in which the arm 16A and the gripper 16B of the controlled part 16 are connected to be integrated, and the degree of stiffness of the controlled part 16 is high, and if the controlled part 16 comes into contact with an obstacle, the impact is large.

[0017] In this embodiment, the state of the controlled part 16 as shown in X in Fig. 3 is referred to as a first mode. On the other hand, in this embodiment, the state of the controlled part 16 as shown in Y in Fig. 3 is referred to as a second mode.

[0018] In the present embodiment, when controlling the operation of the robot 12 when executing a task, the control device 14 moves the controlled part 16 of the robot 12 while switching between the first mode and the second mode. Specifically, the control device 14 controls the controlled part 16 so that the controlled part 16 is in the first mode at a position where the controlled part 16 is likely to contact an obstacle. On the other hand, the control device 14 controls the controlled part 16 so that the controlled part 16 is in the second mode at a position where the controlled part 16 is unlikely to contact an obstacle. As shown by X in FIG. 3, when the controlled part 16 is not in contact with an obstacle, the length l1 of the spring 16C is almost the same as the natural length. On the other hand, as shown by Z in FIG. 3, when the controlled part 16 is in contact with an obstacle, the length l2 of the spring 16C is shorter than the natural length. In the present embodiment, this principle is utilized to control the switching between the first mode and the second mode.

[0019] Figures 4 and 5 are diagrams for explaining the control of the control target part 16. The AR shown in Figures 4 and 5 is an area representing a place where the control target part 16 has visited in the past and where the control target part 16 is unlikely to come into contact with an obstacle.

[0020] As shown in Fig. 4, the control device 14 controls the control target part 16 to be in the second mode within the area AR. By setting the control target part 16 to the second mode, it is possible to move the control target part 16 quickly, for example. On the other hand, as shown in Fig. 5, the control device 14 controls the control target part 16 to be in the first mode outside the area AR.

[0021] As a result, even if the robot 12 cannot obtain information about the environment in advance, the robot 12 can execute the peg-in-hole task while preventing the control target portion 16 of the robot 12 from coming into contact with an obstacle in a hard state. A specific description will be given below.

[0022] In this embodiment, the motion of the robot 12 is controlled using a known Probabilistic Ensembles with Trajectory Sampling (PETS) algorithm. The PETS algorithm is based on model-based reinforcement learning, a Probabilistic Ensemble Model (PE), and a Trajectory Sampling Strategy (TS∞). The PETS algorithm is disclosed in the following references:

[0023] References: Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018). "Deep reinforcement learning in a handful of trials using probabilistic dynamics models.", Advances in neural information processing systems, 31.

[0024] [Probabilistic Ensemble Model] A known method, the Probabilistic Ensemble Model (PE), is a model that predicts the state at the next time point when the current state and the current action are given. For example, the Probabilistic Ensemble Model is realized by an ensemble of multiple neural network models. In this case, each of the multiple neural network models is trained independently. Each of the multiple neural network models is trained based on the action data a obtained by the controlled object portion 16 of the robot 12 visiting various positions. t and state data s t and state data s t+1 Supervised machine learning is performed using the combination of these as training data.

[0025] In addition, the Probabilistic Ensemble Model uses the reward r t+1In this case, for example, the Probabilistic Ensemble Model uses the behavioral data a t and state data s t Given state data s t+1 and reward r t+1 The probability distribution output from the Probabilistic Ensemble Model is expressed by the following equation (1).

[0026]

number

[0027] [Trajectory Sampling Strategy:TS∞] In the Trajectory Sampling Strategy, which is a known method, a trajectory of state data at each time is generated. Specifically, in the Trajectory Sampling Strategy, a predicted value of state data s at each time is obtained by the above-mentioned Probabilistic Ensemble Model. In this embodiment, the state data s at time t+1 is t+1 is the behavioral data at time t t This is because the state transitions to the state at time t+1 depending on the action at time t.

[0028] FIG. 6 is a diagram for explaining the Trajectory Sampling Strategy. The Probabilistic Ensemble Model of this embodiment is composed of a plurality of neural network models. For example, as shown in FIG. 6, the Probabilistic Ensemble Model is composed of first to n-th neural network models, which are n neural network models. In this case, each of the n neural network models has a different initial value of a weight parameter, but is trained using the same teacher data. For this reason, each of the predicted values ​​output from each of the first to n-th neural network models is slightly different. However, by ensembling the first to n-th neural network models (for example, averaging the predicted values), it is possible to obtain a highly accurate predicted value.

[0029] In the Trajectory Sampling Strategy, as shown in FIG. 6, a series of predicted values ​​output from each of the first to n-th neural network models is generated. Specifically, for the first neural network model, behavioral data a t and state data s at time t t When input is made, the state data s at time t+1 is t+1,1 Then, the predicted value of the behavioral data a at time t+1 is output to the first neural network model. t+1 and state data s at time t+1 t+1,1 When the predicted value of and are input, the state data s at time t+2 t+2,1 The predicted value of is output for the first neural network model. t+2 and state data s at time t+2 t+2,1 When the predicted value of and the state data s at time t+3 are input, t+3,1 In this manner, the predicted values ​​of the state data s from time t+1 to time t+H are obtained.

[0030] Moreover, similarly to the first neural network model, the second neural network model is used to obtain predicted values ​​of the state data s from time t+1 to time t+H.

[0031] In this manner, predicted values ​​of state data s from time t+1 to time t+H are output from each of the first to n-th neural network models. The state data at each time is estimated by ensembling the trajectories of the predicted values ​​output from each of the first to n-th neural network models.

[0032] [PETS algorithm] The PETS algorithm is an algorithm based on the above-mentioned Probabilistic Ensemble Model and Trajectory Sampling Strategy. In the PETS algorithm, the following processes (1) to (5) are executed.

[0033] (1) Based on training data, which is a combination of state data and action data, multiple neural network models that constitute a Probabilistic Ensemble Model are trained. (2) Using the Trajectory Sampling Strategy, a series of predicted values ​​of behavioral data from time t+1 to time t+H is generated. (3) From the sequence of predicted values ​​of the behavioral data from time t+1 to time t+H, select the sequence of behavioral data that maximizes the sum of rewards. (4) Controlling the robot 12 to execute the selected sequence of behavioral data. (5) Based on new state data obtained by executing the sequence of behavioral data, multiple neural network models constituting the Probabilistic Ensemble Model are updated.

[0034] [Proposed method] In this embodiment, the movement of the robot 12 is controlled based on the PETS algorithm. conti ∈RD is defined. Continuous state data s conti is a state vector representing the state of the robot 12 and is represented by a continuous value. D represents the dimension of the vector. Continuous state data s conti is a state vector having, for example, the velocity data of the arm 16A as an element. conti Based on this, the length l(s conti ) and the position data p of arm 16A arm (s conti ) can be calculated.

[0035] In this embodiment, the continuous action data a conti ∈R F is defined. Continuous action data a conti is a state vector representing the behavior of the robot 12 and is represented by a continuous value. F represents the dimension of the vector. conti contains velocity data of the controlled object part 16 of the robot 12 as a component.

[0036] In this embodiment, the mode of the controlled part 16 is switched to the first mode or the second mode. In this embodiment, the above-mentioned Probabilistic Ensemble Model is prepared for each mode. The first model, which is the Probabilistic Ensemble Model for the first mode, is expressed by the following formula (2). On the other hand, the second model, which is the Probabilistic Ensemble Model for the second mode, is expressed by the following formula (3).

[0037]

number

[0038] The mode of the controlled part 16 is represented by discrete value state data sdisc ∈{0,1}, where 0 represents the first mode and 1 represents the second mode. conti ∈{0,1} represents the behavior of switching modes. Therefore, the following formula (4) holds. The following formula (4) represents that the behavior at time t becomes the state at time t+1.

[0039]

number

[0040] In this embodiment, a known model-based reinforcement learning is executed so that the robot 12 executes the peg-in-hole task. For this purpose, a reward function is designed in advance so that the reward is maximized when the peg T is inserted into the hole H, and a policy is learned so that this reward function is maximized. In this embodiment, the above-mentioned PETS algorithm and a known Model Predictive Control (MPC) are used. However, the following two points must be considered.

[0041] (1) Existing model-based reinforcement learning cannot handle dynamics that mix discrete and continuous inputs. (2) In reinforcement learning, the robot 12 may take dangerous actions, but it is necessary to stop such dangerous actions.

[0042] To address the first point above, this embodiment proposes a PETS algorithm that can handle both discrete and continuous inputs. To address the second point above, this embodiment introduces a blocker.

[0043] [Hybrid PETS algorithm] As described above, the conventional PETS algorithm cannot simultaneously handle the speed data of the arm 16A, which is a continuous input, and the switching between the first mode and the second mode, which is a discrete input. Therefore, in this embodiment, a PETS algorithm is proposed that can handle hybrid actions according to continuous values ​​and discrete values.

[0044] First, let the prediction horizon be H. Let a sequence of discrete behavioral data a, which represents a switching command from time t to time t+H-1, be disc t:t+H-1 Consider the case where is fixed. In this case, the dynamics of the continuous value state and the continuous value behavior at time t' are expressed by the following equation (5).

[0045]

number

[0046] In the above formula (5), f soft represents the first model, which is the dynamics model for the first mode, and f rigid represents the second model, which is the dynamics model for the second mode. disc t’ Depending on whether t is 0 or 1, either the output of the first model or the output of the second model is adopted.

[0047] As mentioned above, the sequence of discrete behavioral data a disc t:t+H-1 When is fixed, the discrete state data sequence s disc t:t+H-1 is determined in the same way. Therefore, the sequence of discrete behavior data a disc t:t+H-1 and a sequence of discrete state data s disc t:t+H-1 When is fixed, the continuous action data a conti and continuous state data s contiSince we only need to deal with the sequence of all possible discrete action data a, we can use the same framework as the conventional PETS algorithm. disc t:t+H-1 For the optimal continuous action data a conti t:t+H-1 By planning a series of discrete behavior data and selecting the best series of discrete behavior data from among them, it is possible to obtain optimal series of discrete behavior data and series of continuous behavior data.

[0048] However, when calculating optimal continuous behavior data sequences for all possible discrete behavior data sequences, enormous computational costs may be incurred.

[0049] Therefore, in this embodiment, an assumption is made that mode switching occurs only once while reinforcement learning is being performed, and a sequence of discrete behavior data is set. Specifically, the sequence of discrete behavior data is set fixedly as shown in the following formula (6). In the following formula (6), the sequence A of discrete behavior data to be searched is d t In addition, as shown in the following equation (6), the discrete behavior data a is t From 1-s t After the time when it switches to, a is 1-s t It becomes.

[0050]

number

[0051] As shown in the above formula (6), the sequence of possible discrete action data is fixed, and then the continuous action data is optimized using the conventional PETS algorithm. Specifically, the continuous action data is optimized so as to maximize the reward in reinforcement learning.

[0052] [blocker] As described above, in this embodiment, the hybrid PETS algorithm is introduced. However, when the hybrid PETS algorithm is directly applied, there is a possibility that the robot 12 will select a dangerous action at a stage where learning has not been sufficiently performed. Therefore, in this embodiment, a blocker is introduced. The blocker is also disclosed in the following reference document.

[0053] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, Yuval Tassa, "Safe Exploration in Continuous Action Spaces", 2018

[0054] In this embodiment, a blocker is used to suppress dangerous behavior of the robot 12. When the blocker detects a dangerous behavior of the robot 12, it changes the behavior of the robot 12. In this embodiment, a penalty is added to the reward when the robot 12 performs a dangerous behavior in model-based reinforcement learning, which will be described later.

[0055] In this embodiment, a dangerous behavior is defined as a case where the control target part 16 comes into contact with an obstacle while the control target part 16 is in the second mode.

[0056] Specifically, the blocker of this embodiment first calculates the position data p arm Based on this, the length of the spring 16C of the controlled part 16 is predicted. In this case, the length of the spring 16C is predicted assuming that the controlled part 16 is in the first mode.

[0057] When the difference between the predicted value of the length of the spring 16C and the natural length of the spring 16C is equal to or less than a predetermined threshold (hereinafter simply referred to as the "threshold"), the peg T or the controlled part 16 is unlikely to be in contact with an obstacle. Therefore, when the difference between the predicted value of the length of the spring 16C and the natural length of the spring 16C is equal to or less than the threshold, the controlled part 16 remains in the second mode. For example, the threshold is set in advance by the user.

[0058] On the other hand, if the difference between the predicted value of the length of the spring 16C and the natural length of the spring 16C is greater than the threshold, it is highly likely that the peg T or the controlled part 16 is in contact with an obstacle. Therefore, if the difference between the predicted value of the length of the spring 16C and the natural length of the spring 16C is greater than the threshold, the controlled part 16 is switched to the first mode.

[0059] In this embodiment, when predicting the length of the spring 16C, a plurality of neural network models constituting the above-mentioned Probabilistic Ensemble Model are used. Each of the plurality of neural network models is based on the position data p arm When this is input, it outputs a predicted value for the length of the spring 16C.

[0060] In this embodiment, the position data p of the arm 16A of the controlled object 16 obtained when the controlled object 16 is in the first mode is arm and the correct data 1 for the length of the spring 16C are acquired as training data. In this embodiment, a plurality of neural network models constituting a Probabilistic Ensemble Model are trained based on this training data.

[0061] The blocker is the average of the predicted values ​​output from the multiple neural network models that make up the Probabilistic Ensemble Model. pred and the deviation of the ensemble σ ale and the deviation of the prediction model σ epi Calculate the deviation of the ensemble σ aleand the deviation of the prediction model σ epi is the deviation of the outputs of multiple neural network models. The deviation of the ensemble σ ale is a value that indicates the uncertainty due to noise. The deviation of the prediction model σ epi is a value indicating the uncertainty due to a lack of data. Then, the blocker judges whether the current position of the arm 16A is dangerous or not according to the following equation (7).

[0062]

number

[0063] In the above formula (7), the average predicted length of spring 16C, l pred is the natural length l natural The average predicted length of spring 16C is close to l pred is the natural length l natural and the difference between is less than or equal to a threshold, and the deviation of the ensemble σ ale and the deviation of the prediction model σ epi If the average value l of the predicted length of the spring 16C is less than or equal to the threshold value ε, the position of the arm 16A of the robot 12 is determined to be safe. pred is the natural length l natural The difference between the ensemble deviation σ ale is greater than the threshold, or the deviation of the prediction model σ epi If is greater than the threshold ε, the position of the arm 16A of the robot 12 is determined to be dangerous. Based on this determination result, the discrete behavior data a disc Specifically, the discrete action data a disc is modified according to the following equation (8):

[0064]

number

[0065] In the initial stage of the search by the robot 12 moving the arm 16A, the training data {p arm , l} is small, so the deviation of the ensemble σ ale becomes large and most positions are determined to be dangerous. Therefore, in the initial stage of the search, the control target part 16 of the robot 12 is set to the first mode in most positions.

[0066] On the other hand, as the search progresses, the training data {p arm , l} increases, the deviation of the ensemble σ ale becomes smaller, and the average predicted length of spring 16C, l pred is the natural length l natural A position is determined to be safe when the difference between the position and the target position is equal to or smaller than the threshold value. Therefore, as the search progresses, the controlled part 16 of the robot 12 is often set to the second mode.

[0067] In the hybrid PETS algorithm of this embodiment, the behavior is modified by a blocker and the reward is modified by a penalty. Specifically, as described above, when the robot 12 performs a dangerous behavior, a penalty is added (for example, the reward in the model-based reinforcement learning is reduced), thereby executing the model-based reinforcement learning.

[0068] [Control System 10] Fig. 7 is a block diagram showing a schematic configuration of the control system 10 of this embodiment. As shown in Fig. 7, the control system 10 includes a sensor group 11, a robot 12, a control device 14, and a control target part 16 installed on the robot 12.

[0069] The sensor group 11 sequentially detects the state of the robot 12. For example, the sensor group 11 detects the position data p arm are detected sequentially.

[0070] The robot 12 operates in response to control commands output from a control device 14, which will be described later. A control target part 16 is a part that the robot 12 is provided with, including an arm 16A and a gripper 16B.

[0071] Fig. 8 is a block diagram showing the hardware configuration of the control device 14 according to this embodiment. As shown in Fig. 8, the control device 14 has a CPU (Central Processing Unit) 42, a memory 44, a storage device 46, an input / output I / F (Interface) 48, a storage medium reading device 50, and a communication I / F 52. Each component is connected to each other via a bus 54 so as to be able to communicate with each other.

[0072] The storage device 46 stores control programs for executing each process described below. The CPU 42 is a central processing unit, and executes various programs and controls each component. That is, the CPU 42 reads the programs from the storage device 46 and executes the programs using the memory 44 as a working area. The CPU 42 controls each component and performs various arithmetic processes according to the programs stored in the storage device 46.

[0073] The memory 44 is made up of a RAM (Random Access Memory) and temporarily stores programs and data as a working area. The storage device 46 is made up of a ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., and stores various programs including an operating system, and various data.

[0074] The input / output I / F 48 is an interface for inputting data from the outside and outputting data to the outside. In addition, an input device for performing various inputs, such as a keyboard or a mouse, and an output device for outputting various information, such as a display or a printer, may be connected. A touch panel display may be used as the output device to function as the input device.

[0075] The storage medium reader 50 reads data stored in various storage media such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, Blu-ray Disc, and USB (Universal Serial Bus) memory, and writes data to the storage media.

[0076] The communication I / F 52 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0077] Next, the functional configuration of the control device 14 will be described. As shown in Fig. 7, the control device 14 functionally includes an acquisition unit 17, a generation unit 18, a control unit 20, and a learning unit 22. A data storage unit 24 and a trained model storage unit 26 are provided in a predetermined storage area of ​​the control device 14. Each functional configuration is realized by the CPU 42 reading out each program stored in the storage device 46, expanding it in the memory 44, and executing it.

[0078] The data storage unit 24 stores various data obtained by the sensor group 11, sequence data of past control commands, and the like.

[0079] The trained model storage unit 26 stores a plurality of neural network models constituting the above-mentioned Probabilistic Ensemble Model. In this embodiment, the plurality of neural network models constituting the Probabilistic Ensemble Model are simply referred to as dynamics models. The dynamics model in this embodiment is a model that represents the state of the robot 12 and the surrounding environment. FIG. 9 is a diagram for explaining the dynamics model.

[0080] As shown in FIG. 9, the dynamics model is generated by using continuous state data s , which represents the continuous-valued state of the robot 12 at time t. conti tand continuous action data a representing the continuous value of the action of the robot 12 at time t conti t When input is made, the continuous state data s at time t+1 is conti t As shown in FIG. 9, the dynamics model outputs the position data p arm When the input is made, the predicted value l of the length of the spring 16C of the controlled part 16 is pred is also the model that outputs

[0081] The acquisition unit 17 acquires position data p of the arm 16A of the control target part 16 provided in the robot 12. arm For example, the acquisition unit 17 acquires the position data p of the arm 16A acquired by the sensor group 11. arm Get the.

[0082] The generation unit 18 applies the position data p acquired by the acquisition unit 17 to the dynamics model stored in the trained model storage unit 26. arm By inputting the above, the generation unit 18 generates predicted deformation degree data representing a predicted value of the deformation degree of the control target portion 16. Specifically, the generation unit 18 inputs the position data p arm By inputting the above, the predicted value l of the length of the spring 16C of the controlled part 16 is obtained. pred The predicted length of spring 16C, l pred is an example of predicted deformation degree data.

[0083] The control unit 20 receives a sequence a of continuous behavior data from time t+1 to time t+H generated by the learning unit 22 (to be described later) in the previous control step. conti t:t+H-1 and the sequence of discrete behavioral data a from time t+1 to time t+H disc t:t+H-1 The operation of the robot 12 is controlled in accordance with the above.

[0084] Here, the control unit 20 controls the mode switching when controlling the robot 12 according to the above-mentioned sequence data. The above-mentioned blocker function is realized by the mode switching control by the control unit 20.

[0085] Specifically, the control unit 20 calculates the predicted value l of the length of the spring 16C generated by the generation unit 18. pred and the natural length l of spring 16C natural If the difference between is equal to or greater than the threshold, the control unit 20 controls the control target part 16 to be in the first mode. The first mode is a mode in which the arm 16A and the gripper 16B are connected by the spring 16C. On the other hand, the control unit 20 controls the predicted value l of the length of the spring 16C generated by the generation unit 18. pred and the natural length l of spring 16C natural is less than the threshold value, the control target part 16 is controlled to be in the second mode. The second mode is a mode in which the arm 16A and the gripper 16B are connected to be integral with each other.

[0086] In addition, the control unit 20 receives position data p of the arm 16A of the control target part 16 while the control target part 16 is being moved. arm and correct deformation degree data that indicates the actual deformation degree of the spring 16C of the control target portion 16 are acquired as teacher data. The correct deformation degree data is detected by the sensor group 11, for example.

[0087] The learning unit 22 re-learns the dynamics model stored in the trained model storage unit 26 by executing known supervised mechanics based on the training data acquired by the control unit 20.

[0088] Furthermore, the learning unit 22 generates a series of continuous behavior data to be used in the next control step in the control unit 20 based on the model-based reinforcement learning. In this case, the learning unit 22 optimizes the continuous behavior data and discrete behavior data representing the continuous-value behavior of the robot 12 while fixing the discrete behavior data representing the discrete-value behavior of the robot 12 representing switching between the first mode and the second mode in the model-based reinforcement learning. Then, in the next control step, the control unit 20 controls the operation of the robot 12 according to the optimized continuous behavior data and discrete behavior data.

[0089] Specifically, the learning unit 22 first applies the continuous state data s conti and the previous continuous action data a conti By repeatedly inputting, the sequence of continuous state data s from time t+1 to time t+H is obtained. conti t:t+H-1 Next, the learning unit 22 acquires a sequence a of discrete behavior data representing the discrete-value behavior of the robot 12, which represents switching between the first mode and the second mode, from time t+1 to time t+H. disc t:t+H-1 Fix the.

[0090] Then, the learning unit 22 calculates the sequence s of continuous state data from time t+1 to time t+H. conti t:t+H-1 and a sequence of discrete behavioral data a from fixed time t+1 to time t+H disc t:t+H-1 Based on this, we calculate the sequence a of continuous action data from time t+1 to time t+H so that the evaluation function that represents the sum of rewards in model-based reinforcement learning is maximized. conti t:t+H-1 and the sequence of discrete behavioral data a from time t+1 to time t+H disc t:t+H-1The continuous behavior data is, for example, a control command for the robot 12. For example, the continuous behavior data is a speed command for the arm 16A. The discrete behavior data is a control command representing a mode switch and is expressed by 0 or 1.

[0091] In the next control step, the control unit 20 determines the sequence a of continuous behavior data from time t+1 to time t+H generated by the learning unit 22. conti t:t+H-1 and the sequence of discrete behavioral data a from time t+1 to time t+H disc t:t+H-1 The operation of the robot 12 is controlled in accordance with the above.

[0092] Next, the operation of the control device 14 according to this embodiment will be described.

[0093] When the peg T is placed on the gripper 16B and the control device 14 receives a predetermined instruction signal, the CPU 42 of the control device 14 reads out a control program from the storage device 46, loads it into the memory 44, and executes it. As a result, the CPU 42 functions as each functional component of the control device 14, and the control processing shown in Figs. 10 and 11 is repeatedly executed.

[0094] In step S100, the acquisition unit 17 acquires the position data p of the arm 16A acquired by the sensor group 11. arm Get the.

[0095] In step S102, the generation unit 18 applies the position data p of the arm 16A acquired in step S100 to the dynamics model stored in the trained model storage unit 26. arm By inputting the above, the predicted length of spring 16C, l pred Get the.

[0096] In step S104, the control unit 20 determines whether a predetermined condition for mode switching is satisfied based on the data generated in step S102. Specifically, the predicted value l of the length of the spring 16C acquired in step S102pred and the natural length l of spring 16C natural If the difference between the predicted value l of the length of the spring 16C acquired in step S102 is less than the threshold value, the process proceeds to step S106. pred and the natural length l of spring 16C natural If the difference between is equal to or greater than the threshold, the process proceeds to step S108.

[0097] In step S106, the control unit 20 calculates the sequence a of continuous behavior data from time t+1 to time t+H generated by the learning unit 22 in the previous step. conti t:t+H-1 and the sequence of discrete behavioral data a from time t+1 to time t+H disc t:t+H-1 In response to the above, the control unit 20 controls the operation of the robot 12. At this time, the control unit 20 controls the control target part 16 to be in the second mode. If the current mode of the control target part 16 is the second mode, the control unit 20 keeps the mode of the control target part 16 as it is.

[0098] In step S108, the control unit 20 calculates the sequence a of continuous behavior data from time t+1 to time t+H generated by the learning unit 22 in the previous step. conti t:t+H-1 and the sequence of discrete behavioral data a from time t+1 to time t+H disc t:t+H-1 The control unit 20 controls the operation of the robot 12 in response to the above. At this time, the control unit 20 controls the control target part 16 to be in the first mode. If the current mode of the control target part 16 is the first mode, the control unit 20 keeps the mode of the control target part 16 as it is.

[0099] Next, the control device 14 executes a control process shown in FIG.

[0100] In step S200, the learning unit 22 calculates the position data p of the arm 16A of the control target part 16 while the control target part 16 is being moved. armand the correct deformation degree data representing the actual deformation degree of the spring 16C of the control target part 16 are obtained as teacher data.

[0101] In step S202, the learning unit 22 re-learns the dynamics model stored in the trained model storage unit 26 by executing known supervised mechanics based on the training data acquired in step S200.

[0102] In step S204, the learning unit 22 generates a sequence of continuous behavior data and a sequence of discrete behavior data to be used in the next control step in the control unit 20 based on model-based reinforcement learning.

[0103] The control processes shown in Figures 10 and 11 are repeated, and control signals are repeatedly output to the robot 12, thereby properly executing the peg-in-hole task.

[0104] As described above, the control device according to the present embodiment acquires position data of a control target part provided in a robot, and inputs the acquired position data to a dynamics model to generate a predicted value of a spring length, which is an example of a degree of deformation of the control target part. When controlling the robot, the control device controls the control target part to be in a first mode if the difference between the generated predicted value of the spring length and the natural length of the spring is equal to or greater than a threshold value. When the difference between the generated predicted value of the spring length and the natural length of the spring is equal to or greater than a threshold value, the control device controls the control target part to be in a second mode. This makes it possible to execute a task while preventing the robot from coming into contact with an obstacle in a stiff state when the robot is unable to acquire information about the environment in advance.

[0105] Although the embodiment and examples have been described above, it goes without saying that the present invention can be embodied in various forms without departing from the gist of the present invention.

[0106] For example, in the above embodiment, the target task is a peg-in-hole task, but the target task is not limited to this. Any task that can be executed by the robot 12 may be used.

[0107] Furthermore, the configuration of the state data and the action data in the above embodiment can be modified as appropriate.

[0108] In the above embodiment, the length of the spring is predicted by the dynamics model, and the first mode and the second mode are switched according to the length. However, the present invention is not limited to this. For example, the control device 14 may receive position data p arm In this case, for example, the control device 14 may switch between the first mode and the second mode depending on whether the arm 16A has visited the position represented by the position data p arm If the control target site 16 has never visited the position represented by the position data p arm When the degree of contact of the controlled part 16 with the obstacle is equal to or greater than a predetermined value, the control device 14 controls the controlled part 16 to enter the first mode in which the degree of flexibility of the controlled part 16 is equal to or greater than a predetermined value. arm In the case where the control target site 16 has visited the position represented by the position data p arm If the degree of contact of the controlled part 16 with the obstacle is less than a predetermined value in the first mode, the controlled part 16 is controlled so as to enter the second mode in which the degree of hardness of the controlled part 16 is equal to or greater than a predetermined value. Whether or not the degree of contact of the controlled part 16 with the obstacle is equal to or greater than a predetermined value is determined by data output from the dynamics model as described above. Note that a specific value for determining whether or not the degree of contact with the obstacle is equal to or greater than a predetermined value may be preset by, for example, a user. Also, the predetermined value representing the degree of softness of the controlled part 16 varies depending on the mechanism that realizes the controlled part 16.

[0109] In addition, in the above embodiment, it is assumed that the mode switching occurs only once, but the present invention is not limited to this. For example, the mode switching may occur multiple times.

[0110] In addition, various processors other than the CPU may execute each process executed by reading the software (program) in the above embodiment. Examples of the processor in this case include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. Each process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). More specifically, the hardware structure of these various processors is an electric circuit that combines circuit elements such as semiconductor elements.

[0111] In the above embodiment, the programs are stored (installed) in advance in a storage device, but the present invention is not limited to this. The programs may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, a Blu-ray disc, or a USB memory. The programs may be downloaded from an external device via a network.

[0112] (Additional Note) The following additional notes are given regarding aspects of the present disclosure.

[0113] (Appendix 1) an acquisition unit that acquires position data of a control target portion of the robot; if the control target part has never been to the position represented by the position data before, or if the degree of contact of the control target part with an obstacle at the position is equal to or greater than a predetermined value, the control target part is controlled to enter a first mode in which the degree of softness of the control target part is equal to or greater than a predetermined value; a control unit that controls the control target part to enter a second mode in which the degree of hardness of the control target part is equal to or greater than a predetermined value when the control target part has previously visited the position represented by the position data and the degree of contact of the control target part with an obstacle at the position is less than a predetermined value; A control device comprising:

[0114] a generation unit that generates predicted deformation degree data as a degree of contact of the control target part with an obstacle by inputting the position data acquired by the acquisition unit into a dynamics model that outputs predicted deformation degree data representing a predicted value of a deformation degree of the control target part when position data of the control target part is input, When controlling the robot, the control unit controls the control target part to be in the first mode when the generated predicted deformation degree data is equal to or greater than a predetermined threshold, and controls the control target part to be in the second mode when the generated predicted deformation degree data is less than a predetermined threshold. 2. The control device according to claim 1.

[0115] (Appendix 3) Further comprising a learning section; The control unit acquires a combination of position data of the control target part and correct deformation degree data representing an actual deformation degree of the control target part, which is obtained while the control target part is being moved, and stores the combination in a storage unit; the learning unit reads out the combination from the storage unit, and causes the dynamics model to learn based on the combination. 3. The control device according to claim 2.

[0116] (Appendix 4) the learning unit optimizes continuous behavior data representing a continuous-value behavior of the robot and the discrete behavior data while fixing discrete behavior data representing a discrete-value behavior of the robot that represents switching between the first mode and the second mode in model-based reinforcement learning; The control unit controls an operation of the robot in accordance with the optimized continuous state data and the optimized discrete behavior data. 4. The control device according to claim 3.

[0117] (Appendix 5) The learning unit is a dynamics model which outputs the continuous state data at time t+1 when continuous state data representing a continuous value state of the robot at time t and the continuous behavior data at time t are input, and by repeatedly inputting the continuous state data at a previous time and the continuous behavior data at a previous time, a series of the continuous state data from time t+1 to time t+H is obtained; The sequence of the discrete behavior data from time t+1 to time t+H is fixed; Based on the acquired sequence of continuous state data from time t+1 to time t+H and the fixed discrete behavior data from time t+1 to time t+H, the sequence of continuous behavior data from time t+1 to time t+H and the sequence of discrete behavior data from time t+1 to time t+H are acquired such that an evaluation function representing a sum of rewards in model-based reinforcement learning is maximized; the control unit controls an operation of the robot in accordance with the series of the continuous behavior data from time t+1 to time t+H and the series of the discrete behavior data from time t+1 to time t+H. 5. The control device according to claim 4.

[0118] (Appendix 6) the control target portion represents a portion including an arm and a gripper provided to the robot, the first mode is a mode in which the arm and the gripper are connected by an elastic body, The second mode is a mode in which the arm and the gripper are connected to be integrated. The control device according to any one of Supplementary notes 1 to 5.

[0119] (Appendix 6) Acquire position data of a control target part of the robot, if the control target part has never been to the position represented by the position data before, or if the degree of contact of the control target part with an obstacle at the position is equal to or greater than a predetermined value, the control target part is controlled to enter a first mode in which the degree of softness of the control target part is equal to or greater than a predetermined value; When the control target part has previously visited the position represented by the position data and the degree of contact of the control target part with an obstacle at the position is less than a predetermined value, the control target part is controlled to enter a second mode in which the degree of hardness of the control target part is equal to or greater than a predetermined value. A method of controlling how processing is executed by a computer.

[0120] (Appendix 7) Acquire position data of a control target part of the robot, if the control target part has never been to the position represented by the position data before, or if the degree of contact of the control target part with an obstacle at the position is equal to or greater than a predetermined value, the control target part is controlled to enter a first mode in which the degree of softness of the control target part is equal to or greater than a predetermined value; When the control target part has previously visited the position represented by the position data and the degree of contact of the control target part with an obstacle at the position is less than a predetermined value, the control target part is controlled to enter a second mode in which the degree of hardness of the control target part is equal to or greater than a predetermined value. A control program that causes a computer to execute processing. [Explanation of symbols]

[0121] 10. Control System 14 Control device 16A Arm 16B Gripper 16C Spring 17 Acquisition Department 18 Generation part 20 Control section 22 Learning Department 24 Data storage unit 26 Trained model memory

Claims

1. an acquisition unit that acquires position data of a control target portion of the robot; if the control target part has never been to the position represented by the position data before, or if the degree of contact of the control target part with an obstacle at the position is equal to or greater than a predetermined value, the control target part is controlled to enter a first mode in which the degree of softness of the control target part is equal to or greater than a predetermined value; a control unit that controls the control target part to enter a second mode in which the degree of hardness of the control target part is equal to or greater than a predetermined value when the control target part has previously visited the position represented by the position data and the degree of contact of the control target part with an obstacle at the position is less than a predetermined value; A control device comprising:

2. a generation unit that generates predicted deformation degree data as a degree of contact of the control target part with an obstacle by inputting the position data acquired by the acquisition unit into a dynamics model that outputs predicted deformation degree data representing a predicted value of a deformation degree of the control target part when position data of the control target part is input, When controlling the robot, the control unit controls the control target part to be in the first mode when the generated predicted deformation degree data is equal to or greater than a predetermined threshold, and controls the control target part to be in the second mode when the generated predicted deformation degree data is less than a predetermined threshold. The control device according to claim 1 .

3. Further comprising a learning section; The control unit acquires a combination of position data of the control target part and correct deformation degree data representing an actual deformation degree of the control target part, which is obtained while the control target part is being moved, and stores the combination in a storage unit; the learning unit reads out the combination from the storage unit, and causes the dynamics model to learn based on the combination. The control device according to claim 2.

4. the learning unit optimizes continuous behavior data representing a continuous-value behavior of the robot and the discrete behavior data while fixing discrete behavior data representing a discrete-value behavior of the robot that represents switching between the first mode and the second mode in model-based reinforcement learning; the control unit controls an operation of the robot in accordance with the optimized continuous behavior data and the optimized discrete behavior data. The control device according to claim 3.

5. The learning unit is a dynamics model which outputs the continuous state data at time t+1 when continuous state data representing a continuous value state of the robot at time t and the continuous behavior data at time t are input, and by repeatedly inputting the continuous state data at a previous time and the continuous behavior data at a previous time, a series of the continuous state data from time t+1 to time t+H is obtained; The sequence of the discrete behavior data from time t+1 to time t+H is fixed; Based on the acquired sequence of continuous state data from time t+1 to time t+H and the fixed discrete behavior data from time t+1 to time t+H, a sequence of continuous behavior data from time t+1 to time t+H and a sequence of discrete behavior data from time t+1 to time t+H are acquired such that an evaluation function representing a sum of rewards in model-based reinforcement learning is maximized; the control unit controls an operation of the robot in accordance with the series of the continuous behavior data from time t+1 to time t+H and the series of the discrete behavior data from time t+1 to time t+H. The control device according to claim 4.

6. the control target portion represents a portion including an arm and a gripper provided to the robot, the first mode is a mode in which the arm and the gripper are connected by an elastic body, The second mode is a mode in which the arm and the gripper are connected to be integrated. The control device according to claim 1 or 2.

7. Acquire position data of a control target part of the robot, if the control target part has never been to the position represented by the position data before, or if the degree of contact of the control target part with an obstacle at the position is equal to or greater than a predetermined value, the control target part is controlled to enter a first mode in which the degree of softness of the control target part is equal to or greater than a predetermined value; When the control target part has previously visited the position represented by the position data and the degree of contact of the control target part with an obstacle at the position is less than a predetermined value, the control target part is controlled to enter a second mode in which the degree of hardness of the control target part is equal to or greater than a predetermined value. A method of controlling how processing is executed by a computer.

8. Acquire position data of a control target part of the robot, if the control target part has never been to the position represented by the position data before, or if the degree of contact of the control target part with an obstacle at the position is equal to or greater than a predetermined value, the control target part is controlled to enter a first mode in which the degree of softness of the control target part is equal to or greater than a predetermined value; When the control target part has previously visited the position represented by the position data and the degree of contact of the control target part with an obstacle at the position is less than a predetermined value, the control target part is controlled to enter a second mode in which the degree of hardness of the control target part is equal to or greater than a predetermined value. A control program that causes a computer to execute processing.

Images