Robot operation training device, robot operation training system, robot, and robot operation training method

Abstract

Provided is a robot operation training method capable of re-training a learning model in a short time and at a small hardware cost. The robot operation training device is for assisting learning of a robot to execute an operation, and is characterized by comprising: an operation model; another operation model; an error calculation unit that calculates an error between the operation model and the other operation model by using an operation execution result of the operation model and an operation execution result of the other operation model; a collection range determination unit that determines a collection range of learning data to be collected from a peripheral environment of the robot by using the error calculated by the error calculation unit; a data collection unit that collects learning data by using the collection range determined by the collection range determination unit; and a training unit that trains at least one of the operation model and the other operation model by using the learning data collected by the data collection unit.

Description

ROBOT MOTION LEARNING DEVICE, ROBOT MOTION LEARNING SYSTEM, ROBOT, AND ROBOT MOTION LEARNING METHOD

[0001] The present invention relates to a technology that is effective when applied to a robot motion learning device, a robot motion learning system, a robot, and a robot motion learning method that re-learns a learning model based on learning data collected from the robot's surrounding environment.

[0002] Learning-based robot control models are often used to achieve highly successful manipulation tasks in a variety of different environments. A typical training method for learning-based models involves training in a lab environment and then retraining using data extracted from the field. This allows for a learning-based model that is adapted to the field while maintaining high accuracy. However, because the range of training data that needs to be acquired from the field is unclear, the data acquisition range becomes wide, leading to problems such as extended training times and hardware degradation.

[0003] To solve this problem, for example, Patent Literature 1 proposes an information processing system that can automatically switch learning without external instruction input by executing a model that has undergone reinforcement learning in advance and updating the learning model during execution based on the magnitude of the difference with existing environmental information and reward information. This allows the model to be learned only when learning is necessary, and furthermore, the use of reinforcement learning makes data collection unnecessary, reducing the time spent collecting unnecessary learning data and reducing hardware degradation.

[0004] International Publication No. 2022 / 201796

[0005] However, the information processing system of Patent Document 1 does not consider which range of the environment should be used for re-learning when it is determined that there is a large difference from the existing environmental information or reward information. As a result, learning may be performed over a wide range of the environment, which may take a long time until the difference from the existing environmental information or reward information becomes small.

[0006] Therefore, an object of the present invention is to provide a robot motion learning device, a robot motion learning system, a robot, and a robot motion learning method that are capable of re-learning a learning model in a short time and with low hardware costs.

[0007] In order to solve the above problems, the present invention provides a robot behavior learning device that assists a robot in learning to perform behavior, and is characterized by comprising: an error calculation unit that calculates an error between a certain behavior model and the other behavior model using a certain behavior model, another behavior model, and a behavior execution result of the certain behavior model and a behavior execution result of the other behavior model; a collection range determination unit that determines a collection range of learning data to be collected from the surrounding environment of the robot using the error calculated by the error calculation unit; a data collection unit that collects learning data using the collection range determined by the collection range determination unit; and a learning unit that trains at least one of the certain behavior model and the other behavior model using the learning data collected by the data collection unit.

[0008] The present invention also provides a robot movement learning method for assisting a robot in learning to execute movements, comprising an error calculation step of calculating an error between a certain movement model and another movement model using a movement execution result of the certain movement model and a movement execution result of the other movement model; a collection range determination step of determining a collection range of learning data to be collected from the surrounding environment of the robot using the error; a data collection step of collecting learning data using the collection range; and a learning step of training at least one of the certain movement model and the other movement model using the learning data collected in the data collection step.

[0009] According to the present invention, it is possible to realize a robot motion learning device, a robot motion learning system, a robot, and a robot motion learning method that are capable of relearning a learning model in a short time and with low hardware costs.

[0010] This enables the robot to perform manipulation tasks with a high success rate while reducing training time and hardware costs.

[0011] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0012] FIG. 3 is a block diagram showing a basic configuration of a robot movement learning system according to a first embodiment of the present invention. FIG. 4 is a diagram showing a schematic configuration of a robot system according to a first embodiment of the present invention. FIG. 5 is a flowchart showing a robot movement learning method according to a first embodiment of the present invention. FIG. 6 is a diagram showing a schematic diagram of a learning data collection method in imitation learning of FIG. 3. FIG. 7 is a diagram showing a schematic diagram of a learning data collection method in reinforcement learning of FIG. 3. FIG. 8 is a flowchart showing a modified example of FIG. 3. FIG. 9 is a diagram showing a schematic configuration of a robot system according to a second embodiment of the present invention. FIG. 10 is a flowchart showing a robot movement learning method according to a second embodiment of the present invention. FIG. 11 is a diagram showing a schematic diagram of a learning movement generation method according to a second embodiment of the present invention. FIG. 12 is a block diagram showing a configuration of a robot movement learning system according to a third embodiment of the present invention.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same components are designated by the same reference numerals, and detailed description of overlapping parts will be omitted.

[0014] A robot motion learning method, a robot motion learning device, and a robot motion learning system according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 6. FIG.

[0015] 1 is a block diagram showing the basic configuration of a robot movement learning system 100 according to this embodiment. The robot movement learning system 100 according to this embodiment includes, as its main component, a robot movement learning device (learning support device) 101 that supports learning of a learning model of movement commands to be sent to a robot to be controlled. The robot movement learning device is configured as a computer having a CPU (Central Processing Unit), MPU (Micro Processing Unit), DSP (Digital Signal Processor), processor, etc., as well as memory and peripheral circuits.

[0016] The robot behavior learning device (learning support device) 101 includes an error calculation unit 104 that calculates an error between one behavior model and the other behavior model using a certain behavior model, another behavior model, and the behavior execution results of the certain behavior model and the behavior execution results of the other behavior model; a collection range determination unit 105 that determines a collection range of learning data to be collected from the robot's surrounding environment using the error calculated by the error calculation unit 104; a data collection unit 109 that collects learning data using the collection range determined by the collection range determination unit 105; and a learning unit 110 that learns at least one of the certain behavior model and the other behavior model using the learning data collected by the data collection unit 109.

[0017] In detail, the robot motion learning device (learning support device) 101 includes a first motion model 102, a second motion model 103 having a higher success probability of the robot motion compared to the first motion model 102, an error calculation unit 104 that calculates an error between the first motion model 102 and the second motion model 103 using the motion execution result of the first motion model 102 and the motion execution result of the second motion model 103, a collection range determination unit 105 that determines a collection range of learning data from the surrounding environment of the robot using the error calculated by the error calculation unit 104, and a collection range determination unit 106 that determines a collection range of learning data from the surrounding environment of the robot. The control system includes a learning action generation unit 107 that generates a learning action based on the collection range determined by the range determination unit 105 and sends an action command to cause the action execution unit 106 to execute the action of the control target, an environmental data acquisition unit 108 that acquires environmental information obtained by the action execution unit 106 executing the action, a data collection unit 109 that collects the environmental information from the environmental data acquisition unit 108 and the action command from the learning action generation unit 107 as learning data, and a learning unit 110 that extracts the learning data from the data collection unit 109 and learns a first action model 102. Note that there may be a plurality of first action models and second action models.

[0018] Although FIG. 1 shows an example in which the environmental data acquisition unit 108 is disposed in the robot to be controlled, it may also be disposed in the robot motion learning device (learning support device) 101 as described above.

[0019] Furthermore, while Figure 1 shows an example of a configuration in which the robot motion learning device (learning support device) 101 and the robot to be controlled are independent, the robot motion learning device (learning support device) 101 may be built into the robot and configured as an integrated robot.

[0020] The error calculation unit 104 calculates, as an error, the difference between the motion position when executed using the first motion model 102 and the motion position when executed using the second motion model 103. The error calculation unit 104 may also calculate, as an error, the difference between the motion trajectory when executed using the first motion model 102 and the motion trajectory when executed using the second motion model 103. The error of the motion position, motion trajectory, etc. calculated by the error calculation unit 104 is the difference between the motion execution results using each motion model.

[0021] The collection range determination unit 105 determines, as a collection range, an area in the vicinity of an error that occurs when the robot executes an operation, from among the environmental information acquired by the environmental data acquisition unit 108. The environmental information is information about the environment around the robot.

[0022] The data collection unit 109 has a learning data collection function of collecting data related to the operation of the robot within the collection range determined by the collection range determination unit 105 .

[0023] The learning unit 110 sends update information such as gradients to update the first operation model 102. The first operation model 102 and the second operation model 103 can cause the robot to execute a task by sending an operation command to the operation execution unit 106 of the controlled object.

[0024] Using the robot motion learning device (learning support device) 101 of this embodiment, it is possible to learn the first motion model 102 for application to a new environment in a short time and with low hardware costs.

[0025] A robot movement learning method using the robot movement learning device (learning support device) 101 of this embodiment will be described with reference to FIGS.

[0026] 2 is a diagram showing a schematic configuration of a robot system 200 according to this embodiment. In this embodiment, the robot motion learning system 100 is used to learn the control model of the robot system 200 shown in FIG.

[0027] The robot system 200 is composed of a robot arm 201, a belt conveyor 202, a plurality of fixed target portions 203 each including four through holes, a plurality of base portions 204 each including four screw holes, a plurality of screws 205, a screw feeder 206, a camera 207, and a camera fixing portion 208. The robot arm 201 is composed of, for example, six joints (not shown), a force sensor 209 at the tip, and a gripper 210 with two fingers. The fixed target portion 203 is transported near the robot arm 201 by the belt conveyor 202 and fixed to the base portion 204 with the screws 205.

[0028] The task performed by the robot system 200 is an assembly task, and the operations required for this task will be described below. In this assembly task, after the belt conveyor 202 transports the base part 204 and the fixed object part 203, the robot arm 201 grasps the screw 205 of the screw feeder 206 and inserts the screw 205 into one hole in the fixed object part 203. The robot arm 201 then rotates the joint closest to the tip of the robot arm 201, and screws the screw 205 into the screw hole in the base part 204. By repeating this operation four times, the fixed object part 203 is fixed to the base part 204.

[0029] Because the positions of the robot arm 201 and the screw feeder 206 are fixed, gripping of the screw 205 can be realized by a rule-based program. However, inserting the screw 205 into the hole in the fixing target part 203 is difficult using a rule-based program because the position of the part being transported by the belt conveyor 202 changes each time.

[0030] Therefore, in the robot system 200 of this embodiment, in order to extract rough information about the hole position, the hole position is detected using an image processing algorithm such as template matching on the image from the camera 207, and the first operating model 102 or the second operating model 103, which are data-driven learning models, generate the operation of the robot arm 201 based on the rough hole position information, and move the robot arm 201 based on that, thereby realizing the insertion of the screw 205 into the hole in the fixed target part 203.

[0031] The motion in the air is generated based on image information from the camera 207, but the motion after the screw 205 comes into contact with the surface of the fixed target part 203 is generated based on not only the image information but also the contact information acquired by the force sensor 209. By using the information from the force sensor 209, the motion to search for the insertion hole can be generated more effectively.

[0032] In this assembly task, the robot control model that needs to be learned is the model used to insert the screw 205, and in this embodiment, the first motion model 102 of the robot motion learning system 100 is assumed.

[0033] 3 is a flowchart showing the robot motion learning method of this embodiment, which illustrates a method for learning the first motion model 102 when the robot system 200 is used in a new environment or when it is used to assemble a new part.

[0034] This situation is similar to when a learning model is transferred from a lab environment to the field. It is assumed that the first operating model 102 has been pre-trained in a previous environment similar to the current environment and can therefore, theoretically, be used in the current environment without retraining. It is also assumed that the rule-based picking operation of the screw 205 is valid for the current system. It is further assumed that after the hole is found, the screw 205 can be similarly driven by a rule-based program.

[0035] This robot motion learning method 300 begins with the motion execution unit 106 of the controlled object using a motion command from the first motion model 102 to attempt to insert a screw 205 into one hole in the fixed target part 203 using the robot arm 201 (step S301).

[0036] If the robot arm 201 successfully inserts the screw 205 (step S302a), this means that the first operation model 102 can find the hole with the robot arm 201 without re-learning the first operation model 102, and the learning flow is terminated.

[0037] On the other hand, if the robot arm 201 cannot insert the screw 205 (step S302b), the operation execution unit 106 searches for the hole using the operation command of the second operation model 103, which has a higher success rate for finding the hole (step S303).

[0038] If the hole cannot be found by second movement model 103 (step S304b), the actual position of the hole cannot be found, and therefore re-learning of first movement model 102 is impossible, and the learning flow is terminated.

[0039] If a hole is found by the second movement model 103 (step S304a), the error calculation unit 104 detects the error between the movement from the first movement model 102 and the movement from the second movement model 103 (step S305).

[0040] Next, the collection range determination unit 105 determines the range of the learning movement data for training of the first movement model 102 that should be collected for accomplishing the task based on the movement error (step S306).

[0041] Then, the learning motion generator 107 generates a learning motion and sends this motion command to the motion execution unit 106, which then moves the robot arm 201 based on this motion command.

[0042] Next, the data collection unit 109 stores the operation command and the environmental information obtained during the operation (step S307).

[0043] Next, the learning unit 110 samples data from the data collecting unit 109 and re-learns the first action model 102 (step S308). This re-learning in step S308 is positioned as imitation learning or reinforcement learning, which will be described later.

[0044] Finally, after re-learning the first operation model 102, the first operation model 102 is used to try inserting the screw 205 again (step S301), and the robot operation learning method 300 is repeated until the first operation model 102 can accomplish the task.

[0045] The second operation model 103 may be either a rule-based or learning-based model, as long as it can achieve a task with higher performance than the first operation model 102 .

[0046] An example of a rule-based model for the task of inserting a screw 205 is a spiral model, in which the robot arm 201 moves on a fixed spiral trajectory to search for holes. This spiral model searches for holes over a wide area where holes are likely to be found, so the probability of finding a hole, i.e., the success rate, is high. However, there is a problem in that it can take a long time to find a hole depending on the initial search position. By using a learning model such as the first behavior model 102 used for environmental data, holes can be found in a shorter time, so the learning model is preferable in terms of the lead time for completing a field task.

[0047] The subtask assumed in this embodiment is the insertion of a screw 205 into a hole in a fixed target part 203, and therefore the motion error can be determined as the difference between the center of the hole found by the second motion model 103 and the hole position predicted by the first motion model 102, i.e., a position error. This position error can be calculated using the Euclidean distance equation of formula (1).

[0048]

[0049] In this equation, d is the position error, x1 is the position of the first motion model 102 relative to the x-axis, x2 is the position of the second motion model 103 relative to the x-axis, y1 is the position of the first motion model 102 relative to the y-axis, and y2 is the position of the second motion model 103 relative to the y-axis.

[0050] There are two possible methods for collecting data and learning the first motion model 102 using the motion error (position error) in this embodiment.

[0051] <Imitation Learning> FIG. 4 is a diagram showing a first example of a data collection method for learning the first operation model 102. In FIG.

[0052] This data collection method 400 is a data collection method in which a screw 205 is brought into contact with the surface of a fixed target part 203 at multiple positions on a circle with a radius equal to a maximum distance error 402, and the screw 205 is slid along a sliding motion trajectory 403 to the center of the hole 401 until the screw 205 is inserted into the hole 401, and during the operation, images from a camera 207, environmental data such as force and torque from a force sensor 209, and motion commands such as the joint angles of the robot are collected.

[0053] The maximum distance error 402 is the same as the maximum value of the motion error described above. After collecting data, imitation learning is applied to learning the first motion model 102 using this data collection method 400. Imitation learning is a learning method that uses a mean squared error loss function and a stochastic gradient descent optimizer to minimize the error between the motion (output) of the training dataset and the motion generated by the first motion model 102, and is suitable for the data collection method 400.

[0054] 4 needs to be determined, which may become enormous in some cases, but the amount of movement is limited by the maximum distance error 402, so the amount of movement is smaller than sliding over a wide range. In other words, because the data range obtained from the maximum distance error 402 is smaller, the amount of training data can be minimized, making it possible to reduce the training time and hardware costs, which are the goals of the present invention.

[0055] In this data collection method 400, the collection range determined by the collection range determination unit 105 is a circular area, the movement position obtained by the second movement model 103 is the center of the circular area, and the maximum distance error 402 is the radius of the circular area.

[0056] The data collected for learning the first movement model 102 is the environmental information and movement information obtained when moving from a position at a distance of the maximum distance error 402 from the center of the circular area to the center of the circular area.

[0057] <Reinforcement Learning> FIG. 5 is a diagram showing a second example of a data collection method for learning the first operation model 102. In FIG.

[0058] In this data collection method 500, the robot arm 201 contacts the screw 205 at a plurality of data collection positions 502 in the hole 501, which are represented by circular marks, and within a limit position determined by a data collection limit position 503, which is represented by a triangular mark. The data collection limit position 503 is determined by the maximum distance error 402.

[0059] Similar to data collection method 400, the environmental data collected by data collection method 500 is images from camera 207 or forces and torques from force sensors 209, and the collected motion commands are robot joint angles. However, in data collection method 500, data is not collected at a set frequency (e.g., every 0.1 seconds), but is collected only at data collection position 502 and data collection limit position 503. This allows a map of the area around the hole to be created, and this map can be used to build an offline learning environment.

[0060] The offline learning environment can be used to perform reinforcement learning, which is suitable for the present data collection method 500. The offline learning environment can be used to allow a reinforcement learning agent to learn the first behavior model 102, and enable the first behavior model 102 to perform a task.

[0061] 5 needs to be determined in advance, and in some cases it may become enormous, but since the number of data collection positions 502 is limited by the data collection limit position 503, it is less than collecting data over a wide range. In other words, because the data range obtained from the data collection limit position 503 is smaller, the amount of training data can be minimized, making it possible to reduce training time and hardware costs, which are the goals of the present invention.

[0062] In this data collection method 500, the collection range determined by the collection range determination unit 105 is a rectangular area, the motion position obtained by the second motion model 103 is the center of the rectangular area, and the error determines the data collection limit position 503 of the rectangular area.

[0063] The data obtained from the collection range is a plurality of discretized points within the collection range, and a reinforcement learning environment is constructed using a plurality of the discretized points, and the first behavior model 102 is re-learned using the reinforcement learning environment.

[0064] As described above, by using the robot motion learning device (learning support device) 101 of this embodiment, it is possible to learn the first motion model 102 for application to a new environment in a short time and with low hardware costs.

[0065] <Modification> FIG. 6 is a flowchart showing a modification of FIG.

[0066] The robot movement learning method shown in FIG. 6 differs from the robot movement learning method shown in FIG. 3 in that it has two additional steps, step S606 and step S607, between step S305 and step S306 in the flow of FIG. 3.

[0067] Similar to the robot motion learning method of FIG. 3, after the position error (motion error) is determined in step S605, this variant checks whether it is the first time that the first motion model 102 has failed to find the hole (step S606), and if so, in adjustment step S607, the first motion model 102 is tested again for the same task in the same environment (step S601), simply by subtracting the distance error from the predicted position.

[0068] This is done because the prediction errors of the first motion model 102 may be systematic and therefore can be corrected by a simple calibration of the first motion model 102 without having to retrain the model.

[0069] The additional step of this modification in the update flow of the first operation model 102 can further accelerate the adaptation of the first operation model 102 to a new environment.

[0070] Second Embodiment A robot motion learning method, a robot motion learning device, and a robot motion learning system according to a second embodiment of the present invention will be described with reference to FIGS.

[0071] 7 is a diagram showing a schematic configuration of a robot system 700 according to this embodiment. In this embodiment, the robot motion learning system 100 is used to learn the control model of the robot system 700 shown in FIG.

[0072] This robot system 700 is configured with substantially the same components as the robot system 200 of Example 1 (FIG. 2). However, instead of the fixed target portion 203, the base portion 204, the plurality of screws 205, and the screw feed device 206, the robot system 700 is configured with an object to be grasped 701 and a grasped object collection device 702. In addition, the fingers of the gripper 210 include contact sensors 703.

[0073] The object-to-be-grasped collection device 702 may be a container that an operator periodically places therein, or a mobile robot that transports the collected objects to be grasped 701 to another location when the container is full of the objects to be grasped 701. When mobile robots are used, multiple mobile robots are required, and it is assumed that when one robot becomes full of the objects to be grasped 701 and moves to another location, another mobile robot moves to the same location in its place.

[0074] The task performed by the robot system 700 is a pick-and-place task for palletizing, etc., and the operations required for this task will be described below. In this pick-and-place task, after the belt conveyor 202 transports the object to be grasped 701, the robot arm 201 grasps the object to be grasped 701, moves to the vicinity of an empty space on the grasped object collection device 702, and releases the object to be grasped 701.

[0075] The operation flow of this embodiment will be described with reference to Figures 8 and 9. Figure 8 is a flowchart showing the robot operation learning method of this embodiment, and shows a method for learning the first operation model 102 when the robot system 700 is used in a new environment or when a new part is used by picking and placing it. Figure 9 is a diagram showing a method for generating operations for learning.

[0076] The first behavior model 102 has been pre-trained in a previous environment similar to the current environment, and therefore, theoretically, it is assumed that it can be used in the current environment without re-training. The robot behavior learning method 800 of this embodiment begins with the behavior executing unit 106 attempting to grasp (pick) the object to be grasped 701 with the robot arm 201 using a behavior command from the first behavior model 102 (step S801).

[0077] If the robot arm 201 successfully grasps the object to be grasped 701 (step S802a), this means that the first behavior model 102 can grasp the object to be grasped 701 with the robot arm 201 without re-learning the first behavior model 102, and the learning flow is terminated.

[0078] On the other hand, if the robot arm 201 cannot grasp the object to be grasped 701 (step S802b), the operation execution unit 106 uses the operation commands of the second operation model 103, which can grasp the object to be grasped 701 with a higher success rate, to search for the position of the object to be grasped 701 using a contact sensor 703 or the like, and grasps the object to be grasped 701 (step S803).

[0079] If the second movement model 103 cannot grasp (find) the object to be grasped 701 (step S804b), the first movement model 102 cannot be re-learned, and the learning flow is terminated.

[0080] If the second movement model 103 is able to grasp the object to be grasped 701 (step S804a), the error calculation unit 104 detects the error between the movement from the first movement model 102 and the movement from the second movement model 103 (step S805).

[0081] Next, the collection range determination unit 105 determines the range of the learning movement data for training of the first movement model 102 that should be collected for task accomplishment based on the movement error (step S806).

[0082] Next, the learning motion generator 107 generates a learning motion and sends this motion command to the motion execution unit 106, which then moves the robot arm 201 based on this motion command (step S807).

[0083] This specific operation is shown in Fig. 9. After grasping the object to be grasped 701, the robot arm 201 moves the object to be grasped 701 along a random movement trajectory 902 into a data collection limit range 901 obtained from a maximum grasp position error, which is the difference between the position at which the object to be grasped 701 is grasped by the second operation model 103 and the position of the object to be grasped 701 predicted by the first operation model 102, and then releases the object to be grasped 701.

[0084] Next, the operation command and the environmental information are stored in the data collection unit 109 (step S808), and it is confirmed whether a predetermined target amount of data has been achieved (step S809).

[0085] If the target data amount has not been achieved (step S809b), the picking and moving operation (step S807) of the object to be grasped 701 is repeated. If the target data amount has been achieved (step S809a), the learning unit 110 re-learns the first operation model 102 using the data stored by the data collection unit 109 (step S810).

[0086] Finally, after re-learning the first behavior model 102, the first behavior model 102 is again used to grasp a new object to be grasped 701 (step S801), and this learning flow 800 is repeated until the first behavior model 102 is able to accomplish the task.

[0087] In addition, the method for finding the position of the object to be grasped 701 in the second operating model 103 of this embodiment in steps S803 and S804 is to move the gripper 210 to a height where it will come into contact with the object to be grasped 701, and then move the gripper 210 on a rule-based trajectory such as a spiral trajectory, and when the contact sensor 703 installed on the gripper 210 detects contact, calculate the position of the object to be grasped 701 based on the specified dimensions of the object to be grasped 701.

[0088] In this embodiment, it is necessary to set a target data amount, which may become enormous in some cases, but because the position of the object to be grasped 701 is limited by the data collection limit range 901 obtained from the maximum grip position error, the amount of data required is less than if the object to be grasped 701 were positioned over a wider range. In other words, by using the data collection limit range 901 obtained from the grip position error, the amount of training data can be minimized, enabling the reduction of training time and hardware costs, which are the goals of the present invention.

[0089] As described above, by using the robot motion learning device (learning support device) 101 of this embodiment, it is possible to learn the first motion model 102 for application to a new environment in a short time and with low hardware costs.

[0090] Third Embodiment A robot movement learning method, a robot movement learning device, and a robot movement learning system according to a third embodiment of the present invention will be described with reference to FIG.

[0091] FIG. 10 is a block diagram showing the configuration of a robot movement learning system 100 according to this embodiment, which shows yet another modification of the first embodiment (FIG. 1).

[0092] 10, a robot movement learning device (learning support device) 101 of this embodiment differs from Example 1 (FIG. 1) in that, in addition to the configuration of Example 1 (FIG. 1), it further includes a third movement model 111 that has a higher probability of success in the movement of the robot arm 201 compared to the first movement model 102 and the second movement model 103. The other configurations are the same as those of Example 1 (FIG. 1).

[0093] The error calculation unit 104 calculates the error between the first operating model 102 and the second operating model 103, and the error between the first operating model 102 and the third operating model 111, using the operation execution results of the first operating model 102, the operation execution results of the second operating model 103, and the operation execution results of the third operating model 111.

[0094] Then, if the error between the first operating model 102 and the third operating model 111 is greater than the error between the first operating model 102 and the second operating model 103, the collection range determination unit 105 uses the error between the first operating model 102 and the third operating model 111 to determine the collection range of learning data to be collected from the surrounding environment of the robot arm 201.

[0095] On the other hand, if the error between the first operating model 102 and the second operating model 103 is greater than the error between the first operating model 102 and the third operating model 111, the collection range determination unit 105 uses the error between the first operating model 102 and the second operating model 103 to determine the collection range of learning data to be collected from the surrounding environment of the robot arm 201.

[0096] As in this embodiment, by using a plurality of models to compare with the first operation model 102, it becomes possible to re-learn the first operation model 102 with a higher success rate.

[0097] In the above examples, the differences between the behavior models were explained based on the probability of success of the robot's behavior, but each behavior model may also be created based on other parameters besides success probability, such as the length of time the robot takes to move.

[0098] Furthermore, the present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0099] 100...Robot motion learning system, 101...Robot motion learning device (learning support device), 102...First motion model, 103...Second motion model, 104...Error calculation unit, 105...Collection range determination unit, 106...Movement execution unit, 107...Learning motion generation unit, 108...Environment data acquisition unit, 109...Data collection unit, 110...Learning unit, 111...Third motion model, 200, 700...Robot system, 201...Robot arm, 202...Belt conveyor, 203...Fixed target part, 204...Base part, 205...Screw, 206...Screw Feeding device, 207...camera, 208...camera fixing part, 209...force sensor, 210...gripper, 300, 600, 800...robot movement learning method, 400...data collection method for imitation learning, 401...hole, 402...maximum distance error, 403...sliding movement trajectory, 500...data collection method for reinforcement learning, 501...hole, 502...data collection position, 503...data collection limit position, 701...object to be grasped, 702...grasped object collection device, 703...contact sensor, 901...data collection limit range, 902...random movement trajectory.

Claims

1. A robot motion learning device that assists a robot in learning to execute movements, comprising: a certain behavior model; another behavior model; an error calculation unit that calculates an error between the certain behavior model and the other behavior model using a movement execution result of the certain behavior model and a movement execution result of the other behavior model; a collection range determination unit that determines a collection range of learning data to be collected from the surrounding environment of the robot using the error calculated by the error calculation unit; a data collection unit that collects learning data using the collection range determined by the collection range determination unit; and a learning unit that learns at least one of the certain behavior model and the other behavior model using the learning data collected by the data collection unit.

2. A robot movement learning device as described in claim 1, wherein the one movement model is a first movement model, the other movement model is a second movement model having a higher probability of success in the robot's movement compared to the first movement model, the error calculation unit calculates the error between the first movement model and the second movement model using the movement execution results of the first movement model and the movement execution results of the second movement model, and the learning unit learns the first movement model based on the learning data collected by the data collection unit.

3. A robot motion learning device as described in claim 2, characterized in that the error calculation unit calculates the difference between the motion position when executed according to the first motion model and the motion position when executed according to the second motion model as the error.

4. A robot motion learning device as described in claim 3, characterized in that the collection range determination unit determines, as the collection range, an area of ​​the surrounding environment in the vicinity of where the error occurred when the robot executed an action.

5. A robot motion learning device as described in claim 4, characterized in that the data collection unit has a learning data collection function for collecting data related to the robot's motion within the collection range determined by the collection range determination unit.

6. A robot motion learning device as described in claim 4, characterized in that the collection range is a circular area, the motion position obtained by the second motion model is the center of the area, and the error is the radius of the area.

7. A robot motion learning device as described in claim 6, characterized in that the data collected for learning the first motion model is environmental information and motion information obtained when moving from a position away from the center of the area by the error distance to the center of the area.

8. A robot motion learning device as set forth in claim 7, characterized in that the learning unit re-learns the first motion model by imitation learning using the collected learning data.

9. A robot motion learning device as described in claim 4, wherein the collection range is a rectangular area, the motion position obtained by the second motion model is the center of the area, and the error defines the limits of the area.

10. A robot motion learning device as described in claim 9, wherein the data obtained from the collection range is a plurality of discretized points within the collection range, and the learning unit constructs a reinforcement learning environment using a plurality of the discretized points, and re-learns the first motion model using the reinforcement learning environment.

11. A robot motion learning device according to claim 4, wherein the second motion model is a rule-based model.

12. A robot motion learning device as described in claim 11, wherein the second motion model is utilized by the robot to insert a rod-shaped member into a hole, and the robot arm of the robot inserts the rod-shaped member into the hole by sliding the rod-shaped member around the hole on a trajectory that covers the entire fixed search area.

13. A robot motion learning device as described in claim 11, wherein the second motion model is utilized for the robot to pick up an object, and the robot moves the gripper of the robot in the air on a trajectory that covers an entire fixed search area, and detects the position of the object when contact with the object is detected by a contact sensor installed on the gripper.

14. A robot motion learning device as described in claim 2, further comprising a third motion model having a higher probability of success in the robot's motion compared to the first motion model and the second motion model, and the error calculation unit calculates the error between the first motion model and the second motion model and the error between the first motion model and the third motion model using the motion execution results of the first motion model, the motion execution results of the second motion model, and the motion execution results of the third motion model.

15. A robot motion learning method for assisting a robot in learning to execute movements, comprising: an error calculation step for calculating an error between a certain behavior model and the other behavior model using a motion execution result of the certain behavior model and a motion execution result of the other behavior model; a collection range determination step for determining a collection range of learning data to be collected from the robot's surrounding environment using the error; a data collection step for collecting learning data using the collection range; and a learning step for learning at least one of the certain behavior model and the other behavior model using the learning data collected in the data collection step.

16. A robot motion learning system comprising a robot motion learning device as defined in any one of claims 1 to 14, comprising a robot, the robot having: a motion execution unit that executes motions based on motion commands according to a motion model re-learned by the robot motion learning device; and an environmental data acquisition unit that acquires environmental information obtained by the motion execution by the motion execution unit.

17. A robot incorporating a robot motion learning device as claimed in any one of claims 1 to 14, comprising: a motion execution unit that executes motions based on motion commands from a motion model re-learned by said robot motion learning device; and an environmental data acquisition unit that acquires environmental information obtained by the motion execution by said motion execution unit.

Images