Robot model learning device, robot model machine learning method, robot control device, robot control method, and storage medium

Abstract

A learning device for a robot model obtains the actual value of the robot's position and posture and the actual value of the external force applied to the robot, executes a state transition model that calculates a predicted value of the robot's position and posture based on the actual value of the position and posture in a certain period of time and the action instructions that can be provided to the robot, and an external force model that calculates a predicted value of the external force applied to the robot. The robot model calculates a reward based on the error in the position and posture and the predicted value of the external force. For each control cycle, multiple candidates for the action instruction are generated and provided to the robot model. The action instruction that maximizes the reward is determined based on the rewards calculated corresponding to the multiple candidates for the action instruction, and the external force model is updated so that the difference between the predicted value of the external force calculated by the external force model based on the determined action instruction and the actual value of the external force corresponding to the predicted value of the external force becomes smaller.

Description

Technical Field

[0001] The disclosed technology relates to a learning device for a robot model, a machine learning method for a robot model, a storage medium storing a machine learning program for a robot model, a robot control device, a robot control method, and a storage medium storing a robot control program. Background Art

[0002] In order to automatically obtain the control rules required for the robot to complete the task, the robot model is learned through machine learning.

[0003] For example, in Japanese Patent Gazette No. 2020-055095, a control device for controlling an industrial robot having a function of detecting the force and torque applied to the manipulator is disclosed, comprising: a control unit, which controls the industrial robot based on a control instruction, a data acquisition unit, which acquires at least one of the force and torque applied to the manipulator of the industrial robot as acquired data, and a preprocessing unit, which generates force state data representing information related to the force applied to the manipulator and control instruction adjustment data of the adjustment action of the control instruction related to the manipulator as state data based on the acquired data, and a technology for performing machine learning processing related to the adjustment action of the control instruction related to the manipulator based on the state data. Summary of the Invention

[0004] Technical problem to be solved by the invention

[0005] However, when learning a robot model through machine learning, setting parameters and designing reward functions are difficult, making efficient learning difficult.

[0006] The disclosed technology is developed in view of the above-mentioned points, and aims to provide a learning device for a robot model that can effectively learn a robot model when learning a robot model through machine learning, a machine learning method for a robot model, a storage medium storing a machine learning program for a robot model, a robot control device, a robot control method, and a storage medium storing a robot control program.

[0007] Solutions for solving technical problems

[0008] The first disclosed method is a learning device for a robot model, comprising: an acquisition unit, which acquires actual values ​​of the position and posture of a robot and actual values ​​of external forces applied to the robot; a robot model, wherein the robot model includes a state transition model for calculating a predicted value of the position and posture of the robot in a next period of time based on the actual values ​​of the position and posture in a certain period of time and an action instruction that can be provided to the robot, and an external force model for calculating a predicted value of the external force applied to the robot; a model execution unit, which executes the robot model; a reward calculation unit, which calculates a reward based on an error between the predicted value of the position and posture and a target value of the position and posture to be achieved, and the predicted value of the external force; an action determination unit, which generates multiple candidates of the action instruction for each control cycle and provides them to the robot model, and determines an action instruction that maximizes the reward based on the rewards calculated by the reward calculation unit corresponding to the multiple candidates of the action instruction; and an external force model updating unit, which updates the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action instruction and the actual value of the external force corresponding to the predicted value of the external force becomes smaller.

[0009] According to the above-mentioned first method, it can also be constructed to have a state transition model updating unit, which updates the state transition model so that the error between the predicted value of the position posture calculated by the state transition model based on the determined action instruction and the actual value of the position posture corresponding to the predicted value of the position posture becomes smaller.

[0010] According to the first aspect, when the external force is a correction external force that suppresses expansion of the error, the reward calculation unit may calculate the reward by calculating a predicted value of the correction external force as a reduction factor for the reward.

[0011] According to the first aspect, when the external force is a counter external force that suppresses reduction of the error, the reward calculation unit may calculate the reward by calculating a predicted value of the counter external force as an increasing factor of the reward.

[0012] According to the first method mentioned above, the reward calculation unit may also calculate the reward by using the predicted value of the corrected external force as a reduction factor of the reward when the external force is a correction external force that suppresses the expansion of the error, and calculate the reward by using the predicted value of the counteracting external force as an increase factor of the reward when the external force is an external force that suppresses the reduction of the error, i.e., a counteracting external force.

[0013] According to the above-mentioned first method, the reward calculation unit can also calculate the reward by calculating that the change in the decrease in the reward based on the predicted value of the corrected external force during task execution is smaller than the change in the reward based on the error, and the change in the increase in the reward based on the predicted value of the opposing external force during task execution is smaller than the change in the reward based on the error.

[0014] According to the above-mentioned first method, the external force model can also be constructed to include: a correction external force model that outputs a predicted value of the correction external force when the external force is the correction external force, and an opposing external force model that outputs a predicted value of the opposing external force when the external force is the opposing external force, and the external force model updating unit includes: a correction external force model updating unit that updates the correction external force model so that when the external force is the correction external force, the difference between the predicted value of the correction external force calculated by the correction external force model based on the determined action instruction and the actual value of the external force becomes smaller; and an opposing external force model updating unit that updates the opposing external force model so that when the external force is the opposing external force, the difference between the predicted value of the opposing external force calculated by the opposing external force model based on the determined action instruction and the actual value of the external force becomes smaller.

[0015] According to the first method mentioned above, the robot model can also include an integrated external force model including the correction external force model and the counter-external force model, and the correction external force model and the counter-external force model are neural networks. One or more intermediate layers and at least one of the output layers of the counter-external force model integrate the output of the previous layer of the corresponding layer of the correction external force model through the progressive neural network method. The counter-external force model outputs the predicted value of the external force and the identification information of whether the external force is a correction external force or a counter-external force. The integrated external force model uses the output of the counter-external force model as its own output. The reward calculation unit calculates the reward by using the predicted value of the external force as a reduction factor of the reward when the identification information is represented by the correction external force, and calculates the reward by using the predicted value of the external force as an increase factor of the reward when the identification information is represented by the counter-external force.

[0016] According to the first method mentioned above, it can also be a structure that further includes an acceptance unit for accepting designation of whether the external force is the correction external force or the counteracting external force, and further includes a learning control unit, wherein the learning control unit validates the action of the correction external force model updating unit when the external force is designated as the correction external force, and the learning control unit validates the action of the counteracting external force model updating unit when the external force is designated as the counteracting external force.

[0017] According to the above-mentioned first method, it can also be constructed to further include a learning control unit, which determines whether the external force is the correction external force or the counteracting external force based on the actual value of the position posture and the actual value of the external force. When the result of the determination is the correction external force, the action of the correction external force model updating unit is validated; when the result of the determination is the counteracting external force, the action of the counteracting external force model updating unit is validated.

[0018] A second disclosed method is a machine learning method for a robot model. The method includes preparing a robot model including a state transition model for calculating a predicted value of the robot's position and posture for a next time period based on actual values ​​of the robot's position and posture for a certain time period and an action command that can be provided to the robot, and an external force model for calculating a predicted value of an external force applied to the robot. The method obtains the actual values ​​of the position and posture and the actual values ​​of the external force applied to the robot for each control cycle. For each control cycle, multiple candidates for the action command are generated and provided to the robot model. Based on multiple errors between multiple predicted values ​​of the position and posture calculated by the state transition model for the multiple candidate action commands and target values ​​of the position and posture to be achieved, and multiple predicted values ​​of the external force calculated by the external force model for the multiple candidate action commands, an action command that maximizes the reward is determined based on multiple rewards calculated for the multiple candidate action commands. The method then updates the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action command and the actual value of the external force corresponding to the predicted value of the external force is reduced.

[0019] According to the second method described above, the state transition model may be further updated so that the error between the predicted value of the position posture calculated by the state transition model based on the determined action instruction and the actual value of the position posture corresponding to the predicted value of the position posture becomes smaller.

[0020] According to the second aspect, when the external force is a correction external force that suppresses expansion of the error, the reward may be calculated by calculating a predicted value of the correction external force as a reduction factor for the reward.

[0021] According to the second aspect, when the external force is an external force that suppresses reduction of the error, that is, an opposing external force, the reward may be calculated by calculating a predicted value of the opposing external force as an increasing factor of the reward.

[0022] According to the second method described above, when the external force is a correction external force that suppresses the expansion of the error, the reward can be calculated by using the predicted value of the correction external force as a reduction factor of the reward; and when the external force is an external force that suppresses the reduction of the error, that is, an opposing external force, the reward can be calculated by using the predicted value of the opposing external force as an increase factor of the reward.

[0023] According to the above-mentioned second method, the external force model may also include: a correction external force model that outputs a predicted value of the correction external force when the external force is the correction external force, and an opposing external force model that outputs a predicted value of the opposing external force when the external force is the opposing external force. The correction external force model is updated so that when the external force is the correction external force, the difference between the predicted value of the correction external force calculated by the correction external force model based on the determined action instruction and the actual value of the external force becomes smaller. The opposing external force model is updated so that when the external force is the opposing external force, the difference between the predicted value of the opposing external force calculated by the opposing external force model based on the determined action instruction and the actual value of the external force becomes smaller.

[0024] According to the second aspect, the correction external force may be applied to the robot when the error is increasing, and the counteracting external force may be applied to the robot when the error is decreasing.

[0025] A third disclosed method is a machine learning program for a robot model, which is a machine learning program for a robot model that performs machine learning, including a state transition model that calculates a predicted value of the robot's position and posture for a next period of time based on actual values ​​of the robot's position and posture for a certain period of time and action commands that can be provided to the robot, and an external force model that calculates a predicted value of an external force applied to the robot. The program causes a computer to perform the following processing: for each control cycle, obtaining the actual value of the position and posture and the actual value of the external force applied to the robot; for each control cycle, generating multiple candidates for the action command and providing them to the robot model; determining an action command that maximizes the reward based on multiple errors between multiple predicted values ​​of the position and posture calculated by the state transition model corresponding to the multiple candidate action commands and target values ​​of the position and posture to be achieved, and multiple predicted values ​​of the external force calculated by the external force model corresponding to the multiple candidate action commands; and updating the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action command and the actual value of the external force corresponding to the predicted value of the external force is reduced.

[0026] The fourth disclosed method is a robot control device, comprising: a model execution unit, which executes a state transition model including the actual value of the position and posture of the robot in a certain period of time and the action instructions that can be provided to the robot, calculates the predicted value of the position and posture of the robot in the next period of time, and calculates the predicted value of the external force applied to the robot; an acquisition unit, which acquires the actual value of the position and posture of the robot and the actual value of the external force applied to the robot; a reward calculation unit, which calculates the reward based on the error between the predicted value of the position and posture calculated by the robot model and the target value of the position and posture to be achieved, and the predicted value of the external force calculated by the robot model; and an action determination unit, which generates multiple candidates of the action instruction for each control cycle and provides them to the robot model, and determines the action instruction that maximizes the reward based on the rewards calculated by the reward calculation unit corresponding to the multiple candidates of the action instruction.

[0027] The fifth disclosed method is a robot control method, which prepares a robot model, wherein the robot model includes a state transition model for calculating a predicted value of the robot's position and posture in a next time period based on the actual value of the robot's position and posture in a certain time period and an action instruction that can be provided to the robot, and an external force model for calculating a predicted value of the external force applied to the robot. For each control cycle, the actual value of the position and posture and the actual value of the external force applied to the robot are obtained. For each control cycle, multiple candidates of the action instruction are generated and provided to the robot model. Based on multiple errors between multiple predicted values ​​of the position and posture calculated by the state transition model corresponding to the multiple candidates of the action instruction and target values ​​of the position and posture to be achieved, and multiple predicted values ​​of the external force calculated by the external force model corresponding to the multiple candidates of the action instruction, an action instruction that maximizes the reward is determined based on multiple rewards calculated corresponding to the multiple candidates of the action instruction, and the robot is controlled based on the determined action instruction.

[0028] The sixth disclosed method is a robot control program, which is a program for controlling the robot using a robot model that includes a state transition model that calculates a predicted value of the position and posture of the robot in the next time period based on the actual value of the robot's position and posture in a certain time period and an action instruction that can be provided to the robot, and an external force model that calculates a predicted value of the external force applied to the robot, so that the computer performs the following processing: for each control cycle, obtain the actual value of the position and posture and the actual value of the external force applied to the robot, for each control cycle, generate multiple candidates of the action instruction and provide them to the robot model, based on multiple errors between the multiple predicted values ​​of the position and posture calculated by the state transition model corresponding to the multiple candidates of the action instruction and the target value of the position and posture to be achieved, and the multiple predicted values ​​of the external force calculated by the external force model corresponding to the multiple candidates of the action instruction, determine the action instruction that maximizes the reward based on multiple rewards calculated corresponding to the multiple candidates of the action instruction, and control the robot based on the determined action instruction.

[0029] Effects of the Invention

[0030] According to the disclosed technology, when learning a robot model through machine learning, efficient learning is possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a configuration diagram of a robot system according to the first embodiment.

[0032] Figure 2A 1 is a diagram showing a schematic configuration of the robot 10 .

[0033] Figure 2B This is an enlarged view of the front end side of the robot arm.

[0034] Figure 3 This is a diagram for explaining the correction external force and the counter external force.

[0035] Figure 4 This is a block diagram showing the hardware configuration of a robot control device.

[0036] Figure 5 This is a diagram of the robot model.

[0037] Figure 6 This is a diagram showing the configuration of a robot model according to a modification example.

[0038] Figure 7 This is a diagram showing the configuration of a robot model according to a modification example.

[0039] Figure 8 It is a diagram of the structure of the integrated external force model.

[0040] Figure 9 This is a flowchart of the learning process according to the first embodiment.

[0041] Figure 10 This is a flowchart of the learning process according to the second embodiment.

[0042] Figure 11 This is a configuration diagram of a robot system according to a third embodiment.

[0043] Figure 12 This is a flowchart of the learning process according to the third embodiment.

[0044] Figure 13A is a graph showing a learning curve with respect to position error.

[0045] Figure 13B is a graph showing a learning curve with respect to external force.

[0046] Figure 14A is a graph showing the number of successes of tasks under different frictions.

[0047] Figure 14B It is a graph showing the number of successful tasks for different pin qualities.

[0048] Figure 15 This is a graph showing the number of successful tasks for different pin materials. DETAILED DESCRIPTION

[0049] An example of an embodiment of the disclosed technology is described below with reference to the accompanying drawings. It should be noted that identical or equivalent components and parts in the various drawings are denoted by the same reference numerals. Furthermore, for ease of description, the dimensional ratios in the drawings may be exaggerated and may differ from the actual ratios.

[0050] <First embodiment>

[0051] Figure 1 The configuration of a robot system 1 according to this embodiment is shown. The robot system 1 includes a robot 10 , a state observation sensor 20 , tactile sensors 30A and 30B, a robot control device 40 , a display device 50 , and an input device 60 .

[0052] (robot)

[0053] Figure 2A 、 Figure 2Bis a diagram schematically illustrating the structure of robot 10. In this embodiment, robot 10 is a six-axis vertical multi-jointed robot. A clamp (hand) 12 is provided at the distal end 11a of arm 11 via a flexible portion 13. Robot 10 uses clamp 12 to grasp parts (e.g., pins) and insert them into holes. It should be noted that while robot 10 is a real robot in this embodiment, it could also be a virtual robot in a simulation.

[0054] like Figure 2A As shown, the robot 10 has an arm 11 with six degrees of freedom, each having joints J1 to J6. Each joint J1 to J6 is connected to each other by a motor (not shown) so that the connecting rods can rotate in the directions of arrows C1 to C6. Here, a vertical multi-joint robot is used as an example, but a horizontal multi-joint robot (scalar robot) can also be used. In addition, a six-axis robot is used as an example, but a multi-joint robot with other degrees of freedom, such as five axes or seven axes, can also be used, or a parallel link robot can be used.

[0055] The clamp 12 has a set of clamping parts 12a, which control the clamping parts 12a to clamp the parts. The clamp 12 is connected to the front end 11a of the arm 11 via the soft part 13, and moves as the arm 11 moves. In this embodiment, the soft part 13 is composed of three springs 13a to 13c arranged in a positional relationship in which the base of each spring becomes the vertices of an equilateral triangle, but the number of springs can be arbitrary. In addition, the soft part 13 can be any other mechanism as long as it generates a restoring force to obtain flexibility in response to changes in position. For example, the soft part 13 can also be an elastic body such as a spring or rubber, a damper, an air pressure or hydraulic cylinder, etc. The soft part 13 is preferably composed of a passive element. Through the soft part 13, the front end 11a of the arm 11 and the clamp 12 are configured to be able to move relative to each other by more than 5 mm in the horizontal direction and the vertical direction, preferably more than 1 cm, and more preferably more than 2 cm.

[0056] A mechanism for switching the clamp 12 between a flexible state and a fixed state relative to the arm 11 may be provided.

[0057] In addition, although the configuration shown here is such that the soft portion 13 is provided between the distal end 11a of the arm 11 and the clamp 12, the soft portion 13 may also be provided midway through the clamp 12 (e.g., at a finger joint or midway through the columnar portion of a finger) or midway through the arm (e.g., at any of the joints J1 to J6 or midway through the columnar portion of the arm). Furthermore, the soft portion 13 may be provided at multiple locations among these.

[0058] As described above, the robot system 1 uses machine learning (e.g., model-based reinforcement learning) to obtain a robot model for controlling the robot 10 having a flexible portion 13. The presence of the flexible portion 13 ensures safety even when the gripped part comes into contact with the environment, and also enables insertion operations, even with a slow control cycle. On the other hand, the position of the fixture 12 and the part is uncertain due to the flexible portion 13, making it difficult to obtain an analytical robot model. Therefore, in this embodiment, the robot model is obtained using machine learning.

[0059] (State Observation Sensor)

[0060] The state observation sensor 20 observes the position and posture of the fixture 12 as the state of the robot 10 and outputs the observed position and posture as an actual value. Examples of the state observation sensor 20 include encoders for the robot 10's joints, visual sensors (cameras), and motion capture systems. By attaching a motion capture marker to the fixture 12, the position and posture of the fixture 12 can be determined, and the posture of the part (work object) can be estimated based on the position and posture of the fixture 12.

[0061] Furthermore, the position and posture of the gripper 12 itself and the part held by the gripper 12 can also be detected as the state of the robot 10 using a visual sensor. If there is a soft portion between the gripper 12 and the arm 11, the position and posture of the gripper 12 relative to the arm 11 can be determined using a displacement sensor that detects the displacement of the gripper 12 relative to the arm 11.

[0062] (Tactile sensor)

[0063] Although not shown in FIG2 , Figure 3 As shown, the jig body 12 b of the jig 12 is provided with touch sensors 30A and 30B.

[0064] As an example, the tactile sensors 30A and 30B are positioned along a pair of opposing gripping portions 12a. As an example, the tactile sensors 30A and 30B detect forces in three or six axes, and can detect the magnitude and direction of external forces applied to them. The user grasps the gripper body 12b and moves the gripper 12, applying external force to the gripper 12, causing their hand (fingers) to contact both the tactile sensors 30A and 30B.

[0065] External forces include advisory forces that cause robot 10 to succeed in its task (operation) and adversarial forces that cause it to fail. The advisory forces are external forces that suppress the increase in the error between the predicted values ​​of the robot's 10 position and posture, as predicted by the robot model, and the target values ​​of the position and posture that robot 10 should achieve. Furthermore, adversarial forces are external forces that reduce the increase in the error between the predicted values ​​of the robot's 10 position and posture, as predicted by the robot model, and the target values ​​of the position and posture that robot 10 should achieve.

[0066] Specifically, the tasks performed by the robot 10 are as follows: Figure 3 In the case of the task of inserting a pin 70 into a hole 74 provided in a stage 72, it is necessary to move the pin 70 in the direction of arrow A in order to insert the pin 70 into the hole 74. In this case, the external force applied in the direction of arrow A, which is the correct direction for inserting the pin 70 into the hole 74, is the corrective external force. On the other hand, the external force applied in the direction of arrow B, which is the opposite direction of arrow A and causes the task to fail, is the counteracting external force.

[0067] exist Figure 3 In this case, if the user applies a correction external force in the direction of arrow A by grasping the jig 12, the force detected by the tactile sensor 30A becomes larger than that detected by the tactile sensor 30B, and it can be determined that a correction external force has been applied. On the other hand, if an opposing external force is applied in the direction of arrow B, the force detected by the tactile sensor 30B becomes larger than that detected by the tactile sensor 30A, and it can be determined that an opposing external force has been applied.

[0068] It should be noted that in this embodiment, two tactile sensors 30A and 30B are provided on the clamp body 12b, but the present invention is not limited to this. For example, three or more tactile sensors may be provided at equal intervals around the clamp body 12b. If three or more tactile sensors are provided and their detection results are combined, each tactile sensor may only detect the magnitude of the external force, provided that at least the direction of the external force in a plane perpendicular to the axis of the clamp 12 is known.

[0069] (Robot control device)

[0070] The robot control device 40 functions as a learning device that learns a robot model through machine learning. In addition, the robot control device 40 also functions as a control device that controls the robot 10 using the learned robot model.

[0071] Figure 4 : is a block diagram showing the hardware configuration of the robot control device according to this embodiment. Figure 4As shown, the robot control device 40 has the same configuration as a general computer (information processing device), and includes a CPU (Central Processing Unit) 40A, ROM (Read Only Memory) 40B, RAM (Random Access Memory) 40C, memory 40D, keyboard 40E, mouse 40F, monitor 40G, and communication interface 40H. These components are interconnected for communication via a bus 40I.

[0072] In this embodiment, programs for machine learning robot models and robot control programs are stored in ROM 40B or memory 40D. CPU 40A is a central processing unit that executes various programs and controls various components. Specifically, CPU 40A reads programs from ROM 40B or memory 40D and uses RAM 40C as a work area to execute the programs. CPU 40A controls the aforementioned components and performs various calculations based on the programs recorded in ROM 40B or memory 40D. ROM 40B stores various programs and data. RAM 40C temporarily stores programs or data as a work area. Memory 40D is composed of an HDD (Hard Disk Drive), an SSD (Solid State Drive), or flash memory, and stores various programs, including the operating system, and various data. Keyboard 40E and mouse 40F are examples of input devices 60 and are used for various inputs. Monitor 40G, such as a liquid crystal display, is an example of display device 50. Monitor 40G can also use a touch panel to function as input device 60. The communication interface 40H is an interface for communicating with other devices, and uses a standard such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark), for example.

[0073] Next, the functional configuration of the robot control device 40 will be described.

[0074] like Figure 1 As shown, the robot control device 40 includes, as its functional components, an acquisition unit 41, a model execution unit 42, a reward calculation unit 43, an action determination unit 44, an external force model update unit 45, a learning control unit 46, and a user interface (UI) control unit 47. Each functional component is implemented by CPU 40A reading a machine learning program stored in ROM 40B or memory 40D, expanding it in RAM 40C, and executing it. It should be noted that some or all of the functions may also be implemented using dedicated hardware devices.

[0075] The acquisition unit 41 acquires the actual value of the position and posture of the robot 10 and the actual value of the external force applied to the robot 10. The position and posture of the robot 10, as an example, is the position and posture of the clamp 12 as the end effector of the robot 10. The external force applied to the robot 10, as an example, is the external force applied to the clamp 12 as the end effector of the robot 10. The actual value of the external force is measured by the tactile sensors 30A and 30B. It should be noted that in the case where it is difficult to determine which part of the robot 10 is the end effector, from the perspective of the part of the robot that affects the operation object, it is sufficient to determine the part where the position and posture are appropriately measured and the part where the external force is applied.

[0076] In this embodiment, since the clamp 12 is provided at the distal end 11a of the arm 11 via a flexible portion 13, it is preferable that the clamp 12 be physically and flexibly displaced when an external force is applied, or that the displacement be controlled in response to the external force. It should be noted that the disclosed technology is applicable even when an external force is applied manually to a rigid robot that lacks flexibility.

[0077] In this embodiment, the position and posture of the robot 10 are expressed as values ​​of a maximum of six degrees of freedom, including three degrees of freedom in position and three degrees of freedom in posture. However, it may be expressed as fewer degrees of freedom depending on the movable degrees of freedom of the robot 10. For example, in the case of a robot that does not produce posture changes in the end effector, the "position and posture" may be expressed as only three degrees of freedom in position.

[0078] The model execution unit 42 executes the robot model LM.

[0079] like Figure 5 As shown, the robot model LM includes a state transition model DM that calculates the predicted value of the position and posture of the robot 10 in the next time period based on the actual value (measured value) of the position and posture in a certain time period and the action instructions (candidate value or confirmed value) that can be provided to the robot 10, and an external force model EM that calculates the predicted value of the external force applied to the robot 10.

[0080] It should be noted that the robot model LM is “based on” (=input) and “calculated” (=output) means that when the model execution unit 42 executes the model, the model is executed using input data, and output data is calculated (generated) by executing the model.

[0081] The external force model EM includes a correction external force model EM1 that outputs a predicted value of the correction external force and a counter external force model EM2 that outputs a predicted value of the counter external force.

[0082] The reward calculation unit 43 calculates the reward based on the error between the predicted position and posture value and the target position and posture value to be achieved, as well as the predicted value of the external force. The target position and posture to be achieved can be the position and posture to be achieved when the task is completed, or it can be the position and posture as an intermediate goal before the task is completed.

[0083] When the external force is a correction external force that suppresses expansion of the error, the reward calculation unit 43 calculates the reward by calculating the predicted value of the correction external force as a reduction factor of the reward.

[0084] The external force that suppresses the expansion of the error, that is, the correction external force, may be an external force that slows down the expansion speed when the position and posture error gradually expands, and may not be an external force that turns the expansion of the error into a reduction.

[0085] Calculating the reward using the predicted value of the corrected external force as a factor in reducing the reward means that the reward calculated using the predicted value of the corrected external force calculated by the external force model is smaller than the reward calculated using the predicted value of the corrected external force as 0. It should be noted that reduction does not necessarily mean a decrease in time. Even if the predicted value of the corrected external force calculated by the corrected external force model EM1 is used in the calculation, the reward does not necessarily decrease over time.

[0086] Even when the position and posture errors gradually increase, it is preferable to Figure 3 A correction force is applied when the position and posture error is large (when the center pin 70 is widely separated from the hole 74), and no correction force is applied when the position and posture error is small (when the pin 70 is near the hole 74). In addition, when the position and posture error is large, it is preferred that a larger correction force be applied as the speed at which the position and posture error gradually increases.

[0087] Furthermore, the reward calculation unit 43 calculates the reward by calculating that the change width of the reduction amount of the reward based on the predicted value of the corrected external force during task execution is smaller than the change width of the reward based on the error.

[0088] Furthermore, when the external force is an external force that suppresses reduction of the error, that is, an opposing external force, the reward calculation unit 43 calculates the reward by calculating the predicted value of the opposing external force as an increasing factor of the reward.

[0089] The external force that suppresses the reduction of the error, that is, the counteracting external force, may be an external force that slows down the reduction speed when the position and posture error gradually decreases, and may not be an external force that turns the reduction of the error into expansion.

[0090] Calculating the reward using the predicted value of the external antagonist force as a factor in increasing the reward means that the reward is greater when the predicted value of the external antagonist force calculated by the external antagonist force model is used, compared to the reward when the predicted value of the external antagonist force is set to 0. It should be noted that "increase" does not necessarily mean an increase over time. Even if the predicted value of the external antagonist force calculated by the external antagonist force model EM2 is used in the calculation, the reward does not necessarily increase over time.

[0091] Even when the position and attitude errors gradually decrease, it is preferable to Figure 3 When the center pin 70 is near the hole 74, an opposing external force is applied. When the position and posture error is large, no opposing external force may be applied. In addition, preferably, when the position and posture error is small, the faster the speed at which the position and posture error gradually decreases, the greater the opposing external force is applied.

[0092] Furthermore, the reward calculation unit 43 calculates the reward by calculating that the change width of the increase in the reward based on the predicted value of the resisting external force during task execution is smaller than the change width of the reward based on the error.

[0093] The action determination unit 44 generates multiple action command candidates for each control cycle and provides them to the robot model LM. Based on the rewards calculated by the reward calculation unit 43 corresponding to each of the multiple action command candidates, the action command that maximizes the reward is determined.

[0094] In this embodiment, the action command is a speed command, but it may also be a position command, a torque command, or a combination of speed, position, and torque commands. Furthermore, the action command may be a series of action commands spanning multiple time periods. Furthermore, multiple candidate action commands may be multiple series of candidate action commands.

[0095] Maximizing the reward means that as long as it is maximized as a result of exploration within a limited time, the reward does not need to be the actual maximum value under the circumstances.

[0096] The external force model updating unit 45 updates the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action command and the actual value of the external force corresponding to the predicted value of the external force becomes smaller.

[0097] The external force model updating unit 45 includes a corrected external force model updating unit 45A and a counter external force model updating unit 45B.

[0098] The corrected external force model updating unit 45A updates the corrected external force model EM1 so that the difference between the predicted value of the corrected external force calculated by the corrected external force model EM1 based on the action command determined by the action determination unit 44 and the actual value of the corrected external force becomes smaller.

[0099] The counter-external force model updating unit 45B updates the counter-external force model EM2 so that the difference between the predicted counter-external force value calculated by the counter-external force model EM2 based on the action command determined by the action determination unit 44 and the actual value of the counter-external force becomes smaller.

[0100] Based on the actual values ​​of the position and posture and the actual values ​​of the external force, the learning control unit 46 determines whether the external force is a correction force or a counterforce. If the result of the determination is a correction force, the operation of the correction external force model updating unit 45A is enabled. If the result of the determination is a counterforce, the operation of the counterforce model updating unit 45B is enabled. Furthermore, if the result of the determination is not a correction force, the operation of the correction external force model updating unit 45A is disabled. If the result of the determination is not a counterforce, the operation of the counterforce model updating unit 45B is disabled.

[0101] It should be noted that, in this embodiment, the situation in which the learning control unit 46 automatically determines whether the external force is a correction external force or a counteracting external force based on the actual value of the position posture and the actual value of the external force is described. The learning control unit 46 may also be configured to include a designated acceptance unit for accepting whether the external force is a correction external force or a counteracting external force.

[0102] In this case, the user operates the input device 60 to designate whether the external force applied to the jig 12 is a correction force or a counteracting force, and then applies the external force to the jig 12 .

[0103] Then, when a correction external force is specified, the learning control unit 46 enables the operation of the correction external force model updating unit 45A, and when a counter external force is specified, the learning control unit 46 enables the operation of the counter external force model updating unit 45B. Furthermore, when a correction external force is not specified, the learning control unit 46 disables the operation of the correction external force model updating unit 45A, and when a counter external force is not specified, the learning control unit 46 disables the operation of the counter external force model updating unit 45B.

[0104] It should be noted that in Figure 5 In the example, the state transition model DM, the correction external force model EM1, and the counter external force model EM2 are independent models, but the structure of the robot model LM is not limited to this. Figure 6 The robot model LM1 shown can also be composed of a common component CM, a state transition model-specific component DMa, a modified external force model-specific component EM1a, and a counter-external force model-specific component EM2a. In this case, the common component CM performs common processing for the state transition model DM, the modified external force model EM1, and the counter-external force model EM2. The modified external force model-specific component EM1a performs processing specific to the modified external force model EM1. The counter-external force model-specific component EM2a performs processing specific to the counter-external force model EM2.

[0105] In addition, if Figure 7 The robot model LM2 shown can also be constructed to include Figure 5 In this case, the integrated external force model IM outputs a predicted value of the external force and also outputs identification information for identifying whether the external force is a correction external force or a counter-external force.

[0106] The integrated external force model IM can integrate the modified external force model EM1 and the counter-force model EM2 using a progressive neural network approach. In this case, the modified external force model EM1 and the counter-force model are constructed using neural networks. Then, one or more intermediate layers and at least one of the output layers of the counter-force model EM2 are integrated with the output of the preceding layer of the corresponding layer of the modified external force model EM1 using a progressive neural network (PNN) approach.

[0107] exist Figure 8 In this example, the output layer OUT2 of the counteracting external force model EM2 inputs the output of the intermediate layer MID2A, which is the layer preceding the corresponding output layer OUT1 of the corrected external force model EM1. Furthermore, the output of the intermediate layer MID1A, which is the layer preceding the corresponding intermediate layer MID2A of the corrected external force model EM1, is input to the intermediate layer MID2B of the counteracting external force model EM2.

[0108] For this integrated external force model IM, machine learning of the corrected external force model EM1 is first performed, followed by machine learning of the counterforce model EM2. While the corrected external force is applied and the corrected external force model EM1 is being learned, the weight parameters between each layer of the corrected external force model EM1 are updated to minimize the error between the predicted value of the corrected external force and the actual value of the corrected external force. The counterforce model EM2 is not updated. The weight parameters for the path from one layer of the corrected external force model EM1 (e.g., MID1A) to the next layer (MID2A) are always the same as the weight parameters for the path from the same layer (MID1A) to the counterforce model layer (MID2B). A layer of the counterforce model EM2 (e.g., MID2B) outputs the sum of the weighted input from the corrected external force model layer (e.g., MID1A) to that layer and the weighted input from the preceding layer (MID1B) of the counterforce model EM2 to the succeeding layer (OUT2) of the counterforce model EM2. After learning the corrected external force model EM1, while applying the counterforce to learn the counterforce model EM2, the weight parameters between the layers of the counterforce model EM2 are updated to minimize the error between the predicted counterforce value and the actual counterforce value. The corrected external force model EM1 is not updated. During the application phase after learning the counterforce model EM2, the output of the counterforce model EM2 is used as the predicted external force value, while the output of the corrected external force model EM1 is not used. This external force machine learning process integrates the corrected external force model EM1 and the counterforce model EM2, allowing counterforce learning to proceed without destroying the previously learned results for the corrected external force.

[0109] The integrated external force model IM uses a recognition unit (not shown) to identify whether the predicted external force value is a predicted value for the correcting external force or a predicted value for the counteracting external force, and outputs the recognition result as recognition information. In this case, if the recognition information indicates the predicted value for the correcting external force, the reward calculation unit 43 calculates the reward by using the predicted external force value as a reduction factor for the reward. If the recognition information indicates the predicted value for the counteracting external force, the reward calculation unit 43 calculates the reward by using the predicted external force value as a reduction factor for the reward.

[0110] It should be noted that the progressive neural network method refers to, for example, the method described in the following references.

[0111] (Reference) Rusu et al., “Progressive neural networks,” arXiv preprint arXiv:1606.04671, 2016.

[0112] In addition, regarding progressive neural networks, there are the following reference articles.

[0113] (Reference article) Continuous learning in multiple games

[0114] https: / / wba-initiative.org / wp-content / uploads / 2015 / 05 / 20161008-hack2-noguchi.pdf

[0115] (Robot model learning process)

[0116] Figure 9 3 is a flowchart showing the flow of a machine learning process for learning a robot model LM using machine learning. Figure 9 The machine learning processing is performed by the CPU 40A reading out the machine learning program stored in the ROM 40B or the memory 40D, expanding it in the RAM 40C, and executing it.

[0117] The processing of steps S100 to S108 described below is executed at a fixed time interval according to a control cycle. The control cycle is set to a time during which the processing of steps S100 to S108 can be executed.

[0118] In step S100, the CPU 40A waits for a predetermined time period corresponding to the length of the control cycle to elapse from the start of the previous control cycle. Note that step S100 may be omitted, and the next control cycle may be started immediately after the previous control cycle is completed.

[0119] In step S101 , the CPU 40A acquires actual values ​​(measured values) of the position and posture of the robot 10 from the state observation sensor 20 , and acquires actual values ​​(measured values) of external forces from the tactile sensors 30A and 30B.

[0120] In step S102, CPU 40A, as acquisition unit 41, determines whether the actual position and posture values ​​acquired in step S101 satisfy a predetermined termination condition. Satisfaction of the termination condition may, for example, occur when the difference between the actual position and posture values ​​and the target position and posture values ​​to be achieved is within a predetermined value. In this embodiment, the target position and posture is the position and posture of robot 10 at which robot 10 can insert pin 70 into hole 74.

[0121] If the determination in step S102 is affirmative, the present routine ends. On the other hand, if the determination in step S102 is negative, the process proceeds to step S103.

[0122] In step S103, the CPU 40A, acting as the external force model updater 45, updates the external force model EM. Specifically, based on the actual position and posture values ​​and the actual external force values ​​acquired in step S101, it is first determined whether the external force is a corrective force or an opposing force. For example, if the error between the actual position and posture values ​​and the target position and posture values ​​to be achieved increases, the external force detected as a force acting to suppress the increase in error may be determined as a corrective force, while if the error decreases, the external force detected as a force acting to suppress the decrease in error may be determined as an opposing force. However, this determination method is not limited to this.

[0123] If the external force is determined to be a correction external force, the correction external force model parameters of the correction external force model EM1 are updated so that the difference between the predicted value of the correction external force calculated by the correction external force model EM1 based on the determined action command and the actual value of the correction external force is reduced.

[0124] On the other hand, when the external force is determined to be a counteracting external force, the counteracting external force model parameters of the counteracting external force model EM2 are updated so that the difference between the predicted counteracting external force value calculated by the counteracting external force model EM2 based on the determined action command and the actual value of the counteracting external force becomes smaller.

[0125] In step S104, the CPU 40A, as the action determination unit 44, generates a plurality of candidate action commands (or action command series) for the robot 10. In the present embodiment, n (e.g., 300) velocity command value candidates are randomly generated and output to the model execution unit 42 as candidate action command values.

[0126] In step S105, CPU 40A, acting as model execution unit 42, calculates predicted values ​​for position and posture, as well as predicted values ​​for external force, for each of the multiple candidate motion command values ​​generated in step S104. Specifically, the actual position and posture values ​​and the n candidate motion command values ​​are input to robot model LM, and the predicted values ​​for position and posture, as well as the predicted values ​​for the corrective external force or the counteracting external force, are calculated for each candidate motion command value.

[0127] In step S106 , the CPU 40A as the reward calculation unit 43 calculates a reward for each pair of the predicted value of the position and posture and the predicted value of the corrected external force corresponding to the n candidate values ​​of the action command.

[0128] The reward r1 when the external force is a correction external force can be calculated using the following formula (1).

[0129] r1=-r R -α1||s1 H || 2 ···(1)

[0130] Here, r R The error between the predicted value of the position and attitude and the target value of the position and attitude that should be achieved. H is the correction external force. α1 is a pre-set weight. α1 is set so that the magnitude of the decrease in reward r1 based on the predicted value of the correction external force during task execution is smaller than the magnitude of the change in reward r1 based on the error between the predicted value of position and posture and the target value of position and posture to be achieved.

[0131] On the other hand, the reward r2 when the external force is a counter-force can be calculated using the following formula (2).

[0132] r 2 =-r R +α 2 ||s2 H || 2 ···(2)

[0133] Here, s2 H α2 is a weight that is set in advance. α2 is set so that the increase in reward r2 based on the predicted value of the counteracting external force during task execution is smaller than the increase in reward r2 based on the error between the predicted position and posture and the target position and posture to be achieved.

[0134] As shown in equations (1) and (2) above, when the external force remains constant, the greater the error between the predicted position and posture and the target position and posture to be achieved, the smaller the reward. Furthermore, as shown in equation (1) above, when the error remains constant, the greater the correction force, the smaller the reward. Furthermore, as shown in equation (2) above, when the error remains constant, the greater the counteracting external force, the larger the reward.

[0135] In step S107, the CPU 40A, acting as the action determination unit 44, determines the action command that maximizes the reward and outputs it to the robot 10. For example, a relationship is calculated that represents the correspondence between n action command candidate values ​​and the reward, and the action command candidate value corresponding to the maximum reward on the curve represented by the calculated relationship is determined as the value. Alternatively, the so-called cross-entropy method (CEM) can be used to determine the action command that maximizes the reward. This method results in the action command that maximizes the reward.

[0136] Steps S104 to S106 can be repeated a predetermined number of times. In this case, after executing step S106 for the first time, CPU 40A, as the action determination unit 44, extracts m candidate values ​​for action commands with high rewards from the n candidate values ​​for action commands and rewards, calculates the mean and variance of these m candidate values ​​for action commands, and generates a corresponding normal distribution. In the second step S104, CPU 40A, as the action determination unit 44, generates n new candidate values ​​for speed commands so that they are consistent with the normal distribution calculated by the probability density, rather than random. Similarly, steps S104 to S106 are executed a predetermined number of times. This improves the accuracy of reward maximization.

[0137] The robot 10 moves according to the determined value of the action instruction. The user applies an external force to the robot 10 according to the movement of the robot 10. Specifically, an external force is applied to the clamp 12. The user preferably applies a correction external force to the robot 10 when the error between the predicted value of the position posture and the target value of the position posture to be achieved is expanding, and applies an opposing external force to the robot 10 when the error is shrinking. That is, for example, when the pin 70 is moving in a direction away from the hole 74 due to the movement of the robot 10, the user applies a correction external force to the clamp 12 in a direction in which the pin 70 approaches the hole 74. In addition, for example, when the pin 70 is moving in a direction close to the hole 74 due to the movement of the robot 10, an opposing external force is applied to the clamp 12 in a direction in which the pin 70 is away from the hole 74.

[0138] It should be noted that during machine learning of the external force model, it is preferable to initially apply a corrective external force. This is because applying an opposing external force initially may slow learning. Furthermore, the ratio of applying the corrective external force to the opposing external force can be 1:1, or the ratio of the corrective external force can be increased. Furthermore, the order of applying the corrective external force and the opposing external force can be multiple applications of the corrective external force followed by multiple applications of the opposing external force, or the corrective external force and the opposing external force can be applied alternately.

[0139] Furthermore, instead of humans applying the corrective external force or the counteracting external force, a robot or the like that applies the external force may automatically apply the corrective external force or the counteracting external force.

[0140] In step S108 , the CPU 40A as the model execution unit 42 calculates a predicted value of the external force for the determined value of the action command determined in step S107 , and the process returns to step S100 .

[0141] In this manner, the processing of steps S100 to S108 is repeated for each control cycle until the actual value of the position and posture satisfies the end condition.

[0142] Thus, the robot model LM is learned. Thus, the robot model LM includes the correction external force model EM1 and the counteracting external force model EM2. Since the user learns the robot model LM while applying the correction external force or counteracting external force to the robot 10, learning is efficient, and a robot model LM with excellent robustness against environmental changes, such as changes in the shape and material of parts operated by the robot 10 and changes in the physical properties of the robot 10 over time, can be obtained.

[0143] It should be noted that, in the application phase, the model execution unit 42 Figure 9 The learning process executes the learned robot model LM. The function of the robot control device 40 in the application phase is as follows: Figure 1 The external force model updating unit 45 and the learning control unit 46 are omitted from the functional structure. The robot control process in the operation phase is as follows: Figure 9 The learning process excludes the process of "obtaining the actual value of the external force" in step S101 and the updating process of updating the external force model in step S103, and the program that executes this process becomes the robot control processing program.

[0144] It should be noted that the device that performs the robot model learning process during the learning phase and the device that performs the robot control process during the operation phase can be different devices or the same device. For example, the learning device used for learning can be used directly as the robot control device 40 to perform control using the learned robot model LM. Furthermore, the robot control device 40 can also perform control while continuing to learn.

[0145] <Modification of the First Embodiment>

[0146] In the first embodiment, the state transition model DM is configured to input actual values ​​of position and posture and action commands, but not actual values ​​of external forces. Alternatively, the state transition model DM can be configured to input actual values ​​of external forces. In this case, the state transition model DM calculates predicted values ​​of position and posture based on the actual values ​​of position and posture, action commands, and actual values ​​of external forces. However, the application of corrective external forces or counteracting external forces to the tactile sensors 30A and 30B is limited to the period during which machine learning of the external force models EM1, EM2, EM1a, EM2a, and IM is performed. During the operational phase, the state transition model DM calculates predicted values ​​of position and posture while the input of actual values ​​of external forces remains substantially zero. On the other hand, the external force model calculates predicted values ​​of external forces based on actual values ​​of position and posture and action commands without inputting actual values ​​of external forces, and this modification also applies. The predicted values ​​of external forces influence action determination by being used in reward calculation. Modifications similar to this modification can also be implemented in subsequent embodiments.

[0147] <Second embodiment>

[0148] Next, a second embodiment of the disclosed technology will be described. Components identical to those in the first embodiment are denoted by the same reference numerals, and detailed descriptions thereof will be omitted.

[0149] The robot system 1 according to the second embodiment is the same as that of the first embodiment, and therefore its description is omitted.

[0150] (Robot model learning process)

[0151] Figure 10 This is a flowchart showing the flow of machine learning processing according to the second embodiment. Figure 10 The machine learning process is performed by the CPU 40A reading out the machine learning program stored in the ROM 40B or the memory 40D, expanding it in the RAM 40C, and executing it.

[0152] The processing of steps S100 to S103 and S108 is Figure 9 The processing is the same, so the description is omitted.

[0153] In step S104A, the CPU 40A, as the behavior specifying unit 44 , generates one candidate of an action command (or a series of action commands) for the robot 10 .

[0154] In step S105A, CPU 40A, acting as model execution unit 42, calculates predicted values ​​for position and posture, as well as predicted values ​​for external force, for one candidate action command generated in step S104A. Specifically, the actual position and posture values ​​and the candidate action command values ​​are input to robot model LM, and the predicted values ​​for position and posture, as well as the predicted values ​​for the corrective external force or the counteracting external force, corresponding to the candidate action command value are calculated.

[0155] In step S106A, CPU 40A, as reward calculation unit 43, calculates a reward based on the predicted value of the position and posture corresponding to the candidate value of the action command and the predicted value of the external force. Specifically, if the external force is a corrective force, reward r1 is calculated using equation (1) above; if the external force is an opposing force, reward r2 is calculated using equation (2) above.

[0156] In step S106B, CPU 40A determines whether the reward calculated in step S106A satisfies a predetermined condition. Here, the predetermined condition is satisfied, for example, when the reward exceeds a predetermined value, or when the processing of steps S104A to S106B has been executed a predetermined number of times. The predetermined number of times is set to, for example, 10, 100, or 1000 times.

[0157] In step S107A, the CPU 40A, as the action determination unit 44, determines an action command that maximizes the reward and outputs it to the robot 10. For example, the action command may be an action command that is executed when the reward satisfies a predetermined condition, or an action command that is predicted based on the history of changes in the reward corresponding to changes in the action command to further maximize the reward.

[0158] <Third embodiment>

[0159] Next, a third embodiment of the disclosed technology will be described. Components identical to those in the first embodiment are denoted by the same reference numerals, and detailed descriptions thereof will be omitted.

[0160] (Robot control device)

[0161] Figure 11 This is the functional configuration of the robot control device 40X according to the third embodiment. Figure 1 The robot control device 40 shown is different in that it includes a storage unit 48 and a state transition model updating unit 49. The other configurations are the same as those of the robot control device 40, and therefore description thereof will be omitted.

[0162] The storage unit 48 stores the actual values ​​of the position and posture of the robot 10 acquired by the acquisition unit 41 .

[0163] The state transition model updating unit 49 updates the state transition model DM so that the error between the predicted value of the position and posture calculated by the state transition model DM based on the action command determined by the action determination unit 44 and the actual value of the position and posture corresponding to the predicted value is reduced.

[0164] (Robot model learning process)

[0165] Figure 12 This is a flowchart showing the flow of machine learning processing according to the third embodiment. Figure 12 The machine learning process is performed by the CPU 40A reading out the machine learning program stored in the ROM 40B or the memory 40D, expanding it in the RAM 40C, and executing it.

[0166] Figure 12 Learning Processing and Figure 9 The difference between the learning process and the learning process is that the processing of steps S101A and S103A is added. The other steps are the same as Figure 9 The processing is the same and therefore the description is omitted.

[0167] In step S101A, the CPU 40A as the acquisition unit 41 stores the actual value of the position and posture of the robot 10 acquired in step S101 in the storage unit 48 .

[0168] In step S103A, the CPU 40A, as the state transition model updating unit 49, updates the state transition model DM. Specifically, first, actual values ​​x of positions and postures at, for example, 100 times t, selected randomly, are acquired from the values ​​stored in the storage unit 48. t , speed command value u as action command t , and the actual value x of the position and posture at time t+1 t+1 Next, new state transition model parameters are determined by correcting the previous state transition model parameters. The state transition model parameters are corrected with the goal of minimizing the error between the predicted position and posture at time t+1 calculated from the actual position and posture at time t and the actual position and posture at time t+1.

[0169] Then, the new state transition model parameters are set as the state transition model DM. The new state transition model parameters are stored in the state transition model updating unit 49 in order to be used as the "previous model parameters" in the next control cycle.

[0170] As described above, in the present embodiment, the state transition model DM can be learned together with the corrected external force model EM1 and the counter-external force model EM2.

[0171] <Experimental Example>

[0172] Next, experimental examples of the disclosed technology will be described.

[0173] Figure 13 shows the results of learning the robot model by simulating a task of inserting a pin into a hole while applying a correction external force and a counteracting external force to the robot. In this simulation, the robot model was learned by applying the correction external force seven times and then the counteracting external force seven times.

[0174] Figure 13A 、 Figure 13B The horizontal axis represents the number of times the external force is applied. Figure 13A The vertical axis represents the error in the position of the pin. Figure 13B The vertical axis represents the magnitude of the applied external force.

[0175] exist Figure 13A The figure shows the result of learning only the state transition model without applying external force by the existing method (Baseline), the result of learning the robot model including the corrected external force applied by subject 1 by the proposed method and the corrected external force model and the counter-external force model (Proposed (participant1)), and the result of learning the robot model including the corrected external force applied by subject 2 who is different from subject 1 by the proposed method and the corrected external force model and the counter-external force model (Proposed (participant2)). Figure 13AAs shown in , when the position error at the end of learning is compared between the conventional method and the proposed method (subject 1 and subject 2), it can be seen that the position error in the proposed method is smaller than that in the conventional method. Figure 13B As shown, although the method of applying the external force is different between subjects 1 and 2, the position error is reduced regardless of the subject.

[0176] In addition, Figure 14A In the figure, the results of simulating the number of successful pin insertions by changing the friction coefficient of the table provided with the hole are shown. Figure 14B In the figure, the results of simulating the number of successful pin insertions by changing the mass of the pin are shown. Figure 14A 、 14B As shown, it can be seen that even when the friction coefficient of the table and the mass of the pin are different, the number of successful pin insertions in the proposed method is greater than the number of successful pin insertions in the conventional method.

[0177] In addition, Figure 15 The following table shows the results of the same pin insertion task as the simulation using pins of different materials. The pin materials are metal (Default), plastic, and sponge. Figure 15 As shown, it can be seen that the proposed method has a greater number of successful pin insertions than the conventional method for pins of any material.

[0178] As can be seen from the configuration and operation of the aforementioned embodiment, as well as the experimental examples, applying a corrective external force for machine learning can improve the efficiency of the robot model's machine learning. Furthermore, applying a counteracting external force for machine learning can improve robustness against variations in friction and mass within the grasped object. Furthermore, applying a counteracting external force for machine learning also has the effect of improving learning efficiency.

[0179] It should be noted that the above-mentioned embodiments are merely exemplary configuration examples of the present disclosure. The disclosed technology is not limited to the above-mentioned specific embodiments, and various modifications are possible within the scope of the concept of the technology.

[0180] For example, in the above-mentioned embodiment, the pin fitting operation is described as an example, but the operation to be learned and controlled may be any operation.

[0181] Furthermore, the robot model learning and robot control processes executed by the CPU in the above-described embodiments by reading software (programs) can also be performed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices), whose circuit configuration can be changed after manufacture, such as FPGAs (Field-Programmable Gate Arrays), and ASICs (Application-Specific Integrated Circuits), processors with circuit configurations specifically designed to perform specific processes, i.e., dedicated electrical circuits. Furthermore, the learning and control processes can be performed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is more specifically an electrical circuit that combines circuit elements such as semiconductor elements.

[0182] Furthermore, in the above embodiments, the robot model learning program and the robot control program are pre-stored (installed) in the memory 40D or ROM 40B, but the present invention is not limited thereto. The program may also be provided by recording it on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory device. Furthermore, the program may be downloaded from an external device via a network.

[0183] Regarding the above embodiment, the following supplementary notes are also disclosed.

[0184] (Note 1)

[0185] A robot model learning device, comprising:

[0186] An acquisition unit (41) acquires an actual value of the position and posture of the robot and an actual value of an external force applied to the robot;

[0187] a robot model (LM), the robot model (LM) comprising a state transition model (DM) for calculating a predicted value of the robot's position and posture at a next time period based on actual values ​​of the position and posture at a certain time period and action instructions that can be provided to the robot, and an external force model (EM) for calculating a predicted value of an external force applied to the robot;

[0188] A model execution unit (42) for executing the robot model;

[0189] A reward calculation unit (43) calculates a reward based on an error between the predicted value of the position and posture and the target value of the position and posture to be achieved, and the predicted value of the external force;

[0190] an action determination unit (44) that generates a plurality of candidates for the action instruction for each control cycle and provides the generated candidate candidates to the robot model, and determines an action instruction that maximizes the reward based on the rewards calculated by the reward calculation unit for the plurality of candidates for the action instruction; and

[0191] An external force model updating unit (45) updates the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action instruction and the actual value of the external force corresponding to the predicted value of the external force becomes smaller.

[0192] (Note 2)

[0193] The robot model learning device according to Supplementary Note 1 comprises:

[0194] A state transition model updating unit (49) updates the state transition model so that the error between the predicted value of the position posture calculated by the state transition model based on the determined action instruction and the actual value of the position posture corresponding to the predicted value of the position posture becomes smaller.

[0195] (Note 3)

[0196] The robot model learning device according to Supplement 1 or 2,

[0197] When the external force is a correction external force that suppresses expansion of the error, the reward calculation unit calculates the reward by calculating a predicted value of the correction external force as a reduction factor for the reward.

[0198] (Note 4)

[0199] According to the robot model learning device described in Supplementary Note 3,

[0200] The reward calculation unit calculates the reward by calculating that a change width of a reduction amount of the reward based on a predicted value of the corrected external force during task execution is smaller than a change width of a reduction amount of the reward based on the error.

[0201] (Note 5)

[0202] The robot model learning device according to Supplement 3 or 4,

[0203] The external force model includes: a correction external force model (EM1) outputting a predicted value of the correction external force when the external force is the correction external force;

[0204] The external force model updating unit includes: a corrected external force model updating unit (45A), which updates the corrected external force model so that when the external force is the corrected external force, the difference between the predicted value of the corrected external force calculated by the corrected external force model based on the determined action instruction and the actual value of the corrected external force becomes smaller.

[0205] (Note 6)

[0206] The robot model learning device according to Supplement 1 or 2,

[0207] When the external force is an external force that suppresses reduction of the error, that is, an opposing external force, the reward calculation unit calculates the reward by using a predicted value of the opposing external force as an increasing factor of the reward.

[0208] (Note 7)

[0209] According to the robot model learning device described in Supplementary Note 6,

[0210] The reward calculation unit calculates the reward by calculating that a change width of an increase in the reward based on a predicted value of the counter external force during task execution is smaller than a change width of an increase in the reward based on the error.

[0211] (Note 8)

[0212] The robot model learning device according to Supplement 6 or 7,

[0213] The external force model includes: a counter external force model (EM2) that outputs a predicted value of the counter external force when the external force is the counter external force;

[0214] The external force model updating unit includes: an opposing external force model updating unit (45B), which updates the opposing external force model so that when the external force is the opposing external force, the difference between the predicted value of the opposing external force calculated by the opposing external force model based on the determined action instruction and the actual value of the opposing external force becomes smaller.

[0215] (Note 9)

[0216] The robot model learning device according to Supplement 1 or 2,

[0217] The reward calculation unit calculates the reward by using the predicted value of the corrected external force as a reduction factor of the reward when the external force is a corrected external force that suppresses the expansion of the error. The reward calculation unit calculates the reward by using the predicted value of the counteracting external force as an increase factor of the reward when the external force is an external force that suppresses the reduction of the error, i.e., a counteracting external force.

[0218] (Note 10)

[0219] According to the robot model learning device described in Supplementary Note 9,

[0220] The reward calculation unit calculates the reward by calculating that the change in the decrease in the reward based on the predicted value of the corrected external force during task execution is smaller than the change in the reward based on the error, and the change in the increase in the reward based on the predicted value of the counteracting external force during task execution is smaller than the change in the reward based on the error.

[0221] (Note 11)

[0222] The robot model learning device according to Supplement 9 or 10,

[0223] The external force model includes: a correction external force model (EM1) that outputs a predicted value of the correction external force when the external force is the correction external force, and a counter external force model (EM2) that outputs a predicted value of the counter external force when the external force is the counter external force.

[0224] The external force model updating unit includes: a correction external force model updating unit, which updates the correction external force model so that when the external force is the correction external force, the difference between the predicted value of the correction external force calculated by the correction external force model based on the determined action instruction and the actual value of the external force becomes smaller; and an opposing external force model updating unit (45B), which updates the opposing external force model so that when the external force is the opposing external force, the difference between the predicted value of the opposing external force calculated by the opposing external force model based on the determined action instruction and the actual value of the external force becomes smaller.

[0225] (Note 12)

[0226] According to the robot model learning device described in Supplementary Note 11,

[0227] The robot model includes an integrated external force model (IM) including the correction external force model and the counter external force model.

[0228] The corrected external force model and the counteracted external force model are neural networks,

[0229] At least one of the one or more intermediate layers and the output layer of the counteracting external force model integrates the output of the preceding layer of the corresponding layer of the corrected external force model through a progressive neural network method.

[0230] The integrated external force model outputs the output of the counter external force model as the predicted value of the external force.

[0231] The integrated external force model outputs identification information of whether the outputted predicted value of the external force is a predicted value of the corrected external force or a predicted value of the counteracted external force.

[0232] The reward calculation unit calculates the reward by using the predicted value of the external force as a reduction factor for the reward when the identification information is represented by a predicted value of a corrected external force. The reward calculation unit calculates the reward by using the predicted value of the external force as an increase factor for the reward when the identification information is represented by a predicted value of a counteracting external force.

[0233] (Note 13)

[0234] The robot model learning device according to Supplement 11 or 12,

[0235] further comprising a receiving unit for receiving a designation of whether the external force is the correction external force or the counteracting external force;

[0236] The system further includes a learning control unit that validates the operation of the correction external force model updating unit when the correction external force is specified, and validates the operation of the counter external force model updating unit when the counter external force is specified.

[0237] (Note 14)

[0238] The robot model learning device according to Supplement 11 or 12,

[0239] It also has a learning control unit, which determines whether the external force is the correction external force or the counteracting external force based on the actual value of the position posture and the actual value of the external force. When the result of the determination is the correction external force, the action of the correction external force model updating unit is validated. When the result of the determination is the counteracting external force, the action of the counteracting external force model updating unit is validated.

[0240] (Note 15)

[0241] A machine learning method for a robot model performs the following steps:

[0242] A robot model (LM) is prepared, wherein the robot model (LM) includes a state transition model (DM) for calculating a predicted value of the robot's position and posture at a next time period based on actual values ​​of the robot's position and posture at a certain time period and action instructions that can be provided to the robot, and an external force model (EM) for calculating a predicted value of an external force applied to the robot.

[0243] For each control cycle, the actual value of the position and posture and the actual value of the external force applied to the robot are obtained.

[0244] For each control cycle, a plurality of candidates for the action command are generated and provided to the robot model, and based on a plurality of errors between a plurality of predicted values ​​of the position and posture calculated by the state transition model corresponding to the plurality of candidates for the action command and a target value of the position and posture to be achieved, and a plurality of predicted values ​​of the external force calculated by the external force model corresponding to the plurality of candidates for the action command, an action command that maximizes the reward is determined based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command.

[0245] The external force model is updated so that a difference between a predicted value of the external force calculated by the external force model based on the determined action command and an actual value of the external force corresponding to the predicted value of the external force becomes smaller.

[0246] (Note 16)

[0247] According to the machine learning method of the robot model described in Supplementary Note 15,

[0248] The state transition model is also updated so that an error between a predicted value of the position and posture calculated by the state transition model based on the determined action command and an actual value of the position and posture corresponding to the predicted value of the position and posture becomes smaller.

[0249] (Note 17)

[0250] According to the machine learning method of the robot model described in Supplement 15 or 16,

[0251] When the external force is a correction external force that suppresses expansion of the error, the reward is calculated by calculating a predicted value of the correction external force as a reduction factor for the reward.

[0252] (Note 18)

[0253] According to the machine learning method of the robot model described in Supplementary Note 17,

[0254] The reward is calculated by calculating that the change width of the reduction amount of the reward based on the predicted value of the corrected external force during task execution is smaller than the change width of the reward based on the error.

[0255] (Note 19)

[0256] According to the machine learning method of the robot model described in Supplement 17 or 18,

[0257] The external force model includes: a correction external force model (EM1) outputting a predicted value of the correction external force when the external force is the correction external force;

[0258] The corrected external force model is updated so that, when the external force is the corrected external force, a difference between a predicted value of the corrected external force calculated by the corrected external force model based on the determined action command and an actual value of the external force becomes smaller.

[0259] (Note 20)

[0260] According to the machine learning method of the robot model described in Supplementary Note 19,

[0261] When the error is increasing, the correction external force is applied to the robot.

[0262] (Note 21)

[0263] According to the machine learning method of the robot model described in Supplement 15 or 16,

[0264] When the external force is an external force that suppresses reduction of the error, that is, an opposing external force, the reward is calculated by using a predicted value of the opposing external force as an increasing factor of the reward.

[0265] (Note 22)

[0266] According to the machine learning method of the robot model described in Supplementary Note 21,

[0267] The reward is calculated by calculating that the change width of the increase amount of the reward based on the predicted value of the counter external force during task execution is smaller than the change width of the reward based on the error.

[0268] (Note 23)

[0269] According to the machine learning method of the robot model described in Supplement 21 or 22,

[0270] The external force model includes: a counter external force model (EM2) that outputs a predicted value of the counter external force when the external force is the counter external force;

[0271] The counter external force model is updated so that, when the external force is the counter external force, a difference between a predicted value of the counter external force calculated by the counter external force model based on the determined action instruction and an actual value of the external force becomes smaller.

[0272] (Note 24)

[0273] According to the machine learning method of the robot model described in Supplementary Note 23,

[0274] When the error is decreasing, the counteracting external force is applied to the robot.

[0275] (Note 25)

[0276] According to the machine learning method of the robot model described in Supplement 15 or 16,

[0277] In the case where the external force is a corrective external force that suppresses the expansion of the error, the reward is calculated by calculating the predicted value of the corrective external force as a reduction factor of the reward. In the case where the external force is an external force that suppresses the reduction of the error, that is, a counteracting external force, the reward is calculated by calculating the predicted value of the counteracting external force as an increase factor of the reward.

[0278] (Note 26)

[0279] According to the machine learning method of the robot model described in Supplementary Note 25,

[0280] The reward is calculated by calculating that the change in the decrease in the reward based on the predicted value of the corrected external force during task execution is smaller than the change in the reward based on the error, and the change in the increase in the reward based on the predicted value of the opposing external force during task execution is smaller than the change in the reward based on the error.

[0281] (Note 27)

[0282] According to the machine learning method of the robot model described in Supplement 25 or 26,

[0283] The external force model includes: a correction external force model (EM1) outputting a predicted value of the correction external force when the external force is the correction external force, and a counter external force model (EM2) outputting a predicted value of the counter external force when the external force is the counter external force.

[0284] The correction external force model is updated so that when the external force is the correction external force, the difference between the predicted value of the correction external force calculated by the correction external force model based on the determined action instruction and the actual value of the external force becomes smaller; the counteracting external force model is updated so that when the external force is the counteracting external force, the difference between the predicted value of the counteracting external force calculated by the counteracting external force model based on the determined action instruction and the actual value of the external force becomes smaller.

[0285] (Note 28)

[0286] According to the machine learning method of the robot model described in Supplementary Note 27,

[0287] When the error is increasing, the correction external force is applied to the robot, and when the error is decreasing, the counter external force is applied to the robot.

[0288] (Note 29)

[0289] A machine learning program for a robot model,

[0290] A machine learning program for a robot model (LM) for performing machine learning includes a state transition model (DM) for calculating a predicted value of the robot's position and posture in the next period based on the actual value of the robot's position and posture in a certain period and an action instruction that can be provided to the robot, and an external force model (EM) for calculating a predicted value of an external force applied to the robot, and causes a computer to perform the following processing:

[0291] For each control cycle, the actual value of the position and posture and the actual value of the external force applied to the robot are obtained.

[0292] For each control cycle, a plurality of candidates for the action command are generated and provided to the robot model, and based on a plurality of errors between a plurality of predicted values ​​of the position and posture calculated by the state transition model corresponding to the plurality of candidates for the action command and a target value of the position and posture to be achieved, and a plurality of predicted values ​​of the external force calculated by the external force model corresponding to the plurality of candidates for the action command, an action command that maximizes the reward is determined based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command.

[0293] The external force model is updated so that a difference between a predicted value of the external force calculated by the external force model based on the determined action command and an actual value of the external force corresponding to the predicted value of the external force becomes smaller.

[0294] (Note 30)

[0295] A robot control device comprising:

[0296] A model execution unit (42) executes a robot model (LM) including a state transition model (DM) that calculates actual values ​​of the position and posture of the robot in a certain period of time and action instructions that can be provided to the robot, and calculates a predicted value of the position and posture of the robot in the next period of time, and an external force model (EM) that calculates a predicted value of an external force applied to the robot.

[0297] An acquisition unit (41) acquires the actual value of the position and posture of the robot and the actual value of the external force applied to the robot,

[0298] A reward calculation unit (43) calculates a reward based on an error between a predicted value of a position and posture calculated by the robot model and a target value of the position and posture to be achieved, and a predicted value of an external force calculated by the robot model, and

[0299] An action determination unit (44) generates a plurality of candidates for the action instruction for each control cycle and provides them to the robot model, and determines an action instruction that maximizes the reward based on the rewards calculated by the reward calculation unit corresponding to the plurality of candidates for the action instruction.

[0300] (Note 31)

[0301] A robot control method comprises the following steps:

[0302] A robot model (LM) is prepared, wherein the robot model (LM) includes a state transition model (DM) for calculating a predicted value of the robot's position and posture at a next time period based on actual values ​​of the robot's position and posture at a certain time period and action instructions that can be provided to the robot, and an external force model (EM) for calculating a predicted value of an external force applied to the robot.

[0303] For each control cycle, the actual value of the position and posture and the actual value of the external force applied to the robot are obtained.

[0304] For each control cycle, a plurality of candidates for the action command are generated and provided to the robot model, and based on a plurality of errors between a plurality of predicted values ​​of the position and posture calculated by the state transition model corresponding to the plurality of candidates for the action command and a target value of the position and posture to be achieved, and a plurality of predicted values ​​of the external force calculated by the external force model corresponding to the plurality of candidates for the action command, an action command that maximizes the reward is determined based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command.

[0305] The robot is controlled based on the determined action instruction.

[0306] (Note 32)

[0307] A robot control program,

[0308] A program for controlling a robot using a robot model (LM) including a state transition model (DM) for calculating predicted values ​​of the robot's position and posture in a next period based on actual values ​​of the robot's position and posture in a certain period of time and action instructions that can be provided to the robot, and an external force model (EM) for calculating predicted values ​​of external forces applied to the robot, causing a computer to perform the following processing:

[0309] For each control cycle, the actual value of the position and posture and the actual value of the external force applied to the robot are obtained.

[0310] For each control cycle, a plurality of candidates for the action command are generated and provided to the robot model, and based on a plurality of errors between a plurality of predicted values ​​of the position and posture calculated by the state transition model corresponding to the plurality of candidates for the action command and a target value of the position and posture to be achieved, and a plurality of predicted values ​​of the external force calculated by the external force model corresponding to the plurality of candidates for the action command, an action command that maximizes the reward is determined based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command.

[0311] The robot is controlled based on the determined action instruction.

[0312] The disclosure of Japanese Patent Application No. 2021-20049 is incorporated herein by reference in its entirety. Furthermore, all documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, or technical standard were specifically and individually indicated to be incorporated by reference.

Claims

1. A robot model learning device comprising: an acquisition unit, configured to acquire an actual value of the position and posture of the robot and an actual value of an external force applied to the robot; a robot model comprising a state transition model for calculating a predicted value of the robot's position and posture for a next time period based on the actual value of the position and posture for a certain time period and an action instruction that can be provided to the robot, and an external force model for calculating a predicted value of an external force applied to the robot; A model execution unit, configured to execute the robot model; a reward calculation unit that calculates a reward based on an error between the predicted value of the position and posture and a target value of the position and posture to be achieved, and the predicted value of the external force; an action determination unit that generates a plurality of candidates for the action command for each control cycle and provides the generated candidates to the robot model, and determines an action command that maximizes the reward based on the rewards calculated by the reward calculation unit for the plurality of candidates for the action command; as well as The external force model updating unit updates the external force model so that a difference between a predicted value of the external force calculated by the external force model based on the determined action command and an actual value of the external force corresponding to the predicted value of the external force becomes smaller.

2. The robot model learning device according to claim 1, further comprising: A state transition model updating unit updates the state transition model so that an error between a predicted value of the position and posture calculated by the state transition model based on the determined action instruction and an actual value of the position and posture corresponding to the predicted value of the position and posture becomes smaller.

3. The robot model learning device according to claim 1, When the external force is a correction external force that suppresses expansion of the error, the reward calculation unit calculates the reward by calculating a predicted value of the correction external force as a reduction factor for the reward.

4. The robot model learning device according to claim 1, When the external force is an external force that suppresses reduction of the error, that is, an opposing external force, the reward calculation unit calculates the reward by using a predicted value of the opposing external force as an increasing factor of the reward.

5. The robot model learning device according to claim 1, When the external force is a correction external force that suppresses the expansion of the error, the reward calculation unit calculates the reward by using the predicted value of the correction external force as a reduction factor of the reward. When the external force is an external force that suppresses the reduction of the error, that is, an opposing external force, the reward calculation unit calculates the reward by using the predicted value of the opposing external force as an increase factor of the reward.

6. The robot model learning device according to claim 5, The reward calculation unit calculates the reward by calculating that the change in the decrease in the reward based on the predicted value of the corrected external force during task execution is smaller than the change in the reward based on the error, and the change in the increase in the reward based on the predicted value of the counteracting external force during task execution is smaller than the change in the reward based on the error.

7. The robot model learning device according to claim 5, The external force model includes: a correction external force model that outputs a predicted value of the correction external force when the external force is the correction external force, and a counter external force model that outputs a predicted value of the counter external force when the external force is the counter external force, The external force model updating unit includes: a correction external force model updating unit, which updates the correction external force model so that when the external force is the correction external force, the difference between the predicted value of the correction external force calculated by the correction external force model based on the determined action instruction and the actual value of the external force becomes smaller; and an opposing external force model updating unit, which updates the opposing external force model so that when the external force is the opposing external force, the difference between the predicted value of the opposing external force calculated by the opposing external force model based on the determined action instruction and the actual value of the external force becomes smaller.

8. The robot model learning device according to claim 7, The robot model includes an integrated external force model including the correction external force model and the counter external force model. The corrected external force model and the counteracted external force model are neural networks, At least one of the one or more intermediate layers and the output layer of the counteracting external force model integrates the output of the preceding layer of the corresponding layer of the corrected external force model through a progressive neural network method. The integrated external force model outputs the output of the counter external force model as the predicted value of the external force. The integrated external force model outputs identification information of whether the outputted predicted value of the external force is a predicted value of the corrected external force or a predicted value of the counteracted external force. The reward calculation unit calculates the reward by using the predicted value of the external force as a reduction factor for the reward when the identification information is represented by a predicted value of a corrected external force. The reward calculation unit calculates the reward by using the predicted value of the external force as an increase factor for the reward when the identification information is represented by a predicted value of a counteracting external force.

9. The robot model learning device according to claim 7 or 8, The robot model learning device further includes a receiving unit that receives a designation of whether the external force is the correction external force or the counteracting external force. The learning device of the robot model also includes a learning control unit, which validates the action of the correction external force model update unit when the correction external force is specified, and validates the action of the counteracting external force model update unit when the counteracting external force is specified.

10. The robot model learning device according to claim 7 or 8, The learning device of the robot model also has a learning control unit, which determines whether the external force is the correction external force or the counteracting external force based on the actual value of the position posture and the actual value of the external force. When the result of the determination is the correction external force, the action of the correction external force model updating unit is validated. When the result of the determination is the counteracting external force, the action of the counteracting external force model updating unit is validated.

11. A machine learning method for a robot model, comprising the following steps: Prepare a robot model, the robot model including a state transition model for calculating a predicted value of the robot's position and posture in a next period based on actual values ​​of the robot's position and posture in a certain period and action instructions that can be provided to the robot, and an external force model for calculating a predicted value of an external force applied to the robot. For each control cycle, the actual value of the position and posture and the actual value of the external force applied to the robot are obtained. For each control cycle, a plurality of candidates for the action command are generated and provided to the robot model, and based on a plurality of errors between a plurality of predicted values ​​of the position and posture calculated by the state transition model corresponding to the plurality of candidates for the action command and a target value of the position and posture to be achieved, and a plurality of predicted values ​​of the external force calculated by the external force model corresponding to the plurality of candidates for the action command, an action command that maximizes the reward is determined based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command. The external force model is updated so that a difference between a predicted value of the external force calculated by the external force model based on the determined action command and an actual value of the external force corresponding to the predicted value of the external force becomes smaller.

12. The machine learning method for the robot model according to claim 11, The machine learning method of the robot model also updates the state transition model so that the error between the predicted value of the position posture calculated by the state transition model based on the determined action instruction and the actual value of the position posture corresponding to the predicted value of the position posture becomes smaller.

13. The machine learning method for a robot model according to claim 11 or 12, When the external force is a correction external force that suppresses expansion of the error, the reward is calculated by calculating a predicted value of the correction external force as a reduction factor for the reward.

14. The machine learning method for a robot model according to claim 11 or 12, When the external force is an external force that suppresses reduction of the error, that is, an opposing external force, the reward is calculated by using a predicted value of the opposing external force as an increasing factor of the reward.

15. The machine learning method for a robot model according to claim 11 or 12, In the case where the external force is a corrective external force that suppresses the expansion of the error, the reward is calculated by calculating the predicted value of the corrective external force as a reduction factor of the reward. In the case where the external force is an external force that suppresses the reduction of the error, that is, a counteracting external force, the reward is calculated by calculating the predicted value of the counteracting external force as an increase factor of the reward.

16. The machine learning method for a robot model according to claim 15, The external force model includes: a correction external force model that outputs a predicted value of the correction external force when the external force is the correction external force, and a counter external force model that outputs a predicted value of the counter external force when the external force is the counter external force, The machine learning method of the robot model updates the corrected external force model so that when the external force is the corrected external force, the difference between the predicted value of the corrected external force calculated by the corrected external force model based on the determined action instruction and the actual value of the external force becomes smaller. The machine learning method of the robot model updates the antagonistic external force model so that when the external force is the antagonistic external force, the difference between the predicted value of the antagonistic external force calculated by the antagonistic external force model based on the determined action instruction and the actual value of the external force becomes smaller.

17. The machine learning method for a robot model according to claim 16, When the error is increasing, the correction external force is applied to the robot, and when the error is decreasing, the counter external force is applied to the robot.

18. A storage medium storing a machine learning program of a robot model, The machine learning program is used to perform machine learning on a robot model including a state transition model that calculates a predicted value of the robot's position and posture in a next period of time based on actual values ​​of the robot's position and posture in a certain period of time and action instructions that can be provided to the robot, and an external force model that calculates a predicted value of an external force applied to the robot. The machine learning program causes the computer to perform the following processing: For each control cycle, the actual value of the position and posture and the actual value of the external force applied to the robot are obtained. For each control cycle, a plurality of candidates for the action command are generated and provided to the robot model, and based on a plurality of errors between a plurality of predicted values ​​of the position and posture calculated by the state transition model corresponding to the plurality of candidates for the action command and a target value of the position and posture to be achieved, and a plurality of predicted values ​​of the external force calculated by the external force model corresponding to the plurality of candidates for the action command, an action command that maximizes the reward is determined based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command. The external force model is updated so that a difference between a predicted value of the external force calculated by the external force model based on the determined action command and an actual value of the external force corresponding to the predicted value of the external force becomes smaller.

19. A robot control device comprising: a model execution unit that executes a robot model including a state transition model that calculates a predicted value of the robot's position and posture in a next time period based on actual values ​​of the robot's position and posture in a certain time period and an action command that can be provided to the robot, and an external force model that calculates a predicted value of an external force applied to the robot; an acquisition unit, configured to acquire an actual value of the position and posture of the robot and an actual value of an external force applied to the robot; a reward calculation unit that calculates a reward based on an error between a predicted value of a position and posture calculated by the robot model and a target value of the position and posture to be achieved, and a predicted value of an external force calculated by the robot model; and The action determination unit generates a plurality of candidates for the action command for each control cycle and provides the generated candidates to the robot model, and determines the action command that maximizes the reward based on the rewards calculated by the reward calculation unit for the plurality of candidates for the action command.

20. A robot control method comprising the following steps: Prepare a robot model, the robot model including a state transition model for calculating a predicted value of the robot's position and posture in a next period based on actual values ​​of the robot's position and posture in a certain period and action instructions that can be provided to the robot, and an external force model for calculating a predicted value of an external force applied to the robot. For each control cycle, the actual value of the position and posture and the actual value of the external force applied to the robot are obtained. For each control cycle, a plurality of candidates for the action command are generated and provided to the robot model, and based on a plurality of errors between a plurality of predicted values ​​of the position and posture calculated by the state transition model corresponding to the plurality of candidates for the action command and a target value of the position and posture to be achieved, and a plurality of predicted values ​​of the external force calculated by the external force model corresponding to the plurality of candidates for the action command, an action command that maximizes the reward is determined based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command. The robot is controlled based on the determined action instruction.

21. A storage medium storing a robot control program, The robot control program is configured to control the robot using a robot model including a state transition model for calculating predicted values ​​of the robot's position and posture in a next period based on actual values ​​of the robot's position and posture in a certain period and action instructions that can be provided to the robot, and an external force model for calculating predicted values ​​of external forces applied to the robot. The robot control program causes a computer to perform the following processing: For each control cycle, the actual value of the position and posture and the actual value of the external force applied to the robot are obtained. For each control cycle, a plurality of candidates for the action command are generated and provided to the robot model, and based on a plurality of errors between a plurality of predicted values ​​of the position and posture calculated by the state transition model corresponding to the plurality of candidates for the action command and a target value of the position and posture to be achieved, and a plurality of predicted values ​​of the external force calculated by the external force model corresponding to the plurality of candidates for the action command, an action command that maximizes the reward is determined based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command. The robot is controlled based on the determined action instruction.

Images