Instruction learning device, instruction learning method, and recording medium

Abstract

This instruction learning device: generates a first instruction including at least one instruction to a control target that operates an object; evaluates a second instruction, which satisfies a first condition, in the first instruction on the basis of a second condition; calculates an evaluation amount; and selects and learns a third instruction, which maximizes the evaluation amount, in the second instruction, wherein the control target operates the object on the basis of the third instruction.

Description

Command learning device, command learning method, and recording medium

[0001] The present disclosure relates to an instruction learning device, an instruction learning method, and a recording medium.

[0002] Patent Document 1 discloses a robot control device that avoids collisions between a robot and an obstacle.

[0003] International Publication No. 2022 / 009333

[0004] However, with the robot control device described in Patent Document 1, when learning the robot's movements, the user must manually specify the areas through which the robot will pass.

[0005] The present disclosure aims to provide an instruction learning device that solves the above-mentioned problems.

[0006] According to a first aspect of the present disclosure, there is provided an instruction learning device comprising: a generation unit that generates first instructions including at least one instruction for a control target that operates an object; an evaluation unit that evaluates second instructions from the first instructions that satisfy a first condition based on a second condition and calculates an evaluation amount; and a learning unit that selects and learns a third instruction from the second instructions that maximizes the evaluation amount, wherein the control target operates the object based on the third instruction.

[0007] According to a second aspect of the present disclosure, there is provided an instruction learning method comprising: a generation step of generating first instructions including at least one instruction for a control target that operates an object; an evaluation step of evaluating second instructions from the first instructions that satisfy a first condition based on a second condition and calculating an evaluation quantity; and a learning step of selecting and learning a third instruction from the second instructions that maximizes the evaluation quantity, wherein the control target operates the object based on the third instruction.

[0008] According to a third aspect of the present disclosure, there is provided a recording medium comprising: a generation step of generating first instructions including at least one instruction for a control target that operates an object; an evaluation step of evaluating second instructions from the first instructions that satisfy a first condition based on a second condition and calculating an evaluation amount; and a learning step of selecting and learning a third instruction from the second instructions that maximizes the evaluation amount, wherein the control target operates the object based on the third instruction.

[0009] According to the present disclosure, it is possible to provide an instruction learning device that does not require manual learning operations by a human.

[0010] FIG. 1 is a schematic diagram illustrating an instruction learning device according to some embodiments of the present disclosure; FIG. 2 is a block diagram illustrating a hardware configuration of a computer according to some embodiments of the present disclosure; FIG. 3 is a block diagram illustrating a hardware configuration of a robot controller according to some embodiments of the present disclosure; FIG. 4 is a block diagram illustrating a configuration of an instruction learning device according to some embodiments of the present disclosure; FIG. 5 is a sequence chart of an instruction learning device according to some embodiments of the present disclosure; FIG. 6 is an example of a model implemented by a simulator unit of an instruction learning device according to some embodiments of the present disclosure; FIG. 7 is an example of a model implemented by a simulator unit of an instruction learning device according to some embodiments of the present disclosure; FIG. 8 is a block diagram illustrating a configuration of an instruction learning device according to another embodiment of the present disclosure.

[0011] Hereinafter, an instruction learning device according to some embodiments of the present disclosure will be described with reference to the drawings.

[0012] 1 is a schematic diagram illustrating an instruction learning device 5 according to some embodiments of the present disclosure. The instruction learning device 5 includes a computer 1, a robot controller 2, and a robot 3. In this embodiment, the instruction learning device 5 learns instructions for the robot 3.

[0013] The computer 1 is a general-purpose computer. The computer 1 is communicably connected to the robot controller 2. The computer 1 generates commands to control the robot 3 in response to an operator's operation. The computer 1 transmits the generated commands to the robot controller 2.

[0014] The robot controller 2 is a device that controls the robot 3. The robot controller 2 includes a servo amplifier, a circuit board, a teach pendant, etc. The robot controller 2 is communicatively connected to the computer 1 and the robot 3. The robot controller 2 receives commands to control the robot 3 from the computer 1, and transmits control signals corresponding to the commands to the robot 3.

[0015] The robot 3 is fixed to a fixed surface such as a floor, a wall, a jig, a workbench, or other device, and manipulates an object 4. The robot 3 receives a control signal from the robot controller 2 and operates in accordance with the control signal. In this embodiment, a case will be described where the robot 3 is a six-axis robot. The robot 3 includes a lower arm 31, an upper arm 32, and an end effector 33.

[0016] The lower arm 31 is connected to the fixed surface via a first axis 311 and a second axis 312. The first axis 311 is a movable part of the robot 3 that connects the fixed surface and the second axis 312, and moves around an axis perpendicular to the fixed surface. The second axis 312 is a movable part of the robot 3 that connects the first axis 311 and the lower arm 31, and moves around an axis parallel to the fixed surface.

[0017] The upper arm 32 is connected to the lower arm 31 via a third axis 313 and a fourth axis 314. The third axis 313 is a movable part of the robot 3 that connects the lower arm 31 and the fourth axis 314, and moves around an axis perpendicular to the central axis of the lower arm 31. The fourth axis 314 is a movable part of the robot 3 that connects the third axis 313 and the upper arm 32, and moves around the central axis of the upper arm 32.

[0018] The end effector 33 is connected to the upper arm 32 via a fifth axis 315 and a sixth axis 316. The fifth axis 315 is a movable part of the robot 3 that connects the upper arm 32 and the sixth axis 316, and moves around the central axis of the upper arm 32. The sixth axis 316 is a movable part of the robot 3 that connects the fifth axis 315 and the end effector 33, and moves around an axis perpendicular to the central axis of the upper arm 32. The first axis 311 to the sixth axis 316 each include an actuator, a reducer, an encoder, and a transmission mechanism. By including these hardware configurations in the first axis 311 to the sixth axis 316, it is possible to control the rotation angle, rotation speed, etc. of the first axis 311 to the sixth axis 316.

[0019] In this embodiment, the configuration of the robot 3 is not particularly limited. The robot 3 may have an appropriate configuration depending on the shape of the object 4, the properties of the object 4, the work to be performed on the object 4, and the like. When the robot 3 moves the object 4 from one place to another, the end effector 33 may be, for example, a two-finger gripper, a three-finger gripper, a suction gripper, a magnetic gripper, or the like. When the robot 3 screws a part to the object 4, the end effector 33 may be a screwdriver or the like. When the object 4 is an automobile engine part or the like and the robot 3 measures the three-dimensional shape of the object 4, the end effector 33 may be a CMM (Coordinate Measuring Machine) stylus. The material of the CMM stylus shaft may be, for example, stainless steel, ceramic, carbon fiber, aluminum, titanium, or the like. The type of the CMM stylus ball may be, for example, ruby, silicon nitride, zirconia, or the like. When the target object 4 is a semiconductor wafer and the robot 3 dices the target object 4, the end effector 33 may be a dicing blade (circular blade). The robot 3 may also have a plurality of end effectors 33 that can be replaced depending on the process.

[0020] The robot 3 may further include sensors. For example, the end effector 33 of the robot 3 may include a speed sensor, an acceleration sensor, a distance sensor, a temperature sensor, a pressure sensor, etc. The robot 3 can appropriately manipulate the object 4 based on information obtained by the sensors. For example, if the end effector 33 includes a distance sensor, the distance between the end effector 33 and the object 4 can be sequentially obtained, and the computer 1 can also obtain the three-dimensional shape of the object 4 via the robot 3 before the robot 3 grasps the object 4.

[0021] The object 4 is an object manipulated by the robot 3. There may be one object 4 or multiple objects 4. The object 4 may be, for example, a packaging box for storing products, agricultural produce, industrial products, semiconductor wafers, etc., and the object 4 may be in the form of a solid, liquid, or gas as long as it is an object that can be manipulated by the robot 3. The object 4 may also include multiple different substances, etc. For example, the object 4 may be a paper cup, a liquid contained in the paper cup, and a plastic lid that seals the top of the paper cup. If the object 4 is gas, the robot 3 may be installed in a sealed room, etc.

[0022] In this embodiment, the computer 1 and the robot controller 2 are provided as separate hardware configurations, but the computer 1 and the robot controller 2 may be provided as the same hardware configuration. When the computer 1 and the robot controller 2 are provided as the same hardware configuration, the computer 1 may have the functions of the robot controller 2 as hardware or software functions.

[0023] 2 is a block diagram showing a hardware configuration of a computer 1 according to some embodiments of the present disclosure. The computer 1 includes a CPU 11, a ROM 12, a RAM 13, a storage device 14, a display device 15, an imaging device 16, an input / output IF 17, and a communication IF 18. The CPU 11, the ROM 12, the RAM 13, the storage device 14, the display device 15, the imaging device 16, the input / output IF 17, and the communication IF 18 are connected to each other via a bus 19 so as to be able to communicate with each other.

[0024] The CPU 11 is a central processing unit. The CPU 11 controls each part of the computer 1 using an application program. The ROM 12 is a read-only memory. The ROM 12 is made up of non-volatile memory and stores application programs for controlling each part of the computer 1. The RAM 13 is a random access memory. The RAM 13 provides a memory area necessary for the operation of the CPU 11. The storage device 14 is a large-capacity storage device such as a hard disk drive.

[0025] The display device 15 may be a liquid crystal display, an organic electroluminescence (EL) display, etc. The imaging device 16 is an image capturing device such as a camera, and captures images of the robot 3 and the target object 4 in accordance with a user's operation or a command from an application, etc. The imaging device 16 includes an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a control circuit, a signal processing circuit, etc.

[0026] The input / output IF 17 is an input / output interface for connecting a mouse, a trackball, a keyboard, a speaker, etc. An operator may operate the computer 1 via the input / output IF 17. The communication IF 18 performs mutual data communication between the computer 1 and the robot controller 2 via wired communication and / or wireless communication. The communication IF 18 may be communicatively connected to sensors, etc. provided in the robot 3, and may receive signals indicating the distance between the robot 3 and the object 4, the strength of the force applied by the robot 3 to the object 4, the temperature of the object 4, the moving speed of the object 4, etc.

[0027] 3 is a block diagram showing the hardware configuration of a robot controller 2 according to some embodiments of the present disclosure. The robot controller 2 includes a CPU 21, a ROM 22, a RAM 23, a storage device 24, an input / output IF 25, and a communication IF 26. The CPU 21, the ROM 22, the RAM 23, the storage device 24, the input / output IF 25, and the communication IF 26 are connected to each other via a bus 29 so as to be able to communicate with each other.

[0028] The CPU 21 is a central processing unit. The CPU 21 controls each part of the robot controller 2 using an application program. The ROM 22 is a read-only memory. The ROM 22 is made up of non-volatile memory and stores application programs for controlling each part of the robot controller 2. The RAM 23 is a random access memory. The RAM 23 provides a memory area necessary for the operation of the CPU 21. The storage device 24 is a large-capacity storage device such as a hard disk drive.

[0029] The input / output IF 25 is a user interface that accepts input from an operator. The input / output IF 25 includes an operation lever, a teach pendant, an emergency stop button, etc. The operator can send control signals to the robot controller 2 via the input / output IF 25 to operate the robot 3. The communication IF 26 performs mutual data communication between the robot controller 2 and other devices via wired communication and / or wireless communication. The communication IF 26 performs mutual data communication between the computer 1 and the robot controller 2, and between the robot controller 2 and the robot 3.

[0030] Fig. 4 is a block diagram showing the configuration of an instruction learning device according to some embodiments of the present disclosure. Fig. 5 is a sequence chart of an instruction learning device according to some embodiments of the present disclosure. The configuration and functions of the instruction learning device 5 will be described using the configuration of the instruction learning device 5 shown in Fig. 4 and the sequence chart of the instruction learning device 5 shown in Fig. 5.

[0031] The computer 1 includes a detection unit 101, a generation unit 102, an imitation learning unit 103, a simulator unit 104, an evaluation unit 105, a communication unit 106, and a database 107. The robot controller 2 includes a determination unit 201, a control unit 202, and a communication unit 203. The computer 1 can capture images of the robot 3 and the target object 4 using the imaging device 16.

[0032] The detection unit 101 captures images of the robot 3 and the object 4 and detects the hardware configuration of the robot 3, the positional relationship between the robot 3 and the object 4, the shape of the object 4, etc. Based on information about the hardware configuration of the robot 3 and information from sensors and the like possessed by the robot 3, the detection unit 101 may detect the hardware configuration of the robot 3, the positional relationship between the robot 3 and the object 4, the shape of the object 4, etc. The detection unit 101 may acquire information about the hardware configuration of the robot 3 and information from sensors and the like possessed by the robot 3 from the storage device 14 or the database 107. The detection unit 101 may also capture an image of the object 4 after the robot 3 has actually operated it. Based on the configuration of the robot 3, the position of the robot 3, the shape of the object 4, the position of the object 4, etc., the detection unit 101 converts the positions of the robot 3 and the object 4 into coordinate data and stores it in the database 107.

[0033] The generation unit 102 generates a parameter set (first command) for controlling the robot 3. The robot 3 receives a control signal corresponding to the parameter set and operates based on the values ​​included in the parameter set. In other words, the parameter set may be values ​​that specify a target operation of the robot 3. In this embodiment, the parameter set may include a value for each of the first axis 311 to the sixth axis 316, or may include a set of values ​​for multiple axes among the first axis 311 to the sixth axis 316. If the parameters for the first axis 311 to the sixth axis 316 are parameter P1 to parameter P6, respectively, the parameter set may include a value for rotating parameter P1 at an arbitrary angle (for example, P1 = θ1).

[0034] Alternatively, the parameter set may include a set of values ​​for rotating the parameter P1, the parameter P3, and the parameter P4 to any angle (e.g., {P1, P3, P4} = {θ1, θ3, θ4}). The parameter set may further include a target time for the operation of the robot 3. When the parameter P1 is to be rotated to any angle by a certain time t1, the parameter set may include {P1, T1} = {θ1, t1}. When the parameter P1, the parameter P3, and the parameter P4 are to be rotated to any angle by certain times t1, t3, and t4, respectively, the parameter set may include {P1, P3, P4, T1, T3, T4} = {θ1, θ3, θ4, t1, t3, t4}. Furthermore, the parameter T for time may be the time required for the robot 3 to operate from a certain state and reach a state corresponding to a target value.

[0035] The combination of parameters included in the parameter set is not limited to the above example. For example, the parameter set may be an operation sequence. For example, when rotating the first axis 311 to an angle θ1 over a time t1 and then rotating the third axis 313 to an angle θ3 over a time t2, the parameter set may include a combination of multiple parameter sets such as {{P1, T1} = {θ1, t1}, {P3, T3} = {θ3, t3}}. The parameter set may also include a waiting time, and the above operation sequence may include a parameter W for the waiting time. Furthermore, the parameter set may include operation settings of sensors possessed by the robot 3. The description of the parameter set is not limited to the above example description, and may correspond to a description specified by the manufacturer of the robot 3 or the robot controller 2.

[0036] The generation unit 102 generates a parameter set according to the hardware configuration of the robot 3 (step S100). Based on the hardware configuration of the robot 3, the generation unit 102 may generate a parameter set that includes all of the values ​​that can be set for the robot 3. For example, a parameter set may be generated that sets the values ​​of parameter P1 to parameter P2 to be 0 to 360 degrees, ..., parameter P6 to be 0 to 360 degrees for the first axis 311 to the sixth axis 316, respectively (i.e., a brute-force parameter set). Alternatively, based on the coordinate data of the robot 3 and the coordinate data of the object 4 detected by the detection unit 101, the generation unit 102 may generate a parameter set that includes some of the values ​​that can be set for the robot 3 (i.e., a parameter set based on the object 4). For example, when the robot 3 moves the object 4 from one workbench to another, and the range of the rotation angle of the first axis 311 is 45-135 degrees, and the robot 3 can sufficiently manipulate the object 4, the generation unit 102 may generate a parameter set in which the parameter P1 is 45-135 degrees. Alternatively, the operator may set the hardware configuration of the robot 3 in the computer 1 or the robot controller 2 in advance, and the generation unit 102 may generate a parameter set including values ​​that can be set in the robot 3 based on the setting (i.e., a parameter set based on prior information). The generation unit 102 may generate a parameter set based on a simulation model of the robot 3 included in the simulator unit 104 (a parameter set based on an evaluation model). Alternatively, the generation unit 102 may generate a random parameter set using random numbers or the like (i.e., a random parameter set). When the generation unit 102 generates a parameter set using random numbers or the like, the parameter set may include an action sequence that the robot 3 cannot actually execute. The generator 102 may generate a parameter set based on a parameter set of another robot having components similar to those of the robot 3 (i.e., a parameter set based on similar components).

[0037] The generation unit 102 transmits the parameter set to the determination unit 201 via the communication unit 106 (step S101). The generation unit 102 may generate one parameter set and sequentially transmit the one parameter set to the determination unit 201. Alternatively, the generation unit 102 may generate multiple parameter sets, such as an action sequence, and transmit the multiple parameter sets to the determination unit 201.

[0038] The communication unit 106 manages communication between the computer 1 and the robot controller 2. For example, the communication unit 106 transmits a parameter set from the generation unit 102 or the imitation learning unit 103 to the robot controller 2. The communication unit 106 also receives a parameter set and a determination result determined by the determination unit 201 from the robot controller 2 and transmits them to the generation unit 102 and / or the imitation learning unit 103. The communication unit 106 may transmit the parameter set and the determination result to the database 107. Note that, although the imaging device 16 is directly connected to the detection unit 101 in FIG. 4 , the imaging device 16 may also be connected to the detection unit 101 via the communication unit 106. The communication unit 106 may also receive data from a sensor provided in the robot 3 or the target object 4. When the imaging device 16 connects to the detection unit 101 via the communication unit 106, the imaging device 16 transmits an image or video to the detection unit 101 via the communication unit 106.

[0039] The communication unit 203 manages communication between the computer 1 and the robot controller 2. For example, the communication unit 203 receives the parameter set generated by the generation unit 102 and the parameter set output by the imitation learning unit 103 from the computer 1, and transmits them to the determination unit 201. The communication unit 203 also transmits the parameter set determined by the determination unit 201 and the determination result to the computer 1. Furthermore, the communication unit 203 transmits the control signal generated by the control unit 202 to the robot 3. The communication unit 203 may receive data from a sensor provided in the robot 3.

[0040] The database 107 is a database that stores coordinate data based on the position of the robot 3 detected by the detection unit 101, coordinate data based on the position of the object 4, a learning model in the generation unit 102, a learning model in the imitation learning unit 103, a simulation model of the robot 3 in the simulator unit 104, evaluation conditions in the evaluation unit 105, etc. The database 107 stores parameter sets generated by the generation unit 102. Furthermore, the database 107 may store parameter sets for robots that have components different from those of the robot 3. Machine learning in the generation unit 102 may be performed using a parameter set for a robot that has different components. The database 107 may store parameter sets for a robot 3 that has different components or different functions. When the generation unit 102 generates a parameter set, it may obtain a parameter set for a different robot that has components similar to those of the robot 3 from the database 107 and generate the parameter set.

[0041] The determination unit 201 receives a parameter set from the generation unit 102 via the communication unit 203. The determination unit 201 determines a parameter set based on a predetermined condition (first condition) (step S110). For example, suppose the angle at which the second axis 312 can be moved is 10-170 degrees (i.e., the predetermined condition is that the angle must be within a range of 10-170 degrees), and the value of parameter P2 included in the parameter set is set to 5 degrees. In this case, the value of parameter P2 is outside the range of angles at which the second axis 312 can be moved, so the determination unit 201 determines that the parameter P2 included in the parameter set cannot be set. On the other hand, suppose the angle at which the second axis 312 can be moved is 10-170 degrees (i.e., the predetermined condition is that the angle must be within a range of 10-170 degrees), and the value of parameter P2 included in the parameter set is set to 55 degrees. In this case, the value of parameter P2 is within the range of angles at which the second axis 312 can be moved, so the determination unit 201 determines that the parameter P2 included in the parameter set can be set. The judgment unit 201 compares the movable range of the components (e.g., movable parts such as the first axis 311) of the robot 3 with the parameter values ​​corresponding to the components included in the parameter set, and determines whether the parameter values ​​can be set to the components.

[0042] The determination unit 201 may further determine whether parameters included in the parameter set may damage components of the robot 3. For example, assume that the parameter set includes parameters for rotating the first axis 311 from an angle of 0 degrees to an angle of 180 degrees in 0.1 seconds. Even if the value of parameter P1 is within the range of angles within which the first axis 311 can move, if the target time required for the first axis 311 to reach the value of parameter P1 is insufficient for the value of parameter P1, the determination unit 201 may determine that the parameter P1 and the target time cannot be set. In other words, if it is difficult for the components of the robot 3 to achieve a transition from the current state of the components of the robot 3 to the target state within the target time, the determination unit 201 may determine that the parameter P1 and the target time cannot be set. The determination unit 201 may determine that parameters that do not damage the components of the robot 3 can be set to the components. The determination unit 201 may determine the parameters included in the parameter set based on constraints, etc., when operating the object 4. For example, although the first axis 311 can move from 0 to 359 degrees, if there is a structure or the like adjacent to the robot 3, the determination unit 201 may determine the parameters so as to limit the range of angles within which the first axis 311 can move.

[0043] The determination unit 201 transmits the parameter set and the determination result to the generation unit 102 via the communication unit 203 (step S111). Furthermore, the determination unit 201 transmits the parameter set and the determination result to the imitation learning unit 103 via the communication unit 203 (step S112). Note that the determination unit 201 does not need to transmit the parameter set and the determination result separately to the generation unit 102 and the imitation learning unit 103. For example, the computer 1 may receive the parameter set and the determination result from the determination unit 201 and store the parameter set and the determination result in the database 107. Furthermore, the computer 1 may transmit the parameter set and the determination result to the generation unit 102 and the imitation learning unit 103.

[0044] The generation unit 102 may perform machine learning based on the transmitted parameter set and the received parameter set and determination result (step S113). The generation unit 102 may include, for example, a machine learning model based on a deep neural network. The generation unit 102 may perform machine learning using the received parameter set and determination result as training data. For example, if the determination unit 201 determines that the generated parameter set is not executable by the robot 3, the generation unit 102 may learn to reduce the evaluation of the generated parameter set and generate a parameter set that increases the evaluation. Note that step S113 is not an essential component in this embodiment and may be an additional component in this embodiment. In other words, the generation unit 102 may have a function of only generating a parameter set.

[0045] The imitation learning unit 103 learns a parameter set that enables the robot 3 to appropriately operate the object 4, based on a parameter set (second command) that satisfies a predetermined condition. The imitation learning unit 103 includes, for example, a reinforcement learning model. The imitation learning unit 103 performs reinforcement learning based on the received parameter set and the evaluation amount of the parameter set received from the evaluation unit 105. In step S112, upon receiving the parameter set and the determination result from the determination unit 201, the imitation learning unit 103 transmits the parameter set and the determination result to the simulator unit 104 (step S120).

[0046] The simulator unit 104 is a simulator that simulates the configuration of the robot 3. The simulator unit 104 has a simulation model of the robot 3 based on, for example, CAD (Computer Aided Design) data or the like. The simulation model of the robot 3 may be a combination of multiple two-dimensional models or a three-dimensional model. In this embodiment, the simulation model of the robot 3 will be described as a three-dimensional model. The simulator unit 104 further has coordinate data of the object 4 captured by the detection unit 101. The simulator unit 104 may generate a simulation model of the object 4 based on the coordinate data of the object 4. Based on the simulation model of the robot 3 and the coordinate data of the object 4, the simulator unit 104 can simulate the operation of the robot 3 according to the parameter set.

[0047] When the simulator unit 104 receives the parameter set and the determination result from the imitation learning unit 103, it operates the simulation model of the robot 3 using the parameter set that the determination result indicates can be set in the robot 3 (step S121). The simulator unit 104 inputs the parameter set into the simulation model of the robot 3 and simulates the behavior of the robot 3.

[0048] The evaluation unit 105 evaluates the parameter set based on the operation of the simulator unit 104 and predetermined evaluation conditions (second conditions) (step S122). For example, the evaluation conditions may include, when the robot 3 operates the object 4 to achieve a predetermined goal, minimizing the difference in distance between the position (current position) of the object 4 before movement and the position (target position) of the object 4 after movement by the robot 3; minimizing the difference between the target posture of the object 4 and the posture of the object 4; minimizing the difference between the target state of the object 4 and the state of the object 4; minimizing the distance between the robot 3 and the object 4; minimizing the power consumption of the robot 3; minimizing the number of commands to the robot 3; minimizing the time it takes for the robot 3 to operate the object 4; minimizing the difference in distance between the positions of each component of the robot 3 before achieving the predetermined goal and the positions of each component of the robot 3 after achieving the predetermined goal, etc. Furthermore, the evaluation conditions may include the quality of the object 4, the shape of the object 4, the detection results of defects in the object 4, etc. The evaluation condition may be one of the above conditions or a combination of multiple conditions.

[0049] As an example of manipulating an object 4 by the robot 3, when the robot 3 moves the object 4 from one location to another (target position), the robot 3 grasps the object 4 with the end effector 33 and moves the object 4 to the target position. In this example, it is desirable to minimize the difference in distance between the position of the object 4 and the target position. Therefore, the evaluation condition includes minimizing the difference in distance between the target position of the object 4 and the position of the object 4 after movement. Furthermore, the evaluation condition may include minimizing the difference between the target posture of the object 4 and the posture of the object 4 after movement. If the goal is to reduce the power consumption of the robot 3, the evaluation condition may include minimizing the power consumption of the robot 3 when moving the object 4 from one location to another, and the evaluation condition may include minimizing the number of commands used to operate the robot 3 when moving the object 4 from one location to another. If the goal is to shorten the operation time of the object 4, the evaluation condition may include minimizing the time required to move the object 4 from one location to the target position. When the evaluation condition is to minimize the difference in distance between the positions of the components of the robot 3 before and after the object 4 is moved, the evaluation condition may include, for example, minimizing the difference in distance between the three-dimensional coordinates of the first axis 311-sixth axis 316 before and after the object 4 is moved. When the evaluation condition is the quality of the object 4 or the shape of the object 4, the evaluation condition may include not damaging the object 4 when the end effector 33 grasps the object 4.

[0050] 6 and 7 show an example of a model generated by the simulator unit of the command learning device according to some embodiments of the present disclosure, after the robot 3 has operated based on the parameter set. Using the examples of FIGS. 6 and 7, how the evaluation unit 105 evaluates the parameter set will be described. For simplicity's sake, a case will be described in which the evaluation conditions are to minimize the difference in distance between the target position and the position of the object 4 after movement, and to minimize the difference between the target posture and the posture of the object 4 after movement.

[0051] In FIG. 6 , after the robot 3 moves the object 4 using the parameter set, the object 4 rolls over at a position away from the target position. The difference in distance between the object 4 and the target position is large, and the difference between the target posture and the posture of the object 4 is also large. Therefore, in the example of FIG. 6 , the evaluation unit 105 evaluates the evaluation amount of the parameter set to be low. On the other hand, in FIG. 7 , after the robot 3 moves the object 4 using the parameter set, the object 4 is near the target position and has taken the target posture. The difference in distance between the object 4 and the target position is smaller than in the example of FIG. 6 , and the difference between the target posture and the posture of the object 4 is also smaller. Therefore, in the example of FIG. 7 , the evaluation unit 105 evaluates the evaluation amount of the parameter set to be high compared to the example of FIG. 6 .

[0052] The evaluation unit 105 transmits the evaluation value of the parameter set to the imitation learning unit 103 based on the result of the simulator unit 104 inputting the parameter set into the simulation model of the robot 3 (step S123). The imitation learning unit 103 performs reinforcement learning based on the parameter set received in step S112 and the evaluation value of the parameter set received in step S123 (step S124). That is, the imitation learning unit 103 performs reinforcement learning using the simulator unit 104 and the evaluation unit 105 as a simulation model of the environment in reinforcement learning. The imitation learning unit 103 does not input the parameter set into the robot 3 and operate the object 4, but inputs the parameter set into the simulation model of the robot 3 in the simulator unit 104 and operates the simulation model of the object 4. The imitation learning unit 103 receives from the evaluation unit 105 an evaluation value corresponding to the simulation execution result of the parameter set obtained in the simulator unit 104, and performs reinforcement learning according to the evaluation value. The imitation learning unit 103 learns the parameter set so as to select a parameter set with a large evaluation value. For example, the imitation learning unit 103 learns a parameter set so as to select a parameter set (third command) that causes the robot 3 to operate as shown in Fig. 7, and learns a parameter set so as not to select a parameter set that causes the robot 3 to operate as shown in Fig. 6. The imitation learning unit 103 transmits the parameter set received in step S112 to the determination unit 201 via the communication unit 106 (step S125). Note that the imitation learning unit 103 may transmit to the determination unit 201 a parameter set whose evaluation amount is equal to or greater than a predetermined threshold.

[0053] According to the above-described process, the generation unit 102 generates parameter sets, and the determination unit 201 determines parameter sets that can be set for the robot 3. Furthermore, the imitation learning unit 103 learns and selects appropriate parameter sets from the determined parameter sets, thereby enabling the imitation learning unit 103 to learn parameter sets that make the robot 3 satisfy predetermined evaluation conditions without directly operating the robot 3 with a human hand. This makes it possible to provide an instruction learning device that does not require human manual operation of the robot for learning.

[0054] The determination unit 201 receives a parameter set from the imitation learning unit 103 via the communication unit 203. The determination unit 201 determines a parameter set based on predetermined conditions (step S130). The determination by the determination unit 201 in step S130 is the same as the determination by the determination unit 201 in step S110. The determination unit 201 transmits the parameter set determined to be settable to the components of the robot 3 to the control unit 202 (step S131). Furthermore, the determination unit 201 may transmit the parameter set and the determination result to the generation unit 102 via the communication unit 203 (step S132).

[0055] The generation unit 102 may perform machine learning based on the transmitted parameter set and the received parameter set and determination result (step S133). Note that the machine learning process of the generation unit 102 in step S133 may be the same as the machine learning process of the generation unit 102 in step S113. The generation unit 102 may perform weighted learning of the parameter set based on the evaluation amount received by the imitation learning unit 103 in step S123. In step S113, if the generation unit 102 has the function of only generating a parameter set and does not perform machine learning, the generation unit 102 does not perform machine learning in step S133. In other words, the machine learning processes in steps S113 and S133 may be additional configurations.

[0056] According to the above-described processing, when the generation unit 102 performs machine learning processing in steps S113 and S133, the generation unit 102 does not generate a random parameter set, thereby preventing a decrease in the performance of the imitation learning unit 103 that performs reinforcement learning processing.

[0057] The control unit 202 receives the parameter set and generates a control signal that can be executed by the robot 3 based on the parameter set (step S140). The control signal may be a signal that controls the actuators, reducers, encoders, and transmission mechanisms of the first axis 311 to the sixth axis 316 and the end effector 33. The control signal may also include a signal that controls the operation mode of a sensor of the robot 3, etc. The control unit 202 transmits the control signal to the robot 3 via the communication unit 203 (step S141). The robot 3 receives the control signal and operates according to the parameter set included in the control signal.

[0058] The computer 1 and the robot controller 2 may repeat the processes of steps S100 to S130, S132, and S133 (i.e., the processes of steps S100 to S133 excluding step S131) ​​until a parameter set that fully satisfies predetermined evaluation conditions is obtained, and then execute the processes of steps S131, S140, and S141. This allows the robot 3 to operate the object 4 while reducing the possibility of damage to the robot 3 and the object 4.

[0059] Of the parameter sets that can be set for the robot 3, the imitation learning unit 103 performs reinforcement learning on the parameter set that maximizes a predetermined evaluation amount. Based on the reinforcement learning, the imitation learning unit 103 can select a parameter set that satisfies predetermined evaluation conditions for the robot 3. This makes it possible to provide an instruction learning device that does not require manual learning operation of the robot by a human.

[0060] 8 is a block diagram showing the configuration of an instruction learning device according to another embodiment of the present disclosure. The instruction learning device 9 includes a generation unit 91, an evaluation unit 92, and a learning unit 93. The generation unit 91 generates parameters for controlling a controlled object, the evaluation unit 92 evaluates the parameters, and the learning unit 93 learns parameters that maximize the evaluation amount of the evaluation unit 92, and selects the parameters.

[0061] Furthermore, the scope of each embodiment also includes a processing method in which a program that operates the configuration of each embodiment to realize the functions of the above-described embodiments is recorded on a recording medium, the program recorded on the recording medium is read as code, and the program is executed on a computer. In other words, a computer-readable recording medium is also included in the scope of each embodiment. Furthermore, each embodiment includes not only a recording medium on which the above-described computer program is recorded, but also the computer program itself.

[0062] Examples of the recording medium that can be used include a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM (Compact Disc-Read Only Memory), a magnetic tape, a non-volatile memory card, and a ROM. In addition, the scope of each embodiment is not limited to programs that execute processing by themselves, but also includes programs that execute processing by operating on an OS (Operating System) in cooperation with other software and functions of an expansion board.

[0063] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and detailed description of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0064] According to this embodiment, it is possible to provide an instruction learning device that does not require a human to manually operate a robot for learning.

[0065] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0066] (Supplementary Note 1) An instruction learning device comprising: a generation unit that generates first instructions including at least one instruction for a control target that operates an object; an evaluation unit that evaluates second instructions from the first instructions that satisfy a first condition based on a second condition and calculates an evaluation amount; and a learning unit that selects and learns a third instruction from the second instructions that maximizes the evaluation amount, wherein the control target operates the object based on the third instruction.

[0067] (Supplementary Note 2) The instruction learning device according to Supplementary Note 1, wherein the controlled object includes at least one component, and the first condition includes that the first instruction can be set to the at least one component.

[0068] (Supplementary Note 3) The command learning device according to Supplementary Note 1, wherein the control object includes a plurality of components, and the first condition includes that when one of the plurality of components operates based on the first command and another of the plurality of components operates based on the first command, the one component and the other component do not physically interfere with each other.

[0069] (Supplementary Note 4) The instruction learning device according to Supplementary Note 1, wherein the control target includes at least one component, and the generation unit generates the first instruction based on the at least one component.

[0070] (Supplementary Note 5) The instruction learning device according to Supplementary Note 2, wherein the generation unit generates the first instruction including a combination of any value within a range of values ​​that one of the components can take and any value within a range of values ​​that another of the components can take.

[0071] (Supplementary Note 6) The instruction learning device according to Supplementary Note 1, wherein the generation unit generates a fourth instruction based on the first instruction and the second instruction.

[0072] (Supplementary Note 7) The instruction learning device according to Supplementary Note 1, wherein the generating unit generates a fourth instruction based on the first instruction and the third instruction.

[0073] (Supplementary Note 8) The command learning device described in Supplementary Note 1, wherein the second condition includes at least one of: minimizing the power consumption of the controlled object; minimizing the number of commands when the controlled object operates the object; minimizing the time required for the controlled object to operate the object; and minimizing the difference between the three-dimensional position of at least one component included in the controlled object before the controlled object operates the object and the three-dimensional position of the at least one component after the controlled object operates the object.

[0074] (Supplementary Note 9) The instruction learning device according to Supplementary Note 8, wherein the evaluation unit includes a model that simulates the controlled object, and the model executes the second instruction and evaluates the second instruction based on the second condition.

[0075] (Supplementary Note 10) The command learning device according to Supplementary Note 1, further comprising an imaging unit that captures images of the control target and the object, wherein the imaging unit acquires a three-dimensional shape of the object.

[0076] (Supplementary Note 11) The command learning device according to Supplementary Note 1, further comprising a determination unit that determines whether or not the behavior of the control target based on the first command satisfies the first condition.

[0077] (Supplementary Note 12) A command learning method comprising: a generating step of generating first commands including at least one command for a control target that operates an object; an evaluating step of evaluating second commands among the first commands that satisfy a first condition based on a second condition and calculating an evaluation amount; and a learning step of selecting and learning a third command from the second commands that maximizes the evaluation amount, wherein the control target operates the object based on the third command.

[0078] (Supplementary Note 13) The command learning method according to Supplementary Note 12, wherein the controlled object includes at least one component, and the first condition includes that the first command can be set to the at least one component.

[0079] (Supplementary Note 14) The command learning method according to Supplementary Note 12, wherein the control object includes a plurality of components, and the first condition includes that when one component of the plurality of components operates based on the first command and another component of the plurality of components operates based on the first command, the one component and the other component do not physically interfere with each other.

[0080] (Supplementary Note 15) The instruction learning method according to Supplementary Note 12, wherein the control target includes at least one component, and the generation unit generates the first instruction based on the at least one component.

[0081] (Supplementary Note 16) The instruction learning method according to Supplementary Note 13, wherein the generation unit generates the first instruction including a combination of any value within a range of values ​​that one of the components can take and any value within a range of values ​​that another of the components can take.

[0082] (Supplementary Note 17) The instruction learning method according to Supplementary Note 12, wherein the generating unit generates a fourth instruction based on the first instruction and the second instruction.

[0083] (Supplementary Note 18) The instruction learning method according to Supplementary Note 12, wherein the generating unit generates a fourth instruction based on the first instruction and the third instruction.

[0084] (Supplementary Note 19) The command learning method described in Supplementary Note 12, wherein the second condition includes at least one of: minimizing the power consumption of the controlled object; minimizing the number of commands when the controlled object operates the object; minimizing the time required for the controlled object to operate the object; and minimizing the difference between the three-dimensional position of at least one component included in the controlled object before the controlled object operates the object and the three-dimensional position of the at least one component after the controlled object operates the object.

[0085] (Supplementary Note 20) The instruction learning method according to Supplementary Note 19, wherein the evaluation unit includes a model that simulates the controlled object, and the model executes the second instruction and evaluates the second instruction based on the second condition.

[0086] (Supplementary Note 21) The command learning method according to Supplementary Note 12, further comprising an imaging unit that captures images of the control target and the object, wherein the imaging unit acquires a three-dimensional shape of the object.

[0087] (Supplementary Note 22) The command learning method according to Supplementary Note 12, further comprising a determination unit that determines whether or not the behavior of the control target based on the first command satisfies the first condition.

[0088] (Supplementary Note 23) A recording medium having a program recorded thereon, comprising: a generating step of generating first instructions including at least one instruction for a control target that operates an object; an evaluating step of evaluating second instructions from the first instructions that satisfy a first condition based on a second condition and calculating an evaluation amount; and a learning step of selecting and learning a third instruction from the second instructions that maximizes the evaluation amount, wherein the control target operates the object based on the third instruction.

[0089] (Supplementary Note 24) The recording medium according to Supplementary Note 23, wherein the control target includes at least one component, and the first condition includes that the first command can be set to the at least one component.

[0090] (Appendix 25) The recording medium described in Appendix 23, wherein the control object includes a plurality of components, and the first condition includes that when one component of the plurality of components operates based on the first command and another component of the plurality of components operates based on the first command, the one component and the other component do not physically interfere with each other.

[0091] (Supplementary Note 26) The recording medium according to Supplementary Note 23, wherein the control target includes at least one component, and the generation unit generates the first command based on the at least one component.

[0092] (Supplementary Note 27) The recording medium according to Supplementary Note 24, wherein the generation unit generates the first instruction including a combination of any value within a range of values ​​that one of the components can take and any value within a range of values ​​that another of the components can take.

[0093] (Supplementary Note 28) The recording medium according to Supplementary Note 23, wherein the generating unit generates a fourth instruction based on the first instruction and the second instruction.

[0094] (Supplementary Note 29) The recording medium according to Supplementary Note 23, wherein the generating unit generates a fourth instruction based on the first instruction and the third instruction.

[0095] (Appendix 30) The recording medium described in Appendix 23, wherein the second condition includes at least one of: minimizing the power consumption of the controlled object; minimizing the number of commands when the controlled object operates the object; minimizing the time required for the controlled object to operate the object; and minimizing the difference between the three-dimensional position of at least one component included in the controlled object before the controlled object operates the object and the three-dimensional position of the at least one component after the controlled object operates the object.

[0096] (Supplementary Note 31) The recording medium according to Supplementary Note 30, wherein the evaluation unit includes a model that simulates the controlled object, and the model executes the second command and evaluates the second command based on the second condition.

[0097] (Supplementary Note 32) The recording medium according to Supplementary Note 23, further comprising an imaging unit that captures images of the control target and the object, wherein the imaging unit acquires a three-dimensional shape of the object.

[0098] (Supplementary Note 33) The program according to Supplementary Note 23, further comprising a determination unit that determines whether or not an operation of the control target based on the first command satisfies the first condition.

[0099] This application claims priority based on Japanese Patent Application No. 2023-142841, filed September 4, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0100] According to the present disclosure, it is possible to provide an instruction learning device that does not require manual learning operations by a human.

[0101] 1: Computer 11: CPU 12: ROM 13: RAM 14: Storage device 15: Display device 16: Imaging device 17: Input / output IF 18: Communication IF 19: Bus 2: Robot controller 21: CPU 22: ROM 23: RAM 24: Storage device 25: Input / output IF 26: Communication IF 29: Bus 3: Robot 31: Lower arm 32: Upper arm 33: End effector 311: First axis 312: Second axis 313: Third axis 314: Fourth axis 315: Fifth axis 316: Sixth axis 4: Object 5: Command learning device 9: Command learning device 91: Generation unit 92: Evaluation unit 93: Learning unit

Claims

1. A command learning device comprising: a generation unit that generates first commands including at least one command for a control object that manipulates an object; an evaluation unit that evaluates second commands among the first commands that satisfy a first condition based on a second condition and calculates an evaluation amount; and a learning unit that selects and learns a third command from the second commands that maximizes the evaluation amount, wherein the control object manipulates the object based on the third command.

2. The instruction learning device according to claim 1, wherein the controlled object includes at least one component, and the first condition includes that the first instruction can be set to the at least one component.

3. The instruction learning device of claim 1, wherein the controlled object includes a plurality of components, and the first condition includes that when one component of the plurality of components operates based on the first instruction and another component of the plurality of components operates based on the first instruction, the one component and the other component do not physically interfere with each other.

4. The instruction learning device according to claim 1 or 2, wherein the control object includes at least one component, and the generation unit generates the first instruction based on the at least one component.

5. An instruction learning device as described in any one of claims 2 to 4, wherein the generation unit generates the first instruction including a combination of any value within a range of values ​​that one of the components can take and any value within a range of values ​​that another of the components can take.

6. An instruction learning device according to any one of claims 1 to 5, wherein the generation unit generates a fourth instruction based on the first instruction and the second instruction.

7. An instruction learning device according to any one of claims 1 to 6, wherein the generation unit generates a fourth instruction based on the first instruction and the third instruction.

8. An instruction learning device as described in any one of claims 1 to 7, wherein the second condition includes at least one of: minimizing the power consumption of the control object; minimizing the number of commands when the control object operates the object; minimizing the time required for the control object to operate the object; and minimizing the difference between the three-dimensional position of at least one component contained in the control object before the control object operates the object and the three-dimensional position of the at least one component after the control object operates the object.

9. An instruction learning device according to any one of claims 1 to 8, wherein the evaluation unit includes a model that simulates the controlled object, and the model executes the second instruction and evaluates the second instruction based on the second condition.

10. An instruction learning device according to any one of claims 1 to 9, further comprising an imaging unit that images the control target and the object, the imaging unit acquiring a three-dimensional shape of the object.

11. An instruction learning device according to any one of claims 1 to 10, further comprising a judgment unit that judges whether or not the operation of the controlled object based on the first instruction satisfies the first condition.

12. A command learning method comprising: a generating step of generating first commands including at least one command for a control object that operates an object; an evaluating step of evaluating second commands among the first commands that satisfy a first condition based on a second condition and calculating an evaluation amount; and a learning step of selecting and learning a third command among the second commands that maximizes the evaluation amount, wherein the control object operates the object based on the third command.

13. The method for learning instructions according to claim 12, wherein the controlled object includes at least one component, and the first condition includes that the first instruction can be set to the at least one component.

14. The command learning method of claim 12, wherein the controlled object includes a plurality of components, and the first condition includes that when one component of the plurality of components operates based on the first command and another component of the plurality of components operates based on the first command, the one component and the other component do not physically interfere with each other.

15. The instruction learning method according to claim 12 or 13, wherein the controlled object includes at least one component, and the generation unit generates the first instruction based on the at least one component.

16. A method for learning instructions as described in any one of claims 13 to 15, wherein the generation unit generates the first instruction including a combination of any value within a range of values ​​that one of the components can take and any value within a range of values ​​that another of the components can take.

17. The instruction learning method according to any one of claims 12 to 16, wherein the generation unit generates a fourth instruction based on the first instruction and the second instruction.

18. An instruction learning method according to any one of claims 12 to 17, wherein the generation unit generates a fourth instruction based on the first instruction and the third instruction.

19. A command learning method as described in any one of claims 12 to 18, wherein the second condition includes at least one of: minimizing the power consumption of the control object; minimizing the number of commands when the control object operates the object; minimizing the time required for the control object to operate the object; and minimizing the difference between a three-dimensional position of at least one component contained in the control object before the control object operates the object and a three-dimensional position of the at least one component after the control object operates the object.

20. A recording medium having a program recorded thereon, comprising: a generation step of generating first instructions including at least one instruction for a control object that operates an object; an evaluation step of evaluating second instructions from the first instructions that satisfy a first condition based on a second condition and calculating an evaluation amount; and a learning step of selecting and learning a third instruction from the second instructions that maximizes the evaluation amount, wherein the control object operates the object based on the third instruction.

Images