Robot control device, robot control method, and program

Abstract

To make it possible to suppress work complication while further reliably holding a work-piece.SOLUTION: A robot control device comprises: an image acquisition part which acquires a top face vision image of a work-piece; an image generation network which generates a side face vision image of the work-piece on the basis of the top face vision image of the work-piece acquired by the image acquisition part; a depth information acquisition part which acquires depth information of the work-piece on the basis of the side face vision image of the work-piece generated by the image generation network; a route learning part which has the depth information of the work-piece and joint information of a robot arm as input information, and has route information of the robot arm until the robot arm holds and lifts the work-piece as output information, and performs reinforcement learning so that an interference amount between the work-piece and other object around the work-piece until the robot arm holds and lifts the work-piece becomes minimum; and a control part which controls the robot arm on the basis of the route information of the robot arm outputted from the route learning part after learning.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a robot control device, a robot control method, and a program for controlling a robot arm. [Background technology]

[0002] 2. Description of the Related Art There is known a robot control device that identifies the type of a workpiece based on the position of the center of gravity of the workpiece, and holds the workpiece in a holding method suitable for the identified type of workpiece (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-067995 Summary of the Invention [Problem to be solved by the invention]

[0004] In the robot control device, the workpiece can be held more reliably by holding the workpiece in a holding method suitable for the type of workpiece. However, since the robot needs to be taught before holding the workpiece, the teaching work increases as the number of types of workpieces increases, and this may complicate the work.

[0005] The present disclosure has been made to solve such problems, and has as its main objective the provision of a robot control device, a robot control method, and a program that can hold a workpiece more securely while reducing the complexity of the work. [Means for solving the problem]

[0006] In order to achieve the above object, one aspect of the present disclosure is to A robot control device that controls a robot arm to hold and lift a workpiece, An image acquisition unit that acquires a top view image of the workpiece; an image generation network that generates a side view image of the workpiece based on the top view image of the workpiece acquired by the image acquisition unit; a depth information acquisition unit that acquires depth information of the workpiece based on a side view image of the workpiece generated by the image generation network; a path learning unit that uses depth information of the workpiece and joint information of the robot arm as input information, uses path information of the robot arm until the robot arm holds and lifts the workpiece as output information, and performs reinforcement learning so as to minimize an amount of interference between the workpiece and other objects around the workpiece until the robot arm holds and lifts the workpiece; a control unit that controls the robot arm based on the path information of the robot arm output from the path learning unit after learning; Equipped with Robot Control Device It is. In this aspect, the control unit controls the robot arm to hold and lift the workpiece based on the path information of the robot arm output from the path learning unit after learning, the path learning unit uses, as input information, joint information of the robot arm when the robot arm holds and lifts a workpiece and depth information of the workpiece, and re-learns so as to minimize an amount of interference between the workpiece and other objects around the workpiece; The control unit may control the robot arm based on the path information of the robot arm output from the path learning unit after the re-learning. In this aspect, the image generation network may be configured as a generative adversarial network. In order to achieve the above object, one aspect of the present disclosure is to 1. A method for controlling a robot arm to hold and lift a workpiece, comprising: acquiring a top view image of the workpiece; generating a side view image of the workpiece based on the acquired top view image of the workpiece; acquiring depth information of the workpiece based on the generated side view image of the workpiece; a step of performing reinforcement learning by a path learning unit using depth information of the workpiece and joint information of the robot arm as input information, path information of the robot arm until the robot arm holds and lifts the workpiece as output information, so that an amount of interference between the workpiece and other objects around the workpiece is minimized until the robot arm holds and lifts the workpiece; controlling the robot arm based on the path information of the robot arm output from the path learning unit after learning; Including, Robot Control Method It is. In order to achieve the above object, one aspect of the present disclosure is to A program for controlling a robot arm to hold and lift a workpiece, A process of generating a side view image of the workpiece based on the top view image of the workpiece; A process of acquiring depth information of the workpiece based on the generated side view image of the workpiece; a process of performing reinforcement learning by a path learning unit using depth information of the workpiece and joint information of the robot arm as input information, path information of the robot arm until the robot arm holds and lifts the workpiece as output information, so that an amount of interference between the workpiece and other objects around the workpiece is minimized until the robot arm holds and lifts the workpiece; A process of controlling the robot arm based on the path information of the robot arm output from the path learning unit after learning; A program that causes a computer to execute It is. Effect of the Invention

[0007] According to the present disclosure, it is possible to provide a robot control device, a robot control method, and a program that can more reliably hold a workpiece while preventing the work from becoming complicated. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a schematic system configuration of a robot system according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing a schematic system configuration of a robot control device according to an embodiment of the present invention. [Diagram 3] 3A to 3C are diagrams showing an example of a top view image, a side view image, and depth information of a workpiece; [Figure 4] 3 is a flowchart showing a flow of a robot control method according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the present embodiment will be described with reference to the drawings. Fig. 1 is a block diagram showing a schematic system configuration of a robot system according to the present embodiment. A robot control device 1 according to the present embodiment controls a robot arm 2 so as to hold and lift a workpiece placed at a predetermined position and move it to a target position, for example.

[0010] The robot control device 1 has a typical computer hardware configuration including, for example, a processor 1a such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), an internal memory 1b such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a storage device 1c such as an HDD (Hard Disk Drive) or an SDD (Solid State Drive), an input / output I / F 1d for connecting peripheral devices such as a display, and a communication I / F 1e for communicating with devices outside the device.

[0011] The robot arm 2 is configured as a multi-joint arm having, for example, a joint section (wrist joint, elbow joint, shoulder joint, etc.) 21 that connects multiple link members rotatably around a yaw axis, a pitch axis, and a roll axis, and an end effector 22 provided at the tip of the joint section and holding a workpiece.

[0012] Each joint 21 is provided with a rotation sensor 23, an actuator 24, and a force sensor 25. The rotation sensor 23 is configured with an encoder or the like, detects rotation information of each joint 21 (joint angle, joint angular velocity, joint angular acceleration, etc.), and outputs the information to the robot control device 1.

[0013] The actuator 24 is composed of a servo motor or the like, and rotates and drives each joint 21 in response to a control signal from the robot control device 1. The force sensor 25 is composed of a six-axis force sensor that detects torque information of the joint 21, such as the magnitude of force (Fx, Fy, Fz) acting in each direction of three axes (x-axis, y-axis, z-axis) and the magnitude of moment (Mx, My, Mz) around each axis, and outputs the torque information of the joint 21 to the robot control device 1. Each joint 21 is provided with a reduction mechanism or the like.

[0014] The end effector 22 is configured as, for example, a gripping mechanism having a plurality of fingers for gripping a workpiece. The gripping mechanism of the end effector 22 is configured to grip a workpiece by operating the fingers with an actuator such as a servo motor. The end effector 22 may be configured as a suction mechanism that sucks and holds the workpiece by suction force using magnetic force, air pressure, or the like.

[0015] 2 is a block diagram showing a schematic system configuration of a robot control device according to this embodiment. The robot control device 1 according to this embodiment includes an image generation network 11 that generates a side view image of a workpiece, a depth information acquisition unit 12 that acquires depth information of the workpiece, a path learning unit 13 that learns the path of the robot arm 2, and a control unit 14 that controls the operation of the robot arm 2.

[0016] However, for example, when a robot arm looks down at a part in a box, grasps it, lifts it up and takes it out, it does not know how high it should be lifted because it does not know the depth information of the workpiece. Therefore, when the robot arm lifts the workpiece, there is a risk that the workpiece will interfere with other objects in the vicinity and fall.

[0017] In contrast, in this embodiment, the image generation network 11 generates a side view image of the workpiece viewed from the side from a top view image of the workpiece viewed from above. Then, the depth information acquisition unit 12 acquires depth information of the workpiece based on the side view image of the workpiece generated by the image generation network 11. As a result, even when the robot arm 2 grasps, lifts, and removes the workpiece from above, the workpiece can be lifted to an appropriate height and removed reliably, taking into account the depth information of the workpiece.

[0018] For example, a camera 26 is provided on the end effector 22 or link of the robot arm 2, on environmental equipment above the workpiece, etc. The camera 26 is a specific example of an image acquisition unit. Before the robot arm 2 holds the workpiece, the camera 26 of the robot arm 2 captures an image of the workpiece from above. The camera 26 may output the captured image to the image generation network 11 as a top view image of the workpiece. The image generation network 11 generates a side view image of the workpiece based on the top view image of the workpiece from the camera 26.

[0019] The image generation network 11 may be configured as, for example, a generative adversarial network. A generative adversarial network (GAN) is composed of two networks: a generator that generates an image, and a discriminator that judges whether the image is correct or not.

[0020] The generative network learns to deceive the discriminative network. Meanwhile, the discriminative network learns to discriminate more accurately. In this way, the generative network and the discriminative network are configured to compete with each other and improve their respective accuracy. In this embodiment, by utilizing the characteristics of such a generative adversarial network, a highly accurate side view image of a workpiece can be generated from a top view image of the workpiece.

[0021] Fig. 3 is a diagram showing an example of a top view image, a side view image, and depth information of a workpiece. For example, as shown in Fig. 3, the image generation network 11 generates a side view image (b) of the workpiece based on a top view image (a) of the workpiece from the camera 26, and outputs the generated image to the depth information acquisition unit 12. The depth information acquisition unit 12 calculates depth information D of the workpiece based on the side view image (b) of the workpiece output from the image generation network 11.

[0022] Although the image generation network 11 is configured as a generative adversarial network, it is not limited to this and may be configured as any learning type network as long as it can generate a highly accurate side view image of a workpiece from a top view image of the workpiece.

[0023] The depth information acquisition unit 12 outputs the calculated depth information of the workpiece to the path learning unit 13. The path learning unit 13 performs reinforcement learning on the path information of the robot arm 2 from holding the workpiece to lifting it up.

[0024] In addition, the path information of the robot arm 2 until it holds and lifts the workpiece includes, for example, the path from when the end effector 22 of the robot arm 2 starts moving from a predetermined starting position, grasps and lifts the workpiece, and moves to the target position.

[0025] The route learning unit 13 is configured as a neural network including, for example, an input layer, a hidden layer, and an output layer. The route learning unit 13 may perform deep reinforcement learning using, for example, DQN (Deep Q-Network).

[0026] The path learning unit 13 receives depth information of the workpiece and joint information of the robot arm 2 as input information, and path information of the robot arm 2 as output information, and performs reinforcement learning to minimize the amount of interference between the workpiece and other objects around the workpiece until the robot arm 2 holds and lifts the workpiece.

[0027] By operating according to this path information, the robot arm 2 can minimize the amount of interference between the workpiece and other objects around it when it holds and lifts it. This can significantly increase the success rate of holding and lifting a workpiece by the robot arm 2. This makes it possible to automate the picking up of medium-sized parts that are large and difficult to pull out of a box, for example, and is expected to greatly advance factory automation.

[0028] The joint information of the robot arm 2 includes rotation information and torque information of each joint 21 of the robot arm 2. The path information of the robot arm 2 is, for example, the trajectory of the center of gravity of the end effector 22 until the robot arm 2 holds and lifts the workpiece.

[0029] Furthermore, minimizing the amount of interference between the workpiece and other objects around the workpiece means, for example, that the maximum or average torque value of the joint 21 detected by the force sensor 25 of the joint 21 during the time until the robot arm 2 holds and lifts the workpiece is minimized, or that the time until the robot arm 2 holds and lifts the workpiece is minimized.

[0030] The control unit 14 controls the operation of the robot arm 2 by controlling the actuators 24 of each joint 21 of the robot arm 2 based on the path information of the robot arm 2 output from the path learning unit 13 after learning. The control unit 14 performs, for example, feedback control or robust control on the actuators 24 of each joint 21 of the robot arm 2 so that the end effector 22 of the robot arm 2 moves on a trajectory according to the path information of the robot arm 2 output from the path learning unit 13 after learning.

[0031] As described above, the robot arm 2 can hold and lift a desired workpiece more reliably by simply operating according to the path information of the robot arm 2 output from the learned path learning unit 13. Therefore, since there is no need to perform complicated work such as teaching the robot arm 2, it is possible to hold and lift the workpiece more reliably while suppressing the complication of the work.

[0032] The path learning unit 13 may perform re-learning using the sensor values ​​obtained when the robot arm 2 is operated. This makes it possible to generate an optimal path for the robot arm 2 with less interference between the workpiece and other objects around the workpiece.

[0033] For example, the control unit 14 controls the robot arm 2 so that the end effector 22 holds and lifts a workpiece based on the path information of the robot arm 2 output from the path learning unit 13 after learning. At this time, the rotation sensor 23 of each joint 21 of the robot arm 2 detects the rotation information of each joint 21, and the force sensor 25 of each joint 21 detects the torque information of each joint 21.

[0034] The path learning unit 13 may perform re-learning using as input information the rotation information detected by the rotation sensor 23, the torque information detected by the force sensor 25, and the depth information of the workpiece from the depth information acquisition unit 12. Based on the path information of the robot arm 2 output from the path learning unit 13 after re-learning, the control unit 14 controls the robot arm 2 to hold, lift, and move the workpiece by the end effector 22.

[0035] Next, an example of a robot control method according to the present embodiment will be described with reference to a flowchart of FIG 4.

[0036] First, the control unit 14 calculates depth information indicating the height position of the workpiece based on the image of the workpiece captured by the camera 26. Based on the calculated depth information, the control unit 14 calculates the holding point of the workpiece when the end effector 22 of the robot arm 2 holds the workpiece (step S101).

[0037] The control unit 14 calculates a path of the robot arm 2 for the end effector 22 of the robot arm 2 to hold the workpiece at the calculated workpiece holding point. The control unit 14 controls the robot arm 2 so that the robot arm 2 operates according to the calculated path (step S102).

[0038] When the robot arm 2 operates and comes above the workpiece, the camera 26 acquires a top view image of the workpiece and outputs it to the image generation network 11 (step S103).

[0039] The image generation network 11 generates a side view image of the workpiece based on the top view image of the workpiece from the camera 26 (step S104).

[0040] The depth information acquisition unit 12 calculates depth information of the workpiece based on the side view image of the workpiece generated by the image generation network 11 (step S105).

[0041] The path learning unit 13 receives as input the depth information of the workpiece and the joint information of the robot arm 2, and outputs the path information of the robot arm 2, and performs reinforcement learning to minimize the amount of interference between the workpiece and other objects around the workpiece until the robot arm 2 holds and lifts the workpiece (step S106).

[0042] The control unit 14 controls the robot arm 2 based on the path information of the robot arm 2 output from the path learning unit 13 after learning, thereby holding and lifting the workpiece (step S107).

[0043] As described above, the robot control device 1 according to this embodiment includes a camera 26 that acquires a top view image of the workpiece, an image generation network 11 that generates a side view image of the workpiece based on the top view image of the workpiece acquired by the camera 26, a depth information acquisition unit 12 that acquires depth information of the workpiece based on the side view image of the workpiece generated by the image generation network 11, a route learning unit 13 that receives as input information the depth information of the workpiece and joint information of the robot arm 2 and outputs path information of the robot arm 2, and performs reinforcement learning to minimize the amount of interference between the workpiece and other objects around the workpiece until the robot arm 2 holds and lifts the workpiece, and a control unit 14 that controls the robot arm 2 based on the path information of the robot arm 2 output from the route learning unit 13 after learning.

[0044] According to the robot control device 1 of this embodiment, the image generation network 11 generates a side view image of the workpiece from a top view image of the workpiece. Then, the depth information acquisition unit 12 acquires depth information of the workpiece based on the side view image of the workpiece generated by the image generation network 11. As a result, even when the robot arm 2 holds, lifts, and removes the workpiece while viewing it from above, the workpiece can be lifted to an appropriate height and removed reliably, taking into account the depth information of the workpiece.

[0045] For example, the robot arm 2 can more reliably hold and lift a workpiece having a large depth dimension (a workpiece having a large area at the back that cannot be seen from above) by taking into account the depth information of the workpiece. This allows the robot arm 2 to hold and lift workpieces that were previously difficult to hold and lift, thereby increasing the variety of workpieces that can be held and lifted.

[0046] Although some embodiments of the present disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0047] The present disclosure can also be implemented by causing a processor to execute a computer program to perform the processes shown in FIG.

[0048] The program can be stored and provided to a computer using various types of non-transitory computer readable media. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memories)).

[0049] The program may be provided to the computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include an electric signal, an optical signal, and an electromagnetic wave. The transitory computer readable medium can provide the program to the computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0050] Each component of the robot control device 1 according to the embodiment described above can be realized not only by a program, but also in part or in whole by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). [Explanation of symbols]

[0051] REFERENCE SIGNS LIST 1 robot control device, 2 robot arm, 11 image generation network, 12 depth information acquisition unit, 13 path learning unit, 14 control unit, 21 joint unit, 22 end effector, 23 rotation sensor, 24 actuator, 25 force sensor, 26 camera

Claims

1. A robot control device that controls a robot arm to hold and lift a workpiece, An image acquisition unit that acquires a top view image of the workpiece; an image generation network that generates a side view image of the workpiece based on the top view image of the workpiece acquired by the image acquisition unit; a depth information acquisition unit that acquires depth information of the workpiece based on a side view image of the workpiece generated by the image generation network; a path learning unit that uses depth information of the workpiece and joint information of the robot arm as input information, uses path information of the robot arm until the robot arm holds and lifts the workpiece as output information, and performs reinforcement learning so as to minimize an amount of interference between the workpiece and other objects around the workpiece until the robot arm holds and lifts the workpiece; a control unit that controls the robot arm based on the path information of the robot arm output from the path learning unit after learning; Equipped with Robot control device.

2. The robot control device according to claim 1, the control unit controls the robot arm to hold and lift the workpiece based on the path information of the robot arm output from the path learning unit after learning, the path learning unit uses, as input information, joint information of the robot arm when the robot arm holds and lifts a workpiece and depth information of the workpiece, and re-learns so as to minimize an amount of interference between the workpiece and other objects around the workpiece; The control unit controls the robot arm based on the path information of the robot arm output from the path learning unit after the re-learning. Robot control device.

3. The robot control device according to claim 1 or 2, The image generation network is configured as a generative adversarial network. Robot control device.

4. 1. A method for controlling a robot arm to hold and lift a workpiece, comprising: acquiring a top view image of the workpiece; generating a side view image of the workpiece based on the acquired top view image of the workpiece; acquiring depth information of the workpiece based on the generated side view image of the workpiece; a step of performing reinforcement learning by a path learning unit using depth information of the workpiece and joint information of the robot arm as input information, path information of the robot arm until the robot arm holds and lifts the workpiece as output information, so that an amount of interference between the workpiece and other objects around the workpiece is minimized until the robot arm holds and lifts the workpiece; controlling the robot arm based on the path information of the robot arm output from the path learning unit after learning; Including, A method for controlling a robot.

5. A program for controlling a robot arm to hold and lift a workpiece, A process of generating a side view image of the workpiece based on the top view image of the workpiece; A process of acquiring depth information of the workpiece based on the generated side view image of the workpiece; a process of performing reinforcement learning by a path learning unit using depth information of the workpiece and joint information of the robot arm as input information, path information of the robot arm until the robot arm holds and lifts the workpiece as output information, so that an amount of interference between the workpiece and other objects around the workpiece is minimized until the robot arm holds and lifts the workpiece; A process of controlling the robot arm based on the path information of the robot arm output from the path learning unit after learning; A program that causes a computer to execute the following.

Images