Robot control system, robot control method and program

Abstract

To provide a robot steering system with increased learning model accuracy.SOLUTION: There is provided a robot steering system comprising: a synthetic image generation unit 209 which generates a plurality of synthetic images under random conditions from an object and CAD data of a workpiece 105 when the object, the CAD data of the workpiece, and a feature point 107 of the workpiece are input; an information processing device 215 which searches for a route by using a position of an end effector 103 of a robot and a position of the feature point, and moves the end effector to the feature point along the searched route; and an imaging unit 109 which images the object and the workpiece. The information processing device comprises: a storage unit 217 which stores a learned machine learning device which performs learning by inputting a plurality of training datasets composed of combinations of the synthetic images and the position of the feature point; and an arithmetic operation unit 219 which outputs the position of the feature point when a captured image of the workpiece is input to the learned machine learning device read from the storage unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a robot manipulation system, a robot manipulation method, and a program. [Background technology]

[0002] Patent Document 1 discloses an image processing system. The image processing system detects an image of an object from an image captured of the object. The image processing system includes a first detection device that detects the image of the object from the image based on a model pattern that represents the characteristics of the image of the object. The image processing system also includes a learning device that uses the image used for detection by the first detection device as input data and the detection result by the first detection device as training data to learn a learning model. The image processing system also includes a second detection device that detects the image of the object from the image based on the learning model learned by the learning device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-26599 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology of Patent Document 1 uses a workpiece model for learning and detects the workpiece from an image. However, because actual images vary in lighting conditions, workpiece mounting position, photographing conditions, etc., learning using model data may result in an inaccuracy of the learning model that is not as desired. Therefore, the purpose of this disclosure is to improve the accuracy of the learning model by having an information processing device generate multiple composite images of objects, workpieces, and feature points from CAD (Computer-Aided Design) data and learn from them. [Means for solving the problem]

[0005] The robot manipulation system of the present disclosure comprises: a composite image generation unit that receives CAD data of an object and a workpiece and feature points of the workpiece and generates a plurality of composite images under random conditions from the CAD data of the object and the workpiece; an information processing device that searches for a path using a position of an end effector of a robot and the positions of the feature points, and moves the end effector to the feature points along the path that has been searched; an imaging device that images the object and the workpiece, the information processing device includes a storage unit that stores a trained machine learning machine that learns by inputting a plurality of training data sets configured by combinations of the synthetic image and the positions of the feature points; The robot control system includes a calculation unit that outputs the positions of the feature points when an image of the workpiece is input to the trained machine learning machine read from the memory unit.

[0006] With the above configuration, the information processing device generates multiple composite images of objects, workpieces, and feature points from CAD data, and then learns them, improving the accuracy of the learning model. This makes it possible to provide a system that extracts feature points of a workpiece and controls a robot without teaching.

[0007] The robot manipulation system of the present disclosure comprises: The random conditions are characterized by various angles and positions of the object and the workpiece, various distances to the object and the workpiece, various exposure conditions, and various backgrounds.

[0008] The above configuration can improve the accuracy of extracting feature points of a workpiece.

[0009] The robot manipulation system of the present disclosure comprises: The plurality of composite images are generated with different sizes and aspect ratios.

[0010] The above configuration can improve the accuracy of workpiece recognition.

[0011] The robot manipulation system of the present disclosure comprises: The feature points are gripping points or welding points.

[0012] With the above configuration, the robot can grip or weld a workpiece.

[0013] The robot control method of the present disclosure includes: A step of inputting CAD data of an object and a workpiece and feature points of the workpiece; generating a plurality of composite images of the object and the workpiece under random conditions using CAD data; creating a trained machine learning machine that learns by inputting a plurality of training data sets each composed of a combination of the synthetic image and the position of the feature point; taking an image of the object and the workpiece; a step of inputting a captured image of the workpiece to the trained machine learning machine, thereby outputting the positions of the feature points; a step of searching for a path using the position of the end effector of the robot and the feature point, and moving the end effector to the feature point along the searched path.

[0014] With the above configuration, the information processing device generates multiple composite images of objects, workpieces, and feature points from CAD data, and then learns them, improving the accuracy of the learning model. This provides a method for extracting feature points of a workpiece and operating a robot without teaching.

[0015] The program of the present disclosure is A step of inputting CAD data of an object and a workpiece and feature points of the workpiece; The program causes an information processing device to execute a step of generating multiple composite images using CAD data of the object and the workpiece at various angles and positions of the object and the workpiece, various distances to the object and the workpiece, various exposure conditions, and various backgrounds.

[0016] The above configuration provides a program that can generate multiple composite images of an object, a workpiece, and feature points from CAD data. [Effects of the Invention]

[0017] This disclosure enables an information processing device to generate multiple composite images of an object, a workpiece, and its feature points from CAD data, and then learn the generated images, thereby improving the accuracy of the learning model. This makes it possible to provide a system that extracts the feature points of a workpiece and controls a robot without teaching. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a schematic diagram of a robot according to an embodiment; [Figure 2] FIG. 1 is a block diagram of a robot manipulation system according to an embodiment. [Figure 3] FIG. 1 is a schematic diagram of a workpiece according to an embodiment. [Figure 4] 1 is a flowchart of a robot control method according to an embodiment. [Figure 5] 1 is a flowchart of a robot control method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] Embodiment Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and duplicate explanations are omitted as necessary.

[0020] (Description of Robot According to Embodiment) 1 is a schematic diagram of a robot according to an embodiment, and the robot according to the embodiment will be described with reference to FIG.

[0021] The robot 101 includes an arm. The tip of the arm includes an end effector 103. The end effector 103 can grip or weld a workpiece 105. In other words, the robot 101 is a welding or transport robot.

[0022] The robot 101 grips or welds a feature point 107 of the workpiece 105. That is, the feature point 107 is a welding point or a gripping point. The robot 101 grips the feature point 107 to transport the workpiece 105 or welds it at the feature point 107. A single workpiece 105 may be provided with multiple feature points 107. For example, if a single workpiece 105 has multiple welding points, the multiple welding points become the feature points.

[0023] The imaging device 109 captures an image of the workpiece 105 and the robot 101. Specifically, the imaging device 109 captures an image of the workpiece 105, and the information processing device 215 recognizes the feature point 107. The information processing device 215 also recognizes the position of the end effector 103 of the robot 101. The imaging device 109 is used to move the end effector 103 of the robot 101 to the feature point 107 of the workpiece 105. In other words, the imaging device 109 is used to search for a path between the end effector 103 and the feature point 107, as indicated by the dotted line.

[0024] If the imaging device 109 is attached to a predetermined position on the robot 101, a path search can be performed simply by taking an image of the workpiece 105. This is because the information processing device 215 knows the position of the end effector 103 of the robot 101.

[0025] (Description of a robot control system according to an embodiment) Fig. 2 is a block diagram of a robot manipulation system according to an embodiment. Fig. 3 is a schematic diagram of a workpiece according to an embodiment. The robot manipulation system according to an embodiment will be described with reference to Figs. 2 and 3.

[0026] The robot control system 200 according to the embodiment includes a cloud 203, a server 205, or a web server 204. The robot control system 200 also includes an information processing device 207, an information processing device 215, a robot 101, and an imaging device 109.

[0027] The cloud 203 or the server 205 stores CAD data of the workpiece 105 designed by the designer 201 and feature points 107 of the workpiece 105. If the data can be stored, a personal computer's storage may be used instead of the cloud 203 or the server 205. Every time the designer 201 changes the design of the workpiece 105, the data is stored in the cloud 203 or the server 205.

[0028] The web server 204 stores CAD data and textures of objects. An object is an object on which a workpiece 105 is placed in a factory, such as a desk, chair, or jig. The information processing device 207 is connected to the web server 204 to widely collect CAD data of objects, but may also be connected to a closed server 205 or the cloud 203.

[0029] The information processing device 207 includes a composite image generation unit 209 , a composite image storage unit 211 , and a machine learning device 213 .

[0030] The composite image generation unit 209 captures images of the workpiece 105 and the object under random conditions and projects the feature points 107 to obtain a composite image. For example, if the object is a desk, the composite image generation unit 209 generates a composite image of the workpiece 105 placed on the desk. Random conditions include changing the texture, lighting, the angle of view of the imaging device, the filter, the aspect ratio, the size, and the background. Changing the texture means changing the color and texture. Changing the lighting conditions means changing the position, angle, color, and intensity of the lighting. Changing the angle of view of the imaging device means changing the position and orientation. Changing the filter means adding blur, noise, smog, and halation. These conditions are referred to as various angles and positions of the object and workpiece, various distances to the object, various exposure conditions, various backgrounds, etc.

[0031] The composite image storage unit 211 stores multiple composite images generated by the composite image generation unit 209. For use in machine learning, several million composite images are required. Normally, it takes several months to acquire images for learning. However, by having the composite image generation unit 209 create these composite images, materials can be collected in a few days.

[0032] The machine learning machine 213 learns by inputting multiple training data sets. Each training data set is composed of a combination of a synthetic image and the positions of feature points 107. The machine learning machine 213 learns the feature points 107 of the workpiece 105 by supervised learning. Supervised learning is a method in which, when the correct answer for the data to be learned is predetermined, a large amount of training data sets composed of combinations of the data to be learned (input) and the correct answer (output) are given in advance to learn the relationship between input and output. Because a synthetic image generated from CAD data is used, the coordinates of the feature points can be accurately determined.

[0033] A training dataset is provided to an untrained machine learning machine. The training dataset is composed of a composite image of a workpiece 105 as input and feature points 107 as output (ground truth labels). For example, the composite image of the workpiece 105 is text data indicating three-dimensional coordinates. Furthermore, for example, feature point data (vertex IDs) is text data indicating three-dimensional coordinates.

[0034] A trained machine learning machine 213 is created by inputting multiple composite images of workpieces into a machine learning machine using a neural network and making it learn. When an image of a workpiece 105 is input, the trained machine learning machine 213 outputs feature points 107. The machine learning machine uses, for example, deep learning to create a multi-layered neural network. Work The trained machine learning machine 213 may be a convolutional neural network (CNN) model having a convolutional layer and a pooling layer.

[0035] The information processing device 215 includes a storage unit 217 and a calculation unit 219. The information processing device 215 is connected to the robot 101 and the imaging device 109.

[0036] The memory unit 217 stores the trained machine learning machine 213. The calculation unit 219 is connected to the memory unit 217 and the imaging device 109. The calculation unit 219 outputs the positions of the feature points 107 of the workpiece 105 by inputting a captured image of the workpiece to the trained machine learning machine 213 read from the memory unit 217.

[0037] The imaging device 109 captures an image of the robot 101 and the workpiece 105, or captures an image of the workpiece 105. The imaging device 109 may capture an image of the workpiece 105 placed on an object such as a desk. In other words, the imaging device 109 may capture an image of the workpiece together with the object. Therefore, the captured image includes the workpiece and the object. The information processing device 215 uses the captured image captured by the imaging device 109 to search for a path between the position of the end effector 103 and the position of the feature point 107. The information processing device 215 moves the end effector 103 to the feature point 107 along the searched path.

[0038] Such a robot manipulation system 200 can improve the accuracy of the learning model by generating multiple composite images from CAD data and learning from them. Here, the composite image includes an object and a workpiece 105. The composite image is accompanied by coordinate data of feature points 107. Also, normally, path search is not possible unless the feature points 107 of the workpiece 105 and the positions of the end effector 103 are taught. However, the robot manipulation system of the embodiment can automatically detect the feature points 107 of the workpiece and the positions of the end effector 103 and perform path search, thereby providing a system for manipulating a robot without teaching.

[0039] Furthermore, the composite image generation unit 209 generates a composite image including the workpiece 105 and the object. The composite image generation unit 209 generates a plurality of composite images under various conditions. In each training data set, data on the feature points 107 is associated with the composite image. The machine learning machine 213 is obtained by machine learning using a plurality of training data sets. Therefore, it is possible to extract feature points with high accuracy from a captured image of the workpiece 105 placed on the object.

[0040] (Description of a robot control method according to an embodiment) Fig. 4 is a flowchart of a robot control method according to an embodiment. Fig. 5 is a flowchart of a robot control method according to an embodiment. The robot control method according to an embodiment will be described with reference to Figs. 4 and 5.

[0041] First, the information processing device 207 monitors the CAD server or cloud (step S401). Next, the information processing device 207 determines whether the CAD data has been updated (step S402). The CAD server or cloud is monitored to determine whether the designer 201 has updated the CAD data. If the CAD data has not been updated (NO in step S402), the information processing device 207 monitors the CAD server or cloud again (step S401). If the CAD data has been updated (YES in step S402), the information processing device 207 reads the CAD data, feature points 107, and textures of the workpiece 105 and object (step S403). The information processing device 207 reads the updated CAD data of the workpiece 105 and object, the feature points 107 of the workpiece 105, and the textures of the object.

[0042] Next, the CAD data of the workpiece and object are randomly arranged (step S404). The workpiece 105 and object are randomly arranged and photographed in the CAD space of the composite image generation unit 209 of the information processing device 207. Next, the vertex positions in the CAD data of the workpiece are extracted using the vertex IDs in the feature point list (step S405). The composite image generation unit 209 finds vertices of the workpiece 105 that match the vertex IDs in the feature point list. Next, labels and segmentations are attached to the extracted vertices (step S406). The composite image generation unit 209 attaches labels and segmentations indicating that the vertices of the workpiece 105 are important. Next, the textures of the workpiece and object are randomly changed and photographed (step S407). Next, the lighting conditions are randomly changed and photographed (step S408). Next, the angle of view and filter of the imaging device model are randomly changed and photographed (step S409). The vertex positions are projected onto the photographed data to create an annotated composite image (step S410). The composite image generating unit 209 generates a composite image by adding feature points to the captured data.

[0043] Next, it is determined whether the designated number of images has been captured (step S411). If the designated number of images has not been captured (NO in step S411), the process returns to step S404, and images of the workpiece 105 and the object are captured. If the designated number of images has been captured (YES in step S411), the process continues to step S501 in FIG. 5.

[0044] If the specified number of images has been reached (YES in step S411), multiple composite images are added to the dataset (step S501). The information processing device 207 adds the composite images to the training dataset. Next, the dataset is input into the learning device (step S502). The information processing device 207 adds the dataset to the machine learning device 213 and causes it to learn. Next, the output model is sent to the information processing device, and the model of the information processing device is updated (step S503). The information processing device 207 sends the model to the information processing device 215, and updates the model stored in the storage unit 217.

[0045] Next, an image of the workpiece is acquired from the imaging device (step S504). The imaging device 109 captures an image of the workpiece 105. Next, the feature points of the workpiece are estimated (step S505). The calculation unit 219 of the information processing device 215 estimates the feature points 107 of the captured image of the workpiece 105 from the learning results. Next, a path search is performed using the position of the robot's end effector and the position of the feature points (step S506). The information processing device 215 performs a path search using the position of the end effector 103 of the robot 101 and the position of the feature points 107. Finally, after the path search, the robot is started (step S507) and the process ends. After the path search, the information processing device 215 operates the robot 101 and moves the end effector 103 to the feature point 107.

[0046] By generating a composite image of multiple objects, workpieces, and their feature points from CAD data and learning from the composite image, the accuracy of the learning model can be improved, providing a method for extracting the feature points of a workpiece and operating a robot without teaching.

[0047] Furthermore, some or all of the processes in the information processing device 207, the composite image generation unit 209, and the information processing device 215 described above can be realized as a computer program. Such a program can be stored on various types of non-transitory computer-readable media and supplied to a computer. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0048] By executing the above-described program of the composite image generating unit 209, a composite image of a plurality of objects, workpieces, and feature points can be generated from CAD data.

[0049] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present disclosure. For example, the information processing device 207 and the information processing device 215 are separate devices, but they may be combined into one device. Some of the functions of the information processing device 207 may be located in the cloud. [Explanation of symbols]

[0050] 101 Robot 103 End Effector 105 Work 107 minutiae 109 Imaging Device 200 Robot Control System 201 Designer 203 Cloud 204 Web Server 205 Server 207 Information processing equipment 209 Synthetic Image Generation Unit 211 Composite Image Storage Unit 213 Machine Learning Machine 215 Information processing equipment 217 Memory section 219 Arithmetic section

Claims

1. a composite image generation unit that receives CAD data of an object and a workpiece and feature points of the workpiece and generates a plurality of composite images under random conditions from the CAD data of the object and the workpiece; an information processing device that searches for a path using a position of an end effector of a robot and the positions of the feature points, and moves the end effector to the feature points along the path that has been searched; an imaging device that captures images of the end effector of the robot, the object, and the workpiece; The information processing device includes: a storage unit that stores a trained machine learning machine that learns by inputting a plurality of training data sets configured by combinations of the synthetic image and the positions of the feature points; a calculation unit that outputs the positions of the feature points by inputting a captured image of the workpiece to the trained machine learning machine read from the storage unit, The information processing device searches for the path using images of the robot's end effector and the feature points captured by the imaging device.

2. The robotic system of claim 1 , wherein the random conditions are different angles and positions of the object and the workpiece, different distances to the object and the workpiece, different exposure conditions, and different backgrounds.

3. The robotic manipulation system of claim 1 , wherein the composite images are generated with varying sizes and aspect ratios.

4. The robotic manipulation system of claim 1 , wherein the feature point is a grip point or a welding point.

5. A step of inputting CAD data of an object and a workpiece and feature points of the workpiece; generating a plurality of composite images under random conditions using the CAD data of the object and the workpiece; creating a trained machine learning machine that learns by inputting a plurality of training data sets each composed of a combination of the synthetic image and the position of the feature point; imaging an end effector of a robot, the object, and the workpiece; a step of inputting a captured image of the workpiece to the trained machine learning machine, thereby outputting the positions of the feature points; A robot operation method comprising: a step of searching for a path using the positions of the end effector of the robot and the feature points, and moving the end effector to the feature points along the searched path, wherein the path is searched for using captured images of the end effector of the robot and the feature points.

6. A step of inputting CAD data of an object and a workpiece and feature points of the workpiece; generating a plurality of composite images of the object and the workpiece using CAD data of the object and the workpiece at different angles and positions of the object and the workpiece, different distances to the object and the workpiece, different exposure conditions, and different backgrounds; creating a trained machine learning machine that learns by inputting a plurality of training data sets each composed of a combination of the synthetic image and the position of the feature point; imaging an end effector of a robot, the object, and the workpiece; a step of inputting a captured image of the workpiece to the trained machine learning machine, thereby outputting the positions of the feature points; a step of searching for a path using the positions of the end effector of the robot and the feature points, and moving the end effector to the feature points along the path that has been searched for, wherein the path is searched for using captured images of the end effector of the robot and the feature points.

Images