Robot system, control method, image processing apparatus, image processing method, method of manufacturing products, program, and recording medium

Abstract

To shorten the time required for image processing related to recognition of a workpiece.SOLUTION: A robot system comprises: a robot; an imaging device; an image processing unit which specifies at least one region where a prescribed object in a prescribed attitude exists in an image acquired by imaging a plurality of objects by the imaging device and acquires information about the position and / or attitude of the prescribed object in the region; and a control unit which controls the robot so as to make the robot hold the prescribed object based on the information about the position and / or attitude of the prescribed object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to image processing. [Background technology]

[0002] In factories, tasks such as kitting, which involves arranging workpieces in their designated positions, and assembly, which involves fitting or inserting workpieces into other workpieces to assemble items, are performed. Industrial robots are used in these tasks, and factory automation is being implemented. Among these tasks, there is a picking operation, which involves taking out workpieces one by one from a pile of multiple workpieces.

[0003] Patent Document 1 describes image processing such as pattern matching, which involves capturing images of multiple workpieces stacked in a random order using a camera and comparing the captured images with a pre-acquired teaching model. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2000-293695 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, conventional methods required significant time for image processing related to workpiece recognition. Therefore, from the perspective of productivity in the production line, there was a need to reduce the time required for image processing related to workpiece recognition.

[0006] Therefore, the present invention aims to shorten the time required for image processing related to workpiece recognition. [Means for solving the problem]

[0007] The robot system of the present invention is characterized by comprising: a robot; an imaging device; an image processing unit that identifies at least one region in an image of multiple objects captured by the imaging device in which a predetermined object is in a predetermined posture, and acquires information on the position and / or posture of the predetermined object in the region; and a control unit that controls the robot to hold the predetermined object based on the information on the position and / or posture of the predetermined object.

[0008] The present invention relates to a control method for a robot system comprising a robot and an imaging device, characterized in that an image processing unit identifies at least one region in an image of multiple objects captured by the imaging device in which a predetermined object is in a predetermined posture, acquires information on the position and / or posture of the predetermined object in the region, and a control unit controls the robot to hold the object based on the information on the position and / or posture of the predetermined object.

[0009] The present invention is characterized by comprising an image processing unit that identifies at least one region in an image in which multiple objects are captured that contains a predetermined object in a predetermined posture, and acquires information on the position and / or posture of the predetermined object in the region.

[0010] The present invention relates to an image processing method characterized in that an image processing unit identifies at least one region in an image in which multiple objects are captured that contains a predetermined object in a predetermined posture, and the image processing unit acquires information on the position and / or posture of the predetermined object in the region. [Effects of the Invention]

[0011] According to the present invention, the time required for image processing related to workpiece recognition can be reduced. [Brief explanation of the drawing]

[0012] [Figure 1]It is an explanatory diagram showing a schematic configuration of a robot system according to the first embodiment. [Figure 2] It is an explanatory diagram of an image processing apparatus according to the first embodiment. [Figure 3] It is a block diagram of a computer system in a robot system according to the first embodiment. [Figure 4] (a) is a block diagram showing the functions of a CPU according to the first embodiment. (b) is a schematic diagram for explaining the state of a workpiece. [Figure 5] It is a flowchart of a robot control method according to the first embodiment. [Figure 6] (a) is a schematic diagram showing an example of an image according to the first embodiment. (b) is a schematic diagram for explaining the detection process according to the first embodiment. [Figure 7] (a) is an explanatory diagram of a tagging operation according to the first embodiment. (b) is an explanatory diagram of a tagging operation according to the first embodiment. [Figure 8] (a) is a schematic diagram showing an example of a recognition result according to the first embodiment. (b) is a schematic diagram for explaining a method of calculating the exposure of a workpiece according to the first embodiment. [Figure 9] (a) is a graph showing an example of the height of a workpiece corresponding to each candidate region according to the first embodiment. (b) is a table showing an example of priorities according to the first embodiment. [Figure 10] (a) is a schematic diagram of an image obtained from a camera according to the first embodiment. (b) is a schematic diagram for explaining the pattern matching process of a comparative example. (c) is a schematic diagram for explaining the pattern matching process according to the first embodiment. [Figure 11] It is a diagram showing an example of the display on a display according to the first embodiment. [Figure 12] It is a flowchart of a robot control method according to the second embodiment. [Figure 13] It is a block diagram showing the functions of a CPU according to the third embodiment. [Figure 14](a) is a schematic diagram illustrating the trimming process according to the third embodiment. (b) is a schematic diagram illustrating the trained model according to the third embodiment. [Figure 15] This is a flowchart showing a robot control method according to the fourth embodiment. [Figure 16] (a) is a schematic diagram showing an example of an image according to the fifth embodiment. (b) is a schematic diagram for explaining the detection process according to the fifth embodiment. [Figure 17] This is a flowchart of the robot control method according to the sixth embodiment. [Figure 18] This figure shows a plurality of candidate holding positions according to the sixth embodiment. [Figure 19] This is a flowchart of the robot control method according to the seventh embodiment. [Figure 20] Figures (a) to (d) are schematic diagrams showing two workpieces overlapping according to the seventh embodiment. [Figure 21] This diagram illustrates interference detection between a modified robot hand and a workpiece. [Modes for carrying out the invention]

[0013] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

[0014] [First Embodiment] Figure 1 is an explanatory diagram showing the schematic configuration of a robot system 10 according to the first embodiment. The robot system 10 comprises a robot 100, an image processing device 200, a robot controller 300 which is an example of a control device, and an imaging system 400 which is an example of an imaging device. The robot 100 is an industrial robot, is placed on a production line, and is used for manufacturing goods.

[0015] Robot 100 is a manipulator. Robot 100 is fixed to a stand, for example. A container 30 with an open top and a mounting platform 40 are arranged around robot 100. Multiple workpieces W are loosely packed inside container 30. Each workpiece W is an example of an object, for example, a part. Multiple workpieces W in container 30 are picked up one by one by robot 100 and transported to a predetermined position on the mounting platform 40. Multiple workpieces W are the same shape, the same size, and the same color, and are randomly arranged inside container 30. Workpieces W are flat members, and their front and back surfaces have different shapes.

[0016] Robot 100 and robot controller 300 are connected by wiring to enable communication. Robot controller 300 and image processing device 200 are connected by wiring to enable communication. Imaging system 400 and image processing device 200 are connected by wired or wireless connection to enable communication.

[0017] The robot 100 includes a robot arm 101 and an end effector, i.e., a robot hand 102 which is an example of a holding mechanism. The robot arm 101 is a vertically articulated robot arm. The robot hand 102 is supported by the robot arm 101. The robot hand 102 is attached to a predetermined part of the robot arm 101, for example, the tip of the robot arm 101. The robot hand 102 is configured to hold a workpiece W. Although the description will focus on the case where the holding mechanism is the robot hand 102, it is not limited to this, and for example, the holding mechanism may be a suction mechanism capable of holding a workpiece by suction. In the first embodiment, the robot hand 102 is configured to grip the workpiece W.

[0018] With the above configuration, the robot arm 101 can move the robot hand 102 to any position, allowing the robot 100 to perform a desired task. For example, by preparing a workpiece W and other workpieces as materials, and having the robot 100 perform the process of assembling workpiece W onto the other workpieces, an assembled workpiece can be manufactured as a deliverable. Thus, an article can be manufactured by the robot 100. In this embodiment, the manufacturing of an article by assembling workpieces by the robot 100 has been described as an example, but it is not limited to this. For example, the robot arm 101 may be equipped with tools such as cutting tools or grinding tools, and the article may be manufactured by processing the workpiece with the tools.

[0019] The imaging system 400 includes a camera 401, which is an example of a first imaging unit, and a camera 402, which is an example of a second imaging unit. Each camera 401 and 402 is a digital camera. Camera 401 is fixed to a frame (not shown). Camera 401 is positioned to image an area including a plurality of workpieces W placed in the container 30. That is, camera 401 is capable of imaging an area including the workpieces W, which are objects to be held by the robot 100.

[0020] Camera 402 is supported by the robot 100 by being attached to a predetermined part of the robot 100, for example, the robot hand 102. The imaging position of camera 402, that is, the imaging area captured by camera 402, can be freely changed according to the posture of the robot 100. Specifically, camera 402 can be moved by the movement of the robot 100 to a position closer to the multiple workpieces W that are loosely packed in the container 30 than camera 401, and can image a narrower area than camera 401. Furthermore, camera 402 can be moved by the movement of the robot 100 to the air above the workpiece W that is to be held by the robot 100.

[0021] In the first embodiment, the image processing device 200 is configured as a computer. The image processing device 200 can send an imaging command to the camera 401 to cause the camera 401 to take an image. The image processing device 200 can also send an imaging command to the camera 402 to cause the camera 402 to take an image. The image processing device 200 is configured to acquire image I1, which is an example of a first image generated by the camera 401, and to process the acquired image I1. The image processing device 200 is also configured to acquire image I2, which is an example of a second image generated by the camera 402, and to process the acquired image I2. Figure 2 is an explanatory diagram of the image processing device 200 according to the first embodiment. The image processing device 200 includes a main unit 201, a display 202, which is an example of a display device connected to the main unit 201, and a keyboard 203 and a mouse 204, which are examples of input devices connected to the main unit 201.

[0022] In the first embodiment, the robot controller 300 shown in Figure 1 is configured as a computer. The robot controller 300 is configured to control the movement of the robot 100, that is, the posture of the robot 100.

[0023] Figure 3 is a block diagram of the computer system in the robot system 10 according to the first embodiment. The main body 201 of the image processing device 200 includes a CPU (Central Processing Unit) 251, which is an example of a processor. The CPU 251 is an example of an image processing unit. The main body 201 also includes a ROM (Read Only Memory) 252, a RAM (Random Access Memory) 253, and an HDD (Hard Disk Drive) 254 as storage units. The main body 201 also includes a recording disk drive 255 and an input / output interface 256. The CPU 251, ROM 252, RAM 253, HDD 254, recording disk drive 255, and interface 256 are connected to each other by a bus so that they can communicate with one another.

[0024] ROM252 stores the basic programs related to the operation of the computer. RAM253 is a memory device that temporarily stores various data, such as the results of calculations performed by CPU251. HDD254 records the results of calculations performed by CPU251 and various data acquired from external sources, as well as programs 261 that cause CPU251 to execute various processes. Program 261 is application software that CPU251 can execute.

[0025] The CPU 251 can perform the image processing described later by executing the program 261 recorded on the HDD 254. The recording disk drive 255 can read various data and programs recorded on the recording disk 262.

[0026] In the first embodiment, the non-temporary recording medium readable by the computer is the HDD254, and the program 261 is recorded on the HDD254, but this is not the only embodiment. The program 261 may be recorded on any recording medium that is a non-temporary recording medium readable by the computer. Examples of recording media that can be used to supply the program 261 to the computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory, etc.

[0027] The robot controller 300 includes a CPU 351, which is an example of a processor. The CPU 351 is an example of a control unit. The robot controller 300 also includes a ROM 352, RAM 353, and HDD 354 as storage units. Furthermore, the robot controller 300 includes a recording disk drive 355 and an input / output interface 356. The CPU 351, ROM 352, RAM 353, HDD 354, recording disk drive 355, and interface 356 are connected to each other by a bus so that they can communicate with one another.

[0028] ROM352 stores the basic programs related to the operation of the computer. RAM353 is a memory device that temporarily stores various data, such as the results of calculations performed by CPU351. HDD354 records the results of calculations performed by CPU351 and various data acquired from external sources, as well as programs 361 that cause CPU351 to execute various processes. Program 361 is application software that CPU351 can execute.

[0029] The CPU 351 executes control processing by running program 361 recorded on HDD 354, enabling control of the robot 100 shown in Figure 1. The recording disk drive 355 can read various data and programs recorded on recording disk 362.

[0030] In the first embodiment, the non-temporary recording medium readable by the computer is the HDD 354, and the program 361 is recorded on the HDD 354, but this is not the only embodiment. The program 361 may be recorded on any recording medium that is a non-temporary recording medium readable by the computer. Examples of recording media that can be used to supply the program 361 to the computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory, etc.

[0031] In the first embodiment, image processing and control processing are performed by multiple computers, i.e., multiple CPUs 251, 351, but the system is not limited to this. Image processing and control processing may also be performed by a single computer, i.e., a single CPU. In this case, the single CPU can be configured to function as both the image processing unit and the control unit.

[0032] The CPU 251, by executing program 261, functions to cause camera 401 to image an area where multiple workpieces W exist, and to detect workpieces W that can be extracted from the image I1 obtained from the image. That is, the CPU 251 functions to identify a predetermined area containing workpiece images corresponding to the extractable workpieces W. Hereinafter, this predetermined area will be referred to as the search area. The CPU 251 also functions to cause camera 402 to image an area in real space corresponding to the search area by executing program 261, and to measure the position and orientation of the workpieces W using pattern matching processing, which is an example of image processing.

[0033] The CPU 351, by executing program 361, controls the robot 100 to move the camera 402 to a position where it can image the real-world area corresponding to the search area. The CPU 351 also, by executing program 361, moves the robot hand 102 to a position where it can hold the workpiece W measured by the CPU 251. The CPU 351 also, by executing program 361, causes the robot hand 102 to hold the workpiece W and transfer the workpiece W to the mounting table 40.

[0034] Figure 4(a) is a block diagram showing the functions of the CPU 251 according to the first embodiment. By executing the program 261, the CPU 251 functions as a work detection unit 211, a recognition unit 212, a height detection unit 213, a priority determination unit 214, and a measurement unit 215. The processing operations of each camera 401, 402 and each unit 211-215 will be broadly described below.

[0035] Camera 401 can capture an area containing multiple workpieces W and output an image I1 consisting of RGB color gradations. In the first embodiment, the first imaging unit is camera 401, but it is not limited to this. The first imaging unit can be any unit capable of digitally quantifying the characteristics of the workpieces W.

[0036] The workpiece detection unit 211 detects at least one candidate region containing an image of a workpiece W, i.e., a workpiece image, from the image I1 acquired from the camera 401. The recognition unit 212 recognizes the state of the workpiece W based on the candidate region detected by the workpiece detection unit 211. Here, the state of the workpiece W refers to the orientation information of the workpiece W. Figure 4(b) is a schematic diagram illustrating the state of the workpiece W. The state of the workpiece W, i.e., the orientation information of the workpiece W, is information indicating which of the front surface F1 and back surface F2 of the workpiece W is facing upwards, as seen from the camera 401, as shown in Figure 4(b). Front surface F1 is an example of a first surface, and back surface F2 is an example of a second surface different from the first surface. Back surface F2 has a different shape from front surface F1.

[0037] The height detection unit 213 detects the vertical height of the workpiece W relative to a reference position for each workpiece W corresponding to the workpiece image contained in each candidate region, if at least one candidate region is two or more candidate regions. The height detection unit 213 detects the height of the workpiece W using a sensor (not shown) that outputs a signal corresponding to the height of the workpiece W. The sensor (not shown) is, for example, a Time of Flight (ToF) height sensor or a depth sensor that outputs a distance image. Alternatively, the height detection unit 213 may detect the height of the workpiece W using a three-dimensional camera (not shown) that outputs an RGB image and a 3D point cloud. In this case, the three-dimensional camera may be integrated with camera 401. The priority determination unit 214 assigns a priority to each of the multiple candidate regions detected by the workpiece detection unit 211 in order of ease of acquisition of the workpiece by the robot 100, and extracts the candidate region with the highest priority. The candidate region with the highest priority is the search region described above. The workpiece W corresponding to the workpiece image contained in the search region is the object to be held by the robot 100. The object to be held by the robot 100 is an example of a predetermined object.

[0038] Camera 402 captures an imaging area that includes the workpiece W corresponding to the workpiece image contained in the search area within image I1, and is narrower than the imaging area of ​​camera 401. The image I2 obtained by camera 402 is, for example, an RGB grayscale image. Camera 402 outputs a distance image including height information along with image I2, if necessary.

[0039] The measurement unit 215 performs pattern matching on the image I2 acquired from the camera 402 to obtain three-dimensional information regarding the position and orientation of the workpiece W. In this process, the measurement unit 215 performs pattern matching based on the orientation information of the workpiece W obtained by the recognition unit 212. This reduces the amount of computation required.

[0040] The following describes in detail the control method of the robot 100, including the image processing method according to the first embodiment. Figure 5 is a flowchart of the control method of the robot 100 according to the first embodiment.

[0041] The workpiece detection unit 211 sends an imaging command to the camera 401, causing the camera 401 to image an area containing multiple workpieces W (S101). As a result, the camera 401 images multiple workpieces W that are loosely packed in the container 30. Among the multiple workpieces W are objects to be held by the robot 100. The workpiece detection unit 211 acquires an image I1 containing multiple workpieces W from the camera 401.

[0042] Figure 6(a) is a schematic diagram showing an example of image I1 according to the first embodiment. As shown in Figure 6(a), image I1 contains multiple work images WI corresponding to multiple bulk-stacked workpieces W, as a grayscale image. Note that image I1 may also include objects other than the workpieces W, such as the container 30 that stores the workpieces W. In the example in Figure 6(a), an image 30I corresponding to the container 30 is included.

[0043] Next, the workpiece detection unit 211 performs a detection process to detect the workpiece W from the image I1 (S102). Figure 6(b) is a schematic diagram illustrating the detection process according to the first embodiment. If the workpiece W detection process is successful, as shown in Figure 6(b), two or more regions enclosed by dashed lines are obtained in the image I1, for example, four candidate regions A, B, C, D. Each candidate region A to D is a rectangular region that encloses one workpiece image WI. Each candidate region A to D contains one workpiece image WI, and the posture information of the workpiece W corresponding to the workpiece image WI is associated with it. Therefore, in step S102, the workpiece detection unit 211 extracts two or more workpiece images WI from the image I1 by detecting two or more candidate regions A to D.

[0044] In image I1, multiple workpieces W are captured in a scattered state, and the imaging area is also large, so it takes time to extract the workpieces W to be held by the robot 100 using pattern matching. In the first embodiment, candidate regions A to D are detected by a process other than pattern matching. An image recognition method called object detection is used to detect candidate regions A to D, and in the first embodiment, a learning-based image recognition method using deep learning will be explained as an example.

[0045] In object detection, the work detection unit 211 uses a trained model 263 to find a work image WI corresponding to a work W in image I1 and outputs candidate rectangular regions A to D surrounding the work image WI. Each candidate region A to D contains a work image WI corresponding to a work W in a predetermined orientation. That is, the work detection unit 211 identifies candidate regions A to D in image I1 where a work image WI corresponding to a work W in a predetermined orientation exists. In the first embodiment, the predetermined orientation is either orientation P10, which is the first orientation in which the surface F1 of the work W shown in Figure 4(b) is imaged, or orientation P20, which is the second orientation in which the back surface F2 of the work W is imaged. Therefore, each candidate region A to D contains a work image WI corresponding to a work W in orientation P10 or orientation P20.

[0046] To find workpiece W using deep learning, it is necessary to train the model on the features of workpiece W in images captured by camera 401. For training, multiple training datasets are prepared, each consisting of input and output data. Raw grayscale images are used as input data, and tagged data, which includes grayscale images and tag information attached to the grayscale images, is used as output data. The tagging process is performed by a human operator. Figure 7(a) is an explanatory diagram of the tagging process according to the first embodiment. For training, a large number of grayscale images I0 are prepared. Figure 7(a) shows one of these grayscale images I0. In the first embodiment, image I0 is obtained by imaging an object with a shape corresponding to the workpiece W held by the robot hand 102. Hereinafter, the object used for training will be referred to as workpiece W, and the image corresponding to the object included in image I0 will also be referred to as workpiece image WI.

[0047] The operator designates a rectangular region R0 that encloses the workpiece image WI contained in image I0, and associates information about the state of the workpiece W with this region R0. Region R0 is a part of image I0. Region R0 is specified by the starting coordinates P1 and ending coordinates P2. That is, the rectangular area with the starting coordinates P1 and ending coordinates P2 as diagonals is designated as region R0. The information indicating the state of the workpiece W is orientation information with a defined domain, for example, information indicating the front surface F1 and back surface F2 of the workpiece W. The information indicating the state of the workpiece W is assigned in association with region R0.

[0048] Figure 7(b) is an explanatory diagram of the tagging operation according to the first embodiment. First, as shown in Figure 7(b), the case where the surface F1 of the workpiece W is facing upward will be explained. When the surface F1 of the workpiece W is facing upward, the surface F1 of the workpiece W will be imaged. For the orientation of the workpiece W in which the axis C1 perpendicular to the surface F1 is included within a solid angle α1 with respect to a specific plane, for example, the axis C0 perpendicular to the horizontal plane, orientation information T1 indicating that the surface F1 of the workpiece W is facing upward is assigned. When the back surface F2 of the workpiece W is facing upward, the back surface F2 of the workpiece W will be imaged. Similarly, when the back surface F2 of the workpiece W is facing upward, orientation information indicating that the back surface F2 of the workpiece W is facing upward is assigned. The CPU 251 acquires a trained model 263 using a plurality of images I0 including the image of the surface F1 of the workpiece W and the image of the back surface F2 of the workpiece W. Note that even if the surfaces are different, the same orientation information may be assigned if they look and have the same shape. For example, if the front surface F1 and the back surface F2 have the same appearance and shape, both may be assigned pose information T1. Conversely, if the shapes are the same but the appearances are different, for example, if the numbers on the faces of a die are different, it becomes possible to recognize the numbers on the faces of the die by assigning different pose information to each face of the die. In this way, the operator can register multiple training datasets with the image processing device 200 by specifying the region R0 and pose information for the raw grayscale image I0 obtained from the camera 401. The CPU 251 of the image processing device 200 can then obtain a trained model 263 by performing machine learning using a predetermined learning algorithm with the multiple training datasets. The trained model 263 is stored, for example, in the HDD 254.

[0049] As learning algorithms, algorithms such as SSD (Single Shot Multibox Detector) and YOLO (You Look Only Once) can be used. However, the algorithm is not limited to these and any learning algorithm can be used as long as it can output candidate regions A to D, including information indicating the state of work W. Furthermore, as mentioned above, actual captured images may be used for preparing the training dataset, or images generated in a virtual space such as a physical simulator may be used.

[0050] In the first embodiment, the CPU 251 acquires a trained model 263 used to detect candidate regions by training a neural network on numerous patterns of grayscale features of the workpiece image WI corresponding to the workpiece W, as well as the contours of the workpieces captured in the grayscale image I0, i.e., edge information. As a result, even if multiple workpieces W are piled up in various random states, the CPU 251 can recognize a specific workpiece W from among the multiple randomly piled workpieces W, i.e., extract a candidate region.

[0051] The recognition unit 212 recognizes the state of the workpiece W corresponding to each of the multiple candidate regions A to D detected by the workpiece detection unit 211 using the trained model 263 (S103). This step S103 can be performed together with the detection of candidate regions A to D in step S102 by using algorithms such as SSD or YOLO. In this way, the recognition unit 212 can determine the orientation of the workpiece W corresponding to each candidate region A to D as the state of the workpiece W corresponding to each candidate region A to D using the pre-set trained model 263.

[0052] Figure 8(a) is a schematic diagram showing an example of the recognition result according to the first embodiment. Figure 8(a) illustrates a case where all of the multiple candidate regions A to D are assigned attitude information T1 indicating that the surface F1 of the workpiece W is facing upwards. However, the accuracy of the attitude information of the workpiece W obtained in step S103 is low for the robot 100 to hold the workpiece W. Therefore, in the first embodiment, the attitude of the workpiece W is determined with high accuracy by the pattern matching process in step S108, which will be described later.

[0053] Next, the height detection unit 213 detects the vertical height of the workpiece W corresponding to the workpiece image WI contained in each of the multiple candidate regions A to D (S104). The height detection unit 213 detects the maximum height of the part of the workpiece W corresponding to each candidate region A to D. This makes it possible to compare the height difference of the workpiece W between the multiple candidate regions A to D. For the robot 100, among the multiple workpieces W that are piled up, it is easier to pick up workpieces W that are at a higher position than workpieces W that are at a lower position. This is because workpieces W at a lower position are more likely to act as a fulcrum supporting workpieces W above them, and when picking up a workpiece W, other workpieces W are more likely to collapse or other workpieces W are more likely to be pulled along with the workpiece W being picked up. Therefore, in order to use this for prioritizing the workpieces W in the next step S105, the height detection unit 213 detects the height of the workpiece W corresponding to each candidate region A to D.

[0054] Next, the priority determination unit 214 selects the workpiece W to be held by the robot 100 (S105). The priority determination unit 214 assigns a priority to each candidate region A to D based on multiple factors. Each factor indicates how easy it is for the robot 100 to hold the workpiece W, that is, the success rate of retrieval. For example, there are three factors 1 to 3 shown below.

[0055] Factor 1: Exposure of workpiece W x 1 The exposure degree x1 of workpiece W refers to the ratio of the visible area to the total area of ​​a specific surface of workpiece W, such as the front surface F1 or back surface F2. A lower exposure degree x1 of workpiece W indicates a denser overlap of multiple workpiece Ws, while a higher exposure degree x1 indicates a sparser overlap. Therefore, a higher exposure degree x1 of workpiece W reduces the probability of failure in retrieving that workpiece W, thus increasing the success rate of retrieving that workpiece W. The calculation method for this factor 1 will now be explained.

[0056] Figure 8(b) is a schematic diagram illustrating the method for calculating the exposure degree x1 of a workpiece according to the first embodiment. The workpiece image WI in image I1 is used to calculate the exposure degree x1 of the workpiece W. The exposure degree x1 is a numerical value in the range of 0 to 1.

[0057] When focusing on a single work image WI, x1 = 1 if the entire area of ​​a specific surface of work W is visible in the work image WI. If half the area of ​​a specific surface of work W is visible in the work image WI, x1 = 0.5. In this way, the exposure x1 of work W is calculated based on the area of ​​the exposed portion of work W as seen from camera 401. This calculation result can be obtained by training a pre-trained object detection model 263, such as SSD or YOLO, with only work W where x1 = 1. That is, object detection methods such as SSD and YOLO can output the similarity of objects to the trained data set using a probability density function such as the softmax function, and have the characteristic that the similarity is reduced according to the ratio of visible area. The exposure x1 of work W can be used to determine this. Note that the method for measuring the exposure x1 of work W is not limited to this, and other methods may be used.

[0058] Factor 2: Spread of work W x 2 The degree of scattering x2 of workpieces W is the positional relationship between multiple candidate regions. In a scattered state of multiple workpieces W, the position and orientation of each workpiece W are random. Therefore, within the region where each workpiece W exists, there is a bias towards densely packed and sparsely packed areas of workpieces W. In densely packed areas of workpieces W, entanglement between workpieces W and collapse due to external forces are more likely to occur probabilistically. Therefore, in areas where workpieces W are sparse, that is, in areas separated from densely packed areas of workpieces W and isolated, the success rate of extraction tends to be relatively higher. For these reasons, it is preferable to take into account the degree of scattering x2, which indicates the degree of isolation of workpieces W. For example, in the case of candidate region A shown in Figure 6(b), the degree of scattering x2 can be calculated as shown in the following equation (1).

number

[0059] Here, S(A) represents the area of ​​candidate region A. The dispersion degree x2 is calculated by subtracting the common area with other candidate regions B to D from the area of ​​candidate region A itself. Therefore, the more densely packed the workpiece W is, the closer the dispersion degree x2 approaches 0, and the more isolated the workpiece W is, the closer the dispersion degree x2 approaches 1. Consequently, the higher the dispersion degree x2, the higher the priority for extraction. The difference between exposure degree x1 and dispersion degree x2 will be explained. Exposure degree x1 takes into account the hidden parts of workpiece W that could not be detected by the object detection process, whereas dispersion degree x2 differs in that it takes into account the density of workpiece W that could be detected by the object detection process.

[0060] Factor 3: Height of the workpiece W in a bulk state x 3 Height x3 is the vertical height of the workpiece W relative to the ground. In the first embodiment, the ground is the bottom surface of the container 30.

[0061] When robot 100 approaches a workpiece W to be retrieved from above multiple workpieces W in a scattered state, the closer the workpiece W to be retrieved is to the upper part of the multiple workpieces W, the higher the success rate of retrieving the workpiece W. If there are multiple workpieces W with the same exposure x1 and dispersion x2, among these multiple workpieces W, the workpiece W at a higher position is prioritized in order to increase the success rate of retrieval. For example, the height x3 of the workpiece W corresponding to candidate region A is calculated by the following equation (2).

number

[0062] Here, H(A) represents the height value of the workpiece W corresponding to candidate region A, and is a value detected by the height detection unit 213. The height information H(A) may be the maximum height or the average value of the workpiece W corresponding to candidate region A, and can be selected as appropriate.

[0063] Figure 9(a) is a graph showing an example of the height x3 of workpiece W corresponding to each candidate region A to D according to the first embodiment. As shown in Figure 9(a), the height x3 is calculated as the ratio of the height of workpiece W corresponding to each candidate region to the maximum height among all workpiece W corresponding to all candidate regions A to D. In other words, the height x3 approaches 0 as the height of the workpiece W is lower than the maximum height, and approaches 1 as it is closer to the maximum height. Therefore, the higher the height x3, the higher the priority for retrieval.

[0064] Next, the priority determination unit 214 uses the following formula (3), which includes the three factors 1 to 3 described above, as multiple factors, to determine the index f used to determine the priority. i The index f is calculated by multiplying each factor 1 to 3 by a weight coefficient and summing them up, as shown in equation (3). i Calculate the index f. i This indicates the ease of removing the workpiece in each candidate area.

number

[0065] Here, the subscript i means the number assigned to each of the plurality of candidate regions. For example, when four candidate regions A to D are detected, i takes values from 1 to 4. The index f i is an evaluation value indicating the ease of taking out for each candidate region. Also, α, β, and γ are coefficients that sum to 1, and the values of the coefficients can be freely set depending on which factor is emphasized. The index f corresponding to the work W i The higher it is, the easier it is for the robot 100 to hold the work W.

[0066] The priority determination unit 214 calculates the index f i for all candidate regions A to D. Then, the priority determination unit 214 determines the priority by comparing the magnitude relationship of the indices f i corresponding to all candidate regions A to D. That is, the priority determination unit 214 calculates the index f i using a plurality of factors 1 to 3, and determines the priority of the work W to be taken out from the container 30 based on the index f i .

[0067] In the first embodiment, the priority determination unit 214 determines the candidate region with the largest index f i as the search region for searching for the position and orientation of the work W to be used in the next step S106. That is, the priority determination unit 214 sets the candidate region with the highest priority among the two or more detected candidate regions as the search region. The search region includes the work image corresponding to the object to be held by the robot 100. Thereby, the priority determination unit 214 sets one of the two or more works W as the object to be held by the robot 100. More specifically, the priority determination unit 214 obtains the priority for each of the two or more works W, and selects the one with the highest priority among the two or more works W as the object to be held by the robot 100.

[0068] Figure 9(b) is a table showing an example of priority according to the first embodiment. For example, if α=0.3, β=0.4, and γ=0.3 in equation (3), the weight coefficient β of the dispersion x2 is given more weight, and candidate region D has the highest priority. Note that index f i The calculation of the priority was explained using the example of using three factors 1 to 3, but it is not limited to this, and if there are other factors related to the success rate of extraction, they may be added to equation (3). In this way, the priority determination unit 214 calculates the priority for each of the two or more workpieces W using multiple factors 1 to 3.

[0069] In step S102, if the workpiece detection unit 211 detects only one candidate region, the priority determination unit 214 determines that candidate region as the search region in step S105. Therefore, in this case, the priority calculation process and the height detection process for that purpose can be omitted.

[0070] Next, the CPU 351 of the robot controller 300 moves the robot 100 to a position where the camera 402 is close to the workpiece W, which is the object selected by the priority determination unit 214 (S106). Here, the position where the camera 402 is close to the workpiece W is a position above the workpiece W, where the real space region corresponding to the search area is included within the imaging area of ​​the camera 402. At this time, it is preferable to align the center of the imaging area of ​​the camera 402 with the center of the real space region corresponding to the search area. In this way, by bringing the camera 402 close to the workpiece W, which is the object selected by the priority determination unit 214, the imaging target can be narrowed down to one of the multiple workpieces W that are piled up, and the resolution of the image I2 obtained by imaging the real space region corresponding to the search area can be increased. For this reason, the processing time of the pattern matching process performed in step S108, which will be described later, can be shortened.

[0071] Next, the measurement unit 215 sends an imaging command to the camera 402, causing the camera 402 to image the workpiece W, which is the object selected by the priority determination unit 214 (S107). The image I2 acquired from the camera 402 may be a grayscale image or a three-dimensional point cloud image including depth information, as long as it is in a format that can be used for pattern matching processing in the next step S108. In the first embodiment, the image I2 is a grayscale image. Figure 10(a) is a schematic diagram of the image I2 obtained by imaging by the camera 402 according to the first embodiment. The image I2 is, for example, as shown in Figure 10(a), an image in which the workpiece W, which is the object selected by the priority determination unit 214, is captured as a workpiece image WI.

[0072] Next, the measurement unit 215 performs pattern matching processing on the image I2 (S108) to determine the position and orientation of the workpiece W, which is the object selected by the priority determination unit 214. In the first embodiment, the accuracy of the orientation information of the workpiece W obtained by the pattern matching processing is higher than the accuracy of the orientation information of the workpiece W obtained in step S103. The pattern matching processing is performed on the image I2 output by the camera 402 using CAD data 260 that has been previously stored in a storage device such as the HDD 254 in Figure 3. The CAD data 260 is a model that shows the shape of the workpiece W, which is the object, and holds the edge information, i.e., line segment information, of the workpiece W. In the first embodiment, the measurement unit 215 performs preprocessing such as edge detection on the image I2, which is a grayscale image, and compares the line segment information obtained therefrom with the line segment information of the CAD data 260 to calculate the degree of agreement as a score.

[0073] Here, we will explain the pattern matching process for the comparative example. Figure 10(b) is a schematic diagram illustrating the pattern matching process for the comparative example. In the pattern matching process for the comparative example, as shown in Figure 10(b), line segment information from 260 CAD data points for all orientations in each 360° axis is used. Then, the matching is performed comprehensively in an interlaced manner starting from the upper left pixel of image I2. For this reason, the order of the calculation process is O(n), based on the number of axes representing the orientation. 3This results in a large amount of computation and a long processing time.

[0074] Figure 10(c) is a schematic diagram illustrating the pattern matching process according to the first embodiment. In the first embodiment, the measurement unit 215 performs matching processing based on the posture information obtained by the recognition unit 212. More specifically, the measurement unit 215 determines the posture of the CAD data model used in the pattern matching process based on the posture of the workpiece W recognized by the recognition unit 212. For example, since the recognition unit 212 has already recognized that the posture of the workpiece W included in the search area is within a predetermined angle α1 with respect to a specific plane, for example, an axis perpendicular to the horizontal plane, the posture of the edges of the CAD data 260 used for matching in the pattern matching process can be reduced to within an angle α1 on each axis. As a result, the more precisely the angles recognized by the recognition unit 212 are set, the greater the load on the pattern matching process can be reduced, and the processing time can be shortened.

[0075] The recognition results obtained in this way may be displayed on the screen of the display 202 as shown in Figure 11 to inform the operator, i.e., the user. The CPU 251 displays the work display unit 202a and the work detail display unit 202b on the screen of the display 202. The work detail display unit 202b includes the detailed content display unit 202c. In the work display unit 202a, the candidate areas acquired in step S102 are displayed along with the priority acquired in step S105. In the work detail display unit 202b, the number of detected workpieces and the number of the workpiece to be retrieved are displayed. In the example shown in Figure 11, 4 is displayed as the number of detected workpieces, and the workpiece with priority 1 is displayed as the workpiece number. Also in the example shown in Figure 11, the detailed content display unit 202c displays the detailed content of the workpiece with priority 1. In the detailed content display unit 202c, factor information such as exposure, dispersion, and height acquired in step S105, and the coordinate information and score information of the workpiece acquired in step S108 are displayed. In the example shown in Figure 11, the coordinate information of the workpiece is displayed as (100,100,200,0°,0°,45°). Also in the example shown in Figure 11, the score information is displayed as 0.95, the exposure is displayed as 0.9, the dispersion is displayed as 0.8, and the height is displayed as 0.8. As a result, by looking at the screen of display 202, the worker can easily determine which workpiece to prioritize picking up from the randomly stacked workpieces and what the condition of the workpiece being picked up is.

[0076] Furthermore, since the center of the search area coincides with the center of the imaging area of ​​camera 402, the workpiece image WI to be searched is located near the center of image I2. Therefore, instead of interlacing the entire image I2 with the model, it is possible to narrow down the pattern matching process to the area near the center of image I2, taking into account the size of the workpiece W. Combining these processes, the processing time required for pattern matching can be reduced.

[0077] Next, the CPU 351 of the robot controller 300 controls the robot 100 to move to a holding position where the robot hand 102 will hold the workpiece W, based on the position and orientation information of the workpiece W obtained as a result of the pattern matching process (S109). Then, the CPU 351 controls the robot 100 to have the robot hand 102 hold the workpiece W (S110). Then, the CPU 351 controls the robot 100 to transport the workpiece W to a predetermined position on the platform 40 (S111). When taking the next workpiece W from the container 30, the process returns to step S101 and the process from step S101 to step S110 is repeated, allowing the robot 100 to continuously pick up the bulk-stacked workpieces W one by one.

[0078] By performing the picking operation of workpieces W according to the flowchart above, it is possible to narrow down the search area even when workpieces W are in a loose stack. Furthermore, by prioritizing the selection of workpieces that are easy for the robot 100 to pick up, the success rate of the robot 100 acquiring workpieces can be increased. In addition, by using the trained model 263, the number of easily pickable workpieces that can be recognized can be increased. Moreover, while conventional methods performed pattern matching without narrowing down the area of ​​the acquired image, in this first embodiment, pattern matching is performed by narrowing down to the search area, thus shortening the processing time required for pattern matching. As a result, the productivity of the robot 100 in handling goods is improved.

[0079] [Second Embodiment] A second embodiment will now be described. Figure 12 is a flowchart of the robot control method according to the second embodiment. In the second embodiment, the configuration of the robot system is the same as in the first embodiment, so its explanation will be omitted. Also, the processes of steps S201 to S211 shown in Figure 12 are the same as the processes of steps S101 to S111 shown in Figure 5, which were described in the first embodiment, so their explanation will be omitted.

[0080] After removing the workpiece W from the container 30, in step S212, the priority determination unit 214 determines whether there are any candidate regions other than the candidate region selected as the search region in step S205.

[0081] If a candidate area exists (S212: YES), the priority determination unit 214 returns to the process in step S205 and sets the candidate area with the next highest priority as the new search area. In this case, the calculation of priority is omitted. That is, in step S205, the priority determination unit 214 selects another workpiece W with the next highest priority from among the two or more workpieces W other than the object previously selected by the priority determination unit 214. The other workpiece W selected by the priority determination unit 214 is the object to be held by the robot 100 next.

[0082] In the example of the first embodiment, after removing the workpiece W corresponding to the candidate region D from the container 30, the ease of removal index f i The next best option is work W, which corresponds to candidate region B. Therefore, the priority determination unit 214 sets candidate region B as a new search region. This completes the processing in steps S201 to S204, and the index f in step S205. i The calculation process can be omitted.

[0083] Here, the processing in steps S201 to S204, and the index f in step S205. i The calculation process can be omitted if the position and orientation of the next workpiece W have not changed significantly. Therefore, a separate monitoring device may be provided to monitor the position and orientation of the workpiece W. Alternatively, after the robot 100 removes the workpiece W from the container 30, the CPU 251 may have the camera 401 take another image and determine whether the state has changed in the candidate region excluding the search area by comparing the difference between the two images. It should be noted that either the second embodiment or its modified version may be combined with either the first embodiment or its modified version described above.

[0084] [Third Embodiment] A third embodiment will now be described. In the first embodiment, in the process of step S105 in Figure 5, the priority determination unit 214 uses three factors 1 to 3 to determine index f i Calculate the index f i The case of determining the priority of workpieces W to be removed from container 30 based on this was explained. However, the index f i The number of factors used in the calculation is not limited to three. In the third embodiment, index f i In addition to the three factors 1 to 3 described in the first embodiment, we will now describe a case in which another factor is added as a factor used in the calculation. The factor added in the third embodiment is a factor of the success rate of workpiece extraction based on past extraction results. Note that the configuration of the robot system according to the third embodiment is the same as in the first embodiment, so the explanation will be omitted.

[0085] Figure 13 is a block diagram showing the functions of the CPU 251 according to the third embodiment. By executing the program 261, the CPU 251 functions as a workpiece detection unit 211, a recognition unit 212, a height detection unit 213, a priority determination unit 214, and a measurement unit 215, in addition to a verification unit 216. The verification unit 216 confirms whether the robot 100 succeeded or failed in holding the workpiece W in step S110 of Figure 5.

[0086] In the third embodiment, a physical detection sensor (not shown) is provided on the robot hand 102 to check for the presence or absence of the workpiece W, and the confirmation unit 216 checks whether the holding of the workpiece W was successful or unsuccessful based on the signal from the physical detection sensor (not shown). Alternatively, the confirmation unit 216 may acquire an image of the robot hand 102 from an imaging device (not shown) and check whether the holding of the workpiece W was successful or unsuccessful based on that image.

[0087] The verification unit 216 is configured to output a binary value depending on whether or not workpiece W is present. For example, the verification unit 216 outputs "1" if it confirms that workpiece W is present, and "0" if it confirms that workpiece W is absent.

[0088] Next, the verification unit 216 trims the search area corresponding to the workpiece W from the image I1. Figure 14(a) is a schematic diagram illustrating the trimming process according to the third embodiment. In the example in Figure 14(a), the candidate area D is being trimmed.

[0089] Figure 14(b) is a schematic diagram illustrating the trained model according to the third embodiment. A dataset is prepared containing information on whether the holding of work W was successful or unsuccessful, and a trimmed image. This allows the neural network to learn the correlation between the trimmed image and the information on whether or not the holding of work W was successful. The neural network used here can be any model used in classification, such as the VGG (Visual Geometry Group) algorithm. In this way, a trained model is created in which the trimmed image and the success or failure of holding are associated. This trained model can be stored in a storage device such as the HDD254 in Figure 3. This allows the priority determination unit 214 to estimate the success rate x4 when it tried to have the robot 100 hold work W in the past, based on past retrieval results, for each candidate region obtained in the process of step S102 in Figure 5. Therefore, adding the success rate x4 to equation (3) gives the index f i This is expressed as shown in equation (4) below.

number

[0090] In this way, by determining the priority of the workpieces to be retrieved, including the success rate of past retrieval results, the success rate of retrieval can be increased. Furthermore, the third embodiment and any of its variations may be combined with any of the various embodiments and their variations described above.

[0091] [Fourth Embodiment] A fourth embodiment will now be described. In the imaging system 400 shown in Figure 1, either camera 401 or camera 402 may be omitted. In the fourth embodiment, the case in which camera 402 is omitted in the robot system 10 shown in Figure 1 will be described. Figure 15 is a flowchart of the robot control method according to the fourth embodiment. The flowchart shown in Figure 15 shows the case in which close-range imaging is not performed.

[0092] The processes in steps S401 to S405 of the fourth embodiment are the same as the processes in steps S101 to S105 shown in Figure 5 of the first embodiment, so their explanation will be omitted.

[0093] In step S406, the measurement unit 215 obtains image data of the search region by cropping the search region described in the first embodiment from the image I1 acquired from the camera 401. For example, as shown in Figure 14(a), if candidate region D is determined to be the search region, region D is cropped. The search region is a sub-region of image I1. The workpiece W, which is the target object, is captured in the search region.

[0094] In step S407, the measurement unit 215 performs pattern matching on the trimmed search area. At this time, the measurement unit 215 also performs pattern matching based on the posture information of the workpiece W recognized by the recognition unit 212. This reduces the amount of computation required for pattern matching and shortens the processing time required for pattern matching.

[0095] The processing in steps S408 to S410 is the same as the processing in steps S109 to S111 shown in Figure 5 in the first embodiment, so the explanation is omitted.

[0096] Furthermore, if the camera 401 can narrow down the search area and capture an image using a zoom lens or the like, the camera 402 may be omitted in the imaging system 400, and image I2 may be generated by camera 401. Alternatively, camera 401 may be omitted in the imaging system 400. In this case, the position of camera 402 may be adjusted by the movement of the robot 100, and image I1 may be generated by camera 402. It should also be noted that the fourth embodiment and any of its modifications may be combined with any of the various embodiments and their modifications described above.

[0097] [Fifth Embodiment] A fifth embodiment will now be described. In the first embodiment described above, the work detection unit 211 in Figure 4(a) was described as detecting work W as a rectangular candidate region using SSD or YOLO as a deep learning learning algorithm. However, the shape of the detected candidate region is not limited to a rectangle. In the fifth embodiment, a case in which an algorithm for detecting work W in a region other than a rectangle is used will be described. An example of an algorithm that can be used in the fifth embodiment is Instance Segmentation. Figure 16(a) is a schematic diagram showing an example of image I2 according to the fifth embodiment. For learning, many images I2 like Figure 16(a) are prepared. The operator sets a region R1 that surrounds the work image WI along the contour of the work image WI contained in image I2, and associates region R1 with information indicating the state of the work image WI surrounded by region R1. Then, the image processing device 200 is trained with this information using an algorithm and a neural network. The information indicating the state of work W and the training by the neural network are the same as those described using Figure 7(b) of the first embodiment, so the explanation will be omitted.

[0098] Figure 16(b) is a schematic diagram illustrating the detection process according to the fifth embodiment. From Figure 16(b), the work detection unit 211 uses Instance Segmentation as its algorithm and is trained to output regions along the contour of the work image WI as candidate regions. In this way, the work detection unit 211 can output candidate regions E to J as shown in Figure 16(b). These candidate regions E to J are linked to information about the state of the work W. Therefore, when the measurement unit 215 performs pattern matching processing, it can perform pattern matching processing within the contour of the target work. Thus, in the case of a rectangle, there is a possibility that the processing target region will include areas outside the contour of the work, which are unnecessary for pattern matching processing. However, by performing pattern matching processing within the contour of the work, the time required for pattern matching processing can be shortened. In the fifth embodiment, Instance Segmentation was described as an example of an algorithm that can output the result of linking regions along the contour of the work image WI with information about the state of the work W, but other algorithms with similar functionality may be used. Furthermore, the fifth embodiment and any of its variations may be combined with any of the various embodiments and their variations described above.

[0099] [Sixth Embodiment] A sixth embodiment will now be described. In the first embodiment described above, the process of moving the robot hand 102 to the holding position in step S109 of Figure 5 and holding the workpiece in step S110 was explained. The holding position is ultimately one location, but there may be more than one candidate holding position for a single workpiece. In the sixth embodiment, a method for determining one holding position from among multiple candidate holding positions to increase the success rate of retrieval will be described.

[0100] Figure 17 is a flowchart of the robot control method according to the sixth embodiment. In the sixth embodiment, the configuration of the robot system is the same as in the first embodiment, so its explanation is omitted. Also, the processes of steps S601 to S608 and S610 to S612 in Figure 17 are the same as the processes of steps S101 to S108 and S109 to S111 shown in Figure 5, which were explained in the first embodiment, so their explanation is omitted. Figure 18 is a schematic diagram showing a plurality of holding position candidates set for the workpiece W.

[0101] In the sixth embodiment, a step S609 is added after the pattern matching process in step S608, in which a holding position is selected from among a plurality of holding position candidates. The plurality of holding position candidates are determined by the operator, as shown in Figure 18. Figure 18 shows the reference position and orientation O0 of the workpiece W, the position and orientation O1 of holding position candidate K1, and the position and orientation O2 of holding position candidate K2. That is, the information of position and orientation O1 and O2 can be arbitrarily determined by the operator, and the relative position and orientation relationship of position and orientation O1 and O2 as seen from the reference position and orientation O0 can be known by the operator.

[0102] When pattern matching is performed in step S608, the reference position and orientation O0 of the workpiece W is determined as the position and orientation of the workpiece W as viewed from another coordinate system (not shown). The other coordinate system mentioned here is, for example, the coordinate system of the robot system 10. Since the relative positions and orientations of positions and orientations O1 and O2 as viewed from the reference position and orientation O0 are known, the positions and orientations of each holding position candidate K1 and K2 as viewed from the other coordinate system can be obtained. Similarly, the height of each holding position candidate K1 and K2 from the ground, i.e., the height from the bottom surface of the container 30, is obtained by the method described in the first embodiment.

[0103] In step S609, the CPU 251 selects one of these candidate holding positions K1 and K2 to be the final holding position. For this selection method, one of the following factors can be used, for example. Factor 1: It is located at the highest point from the ground, i.e., the bottom surface of container 30. Factor 2: It is located closest to the center of gravity of the workpiece. Factor 3: The robot arm 101 can approach in the shortest possible time. Taking these factors into account is advantageous in the following ways:

[0104] When using Factor 1, considering the state of the loosely stacked parts, the higher a workpiece is located in the upper part of the loose stack, the more likely it is to be exposed to other workpieces. Conversely, the lower a workpiece is located in the loose stack, the more likely it is to be densely packed and in close contact with other workpieces. Therefore, by selecting the upper holding position candidate from among multiple holding position candidates as the holding position, it is possible to reduce the chance of other workpieces being removed simultaneously when the target workpiece is removed.

[0105] When using factor 2, centrifugal force is generated on the workpiece when the robot 100 moves the gripped workpiece after it has grasped it. In this case, a large centrifugal force is generated near the center of gravity of the workpiece, so gripping the workpiece closer to its center of gravity makes it easier to generate a counterforce against the moment of inertia generated by the centrifugal force, thereby reducing the likelihood of the workpiece falling off the robot hand 102 during movement.

[0106] When using factor 3, the approach can be made in the shortest time, and the robot arm 101 moves only as much as possible, thus reducing the chance of failure to pick up the workpiece due to interference between the robot arm 101 and other workpieces.

[0107] As described above, according to the sixth embodiment, a holding position that can reduce the failure of workpiece removal can be selected from among multiple holding position candidates, depending on the condition of the workpiece at the time of workpiece removal. Therefore, the success rate of workpiece removal can be improved. Of course, it is also possible to combine two or more of these multiple factors in a complex manner to ultimately select one holding position. Furthermore, it is also possible to implement the sixth embodiment and any of its modified versions in combination with any of the various embodiments and their modified versions described above.

[0108] [Seventh Embodiment] A seventh embodiment will now be described. In the sixth embodiment described above, after performing pattern matching in step S608 of Figure 17, the process proceeds to step S609, which is the process of selecting the holding position. In the seventh embodiment, a control method is described in which a step is added to determine interference between the robot hand 102 and workpieces W other than the workpiece W to be picked up, before selecting the holding position. Figure 19 is a flowchart of the robot control method according to the seventh embodiment. Note that in the seventh embodiment, the configuration of the robot system is the same as in the first embodiment, so the explanation is omitted. In Figure 19, the processes of steps S701 to S708 and S710 to S713 are the same as the processes of steps S601 to S612 shown in Figure 17, which were described in the sixth embodiment, so the explanation is omitted.

[0109] In the seventh embodiment, the CPU 251 obtains multiple holding position candidates by performing pattern matching in step S708, similar to the sixth embodiment. Next, in step S709, the CPU 251 determines whether the robot hand 102 can approach each holding position candidate without interfering with any workpieces W other than the workpiece W to be picked up. The height information obtained in step S704 is used for interference determination.

[0110] Figures 20(a) to 20(d) are diagrams that virtually and schematically show the state in which the workpiece W1 to be removed and the other workpiece W2 overlap in the seventh embodiment. For explanatory purposes, workpiece W2 is shown as transparent in Figures 20(a) to 20(d). Figure 20(a) shows the workpiece W1 detected in the pattern matching process of step S708 and its reference position and orientation O0, and the candidate holding positions K1 and K2 at positions and orientations O1 and O2, showing the state in which another workpiece W2 is covering the top of candidate holding position K1.

[0111] Figure 20(b) shows the vertical height detected in the state of Figure 20(a) as point cloud information. Figures 20(c) and 20(d) show the placement of a virtual robot hand 102. In the state of Figure 20(a), if, for example, an attempt is made to pick up workpiece W1 using a candidate holding position K1 with position and orientation O1, the robot hand 102 may interfere with workpiece W2, potentially resulting in failure to pick up workpiece W1. Also, because the orientation of workpiece W2 is tilted, the robot hand 102 may not be able to properly hold workpiece W2, potentially resulting in failure to hold workpiece W2 as well.

[0112] In the state shown in Figure 20(a), the height information from step S704 can be obtained as a virtual model with point cloud information of position and height, as shown in Figure 20(b). In this case, since the position and orientation O1 and the point cloud information are known, by further adding shape information of the robot hand 102, the virtual robot hand 102 can be virtually placed at the holding position candidate K1, which is at position and orientation O1, as shown in Figure 20(c). Here, the shape information of the virtual robot hand 102 refers to the size of the robot hand 102 and the position information of the contact surface where the robot hand 102 and the workpiece come into contact, such as the CAD (Computer Aided Design) information of the robot hand 102.

[0113] Based on the above, it is possible to determine whether the point cloud of the virtual robot hand 102 and the workpiece W2 interfere with each other. Similarly, as shown in Figure 20(d), when the virtual robot hand 102 is placed at the candidate holding position K2 at position and orientation O2, it is possible to determine whether the point cloud of the robot hand 102 and the workpiece W2 interfere with each other using the same process. Therefore, in the examples of Figures 20(a) to 20(d), it can be determined that the candidate holding position K1 at position and orientation O1 will interfere with the robot hand 102 and workpiece W2 other than the workpiece W1 to be picked up. Also, in the examples of Figures 20(a) to 20(d), it can be determined that the candidate holding position K2 at position and orientation O2 will allow the robot hand 102 to pick up workpiece W1 without interference between the robot hand 102 and workpiece W2 other than the workpiece W1 to be picked up. Then, in step S710, the CPU 251 selects a suitable holding position from among the candidate holding positions in which the robot hand 102 does not interfere with workpiece W2 other than the workpiece W1 to be picked up. Then, in steps S711 and S712, the robot 100 takes out the workpiece W1, and in step S713, it transports the workpiece W1.

[0114] As described above, according to the seventh embodiment, by adding the interference determination process in step S709, it is possible to determine whether or not the robot hand 102 will interfere with another workpiece before approaching the workpiece to be picked up. In the next step S710, any candidate holding position that is determined to interfere is not selected. In this way, interference between the robot hand 102 and the workpiece W2 can be avoided, and the workpiece W1, which is in an easy-to-pick-up position, can be reliably held, thereby improving the success rate of workpiece pick-up. Note that the seventh embodiment and any of its modifications may be combined with any of the various embodiments and their modifications described above.

[0115] [Differentiation] In the seventh embodiment described above, the case in which interference determination in step S709 is performed using height information obtained in step S704 was explained. However, it is also possible to perform interference determination using the workpiece contour information described in the fifth embodiment described above. This will be described in detail below.

[0116] Figure 21 shows the workpiece W1 detected in the pattern matching process of step S708, along with its reference position and orientation O0, and candidate holding positions K1 and K2, which correspond to positions and orientations O1 and O2. It also shows the projection of these positions and orientations O1 and O2 onto image I1, which represents the contour region of the workpiece obtained in the workpiece detection in step S702. Region RW1 represents the contour region of workpiece W1, and region RW2 represents the contour region of workpiece W2. For illustrative purposes, workpiece W2 is shown transparently in Figure 21. Interference detection can be performed using such a projected image. For example, since candidate holding position K1 at position and orientation O1 of workpiece W1 is located within region RW2, which is the contour of workpiece W2, the CPU 251 can determine that if the robot hand 102 approaches candidate holding position K1, the robot hand 102 will interfere with workpiece W2. On the other hand, since the candidate holding position K2 of the workpiece W1, which is in position and orientation O2, is outside the region RW2, the CPU 251 can determine that if the robot hand 102 approaches the candidate holding position K2, the robot hand 102 can pick up the workpiece W1 without interfering with the workpiece W2.

[0117] As described above, according to this modified version, by adding the interference determination process in step S709, it is possible to determine whether or not the robot hand 102 will interfere with another workpiece before approaching the workpiece to be picked up. In the next step S710, any candidate holding position that is determined to interfere is not selected. In this way, interference between the robot hand 102 and the workpiece W2 can be avoided, and the workpiece W1, which is in an easy-to-pick-up position, can be reliably held, thereby improving the success rate of workpiece pick-up. Note that this modified version may be implemented in combination with any of the various embodiments and their modifications described above.

[0118] The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention. Furthermore, the effects described in the embodiments are merely a list of the most preferred effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0119] In the embodiments described above, the case in which the robot arm 101 is a vertically articulated robot arm was explained, but the invention is not limited to this. The robot arm may be various types of robot arms, such as a horizontally articulated robot arm, a parallel link robot arm, or a Cartesian robot. Furthermore, the mechanism for holding the workpiece may be realized by a machine that can automatically perform movements such as extension and retraction, bending and straightening, vertical movement, horizontal movement or rotation, or combinations thereof, based on information stored in a memory device provided in the control device.

[0120] Furthermore, although the above-described embodiment described a case in which the image processing device 200 and the robot controller 300 are configured as separate computers, the invention is not limited to this. The image processing device 200 and the robot controller 300 may be configured as a single computer. In this case, the computer's CPU can be configured to function as an image processing unit and a control unit by executing a program.

[0121] Furthermore, although the case in which the image processing device 200 has one CPU 251 has been described, it is not limited to this, and the image processing device 200 may have multiple CPUs or multiple computers. In this case, the functions of the image processing unit may be shared among the multiple CPUs or multiple computers.

[0122] Furthermore, in the above-described embodiment, the imaging system 400, which is an imaging device, was described as having a first imaging unit, camera 401, and a second imaging unit, camera 402, but it is not limited to this. For example, if pattern matching is possible within the search area in image I1 obtained by the imaging operation of camera 401, camera 402, i.e., image I2, may be omitted. Also, if image I1 and image I2 can be captured by camera 402 through the operation of the robot arm 101, camera 401 may be omitted. In addition, in the above-described embodiment, the position and orientation of the workpiece were measured, but it is also possible to measure only the position of the workpiece or only the orientation of the workpiece.

[0123] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0124] 10…Robot system, 100…Robot, 200…Image processing device, 251…CPU (Image processing unit), 351…CPU (Control unit), 400…Imaging system (Imaging device)

Claims

1. A robot system including a robot controlled based on information acquired from an imaging device and a control unit, The control unit: Identifying at least one region in which a predetermined object having a predetermined posture is present in an image in which a plurality of objects are captured by the imaging device; controlling the robot to hold the predetermined object based on information on the position and / or posture of the predetermined object in the area; A robot system comprising:

2. The control unit acquires information on the position and / or posture of the predetermined object based on a model indicating the shape of the predetermined object.

2. The robot system according to claim 1 .

3. The control unit identifies the area using a trained model that has undergone machine learning.

3. The robot system according to claim 1 or 2.

4. The predetermined posture is a first posture in which a first surface of the predetermined object is imaged, or a second posture in which a second surface of the predetermined object different from the first surface is imaged.

4. The robot system according to claim 1, wherein the first and second electrodes are arranged in a first direction.

5. The control unit identifies the area in which the specified object in the first posture exists or the area in which the specified object in the second posture exists.

5. The robot system according to claim 4.

6. The first surface is a front surface of the specified object, and the second surface is a back surface of the specified object.

6. The robot system according to claim 4 or 5.

7. The trained model is obtained using an image of a first surface of an object and / or an image of a second surface of the object different from the first surface.

4. The robot system according to claim 3.

8. The control unit displays the area on a display device.

8. The robot system according to claim 1, wherein the first and second movable members are arranged in a first and second directions.

9. The control unit displays information regarding the state of the specified object on a display device.

9. The robot system according to claim 1, wherein the first and second electrodes are arranged in a first direction.

10. The control unit identifies two or more of the regions and obtains a priority in holding for the predetermined objects corresponding to the two or more regions.

10. The robot system according to claim 1, wherein the first and second components are arranged in a first direction.

11. The control unit displays the priority on a display device. The robot system according to claim 10 .

12. The control unit obtains the priority for the predetermined object corresponding to the two or more regions using a plurality of factors.

12. The robot system according to claim 10 or 11.

13. One of the factors is a success rate of past attempts to have the robot hold an object. The robot system according to claim 12 .

14. The plurality of factors includes at least one of an exposure degree of the predetermined object, a scattering degree of the predetermined object, and a height of the predetermined object in a bulk state.

14. The robot system according to claim 12 or 13.

15. The control unit performs image processing to obtain information on the position and / or posture of the predetermined object in the region.

15. The robot system according to claim 1 .

16. The control unit acquires information on the position and / or posture of the predetermined object by performing pattern matching as the image processing. The robot system according to claim 15 .

17. The control unit performs the pattern matching in the region and does not perform the pattern matching outside the region.

17. The robot system according to claim 16.

18. The imaging device includes a first imaging unit capable of imaging the plurality of objects and a second imaging unit capable of imaging a narrower area than the first imaging unit, The first imaging unit captures images of the plurality of objects to identify the region, and the second imaging unit captures images of the objects in the region to obtain information on the position and / or posture of the objects.

18. The robot system according to claim 1 .

19. Acquiring information on the position and / or posture of the specified object by performing pattern matching on an image of the specified object captured by the second imaging unit.

20. The robot system of claim 18.

20. The second imaging unit is provided on the robot.

20. The robot system according to claim 18 or 19.

21. The control unit: a first imaging process for imaging the plurality of objects and identifying the region in which the predetermined object having the predetermined posture is present; A second imaging process is executed to obtain information regarding the position and / or the posture of the predetermined object present in the area by imaging with higher resolution than that of the first imaging process.

18. The robot system according to claim 1 .

22. The control unit: a first imaging process for imaging the plurality of objects and identifying the region in which the predetermined object having the predetermined posture is present; A second imaging process is executed to capture an image of an imaging area smaller than the imaging area captured in the first imaging process, thereby acquiring information regarding the position and / or orientation of the predetermined object present in the imaging area.

18. The robot system according to claim 1 .

23. The control unit identifies the region based on a contour of the predetermined object.

23. The robot system according to claim 1 .

24. a plurality of holding positions for holding the predetermined object by the robot are set on the predetermined object; The control unit acquires a priority order of the plurality of holding positions based on a state of the predetermined object.

24. The robotic system of claim 23.

25. The control unit obtains a priority order for the plurality of holding positions based on at least one of the height of each of the plurality of holding positions for the specified object in a bulk stacked state, the center of gravity of the specified object, and the time required for the robot to approach each of the plurality of holding positions for the specified object.

25. The robotic system of claim 24.

26. a plurality of holding positions for holding the predetermined object by the robot are set on the predetermined object; the control unit identifies, among the plurality of holding positions, a holding position at which the robot interferes with an object other than the predetermined object when holding the predetermined object.

24. The robot system according to claim 1 .

27. ​​The control unit determines whether the robot will interfere with an object other than the predetermined object based on a height of the predetermined object and a height of the object other than the predetermined object, and / or a contour of the predetermined object and a contour of the object other than the predetermined object.

27. The robotic system of claim 26.

28. A method for controlling a robot system having a robot controlled based on information acquired from an imaging device, comprising: Identifying at least one region in which a predetermined object having a predetermined posture is present in an image in which a plurality of objects are captured by the imaging device; controlling the robot to hold the object based on information on the position and / or posture of the predetermined object in the area; A control method comprising:

29. Identifying at least one region in which a predetermined object having a predetermined posture is present in an image in which a plurality of objects are captured; executing a process of acquiring information on the position and / or orientation of the predetermined object in the region; 13. An image processing device comprising:

30. Identifying at least one region in which a predetermined object having a predetermined posture exists in an image in which a plurality of objects are captured, executing a process of acquiring information on the position and / or orientation of the predetermined object in the region; 13. An image processing method comprising:

31. A method for manufacturing an article, comprising the steps of: manufacturing an article by using the robot system according to any one of claims 1 to 27;

32. A program for causing a computer to execute the control method according to claim 28 or the image processing method according to claim 30.

33. A computer-readable recording medium having the program according to claim 32 recorded thereon.

Images