Robotic system, control method, image processing device and method, and product manufacturing method

Abstract

A robot system, a control method, an image processing apparatus and method, and a product manufacturing method are provided. A robot system includes a robot, an imaging device, an image processing section, and a control section. The image processing section is configured to specify at least one region in which a predetermined object having a predetermined posture is present among images of a plurality of objects captured by the imaging device, and obtain information about a position and / or a posture of the predetermined object in the region. The control section is configured to control the robot based on the information about the position and / or the posture of the predetermined object so that the robot holds the predetermined object.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to image processing. BACKGROUND

[0002] For example, in a factory, a work of placing workpieces at predetermined positions, and an assembly work of assembling a product by fitting or inserting a workpiece into another workpiece are performed. In these types of work, industrial robots are used to automate the factory. These types of work include a picking work of picking up workpieces one by one from a plurality of workpieces that are stacked in bulk.

[0003] Japanese Patent Application Publication No. 2000-293695 describes a technique in which an image of a plurality of workpieces that are stacked in bulk is captured by a camera, and image processing such as pattern matching is performed. In pattern matching, the captured image is compared with a teaching model that is obtained in advance. SUMMARY

[0004] According to a first aspect of the present application, a robot system includes a robot, an imaging device, an image processing section, and a control section. The image processing section is configured to specify at least one region in which a predetermined object having a predetermined posture is present in an image of a plurality of objects captured by the imaging device, and to obtain information about a position and / or a posture of the predetermined object in the region. The control section is configured to control the robot based on the information about the position and / or the posture of the predetermined object, so that the robot holds the predetermined object.

[0005] According to a second aspect of the present application, a control method of a robot system including a robot and an image capturing device includes specifying, by an image processing section, at least one region in which a predetermined object having a predetermined posture is present in an image of a plurality of objects captured by an imaging device, and obtaining, by the image processing section, information about a position and / or a posture of the predetermined object in the region, and controlling, by a control section, the robot based on the information about the position and / or the posture of the predetermined object, so that the robot holds the predetermined object.

[0006] According to a third aspect of the present application, an image processing apparatus includes an image processing section configured to specify at least one region in which a predetermined object having a predetermined posture is present in an image of a plurality of objects captured, and to obtain information about a position and / or a posture of the predetermined object in the region.

[0007] According to a fourth aspect of the present application, an image processing method includes specifying, by an image processing section, at least one region in which a predetermined object having a predetermined posture is present in an image of a plurality of objects captured, and obtaining, by the image processing section, information about a position and / or a posture of the predetermined object in the region.

[0008] Further features of the present application will become apparent from the following description of example embodiments with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 is a diagram showing a schematic configuration of a robot system of the first embodiment.

[0010] Figure 2 is a diagram showing an image processing apparatus of the first embodiment.

[0011] Figure 3 is a block diagram of a computer system of the robot system of the first embodiment.

[0012] Figure 4A is a block diagram showing a function of a CPU of the first embodiment.

[0013] Figure 4B is a diagram for showing a state of a workpiece.

[0014] Figure 5 is a flowchart showing a control method of a robot of the first embodiment.

[0015] Figure 6A is a diagram showing one example of an image of the first embodiment.

[0016] Figure 6B is a diagram for showing a detection process of the first embodiment.

[0017] Figure 7A is a diagram showing a marking work of the first embodiment.

[0018] Figure 7B is a diagram showing a marking work of the first embodiment.

[0019] Figure 8A is a diagram showing one example of a recognition result of the first embodiment.

[0020] Figure 8B is a diagram for showing a method of calculating an exposure degree of a workpiece of the first embodiment.

[0021] Figure 9A is a diagram showing one example of a height of a workpiece corresponding to each candidate region of the first embodiment.

[0022] Figure 9B is a table showing one example of a priority of the first embodiment.

[0023] Figure 10A is a diagram of an image obtained from a camera of the first embodiment.

[0024] Figure 10Bis a schematic diagram for illustrating a pattern matching process of a comparative example.

[0025] Figure 10C is a schematic diagram for illustrating a pattern matching process of the first embodiment.

[0026] Figure 11 is a diagram showing one example of display by the display of the first embodiment.

[0027] Figure 12 is a flowchart showing a control method of the robot of the second embodiment.

[0028] Figure 13 is a block diagram showing a function of the CPU of the third embodiment.

[0029] Figure 14A is a schematic diagram for illustrating a pruning process of the third embodiment.

[0030] Figure 14B is a schematic diagram for illustrating a learning model of the third embodiment.

[0031] Figure 15 is a flowchart showing a control method of the robot of the fourth embodiment.

[0032] Figure 16A is a schematic diagram showing one example of an image of the fifth embodiment.

[0033] Figure 16B is a schematic diagram for illustrating a detection process of the fifth embodiment.

[0034] Figure 17 is a flowchart showing a control method of the robot of the sixth embodiment.

[0035] Figure 18 is a diagram showing a plurality of holding position candidates of the sixth embodiment.

[0036] Figure 19 is a flowchart showing a control method of the robot of the seventh embodiment.

[0037] Figure 20A is a schematic diagram showing a state in which two workpieces overlap with each other in the seventh embodiment.

[0038] Figure 20B is a schematic diagram showing a state in which two workpieces overlap with each other in the seventh embodiment.

[0039] Figure 20C is a schematic diagram showing a state in which two workpieces overlap with each other in the seventh embodiment.

[0040] Figure 20Dis a diagram showing a state in which two workpieces overlap each other in the seventh embodiment.

[0041] Figure 21 is a diagram for showing interference determination of a modification example made for interference between a robot hand and a workpiece. DETAILED DESCRIPTION

[0042] In the conventional method, image processing for identifying a workpiece takes time. For this reason, it is desired to shorten the time required for image processing made for identifying a workpiece, to improve product productivity in a production line.

[0043] An object of the present disclosure is to shorten the time required for image processing made for identifying a workpiece.

[0044] Hereinafter, some embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0045] First Embodiment

[0046] Figure 1 is a diagram showing a schematic configuration of a robot system 10 of the first embodiment. The robot system 10 includes a robot 100, an image processing device 200, a robot controller 300 as one example of a control device, and an imaging system 400 as one example of an imaging device. The robot 100 is an industrial robot, disposed in a production line, and used for manufacturing a product.

[0047] The robot 100 is a manipulator. For example, the robot 100 is fixed to a base. A container 30 having an upper opening and a stage 40 are disposed around the robot 100. In the container 30, a plurality of workpieces W are bulk-stacked. Each workpiece W is one example of an object; and is, for example, a part. The plurality of workpieces W in the container 30 are held one by one by the robot 100, and transported to a predetermined position on the stage 40. The plurality of workpieces W have the same shape, the same size, and the same color; and are arranged at random in the container 30. Each workpiece W is a plate-like member, having a front surface and a back surface of mutually different shapes.

[0048] The robot 100 and the robot controller 300 are communicatively connected to each other. The robot controller 300 and the image processing device 200 are communicatively connected to each other. The imaging system 400 and the image processing device 200 are communicatively connected to each other by wire or wirelessly.

[0049] The robot 100 includes a robot arm 101 and a robot hand 102 that is one example of an end effector or holding mechanism. The robot arm 101 is a vertical multi-joint robot arm. The robot hand 102 is supported by the robot arm 101. The robot hand 102 is attached at a predetermined portion of the robot arm 101, for example, a distal end portion of the robot arm 101. The robot hand 102 is capable of holding a workpiece W. Note that although a case where the holding mechanism is the robot hand 102 will be described, the present disclosure is not limited thereto. For example, the holding mechanism can be a suction mechanism that holds the workpiece W by suctioning the workpiece W. In the first embodiment, the robot hand 102 can hold the workpiece W.

[0050] In the above-described configuration, the robot hand 102 is moved to a predetermined position by the robot arm 101 so that the robot 100 can perform a desired work. For example, a workpiece W and another workpiece are prepared as materials, and the workpiece W is assembled to the other workpiece by the robot 100 to manufacture an assembled workpiece as a product. In this way, a product can be manufactured by the robot 100. Note that although a case where a product is manufactured by the robot 100 assembling one workpiece to another workpiece has been described as an example in the present embodiment, the present disclosure is not limited thereto. For example, a product can be manufactured by attaching a tool such as a cutting tool or a grinding tool to the robot arm 101 and processing a workpiece by the tool.

[0051] The imaging system 400 includes a camera 401 that is one example of a first imaging unit and a camera 402 that is one example of a second imaging unit. Each of the cameras 401 and 402 is a digital camera. The camera 401 is fixed to a frame (not shown). The camera 401 is located at a position at which the camera 401 can take an image of a region that contains a plurality of workpieces W arranged in the container 30. That is, the camera 401 can take an image of a region that contains workpieces W that are objects to be held by the robot 100.

[0052] The camera 402 is attached to a predetermined portion of the robot 100 such as the robot hand 102 so as to be supported by the robot 100. The position of the camera 402 at which the camera 402 images, that is, the imaging region of which an image is taken by the camera 402, can be freely changed depending on the posture of the robot 100. Specifically, by the movement of the robot 100, the camera 402 can be moved to a position closer to the plurality of workpieces W that are bulk-stowed in the container 30 than the position of the camera 401. Therefore, the camera 402 can take an image of a region smaller than the region of the image taken by the camera 401. In addition, by the movement of the robot 100, the camera 402 can be moved to a position above one workpiece W among the plurality of workpieces W that is an object to be held by the robot 100.

[0053] In the first embodiment, the image processing apparatus 200 is a computer. The image processing apparatus 200 transmits an imaging command to the camera 401, thereby causing the camera 401 to perform imaging. In addition, the image processing apparatus 200 transmits an imaging command to the camera 402, thereby causing the camera 402 to perform imaging. The image processing apparatus 200 obtains an image I1 that is an example of a first image captured by the camera 401, and processes the image I1. In addition, the image processing apparatus 200 obtains an image I2 that is an example of a second image captured by the camera 402, and processes the image I2. Figure 2 is a diagram illustrating the image processing apparatus 200 of the first embodiment. The image processing apparatus 200 includes a main body 201, a display 202 that is an example of a display connected to the main body 201, and a keyboard 203 and a mouse 204 that are examples of input devices connected to the main body 201.

[0054] In the first embodiment, Figure 1 The robot controller 300 illustrated is a computer. The robot controller 300 controls the motion of the robot 100, that is, the posture of the robot 100.

[0055] Figure 3 is a block diagram of a computer system of the robot system 10 of the first embodiment. The main body 201 of the image processing apparatus 200 includes a central processing unit (CPU) 251 that is an example of a processor. The CPU 251 is an example of an image processing section. The main body 201 further includes a read only memory (ROM) 252, a random access memory (RAM) 253, and a hard disk drive (HDD) 254 that function as a storage section. The main body 201 further includes a recording disk drive 255 and an interface 256 that is an input / output interface. The CPU 251, the ROM 252, the RAM 253, the HDD 254, the recording disk drive 255, and the interface 256 are communicatively connected to each other via a bus.

[0056] The ROM 252 stores a basic program related to the operation of the computer. The RAM 253 is a storage device that temporarily stores various types of data, such as a result of a calculation process performed by the CPU 251. The HDD 254 stores various types of data, such as a result of a calculation process performed by the CPU 251 and data obtained from an external device, and a program 261 that causes the CPU 251 to perform various types of processing. The program 261 is application software that can be executed by the CPU 251.

[0057] The CPU 251 performs the image processing described later by executing the program 261 stored in the HDD 254. The recording disk drive 255 reads various types of data and programs stored in a recording disk 262.

[0058] In the first embodiment, the HDD 254 is a computer-readable non-transitory recording medium, and stores the program 261. However, the present disclosure is not limited to this. The program 261 can be recorded in any recording medium as long as the recording medium is a computer-readable non-transitory recording medium. For example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, a non-volatile memory, or the like can be used as the recording medium that provides the program 261 to the computer.

[0059] The robot controller 300 includes a CPU 351 as one example of a processor. The CPU 351 is one example of a control portion. The robot controller 300 further includes a ROM 352, a RAM 353, and a HDD 354 that function as storage portions. The robot controller 300 further includes a recording disk drive 355 and an interface 356 as an input / output interface. The CPU 351, the ROM 352, the RAM 353, the HDD 354, the recording disk drive 355, and the interface 356 are communicatively connected to each other via a bus.

[0060] The ROM 352 stores a basic program related to the operation of the computer. The RAM 353 is a storage device that temporarily stores various types of data, such as a result of a calculation process by the CPU 351. The HDD 354 stores various types of data, such as a result of a calculation process by the CPU 351 and data obtained from an external device, and a program 361 (i.e., the program 361 is recorded in the HDD 354) that causes the CPU 351 to execute various types of processing. The program 361 is application software that can be executed by the CPU 351.

[0061] The CPU 351 executes control processing by executing the program 361 stored in the HDD 354, thereby controlling Figure 1 the movement of the robot 100 illustrated. The recording disk drive 355 reads various types of data and programs stored in the recording disk 362.

[0062] In the first embodiment, the HDD 354 is a computer-readable non-transitory recording medium, and stores the program 361. However, the present disclosure is not limited to this. The program 361 can be recorded in any recording medium as long as the recording medium is a computer-readable non-transitory recording medium. For example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, a non-volatile memory, or the like can be used as the recording medium that provides the program 361 to the computer.

[0063] Note that although the image processing and the control processing are executed by a plurality of computers (i.e., the CPU 251 and the CPU 351) in the first embodiment, the present disclosure is not limited to this. For example, the image processing and the control processing can be executed by a single computer (i.e., a single CPU). In this case, the single CPU can function as the image processing portion and the control portion.

[0064] The CPU 251 executes the program 261, thereby causing the camera 401 to capture an image of a region in which a plurality of workpieces W exist, and detects a workpiece W that can be picked up by using the image I1 captured by the camera 401. That is, the CPU 251 specifies a region of the image I1 that contains a workpiece image corresponding to the workpiece W that can be picked up. Hereinafter, this region is referred to as a search region. In addition, the CPU 251 executes the program 261, thereby causing the camera 402 to capture an image of a region in real space corresponding to the search region. Then, the CPU 251 measures the position and the posture of the workpiece W by performing a pattern matching process as one example of image processing.

[0065] The CPU 351 executes the program 361, thereby moving the camera 402 to a position at which the camera 402 can capture an image of a region in real space corresponding to the search region by controlling the robot 100. In addition, the CPU 351 executes the program 361, thereby moving the robot hand 102 to the position of the workpiece W that has been measured by the CPU 251 and at which the robot hand 102 is to hold the workpiece W. In addition, the CPU 351 executes the program 361, thereby causing the robot hand 102 to hold the workpiece W and move the workpiece W to the stage 40.

[0066] Figure 4A is a block diagram illustrating the functions of the CPU 251 of the first embodiment. The CPU 251 functions as the workpiece detection portion 211, the recognition portion 212, the height detection portion 213, the priority determination portion 214, and the measurement portion 215 by executing the program 261. Next, an operation outline of the cameras 401 and 402 and the portions 211 to 215 will be described.

[0067] The camera 401 captures an image of a region in which a plurality of workpieces W exist, and outputs the image as an RGB grayscale image I1. Note that although the first imaging unit is the camera 401 in the first embodiment, the present disclosure is not limited to this. The first imaging unit can be any unit as long as the unit can digitally convert a feature of a workpiece W into a numerical value.

[0068] The workpiece detection section 211 detects at least one candidate region from the image I1 obtained from the camera 401, each of the candidate regions containing an image of a single workpiece W (i.e., a workpiece image). The recognition section 212 recognizes the state of the workpiece W corresponding to the candidate region detected by the workpiece detection section 211. The state of the workpiece W refers to the posture information of the workpiece W. Figure 4B is a schematic diagram for showing the state of the workpiece W. As shown in Figure 4B , the state of the workpiece W, i.e., the posture information of the workpiece W refers to information indicating which one of the front surface Fl and the back surface F2 of the workpiece W is facing upward when viewed from the camera 401. The front surface Fl is one example of the first surface, and the back surface F2 is one example of the second surface different from the first surface. The shape of the back surface F2 is different from the shape of the front surface Fl.

[0069] In a case where the at least one candidate region is two or more candidate regions, the height detection section 213 detects the height of the workpiece W corresponding to the workpiece image contained in the candidate region. The height of the workpiece W is the height in the vertical direction with respect to a reference position. For example, the height detection section 213 uses a sensor (not shown) that outputs a signal corresponding to the height of the workpiece W; and detects the height of the workpiece W from the signal from the sensor. The sensor (not shown) can be a ToF (Time of Flight) height sensor or a depth sensor that outputs a distance image. In another case, the height detection section 213 can detect the height of the workpiece W by using a three-dimensional camera (not shown) that outputs an RGB image and a 3D point group. In this case, the three-dimensional camera can be integrated with the camera 401. The priority determination section 214 assigns priorities to the plurality of candidate regions detected by the workpiece detection section 211 in order of easiness of picking up the workpiece by the robot 100; and extracts the candidate region having the highest priority. Thus, the candidate region having the highest priority is the above-mentioned search region. In addition, the workpiece W corresponding to the workpiece image contained in the search region is the object to be held by the robot 100. The object to be held by the robot 100 is one example of the predetermined object.

[0070] The camera 402 captures an image of a region containing the workpiece W corresponding to the workpiece image contained in the search region of the image I1, and smaller than the imaging region of the camera 401. For example, the image I2 captured by the camera 402 is an RGB grayscale image. In addition to the image I2, the camera 402 outputs a distance image containing height information, if necessary.

[0071] The measurement section 215 performs a pattern matching process on the image I2 obtained from the camera 402, thereby obtaining three-dimensional information on the position and posture of the workpiece W. Specifically, the measurement section 215 performs the pattern matching process based on the posture information of the workpiece W determined by the recognition section 212. Thus, the amount of calculation can be reduced.

[0072] Hereinafter, the control method of the robot 100 including the image processing method of the first embodiment will be specifically described. Figure 5 is a flowchart showing the control method of the robot 100 of the first embodiment.

[0073] The workpiece detection section 211 transmits a photographing command to the camera 401, thereby causing the camera 401 to capture an image of a region containing a plurality of workpieces W (S101). In this operation, the camera 401 captures an image of a plurality of workpieces W that are bulk-stacked in the container 30. Among the plurality of workpieces W, there is included an object to be held by the robot 100. The workpiece detection section 211 obtains an image II from the camera 401, the image II containing images of the plurality of workpieces W.

[0074] Figure 6A is a schematic view showing one example of the image II of the first embodiment. As Figure 6A shown, the image II contains a plurality of workpiece images WI that correspond to the plurality of workpieces W bulk-stacked, as a gray-scale image. Note that the image II can contain an image of an object other than the workpieces W (for example, the container 30 that houses the workpieces W). In Figure 6A the example, the image II contains an image 30I that corresponds to the container 30.

[0075] Then, the workpiece detection section 211 performs a detection process of detecting the workpieces W from the image II (S102). Figure 6B is a schematic view for illustrating the detection process of the first embodiment. As Figure 6B shown, if the detection process of the workpieces W is successful, two or more regions are obtained in the image II. In Figure 6B , for example, four candidate regions A, B, C, and D enclosed by dotted lines are obtained in the image II. Each of the candidate regions A, B, C, and D is a rectangular region that encloses a single workpiece image WI. Each of the candidate regions A, B, C, and D contains a single workpiece image WI and is associated with posture information of a workpiece W that corresponds to the workpiece image WI. Thus, in step S102, the workpiece detection section 211 detects two or more candidate regions A to D, thereby extracting two or more workpiece images WI from the image II.

[0076] Note that the image II contains images of the plurality of workpieces W bulk-stacked and has a large photographing region. Therefore, it takes time to extract the workpieces W that are objects to be held by the robot 100 by using a pattern matching process. For this reason, in the first embodiment, the candidate regions A to D are detected by using a process other than the pattern matching process. Specifically, an image recognition method called object detection is used to detect the candidate regions A to D. In the first embodiment, a learning-based image recognition method using deep learning will be described as an example.

[0077] In the object detection, the workpiece detection section 211 uses the learning model 263; and searches for the workpiece images WI corresponding to the workpieces W in the image I1. Then, the workpiece detection section 211 outputs the rectangular candidate regions A to D, each of which encloses the corresponding workpiece image WI. Thus, each of the candidate regions A to D contains the workpiece image WI corresponding to the workpiece W having the predetermined posture. That is, the workpiece detection section 211 specifies the candidate regions A to D in the image I1, in each of which the workpiece image WI corresponding to the workpiece W having the predetermined posture is formed. In the first embodiment, the predetermined posture is the posture P10 or P20. As shown, the posture P10 is a first posture in which the image of the front face Fl of the workpiece W is captured, and the posture P20 is a second posture in which the image of the back face F2 of the workpiece W is captured. Thus, each of the candidate regions A to D contains the workpiece image WI corresponding to the workpiece W having the posture P10 or P20. Figure 4B

[0078] In order to search for the workpiece W by using deep learning, it is necessary to teach the robot the features of the image of the workpiece W captured by the camera 401. In the teaching, a plurality of learning data sets are prepared, each of which includes input data and output data. For the input data, the original gray-scale image is used; and for the output data, the gray-scale image and the label data are used. Each piece of label data is data that provides the corresponding label information to the gray-scale image. The labeling work is performed by an operator. Figure 7A is a diagram showing the labeling work of the first embodiment. In the teaching, a number of gray-scale images I0 are prepared. Figure 7A One of the gray-scale images I0 is shown. Note that the image I0 of the first embodiment is obtained by capturing an image of an object having a shape corresponding to the workpiece W to be held by the robot hand 102. Hereinafter, the object used for the teaching is also referred to as the workpiece W, and the image contained in the image I0 and corresponding to the object is also referred to as the workpiece image WI.

[0079] The operator specifies a rectangular region R0 that encloses the workpiece image WI included in the image I0, and associates the region R0 with information indicating the state of the corresponding workpiece W. The region R0 is a part of the image I0. The region R0 is specified using a start point coordinate PI and an end point coordinate P2. That is, a rectangular region having a diagonal specified by the start point coordinate PI and the end point coordinate P2 is specified as the region R0. The information indicating the state of the workpiece W is posture information of the workpiece W having a defined range. For example, the information indicating the state of the workpiece W is information indicating the front face Fl or the back face F2 of the workpiece W. The information indicating the state of the workpiece W is provided to the region R0, with which the region R0 is associated.

[0080] Figure 7B ​is a diagram showing the marking work of the first embodiment. First, the case where the front surface Fl of the workpiece W is upward will be described. Figure 7B The case where the front surface Fl of the workpiece W is upward will be described. In the case where the front surface Fl of the workpiece W is upward, an image of the front surface Fl of the workpiece W is captured. If the workpiece W takes a posture in which the axis Cl perpendicular to the front surface Fl is within a solid angle al defined with respect to the axis Co perpendicular to a predetermined plane such as a horizontal plane, posture information Tl indicating that the front surface Fl of the workpiece W is upward is provided. In the case where the back surface F2 of the workpiece W is upward, an image of the back surface F2 of the workpiece W is captured. If the back surface F2 is upward, posture information indicating that the back surface F2 of the workpiece W is upward is provided. The CPU 251 obtains the learning model 263 by using a plurality of images I0 including the captured image of the front surface Fl of the workpiece W and the captured image of the back surface F2 of the workpiece W. Note that if different surfaces of a workpiece have the same appearance and the same shape, the same posture information can be provided to the surfaces even if the surfaces are different surfaces. For example, if the front surface Fl and the back surface F2 have the same appearance and the same shape, the posture information Tl can be provided to both the front surface Fl and the back surface F2. In contrast, if different surfaces have different appearances even if they have the same shape, different posture information can be provided to the surfaces. For example, the surfaces of a die have different numbers. Therefore, different posture information can be provided to the surfaces of the die so that the numbers of the die surfaces can be recognized. In this way, the operator specifies the region R0 and the posture information with respect to the original gray-scale image I0 obtained from the camera 401, so that a plurality of learning data sets are registered in the image processing apparatus 200. The CPU 251 of the image processing apparatus 200 performs machine learning by using a predetermined learning algorithm and the plurality of learning data sets, so that the learning model 263 that has been subjected to machine learning is obtained. The learning model 263 is stored in the HDD 254, for example.

[0081] The learning algorithm used can be an SSD (Single Shot Multi Box Detector), a YOLO (You Only Look Once), or the like. Note that the learning algorithm is not limited to the above-described algorithm and can be any algorithm as long as the algorithm can output the candidate regions A to D and the information indicating the state of the workpiece W. In addition, in order to prepare the above-described learning data sets, actually captured images can be used as described above. However, in another case, images created in a virtual space such as in a physical simulator can be used.

[0082] In the first embodiment, the CPU 251 obtains the learning model 263 for detecting the candidate region by using not only the contour of the workpiece (i.e., the edge information) of the grayscale image I0 but also the shadow feature of the workpiece image WI corresponding to the workpiece W. The contour and the shadow feature of the workpiece image WI are obtained by causing the neural network to learn many patterns. Thus, even if the plurality of workpieces W are differently bulk-stacked, the CPU 251 can identify a certain workpiece W from among the bulk-stacked plurality of workpieces W. That is, the CPU 251 can extract the candidate region.

[0083] The recognition part 212 identifies the state of the workpiece W with respect to each of the plurality of candidate regions A to D that have been detected by the workpiece detection part 211 using the learning model 263 (S103). Step S103 can be performed together with step S102, which detects the candidate regions A to D by using an algorithm such as SSD or YOLO. In this way, the recognition part 212 determines the posture of the workpiece W corresponding to each of the candidate regions A to D as the state of the workpiece W by using the preset learning model 263.

[0084] Figure 8A is a schematic diagram showing one example of the recognition result of the first embodiment. In Figure 8A In the first embodiment, the posture information T1 indicating the posture in which the front surface F1 of the workpiece W faces upward is provided to all of the plurality of candidate regions A to D as an example. Note that the accuracy of the posture information of the workpiece W determined in step S103 is not sufficient for the robot 100 to hold the workpiece W. For this reason, in the first embodiment, the posture of the workpiece W is determined with high accuracy in the pattern matching process of step S108 described later.

[0085] Then, the height detecting section 213 detects the height of the workpieces W in the vertical direction, which corresponds to the workpiece images WI contained in each of the plurality of candidate regions A to D (S104). Specifically, the height detecting section 213 detects the maximum height of the portion of the workpiece W corresponding to each of the candidate regions A to D. By this operation, the height difference between the workpieces W corresponding to the plurality of candidate regions A to D can be compared with each other. The workpiece W located at a higher position can be more easily picked up from the bulk-stacked plurality of workpieces W than the workpiece W located at a lower position. This is because the workpiece W located at a lower position is more likely to serve as a support point that supports other workpieces W located at a higher position in probability. Therefore, if the workpiece W located at a lower position is picked up, other workpieces W easily collapse, or another workpiece W easily moves with the workpiece W to be picked up. Thus, the height detecting section 213 detects the height of the workpiece W corresponding to each of the candidate regions A to D to assign the priority of the workpiece W using the height of the workpiece W in the next step S105.

[0086] Then, the priority determining section 214 selects the workpiece W as an object to be held by the robot 100 (S105). Specifically, the priority determining section 214 assigns the priority to the candidate regions A to D according to a plurality of factors. Each factor indicates the easiness of the robot 100 to hold the workpiece W, i.e., the success rate of picking up. For example, the factors are three factors 1 to 3 described below.

[0087] Factor 1: Exposure degree x1 of workpiece W

[0088] The exposure degree x1 of the workpiece W is the ratio of the area of the visible portion of a certain surface (e.g., the front face Fl or the back face F2) of the workpiece to the area of the entire surface. Therefore, as the exposure degree x1 decreases, the plurality of workpieces W are more densely overlapped with each other. In contrast, when the exposure degree x1 increases, the plurality of workpieces W are more sparsely overlapped with each other. Thus, as the exposure degree x1 of the workpiece W increases, the probability of the workpiece W picking up failure decreases, and thus the workpiece W picking up success rate increases. The method of calculating the factor 1 will be described.

[0089] Figure 8B is a schematic diagram for illustrating the method of calculating the exposure degree x1 of the workpiece of the first embodiment. To calculate the exposure degree x1 of the workpiece W, the corresponding workpiece image WI of the image II is used. The exposure degree x1 is a numerical value in the range equal to or greater than 0 and equal to or less than 1.

[0090] In determining the exposure degree xl in the single workpiece image WI, if the entire surface of the workpiece W is regarded as the workpiece image WI, the exposure degree xl is 1. If half of the area of the surface of the workpiece W is regarded as the workpiece image WI, the exposure degree xl is 0.5. In this way, the exposure degree xl of the workpiece W is calculated based on the area of the exposed portion of the workpiece W seen from the camera 401. Such a calculation result can be obtained by teaching the robot 100 only the state in which the workpiece W having the exposure degree xl of 1 exists, and teaching when creating the learning model 263 by using object detection such as SSD or YOLO. That is, since object detection such as SSD or YOLO can output the similarity of the object with respect to the learning data group, the similarity decreases as the ratio of the area of the visible portion to the area of the entire surface according to the probability density function such as a softmax function. This feature of the object detection can be used to determine the exposure degree xl of the workpiece W. Note that since the method of measuring the exposure degree xl of the workpiece W is not limited to the above-described method, other methods can also be used to determine the exposure degree xl of the workpiece W.

[0091] Factor 2: Dispersion degree x2 of the workpiece W

[0092] The dispersion degree x2 of the workpiece W is the positional relationship between a plurality of candidate regions. In a plurality of workpieces W that are bulk-stacked, each workpiece W takes a position and a posture at random. Therefore, in a region in which the workpiece W exists, uneven distribution occurs. For example, a dense portion and a sparse portion of the workpiece W are formed. In the dense portion in which the workpiece W exists densely, the workpiece W is easily caught with each other in probability, and is easily collapsed due to an external force. Therefore, in a region in which the workpiece W exists sparsely, that is, in a region that is separated and isolated from the dense portion of the workpiece W, the success rate of picking up the workpiece W tends to increase. From this, it is appropriate to add the dispersion degree x2 indicating the degree of isolation of the workpiece W. For example, Figure 6B The dispersion degree x2 of the candidate region A shown is calculated by using the following equation (1).

[0093]

[0094] In formula (1), S(A) represents the area of the candidate region A. The dispersion x2 is calculated by subtracting the area of the portion shared by the candidate region A and other candidate regions B to D from the area of the candidate region A itself. Therefore, the dispersion x2 becomes closer to 0 as the workpiece W exists in a denser portion, and becomes closer to 1 as the workpiece W exists in a sparser portion. Therefore, the priority of the pickup increases as the dispersion x2 increases. The difference between the exposure x1 and the dispersion x2 will be described. The exposure x1 and the dispersion x2 differ from each other in that the exposure x1 represents a hidden portion of the workpiece W that is not detected in the object detection processing, and the dispersion x2 represents the density or sparsity of the workpiece W that has been detected in the object detection processing.

[0095] Factor 3: Height X3 of the bulk-stacked workpieces

[0096] The height X3 is the height of the workpiece W in the vertical direction with respect to the ground. In the first embodiment, the ground is the bottom surface of the container 30.

[0097] In a case where the robot 100 approaches the workpiece W as an object to be picked up from above the plurality of workpieces W that are bulk-stacked, the success rate of picking up the workpiece W increases when the workpiece W is located closer to the top of the plurality of workpieces W. If the plurality of workpieces W have the same exposure x1 and the same dispersion x2, in order to increase the success rate of the pickup, the priority is given to the workpiece W located at a higher position among the plurality of workpieces W. For example, the height x3 of the workpiece W corresponding to the candidate region A is calculated by using the following formula (2).

[0098]

[0099] In formula (2), H(A) represents the height value of the workpiece W corresponding to the candidate region A, and is a value detected by the height detection portion 213. The parameter H(A) as the height information can represent the maximum value or the average value of the height of the workpiece W corresponding to the candidate region A, and the maximum value or the average value can be appropriately selected.

[0100] Figure 9A is a graph showing one example of the height x3 of the workpiece corresponding to the candidate regions A to D of the first embodiment. As shown in Figure 9A the height x3 is the ratio of the height of the workpiece W of each of the candidate regions A to D to the maximum height of all the workpieces W corresponding to the candidate regions A to D. That is, as the height of the workpiece W decreases with respect to the maximum height, the height x3 becomes closer to 0, and as the height of the workpiece W becomes closer to the maximum height, the height x3 becomes closer to 1. Therefore, the priority of the pickup increases as the height x3 increases.

[0101] Then, the priority determination section 214 calculates an index f for determining the priority by using the following equation (3) i Equation (3) includes the three factors that have been described above as the plurality of factors. Specifically, as shown in Equation (3), the priority determination section 214 calculates the index f by multiplying each of the factors 1 to 3 by a corresponding weight coefficient and adding the factors i . The index f i indicates the easiness of picking up the workpiece in each candidate region.

[0102] f i = αx 1i + βx 2i + γx 3i (3)

[0103] Note that α + β + γ = 1.

[0104] The subscript i indicates a number assigned to each of the plurality of candidate regions. For example, if four candidate regions A to D are detected, the subscript i has a value of one of 1 to 4. The index f i is an evaluation value indicating the easiness of picking up the workpiece in each candidate region. In addition, the weight coefficients α, β, and γ are coefficients whose sum is 1. These weight coefficients can be freely set depending on which factor is valued. As the index f i corresponding to the workpiece W increases, the robot 100 can more easily hold the workpiece W.

[0105] The priority determination section 214 calculates the index f i for each of the candidate regions A to D. Then, the priority determination section 214 determines the priority by comparing one index f i corresponding to each of the candidate regions A to D with the other indexes. That is, the priority determination section 214 calculates the index f i by using the plurality of factors 1 to 3; and determines the priority of the workpiece W to be picked up from the container 30 based on the index f i .

[0106] In the first embodiment, the priority determination section 214 determines the priority of the workpiece W to be picked up from the container 30 based on the index f iThe candidate region having the highest priority is determined as a search region used in the next step S106 in which the position and posture of the work W are searched. That is, the priority decision portion 214 selects the candidate region having the highest priority from among the two or more detected candidate regions, and determines the candidate region as the search region. The search region contains the work image corresponding to the object to be held by the robot 100. In this way, the priority decision portion 214 determines one of the two or more work W as the object to be held by the robot 100. More specifically, the priority decision portion 214 determines the priority of each of the two or more work W; and selects the work W having the highest priority from among the two or more work W, and determines the work W as the object to be held by the robot 100.

[0107] Figure 9B is a table showing one example of the priority of the first embodiment. For example, if a = 0.3, β = 0.4, and γ = 0.3 in Equation (3), the weight coefficient β of the dispersion x2 is given importance, and the highest priority is given to the candidate region D. Note that although the case where three factors 1 to 3 are used to calculate the index f i is described, but the present disclosure is not limited thereto. If there is another factor related to the pickup success rate, the factor can be added to Equation (3). In this way, the priority decision portion 214 calculates the priority of each of the two or more work W by using a plurality of factors 1 to 3.

[0108] Note that if the work detection portion 211 detects only one candidate region in step S102, the priority decision portion 214 determines the candidate region as the search region in step S105. Therefore, in this case, the priority calculation processing and the height detection processing for the priority calculation processing can not be performed.

[0109] Then, the CPU 351 of the robot controller 300 moves the robot 100 so that the camera 402 is moved to a position at which the position of the camera 402 is close to the workpiece W that is the object selected by the priority determination section 214 (S106). Note that the position of the camera 402 close to the workpiece W is a position above the workpiece W at which the imaging area of the camera 402 contains a real space region corresponding to the search region. When the position of the camera 402 is close to the workpiece W, it is preferable that the center of the imaging area of the camera 402 is aligned with the center of the real space region corresponding to the search region. Since the camera 402 is moved close to the workpiece W that is the object selected by the priority determination section 214, the target to be imaged can be narrowed down to one of the plurality of workpieces W in bulk storage, and the resolution of the image I2 (which can be obtained by imaging the real space region corresponding to the search region) can be improved. Thus, the processing time required for the pattern matching processing performed in the step S108 described later can be shortened.

[0110] Then, the measurement section 215 sends an imaging command to the camera 402, thereby causing the camera 402 to image the workpiece W that is the object selected by the priority determination section 214 (S107). The image I2 obtained from the camera 402 can be a gray scale image or a three-dimensional point cloud image containing depth information, as long as the image I2 has a format that can be used for the pattern matching processing in the next step S108. In the first embodiment, the image I2 is a gray scale image. Figure 10A is a schematic view of the image I2 of the object of the first embodiment imaged by the camera 402. As an example as shown in Figure 10A , the image I2 is an image containing a workpiece image WI of the workpiece W that is the object selected by the priority determination section 214.

[0111] Then, the measurement section 215 performs the pattern matching processing on the image I2 (S108), thereby determining the position and posture of the workpiece W that is the object selected by the priority determination section 214. The accuracy of the posture information of the workpiece W obtained by the pattern matching processing in the first embodiment is higher than the accuracy of the posture information of the workpiece W obtained in the step S103. The pattern matching processing is performed on the image I2 output by the camera 402 by using the CAD data 260 pre-stored in a storage device such as the HDD 254 shown in Figure 3 , which is a model representing the shape of the workpiece W that is the object, and contains ridge line information or line segment information of the workpiece W. In the first embodiment, the measurement section 215 performs a pre-processing such as edge detection on the gray scale image I2, compares the line segment information obtained by the pre-processing with the line segment information contained in the CAD data 260, and calculates the degree of agreement as a score.

[0112] Here, the pattern matching processing of the comparative example will be described. Figure 10B is a schematic diagram for illustrating the pattern matching processing of the comparative example. As shown in Figure 10B , the pattern matching processing of the comparative example is performed by using the line segment information contained in the CAD data 260 and created for all postures obtained around each axis from 0 to 360 degrees. In addition, the pattern matching processing is performed thoroughly from the upper left pixel of the image I2 in an interlaced manner. Therefore, since the calculation complexity is O(n 3 ) based on the number of axes representing the postures, the amount of calculation and the processing time increase.

[0113] Figure 10C is a schematic diagram for illustrating the pattern matching processing of the first embodiment. In the first embodiment, the measuring section 215 performs the pattern matching processing based on the posture information determined by the recognizing section 212. Specifically, the measuring section 215 determines the postures of the models as the CAD data used in the pattern matching processing based on the postures of the workpiece W recognized by the recognizing section 212. For example, since the recognizing section 212 recognizes the workpiece W whose image is contained in the search region and whose posture defined with respect to the axis perpendicular to the predetermined plane such as a horizontal plane is within a predetermined angle al, the postures of the ridgelines contained in the CAD data 260 and used in the comparison in the pattern matching processing can be reduced to within the angle al around each axis. Therefore, as the angle recognized by the recognizing section 212 is set more finely, the load of the pattern matching processing can be reduced more significantly, and the processing time can be reduced accordingly.

[0114] The recognition result obtained in this way can be displayed on the screen of the display 202 as shown in Figure 11 to inform the operator or the user of the result. The CPU 251 displays the workpiece display section 202a and the workpiece detailed display section 202b on the screen of the display 202. The workpiece detailed display section 202b contains the detailed data display section 202c. In the workpiece display section 202a, the candidate regions obtained in step S102 are displayed together with the respective priorities obtained in step S105. In the workpiece detailed display section 202b, the number of the detected workpieces and the number of the workpieces to be picked up are displayed. In the example shown in Figure 11 , "4" is displayed as the number of the detected workpieces, and the number of the workpieces having the priority 1 is displayed. In addition, in the example shown in Figure 11 , the detailed data of the workpiece having the priority 1 is displayed in the detailed data display section 202c. In the detailed data display section 202c, the factor information such as the exposure, the dispersion, and the height obtained in step S105, and the coordinate information and the score information obtained in step S108 are displayed. In the example shown in Figure 11In the example shown, "(100,100,200,0°,0°,45°)" is displayed as the workpiece coordinates. Additionally, in Figure 11 In the example shown, "0.95" is displayed as the score information, "0.9" as the exposure, "0.8" as the dispersion, and "0.8" as the height. Therefore, by observing the screen of display 202, the operator can easily use priority checks to determine which of the bulk-stacking workpieces to pick up and its condition.

[0115] Furthermore, since the center of the search area is aligned with the center of the camera area of ​​camera 402, the workpiece image WI to be searched is located at or near the center of image I2. Therefore, without comparing the entire image I2 with the model in an interlaced manner, pattern matching processing is performed only on the central portion of image I2, taking into account the size of the workpiece W. Adding this processing can shorten the processing time required for pattern matching.

[0116] Then, the CPU 351 of the robot controller 300 controls the robot 100 based on the position and posture information of the workpiece W obtained through pattern matching processing, causing the robot arm 102 to move to the holding position where the robot arm 102 will hold the workpiece W (S109). Then, the CPU 351 controls the robot 100 so that the robot arm 102 holds the workpiece W (S110). Then, the CPU 351 controls the robot 100 so that the robot 100 transports the workpiece W to a predetermined position on the platform 40 (S111). If the robot 100 needs to pick up another workpiece W from the container 30, the process returns to step S101 and repeats steps S101 to S110. In this way, the robot 100 can continuously pick up the bulk-stacking workpieces W one by one.

[0117] Therefore, by performing the picking operation on workpiece W according to the above flowchart, the search area can be narrowed even if workpiece W is piled up in bulk. Furthermore, since workpieces that can be easily picked up by robot 100 are selected using priority, the success rate of robot 100 picking up workpieces is improved. Additionally, since the learning model 263 is used, the number of workpieces identified as easily picked up can be increased. In conventional techniques, pattern matching is performed without narrowing the area of ​​the acquired image. However, in the first embodiment, the area is narrowed down to the search area for pattern matching. Therefore, the processing time required for pattern matching can be shortened. Thus, the productivity of products manufactured by robot 100 is improved.

[0118] Second Embodiment

[0119] Next, the second embodiment will be described. Figure 12is a flowchart showing a control method of the robot of the second embodiment. Note that since the configuration of the robot system of the second embodiment is the same as that of the first embodiment, the description thereof will be omitted. In addition, since the steps S101 to S111 shown and described in the first embodiment are the same as the steps S201 to S211 Figure 12 shown and described in the first embodiment are the same as the steps S101 to S111 shown and described in the first embodiment, the description thereof will be omitted. Figure 5

[0120] After the robot 100 picks up the workpiece W from the container 30, the priority determination section 214 determines whether there is another candidate region other than the candidate region selected in step S205 as a search region in step S212.

[0121] If there is another candidate region (S212: YES), the priority determination section 214 returns to step S205 and sets another candidate region having the next priority as a new search region. In this case, the calculation of the priority is omitted. That is, in step S205, the priority determination section 214 selects another workpiece W having the next highest priority from two or more workpieces W other than the workpiece W that has been the object of the previous selection by the priority determination section 214. The other workpiece W selected by the priority determination section 214 is the next object to be held by the robot 100.

[0122] In the example of the first embodiment, after picking up the workpiece W corresponding to the candidate region D from the container 30, the following picking easiness index f i indicates the workpiece W corresponding to the candidate region B. Therefore, the priority determination section 214 sets the candidate region B as a new search region. In this way, the calculation processing for the index f i in step S205 can be omitted.

[0123] Note that in the case where the position and posture of the subsequent workpiece W do not change significantly, the calculation processing for the index f i in steps S201 to S204 and in step S205 can be omitted. Therefore, a monitoring device can be additionally arranged to monitor the position and posture of the workpiece W. In another case, after the robot 100 picks up the workpiece W from the container 30, the CPU 251 can cause the camera 401 to take an image of the workpiece again, and can determine whether the state of the candidate region other than the search region has changed by checking the difference between the two images, one of which is taken before the workpiece W is picked up and the other of which is taken after the workpiece W is picked up. Note that the second embodiment and one of its variations can be combined with the first embodiment and one of its variations.

[0124] Third Embodiment​

[0125] Next, the third embodiment will be described. In the first embodiment described above, in Figure 5 In step S105, the priority determination part 214 calculates the index f using three factors 1 to 3. i And determine the priority of the workpiece W to be picked up from container 30. However, the index f used for calculation i The number of factors is not limited to three. In the third embodiment, the following situation will be described: adding another factor in addition to the three factors 1 to 3 described in the first embodiment as a factor for calculating the index f. i The factors added in the third embodiment are those related to the success rate of workpiece pickup, which are determined based on past pickup records. Note that since the construction of the robot system in the third embodiment is the same as that in the first embodiment, its description will be omitted.

[0126] Figure 13 This is a block diagram illustrating the function of the CPU 251 in the third embodiment. In addition to the workpiece detection section 211, the identification section 212, the height detection section 213, the priority determination section 214, and the measurement section 215, the CPU 251 also functions as an inspection section 216 by executing program 261. The inspection section 216 inspects... Figure 5 In step S110, the robot 100 holds the workpiece W successfully or unsuccessfully.

[0127] In the third embodiment, an object detection sensor (not shown) is arranged on the robot arm 102 to detect the presence of the workpiece W, and the inspection section 216 checks whether the robot 100 has successfully or failed to hold the workpiece W based on the signal from the object detection sensor (not shown). Note that the inspection section 216 can acquire an image of the robot arm 102 captured and transmitted by a camera device (not shown), and can check whether the robot 100 has successfully or failed to hold the workpiece W based on the image.

[0128] The inspection section 216 outputs two values ​​based on whether the workpiece W exists or not. For example, if the inspection section 216 detects the presence of the workpiece W, it outputs "1", and if the inspection section 216 detects that the workpiece W does not exist, it outputs "0".

[0129] Then, the inspection section 216 trims the search area corresponding to the workpiece W from the image Il. Figure 14A This is a schematic diagram illustrating the pruning process of the third embodiment. Figure 14A In the example shown, candidate region D is pruned.

[0130] Figure 14BThis is a schematic diagram illustrating the learning model of the third embodiment. A dataset is prepared containing trimmed images and information instructing robot 100 whether holding workpiece W is successful or failed. Using this dataset, a neural network can learn the correlation between the trimmed images and the information instructing robot 100 whether holding workpiece W is successful or failed. The neural network used can be a model for class separation. For example, the neural network can be implemented using an algorithm such as VGG (Visual Geometric Groups). In this way, a learning model is created in which the trimmed images are correlated with each other regarding success or failure in holding. This learning model can be stored in a database such as... Figure 3 The HDD254 shown is a storage device. Therefore, the priority determination section 214 can estimate the success rate x4 of the robot 100's past attempts to hold the workpiece W. Based on past picking records, for... Figure 5 The success rate x4 is estimated for each candidate region used in step S102. Therefore, if the success rate x4 is added to equation (3), the exponent f i It is represented by the following formula (4).

[0131] f i =αx 1i +βx 2i +γx 3i +δx 4i (4)

[0132] In formula (4), α+β+γ+δ=1.

[0133] Since the priority of picking up workpieces is determined by taking into account the success rate of past picks, the pick success rate can be improved. Note that the third embodiment and its variations can be combined with one of the various embodiments and variations described above.

[0134] Fourth embodiment

[0135] Next, the fourth embodiment will be described. Figure 1 In the camera system 400 shown, one of cameras 401 and 402 may not be arranged. In the fourth embodiment, it will be described in... Figure 1 The robot system 10 shown does not have a camera 402. Figure 15 This is a flowchart illustrating the robot control method of the fourth embodiment. Figure 15 No close-up images were captured in the flowchart shown.

[0136] Because steps S401 to S405 of the fourth embodiment are... Figure 5 The steps S101 to S105 shown and described in the first embodiment are the same, so their description will be omitted.

[0137] In step S406, the measurement section 215 crops the search region described in the first embodiment from the image Il obtained from the camera 401; thereby obtaining image data of the search region. For example, if the candidate region D is determined as follows... Figure 14A The candidate region D is then cropped out from the search area shown. The search area is a portion of image I1. The search area contains the image of the object W.

[0138] In step S407, the measurement unit 215 performs pattern matching processing on the already trimmed search area. Similarly, in this case, the measurement unit 215 performs pattern matching processing based on the posture information of the workpiece W identified by the recognition unit 212. Therefore, the computational load required for pattern matching processing can be reduced, and thus the processing time required for pattern matching processing can be shortened accordingly.

[0139] Because steps S408 to S410 and Figure 5 The steps S109 to S111 shown and described in the first embodiment are the same, so their description will be omitted.

[0140] Please note that if the imaging area of ​​camera 401 can be narrowed down to the search area and an image of the search area can be captured by using a zoom lens or the like, then camera 402 may not be required in the imaging system 400, and image I2 can be generated by camera 401. In another case, camera 401 may not be required in the imaging system 400. In this case, the position of camera 402 can be adjusted by moving robot 100, and image I1 can be generated by camera 402. Please note that the fourth embodiment and its variations can be combined with one of the various embodiments and their variations described above.

[0141] Fifth embodiment

[0142] Next, the fifth embodiment will be described. In the first embodiment described above, Figure 4A The workpiece detection section 211 shown detects workpiece W as rectangular candidate regions using a deep learning algorithm such as SSD or YOLO. However, the shape of the candidate region to be detected is not limited to rectangles. In the fifth embodiment, an algorithm that detects workpiece W by using non-rectangular regions will be described. Examples of algorithms that can be used in the fifth embodiment include instance segmentation. Figure 16A This is a schematic diagram illustrating an example of image I2 of the fifth embodiment. During the teaching, many examples such as... Figure 16AThe operator specifies a region R1 that surrounds the workpiece image WI along the contour of the workpiece image WI contained in the image I2, and associates the region R1 with information indicating the state of the workpiece image WI surrounded by the region R1. Then, the operator causes the image processing apparatus 200 to learn these information by using an algorithm and a neural network. Since the information indicating the state of the workpiece W and the learning by the neural network are the same as those of the first embodiment, the description thereof will be omitted. Figure 7B The first embodiment described above is the same as the first embodiment described above, and thus the description thereof will be omitted.

[0143] Figure 16B is a schematic diagram illustrating a detection process of the fifth embodiment. As Figure 16B indicated, the workpiece detection portion 211 is caused to learn by using an instance segmentation algorithm so that the workpiece detection portion 211 outputs candidate regions each of which is formed along the contour of a corresponding workpiece image WI. In this way, as Figure 16B indicated, the workpiece detection portion 211 can output the candidate regions E to J. The candidate regions E to J are associated with information about the respective states of the workpiece W. Therefore, when the measurement portion 215 performs the pattern matching process, the pattern matching process can be performed on the region surrounded by the contour of the workpiece. If the candidate region is rectangular, the pattern matching process will also be performed on the portion of the candidate region outside the contour of the workpiece. Since the portion is not necessary for the pattern matching process, the pattern matching process on the portion of the candidate region inside the contour of the workpiece can reduce the time required for the pattern matching process. Note that, although instance segmentation is used as an example in the fifth embodiment, other algorithms can be used as long as the algorithm has the same function of being able to output a region formed along the contour of the workpiece image WI associated with information about the state of the workpiece W. Note that, the fifth embodiment and one of its variations can be combined with one of the above-described various embodiments and one of their variations.

[0144] Sixth Embodiment

[0145] Next, the sixth embodiment will be described. In the above-described first embodiment, the robot hand 102 moves to the holding position in step S109, and holds the workpiece in step S110. The holding position is eventually a single position. However, it can not be necessary to set a single holding position candidate for a single workpiece. For example, a plurality of holding position candidates can be set for a single workpiece. In the sixth embodiment, in order to improve the pickup success rate, a single holding position candidate is eventually selected from a plurality of holding position candidates. Figure 5

[0146] Figure 17 is a flowchart illustrating a control method of a robot of the sixth embodiment. Note that, since the configuration of the robot system of the sixth embodiment is the same as that of the first embodiment, the description thereof will be omitted. In addition, since the configuration of the robot system of the sixth embodiment is the same as that of the first embodiment, the description thereof will be omitted.​Figure 17 Steps S601 to S608 and S610 to S612 and Figure 5 The steps S101 to S108 and S109 to S111 shown and described in the first embodiment are the same, so their description will be omitted. Figure 18 This is a schematic diagram showing multiple candidate holding positions set for workpiece W.

[0147] In the sixth embodiment, after the pattern matching process in step S608, an additional step S609 is performed to select a holding position from multiple holding position candidates. For example... Figure 18 As shown, multiple position candidates are predetermined by the operator. Figure 18 The reference position and orientation O0 of workpiece W, the position and orientation O1 of holding position candidate K1, and the position and orientation O2 of holding position candidate K2 are shown. That is, the information about position and orientation O1 and the information about position and orientation O2 can be freely set by the operator. Therefore, the operator knows the position and orientation O1 and O2 observed from the reference position and orientation O0 and relative to the reference position and orientation O0.

[0148] During pattern matching processing in step S608, a reference position and orientation O0, serving as the position and orientation of the workpiece W, are obtained as the position and orientation observed from another coordinate system (not shown). This other coordinate system is, for example, the coordinate system of the robot system 10. Furthermore, since the positions and orientations O1 and O2 observed from and relative to the reference position and orientation O0 are known, the positions and orientations of the holding position candidates K1 and K2, observed from the other coordinate system, can be obtained. The heights of the holding position candidates K1 and K2 relative to the ground, i.e., relative to the bottom surface of the container 30, are obtained using the method described in the first embodiment.

[0149] In step S609, CPU 251 ultimately selects one of the holding position candidates K1 and K2. For example, the selection method can use any of the following factors.

[0150] Factor 1: A factor used to select the holding position at the highest point relative to the ground or the bottom surface of container 30.

[0151] Factor 2: Factors used to select the holding position closest to the workpiece's center of gravity.

[0152] Factor 3: Factors used to select the position that the robotic arm 101 can approach and maintain in the shortest time.

[0153] The use of these factors is beneficial in the following ways.

[0154] The advantages of using Factor 1 are as follows. The workpieces are stacked in bulk. Therefore, if a workpiece is located in an upper position among the bulk-stacked workpieces, the workpiece is more likely to be exposed from other workpieces. In contrast, if a workpiece is located in a lower position among the bulk-stacked workpieces, a plurality of workpieces including the workpiece are denser, and the workpiece is more likely to be in close contact with other workpieces. Therefore, if an upper holding position candidate among the plurality of holding position candidates is selected as the holding position, it is possible to reduce the likelihood that a workpiece other than the target workpiece is also picked up when the target workpiece is picked up.

[0155] The advantages of using Factor 2 are as follows. After the robot 100 holds the workpiece, when the robot 100 moves the workpiece held by the robot 100, a centrifugal force is generated in the workpiece. In this case, the centrifugal force has a large value at or near the center of gravity of the workpiece. Therefore, if a portion of the workpiece at or near the center of gravity of the workpiece is held by the robot hand 102, it is easy to generate a reaction force against the moment of inertia generated by the centrifugal force. As a result, it is possible to reduce the likelihood that the workpiece falls from the robot hand 102 when moving.

[0156] The advantages of using Factor 3 are as follows. If Factor 3 is used, the robot hand 102 can approach the workpiece in the shortest time, and the amount of movement of the robot arm 101 becomes the smallest. Therefore, it is possible to reduce the likelihood of pick-up failure due to interference between the robot arm 101 and other workpieces other than the target workpiece.

[0157] As described above, in the sixth embodiment, it is possible to select a holding position from which pick-up failure can be reduced from among the plurality of holding position candidates in accordance with the state of the workpiece obtained when the workpiece is to be picked up. Therefore, it is possible to improve the success rate of picking up the workpiece. In order to finally select a single holding position, two or more of the plurality of factors can be combined with each other. Note that one of the sixth embodiment and its variations can be combined with one of the above-described various embodiments and its variations.

[0158] Seventh Embodiment

[0159] Next, the seventh embodiment will be described. In the above-described sixth embodiment, the pattern matching process is performed in step S608 of Figure 17 S609. In the seventh embodiment, a control method will be described in which, before the holding position is selected, interference determination is performed to determine interference between the robot hand 102 and workpieces W other than the workpiece W to be picked up. Figure 19 is a flowchart illustrating a control method of a robot of the seventh embodiment. Note that since the configuration of the robot system of the seventh embodiment is the same as that of the first embodiment, the description thereof will be omitted. In addition, since Figure 19 S701 to S708 and S710 to S713 ofFigure 17 The steps S601 to S612 shown and described in the sixth embodiment are the same, and thus the description thereof will be omitted.

[0160] In the seventh embodiment, as in the sixth embodiment, the CPU 251 performs pattern matching in step S708, thereby obtaining a plurality of holding position candidates. In step S709, the CPU 251 determines, for each holding position candidate, whether the robot hand 102 can approach the workpiece W without interference from workpieces W other than the workpiece W to be picked up. For the interference determination, the height information obtained in step S704 is used.

[0161] Figures 20A-20D The state in which the workpiece W1 to be picked up and another workpiece W2 overlap each other in the seventh embodiment is shown virtually and schematically. For ease of description, in Figures 20A-20D , the workpiece W2 is shown so that the workpiece W1 can be seen through the workpiece W2. Figure 20A The workpiece W1, the reference position and posture O0 of the workpiece W1, and the holding position candidates K1 and K2 at the positions and postures O1 and O2 detected by the pattern matching process of step S708 are shown. In addition, another workpiece W2 covers the top of the holding position candidate K1.

[0162] Figure 20B The height of the workpiece in the state shown in Figure 20A is detected and represented as point cloud information. Figure 20C and Figure 20D are diagrams in which the virtual robot hand 102 is arranged. In the state shown in Figure 20A , if the workpiece W1 is picked up by using, for example, the holding position candidate K1 having the position and posture O1, the robot hand 102 will interfere with the workpiece W2, possibly resulting in a failure to pick up the workpiece W1. In addition, since the posture of the workpiece W2 is tilted, the robot hand 102 will not be able to successfully hold the workpiece W2 either, possibly resulting in a failure to pick up the workpiece W2.

[0163] In the state shown in Figure 20A , the height information obtained in step S704 can be represented as a virtual model indicating the point cloud information about the position and height of the workpiece as shown in Figure 20B . In this case, the position and posture O1 and the point cloud information are known. Therefore, if the shape information of the robot hand 102 is added to the virtual model, the virtual robot hand 102 can be as shown in Figure 20CThe virtual robot hand 102 is virtually arranged on the holding position candidate K1 at the position and posture O1. The shape information of the virtual robot hand 102 refers to the size of the robot hand 102, and the position information of the abutting face at which the robot hand 102 abuts against the workpiece. For example, the shape information of the virtual robot hand 102 can be CAD (Computer Aided Design) information of the robot hand 102.

[0164] In this way, it can be determined whether the virtual robot hand 102 will interfere with the point group of the workpiece W2. Similarly, also in the case where the virtual robot hand 102 is arranged on the holding position candidate K2 at the position and posture O2, it can be determined whether the robot hand 102 will interfere with the point group of the workpiece W2 by performing the same processing. Thus, in the example shown in Figures 20A-20D Figures 20A-20D In the example shown in Figures 20A-20D In step S710, the CPU 251 selects a suitable holding position candidate from among the holding position candidates in which the robot hand 102 will not interfere with the workpiece W2 other than the workpiece W1 to be picked up. Then, the robot 100 picks up the workpiece W1 in steps S711 and S712, and transports the workpiece W1 in step S713.

[0165] As described above, in the seventh embodiment, since the interference determination of step S709 is added, it can be determined whether the robot hand 102 will interfere with the workpiece W2 other than the workpiece W1 to be picked up before the robot hand 102 approaches the workpiece. Note that the holding position candidate of the position at which it has been determined that the robot hand 102 will interfere with the workpiece W2 is not selected in the following step S710. Thus, interference between the robot hand 102 and the workpiece W2 can be avoided, and the robot hand 102 can reliably hold the workpiece W1 in a posture in which the robot hand 102 easily picks up the workpiece W1. As a result, the success rate of picking up these workpieces can be improved. Note that the seventh embodiment and one of its variations can be combined with the above-described various embodiments and one of their variations.

[0166] Variation

[0167] In the above-described seventh embodiment, the interference determination of step S709 is performed by using the height information obtained in step S704. However, the interference determination can be performed by using the contour information of the workpiece described in the fifth embodiment. Hereinafter, the interference determination performed by using the contour information will be described in detail.

[0168] Figure 21The workpiece W1 detected by the pattern matching process of step S708, the reference position and posture O0 of the workpiece W1, and the holding position candidates K1 and K2 at the positions and postures O1 and O2 are shown. In addition, the positions and postures O1 and O2 are projected onto the image I1, which indicates a region defined by the outline of the workpiece detected in step S702. The region RW1 is a region defined by the outline of the workpiece W1, and the region RW2 is a region defined by the outline of the workpiece W2. For ease of description, in Figure 21 In the present variant, since the interference determination of step S709 is added, it is possible to determine whether the robot hand 102 will interfere with the workpiece W2 other than the workpiece W1 to be picked up before the robot hand 102 approaches the workpiece. Note that the holding position candidate at the position where it has been determined that the robot hand 102 will interfere with another workpiece is not selected in the next step S710. Thus, it is possible to avoid interference between the robot hand 102 and the workpiece W2, and the robot hand 102 can reliably hold the workpiece W1 in a posture in which the robot hand 102 easily picks up the workpiece W1. As a result, it is possible to improve the success rate of picking up these workpieces. Note that the present variant can be combined with one of the above-described various embodiments and variants thereof.

[0169] As described above, in the present variant, since the interference determination of step S709 is added, it is possible to determine whether the robot hand 102 will interfere with the workpiece W2 other than the workpiece W1 to be picked up before the robot hand 102 approaches the workpiece. Note that the holding position candidate at the position where it has been determined that the robot hand 102 will interfere with another workpiece is not selected in the next step S710. Thus, it is possible to avoid interference between the robot hand 102 and the workpiece W2, and the robot hand 102 can reliably hold the workpiece W1 in a posture in which the robot hand 102 easily picks up the workpiece W1. As a result, it is possible to improve the success rate of picking up these workpieces. Note that the present variant can be combined with one of the above-described various embodiments and variants thereof.

[0170] The present application is not limited to the above-described embodiments, and various changes can be made within the technical concept of the present application. In addition, the effects described in the embodiments are merely the most appropriate effects of the present application. Thus, the effects of the present application are not limited to those described in the embodiments.

[0171] In the above-described embodiments, a case where the robot arm 101 is a vertical multi-joint robot arm has been described. However, the present disclosure is not limited to this. For example, the robot arm can be any one of various robot arms such as a horizontal multi-joint robot arm, a parallel link robot arm, and a Cartesian coordinate robot arm. In addition, the mechanism that holds the work can be implemented by a machine that is capable of automatically performing a telescopic motion, a bending and stretching motion, an up-and-down motion, a left-and-right motion, a pivoting motion, or a combined motion thereof, in accordance with information data stored in a storage device of a control device.

[0172] In the above-described embodiments, a case where the image processing apparatus 200 and the robot controller 300 are separate computers from each other has been described. However, the present disclosure is not limited to this. For example, the image processing apparatus 200 and the robot controller 300 can be a single computer. In this case, a CPU of the computer can function as the image processing portion and the control portion by executing a program.

[0173] In addition, although a case where the image processing apparatus 200 includes a single CPU 251 has been described, the present disclosure is not limited to this. For example, the image processing apparatus 200 can include a plurality of CPUs or computers. In this case, the functions of the image processing portion can be divided and allocated to the plurality of CPUs or computers.

[0174] In addition, although a case where the image processing apparatus 200 includes a single CPU 251 has been described, the present disclosure is not limited to this. For example, the image processing apparatus 200 can include a plurality of CPUs or computers. In this case, the functions of the image processing portion can be divided and allocated to the plurality of CPUs or computers.

[0175] Other Embodiments

[0176] The present application can also be implemented by providing a program that performs one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium and reading and executing the program by one or more processors included in the system or the apparatus. In addition, the present application can also be implemented by using a circuit such as an ASIC that performs one or more functions.

[0177] The present application can shorten the time required for image processing performed for identifying a work.

[0178] Other Embodiments The present application can shorten the time required for image processing performed for identifying a work.

[0179] Embodiment(s) of the present application can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on storage media (which can also be referred to more fully as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above-described embodiment(s) and / or that includes one or more circuits (e.g., application specific integrated circuits (ASICs)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage media to perform the functions of one or more of the above-described embodiment(s) and / or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer can comprise one or more processors (e.g., central processing units (CPUs), micro processing units (MPUs)) and can include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions can be provided to the computer, for example, from a network or the storage media. The storage media can include, for example, one or both of a hard disk, a random access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)®), a flash memory device, a memory card, and the like. TM

[0180] Embodiment(s) of the present application can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on storage media (which can also be referred to more fully as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above-described embodiment(s) and / or that includes one or more circuits (e.g., application specific integrated circuits (ASICs)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage media to perform the functions of one or more of the above-described embodiment(s) and / or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer can comprise one or more processors (e.g., central processing units (CPUs), micro processing units (MPUs)) and can include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions can be provided to the computer, for example, from a network or the storage media. The storage media can include, for example, one or both of a hard disk, a random access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)®), a flash memory device, a memory card, and the like.

[0181] While the present application has been described with respect to exemplary embodiments, it will be recognized that the application is not limited thereto. The scope of the claims should be construed to encompass all such modifications and alterations in accordance with the widest interpretation.​

Claims

1. A robot system comprising: robot; Camera device; The image processing section is configured to designate at least one region in an image of a plurality of objects captured by the camera device, in which a predetermined object with a predetermined pose exists, and to obtain information about the position and / or pose of the predetermined object in the region. as well as The control unit is configured to control the robot based on information about the position and / or posture of the predetermined object, so that the robot holds the predetermined object. The image processing section performs the following: A first imaging process is used to capture images of the plurality of objects and specify the area where the predetermined object exists in the predetermined pose. The second shooting process is used to capture an image of a shooting area smaller than the shooting area captured in the first shooting process, and to obtain information about the position and / or posture of the predetermined object in the area.

2. The robot system according to claim 1, wherein, The image processing section determines the pose of the model based on the predetermined pose, the model being used to obtain information about the position and / or pose of the predetermined object and being constructed to indicate the shape of the predetermined object.

3. The robot system according to claim 1, wherein, The image processing unit uses a machine learning model to specify the region where the predetermined object exists in a first pose or the region where the predetermined object exists in a second pose. In the first pose, it captures an image of a first surface of the predetermined object, and in the second pose, it captures an image of a second surface of the predetermined object that is different from the first surface.

4. The robot system according to claim 3, wherein, The first surface is the front of the predetermined object, and the second surface is the back of the predetermined object.

5. The robot system according to claim 3, wherein, The image processing section obtains a learning model by using an image obtained by photographing a first surface of an object and / or an image obtained by photographing a second surface of an object that is different from the first surface of the object.

6. The robot system according to claim 1, wherein, The image processing unit performs the capture of images in the second capture process at a higher resolution than the capture of images in the first capture process.

7. The robot system according to claim 1, wherein, The image processing section displays information about the status of the predetermined object on the display device.

8. The robot system according to claim 1, wherein, The at least one region is two or more regions, and The image processing section specifies the two or more regions and obtains the priority of predetermined objects corresponding to the two or more regions.

9. The robot system according to claim 8, wherein, The image processing section displays the priority on the display device.

10. The robot system according to claim 8, wherein, The image processing section uses multiple factors to determine the priority of predetermined objects corresponding to the two or more regions.

11. The robot system according to claim 10, wherein, One of the factors is the success rate achieved by the robot in its past attempts to hold the workpiece.

12. The robot system according to claim 10, wherein, The multiple factors include at least one of the exposure of the predetermined object, the dispersion of the predetermined object, and the height of the predetermined object when it is piled up in bulk.

13. The robot system according to claim 1, wherein, The image processing unit obtains the position and / or pose of the predetermined object by performing pattern matching on the region.

14. The robot system according to claim 13, wherein, The image processing section performs pattern matching on the region in question but not on other regions outside of that region.

15. The robot system according to claim 1, wherein, The control unit causes the robot to first hold a predetermined object among the plurality of objects that corresponds to the region.

16. The robot system according to claim 1, wherein, The imaging device includes a first imaging unit and a second imaging unit. The first imaging unit is configured to capture images of the plurality of objects, and the second imaging unit is configured to capture images of an area smaller than the area captured by the first imaging unit. The image processing section includes: The first camera unit captures images of the plurality of objects and designates the region, and information about the position and / or posture of the predetermined object is obtained based on the image of the predetermined object captured by the second camera unit corresponding to the region.

17. The robot system according to claim 16, wherein, The image processing unit obtains information about the position and / or posture of the predetermined object by performing pattern matching on the image of the predetermined object captured by the second camera unit.

18. The robot system according to claim 16, wherein, The second camera unit is mounted on the robot.

19. The robot system according to claim 1, wherein, The image processing section specifies the region based on the outline of the predetermined object.

20. The robot system according to claim 19, wherein, Multiple holding positions are set in the predetermined object so that the robot holds the predetermined object, and The image processing section obtains the priority of the plurality of holding positions based on the state of the predetermined object.

21. The robot system according to claim 20, wherein, The image processing unit determines the priority of the plurality of holding positions based on at least one of the height of each holding position of the predetermined object in bulk stacking, the center of gravity of the predetermined object, and the time required for the robot to approach each of the plurality of holding positions of the predetermined object.

22. The robot system according to claim 1, wherein, Multiple holding positions are set in the predetermined object so that the robot holds the object, and The image processing section specifies one of the plurality of holding positions that causes the robot to interfere with the holding of other objects besides the predetermined object when the robot holds the predetermined object.

23. The robot system according to claim 22, wherein, The image processing unit determines whether the robot will interfere with the objects other than the predetermined object based on the height of the predetermined object, the height of other objects besides the predetermined object, and / or the outline of the predetermined object and the outline of other objects besides the predetermined object.

24. A control method for a robot system, the robot system comprising a robot and a camera device, the control method comprising: The image processing unit designates at least one region in an image of a plurality of objects captured by the camera device, indicating the presence of a predetermined object with a predetermined posture, and obtains information about the position and / or posture of the predetermined object in the region. as well as The control unit controls the robot based on information about the position and / or posture of the predetermined object, so that the robot holds the predetermined object. The image processing unit performs a first imaging process, which captures images of the plurality of objects and specifies the area where the predetermined object exists in the predetermined pose. The image processing unit performs a second shooting process to capture an image of a shooting area smaller than the shooting area captured in the first shooting process, and obtains information about the position and / or posture of the predetermined object in the area.

25. An image processing apparatus comprising: The image processing section is configured to designate at least one region in an image of multiple objects containing a predetermined object with a predetermined pose, and to obtain information about the position and / or pose of the predetermined object within the region. The image processing section performs the following: A first imaging process is used to capture images of the plurality of objects and specify the area where the predetermined object exists in the predetermined pose. The second shooting process is used to capture an image of a shooting area smaller than the shooting area captured in the first shooting process, and to obtain information about the position and / or posture of the predetermined object in the area.

26. An image processing method, comprising: The image processing unit designates at least one region in an image of multiple objects to contain a predetermined object with a predetermined pose. as well as The image processing unit obtains information about the position and / or pose of a predetermined object in the region. The image processing unit performs a first imaging process, which captures images of the plurality of objects and specifies the area where the predetermined object exists in the predetermined pose. The image processing unit performs a second shooting process to capture an image of a shooting area smaller than the shooting area captured in the first shooting process, and obtains information about the position and / or posture of the predetermined object in the area.

27. A method for manufacturing a product using a robot system according to any one of claims 1 to 23.

28. A computer-readable non-transitory recording medium storing a program that causes a computer to perform the control method according to claim 24.

29. A computer-readable non-transitory recording medium storing a program that causes a computer to perform the image processing method according to claim 26.

Images