Robot system and method for controlling robot system

Abstract

A robot system 1 for picking individual objects OB from an irregularly loaded collection of the objects OB, said robot system 1 being characterized by comprising: an imaging means 2 that captures images (IM2, IM3) of the collection of the objects OB; a target object detection unit 31 that identifies an image region OTI within the image of the collection of the objects OB which corresponds to a target object OT for picking and detects the position of the target object OT on the basis of the captured image via object detection based on a rule base which is based on a prescribed algorithm, and an estimation model which is generated via machine learning; and a parallel link–type robot (1X, 32) that picks the target object OT from among the collection of the objects OB and transfers the same to a prescribed position on the basis of the detected position of the target object OT.

Description

Robot system and control method for robot system

[0001] The present disclosure relates to a robot system that picks and transfers individual objects from a collection of irregularly stacked objects, and a control method for the robot system, and in particular to a robot system that picks irregularly shaped objects that are flexible and change shape.

[0002] In recent years, picking has become common in factory production lines, logistics warehouses, etc., where objects to be supplied to a process, such as finished products, intermediate products, and components, are removed from containers and transferred to subsequent processes. This picking requires the ability to quickly detect the position and orientation of each object from a collection of irregularly stacked objects, and for this reason, picking has traditionally been performed manually.

[0003] In response to this, for example, Patent Document 1 proposes a robot device that has an image processing function, detects the position and orientation of each workpiece of the same shape that is piled up, and automatically picks up each detected workpiece.

[0004] Japanese Patent Application Laid-Open No. 2000-288974

[0005] However, the conventional robot device described in Patent Document 1 has a configuration in which each detected workpiece is picked up by a vertical articulated robot, which makes it difficult to increase the speed of the picking operation, and has the problem that it is difficult to supply objects at a speed that satisfies the processing capacity of a packaging machine in a downstream process, for example. The reason for this is that an articulated robot has a redundant structure in which multiple joints connected in series operate in parallel, which gives a high degree of freedom in position control during the picking operation, but on the other hand, the calculation processing becomes complex, making it difficult to improve the processing speed.

[0006] In recent years, there has also been a demand for the ability to pick irregular objects, such as pillow packages containing flexible, shape-changing products such as food, confectionery, cosmetics, and daily necessities. In this case, it is necessary to establish a method for accurately detecting the position and orientation of each irregular object, each with a different shape, that exists individually among a collection of irregularly stacked objects.

[0007] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a robot system and a control method for a robot system that speeds up picking operations for irregular objects, such as pillow packages, each of which has a different shape among a collection of objects, thereby improving object supply capacity.

[0008] A robot system according to one aspect of the present disclosure is a robot system that picks individual objects from a collection of irregularly stacked objects, and is characterized by comprising: an imaging means that captures an image of the collection of objects; a target object detection unit that, based on the captured image, identifies an image portion corresponding to an object to be picked from the image of the collection of objects by using a rule base based on a predetermined algorithm and object detection based on an inference model generated by machine learning, and detects the position of the target object; and a parallel link robot that, based on the detected position of the target object, picks the target object from the collection of objects and transfers it to a predetermined position.

[0009] According to one aspect of the present disclosure, it is possible to provide a robot system and a control method for a robot system that speeds up picking operations for irregular objects, such as pillow packages, that have individually different shapes among a collection of objects, thereby improving object supply capacity.

[0010] 7 is a perspective view schematically illustrating the configuration of a robot system 1 according to an embodiment. FIG. 8 is a functional block diagram illustrating an overview of the functional configuration of the robot system 1. (a) and (b) are front views illustrating an overview of the operation of the robot system 1. FIG. 8 is a functional block diagram illustrating the configuration of a target object detection unit 31. (a), (b), and (c) are schematic diagrams illustrating the state of a two-dimensional image related to object detection and the operation of the object detection unit. (a), (b), and (c) are schematic diagrams illustrating the operation of a target object identification unit. FIG. 8 is a flowchart illustrating an overview of object picking and transfer operations in the robot system 1. FIG. 9 is a flowchart illustrating an example of target object detection processing based on a rule base in step S3 of FIG.

[0011] Overview of a form for implementing the present invention A robot system according to an embodiment of the present disclosure is a robot system that picks individual objects from a collection of objects that are irregularly stacked, and is characterized by including: an imaging means that captures an image of the collection of objects; a target object detection unit that, based on the captured image, identifies an image portion corresponding to an object to be picked from the image of the collection of objects by object detection based on a rule base based on a predetermined algorithm and an inference model generated by machine learning, and detects the position of the target object; and a parallel link robot that, based on the detected position of the target object, picks the target object from the collection of objects and transfers it to a predetermined position.

[0012] With this configuration, object detection using rules and machine learning makes it possible to pick irregular objects, such as pillow packaging, which have individually different shapes among a collection of objects.Furthermore, simple control using a parallel link robot can speed up the picking operation for detected irregular objects, providing a robot system that improves object supply capacity, thereby realizing object supply that contributes to shortening the takt time of the process.

[0013] In another aspect, in any of the aspects described above, the rule base may be configured as a standard set based on one or more elements selected from the height corresponding to the object image portion shown in the image, discontinuity in the image contour of the object image portion, projected area of ​​the object image portion, distinction between the front and back of the object image portion, and accuracy of detection of the object image portion.

[0014] This configuration makes it possible to realize object detection based on a rule base that is more suited to the state of irregular objects present in an aggregate, and to detect irregular objects more accurately and quickly.

[0015] In another aspect, in any of the aspects described above, the image may include a two-dimensional image of a collection of objects viewed in a plane, and further a three-dimensional image to which height data has been assigned, and the target object detection unit may be configured to detect an image portion of a target object candidate shown in the image using the inference model based on the two-dimensional image, calculate height data corresponding to the detected image portion of the target object candidate based on the three-dimensional image, and identify the image portion corresponding to the target object based on the data of the image portion of the target object candidate and the height data.

[0016] With this configuration, object detection using rule-based and machine learning can accurately and quickly detect irregular objects with different shapes that exist among a collection of irregularly stacked objects.

[0017] In another aspect, in any of the aspects described above, after the robot picks up the target object and before transferring it to the predetermined position, the imaging means may capture an image of the collection of objects from which the target object has been removed, and the target object detection unit may be configured to detect the position of a new target object.

[0018] This configuration reduces the operation standby time of a parallel link robot, which can be made faster through simple control, and further speeds up picking operations, thereby improving object supply capacity.

[0019] In another aspect, a control method for a robot system according to an embodiment may be a control method for a robot system that picks individual objects from a collection of irregularly stacked objects, and may be configured to capture an image of the collection of objects, and based on the captured image, identify an image portion of the image of the collection of objects that corresponds to an object to be picked using a rule base based on a predetermined algorithm and object detection based on an inference model generated by machine learning, detect the position of the target object, and, based on the detected position of the target object, pick the target object from the collection of objects and transfer it to a predetermined position using a parallel link robot.

[0020] This configuration makes it possible to realize a control method for a robot system that speeds up the picking operation for irregularly shaped objects and improves the object supply capacity.

[0021] The configuration of a robot system 1 according to an embodiment will be described with reference to the drawings. In this specification, the X, Y, and Z directions in each drawing may be referred to as the width, depth, and height directions, respectively, and the positive height direction may be referred to as the "up" direction and the negative height direction as the "down" direction.

[0022] In addition, the scale of the components in each drawing is not necessarily the same as the actual one. Also, for ease of understanding, illustrations of covers and the like may be omitted. For example, some components, transport paths, etc. may be illustrated schematically or outlined. Furthermore, in this specification, the symbol "to" used to indicate a numerical range includes both ends of the range. Furthermore, the materials, numerical values, etc. described in this embodiment are merely preferred examples and are not limited thereto. Furthermore, appropriate modifications are possible within the scope of the technical concept of this disclosure. Furthermore, combinations of parts of the configurations of other embodiments are possible within the scope of no contradiction.

[0023] <Configuration of Robot System 1> The configuration of the robot system 1 will be described with reference to the drawings. Fig. 1 is a perspective view that schematically shows the configuration of the robot system 1 according to the embodiment.

[0024] (Overall configuration) The robot system 1 is a robot system that picks up an object OB, such as a finished product, an intermediate product, or a component, to be supplied to a process in a manufacturing process or the like, from a container CT used for supply, and transfers the object OB to, for example, a transport means TM as a predetermined position within the process.

[0025] As shown in FIG. 1, the robot system 1 includes a parallel link type robot 1A that functions as a transport means for transferring objects OB, an imaging means 2 that captures images of a collection of objects OB loaded in a container CT, and a control unit 3 that controls the robot 1A based on the images.

[0026] (Overview of Configuration of Each Part) Below, we will explain an overview of the configuration of each part in the robot system 1. The object OB to be picked by the robot system 1 is composed of, for example, products such as finished products, work-in-progress products, intermediate products, components, packaging, etc. that are supplied to processes on a production line, a logistics warehouse, etc. Furthermore, the object OB includes not only fixed objects with fixed shapes such as rigid bodies, but also irregular objects with individually different shapes such as pillow packages that contain flexible, shape-changing objects such as food, confectionery, cosmetics, and liquid daily necessities.

[0027] A pillow package has a body OBd in which the packaged item and gas are sealed in a packaging member made of resin film, and end seal portions OBe at both ends of the body OBd. For example, the pillow package is formed by sealing the packaged item together with the gas in a cylindrical packaging member and then sealing the front and back of the packaged item. The sealed gas provides appropriate cushioning and ease of deformation.

[0028] In the object OB, the resin film of the pillow package may be made of a film member having a translucent or mirror-like outer surface. Information or a design may be printed on the front and / or back of the pillow package, or a printed label may be attached.

[0029] These objects OB are generally supplied to the process in the form of a collection of objects OB stacked irregularly in a mountain shape on the bottom of a container CT used as a returnable box, as shown in Fig. 1. In such a collection of objects OB, if the objects OB are irregularly shaped objects such as pillow packages, the individual objects OB are present in an irregularly stacked state in the container CT, with not only their positions and orientations but also their shapes being individually different.

[0030] The transport means TM places the supplied object OB on it and transports it (T １ ) and transports the material into the process. As the transport means TM, various conveyors can be mentioned.

[0031] The robot 1A is a robot mechanism equipped with a parallel link type robot arm mechanism that functions as a manipulator to take out an object OB from a container CT and transfer it to a transport means TM. As shown in FIG. 1 , the robot 1A has a suction head 11, a robot hand 12, a robot arm 13, and a robot main body 14.

[0032] The suction head 11 is supported at the lower end of the robot hand 12, has a suction portion 11a on its underside, and has the function of suction-holding an object OB from above. The suction portion 11a has an air intake hole (not shown) on its underside. The air intake hole is connected to a vacuum pump or the like (not shown). With the lower surface of the suction portion 11a in contact with the upper surface of the object OB loaded in the container CT, negative pressure is applied through the air intake hole, thereby suction-holding the object OB.

[0033] The robot hand 12 is supported at the tip of the robot arm 13 and supports the suction head 11 so that it can rotate freely within the XY plane, thereby changing the orientation of the suction head 11 .

[0034] The suction head 11 and / or the robot hand 12 may have a structure that allows them to be held movably in the Z direction. With such a structure, even if the suction head 11 and the object OB come into strong contact when suctioning and holding the object OB, it is possible to prevent problems such as the object OB being torn.

[0035] The robot arm 13 is a so-called parallel link robot, with multiple arms (three in this example) extending radially in the XY plane and supported by a robot main body 14 installed on, for example, the ceiling of the facility. The robot arm 13 supports the suction head 11 via the robot hand 12 so that it can rotate freely within the XY plane, and can move the suction head 11 to a predetermined position in three-dimensional space, for example, with the suction portion 11a facing downward.

[0036] In a parallel link robot, each of the multiple robot arms 13 includes a drive means (e.g., a motor) at a fulcrum portion rotatably joined to the robot main body 14. With this configuration, the parallel link robot can quickly change the position of the suction part 11a in the X, Y, and Z directions in three-dimensional space by changing the angle of each robot arm 13 around the fulcrum using a simple control method that independently drives the drive means for each robot arm 13 based on a control signal issued from the control unit 3.

[0037] The imaging means 2 is, for example, a camera using a CCD (Charge Coupled Device) image sensor, which captures images of the collection of objects OB irregularly stacked in the container CT. The imaging means 2 includes a 2D imaging means 21 that captures a two-dimensional image of the collection of objects OB from vertically above the bottom surface of the container CT. The imaging means 2 also includes a 3D imaging means 22 that generates a three-dimensional image. The 3D imaging means 22 may be configured to capture multiple images of the collection of objects OB from different positions diagonally above, compare them, and calculate the depth of the image portion in the planar direction. Alternatively, the 3D imaging means 22 may be a time-of-flight (ToF) camera that generates a three-dimensional image by irradiating the collection of objects OB with infrared light and measuring the time it takes for the reflected light to be received. Alternatively, a pattern projection method such as stripes or random dots may be used.

[0038] The control unit 3 is electrically connected to the imaging means 2 and the robot 1A, and controls the operation of each unit by outputting a control signal to the robot 1A based on an image captured by the imaging means 2. The control unit 3 is realized, for example, as a computer including a general CPU (Central Processing Unit) and RAM (Random Access Memory), and a program executed by these. The control unit 3 realizes functions related to the control method of the robot system 1 by reading the control program related to the robot system 1 from a storage device or the like into the RAM and executing it.

[0039] FIG. 2 is a functional block diagram showing an outline of the functional configuration of the robot system 1. FIGS. 3A and 3B are front views illustrating an outline of the operation of the robot system 1. As shown in FIG. 2, the control unit 3 includes a target object detection unit 31 and a robot control unit 32. Of these, the target object detection unit 31 detects the position and orientation of an object image portion corresponding to an object OB irregularly loaded in the container CT based on an image captured by the imaging means 2, identifies an image portion corresponding to a target object OT to be picked, and outputs information regarding the position and orientation of the corresponding target object OT to the robot control unit 32, as shown in FIG.

[0040] The robot control unit 32 also controls the units constituting the robot 1A so that they operate in a linked manner. Specifically, the robot control unit 32 issues a control signal to control the drive means for each robot arm 13, and adjusts the angle (θ １ , θ ２ , θ ３ ), the position of the suction part 11a in the X, Y, and Z directions in the three-dimensional space is changed, and the suction part 11a of the suction head 11 is moved to the position of the upper surface of the target object OT in the three-dimensional space. Also, the robot control unit 32 issues a control signal to control the suction from the suction part 11a and its release (P ＯＮ／ＯＦＦ Furthermore, the robot control unit 32 controls the direction (θ ＸＹ ) to control the

[0041] As a result, as shown in FIG. 3(a), the object OT to be picked is picked up from the collection of objects OB irregularly loaded in the container CT for carrying in (P ＯＮ ) and take it out (M1), and then, as shown in FIG. 3(b), after transferring it to a predetermined position (M ２ ), release negative pressure (P ＯＦＦ ) to realize the function of transferring the target object OT to the transport means TM. Then, another object OB is selected as a new target object OT, and by repeating the above operation, each object OB is taken out from the collection of irregularly stacked objects OB and transferred to the transport means TM.

[0042] (Functions of the target object detection unit 31) Next, the functions of the target object detection unit 31 will be described in detail. Fig. 4 is a functional block diagram showing the configuration of the target object detection unit 31. As shown in Fig. 4, the target object detection unit 31 includes a 3D image acquisition unit 311, a 2D image acquisition unit 312, a point cloud data generation unit 313, an object detection unit 314, an inference model generation unit 315, a target object identification unit 316, and a rule base storage unit 317.

[0043] The 3D image acquisition unit 311 is a circuit that acquires a 3D image IM3 of the collection of objects OB loaded in the container CT from the 3D imaging means 22 and outputs it to a subsequent stage.

[0044] The 2D image acquisition unit 312 is a circuit that acquires a two-dimensional image IM2 of a collection of objects OB viewed in plan from the 2D imaging means 21 and outputs the image to a subsequent stage. The 3D image acquisition unit 311 and the 2D image acquisition unit 312 can be, for example, an image capture board or other device that captures image data into a processing device such as a computer.

[0045] The point cloud data generation unit 313 is a processing unit that generates point cloud data, which is three-dimensional voxel data of the collection of objects OB, based on the three-dimensional image IM3 of the collection of objects OB. The generated point cloud data is output to the target object identification unit 316.

[0046] The object detection unit 314 is a processing unit that detects image portions of multiple objects shown in the two-dimensional image IM2 using an inference model generated in advance by machine learning based on the two-dimensional image IM2. Figures 5(a), 5(b), and 5(c) are schematic diagrams showing aspects of two-dimensional images related to object detection and explaining the operation of the object detection unit.

[0047] As shown in Figure 5(a), the object detection unit 314 detects the position and orientation of the object image OBI in the X-Y plane by detecting the image contour OLI of the object image portion OBI (hereinafter sometimes referred to as the "object image OBI") corresponding to multiple objects OB shown in the two-dimensional image IM2.

[0048] At this time, the object detection unit 314 may take into consideration the overlap of the objects OB and, based on the discontinuity of the image contour OLI of the object image OBI, select an object image OBI whose image contour OLI does not have any discontinuous parts as the image portion OPI of the target object candidate for picking (the shaded part in FIG. 5( a) ; hereinafter, this may be referred to as the "target candidate image OPI"). The reason for this is based on the inventor's finding that an object OB whose image contour OLI does not have any discontinuous parts is located at the top layer of a collection of multiple objects OB that are arranged in an irregularly overlapping state, and is therefore easy to pick.

[0049] Furthermore, the object detection unit 314 may select an object image OBI with a large image area (the grid-hatched portion in FIG. 5A) from the picking target candidate images OPI based on the object projection area on the XY plane. The reason for this is based on the inventor's finding that an object OB with a large object image OBI area is an object with a small tilt among a collection of irregularly stacked objects OB, and is therefore easier to pick.

[0050] 5B, when detecting the image contour OLI of the object image portion, if the object is a package made of a translucent material through which the contents PR are visible from the outside, the outer edge of the image of the package may be detected as the image contour OLI of the object image OBI.

[0051] Furthermore, as shown in Figure 5(c), when a label LB is attached to the front or back of the outer surface of an object, the orientation of the object image OBI in the XY plane can be detected or the front and back of the object can be distinguished based on the position of the label LB in the object image OBI.

[0052] These conditions can be reflected in object detection using the inference model by providing a two-dimensional image IM2 containing the above conditions as training data to the inference model generation unit 315 when constructing an inference model using machine learning, as described below.

[0053] The object detection unit 314 outputs the detected picking target candidate image OPI and information about the position of the target candidate image OPI in the XY plane to the target object identification unit 316. Furthermore, information about the orientation of the target candidate image OPI in the XY plane and information about distinguishing between the front and back may be output.

[0054] The inference model generation unit 315 is a processing unit that generates an inference model in the learning phase and provides the inference model to the object detection unit 314. The inference model generation unit 315 can generate an inference model related to object detection by providing, as training data, a two-dimensional image of a collection of objects OB that has been acquired in advance by the 2D image acquisition unit 312 in a planar view, and a result of annotation related to object detection performed by an operator using the image, and performing machine learning on the two-dimensional image.

[0055] For example, deep learning using a neural network (CNN: Convolutional Neural Network) can be used for the machine learning in the inference model generation unit 315, and known software can be used. In addition, in the operation phase, the inference model generation unit 315 may be excluded from the implemented functions of the target object detection unit 31.

[0056] The target object identification unit 316 is a circuit that receives as input the position information in the XY plane of the picking target candidate image OPI obtained from the object detection unit 314 and the point cloud data, which is three-dimensional voxel data of a collection of objects OB obtained from the point cloud data generation unit 313, and identifies the image portion OBI corresponding to the picking target object OT from the target candidate image OPI based on a rule base.

[0057] The criteria of the rule base are stored in the rule base storage unit 317 and are selectively provided from the rule base storage unit 317 to the target object identification unit 316 as appropriate based on operation input, etc. The rule base to be applied may be different depending on the type of object OB to be picked, etc.

[0058] For example, the target object identification unit 316 may calculate height data corresponding to one or more target candidate images OPI detected by the object detection unit 314, and identify an image portion corresponding to the target object OT from among the target candidate images OPI based on the magnitude of the value of the height data as a rule base. In this case, the target candidate image OPI with the largest height data can be selected as the image portion corresponding to the target object OT by referring to the height data of point cloud data whose position coordinates in the XY plane are equivalent to those of the target candidate image OPI.

[0059] FIG. 6 is a diagram illustrating the processing performed by the target object identification unit 316. It is a three-dimensional schematic diagram showing a target candidate image OPI and point cloud data PD, which is part of point cloud data representing a collection of objects OB and has equivalent position coordinates in the X-Y plane to the target candidate image OPI, superimposed on one another. In the example shown in FIG. 6, of the six target candidate images OPI detected from multiple object images OBI, the object with the largest height data value in the point cloud data PD included in the target candidate image OPI (shown with a bold image outline in FIG. 6) is selected as the image portion OTI (hereinafter sometimes referred to as the "target object image OTI") corresponding to the object OT to be picked. Comparison of height data between target candidate images OPI may be based on the average value, minimum value, difference between maximum and minimum values ​​of the height data in the included point cloud data PD, or a combination thereof.

[0060] The reason for selecting the target candidate image OPI with the largest height data value as the target object image OTI is that the corresponding target object OT is located at the top layer of a collection of multiple objects OB arranged in an irregularly overlapping state, and is therefore considered to be easy to pick.

[0061] As a rule-based criterion for identifying the target object image OTI, in addition to height data, the same criteria as those used for selecting the target candidate image OPI in the object detection unit 314 may be used.

[0062] That is, based on the discontinuity of the image contour OLI, a target candidate image OPI having no discontinuous portion in the image contour OLI may be selected as the target object image OTI for picking.

[0063] Alternatively, based on the object projection area, a target candidate image OPI having a large area in the XY plane may be selected as the target object image OTI.

[0064] Furthermore, the orientation of the target candidate image OPI in the XY plane may be detected, or the front and back may be distinguished based on the position of the label LB in the target candidate image OPI.

[0065] Furthermore, among the target candidate images OPI detected by the object detection unit 314, the one with a high degree of detection accuracy may be selected as the target object image OTI.

[0066] As described above, these criteria are the criteria used to detect the target candidate image OPI in the machine learning inference model in the object detection unit 314. However, by reapplying these criteria as rule-based criteria in the identification of the target object image OTI in the target object identification unit 316, it is possible to achieve object detection that is more suited to the state of irregular objects that exist individually with different shapes among a collection of irregularly stacked objects, and to detect the position and orientation of each of the irregular objects with higher accuracy.

[0067] The target object identification unit 316 receives information about the position and orientation of the target object OT corresponding to the detected target object image OTI (for example, position (X, Y, Z), orientation θ Ｘ－Ｙ , type of front or back, etc.) to the robot control unit 32.

[0068] <Operation of Robot System 1> The following describes the operation of the robot system 1 having the above configuration. Figure 7 is a flowchart showing an outline of the object picking and transfer operation in the robot system 1. As shown in Figure 7, first, in the robot system 1, the 2D imaging means 21 acquires a two-dimensional image IM2 that shows a planar view of a collection of objects OB that are irregularly loaded in a container CT (step S11), and the target object detection unit 31 detects multiple object images OBI shown in the two-dimensional image IM2 using an inference model based on machine learning, and outputs information regarding a target candidate image OPI for picking and the position of the target candidate image OPI in the X-Y plane (step S21).

[0069] In parallel with the above processing, the 3D imaging means 22 acquires a three-dimensional image IM3 of the collection of objects OB (step S12), and the target object detection unit 31 generates and outputs point cloud data PD, which is three-dimensional voxel data of the collection of objects OB (step S22).

[0070] Next, in step S3, the target object detection unit 31 receives position information in the XY plane of the target candidate image OPI to be picked and point cloud data PD of the collection of objects OB as input, identifies the target object image OTI to be picked from the target candidate images OPI based on the rule base, and outputs information on the position and orientation of the target object OT to the robot control unit 32. At this time, the rule base to be applied may be a criterion set based on one or more elements selected from 1) the height corresponding to the target candidate image OPI, 2) discontinuity of the image contour of the target candidate image OPI, 3) the projected area of ​​the target candidate image OPI in the XY plane, 4) distinction between the front and back of the target candidate image OPI, and 5) the accuracy of detection of the object image OBI.

[0071] Fig. 8 is a flowchart showing an example of the rule-based target object detection process in step S3. As shown in Fig. 8, for a target candidate image OPI shown in an image, the height of the object (step S31), discontinuity of the object image contour (step S32), and the projected area of ​​the object image (step S33) may be determined in this order, and in step S34, these results may be comprehensively evaluated to identify the target object image OTI for picking.

[0072] Meanwhile, the robot control unit 32 performs the following process in parallel with the process of steps S11 to S3 in the target object detection unit 31.

[0073] First, it is determined whether the process is the first detection process (step S4), and if it is the first detection, the process proceeds to step S6. On the other hand, if it is not the first detection, the object OB has been sucked and held by the suction head 11 by the previous picking, so the object OB is moved to a predetermined position and transferred (step S5), and then the process proceeds to step S6.

[0074] Next, in step S6, it is determined whether or not the detection process is to be ended. If the detection process is not to be ended, in step S7, a picking operation of the target object OT is performed.

[0075] Specifically, the robot control unit 32 calculates the angle (θ １ , θ ２ , θ ３ ) to move the suction portion 11a of the suction head 11 to a position in three-dimensional space on the top surface of the target object OT. Then, a picking operation is performed to suck, hold, and remove the target object OT from the collection of irregularly stacked objects.

[0076] Then, the process returns to steps S11, S12, and S4, and in the next iteration, the object OB, which has already been picked up and is being sucked and held by the suction unit 11a, is moved to a predetermined position and transferred to the transport means TM (step S5). On the other hand, if it is determined in step S6 that the detection process is to be ended, the process ends.

[0077] Through the above operations, the robot system 1 can detect irregular objects, such as pillow packages, each with a different shape, from a collection of irregularly stacked objects OB, and achieve high-speed picking operations through simple control utilizing the parallel link type robot 1A.

[0078] <Effects> As described above, the robot system 1 is a robot system that picks up individual objects OB from a collection of objects OB that are irregularly stacked, and is characterized by comprising: an imaging means 2 that captures an image (IM2, IM3) of the collection of objects OB; a target object detection unit 31 that identifies a target object image OTI to be picked from the image (IM2, IM3) of the collection of objects OB by object detection based on a rule base based on a predetermined algorithm and an inference model generated by machine learning, based on the captured images (IM2, IM3), and detects the position of the target object OT; and a parallel link type robot (1X, 32) that picks the target object OT from the collection of objects OB based on the detected position of the target object OT, and transfers it to a predetermined position.

[0079] As described above, conventional picking robot devices have a high degree of freedom in position control during the picking operation for each detected object. However, because the picking operation is performed by an articulated robot, which requires complex calculations and makes it difficult to improve processing speed, it is difficult to speed up the picking operation. For example, there is a problem in that it is difficult to supply objects at a speed that meets the processing capacity of the packaging machine in the downstream process. This is, for example, one of the factors that causes an increase in takt time in the packaging process.

[0080] Furthermore, in order to pick irregular objects such as pillow packages containing flexible products whose shapes change, it was necessary to establish a method for detecting the position and posture of each irregular object, each with a different shape, accurately and quickly among a collection of irregularly stacked objects.

[0081] In contrast, the robot system 1 employs object detection based on a rule base based on a predetermined algorithm and an inference model generated by machine learning in the target object detection unit 31, thereby enabling accurate and fast detection of the position and orientation of each of irregularly shaped objects, such as pillow packages, that exist individually in different shapes among a collection of irregularly stacked objects OB. Furthermore, object detection is possible for pillow packages made of translucent films, mirrored bodies, etc.

[0082] Furthermore, by employing a parallel-link robot, the robot system 1 employs a configuration in which the position of the suction unit 11a in the X, Y, and Z directions in three-dimensional space can be easily changed by changing the angle around the fulcrum of each robot arm 13 through a simple control method of independently driving the drive means arranged at the fulcrum connection portion of the robot main body 14 of each of the multiple robot arms 13. This reduces the processing load on the robot control unit 32 related to position control of the robot hand 12, and can increase the object supply capacity by 2.5 to 3.3 times, or even four times or more, compared to picking by conventional articulated robots, thereby achieving faster picking operations.

[0083] As described above, the robot system 1 uses rule-based and machine learning object detection to enable picking of irregularly shaped objects, such as pillow packages, which have individually different shapes. Furthermore, simple control using the parallel link robot 1A can speed up the picking operation for the detected irregularly shaped objects, improving object supply capacity. As a result, object supply can be achieved that contributes to shortening the takt time of the process.

[0084] Furthermore, in the robot system 1, the rule base may be a standard set based on one or more elements selected from the following shown in the images (IM2, IM3): 1) the height corresponding to the object image portion, 2) discontinuity in the image contour of the object image portion, 3) the projected area of ​​the object image portion, 4) distinction between the front and back of the object image portion, and 5) the accuracy of detection of the object image portion.

[0085] This configuration makes it possible to realize object detection based on a rule base that is more suited to the state of irregular objects present in an aggregate, and to detect irregular objects more accurately and quickly.

[0086] Furthermore, in the robot system 1, the image includes a two-dimensional image IM2 that shows a collection of objects in a planar view, and a three-dimensional image IM3 to which height data has been added, and the target object detection unit 31 may be configured to detect a target candidate image OPI shown in the image using an inference model based on the two-dimensional image IM2, and calculate height data corresponding to the detected target candidate image OPI based on the three-dimensional image IM3, and identify the target object image OTI based on the data of the target candidate image OPI and the height data.

[0087] This configuration makes it possible to accurately and quickly detect irregular objects with individually different shapes that exist among a collection of irregularly stacked objects, and specifically realize a robot system and a control method for a robot system that enable picking of irregular objects.

[0088] Furthermore, in the robot system 1, after the robot 1A picks up the target object OT, until it transfers it to a predetermined position, the imaging means 2 may capture an image of the collection of objects OB from which the target object OT has been removed, and the target object detection unit 31 may be configured to detect the position of the new target object OT.

[0089] With this configuration, it is possible to specifically realize a robot system and a control method for a robot system that can reduce the operation wait time of a parallel link robot, which can be made faster through simple control, further speed up picking operations, and improve object supply capacity.

[0090] Although the robot system according to the embodiment has been described, the present disclosure is not limited to the above embodiment except for its essential characteristic components. For example, the present disclosure also includes various modifications that a person skilled in the art can conceive of to the embodiment, and modifications realized by arbitrarily combining the components and functions of each embodiment without departing from the spirit of the present invention. Below, modifications are described as examples of such modifications.

[0091] (1) In the robot system 1 according to the embodiment, the target object detection unit 31 may detect the orientation of the target object image OTI in the X-Y plane, which is the planar orientation in the image, and the robot 1A may correct the orientation by rotating the corresponding target object OT in the horizontal plane based on information related to the orientation of the target object image OTI. This allows the target objects OT picked up from a collection of irregularly stacked objects OB to be transferred on the transport means TM in the same orientation, facilitating subsequent processing.

[0092] (2) In the robot system 1 according to the embodiment, the target object detection unit 31 may further detect whether the upwardly facing surface of the target object OT is front or back, and the robot 1A may transfer the target object OT to a different location based on the front or back. For example, when a face-down object is picked up, the robot 1A may transfer the object to an inverting device instead of the transport unit TM.

[0093] (3) One aspect of the present disclosure is not limited to the above-described embodiment, and the following cases are also included in one aspect of the present disclosure. For example, one aspect of the present disclosure also includes a case where all or part of the control unit of a robot system is configured as a computer system including a microprocessor, recording media such as ROM and RAM, a hard disk unit, etc. The control method of each robot system achieves its function by the microprocessor operating in accordance with the computer program.

[0094] Furthermore, some or all of the functions of the robot system control method according to the embodiments may be realized by a processor such as a CPU executing a program. A non-transitory computer-readable recording medium on which a program for implementing the robot system control method is recorded may also be used. It goes without saying that the program can be distributed via a transmission medium such as the Internet.

[0095] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or time-shared by a single piece of hardware or software.

[0096] The order in which the steps are performed is merely an example for specifically explaining the present invention, and other orders may be used. Also, some of the steps may be performed simultaneously (in parallel) with other steps.

[0097] <<Supplementary Information>> The above-described embodiments each illustrate a preferred specific example of the present invention. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the embodiments, those not recited in the independent claims that represent the highest concept of the present invention are described as optional components that constitute more preferred embodiments.

[0098] The order in which the above methods are performed is merely an example for specifically explaining the present invention, and other orders may be used. Also, some of the above methods may be performed simultaneously (in parallel) with other methods.

[0099] In addition, to facilitate understanding of the invention, the scales of the components in the drawings in the above embodiments may differ from the actual scales. Furthermore, the present invention is not limited to the descriptions of the above embodiments, and can be modified as appropriate within the scope of the gist of the present invention. Furthermore, at least some of the functions of the embodiments and their modifications may be combined.

[0100] A robot system and a control method for a robot system according to one aspect of the present disclosure can be suitably used as an automated picking means for removing objects, such as finished products, intermediate products, and components, to be supplied to a process from a container or the like and transferring them to a subsequent stage in a factory production line, a logistics warehouse, or the like.

[0101] REFERENCE SIGNS LIST 1 Robot system 1A Robot 11 Suction head 12 Robot hand 13 Robot arm 14 Robot main body 2 Imaging means 21 2D imaging means 22 3D imaging means 3 Control unit 31 Object detection control unit 311 3D image acquisition unit 312 2D image acquisition unit 313 Point cloud data generation unit 314 Object detection unit 315 Inference model generation unit 316 Target object detection unit 32 Robot control unit OB Object OT Target object OBI Object image portion (object image) OPI Image portion of target object candidate (target candidate image) OTI Image portion corresponding to target object (target object image) OLI Image contour CT Container TM Transport means

Claims

1. A robot system for picking individual objects from a collection of irregularly stacked objects, comprising: an imaging means for capturing an image of the collection of objects; a target object detection unit for identifying an image portion corresponding to an object to be picked from the image of the collection of objects and detecting the position of the target object by object detection based on a rule base based on a predetermined algorithm and an inference model generated by machine learning based on the captured image; and a parallel link type robot for picking the target object from the collection of objects based on the detected position of the target object and transferring it to a predetermined position.

2. The robot system of claim 1, wherein the rule base is a criterion set based on one or more elements selected from the height corresponding to the object image portion shown in the image, discontinuity in the image contour of the object image portion, the projected area of ​​the object image portion, distinction between the front and back of the object image portion, and accuracy of detection of the object image portion.

3. The robot system of claim 1, wherein the image includes a two-dimensional image of the collection of objects viewed in a plane and a three-dimensional image to which height data has been added, and the target object detection unit detects an image portion of a target object candidate shown in the image by the inference model based on the two-dimensional image, calculates height data corresponding to the detected image portion of the target object candidate based on the three-dimensional image, and identifies the image portion corresponding to the target object based on the data of the image portion of the target object candidate and the height data.

4. The robot system according to claim 1, wherein, after the robot picks up the target object and before transferring it to the predetermined position, the imaging means captures an image of the collection of objects from which the target object has been removed, and the target object detection unit detects the position of a new target object.

5. A control method for a robot system that picks up an individual object from a collection of irregularly stacked objects, comprising the steps of: capturing an image of the collection of objects; identifying an image portion of the image of the collection of objects that corresponds to an object to be picked based on the captured image and object detection based on a rule base based on a predetermined algorithm and an inference model generated by machine learning, and detecting the position of the target object; and using a parallel link type robot to pick the target object from the collection of objects and transfer it to a predetermined position based on the detected position of the target object.

Images