Robot system and control method of the same

Abstract

To increase a speed of picking operation conducted to irregular objects having different shapes in a collection, such as a pillow package, to improve article supply capability.SOLUTION: A robot system 1 for picking individual objects OB from a collection of irregularly loaded objects OB, comprises: imaging means 2 that captures images (IM2, IM3) of the collection of the objects OB; a target object detection unit 31 that identifies an image region OTI within the image of the collection of the objects OB which corresponds to a target object OT for picking and detects a position of the target object OT on the basis of the captured image via object detection based on a rule base which is based on a prescribed algorithm, and an estimation model which is generated via machine learning; and a parallel link-type robot (1X, 32) that picks the target object OT from among the collection of the objects OB and transfers the same to a prescribed position on the basis of the detected position of the target object OT.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a robot system for picking and transferring individual objects from an aggregate of irregularly stacked objects, and a control method for the robot system, and particularly relates to a robot system for picking amorphous objects having flexibility and changing shapes, etc.

Background Art

[0002] In recent years, in a factory production line, a logistics warehouse, etc., picking has been performed to take out objects such as products, intermediate products, and members supplied to a process from a container or the like and transfer them to a subsequent process. In this picking, it is necessary to detect the positions and postures of individual objects from an aggregate of irregularly stacked objects and quickly take them out. Therefore, conventionally, picking has been performed manually.

[0003] On the other hand, for example, in Patent Document 1, a robot device is proposed that has an image processing function, detects the position and posture of each of the stacked workpieces having the same shape, and automatically picks the detected individual workpieces.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, since the conventional robot device described in Patent Document 1 is configured to pick up the detected individual workpieces by a vertically articulated robot, it is difficult to speed up the picking operation. For example, there is a problem that it is difficult to supply an object at a speed that satisfies the processing capacity of the packaging machine in the downstream process. The reason is that since the articulated robot has a structure with redundancy in which a plurality of joints connected in series operate in parallel, while the degree of freedom for position control in the picking operation is high, the computational processing becomes complicated and it is difficult to improve the processing speed.

[0006] In recent years, there has been a demand for enabling picking of irregularly shaped objects such as pillow packages that use products such as foods, confectioneries, cosmetics, and daily necessities, which have flexibility and change in shape, as objects to be packaged. In this case, it is necessary to establish a method for accurately detecting the position and orientation of each irregularly shaped object that exists with different shapes individually in an aggregate of irregularly stacked objects.

[0007] The present disclosure has been made in view of the above problems, and aims to provide a robot system and a control method for the robot system that speed up the picking operation for irregularly shaped objects with different shapes individually in an aggregate of objects such as a pillow package and improve the object supply ability.

Means for Solving the Problems

[0008] A robot system according to an aspect of the present disclosure is a robot system that picks up individual objects from an aggregate of irregularly stacked objects, and includes imaging means for imaging an image of the aggregate of the objects, and based on the imaged image, a rule base based on a predetermined algorithm and object detection based on an inference model generated by machine learning are used to identify an image portion corresponding to a picking target object from the image of the aggregate of the objects, and a target object detection unit for detecting the position of the target object. Based on the detected position of the target object, a parallel link type robot that picks up the target object from the aggregate of the objects and transfers it to a predetermined position is provided.

Effects of the Invention

[0009] According to one aspect of the present disclosure, it is possible to provide a robot system and a control method for the robot system that can speed up the picking operation for irregularly shaped objects having different shapes individually among aggregates of objects such as pillow packages and improve the object supply ability.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0011] ≪Outline of Embodiments for Carrying Out the Present Invention≫ The robot system according to the embodiment in the present disclosure is a robot system that picks individual objects from an aggregate of irregularly stacked objects, and includes imaging means for imaging an image of the aggregate of the objects, and based on the imaged image, an object detection based on a rule base according to a predetermined algorithm and an inference model generated by machine learning to identify an image portion corresponding to a picking target object from the image of the aggregate of the objects, and a target object detection unit for detecting the position of the target object, and based on the detected position of the target object, a parallel link type robot that picks the target object from the aggregate of the objects and transfers it to a predetermined position.

[0012] With such a configuration, it is possible to pick irregularly shaped objects with different shapes individually from an aggregate of objects such as a pillow package by object detection based on a rule base and machine learning, and further, by simple control utilizing a parallel link type robot, the picking operation for the detected irregularly shaped objects can be speeded up, and a robot system that improves the object supply ability can be provided, and object supply that contributes to shortening the tact time of the process can be realized.

[0013] Also, in another aspect, in the aspect described in any of the above, the rule base may be configured as a criterion set based on one or more elements selected from the height corresponding to the object image portion shown in the image, the discontinuity of the image contour of the object image portion, the projected area of the object image portion, the distinction between the front and back of the object image portion, and the accuracy of detection of the object image portion.

[0014] With such a configuration, object detection based on a rule base that is more suitable for the aspect of irregularly shaped objects existing in the aggregate can be realized, and the irregularly shaped objects can be detected more accurately and quickly.

[0015] In another aspect, in any of the aspects described above, the image includes a two-dimensional image of the aggregate of the objects in a plan view and a three-dimensional image with height data added thereto. The target object detection unit detects an image portion of a target object candidate shown in the image by the inference model based on the two-dimensional image, calculates height data corresponding to the detected image portion of the target object candidate based on the three-dimensional image, and specifies an image portion corresponding to the target object based on the data of the image portion of the target object candidate and the height data. Such a configuration may be adopted.

[0016] With such a configuration, by object detection based on rules and machine learning, it is possible to accurately and quickly detect irregularly shaped objects that exist individually with different shapes among the aggregate of objects stacked irregularly.

[0017] In another aspect, in any of the aspects described above, after the robot picks up the target object and before transferring it to the predetermined position, the imaging means captures an image of the aggregate of objects from which the target object has been removed, and the target object detection unit may be configured to detect the position of a new target object.

[0018] With such a configuration, it is possible to reduce the operation waiting time of a parallel link type robot that can be speeded up by simple control, further speed up the picking operation, and improve the object supply ability.

[0019] In another aspect, a control method for a robot system according to an embodiment is a control method for a robot system that picks individual objects from an aggregate of irregularly stacked objects. The method includes imaging an image of the aggregate of the objects, and based on the captured image, identifying an image portion corresponding to a pick target object from the image of the aggregate of the objects by object detection based on a rule base according to a predetermined algorithm and an inference model generated by machine learning, detecting the position of the target object, and based on the detected position of the target object, picking the target object from the aggregate of the objects by a parallel link type robot and transferring it to a predetermined position.

[0020] With such a configuration, it is possible to realize a control method for a robot system that speeds up the picking operation for irregular objects and improves the object supply ability.

[0021] ≪Embodiment≫ The configuration of the robot system 1 according to the embodiment will be described with reference to the drawings. Here, in this specification, the X direction, Y direction, and Z direction in each figure may be the width direction, depth direction, and height direction, respectively, and the positive direction of the height direction may be the "up" direction and the negative direction may be the "down" direction.

[0022] Also, the scales of the members in each drawing are not necessarily the same as the actual ones. Also, for ease of understanding, illustrations such as covers may be omitted. Also, for example, some members, transport paths, etc. may be illustrated schematically or approximately. Also, in this specification, the symbol "~" used to indicate a numerical range includes the numerical values at both ends. Also, the materials, numerical values, etc. described in this embodiment are merely examples of preferred ones and are not limited thereto. Also, appropriate changes are possible without departing from the scope of the technical idea of the present disclosure. Also, combinations of some of the configurations with those of other embodiments are possible as long as there is no contradiction.

[0023] <Regarding the configuration of the robot system 1> The configuration of the robot system 1 will be described with reference to the drawings. FIG. 1 is a perspective view schematically showing the configuration of the robot system 1 according to the embodiment.

[0024] (Overall configuration) The robot system 1 is a robot system that picks up an object OB such as a product, an intermediate product, or a member supplied to a process from a container CT used for supply, for example, in a manufacturing process or the like, and transfers it to a predetermined position in the process, for example, to a transfer means TM.

[0025] As shown in FIG. 1, the robot system 1 includes a parallel link type robot 1A that functions as a transfer means for transferring the object OB, an imaging means 2 that captures an image of an aggregate of objects OB stacked on the container CT, and a control unit 3 that controls the robot 1A based on the image.

[0026] (Outline of the configuration of each part) Hereinafter, an outline of the configuration of each part in the robot system 1 will be described. The object OB to be picked up by the robot system 1 is composed of, for example, products such as finished products, work-in-process products, intermediate products, members, and packages, etc., supplied to the process in a production line or a logistics warehouse. In addition to regular-shaped objects with a fixed shape such as rigid bodies, the object OB includes, for example, pillow packages and other individually shaped irregular objects that have flexibility and a changing shape, such as foods, confectioneries, cosmetics, and liquid daily necessities, with the flexible and shape-changing objects being the objects to be packaged.

[0027] The pillow package has a package body trunk OBd in which an object to be packaged and gas are enclosed in a packaging member made of a resin film, and end seal portions OBe at both ends thereof. For example, the object to be packaged is enclosed in a cylindrical packaging member together with gas, and the front and back of the object to be packaged are sealed to form it. The enclosed gas has appropriate cushioning properties and ease of deformation.

[0028] In the object OB, the resin film of the pillow package may be composed of a film member having translucency and a mirror-like outer surface. Further, information or a pattern may be printed on the front surface and / or the back surface of the pillow package, or a printed label may be attached thereto.

[0029] As shown in FIG. 1, these objects OB are generally supplied to the process in a state of an aggregate of objects OB irregularly stacked in a mountain shape on the bottom surface of a container CT used as a mailing, for example. In such an aggregate of objects OB, when the object OB is an amorphous object such as a pillow package, each object OB exists in an irregularly stacked state in the container CT with not only the position and posture but also the respective shapes being individually different.

[0030] The conveying means TM is a conveyor for loading and conveying (T 1 ) the supplied object OB into the process. Examples of the conveying means TM include various conveyors.

[0031] The robot 1A is a robot mechanism including a parallel link type robot arm mechanism that functions as a manipulator for taking out the object OB from the container CT and transferring it to the conveying means TM. As shown in FIG. 1, the robot 1A includes a suction head 11, a robot hand 12, a robot arm 13, and a robot body portion 14.

[0032] The suction head 11 is supported at the lower end of the robot hand 12, includes a suction portion 11a on the lower surface side, and has a function of sucking and holding the object OB from above. The suction portion 11a is provided with suction holes (not shown) on the lower surface. The suction holes are connected to a vacuum pump or the like (not shown). The object OB can be sucked and held by applying a negative pressure through the suction holes in a state where the lower surface of the suction portion 11a is in contact with the upper surface of the object OB stacked in the container CT.

[0033] The robot hand 12 is supported at the tip of the robot arm 13 and rotatably supports the suction head 11 in the X-Y plane, enabling the orientation of the suction head 11 to be changed.

[0034] Note that the suction head 11 and / or the robot hand 12 may have a structure that allows them to move in the Z direction. By adopting such a structure, even if the suction head 11 and the object OB come into strong contact when the object OB is suction-held, it is possible to suppress problems such as the object OB being torn.

[0035] The robot arm 13 is a so-called parallel-link type robot. A plurality of arms (three in this example) extend radially in the X-Y plane direction and are supported, for example, by a robot main body 14 installed on the ceiling of the facility or the like. The robot arm 13 rotatably supports the suction head 11 in the X-Y plane via the robot hand 12. For example, with the suction part 11a facing downward, the suction head 11 can be moved to a predetermined position in three-dimensional space.

[0036] In a parallel-link type robot, each of the plurality of robot arms 13 includes a driving means (e.g., a motor) at a fulcrum portion that is rotatably joined to the robot main body 14. With such a configuration, in a parallel-link type robot, based on a control signal issued from the control unit 3, by independently driving the driving means for each robot arm 13 using a simple control method, the angle around the fulcrum of each robot arm 13 can be changed, enabling the position of the suction part 11a in the XYZ directions in three-dimensional space to be changed at high speed.

[0037] The imaging means 2 is a camera using, for example, a CCD (Charge Coupled Device) image sensor that captures an image of a collection of objects OB irregularly stacked in the container CT. The imaging means 2 includes 2D imaging means 21 that captures a two-dimensional image of the collection of objects OB in a plan view from vertically above the bottom surface of the container CT. Further, the imaging means 2 includes 3D imaging means 22 that generates a three-dimensional image. The 3D imaging means 22 may be configured to capture a plurality of images of the collection of objects OB from different positions obliquely above and collate them to calculate the depth of the image portion in the plane direction. Alternatively, the 3D imaging means 22 may use a ToF (Time-of-Flight) camera that generates a three-dimensional image by measuring the time from when the collection of objects OB is irradiated with infrared light until the reflected light is received. Alternatively, a pattern projection method such as stripes or random dots may be used.

[0038] The control unit 3 is electrically connected to the imaging means 2 and the robot 1A, and outputs a control signal to the robot 1A based on the image captured by the imaging means 2 to control the operation of each unit. The control unit 3 is realized, for example, as a computer including a general CPU (Central Processing Unit), a RAM (Random Access Memory), and programs executed by these. The control unit 3 realizes the functions related to the control method of the robot system 1 by reading the control program related to the robot system 1 from a storage device or the like into the RAM and executing it.

[0039] FIG. 2 is a functional block diagram showing an overview of the functional configuration of the robot system 1. FIGS. 3(a) and 3(b) are front views for explaining an overview of the operation of the robot system 1. As shown in FIG. 2, the control unit 3 includes an object detection unit 31 and a robot control unit 32. Among these, the object detection unit 31 detects the position and orientation of the object image portion corresponding to the object OB irregularly stacked on the container CT based on the image captured by the imaging means 2, and as shown in FIG. 3, identifies the image portion corresponding to the picking target object OT, and outputs information regarding the position and orientation of the corresponding target object OT to the robot control unit 32.

[0040] Further, the robot control unit 32 controls so that each unit constituting the robot 1A operates in association with each other. Specifically, the robot control unit 32 issues a control signal to control the driving means for each of the robot arms 13, and changes the angles (θ 1 , θ 2 , θ 3 ) around the fulcrum of each robot arm 13, thereby changing the position of the suction unit 11a in the XYZ directions in the three-dimensional space, and moving the suction unit 11a of the suction head 11 to the position of the upper surface of the target object OT in the three-dimensional space. Also, the robot control unit 32 issues a control signal to control the suction from and the release of the suction unit 11a (P ON / OFF ). Furthermore, the robot control unit 32 controls the orientation (θ XY ) of the suction head 11 in the X-Y plane.

[0041] Thereby, as shown in FIG. 3(a), the target object OT for picking is sucked (P ON ) and held and taken out (M1) from the aggregate of the objects OB irregularly stacked on the container CT related to the loading, and after being transferred (M 2 ) to a predetermined position as shown in FIG. 3(b), the negative pressure is released (P OFF ) to transfer the target object OT to the conveying means TM. Then, by selecting another object OB as a new target object OT and repeating the above operation, each of the objects OB is taken out from the aggregate of the objects OB irregularly stacked and transferred to the conveying means TM.

[0042] (Function of the target object detection unit 31) Next, the details of the function of the target object detection unit 31 will be described. FIG. 4 is a functional block diagram showing the configuration of the target object detection unit 31. As shown in FIG. 4, the target object detection unit 31 includes a 3D image acquisition unit 311, a 2D image acquisition unit 312, a point cloud data generation unit 313, an object detection unit 314, an inference model generation unit 315, a target object identification unit 316, and a rule-based storage unit 317.

[0043] The 3D image acquisition unit 311 is a circuit that acquires a three-dimensional image IM3 of the aggregate of objects OB loaded on the container CT from the 3D imaging means 22 and outputs it to the subsequent stage.

[0044] The 2D image acquisition unit 312 is a circuit that acquires a two-dimensional image IM2 of the aggregate of objects OB in a plan view from the 2D imaging means 21 and outputs it to the subsequent stage. For the 3D image acquisition unit 311 and the 2D image acquisition unit 312, for example, a device for capturing image data into a processing device such as a computer, such as an image capture board, can be used.

[0045] The point cloud data generation unit 313 is a processing unit that generates point cloud data, which is three-dimensional voxel data of the aggregate of objects OB, based on the three-dimensional image IM3 of the aggregate of objects OB. The generated point cloud data is output to the target object identification unit 316.

[0046] The object detection unit 314 is a processing unit that detects image portions of a plurality of objects shown in the 2D image IM2 by using an inference model generated in advance by machine learning based on the 2D image IM2. FIGS. 5(a), (b), and (c) are schematic diagrams showing the modes of 2D images related to object detection and explaining the operation of the object detection unit.

[0047] As shown in FIG. 5(a), the object detection unit 314 detects, for example, the image contour OLI of an object image portion OBI (hereinafter sometimes referred to as "object image OBI") corresponding to a plurality of objects OB shown in the 2D image IM2, thereby detecting the position and orientation in the X-Y plane of the object image OBI.

[0048] At this time, the object detection unit 314 may select, as an image portion OPI of a target object candidate for picking (the shaded portion in Fig. 5(a), hereinafter sometimes referred to as "target candidate image OPI"), an object image OBI in which there is no discontinuous portion in the image contour OLI, considering the overlap of the objects OB, based on the discontinuity of the image contour OLI of the object image OBI. The reason is based on the inventor's finding that an object OB in which there is no discontinuous portion in the image contour OLI of the object image OBI is located at the uppermost layer of an aggregate of a plurality of objects OB arranged in an irregularly overlapping state and is easy to pick.

[0049] Further, the object detection unit 314 may select, from the target candidate images OPI for picking, an object image OBI having a large image area (the portion with grid shading in Fig. 5(a)) based on the object projection area in the X-Y plane. The reason is based on the inventor's finding that an object OB with a large area of the object image OBI is an object with less inclination among an aggregate of objects OB stacked irregularly and is easy to pick.

[0050] Also, as shown in Fig. 5(b), in the detection of the image contour OLI of the object image portion, when the object is a package made of a translucent member and the contents PR can be seen through from the outside, the outer edge portion of the image of the package may be detected as the image contour OLI of the object image OBI. The same applies when the object is a mirror surface body.

[0051] Also, as shown in Fig. 5(c), when a label LB is attached to the front or back surface of the outer surface of the object, the orientation of the object image OBI in the X-Y plane may be detected or the front and back of the object may be distinguished depending on the position of the label LB in the object image OBI.

[0052] These conditions can be reflected in the object detection by the inference model by providing the two-dimensional image IM2 including the above conditions as teacher data to the inference model generation unit 315 in the construction of the inference model by machine learning to be described later.

[0053] The object detection unit 314 outputs the detected picking target candidate image OPI and information regarding the position of the target candidate image OPI within the X-Y plane to the target object identification unit 316. Further, information regarding the orientation of the target candidate image OPI within the X-Y plane and information regarding the front / back discrimination may be output.

[0054] The inference model generation unit 315 is a processing unit that generates an inference model in the learning phase and provides the inference model to the object detection unit 314. According to the inference model generation unit 315, a two-dimensional image obtained by viewing a collection of objects OB acquired in advance by the 2D image acquisition unit 312 in a plan view and the result of the operator annotating the image for object detection are provided as teacher data for machine learning, thereby enabling the generation of an inference model related to object detection.

[0055] For the machine learning in the inference model generation unit 315, for example, deep learning (Deep Learning) using a neural network (CNN: Convolutional Neural Network) can be used, and known software can be utilized. Also, in the operation phase, the inference model generation unit 315 may be excluded from the implemented functions of the target object detection unit 31.

[0056] The target object identification unit 316 is a circuit that takes as input the position information of the picking target candidate image OPI within the X-Y plane acquired from the object detection unit 314 and the point cloud data, which is the three-dimensional voxel data of the collection of objects OB acquired from the point cloud data generation unit 313, and identifies the image portion OBI corresponding to the picking target object OT from among the target candidate images OPI based on a rule base.

[0057] The rule base criteria are stored in the rule base storage unit 317 and are selectively supplied from the rule base storage unit 317 to the target object identification unit 316 as appropriate based on operation inputs and the like. The rule base to be applied may be varied according to the type of the object OB to be picked.

[0058] For example, the target object specifying unit 316 may calculate height data corresponding to one or more target candidate images OPI detected by the object detection unit 314, and specify an image portion corresponding to the target object OT from among the target candidate images OPI based on the magnitude of the value of the height data as a rule base. In this case, referring to the height data of the point cloud data whose position coordinates in the X-Y plane are equivalent to those of the target candidate image OPI, the target candidate image OPI with the largest height data can be selected as the image portion corresponding to the target object OT.

[0059] FIG. 6 is a diagram for explaining the processing in the target object specifying unit 316, and is a schematic diagram showing the target candidate image OPI and the point cloud data PD whose position coordinates in the X-Y plane are equivalent to those of the target candidate image OPI, among the point cloud data of the aggregate of the objects OB, in a superimposed and three-dimensional manner. In the example shown in FIG. 6, among the six target candidate images OPI detected from a plurality of object images OBI, the object (displayed with a thick line image contour in FIG. 6) having the largest value of the height data of the point cloud data PD included in the target candidate image OPI is selected as the image portion OTI (hereinafter may be referred to as "target object image OTI") corresponding to the target object OT for picking. The comparison of the height data between the target candidate images OPI may be performed based on the average value, the minimum value, the difference between the maximum and the minimum, or a combination thereof of the height data of the included point cloud data PD.

[0060] The reason for selecting the target candidate image OPI with the largest value of the height data as the target object image OTI is that the corresponding target object OT is located at the uppermost layer of the aggregate of a plurality of objects OB arranged in an irregularly overlapping state and is considered to be easy to pick.

[0061] As a rule base criterion for specifying the target object image OTI, in addition to the height data, the same criteria as those for selecting the target candidate image OPI in the object detection unit 314 may be used.

[0062] That is, based on the discontinuity of the image contour OLI, a target candidate image OPI without a discontinuous part in the image contour OLI may be selected as the target object image OTI for picking.

[0063] Alternatively, based on the object projected area, a target candidate image OPI with a large area in the XY plane of the target candidate image OPI may be selected as the target object image OTI.

[0064] Also, depending on the position of the label LB in the target candidate image OPI, the orientation of the target candidate image OPI in the X-Y plane may be detected, or the front and back may be distinguished.

[0065] Also, among the target candidate images OPI detected by the object detection unit 314, those with a high detection accuracy may be selected as the target object image OTI.

[0066] As described above, these criteria are the criteria used for detecting the target candidate image OPI in the inference model by machine learning in the object detection unit 314. However, in specifying the target object image OTI in the target object specifying unit 316, by applying these criteria again as rule-based criteria, among the aggregates of irregularly stacked objects, it is possible to realize object detection that is more suitable for the modes of amorphous objects that exist with different shapes individually, and to detect the positions and postures of each amorphous object with higher accuracy.

[0067] The target object specifying unit 316 outputs information (for example, position (X, Y, Z), posture θ X-Y , type of front and back, etc.) regarding the position and posture of the target object OT corresponding to the detected target object image OTI to the robot control unit 32.

[0068] <Operation of the robot system 1> Hereinafter, the operation of the robot system 1 having the above configuration will be described. FIG. 7 is a flowchart showing an overview of the object picking and transfer operation in the robot system 1. As shown in FIG. 7, first, in the robot system 1, the 2D imaging means 21 acquires a two-dimensional image IM2 of the collection of objects OB irregularly stacked in the container CT in a plan view (step S11). The target object detection unit 31 detects a plurality of object images OBI shown in the two-dimensional image IM2 by an inference model based on machine learning, and outputs the target candidate image OPI for picking and information regarding the position of the target candidate image OPI in the X-Y plane.

[0069] In parallel with the above processing, the 3D imaging means 22 acquires a three-dimensional image IM3 of the collection of objects OB (step S12), and the target object detection unit 31 generates and outputs point cloud data PD which is three-dimensional voxel data of the collection of objects OB (step S22).

[0070] Next, in step S3, the target object detection unit 31 inputs the position information of the target candidate image OPI for picking in the X-Y plane and the point cloud data PD of the collection of objects OB, and based on a rule-based approach, identifies the target object image OTI for picking from among the target candidate images OPI, and outputs information regarding the position and orientation of the target object OT to the robot control unit 32. At this time, the rule-based approach applied may be a criterion set based on one or more elements selected from 1) the height corresponding to the target candidate image OPI, 2) the discontinuity of the image contour of the target candidate image OPI, 3) the projected area of the target candidate image OPI in the X-Y plane, 4) the distinction between the front and back of the target candidate image OPI, and 5) the accuracy of detection of the object image OBI.

[0071] FIG. 8 is a flowchart showing an example of the object detection process based on the rule base in step S3. As shown in FIG. 8, regarding the target candidate image OPI shown in the image, height determination of the object (step S31), determination of the discontinuity of the object image contour (step S32), and determination of the projected area of the object image (step S33) are sequentially performed. In step S34, these results may be comprehensively evaluated to identify the object image OTI to be picked.

[0072] On the other hand, the robot control unit 32 performs the following processes in parallel with the processes from step S11 to step S3 in the object detection unit 31.

[0073] First, it is determined whether or not the current process is the first detection process (step S4). If it is the first detection, the process proceeds to step S6. On the other hand, if it is not the first detection, since the object OB is sucked and held by the suction head 11 by the previously performed picking, after moving and transferring the object OB to a predetermined position (step S5), the process proceeds to step S6.

[0074] Next, in step S6, it is determined whether or not to end the detection process. If the detection process is not ended, in step S7, a picking operation of the target object OT is performed.

[0075] Specifically, based on the information regarding the position and posture of the target object OT, the robot control unit 32 changes the angles (θ 1 , θ 2 , θ 3 ) around the fulcrums of the respective robot arms 13 of the parallel link type robot 1A, and moves the suction part 11a of the suction head 11 to the position in the three-dimensional space on the upper surface of the target object OT. Then, a picking operation is performed to suck and hold and take out the target object OT from the aggregate of irregularly stacked objects.

[0076] Thereafter, the process returns to steps S11, S12, and S4. In the next iteration, the object OB that has already completed the picking operation and is being held by suction on the suction unit 11a is moved to a predetermined position and transferred to the conveying means TM (step S5). On the other hand, in the determination of step S6, if the detection process is to be terminated, the process is terminated.

[0077] Through the above operations, according to the robot system 1, from the aggregate of objects OB stacked irregularly, irregularly shaped objects with different individual shapes within the aggregate, such as pillow packages, are detected, and a simple control utilizing the parallel link type robot 1A enables the realization of a high-speed picking operation.

[0078] <Effect> As described above, the robot system 1 is a robot system that picks individual objects OB from an aggregate of objects OB stacked irregularly, and includes an imaging means 2 that captures images (IM2, IM3) of the aggregate of objects OB, and based on the captured images (IM2, IM3), a rule base based on a predetermined algorithm and object detection based on an inference model generated by machine learning are used to identify the target object image OTI for picking from the images (IM2, IM3) of the aggregate of objects OB, and a target object detection unit 31 that detects the position of the target object OT, and based on the detected position of the target object OT, a parallel link type robot (1X, 32) that picks the target object OT from the aggregate of objects OB and transfers it to a predetermined position.

[0079] As described above, the conventional picking robot device performs the picking operation with an articulated robot that has a high degree of freedom in position control in the picking operation for the detected individual objects, but on the other hand, has a complex calculation process and is difficult to improve the processing speed. Therefore, it is difficult to increase the speed of the picking operation. For example, it is difficult to supply objects at a speed that satisfies the processing capacity of the packaging machine in the downstream process, which has been one of the factors increasing the tact time in the packaging process.

[0080] In addition, for picking up an amorphous object such as a pillow packaging body that uses a product or the like having flexibility and a changing shape as an object to be packaged, it was necessary to establish a method for accurately and quickly detecting the position and orientation of each amorphous object that exists with different shapes individually among an aggregate of irregularly stacked objects.

[0081] On the other hand, according to the robot system 1, in the target object detection unit 31, by adopting rule-based based on a predetermined algorithm and object detection based on an inference model generated by machine learning, from an aggregate of irregularly stacked objects OB, for a pillow packaging body or the like, the position and orientation of each amorphous object that exists with different shapes individually among the aggregate of objects OB can be detected accurately and quickly. In addition, it is possible to enable object detection for a pillow packaging body made of a translucent film, a mirror surface body, or the like.

[0082] In the robot system 1, by adopting a parallel link type robot, a simple control method of independently driving the robot main body part 14 of each of the plurality of robot arms 13 and the driving means arranged at the fulcrum connection part is used to change the angle around the fulcrum of each robot arm 13, and a configuration that can easily change the position of the suction part 11a in the XYZ directions in the three-dimensional space is adopted. Therefore, the processing load of the robot control unit 32 related to the position control of the robot hand 12 can be reduced, and compared with the picking by a conventional articulated robot, the object supply ability can be increased by 2.5 to 3.3 times, and further more than 4 times, and the high speed of the picking operation can be realized.

[0083] As described above, according to the robot system 1, object detection by rule-based and machine learning enables picking of amorphous objects having different shapes individually such as a pillow packaging body. Furthermore, by simple control utilizing the parallel link type robot 1A, the picking operation for the detected amorphous object can be speeded up, and the object supply ability can be improved. As a result, it is possible to realize object supply that contributes to shortening the tact time of the process.

[0084] Also, in the robot system 1, the rule base may be a criterion set based on one or more elements selected from 1) the height corresponding to the object image portion, 2) the discontinuity of the image contour of the object image portion, 3) the projected area of the object image portion, 4) the distinction between the front and back of the object image portion, and 5) the accuracy of detection of the object image portion, as shown in the images (IM2, IM3).

[0085] With such a configuration, object detection based on a rule base more suitable for the form of the amorphous object existing in the aggregate can be realized, and the amorphous object can be detected more accurately and at high speed.

[0086] Also, in the robot system 1, the images include a two-dimensional image IM2 of the object aggregate in a plan view and a three-dimensional image IM3 with height data added. The target object detection unit 31 detects a target candidate image OPI shown in the image by an inference model based on the two-dimensional image IM2, calculates height data corresponding to the detected target candidate image OPI based on the three-dimensional image IM3, and specifies a target object image OTI based on the data of the target candidate image OPI and the height data. This may be the configuration.

[0087] With such a configuration, it is possible to accurately and quickly detect amorphous objects with different shapes individually in an aggregate of irregularly stacked objects, and a robot system and a control method of the robot system that enable picking of amorphous objects can be specifically realized.

[0088] Also, in the robot system 1, after the robot 1A picks up the target object OT, until it is transferred to a predetermined position, the imaging means 2 images the aggregate of the objects OB from which the target object OT has been removed, and the target object detection unit 31 may be configured to detect the position of a new target object OT.

[0089] With such a configuration, it is possible to specifically realize a robot system and a control method for a robot system that can reduce the operation standby time of a parallel link type robot that can be speeded up by simple control, further speed up the picking operation, and improve the object supply ability.

[0090] ≪Modification Example≫ Although the robot system according to the embodiment has been described, the present disclosure is not limited to the above embodiments except for its essential characteristic components. For example, forms obtained by applying various modifications conceivable by those skilled in the art to the embodiments, and forms realized by arbitrarily combining the components and functions in each embodiment without departing from the gist of the present invention are also included in the present disclosure. Hereinafter, a modification example will be described as an example of such a form.

[0091] (1) In the robot system 1 according to the embodiment, the target object detection unit 31 may detect the orientation of the target object image OTI in the X-Y plane in the planar direction in the image, and the robot 1A may correct the orientation by rotating the corresponding target object OT in the horizontal plane based on the information related to the orientation of the target object image OTI. Thereby, the target objects OT taken out from the aggregate of the irregularly stacked objects OB can be arranged and transferred in the same orientation on the transfer means TM, and the subsequent processing can be facilitated.

[0092] (2) In the robot system 1 according to the embodiment, further, the target object detection unit 31 may detect the front and back of the surface facing upward of the target object OT, and the robot 1A may be configured to transfer the target object OT to different locations based on the front and back. For example, when picking up a backward object, it may be configured to transfer it to an inversion device instead of the transfer means TM.

[0093] (3) One aspect of the present disclosure is not limited to the above embodiments, and the following cases are also included in one aspect of the present disclosure. For example, one aspect of the present disclosure also includes a case where all or part of the control unit of the robot system is configured by a computer system including a recording medium such as a microprocessor, ROM, RAM, a hard disk unit, and the like. When the microprocessor operates according to the computer program, the control method of each robot system achieves its function.

[0094] Further, part or all of the functions of the control method of the robot system according to the embodiment may be realized by a processor such as a CPU executing a program. It may be a non-transitory computer-readable recording medium on which a program for implementing the control method of the robot system is recorded. Needless to say, the above program can be distributed via a transmission medium such as the Internet.

[0095] Also, the division of the functional blocks in the block diagram is an example, and a plurality of functional blocks may be realized as one functional block, one functional block may be divided into a plurality, or part of the functions may be transferred to other functional blocks. Further, the functions of a plurality of functional blocks having similar functions may be processed by a single hardware or software in parallel or time-division.

[0096] Also, the order in which the above steps are executed is for illustration in order to specifically describe the present invention, and other orders may be used. Also, some of the above steps may be executed simultaneously (in parallel) with other steps.

[0097] <<Supplementary Note>> The embodiments described above all show preferred specific examples of the present invention. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, processes, the order of the processes, etc. shown in the embodiments are examples and are not intended to limit the present invention. Also, among the components in the embodiments, those not described in the independent claims indicating the highest concept of the present invention are described as optional components constituting a more preferred form.

[0098] Also, the order in which the above method is executed is for illustrative purposes to specifically describe the present invention, and other orders may be used. Further, a part of the above method may be executed simultaneously (in parallel) with other methods.

[0099] Also, for ease of understanding of the invention, the scales of the components in each figure given in the above embodiments may be different from the actual ones. Further, the present invention is not limited by the descriptions of the above embodiments and can be appropriately changed without departing from the gist of the present invention. Also, at least a part of the functions of each embodiment and its modification may be combined.

Industrial Applicability

[0100] The robot system and the control method of the robot system according to one aspect of the present disclosure can be suitably used as an automated means for picking, in a factory production line, a logistics warehouse, etc., to take out an object such as a product, an intermediate product, a member, etc. supplied to a process from a container or the like and transfer it to the subsequent stage.

Explanation of Signs

[0101] 1 Robot system 1A Robot 11 Suction head 12 Robot hand 13 Robot arm 14 Robot main body 2 Imaging means 21 2D imaging means 22 3D imaging means 3 Control unit 31 Object detection control unit 311 3D image acquisition unit 312 2D image acquisition unit 313 Point cloud data generation unit 314 Object detection unit 315 Inference model generation unit 316 Target object detection unit 32 Robot control unit OB Object OT target object OBI object image part (object image) OPI image part of target object candidate (target candidate image) OTI image part corresponding to the target object (target object image) OLI image contour CT container TM conveying means

Claims

1. A robot system for picking individual objects from an aggregate of irregularly stacked objects, comprising: imaging means for imaging an image of the aggregate of the objects; an object detection unit that, based on the captured image, identifies an image portion corresponding to a target object for picking from the image of the aggregate of the objects by object detection based on a rule base based on a predetermined algorithm and an inference model generated by machine learning, and detects the position of the target object; a parallel link type robot that picks the target object from the aggregate of the objects based on the detected position of the target object and transfers it to a predetermined position. The robot system.

2. The rule base is a criterion set based on one or more elements selected from the height corresponding to the object image portion shown in the image, the discontinuity of the image contour of the object image portion, the projected area of the object image portion, the distinction between the front and back of the object image portion, and the accuracy of detection of the object image portion. The robot system according to claim 1.

3. The image includes a two-dimensional image of the aggregate of the objects in plan view and a three-dimensional image to which height data is added. The object detection unit detects an image portion of a target object candidate shown in the image by the inference model based on the two-dimensional image, and calculates height data corresponding to the detected image portion of the target object candidate based on the three-dimensional image, and identifies the image portion corresponding to the target object based on the data of the image portion of the target object candidate and the height data. The robot system according to claim 1.

4. After the robot picks the target object and before transferring it to the predetermined position, the imaging means images an image of the aggregate of the objects from which the target object has been removed, and the object detection unit detects the position of a new target object. The robot system according to claim 1.

5. A control method for a robot system for picking individual objects from an aggregate of irregularly stacked objects, comprising: imaging an image of the aggregate of the objects; based on the captured image, identifying an image portion corresponding to a target object for picking from the image of the aggregate of the objects by object detection based on a rule base based on a predetermined algorithm and an inference model generated by machine learning, and detecting the position of the target object; Based on the detected position of the target object, a parallel-link type robot picks up the target object from the aggregate of objects and transfers it to a predetermined position. A control method for a robot system.

Images