Method for picking on a sorting line and associated facility

The method improves the efficiency and reliability of item handling in sorting lines by using image processing to calculate a pickability score for each item, allowing for real-time adaptation of handling priorities, thus addressing the challenge of balancing speed and quality in 'pick and place' operations.

WO2024246318A1PCT designated stage expired Publication Date: 2024-12-05SOLYSTIC

Patent Information

Application Number
PCT/EP2024/065081
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2024-05-31
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Sorting lines face challenges in maximizing the speed of 'pick and place' operations while minimizing handling errors, particularly in handling items of varying sizes, shapes, and orientations, which requires a compromise between throughput and quality of entry.

Method used

A method using a robotic manipulator arm that acquires digital color and depth images to calculate a 'pickability score' for each item, weighing factors like location and gripping quality to determine the optimal item for handling, allowing for real-time adaptation of handling priorities based on operational objectives.

Benefits of technology

This approach enhances the efficiency and reliability of item handling by prioritizing either throughput or quality, reducing errors and collisions, and optimizing the handling process based on item characteristics and operational goals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2024065081_05122024_PF_FP_ABST
    Figure EP2024065081_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a method (100) for handling items by means of an individual handling device, the method comprising the steps of acquiring (S10) a digital colour image (NumCol) of a scene comprising items to be handled in a handling area and a pair of digital images (NumDepth) to form a depth map of the scene; identifying (S20) items (VisIT) on the basis of the colour digital image; calculating (S50), for identified visible items (VisIT), a pickability score (SCOR) on the basis of a combination of a first score representative of the location of the relevant item in the scene and at least one second score representative of a graspability quality of the relevant item, each of the first score and of the at least one second score being assigned a respective weighting coefficient; and, in response to the pickability score (SCOR), controlling (S60) the device for individually handling the items.
Need to check novelty before this filing date? Find Prior Art

Description

PICKING PROCESS IN A SORTING LINE AND ASSOCIATED INSTALLATION TECHNICAL FIELD OF THE INVENTION

[0001] The invention relates to a method of handling articles in circulation on an article sorting line, with the aim of generally increasing the quality or speed of handling of these articles. TECHNOLOGICAL BACKGROUND

[0002] The sorting of items such as postal parcels is carried out within a sorting center comprising sorting lines intended to carry out automated sorting of items, each handling a flow of items. The items are handled along the lines by means of collective conveyor systems but require, for example for individual picking or placement, to be picked individually during an operation known as "pick and place", or "pick and place" in English terminology, implemented by means of a robotic manipulator arm, as described for example in document FR 3 019 068 B1. An example of a manipulator arm using suction cups as a device for gripping items is described in document FR 3 031 046 A1. Of course, in order to make sorting lines profitable, they must handle as large a flow of items as possible and the speed of execution of the "pick and place" operation is essential.

[0003] The function of the manipulator arm is to grasp the item, move it to a predetermined location, and then release it. This operation must be rapid yet reliable in order to limit handling errors. An example of handling error is "fly-out," which is an unexpected release of the item while it is being moved at high speed by the arm and can result in the item being ejected from the sorting line.

[0004] Document WO2021198053A1 discloses a method for sorting bulk objects with individual capture of the objects based on scores indicative of characteristic parameters of the objects (distances, textures, shapes, inclinations, height, etc.).

[0005] There is therefore a need to maximize the speed of "pick and place" operations while limiting handling errors, in other words, it is necessary to find a compromise between the speed of operations and the quality of inputs, in a way that allows this compromise to be adapted to the objectives of the sorting line operator, which may vary with the nature of the items to be sorted.

[0006] Items to be sorted moving along a sorting line may arrive at an automated pick and place handling area without any predefined order or location, and items may even be stacked on top of each other.

[0007] Optimizing the individual handling of items, taking into account the quality of capture of these items and their processing flow rate, requires, in order to control a device for individual handling of items, taking into account the positions of each of the items in the handling area as well as the relative positions of the items in relation to each other, and possibly the characteristics of the handling device.

[0008] An aim of the invention is to provide a method for individual handling of articles allowing optimization of the handling of these articles considered as a whole, integrating the determination of an order of handling of the articles, and more specifically the determination of the next article to be handled in a sorting scene.

[0009] To achieve this aim, a first aspect of the invention is a method for handling articles by means of a device for individual handling of the articles, comprising the steps of acquiring a digital color image of a scene comprising articles to be handled in a handling area and a pair of digital images provided to form a depth map of said scene; on the basis of the digital color image, identifying visible articles by means of a digital computer; on the basis of the pair of digital images, forming a depth map of the scene, and associating the identified visible articles with elements of the depth map; on the basis of the digital color image and the depth map, calculating, for at least some of the identified visible articles,a pickability score based on a combination of a first score representative of the location of the item considered in the scene and at least one second score representative of a quality of gripping of the item considered, each of the first score and the at least one second score being assigned a respective weighting coefficient; and in response to the pickability score, controlling the individual item handling device so as to handle one of the visible items, method in which the pickability score is determined on the basis of a combination of the first score representative of the location of the item considered in the scene and at least two second scores each representative of a quality of gripping of the item considered,in which the usability score is calculated by a combination of the first score assigned its first weighting coefficient and a second overall score representative of a combination between the at least two second scores each assigned their respective weighting coefficient, the second overall score being assigned a second overall weighting coefficient, so as to promote either a flow rate of the articles, or a quality of entry of the articles, the method further comprising a preliminary step of adjusting the respective weighting coefficients of the first score and the second overall score, so as to promote either a flow rate of the articles by increasing the first weighting coefficient relative to the second overall weighting coefficient, or a quality of entry of each of the visible articles identified by increasing the second overall weighting coefficient relative to the first weighting coefficient.,

[0010] This method allows the use of a plurality of selection criteria in order to determine the next item to be handled by an individual item handling device on an item sorting line. These criteria give the sorting line operator a very simple way to adapt the control of the handling device to his operational objectives, which can lead to favoring either the handling speed (the time required to handle an item), or the quality of this handling, by limiting the risks of dropping items and collisions between a picked item and the other items in the work area, or even a compromise between the two positions, depending on the type of items to be sorted (shape, nature, weight, etc.).

[0011] According to additional non-limiting characteristics of the first aspect of the invention, considered individually or in any technically feasible combination:

[0012] - the combination may be a weighted sum of the first score and at least one second score assigned their respective weighting coefficients;

[0013] - the combination between the at least two second scores may be a weighted sum of the at least two second scores each affected by their respective weighting coefficient;

[0014] - the method may further comprise a step of determining the usability of the identified visible articles, the articles determined as usable constituting said at least some of the identified visible articles;

[0015] - the pickability of the identified visible articles may be based on at least one criterion chosen from criteria respectively representative of dimensions, a volume, a height and a slope of an article considered, an accessibility of the article considered by the individual handling device of the articles, and the possibility of attributing a gripper configuration to the article considered;

[0016] - the device for individual manipulation of the articles may be a robotic manipulator arm comprising a head provided with a plurality of suction cups capable of individually applying a vacuum to a surface of an article, the gripper configuration being defined by a coordinate of a gripping point, an orientation of the head, and a suction cup combination to be used;

[0017] - the at least one second score, representative of a quality of gripping of the article considered, can be calculated on the basis of a criterion representative of a quality of gripping of an article by the individual article handling device and chosen from a height, a surface and a slope of the article considered, a gripper configuration assigned to the article considered, heights of articles surrounding the article considered, and a trajectory calculated to move the article considered from the handling zone to a destination zone;

[0018] - for the step of calculating the usability score, the respective weighting coefficients of each of the first score and of the at least one second score can be chosen from weighting coefficients each associated with one of a first homogeneous batch of articles and a second homogeneous batch of articles;

[0019] - the procedure may further comprise a step in which the article considered is evaluated as similar according to at least one criterion to the articles of the first batch, automatically or by an operator, and, in response to the evaluation step, the respective weighting coefficients of each of the first score and of the at least one second score may be chosen as being the weighting coefficients associated with the first batch.

[0020] A second aspect of the invention relates to a system configured to implement the method of the first aspect of the invention within a sorting facility.

[0021] In order to achieve this aim, a second aspect of the invention is an article sorting installation configured to implement the article handling method according to the invention, comprising a sorting line equipped with a control system (SYS), the sorting line comprising a device for individual article handling, the control system comprising a digital camera and a digital computer comprising a computer memory functionally connected to the digital camera and to the device for individual article handling, the digital camera being arranged so as to implement the step of acquiring the digital color image of the scene comprising the articles to be handled in the handling zone and the pair of digital images intended to form the depth map of said scene. BRIEF DESCRIPTION OF THE FIGURES

[0022] Other characteristics and advantages of the invention will emerge from the detailed description of the invention which follows with reference to the appended figures in which:

[0023] Represents a portion of an item sort line;

[0024] Represents a configuration of items to be sorted according to a sectional view;

[0025] It represents a process of individual handling of items to be sorted;

[0026] Illustrates a criterion for characterizing the speed of transport of articles;

[0027] Illustrates methods for determining criteria values ​​for choosing an item to be handled;

[0028] Illustrates a method for determining a value of a criterion for choosing an item to be handled;

[0029] Illustrates a method for determining a value of a criterion for choosing an item to be handled;

[0030] Illustrates a method for determining a value of a criterion for choosing an item to be handled;

[0031] Illustrates a method for determining a value of a criterion for choosing an item to be handled;

[0032] Illustrates a table associating scores with coefficients;

[0033] Illustrates variations made to the process of the ; and

[0034] Illustrates a table associating scores with coefficients and used in the context of the variants of the. DETAILED DESCRIPTION OF THE INVENTION

[0035] The principle of the invention is explained with the help of figures 1 to 9.

[0036] The context in which the invention is situated is illustrated using the example of, which represents a portion of a FAC installation for processing articles comprising an SL sorting line equipped with a SYS system for controlling the sorting line.

[0037] The sorting line SL comprises a device for individual handling of articles, here consisting of a robotic manipulator arm MA equipped with a gripping head GH, and a conveyor belt CB moving articles arriving in the direction Dir represented by an arrow. The gripping head GH can be equipped with suction cups SC connected to a pumping system, each suction cup can be controlled individually so as to apply a vacuum to the surface of an article to be grasped. The articles can arrive in bulk in a work zone WZ from the right of the figure, the manipulator arm being responsible for unpacking them, that is to say, grasping them one by one to place them in order on the conveyor belt CB towards the left of the figure.The SYS sorting line control system comprises a CAM digital camera and a CALC digital computer comprising a computer memory functionally connected to the CAM digital camera and to the MA manipulator arm. The MA manipulator arm has the function of individually manipulating IT articles located in the work area to move them from this area to the CB conveyor belt.

[0038] L represents schematically a configuration of articles to be handled by the manipulator arm, corresponding to a sectional view of the work zone WZ. L illustrates various situations that the control system must face when trying to optimize the choice of the article to be chosen for the next action of the manipulator arm.

[0039] In this example, item A is hidden from the camera CAM by item B and therefore cannot be taken into account in the calculations of the CALC calculator, based on images acquired by this camera, even though it might have been preferable to process this item as a priority, being the closest to the conveyor belt CB. Item C is superimposed on item D, the latter therefore not reasonably being chosen as the next item to be handled due to the risk of causing a handling incident such as an item being dropped during an attempt to move it by the manipulator arm. Item F partially covers item E and has an upper surface inclined relative to the work plane WP of the work zone WZ on which the items rest. The fact that this upper surface is inclined may cause difficulties in gripping it by the manipulator arm.

[0040] The objective of the invention is to allow easy adaptation of the control of the manipulator arm to the objectives of the sorting line operator and to the characteristics of the IT articles to be sorted, taking into account the various situations with which the control system may be confronted. To this end, an object of the invention is a method 100 for individual manipulation of the articles to be sorted arriving together in the work area WA.

[0041] Illustrates the method 100, which comprises a step S10 consisting of acquiring at least (i) a digital color image NumCol of a scene encompassing the working area WZ of the manipulator arm MA and (ii) a pair of digital images NumDepth intended to form a depth map of said scene according to known principles of photogrammetry, by means of the digital camera CAM, in which one or more articles IT, intended to be manipulated individually by means of the manipulator arm, are visible. The images are stored in a computer memory of the computer CALC.

[0042] In a step S20, on the basis of the digital color image, the computer CALC identifies visible items VisIT in the image, for example by means of a neural network by carrying out a segmentation operation according to conventional image analysis methods.

[0043] In a step S30, the digital computer CALC forms, according to conventional photogrammetry methods, a depth map MAP3D of the scene on the basis of the images of the pair of digital images intended to form a depth map of said scene. A greater number of images could be used, the essential thing being to be able to generate a map representative of the positioning of the articles of the work area in a three-dimensional space. The surfaces of the articles visible on the images can then be represented, for example, by a cloud of points each located in the three-dimensional space.

[0044] At this step S30, a correspondence is established between the identified visible articles and elements of the depth map such as volumes defined by the point cloud. In this way, the identified visible articles can be characterized by calculations carried out on the basis of the point cloud, that is to say on the basis of the topographic map. For example, an article in the form of a parallelepiped box can be characterized by its lateral dimensions as seen by the digital camera CAM as well as, for example, by its height (in this document, the term "height" of an article means the height of its upper surface, that is to say the distance separating its upper surface from the work plane, as illustrated by the height h in B), or the inclination of its upper surface relative to the work plane WP of the work area on which it rests.

[0045] At a step S40, the digital computer CALC determines the pickability Y / N of each of the identified visible articles, on the basis of one or more geometric criteria deduced from the depth map, and possibly characteristics of the manipulator arm. This involves determining whether or not an article is considered pickable, i.e. as being able to be handled by the manipulator arm, according to predetermined criteria aimed at limiting the risks of deterioration in the quality of sorting, for example by collisions or involuntary release of a grasped article. The choice of criteria is the responsibility of the operator, who will adapt it according to the type of article to be sorted, the manipulator arm, the installation in which the work area and the manipulator arm are integrated, for example for the management of non-pickable articles, and its objectives.The pickability of an item is an attribute indicating the ability of an item to be moved by the manipulator arm.

[0046] A first criterion is that of dimensions: depending on the capacities of the manipulator arm, not all items can be reliably gripped, and items exceeding the capacities of the manipulator arm will be considered ungrippable. For example, these may be items that are too small or too narrow to be gripped by one or more suction cups of a manipulator arm. An item with a volume exceeding a volume threshold defined by the operator may also be considered ungrippable.

[0047] A second criterion is the height of the item (as distinct from its thickness), and more specifically the vertical positioning of its upper surface. In fact, items whose height exceeds the maximum stroke of the manipulator arm in the vertical direction cannot be picked up.

[0048] A third criterion is the slope of the upper surface of the article. Indeed, when, for example, the manipulator arm is designed to grip an article by means of the GH gripper head, which is flat and equipped with a plurality of suction cups, and this head cannot tilt relative to the work surface (limitation due to the degrees of freedom of the manipulator arm), the suction cups will only be able to come into contact so as to apply a vacuum to the surface of the article, and therefore grip this article, if the slope of the surface does not exceed a certain threshold, depending on the size and deformability of the suction cups. If the slope exceeds this threshold (25° for example), the article will be considered ungrippable due to a high risk of the article being released during its movement by the manipulator arm. illustrates these two situations.

[0049] A fourth criterion is based on the possibility or not of assigning a gripper configuration to the article in question, on the basis of the visible part of the article to be moved and its location in the set of articles, possibly partially covered by one or more other articles. In the case of the example of the gripper defined above, if the control system is unable to calculate a configuration defined by a position of the head and a choice of suction cup(s) which must simultaneously apply a vacuum to a surface of an article, then this article is considered ungrippable. For illustration, one can refer to the and to the corresponding passage of the description. Of course, this criterion depends on the equipment for gripping and moving the articles used, and each system integrating this equipment uses its own calculation method for determining a gripping configuration. An example is given by the.

[0050] A fifth and sixth criteria will be explained in connection with Figures 8 and 9.

[0051] At a step S50, the digital computer CALC calculates a pickability score SCOR, preferably only for each of the items considered pickable at the end of step S40. The choice of the item to be picked up in the scene captured at step S10 is determined by this pickability score, the item with the highest score being chosen for the next item manipulation by the manipulator arm.

[0052] The pickability score of an item is made up of a combination of a first score representative of the location of the item in the scene and at least a second score representative of the quality of grasping of the item, each of the scores being assigned a respective weighting coefficient.

[0053] The first score, calculated on the basis of the location of the article considered in the scene, corresponds to a criterion intended to characterize the speed of transport of the article by the manipulator arm. The first score will be calculated in such a way as to have a value all the greater as a distance Dist, separating the article from the location where it must be deposited by the manipulator arm, is short.

[0054] This distance D is illustrated in the configuration of the, with items H and G to be deposited in the drop zone DZ of the work zone WZ. The distance Dist can be defined as the distance between the center of an item and the drop zone DZ.

[0055] The weighting coefficient assigned to the first score will have a value that is all the greater as the operator gives importance to the flow rate of the articles to be handled: moving as a priority the articles closest to their arrival location makes it possible to minimize the movements of the manipulator arm, which makes it possible to maximize the number of articles processed and thus maximize the flow rate of articles.

[0056] It is possible to integrate into the first score other elements having a direct or indirect impact on the flow of the articles. A first example of such an element is a trajectory calculated for the manipulation arm, trajectory which can be affected by the presence of obstacle to the movement of the arm (stack of articles for example). The length of the necessary detour or the starting position of the manipulator arm can optionally be taken into account to refine the first score. A second example of an element likely to be taken into account for the calculation of the first score is a surface of the article which, if it is important and the article is moved, makes it possible to clear the scene and to make visible articles hidden by this package allowing in the long term a better maximization of the flow of the articles.However, the first score will be considered representative of the location of the item in the scene because it is the predominant criterion even if elements such as those mentioned in this paragraph are integrated into the calculation of the first score.

[0057] The at least one second score may represent one or a plurality of geometric or operational criteria characterizing the expected quality of the gripping of an article by the manipulator arm. It is possible to consider only one second score corresponding to a single criterion, but it is preferable to consider a plurality of second scores corresponding to as many criteria which, taken together, characterize the expected quality of the gripping of an article by the manipulator arm. Among these criteria, we can cite the height and the slope of the upper surface of the article, the trajectory that the manipulator arm must follow to move the article, the superposition between the article considered and another article placed on it, or the configuration of the gripper calculated to grip the article.

[0058] The weighting coefficients assigned to the second scores will have values ​​that are all the greater as the operator gives importance to the quality of entry of the items to be handled.

[0059] The scores considered are dimensionless numbers, each corresponding to one of the criteria mentioned above, and can, for example, take values ​​between 0 and 1, according to correspondences with the quantities considered (height, dimensions, slope, number of suction cups, etc.) established by the operator. It is also possible for the operator to assign values ​​lower than 0 if he wishes to give penalties to certain criteria in particular situations, for example, in the case where an item is covered by two or more other items, or if the general configuration of the scene makes an item particularly difficult to move because it is surrounded by higher items.

[0060] If we take the example of a second score based on the slope of the upper surface of the article considered, a second maximum score of 1 could correspond to an upper surface of the article parallel to the work plane, a second score of 0 could correspond to all angles of inclination greater than a threshold indicated by the operator and depending on the characteristics of the gripping head used, scores increasing between 0 and 1 being assigned to angles decreasing from the threshold to 0°. Figures 7 to 9 illustrate the assignment of second scores to other criteria.

[0061] Each pickable item is assigned a pickability score based on a combination of the first score and the second score(s), each weighted by their respective weighting coefficients. This combination can be a simple weighted sum of the scores. As explained above, the item with the highest score is chosen for the next item handling by the manipulator arm. The values ​​of the coefficients, chosen by the sorting line operator, are decisive for the pickability score obtained. The pickability SCOR score can thus be calculated according to Eq. 1:

[0062] Eq.1

[0063] where S1 represents the first score, representative of the location of the item, and C1 its associated weighting coefficient, and S2i represent the second scores, each representative of one of n geometric or operational criteria characterizing the expected quality of the gripping of an item by the manipulator arm, and C2i their associated weighting coefficients.

[0064] If the operator wishes to give priority to the throughput of the sorting line, then he will give a high value to the weighting coefficient C1 assigned to the first score in comparison with the weighting coefficients C2i assigned to the second scores.

[0065] If the operator wishes to prioritize the quality of item capture, then he will give high values ​​to the C2i weighting coefficients assigned to the second scores in comparison with the C1 weighting coefficient assigned to the first score. In addition, the operator can also adjust the weighting coefficients assigned to the second scores relative to each other, depending on the characteristics of the items to be sorted or the individual item handling device.

[0066] Alternatively, the second S2i scores can be combined, for example by means of a weighted sum, so as to obtain a second overall score S2, the operator then only having to adjust two coefficients: the weighting coefficient C1 of the first score and a second overall weighting coefficient C2 associated with the second overall score S2, or even more simply a ratio between these two coefficients, both methods allowing a relative adjustment of the values ​​of the first weighting coefficient and the second overall weighting coefficient. The SCOR score can be calculated according to the equation Eq.2:

[0067] Eq.2

[0068] Or , S2 represents the second overall score and C2 the associated overall weighting coefficient.

[0069] This alternative provides the operator with a very simple means of control to promote either the flow rate of articles or the quality of article entry.

[0070] Regardless of the method used to adjust the weighting coefficients, a pickability score is calculated based on a weighted sum between a first score representative of the location of the item considered in the scene and at least one second score representative of a quality of capture of the item considered, each of the scores being assigned a respective weighting coefficient. These data can be stored in the form of a table Tab in a computer memory, comprising for example the scores S1, S21, S22, S2 associated respectively with the coefficients C1, C21, C22, C2, for the case where n=2. The illustrates such a table.

[0071] It is possible to calculate the pickability scores for each of the visible items identified and to continue the process on this basis, but limiting the calculation of pickability scores to only those items considered pickable makes it possible to limit the amount of calculation to be carried out. This last point is all the more important since this is a process taking place in real time, the calculations having to be carried out sufficiently quickly by the CALC digital calculator so as not to slow down the processing of items on the sorting line and therefore to allow the throughput of items on this line to be maximized.

[0072] Each of the weighting coefficients of the criteria selected to be assigned to the first score and / or to the second scores can be adjusted individually by the operator, for example during a preliminary step S05 of initialization of the sorting line control system. An individual adjustment of the coefficients allows great precision in the adjustment of the score calculation. These coefficients are stored in a computer memory of the control system and will be retrieved there for the SCOR score calculation of step S50.

[0073] At a step S60, in response to the calculation of the pickability score, the sorting line control system sends a command signal COM to the individual item handling device so as to handle an item to be handled. More specifically, the item with the highest pickability score is chosen to be handled. Following this command sent to the handling device, the manipulator arm MA in this example, a new cycle comprising steps S10 to S60 is initiated in order to determine the next item to be handled. Alternatively, the scores could be calculated in such a way that the item to be handled preferably is the one with the lowest non-pickability score, the non-pickability score being worth (1-pickability score). In this case, it is the item with the lowest non-pickability score that would be chosen to be handled.

[0074] Figures 5 to 9 illustrate in concrete terms how the criteria mentioned above can be represented by a numerical quantity for the calculation of the second scores.

[0075] Illustrates in A) the first criterion of pickability: in a top view of the work area WZ, an article I is seen as rectangular, with dimensions L and W, deduced from the digital color image and the depth map using conventional image processing techniques. These dimensions also allow the surface area of ​​the article to be calculated. These dimensions and / or this surface area can constitute the first criterion of pickability. A study of the depth map also allows the thickness of the article to be determined: by studying the height of the points surrounding the article identified in the digital color image, it is possible to determine whether it rests directly on the work surface, in which case the height of the article is equal to its thickness, or on another article, in which case the thickness of the article is equal to the height of the article from which the height of the article on which it rests is subtracted.

[0076] Illustrates in B) the second criterion of usability: a cross-sectional view of the work area shows the height h of the upper surface of an article J placed on the work plane WP of the work area WZ, considering that the camera CAM is located at a height H above the work plane WP. The height h can be deduced from the height H and the distance between the camera CAM and the upper surface of the article C. This height is of course found in the depth map of the scene.

[0077] Illustrates in C) the third criterion of gripping ability: the slope Sl of the upper surface of the article F supported by the work plane WP and the article E. This slope can be calculated for example by considering two bands of pixels parallel to the estimated sides of the article on the basis of the digital color image, forming a cross in the center of the article. A linear regression on the heights of each of these bands makes it possible to calculate the 2 angles of the slope Sl of the article with respect to each of the edges of the article. These angles make it possible to determine, depending on the characteristics of the gripper and the suction cups with which it is equipped, whether this gripper can actually grip the article, as illustrated by the, and therefore whether the article is grippable or not.

[0078] Figures 6 and 7 illustrate the influence of the item capture system in determining which items to capture.

[0079] Illustrates in A) a first situation in which the slope of the upper surface of the article is sufficiently low and the SC suction cups are sufficiently deformable so that they can adapt to the surface of the article. In the second situation illustrated in B), the adaptation capabilities of the suction cups are not sufficient to apply a vacuum to the upper surface of the object. In the second situation, the article must be considered unpickable.

[0080] Illustrates the fourth pickability criterion as well as the establishment of a second corresponding pickability score, with here four different configurations A) to D) for gripping an article, respectively IT1 to IT4, using an adapted number of suction cups SC. In each of these configurations, the barycenter of the activated suction cups is placed in the middle of the gripped article, six suction cups being activated in A), four suction cups in B), two suction cups in C) and one suction cup in D). The dotted lines indicate the portion of the gripping head GH superimposed on the gripped article, only the dotted suction cups being activated.Among the rules to be applied to determine a configuration, we can cite the following: each activated suction cup (connected to the pumping system and sucking air) must completely cover a mask (representation of an item defined by a method of segmentation of the colored digital image) of the item to be grasped, the suction cups activated but not used to grasp the targeted item must not collide with another item, if several configurations are possible, the one involving the most suction cups is chosen to ensure the best possible grasping. If the CALC calculator cannot determine a configuration such as those illustrated by the, then the item will be considered ungraspable.

[0081] When the item is considered to be pickable, the CALC calculator can calculate a second pickability score representative of the quality of gripping of the item on the basis of the determined gripping configuration, on the basis of rules defined by the operator. The latter can, for example, award a maximum score of 1 when the configuration chosen uses all six suction cups of the gripping head to grip the item (as in A) of the), and reduce the score progressively with the reduction in the number of suction cups used.

[0082] Laet illustrate a fifth criterion of prenability, with a situation in which items C and D are superimposed on each other. In order to calculate a prenability score for each of these items, the CALC calculator will be able to determine which item is above the other by exploiting the digital color image and the depth map: the upper surface of each item is conventionally approximated by a plane of general equation z=a.x+b.y+cst (a, b and cst being constants and x, y and z three coordinates in an orthonormal three-dimensional frame), which allows to determine its slope and its height (median, average, height at any point). When the masks of the two items intersect, which is the case of the masks Mask(C) and Mask(D) of the packages C and D defined by a segmentation operation of the NumCol image, as illustrated by la, the CALC calculator engages an algorithm to determine which item is located above the other.

[0083] More specifically, the average heights of the top surfaces of the items are calculated for the intersection area of ​​the respective masks of the two items from the two equations of the plane approximation. The item with the higher average height is considered to be above the one with the lower average height.

[0084] In the example of figures 2 and 8, article C is above article D. As such, and concerning the superposition criterion, a second low score (possibly 0) will be assigned to article D and a second high score (1 for example) to article C.

[0085] A sixth criterion of usability is illustrated by calculating a second score taking into account the environment of an item in the work area, influencing the trajectory of the manipulator arm to move the item. Thus, a zone Z(IT) is defined so as to frame an IT item identified in a depth map of a scene in the work area. Peripheral zones Z1 to Z5 are defined around the perimeter of the zone Z(IT). The maximum heights reached by items present in each of the zones Z1 to Z5 as well as the height of the upper surface of the IT item are deduced from the depth map. The information on these maximum heights and their respective membership in zones Z1 to Z5 constitutes a means of identifying possible obstacles around the IT item and will allow the CALC computer to determine a trajectory of the robotic arm avoiding collisions by picking up the item or by placing it at its destination point.This information can also serve as a basis for calculating a second score, for example determined as higher the smaller the number of maximum heights exceeding the height of the IT article, a score which can take values ​​between 0 and 1, or lower the more the trajectory of the robotic arm passes through areas containing articles of high height.

[0086] An advantage of the approach described above is the possibility of using several sets of weighting coefficients for the criteria concerning the quality of item capture and those concerning the flow rate of items. These sets of coefficients are obtained, possibly by machine learning, on different statically independent bases. One advantage is to then combine these weighting coefficients to improve the performance and robustness of the picking process.

[0087] This combination can take at least two forms: one linear and the other non-linear.

[0088] The first form consists of averaging the weighting coefficients of each set as well as the weighting (via for example the global weighting coefficient C2 defined above) making it possible to adjust the relative influences of all the criteria influencing the flow rate on the one hand and all those influencing the quality of capture on the other hand.

[0089] The second form consists of assigning a new score to each item in the scene by averaging the ranks in the manipulation order obtained for the different sets of coefficients for the item in question. If R represents the rank in the manipulation order of an item in a scene comprising m items, we can define a new SCOR score equal to (mR) / (m-1). The first item to be manipulated has rank 1 and the last has rank m, ranks determined by the original score obtained by means of a set of coefficients for each of the items. Each set of coefficients gives a score. A combined score can be obtained by averaging the scores S obtained for different sets of coefficients.

[0090] This approach can also be advantageously adapted for specific situations, such as, for example, the processing of homogeneous batches. Thus, it is possible to define, possibly by automatic learning (for example according to the principles described in document FR2310269, applied to the determination of the optimum weighting coefficients) on a homogeneous batch of similar articles, for example articles all packaged in white flexible packaging (white Polybags for example). This specific learning makes it possible to determine weighting coefficients (C1) of the criteria influencing the flow rate and those (C2i) influencing the quality of the capture, each adapted to this particular batch of articles. The operator can also manually determine a ratio which determines a compromise between the two families of criteria (flow rate or quality of capture), thus favoring the flow rate or the quality of capture.This ratio can be representative of the quotient between the coefficients C1 and C2 of equation 2. Alternatively, the ratio can be determined automatically by learning on another homogeneous batch according to the same criteria (white polybags in the example above).

[0091] Thus, we can consider a first set of coefficients C11 and C2i1 for a first homogeneous batch Lot1 of articles and a second set of coefficients C12 and C2i2 for a second homogeneous batch Lot2 of articles, as illustrated by the table Tab2 of the, i and n being defined as for the equation Eq. 1. The batches are considered homogeneous according to at least one criterion, which can be chosen from their types of packaging (flexible, rigid), their packaging colors (red, green, blue…), their sizes (possibly divided into categories of length, thickness, diameter…), their shapes (rounded, angular, parallelepiped, cylindrical…) etc… This means that each article of a batch can be considered as similar to the other articles of this batch according to this at least one criterion.In the example of articles packaged in a white polybag cited above, these would be articles considered as forming a homogeneous batch according to two criteria: the color (white) and the type of packaging (flexible, a polybag being a flexible packaging). It is understood that the assessment of the criterion(s) to be considered and the desired level of resemblance are left to the discretion of the operator and can be determined either manually or according to conventional digital image analysis techniques implemented by means of the CALC calculator and the CAM digital camera. The coefficients C11 and C12 represent weighting coefficients associated with first scores each representative of the location of the articles of the first batch and the second batch, respectively.The coefficients C2i1 and C2i2 represent weighting coefficients associated with second scores, each representative of one of n geometric or operational criteria characterizing the expected quality of the grasping of an article by the manipulator arm (i being an integer varying from 1 to n indicating to which criterion i the coefficients C2i1 and C2i2 are associated), respectively. Depending on the type of articles to be sorted, it may be more appropriate to select one or the other of the first set of coefficients and the second set of coefficients, or a combination of these two sets of coefficients.

[0092] Thus, a first variant of the method 100 includes the additional steps illustrated by the.

[0093] In (A), an additional step S05-1 is illustrated, included in the step S05 of initializing the control system. This step S05-1 consists of determining and entering into the memory of the CALC calculator the weighting coefficients associated with homogeneous batches Lot1 and Lot2 of articles. These coefficients can, for example, be deduced from preliminary experience on the sorting line, or determined by automatic learning, possibly using a simulated environment of the sorting line.

[0094] In (B), two additional steps S50-1 and S50-2 are illustrated, included in step S50 for calculating the SCOR score. A first step S50-1 consists of evaluating whether the article in question can be considered similar to the articles of one of the homogeneous batches of step S05, and determining which of these batches the article corresponds to. In step S50-2, in response to this evaluation, the CALC calculator retains, for calculating the SCOR score, the set of coefficients of the batch corresponding to the article.

[0095] A second variant of the method 100 is a simplification of the first variant. In a case in which the article to be sorted is part of a homogeneous batch for which the weighting coefficients are known, for example a homogeneous batch corresponding to one of the batches Lot1 and Lot2 of step S05-1, the sorting line operator can, from the initialization step S05, indicate the set of coefficients to be used. In this case, steps S50-1 and S50-2 are superfluous.

[0096] Both of the above variants improve the relevance of the coefficients, and therefore of the score calculation, and improve the efficiency of the sorting line.

[0097] Of course, the invention is not limited to the implementation methods described above and variant embodiments can be made without departing from the scope of the invention.

Claims

Method (100) for handling articles by means of a device (MA) for individual handling of the articles, comprising the steps of:- acquiring (S10) a digital color image (NumCol) of a scene comprising articles to be handled in a handling area and a pair of digital images (NumDepth) provided to form a depth map of said scene;- on the basis of the digital color image, identifying (S20) visible articles (VisIT) by means of a digital computer (CALC);- on the basis of the pair of digital images (NumDepth), forming (S30) a depth map (MAP depth ) of the scene, and associate the identified visible items with elements of the depth map; - based on the digital color image (NumCol) and the depth map (MAP depth), calculating (S50), for at least some of the identified visible items (VisIT), a pickability score (SCOR) based on a combination between a first score (S1) representative of the location of the item considered in the scene and at least one second score (S21, S22) representative of a grasping quality of the item considered, each of the first score and the at least one second score being assigned a respective weighting coefficient (C1, C21, C22); and- in response to the pickability score (SCOR), controlling (S60) the device (MA) for individual manipulation of the items so as to manipulate one of the visible items, in which the pickability score (SCOR) is determined on the basis of a combination between the first score (S1) representative of the location of the item considered in the scene and at least two second scores (S21, S22) each representative of a grasping quality of the item considered,in which the takeability score (SCOR) is calculated by a combination of the first score (S1) assigned its first weighting coefficient (C1) and a second overall score (S2) representative of a combination between the at least two second scores (S21, S22) each assigned their respective weighting coefficient (C21, C22), the second overall score (S2) being assigned a second overall weighting coefficient (C2), so as to promote either a flow rate of the articles, or a quality of entry of the articles, the method further comprising a preliminary step (S05) of adjusting the respective weighting coefficients (C1, C2) of the first score (S1) and of the second overall score (S2), so as to promote either a flow rate of the articles by increasing the first weighting coefficient (C1) relative to the second overall weighting coefficient (C2),or a quality of capture of each of the visible articles identified by increasing the second overall weighting coefficient (C2) in relation to the first weighting coefficient (C1)., Method according to claim 1, wherein the combination is a weighted sum of the first score (S1) and the at least one second score (S21, S22) assigned their respective weighting coefficients (C1, C21, C22). Method according to claim 1 or 2, in which the combination between the at least two second scores (S21, S22) is a weighted sum of the at least two second scores (S21, S22) each assigned their respective weighting coefficient (C21, C22). Method according to any one of claims 1 to 3, further comprising a step of determining (S40) a pickability (Y / N) of the identified visible articles, the articles determined as pickable constituting said at least some of the identified visible articles. Method according to claim 4, the pickability of the identified visible articles being based on at least one criterion chosen from criteria respectively representative of dimensions, a volume, a height and a slope of an article considered, an accessibility of the article considered by the individual manipulation device of the articles, and the possibility of attributing a gripper configuration to the article considered. The method of claim 5, wherein the device for individually handling the articles is a robotic manipulator arm (MA) comprising a head provided with a plurality of suction cups capable of individually applying a vacuum to a surface of an article, the gripper configuration being defined by a coordinate of a gripping point, an orientation of the head, and a suction cup combination to be used. Method according to any one of claims 1 to 6, the at least one second score, representative of a quality of gripping of the article considered, is calculated on the basis of a criterion representative of a quality of gripping of an article by the device for individual manipulation of the articles and chosen from a height, a surface and a slope of the article considered, a gripper configuration assigned to the article considered, heights of articles surrounding the article considered, and a trajectory calculated to move the article considered from the manipulation zone to a destination zone. Method according to any one of claims 1 to 7, in which, for the step (S50) of calculating the usability score, the respective weighting coefficients (C1, C21, C22) of each of the first score and of the at least one second score are chosen (S50-2) from weighting coefficients (C11, C2i1, C2n1, C12, C2i2 and C2n2) each associated with one of a first homogeneous batch (Batch1) of articles and a second homogeneous batch (Batch2) of articles. Method according to claim 8, further comprising a step (S50-1) in which the article considered is evaluated (S50-1) as similar according to at least one criterion to the articles of the first batch (Batch1), automatically or by an operator, and in which, in response to the evaluation step (S50-1), the respective weighting coefficients (C1, C21, C22) of each of the first score and of the at least one second score are chosen (S50-1) as being the weighting coefficients (C11, C2i1, C2n1) associated with the first batch (Batch1). An article sorting installation (FAC) configured to implement the article handling method according to any one of claims 1 to 9, comprising a sorting line (SL) equipped with a control system (SYS), the sorting line comprising a device (MA) for individual article handling, the control system comprising a digital camera (CAM) and a digital computer (CALC) comprising a computer memory functionally connected to the digital camera (CAM) and to the device (MA) for individual article handling, the digital camera (CAM) being arranged so as to implement the step of acquiring (S10) the digital color image (NumCol) of the scene comprising the articles to be handled in the handling zone and the pair of digital images (NumDepth) intended to form the depth map of said scene.

Citation Information

Patent Citations

  • method AND DEVICE FOR FORMING A HERMETIC WELDED SEAL BETWEEN THE LAYERS OF SYNTHETIC MATERIAL OF A BODY FORMED BY LAYERS OF METAL AND SYNTHETIC MATERIAL

    FR2310269A1

  • Device for supplying flat items for a bucket conveyor

    FR3019068B1

  • Installation for the Separation and Individualization of Heterogeneous Postal Items

    FR3031046A1

  • Autonomous unknown object pick and place

    US20200316782A1

  • Singulation of arbitrary mixed items

    US20210122054A1

Cited By

  • Gripper arrangement, method and computer-readable medium

    EP4768197A1