Picking process in a sorting line and associated installation
The method optimizes article handling on sorting lines by using digital image processing to determine a pickability score, adjusting coefficients for speed or quality, and controlling a robotic arm to handle articles efficiently and reliably.
Patent Information
- Application Number
- FR2023005495
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-06-01
- Publication Date
- 2024-12-06
AI Technical Summary
Existing sorting technologies face challenges in maximizing the speed of 'pick and place' operations while minimizing handling errors, particularly in sorting lines where articles arrive unordered and may be piled up, requiring a compromise between speed and quality of handling.
A method involving a device for individual article handling that uses digital image processing to determine a pickability score based on location and gripping quality, adjusting weighting coefficients to prioritize either throughput or quality, and controlling a robotic manipulator arm to optimize article selection and handling.
Enhances the efficiency and reliability of article handling by ensuring rapid, error-free operations tailored to the specific characteristics and objectives of the sorting line, improving both throughput and quality of handling.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: METHOD FOR PICKING IN A SORTING LINE AND ASSOCIATED INSTALLATION TECHNICAL FIELD OF THE INVENTION
[0001] The invention relates to a method for handling articles in circulation on an article sorting line, with the aim of generally increasing the quality or speed of handling of these articles. TECHNOLOGICAL BACKGROUND
[0002] The sorting of articles such as postal parcels is carried out within a sorting center comprising sorting lines intended to carry out automated sorting of articles, each processing a flow of articles. The articles are handled along the lines by means of collective conveyor systems but require, for example for individual picking or placement, to be picked individually during an operation called "pick and place", or "pick and place" in English terminology, implemented by means of a robotic manipulator arm, as described for example in document FR 3 019 068 Bl. An example of a manipulator arm using suction cups as a device for gripping articles is described in document FR 3 031 046 AL Of course, in order to make the sorting lines profitable, they must process as large a flow of articles as possible and the speed of execution of the "pick and place" operation is essential.
[0003] The function of the manipulator arm is to grasp the item, move it to a predetermined location and then release it. This operation must be rapid while remaining reliable in order to limit handling errors. An example of handling error is "fly-out" in English terminology, which consists of an unexpected release of the item while it is being moved at high speed by the arm and can result in the item being ejected from the sorting line.
[0004] There is therefore a need to maximize the speed of “pick and place” operations while limiting handling errors, in other words, it is necessary to find a compromise between the speed of operations and the quality of inputs, in a way that allows this compromise to be adapted to the objectives of the sorting line operator, which may vary with the nature of the items to be sorted. Statement of the invention
[0005] Items to be sorted moving along a sorting line may arrive at an automated “pick and place” handling area without any predefined order or location, and items may even be piled up on top of each other.
[0006] An optimization of the individual handling of articles taking into account the The quality of capture of these articles and their processing rate requires, in order to control a device for individual handling of the articles, taking into account the positions of each of the articles in the handling area as well as the relative positions of the articles in relation to each other, and possibly the characteristics of the handling device.
[0007] An aim of the invention is to provide a method for individual handling of articles allowing optimization of the handling of these articles considered as a whole, integrating the determination of an order of handling of the articles, and more specifically the determination of the next article to be handled in a sorting scene.
[0008] In order to achieve this aim, a first aspect of the invention is a method for handling articles by means of a device for individual handling of the articles, comprising the steps of acquiring a digital color image of a scene comprising articles to be handled in a handling area and a pair of digital images provided to form a depth map of said scene; on the basis of the digital color image, identifying visible articles by means of a digital computer; on the basis of the pair of digital images, forming a depth map of the scene, and associating the identified visible articles with elements of the depth map;based on the digital color image and the depth map, calculating, for at least some of the identified visible items, a pickability score based on a combination of a first score representative of the location of the item considered in the scene and at least one second score representative of a quality of grasping of the item considered, each of the first score and the at least one second score being assigned a respective weighting coefficient; and in response to the pickability score, controlling the individual item handling device so as to handle one of the visible items.;
[0009] This method allows the use of a plurality of selection criteria so as to determine which is the next article to be handled by an individual article handling device on an article sorting line. These criteria give the sorting line operator a very simple means of adapting the control of the handling device to his operational objectives, which can lead to favoring either the speed of handling or the quality of this handling, by limiting the risks of dropping articles and collisions between a grasped article and the other articles in the work area, or even a compromise between the two positions, depending on the type of articles to be sorted (shape, nature, weight, etc.).
[0010] According to additional non-limiting characteristics of the first aspect of the invention, considered individually or in any technically feasible combination:
[0011] - the combination may be a weighted sum of the first score and the at least one second score affected by their respective weighting coefficients;
[0012] - the method may further comprise a preliminary step of adjusting the coefficients respective weighting factors of the first score and of at least one second score, so as to favor either a speed of flow of the articles or the quality of entry of each of the visible articles identified.;
[0013] - the takeability score can be determined on the basis of a combination between the first score representative of the location of the item considered in the scene and at least two second scores each representative of a quality of capture of the item considered, the pickability score being calculated by a combination of the first score assigned its first weighting coefficient and a second overall score representative of a combination between the at least two second scores each assigned their respective weighting coefficient, the second score being assigned a second overall weighting coefficient;
[0014] - the method may comprise a preliminary step of initializing a system of control of the sorting line includes a relative adjustment of the values of the first weighting coefficient and the second overall weighting coefficient;
[0015] - the combination between the at least two second scores can be a sum weighted by at least two second scores each affected by their respective weighting coefficient;
[0016] - the method may further comprise a step of determining a takeability identified visible items, the items determined to be takeable constituting said at least part of the identified visible items;
[0017] - the usability of the identified visible articles can be based on at least one criterion chosen from criteria respectively representative of dimensions, a volume, a height and a slope of an article considered, an accessibility of the article considered by the individual handling device of the articles, and the possibility of attributing a gripper configuration to the article considered;
[0018] - the device for individual handling of the articles can be a manipulator arm robotic device comprising a head provided with a plurality of suction cups capable of individually applying a vacuum to a surface of an article, the gripper configuration being defined by a coordinate of a gripping point, an orientation of the head, and a suction cup combination to be used;
[0019] - the combination between the first score and the at least one second score can be a linear combination of the first score and the second score each weighted by their respective weighting coefficient;
[0020] - at least one second score, representative of a quality of entry of the article considered, can be calculated on the basis of a criterion representative of a quality of capture of an article by the individual article handling device and selected from a height, a surface and a slope of the article considered, a gripper configuration assigned to the article considered, heights of articles surrounding the article considered, and a trajectory calculated to move the article considered from the handling area to a destination area.
[0021] A second aspect of the invention relates to a system configured to implement the method of the first aspect of the invention within a sorting installation.
[0022] In order to achieve this aim, a second aspect of the invention is an article sorting installation configured to implement the article processing method according to the first aspect of the invention, comprising a device for individual article handling. BRIEF DESCRIPTION OF THE FIGURES
[0023] Other characteristics and advantages of the invention will emerge from the detailed description of the invention which follows with reference to the appended figures in which:
[0024] [Fig.l] [Fig.l] represents a portion of an article sorting line;
[0025] [Fig.2] [Fig.2] represents a configuration of articles to be sorted according to a view in cut ;
[0026] [Fig.3] [Fig.3] represents a method of individual handling of items to be sorted;
[0027] [Fig.4] [Fig.4] illustrates a criterion for characterizing a transport speed of the articles;
[0028] [Fig.5] [Fig.5] illustrates methods for determining criteria values for choosing an item to be handled;
[0029] [Fig.6] [Fig.6] illustrates a method for determining a value of a criterion for choosing an item to be handled;
[0030] [Fig.7] [Fig.7] illustrates a method for determining a value of a criterion for choosing an item to be handled;
[0031] [Fig.8] [Fig.8] illustrates a method for determining a value of a criterion for choosing an item to be handled;
[0032] [Fig.9] [Fig.9] illustrates a method for determining a value of a criterion for choosing an item to be handled; and
[0033] [Fig. 10] [Fig. 10] illustrates a table associating scores with coefficients. DETAILED DESCRIPTION OF THE INVENTION
[0034] The principle of the invention is explained with the aid of figures 1 to 9.
[0035] The context in which the invention is situated is illustrated using the example of the [Fig.l], which represents a portion of a FAC article processing installation comprising an SL sorting line equipped with a SYS system for controlling the line. sorting.
[0036] The sorting line SL comprises a device for individual handling of articles here consisting of a robotic manipulator arm MA equipped with a gripping head GH, and a conveyor belt CB moving articles arriving in the direction Dir represented by an arrow. The gripping head GH can be equipped with suction cups SC connected to a pumping system, each suction cup can be controlled individually so as to apply a vacuum to the surface of an article to be grasped. The articles can arrive in bulk in a work zone WZ from the right of the figure, the manipulator arm being responsible for unpacking them, that is to say grasping them one by one to place them in order on the conveyor belt CB towards the left of the figure.The SYS system for controlling the sorting line comprises a digital camera CAM and a digital computer CALC comprising a computer memory functionally connected to the digital camera CAM and to the manipulator arm MA. The function of the manipulator arm MA is to individually manipulate IT articles located in the work area to move them from this area to the conveyor belt CB.
[0037] [Fig.2] schematically represents a configuration of articles to be handled by the manipulator arm, corresponding to a sectional view of the work zone WZ. [Fig.2] illustrates various situations that the control system must face when it tries to optimize the choice of the article to be chosen for the next action of the manipulator arm.
[0038] In this example, article A is hidden from the camera CAM by article B and therefore cannot be taken into account in the calculations of the calculator CALC, based on images acquired by this camera, even though it might have been preferable to process this article as a priority, being the closest to the conveyor belt CB. Article C is superimposed on article D, the latter therefore not reasonably being able to be chosen as the next article to be handled due to the risk of causing a handling incident such as an article being dropped during an attempt at movement by the manipulator arm. Article F partially covers article E and has an upper surface inclined relative to the work plane WP of the work zone WZ on which the articles rest. The fact that this upper surface is inclined may cause difficulties in gripping by the manipulator arm.
[0039] The objective of the invention is to allow easy adaptation of the control of the manipulator arm to the objectives of the sorting line operator and to the characteristics of the articles IT to be sorted, taking into account the various situations with which the control system may be confronted. To this end, an object of the invention is a method 100 for individual manipulation of the articles to be sorted arriving together in the work area WA.
[0040] [Fig. 3] illustrates the method 100, which comprises a step S10 consisting of acquiring at least (i) a digital color image NumCol of a scene encompassing the working area WZ of the manipulator arm MA and (ii) a pair of digital images NumDepth intended to form a depth map of said scene according to known principles of photogrammetry, by means of the digital camera CAM, in which one or more articles IT, intended to be manipulated individually by means of the manipulator arm, are visible. The images are stored in a computer memory of the computer CALC.
[0041] In a step S20, on the basis of the digital color image, the computer CALC identifies visible items VisIT in the image, for example by means of a neural network by carrying out a segmentation operation according to conventional image analysis methods.
[0042] In a step S30, the digital computer CALC forms, according to conventional photogrammetry methods, a depth map MAP3D of the scene on the basis of the images of the pair of digital images provided to form a depth map of said scene. A greater number of images could be used, the essential thing being to be able to generate a map representative of the positioning of the articles of the work area in a three-dimensional space. The surfaces of the articles visible on the images can then be represented, for example, by a cloud of points each located in the three-dimensional space.
[0043] At this step S30, a correspondence is established between the identified visible articles and elements of the depth map such as volumes defined by the point cloud. In this way, the identified visible articles can be characterized by calculations carried out on the basis of the point cloud, that is to say on the basis of the topographic map. For example, an article in the form of a parallelepiped box can be characterized by its lateral dimensions as seen by the digital camera CAM as well as, for example, by its height (in this document, the term "height" of an article means the height of its upper surface, that is to say the distance separating its upper surface from the work plane, as illustrated by the height h in [Fig. 5] in B), or the inclination of its upper surface relative to the work plane WP of the work area on which it rests.
[0044] In a step S40, the digital computer CALC determines the pickability Y / N of each of the identified visible articles, on the basis of one or more geometric criteria deduced from the depth map, and possibly from characteristics of the manipulator arm. This involves determining whether or not an article is considered pickable, i.e. as being able to be handled by the manipulator arm, according to predetermined criteria aimed at limiting the risks of degradation of the quality sorting, for example by collisions or involuntary release of a seized item. The choice of criteria is the responsibility of the operator, who will adapt it according to the type of item to be sorted, the manipulator arm, the installation in which the work area and the manipulator arm are integrated, for example for the management of non-pickable items, and its objectives. The pickability of an item is an attribute indicating the ability of an item to be moved by the manipulator arm.
[0045] A first criterion is that of dimensions: depending on the capacities of the manipulator arm, not all articles can be reliably grasped, and articles exceeding the capacities of the manipulator arm will be considered ungraspable. These may, for example, be articles that are too small or too narrow to be grasped by one or more suction cups of a manipulator arm. An article with a volume exceeding a volume threshold defined by the operator may also be considered ungraspable.
[0046] A second criterion is that of the height of the article (to be distinguished from its thickness), and more specifically the vertical positioning of its upper surface. Indeed, articles whose height exceeds the maximum stroke of the manipulator arm along the vertical cannot be grasped.
[0047] A third criterion is the slope of the upper surface of the article. Indeed, when, for example, the manipulator arm is designed to grip an article by means of the gripper head GH, flat and provided with a plurality of suction cups, and this head cannot tilt relative to the work surface (limitation due to the degrees of freedom of the manipulator arm), the suction cups will only be able to come into contact so as to apply a vacuum to the surface of the article, and therefore grip this article, if the slope of the surface does not exceed a certain threshold, depending on the size and deformability of the suction cups. If the slope exceeds this threshold (25° for example), the article will be considered ungrippable due to a high risk of the article being released during its movement by the manipulator arm. [Fig. 6] illustrates these two situations.
[0048] A fourth criterion is based on the possibility or not of assigning a gripper configuration to the article in question, on the basis of the visible part of the article to be moved and its location in the set of articles, possibly partially covered by one or more other articles. In the case of the example of the gripper defined above, if the control system is unable to calculate a configuration defined by a position of the head and a choice of suction cup(s) which must simultaneously apply a vacuum to a surface of an article, then this article is considered ungrippable. For illustration, reference may be made to [Fig. 6] and the corresponding passage of the description. Of course, this criterion depends on the equipment for gripping and moving the articles used, and each system integrating this equipment uses its own calculation method for determining a gripping configuration. An example is given by [Fig. 6].
[0049] A fifth and a sixth criteria will be explained in connection with figures 8 and 9.
[0050] At a step S50, the digital calculator CALC calculates a takeability score SCOR, preferably only for each of the items considered to be pickable at the end of step S40. The choice of the item to be picked up in the scene captured in step S10 is determined by this pickability score, the item with the highest score being chosen for the next item manipulation by the manipulator arm.
[0051] The pickability score of an item is made up of a combination between a first score representative of the location of the item in the scene and at least a second score representative of a quality of grasping of the item, each of the scores being assigned a respective weighting coefficient.
[0052] The first score, calculated on the basis of the location of the article considered in the scene, corresponds to a criterion intended to characterize the speed of transport of the article by the manipulator arm. The first score will be calculated so as to have a value that is all the greater the shorter a distance Dist, separating the article from the location where it must be deposited by the manipulator arm, is.
[0053] [Fig.4] illustrates this distance D in the configuration of [Fig.l], with articles H and G to be deposited in the deposit zone DZ of the work zone WZ. The distance Dist can be defined as the distance between the center of an article and the deposit zone DZ.
[0054] The weighting coefficient assigned to the first score will have a value that is all the greater as the operator gives importance to the flow rate of the articles to be handled: moving as a priority the articles closest to their arrival location makes it possible to minimize the movements of the manipulator arm, which makes it possible to maximize the number of articles processed and thus maximize the flow rate of articles.
[0055] It is possible to integrate into the first score other elements having a direct or indirect impact on the flow rate of the articles. A first example of such an element is a trajectory calculated for the manipulation arm, trajectory which can be affected by the presence of an obstacle to the movement of the arm (stack of articles for example). The length of the necessary detour or the starting position of the manipulation arm can optionally be taken into account to refine the first score. A second example of an element likely to be taken into account for the calculation of the first score is a surface of the article which, if it is large and the article is moved, makes it possible to clear the scene and make visible articles hidden by this package, ultimately allowing better maximization of the flow rate of the articles.However, the first score will be considered representative of the location of the item in the scene because it is the predominant criterion even if elements such as those mentioned in this paragraph are integrated into the calculation of the first score.
[0056] The at least one second score may represent one or a plurality of geo-criteria metrics or operational characteristics characterizing the expected quality of the gripping of an article by the manipulator arm. It is possible to consider only one second score corresponding to a single criterion, but it is preferable to consider a plurality of second scores corresponding to as many criteria which, taken together, characterize the expected quality of the gripping of an article by the manipulator arm. Among these criteria, we can cite the height and the slope of the upper surface of the article, the trajectory that the manipulator arm will have to follow to move the article, the superposition between the article considered and another article placed on it, or the configuration of the gripper calculated to grip the article.
[0057] The weighting coefficients assigned to the second scores will have values that are all the greater as the operator gives importance to the quality of capture of the articles to be handled.
[0058] The scores considered are dimensionless numbers each corresponding to one of the criteria mentioned above, and can for example take values between 0 and 1, according to correspondences with the quantities considered (height, dimensions, slope, number of suction cups, etc.) established by the operator. It is also possible for the operator to assign values lower than 0 if he wishes to give penalties to certain criteria in particular situations, for example, in the case where an article is covered by two or more other articles, or if the general configuration of the scene makes an article particularly difficult to move because it is surrounded by higher articles.
[0059] If we take the example of a second score based on the slope of the upper surface of the article considered, a second maximum score of 1 could correspond to an upper surface of the article parallel to the work plane, a second score of 0 could correspond to all the angles of inclination greater than a threshold indicated by the operator and depending on the characteristics of the gripping head used, scores increasing between 0 and 1 being assigned to angles decreasing from the threshold to 0°. Figures 7 to 9 illustrate the assignment of second scores to other criteria.
[0060] Each pickable item is assigned a pickability score based on a combination of the first score and the second score(s), each weighted by their respective weighting coefficients. This combination may be a simple weighted sum of the scores. As explained above, the item with the highest score is chosen for the next item handling by the manipulator arm. The values of the coefficients, chosen by the sorting line operator, are decisive for the pickability score obtained. The pickability SCOR score can thus be calculated according to equation Eq. 1: [°°61] SCOR = C151 + £^^2^2 /
[0062]
[0063]
[0064]
[0065]
[0066]
[0067] where SI represents the first score, representative of the location of the item, and Cl its associated weighting coefficient, and the S2i represent the second scores, each representative of one of n geometric or operational criteria characterizing the expected quality of the grasping of an item by the manipulator arm, and C2i their associated weighting coefficients. If the operator wishes to give priority to the flow rate of the sorting line, then he will give a high value to the weighting coefficient Cl assigned to the first score in comparison with the weighting coefficients C2i assigned to the second scores. If the operator wishes to prioritize the quality of item capture, then he will give high values to the C2i weighting coefficients assigned to the second scores in comparison with the Cl weighting coefficient assigned to the first score. In addition, the operator can also adjust the weighting coefficients assigned to the second scores relative to each other, depending on the characteristics of the items to be sorted or the individual item handling device. Alternatively, the second S2i scores can be combined, for example by means of a weighted sum, so as to obtain a second overall score S2, the operator then only having to adjust two coefficients: the weighting coefficient Cl of the first score and a second overall weighting coefficient C2 associated with the second overall score S2, or even more simply a ratio between these two coefficients, both methods allowing a relative adjustment of the values of the first weighting coefficient and the second overall weighting coefficient. The SCOR score can be calculated according to the equation Eq.2: SC OR = C LS 1 + C2S2 Eq.2 where S? = S2r d rcPæscntc the second overall score and C2 the coefficient of associated overall weighting.
[0068] This alternative provides the operator with a very simple means of control to promote either the flow rate of articles or the quality of entry of articles.
[0069] Whatever the method used to adjust the weighting coefficients, a pickability score is calculated on the basis of a weighted sum between a first score representative of the location of the article considered in the scene and at least one second score representative of a quality of capture of the article considered, each of the scores being assigned a respective weighting coefficient. These data can be stored in the form of a table Tab in a computer memory, comprising for example the scores SI, S21, S22, S2 associated respectively with the coefficients C1, C21, C22, C2, for the case where n=2. [Fig.10] illustrates such a table.
[0070] It is possible to calculate the usability scores for each of the identified visible items and to continue the process on this basis, but to limit the calculation of the scores to Restricting the number of items considered to be retrievable to only those items that are considered retrievable makes it possible to limit the amount of calculations to be carried out. This last point is all the more important since it is a process that takes place in real time, the calculations having to be carried out sufficiently quickly by the CALC digital calculator so as not to slow down the processing of items on the sorting line and therefore to maximise the flow rate of items on this line.
[0071] Each of the weighting coefficients of the criteria selected to be assigned to the first score and / or to the second scores can be adjusted individually by the operator, for example during a preliminary step S05 of initialization of the sorting line control system. An individual adjustment of the coefficients allows great precision in the adjustment of the calculation of the score. These coefficients are stored in a computer memory of the control system and will be retrieved there for the calculation of the SCOR score of step S50.
[0072] At a step S60, in response to the calculation of the pickability score, the sorting line control system sends a command signal COM to the individual item handling device so as to handle an item to be handled. More specifically, the item with the highest pickability score is chosen to be handled. Following this command sent to the handling device, the manipulator arm MA in this example, a new cycle comprising steps S10 to S60 is initiated in order to determine the next item to be handled. Alternatively, the scores could be calculated in such a way that the item to be handled preferably is the one with the lowest non-pickability score, the non-pickability score being worth (1-pickability score). In this case, it is the item with the lowest non-pickability score that would be chosen to be handled.
[0073] Figures 5 to 9 illustrate in a concrete manner how the criteria mentioned above can be represented by a numerical quantity for the calculation of the second scores.
[0074] [Fig.5] illustrates in A) the first criterion of pickability: in a top view of the work area WZ, an article I is seen as rectangular, with dimensions L and W, deduced from the digital color image and the depth map according to conventional image processing techniques. These dimensions also make it possible to calculate the surface area of the article. These dimensions and / or this surface area can constitute the first criterion of pickability. A study of the depth map also makes it possible to determine a thickness of the article: by studying the height of the points surrounding the article identified in the digital color image, it is possible to determine whether it rests directly on the work surface, in which case the height of the article is equal to its thickness, or on another article, in which case the thickness of the article is equal to the height of the article from which the height of the article on which it rests is subtracted.
[0075] [Fig.5] illustrates in B) the second criterion of usability: a sectional view of the work zone shows the height h of the upper surface of an article J placed on the work plane WP of the work zone WZ, considering that the camera CAM is located at a height H above the work plane WP. The height h can be deduced from the height H and the distance between the camera CAM and the upper surface of the article C. This height is of course found in the depth map of the scene.
[0076] [Fig.5] illustrates in C) the third criterion of pickability: the slope SI of the upper surface of the article F supported by the work plane WP and the article E. This slope can be calculated for example by considering two bands of pixels parallel to the estimated sides of the article on the basis of the digital color image, forming a cross in the center of the article. A linear regression on the heights of each of these bands makes it possible to calculate the 2 angles of the slope SI of the article with respect to each of the edges of the article. These angles make it possible to determine, depending on the characteristics of the gripper and the suction cups with which it is equipped, whether this gripper can actually grasp the article, as illustrated by [Fig.6], and therefore whether the article is pickable or not.
[0077] Figures 6 and 7 illustrate the influence of the item capture system in determining which items to capture.
[0078] [Fig.6] illustrates in A) a first situation in which the slope of the upper surface of the article is sufficiently low and the suction cups SC sufficiently deformable so that they can adapt to the surface of the article. In the second situation illustrated in B), the adaptation capabilities of the suction cups are not sufficient to apply a vacuum to the upper surface of the object. In the second situation, the article must be considered as not suitable for gripping.
[0079] [Fig.7] illustrates the fourth gripping criterion as well as the establishment of a second corresponding gripping score, with here four different configurations A) to D) for gripping an article, respectively IT1 to IT4, by means of an adapted number of suction cups SC. In each of these configurations, the barycenter of the activated suction cups is placed in the middle of the gripped article, six suction cups being activated in A), four suction cups in B), two suction cups in C) and one suction cup in D). The dotted lines indicate the portion of the gripping head GH superimposed on the gripped article, only the dotted suction cups being activated.Among the rules to be applied to determine a configuration, we can cite the following: each activated suction cup (connected to the pumping system and sucking air) must completely cover a mask (representation of an article defined by a method of segmentation of the colored digital image) of the article to be grasped, the suction cups activated but not used to grasp the targeted article must not collide with another article, if several configurations are possible, the one involving the most suction cups is chosen to ensure the . best possible capture. If the CALC calculator fails to determine a configuration such as those illustrated in [Fig.7], then the item will be considered unpickable.
[0080] When the article is considered to be pickable, the CALC calculator can calculate a second pickability score representative of a quality of gripping of the article on the basis of the determined gripping configuration, on the basis of rules defined by the operator. The latter can for example grant a maximum score of 1 when the configuration chosen uses all six suction cups of the gripping head to grip the article (as in A) of [Fig.7]), and reduce the score progressively with the reduction in the number of suction cups used.
[0081] [Fig.8] and [Fig.2] illustrate a fifth pickability criterion, with a situation in which items C and D are superimposed on each other. In order to calculate a pickability score for each of these items, the CALC calculator will be able to determine which item is above the other by exploiting the digital color image and the depth map: the upper surface of each item is conventionally approximated by a plane of general equation z=a.x+b.y+cst (a, b and is being constants and x, y and z three coordinates in an orthonormal three-dimensional frame), which makes it possible to determine its slope and its height (median, average, height at any point). When the masks of the two items intersect, which is the case of the masks Mask(C) and Mask(D) of packages C and D defined by a segmentation operation of the NumCol image, as illustrated by [Fig.8], the CALC calculator engages an algorithm to determine which item is located above the other.
[0082] More specifically, the average heights of the upper surfaces of the articles are calculated for the intersection area of the respective masks of the two articles from the two equations of the plane approximation. The article with the higher average height is considered to be above the one with the lower average height.
[0083] In the example of figures 2 and 8, article C is above article D. As such, and concerning the superposition criterion, a second low score (possibly 0) will be assigned to article D and a second high score (1 for example) to article C.
[0084] [Fig.9] illustrates a sixth criterion of usability with the calculation of a second score taking into account the environment of an article in the work zone, influencing the trajectory of the manipulator arm to move the article. Thus, a zone Z(IT) is defined so as to frame an IT article identified in a depth map of a scene of the work zone. Peripheral zones ZI to Z5 are defined on the periphery of the zone Z(IT). The maximum heights reached by articles present in each of the zones ZI to Z5 as well as the height of the upper surface of the IT article are deduced from the depth map. The information on these maximum heights and their respective membership in zones ZI to Z5 constitutes a means of identifying possible obstacles around the IT article and will allow the CALC calculator to determine a trajectory of the robotic arm avoiding collisions by picking up the article or by placing it at its destination point. This information can also serve as a basis for calculating a second score, for example determined as higher the smaller the number of maximum heights exceeding the height of the IT article, a score which can take values between 0 and 1, or lower the more the trajectory of the robotic arm passes through zones containing articles of high heights.
[0085] Of course, the invention is not limited to the embodiments described above and variant embodiments can be made without departing from the scope of the invention.
Claims
Claims
1. Method (400) for handling articles by means of a device (MA) for individual handling of the articles, comprising the steps of: - acquiring (S 10) a digital color image (NumCol) of a scene comprising articles to be handled in a handling area and a pair of digital images (NumDepth) provided to form a depth map of said scene; - on the basis of the digital color image, identifying (S20) visible articles (VisIT) by means of a digital computer (CALC); - on the basis of the pair of digital images (NumDepth), forming (S30) a depth map (MAPdepth) of the scene, and associating the identified visible articles with elements of the depth map;- on the basis of the digital color image (NumCol) and the depth map (MAPdepth), calculating (S50), for at least some of the identified visible articles (VisIT), a pickability score (SCOR) on the basis of a combination between a first score (SI) representative of the location of the article considered in the scene and at least one second score (S21, S22) representative of a quality of grasping of the article considered, each of the first score and the at least one second score being assigned a respective weighting coefficient (Cl, C21, C22); and - in response to the pickability score (SCOR), controlling (S60) the device (MA) for individual manipulation of the articles so as to manipulate one of the visible articles.;
2. Method according to claim 1, in which the combination is a weighted sum of the first score (SI) and the at least one second score (S21, S22) assigned their respective weighting coefficients (Cl, C21, C22).
3. Method according to claim 1 or 2, further comprising a preliminary step (S05) of adjusting the respective weighting coefficients of the first score and of the at least one second score, so as to favor either a speed of flow of the articles or the quality of capture of each of the visible articles identified.
4. Method according to any one of claims 1 to 3, in which the pickability score (SCOR) is determined on the basis of a combination between the first score (SI) representative of the location of the article considered in the scene and at least two second scores (S21, S22) each representative of a quality of capture of the article considered, in which the takeability score (SCOR) is calculated by a combination of the first score (SI) assigned its first weighting coefficient (Cl) and a second overall score (S2) representative of a combination between the at least two second scores (S21, S22) each assigned their respective weighting coefficient (C21, C22), the second score (S2) being assigned a second overall weighting coefficient (C2).
5. Method according to claim 4, comprising a preliminary step (S05) of initializing a system for controlling the sorting line comprising a relative adjustment of the values of the first weighting coefficient and the second global weighting coefficient (C2).
6. Method according to claim 4 or 5, in which the combination between the at least two second scores (S21, S22) is a weighted sum of the at least two second scores (S21, S22) each affected by their respective weighting coefficient (C21, C22).
7. A method according to any one of claims 1 to 6, further comprising a step of determining (S40) a pickability (Y / N) of the identified visible articles, the articles determined as pickable constituting said at least a portion of the identified visible articles.
8. Method according to claim 7, the pickability of the identified visible articles being based on at least one criterion chosen from criteria respectively representative of dimensions, a volume, a height and a slope of an article considered, an accessibility of the article considered by the individual manipulation device of the articles, and the possibility of attributing a gripper configuration to the article considered.
9. The method of claim 8, wherein the device for individually handling the articles is a robotic manipulator arm (MA) comprising a head provided with a plurality of suction cups capable of individually applying a vacuum to a surface of an article, the gripper configuration being defined by a coordinate of a gripping point, an orientation of the head, and a suction cup combination to be used.
10. A method according to any one of claims 1 to 9, wherein the combination between the first score and the at least one second score is a linear combination of the first score and the second score each weighted by their respective weighting coefficient.
11. A method according to any one of claims 1 to 10, the at least a second score, representative of a quality of gripping of the article considered, is calculated on the basis of a criterion representative of a quality of gripping of an article by the individual article handling device and chosen from a height, a surface and a slope of the article considered, a gripper configuration assigned to the article considered, heights of articles surrounding the article considered, and a trajectory calculated to move the article considered from the handling zone to a destination zone.
12. Installation (FAC) for sorting articles configured to implement the article processing method according to any one of claims 1 to 11, comprising a device (MA) for individual manipulation of articles.
Citation Information
Patent Citations
Device for supplying flat items for a bucket conveyor
FR3019068B1
Installation for the Separation and Individualization of Heterogeneous Postal Items
FR3031046A1
Autonomous unknown object pick and place
US20200316782A1
Singulation of arbitrary mixed items
US20210122054A1
Method and computing systems for performing object detection
US20220366694A1
