Product sorting method and apparatus and computer-readable storage medium
The method addresses the challenge of determining stacking relationships between items in boxes by using image analysis and machine learning to identify contact and tilt tendencies, enhancing the success rate of automated picking and reducing damage.
Patent Information
- Application Number
- JP2025503144
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-07-21
- Filing Date
- 2023-02-07
- Publication Date
- 2025-08-20
AI Technical Summary
The challenge of determining stacking relationships between items in boxes to prevent damage during automated picking, particularly when items are stacked vertically or horizontally, leading to incorrect picking orders and potential damage to upper items.
A method and apparatus that determine proximity and constraint relationships between items based on image analysis, inclination angles, and height, using machine learning to identify contact and tilt tendencies, and generate a picking order to minimize damage during automated picking.
Improves the success rate of automated picking by accurately determining the picking order, reducing the risk of items falling and being damaged, and optimizing the sequence to handle complex stacking scenarios.
Smart Images
Figure 2025527159000001_ABST
Abstract
Description
[Technical Field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims the benefit of priority from Chinese Patent Application No. 202210860322.8, filed on July 21, 2022, the entire contents of which are hereby incorporated by reference into this application.
[0002] The present disclosure relates to the field of warehousing and logistics, and in particular to a method and apparatus for picking items, and a computer-readable storage medium. [Background technology]
[0003] With the popularity of online shopping, product distribution is becoming more just-in-time and multi-variant. The types and quantities of goods to be sorted in warehouses and distribution centers are increasing rapidly, and the workload of sorting in distribution warehouses is becoming increasingly heavy. To meet the demand for large-scale sorting, manual sorting is no longer sufficient, and automated sorting has become the key to improving sorting efficiency.
[0004] Items may be stacked in boxes, which means that one item may be covered by another. If the system does not know the stacking relationship between the items, it may result in an incorrect picking order. For example, it is difficult to pick up an item that is covered under another item. In addition, picking up a lower item first may cause an upper item to fall, thereby damaging it. Summary of the Invention [Means for solving the problem]
[0005] According to a first aspect of the present disclosure, there is provided an item picking method comprising the steps of: acquiring picking information for a plurality of items in a box, the picking information comprising picking surfaces and picking points of the plurality of items and an image of the box in which the plurality of items are located; determining proximity relationships between the plurality of items based on the image of the box in which the plurality of items are located; determining constraint relationships between the plurality of items based on the proximity relationships between the plurality of items and at least one of tilt angles or heights of the plurality of items, wherein the tilt angles of the plurality of items are determined based on the picking surfaces and picking points of the plurality of items and the heights of the plurality of items are determined based on the image of the box in which the plurality of items are located; and determining a picking order for the plurality of items based on the constraint relationships.
[0006] In some embodiments, determining a proximity relationship between the plurality of items based on an image of a box in which the plurality of items are located comprises determining that two items of the plurality of items are proximate items to each other if two items of the plurality of items are in contact with each other, and determining that two other items of the plurality of items are proximate items to each other if one item of the plurality of items is in contact with two other items of the plurality of items.
[0007] In some embodiments, determining a constraint relationship between the plurality of items comprises determining that an item has a constraint on a neighboring item if the height of one of the plurality of items is greater than the height of its neighboring item.
[0008] In some embodiments, determining a constraint relationship between a plurality of items comprises grouping the plurality of items based on the inclination angles of the plurality of items, and if an item and its neighboring items belong to the same group, determining a constraint relationship between the item and the neighboring items based on the inclination angles of the item and the neighboring items.
[0009] In some embodiments, when an item and its adjacent items belong to the same group, determining a constraint relationship between the item and its adjacent items based on their inclination angles includes determining a first picking direction based on the area of the picking surfaces of the items belonging to the same group, where the first picking direction indicates that a first picking order of the items belongs to the same group; determining a second picking direction for the item and its adjacent items based on the picking points of the item and its adjacent items, where the second picking direction indicates a second picking order of the item and its adjacent items; and determining a constraint relationship between the item and its adjacent items based on the first picking direction and the second picking direction.
[0010] In some embodiments, determining a first picking direction based on the area of the picking surfaces of items belonging to the same group includes calculating a first vector as the first picking direction if the area ratio of the largest picking surface to the smallest picking surface among the picking surfaces of items belonging to the same group exceeds a ratio threshold, wherein the first vector represents a direction from a picking point on the largest picking surface to a picking point on the smallest picking surface.
[0011] In some embodiments, the step of determining the first picking direction based on the area of the picking surfaces of the items belonging to the same group comprises the steps of: calculating a second vector based on the inclination angle of the items belonging to the same group if the area ratio of the largest picking surface to the smallest picking surface among the picking surfaces of the items belonging to the same group does not exceed a ratio threshold, wherein the second vector represents that the entire inclination direction of the items belongs to the same group; determining the first vector as the first picking direction if the angle between the first vector and the second vector does not exceed a first threshold; and determining the opposite direction of the first vector as the first picking direction if the angle between the first vector and the second vector exceeds the first threshold.
[0012] In some embodiments, the second picking direction is from a picking point on the picking surface of the item to a picking point on the picking surface of an adjacent item.
[0013] In some embodiments, determining a constraint relationship between the item and a proximate item based on the first picking direction and the second picking direction includes: Determining that the item has a constraint relative to a nearby item if the angle between the first picking direction and the second picking direction does not exceed a second threshold.
[0014] In some embodiments, determining a constraint relationship between a plurality of items comprises determining that an item has a constraint on a neighboring item if the item and the neighboring item do not belong to the same group and the height of the item is greater than the height of the neighboring item.
[0015] In some embodiments, the plurality of items have a plurality of picking surfaces, and the step of determining a proximity relationship between the plurality of items based on an image of a box in which the plurality of items are located comprises the steps of: determining two picking surfaces of the plurality of picking surfaces as proximate picking surfaces when the two picking surfaces contact each other; and determining two other picking surfaces of the plurality of picking surfaces as proximate picking surfaces when one picking surface of the plurality of picking surfaces contacts two other picking surfaces of the plurality of picking surfaces.
[0016] In some embodiments, determining a constraint relationship between the plurality of items comprises determining that a picking surface has a constraint with respect to an adjacent picking surface if the height of the picking surface is greater than the height of the adjacent picking surface.
[0017] In some embodiments, the inclination angle of the item comprises an inclination angle of a picking surface of the item, and determining the constraint relationship between the plurality of items comprises grouping the picking surfaces of the plurality of items based on the inclination angles of the picking surfaces, and if a picking surface and its adjacent picking surface belong to the same group, determining a constraint relationship between the picking surface and the adjacent picking surface based on their inclination angles.
[0018] In some embodiments, when a picking surface and its adjacent picking surface belong to the same group, determining a constraint relationship between the picking surface and the adjacent picking surface based on their inclination angles comprises determining a first picking direction based on areas of the picking surfaces belonging to the same group, where the first picking direction indicates a first picking order of the picking surfaces belonging to the same group; determining a second picking direction for the picking surface and the adjacent picking surface based on picking points of the picking surface and the adjacent picking surface, where the second picking direction indicates a second picking order of the picking surface and the adjacent picking surface; and determining a constraint relationship between the picking surface and the adjacent picking surface based on the first picking direction and the second picking direction.
[0019] In some embodiments, determining a first picking direction based on the areas of picking surfaces belonging to the same group comprises calculating a first vector as the first picking direction if the area ratio of the largest picking surface to the smallest picking surface among the picking surfaces belonging to the same group exceeds a ratio threshold, wherein the first vector represents a direction from a picking point on the largest picking surface to a picking point on the smallest picking surface.
[0020] In some embodiments, the step of determining a first picking direction based on the area of the picking surfaces of items belonging to the same group includes the steps of: calculating a second vector based on the inclination angle of the picking surfaces belonging to the same group if the ratio of the largest picking surface to the smallest picking surface among the picking surfaces belonging to the same group does not exceed a ratio threshold, wherein the second vector represents that the entire inclination direction of the picking surfaces belongs to the same group; determining the first vector as the first picking direction if the angle between the first vector and the second vector does not exceed a third threshold; and determining the opposite direction of the first vector as the first picking direction if the angle between the first vector and the second vector exceeds the third threshold.
[0021] In some embodiments, the second picking direction represents a direction from a picking point on a picking surface to a picking point on an adjacent picking surface.
[0022] In some embodiments, determining a constraint relationship between the picking plane and the adjacent picking plane based on the first picking direction and the second picking direction comprises determining that the picking plane has a constraint with respect to the adjacent picking plane if the angle between the first picking direction and the second picking direction does not exceed a fourth threshold.
[0023] In some embodiments, determining a constraint relationship between the plurality of items comprises determining that a picking surface has a constraint on a proximal picking surface if the picking surface and the proximal picking surface do not belong to the same group and if the height of the picking surface is greater than the height of the proximal picking surface.
[0024] In some embodiments, determining a picking order for the plurality of items based on the constraint relationships comprises generating a sorted result based on the picking information, the sorted result representing a difficulty of picking the plurality of items, and determining a picking order for the plurality of items based on the sorted result and the constraint relationships between the plurality of items.
[0025] According to a second aspect of the present disclosure, there is provided an item picking device comprising: an acquisition module for acquiring picking information of a plurality of items in a box, the picking information comprising picking surfaces and picking points of the plurality of items and an image of the box in which the plurality of items are located; a proximity relationship determination module 92 for determining a proximity relationship between the plurality of items according to the image of the box in which the plurality of items are located; a constraint relationship determination module for determining a constraint relationship between the plurality of items based on the proximity relationships between the plurality of items and at least one of an inclination angle or a height of the plurality of items, wherein the inclination angle of the plurality of items is determined based on the picking surfaces and picking points of the plurality of items and the height of the plurality of items is determined based on the image of the box in which the plurality of items are located; and a picking order determination module 94 for determining a picking order for the plurality of items based on the constraint relationships.
[0026] According to a third aspect of the present disclosure, there is provided an item picking apparatus comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to perform an item picking method according to any embodiment of the present disclosure based on instructions stored in the memory.
[0027] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implements an item picking method according to any embodiment of the present disclosure.
[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the disclosure.
[0029] The present disclosure will be more clearly understood from the following detailed description when taken in conjunction with the accompanying drawings, in which: [Brief explanation of the drawings]
[0030] [Figure 1] 1 is a flowchart of a method for picking an item according to some embodiments of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram of stacked articles according to some embodiments of the present disclosure. [Figure 3] FIG. 10 is a schematic diagram of stacked articles according to another embodiment of the present disclosure. [Figure 4a] FIG. 10 is a schematic diagram of determining a first picking direction according to some embodiments of the present disclosure. [Figure 4b] FIG. 10 is a schematic diagram of determining a first picking direction according to another embodiment of the present disclosure. [Figure 4c] FIG. 10 is a schematic diagram of determining a second picking direction according to some embodiments of the present disclosure. [Figure 4d] 1 is a schematic diagram of determining constraint relationships between adjacent items according to some embodiments of the present disclosure. [Figure 5] 10 is a flowchart of generating a constraint relationship matrix according to some embodiments of the present disclosure. [Figure 6] FIG. 10 is a schematic diagram of stacked articles according to yet another embodiment of the present disclosure. [Figure 7] FIG. 10 is a schematic diagram of stacked articles according to a further embodiment of the present disclosure. [Figure 8] 10A-10C are schematic diagrams of determining constraint relationships between proximity picking surfaces according to some embodiments of the present disclosure; [Figure 9] FIG. 1 is a block diagram of an item-picking device according to some embodiments of the present disclosure. [Figure 10] FIG. 10 is a block diagram of an item-picking device according to another embodiment of the present disclosure. [Figure 11] FIG. 1 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION
[0031] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specified, the relative arrangement of components and steps, numerical expressions, and values described in these examples do not limit the scope of the present invention.
[0032] At the same time, it should be understood that for ease of illustration, the dimensions of the various parts shown in the drawings have not been drawn to scale.
[0033] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended as a limitation on the invention, its application, or uses.
[0034] Techniques, methods, and apparatus known to those skilled in the art may not be described in detail, but where appropriate, these techniques, methods, and apparatus should be considered part of this specification.
[0035] Any specific values in all examples shown and described herein should be considered as examples only, and not as limitations, and thus other examples of exemplary embodiments may have different values.
[0036] It should be noted that like reference numbers and characters are indicated by like things in the accompanying drawings, and therefore, once items are defined in the drawings, no further description is necessary in the accompanying drawings.
[0037] Items may be stacked in a box, one item on top of another. Stacking items may have an effect on picking success rates, making it difficult to achieve a firm grip on an item hidden by another. If the picking order is improper, for example, picking a lower item before picking a higher item, the upper item may fall, resulting in damage to the item. In addition to items being stacked vertically, items may lean so that the left item is partially resting on the right item, making it difficult to grasp the right item. Also, if the right item is picked first, the left item may fall and be damaged due to lack of support.
[0038] In the related art, it is difficult to determine the constraint relationships between stacked items, so the items can only be picked according to the order in which they are identified, which may cause damage to the items.
[0039] FIG. 1 illustrates a flowchart of an item picking method according to some embodiments of the present disclosure.
[0040] As shown in FIG. 1, the item picking method includes steps S1 to S4.
[0041] In step S1, picking information of a plurality of items in a box is obtained, where the picking information comprises picking surfaces and picking points of the items and an image of the box in which the items are located.
[0042] For example, a camera is used to capture an image of an item stacked in a box from the top of the box, i.e., taking a top view of the item, and a picking point and picking surface are calculated for the item, where the picking point is the point on the item that the robotic arm touches when gripping or sucking the item, and the picking surface is the surface on which the picking point is located.
[0043] In step S2, proximity relationships between the items are determined based on images of the boxes in which the items are located.
[0044] In some embodiments, two items are determined to be proximate items when they are in contact with each other, and when one item is in contact with two other items, the two other items are determined to be proximate items.
[0045] For example, proximal items include adjacent items and sub-adjacent items, where adjacent items are two items that are in direct contact with each other and sub-adjacent items are adjacent items of adjacent items.
[0046] FIG. 2 shows a schematic diagram of stacked articles according to some embodiments of the present disclosure.
[0047] Figure 2 is a two-dimensional side view of items "a," "b," and "c" stacked vertically in a box. As shown in Figure 2, items "a" and "b" are in contact and are therefore determined to be proximate items. Item "b" is in contact with both items "a" and "c," and there may also be a constraint relationship between items "a" and "c." Therefore, items "a" and "c" are also defined as having a proximate relationship.
[0048] The present disclosure takes into account not only the constraint relationship between two items that are in direct contact, but also the potential constraint relationship between two items that are in indirect contact, thereby looking at a wide range of possible situations and reducing dropouts and improving picking success rates based on constraint relationships.
[0049] In some embodiments, a machine learning model is used to recognize images of boxes in which items are placed and determine whether the items are touching each other.
[0050] For example, two articles are considered to be in contact with each other if they have edges or surfaces that partially or completely overlap.
[0051] In step S3, a constraint relationship between the items is determined based on a proximity relationship between the items and at least one of an inclination angle or a height of the items, where the inclination angle of the item is determined based on a picking plane and a picking point of the item, and the height of the item is determined based on an image of a box in which the item is located.
[0052] In some embodiments, the constraint relationships between the items represent stacking relationships between the items.
[0053] In some embodiments, determining a constraint relationship between the items comprises determining that an item has a constraint on a neighboring item if the height of the item is greater than the height of the neighboring item.
[0054] In Figure 2, if the height of item "a" is higher than the height of item "b", and items "a" and "b" are adjacent items, then item "a" has a constraint on b. Similarly, items "a" and "c" are also adjacent items, and item "a" is higher than c, so item "a" also has a constraint on item c. The height of an item may be the height of the surface of the item, the height of the geometric center of the item, the height of the center of gravity of the item, the height of a picking point on the item, etc., or the multiple heights described above may be comprehensively taken into consideration according to actual needs.
[0055] In some embodiments, determining a constraint relationship between the items comprises grouping a plurality of items based on the inclination angles of the plurality of items, and, if the item and its neighboring items belong to the same group, determining a constraint relationship between the item and its neighboring items based on the inclination angles of the item.
[0056] For example, items with the same tilt tendency may lean against each other. Therefore, items with the same tilt tendency are grouped together, and stacking relationships are determined for items in the same group based on their tilt angles. In addition, grouping can provide advance guidance on the picking order for groups of items with the same tilt tendency. Grouping by tilt tendency can provide a degree of error tolerance when determining the picking order for members in the same group. Even if some members deviate to some extent from the main direction, the overall consistent picking order of the group will not be affected.
[0057] In some embodiments, grouping the plurality of articles based on the tilt angle of the plurality of articles comprises grouping articles having a tilt angle within a predetermined range.
[0058] FIG. 3 shows a schematic diagram of stacked articles according to some embodiments of the present disclosure.
[0059] Figure 3 is a two-dimensional side view of items "a," "b," and "c" stacked horizontally in a box. As shown in Figure 3, items "a," "b," and "c" have similar tilt angles and can be grouped together.
[0060] In some embodiments, when an item and its neighboring items belong to the same group, determining a constraint relationship between the item and the neighboring item based on the inclination angle of the item comprises determining that the item has a constraint on the neighboring item if the angle between the direction of the inclination angle of the group to which the item belongs and the direction from the item to the neighboring item is less than a preset threshold.
[0061] For example, as shown in Figure 3, item "a" lies on a part of item "b." In the constraint relationship calculation, items "a," "b," and "c" are all tilted to the left. Item "a" is to the left of its neighboring item "b," i.e., the direction from item "a" to item "b" is from right to left, which is opposite to the tilt angle direction, so item "a" has no constraint on item "b." Conversely, the direction from item "b" to item "a" is from left to right, which is similar to the tilt angle (direction), so item "b" has a constraint on item "a."
[0062] In some embodiments, determining a constraint relationship between an item and an adjacent item based on the inclination angle of the item comprises: determining a first picking direction based on the area of the picking surface of items belonging to the same group, where the first picking direction indicates a first picking order of the items belonging to the same group; if the item and the adjacent item belong to the same group, determining a second picking direction for the item and the adjacent item based on the picking points of the item and the adjacent item, where the second picking direction indicates a second picking order of the item and the adjacent item; and determining a constraint relationship between the item and the adjacent item based on the first picking direction and the second picking direction.
[0063] In some embodiments, the tilt angle of the item is the angle between the normal vector of the picking surface of the item and the bottom surface of the box.
[0064] To represent the stacking relationship of items in the same group, the first picking direction may be determined according to the following method.
[0065] In some embodiments, when the item and the adjacent item belong to the same group, determining the first picking direction based on the area of the picking surfaces of the items belonging to the same group comprises: calculating a first vector as the first picking direction when a ratio of a maximum picking surface to a minimum picking surface among the picking surfaces of the items belonging to the same group exceeds a ratio threshold, wherein the first vector represents a direction from a picking point on the maximum picking surface to a picking point on the minimum picking surface.
[0066] FIG. 4a shows a schematic diagram of determining a first picking direction according to some embodiments of the present disclosure.
[0067] Figure 4a shows a two-dimensional side view of items "a", "b", and "c". As shown in Figure 4, items "a", "b", and "c" have similar tilt angles and are grouped together.
[0068] Assuming that only one picking surface per item is taken into consideration, the picking surfaces of items "a," "b," and "c" are sorted by area. In Figure 4a, it is assumed that the smallest picking surface is picking surface 1, the largest picking surface is picking surface 2, and the black dots are picking points. The direction from the largest picking surface to the smallest picking surface is indicated by a solid arrow in the figure. Assuming that the area ratio of picking surface 2 to picking surface 1 exceeds a ratio threshold, significant differences in the areas of the picking surfaces of items belonging to the same group are taken into account. The first picking direction is from the largest picking surface to the smallest picking surface.
[0069] If an item has multiple picking surfaces, the picking surface that is easiest to pick is selected based on the difficulty of grasping, or for each picking surface of the item, the angle between the inclination angle of this picking surface and the inclination angle of the item is calculated, and a picking surface with an angle smaller than a preset threshold, i.e., a picking surface that matches the inclination tendency of the item, is selected.
[0070] Generally, the area of the picking surface exposed on the top item is larger, and the area of the picking surface exposed on the covered item is smaller. The area of the picking surface can reflect the stacking relationship between the items, and therefore, based on the area of the picking surface, a first picking direction for items belonging to the same group can be determined.
[0071] In some embodiments, determining a first picking direction for items belonging to the same group based on the area of the picking surfaces of the items belonging to the same group comprises: calculating a second vector based on the inclination angle of the items belonging to the same group if the ratio of the largest picking surface to the smallest picking surface among the picking surfaces of the items belonging to the same group does not exceed a ratio threshold, wherein the second vector represents that the entire inclination direction of the items belongs to the same group; determining the first vector as the first picking direction if the angle between the first vector and the second vector does not exceed a first threshold; and determining the opposite direction of the first vector as the first picking direction if the angle between the first vector and the second vector exceeds the first threshold.
[0072] For example, if there is no significant difference in the areas of the picking surfaces of items belonging to the same group, it is determined whether the direction of the "vector connecting the picking points on the largest picking surface to the picking points on the smallest picking surface" is similar to the overall tilt direction of the group. If they are similar, the picking order of the group members is in the direction of the vector connecting the picking points on the largest picking surface to the picking points on the smallest picking surface. If they are not similar, the picking order of the group members is in the opposite direction to the vector connecting the picking points on the largest picking surface to the picking points on the smallest picking surface.
[0073] In addition, the inclination angle of items belonging to the same group can be selected as the overall inclination direction of the group, or the average or median inclination angle of the group can be calculated as the overall inclination direction of the group, which is not particularly limited in this specification.
[0074] FIG. 4b shows a schematic diagram of determining a first picking direction according to another embodiment of the present disclosure.
[0075] As shown in Figure 4b, assuming that the ratio of picking plane 1 to picking plane 2 does not exceed the ratio threshold, it is also necessary to compare the first vector with the second vector (i.e., the overall tilt direction of the group of items). Assuming that the first threshold is 90 degrees, it is clear that the angle between the first vector and the second vector in Figure 4b does not exceed 90 degrees. Therefore, the first picking direction is the direction of the first vector. Conversely, if the angle between the first vector and the second vector exceeds 90 degrees, the first picking direction is the opposite direction of the first vector. That is, the first picking direction selected last represents a direction closer to the overall tilt trend of the group of items, and as a result, the first picking direction can more accurately represent the stacking situation of the group.
[0076] In some embodiments, the second picking direction represents a direction from a picking point on the picking surface of an item to a picking point on the picking surface of its adjacent item.
[0077] FIG. 4c shows a schematic diagram of determining a second picking direction according to some embodiments of the present disclosure.
[0078] Assuming the current item is "b", as shown in Figure 4c, it is necessary to determine the constraint relationship between item "b" and its neighboring item c. Therefore, the second picking direction is defined as the picking point on item "b" to the picking point on item c.
[0079] In some embodiments, determining a constraint relationship between the item and a nearby item based on the first picking direction and the second picking direction comprises determining that the item has a constraint with respect to a nearby item if the angle between the first picking direction and the second picking direction does not exceed a second threshold.
[0080] FIG. 4d shows a schematic diagram of determining constraint relationships between adjacent items according to some embodiments of the present disclosure.
[0081] The first picking direction and the second picking direction are shown in Figure 4d. Assuming the second threshold is 90 degrees, in Figure 4d, the angle between the first picking direction and the second picking direction is clearly greater than the second threshold. Therefore, item "b" has no constraint on item "c." Conversely, if the angle between the first picking direction and the direction from the picking point of item "c" to the picking point of item "b" is smaller than the threshold, item "c" has a constraint on item "b."
[0082] In some embodiments, determining a constraint relationship between the items comprises determining that an item has a constraint on a neighboring item if the item and the neighboring item do not belong to the same group and the height of the item is greater than the height of the neighboring item.
[0083] For example, if two items do not have similar tilt tendencies, it is not necessary to determine the left-right stacking constraints, but only the longitudinal stacking situation. Therefore, the constraint relationship is determined based on height.
[0084] In some embodiments, determining the constraint relationship between the articles comprises generating a constraint relationship matrix based on at least one of height and tilt angle.
[0085] The values of the elements in the constraint relationship matrix adjacent_matrix are defined as follows: adjacent_matrix[i,j]=0, that is, item i and item j are not adjacent items and have no constraint relationship. adjacent_matrix[i,j]=1, ie, item i and item j are adjacent items, and item i has a constraint on item j. adjacent_matrix[i,j]=-1, ie, item i and item j are adjacent items and item j has a constraint on item i. adjacent_matrix[i,j]=2, ie, item i and item j are adjacent items but do not have a constraint relationship.
[0086] If adjacent_matrix[i,j]=1, then the corresponding adjacent_matrix[j,i]=-1.
[0087] FIG. 5 shows a flowchart of generating a constraint relationship matrix according to some embodiments of the present disclosure.
[0088] As shown in FIG. 5, the generation of the constraint relationship matrix includes steps (1) to (15).
[0089] In step (1), it is determined whether any two items have similar tilt tendencies based on the angle between the normal vector at the picking point on the picking surface of each item and the bottom of the box. Items with matching tilt tendencies are grouped together, and a first picking direction is calculated for each group. The first picking direction, i.e., the sorting direction by picking order, represents a directional space vector indicating the picking order of the items, which is the stacking order of the items.
[0090] In step (2), each current item is used as a candidate for traversing all items.
[0091] In step (3), it is determined whether a constraint relation calculation is performed for all candidate articles and their neighbors, and if two articles touch each other, they are neighbors, and the correspondence between any neighbors is stored in the neighbor set N. If so, the process proceeds to step (10), otherwise the process proceeds to step (4).
[0092] In step (4), the process continues by considering items other than the current candidate item.
[0093] In step (5), it is determined whether the candidate item i and another item j are adjacent items. If so, the process proceeds to step (6); if not, the element adjacent_matrix[i,j] in the constraint matrix is set to 0.
[0094] In step (6), it is determined whether the candidate item i and the neighboring item j belong to a group with the same gradient tendency, and if so, the process proceeds to step (7), otherwise, the process proceeds to step (8).
[0095] In step (7), it is determined whether the first picking direction of the group in which candidate item i and adjacent item j are located matches the second picking direction pointing from candidate item i to adjacent item j, and if so, the element adjacent_matrix[i,j] in the constraint matrix is set to 1; otherwise, the element adjacent_matrix[i,j] is set to -1.
[0096] In step (8), it is determined whether the height of the candidate item is equal to the height of the adjacent item, and if so, the element adjacent_matrix[i,j] in the constraint matrix is set to 2, otherwise the process proceeds to step (9).
[0097] In step (9), it is determined whether the height of the candidate is less than the height of the adjacent item, and if so, the element adjacent_matrix[i,j] in the constraint matrix is set to -1, otherwise the element adjacent_matrix[i,j] is set to 1.
[0098] In step (10), it is determined whether the constraint relationship calculations have been performed for all candidate items and their neighboring items, if so, the process is terminated, otherwise the process proceeds to step (11).
[0099] In step (11), the neighbors of the neighbors of the current candidate item are considered as lower neighbors of the current candidate item, the correspondence between the item and its lower neighbors is stored in a lower neighbor set M, and all lower neighbors of the current candidate item in set M are considered in detail.
[0100] In step (12), it is determined whether the candidate item i and its lower neighbor item j belong to a group with the same gradient tendency, if so, the process proceeds to step (13), if not, the process proceeds to step (14).
[0101] In step (13), it is determined whether the first picking direction of the group in which candidate item i and lower-level neighbor item j are located matches the second picking direction pointing from candidate item i to lower-level neighbor item j, and if so, the element adjacent_matrix[i,j] in the constraint matrix is set to 1; otherwise, the element adjacent_matrix[i,j] is set to -1.
[0102] In step (14), it is determined whether the height of candidate item i is equal to the height of lower-order neighbor item j, and if so, the element adjacent_matrix[i,j] in the constraint matrix is set to 2; if not, the process proceeds to step (15).
[0103] In step (15), it is determined whether the height of candidate item i is lower than the height of lower-order neighbor item j, and if so, the element adjacent_matrix[i,j] in the constraint matrix is set to −1, otherwise the element adjacent_matrix[i,j] is set to 1. In addition, if the element adjacent_matrix[i,j] calculated in step (5) has a value of 0, but when recalculated in step (15) has a value of 1 or −1, the original calculation result is overwritten with the current calculation result.
[0104] The present disclosure determines proximity relationships between items and identifies adjacent items that may have a constraint relationship with each other. Then, based on the proximity relationships and at least one of the height or tilt angle of the items, a stacking constraint relationship between the items is determined, so that horizontal and / or vertical stacking relationships between the items can be identified, thereby avoiding the risk of picking an item being pressed before another item being pressed, solving the problem of items being stacked on each other affecting the picking order, and improving the success rate of the robot arm in picking items.
[0105] In addition to directly determining the constraint relationships between the items, it is also possible to determine the constraint relationships between the picking surfaces of the items, and such constraint relationships reflect the constraint relationships between the items. A method for determining the constraint relationships between the picking surfaces of the items is introduced below.
[0106] The idea of determining constraint relationships between picking surfaces of items is similar to directly determining constraint relationships between items. In the above method of directly determining constraint relationships between items, each item is used as a candidate for determining proximity and constraint relationships between the items. When determining constraint relationships between picking surfaces of items, each picking surface of the item is used as a candidate for determining proximity and constraint relationships between the picking surfaces.
[0107] In some embodiments, the items comprise a plurality of picking surfaces, and determining a proximity relationship between the items based on the image of the box in which the items are located comprises determining two picking surfaces as proximate picking surfaces to each other if the two picking surfaces contact each other.
[0108] If a picking surface contacts two other picking surfaces, the other two picking surfaces are determined to be adjacent picking surfaces to each other.
[0109] FIG. 6 shows a schematic diagram of stacked articles according to some embodiments of the present disclosure.
[0110] Figure 6 is a three-dimensional side view of vertically stacked items "a" and "b." As shown in Figure 6, surface A1 of item "a" and surface B1 of item "b" are proximity picking surfaces, surface A1 of item "a" and surface A2 of item "a" are proximity picking surfaces, and surface A1 of item "a" is a proximity picking surface for both surfaces B1 and B2 of item b. That is, not only can two picking surfaces of different items be proximity picking surfaces, but two picking surfaces of the same item can also be proximity picking surfaces.
[0111] In some embodiments, a machine learning model is used to recognize images of boxes in which items are located and determine whether their picking surfaces are touching each other.
[0112] For example, two picking surfaces are considered to be in contact with each other if the two picking surfaces have edges or surfaces that partially or completely overlap.
[0113] As shown in FIG. 6, since surfaces A1 and B1 have an interface edge, surface A1 of item "a" and surface B1 of item "b" are in contact with each other, and it is determined that surface A1 of item "a" is the proximal picking surface of surface B1 of item b.
[0114] The method for determining the constraint relationship between a picking surface and its adjacent picking surface is similar to the method for determining the constraint relationship between an item and its adjacent items, which will be briefly introduced below. For the same content of these two methods, which will not be repeated here, please refer to the method for determining the constraint relationship between an item and its adjacent items.
[0115] In some embodiments, determining a constraint relationship between the items comprises determining that a picking surface has a constraint with respect to an adjacent picking surface if the height of the picking surface is greater than the height of the adjacent picking surface.
[0116] For example, in Figure 6, surface A1 is a proximal picking surface of surfaces B1 and B2, and its height is higher than that of surfaces B1 and B2. Therefore, surface A1 has a constraint on surfaces B1 and B2 and must be grasped before grasping either surface B1 or surface B2. The height of the picking surface may be the height of one edge of the picking surface, the height of the geometric center of the picking surface, the height of the picking point on the picking surface, etc., or the multiple heights described above may be comprehensively taken into consideration according to actual needs.
[0117] In some embodiments, the tilt angle of the items comprises a tilt angle of a picking surface of the items, and determining the constraint relationship between the items comprises grouping the picking surfaces of the items based on the tilt angles of the picking surfaces, and if a picking surface and its adjacent picking surfaces belong to the same group, determining the constraint relationship between the picking surface and the adjacent picking surface based on their tilt angles. The tilt angle of the picking surface is the tilt angle of a normal vector at a picking point on the picking surface relative to a bottom surface of the box.
[0118] For example, picking surfaces with the same tilt tendency may overlap each other, and therefore, picking surfaces with the same tilt tendency are grouped together, and a stacking relationship is determined from the picking surfaces in the same group based on their tilt angles.
[0119] In some embodiments, grouping the plurality of picking surfaces based on their tilt angle comprises grouping picking surfaces having tilt angles within a predetermined range.
[0120] FIG. 7 shows a schematic diagram of stacked articles according to another embodiment of the present disclosure.
[0121] In FIG. 7, for picking surfaces A1, B1, C1, A2, B2, and C2, based on their tilt angles, picking surfaces A1, B1, and C1 can be grouped together, and picking surfaces A2, B2, and C2 can be grouped together.
[0122] In some embodiments, determining the constraint relationship between a picking surface and its adjacent picking surface comprises: determining a first picking direction based on an area of the picking surface in each group, where the first picking direction indicates a first picking order of the picking surface belonging to the same group; if the picking surface and its adjacent picking surface belong to the same group, determining a second picking direction for the picking surface and its adjacent picking surface based on picking points of the picking surface and the adjacent picking surface, where the second picking direction indicates a second picking order of the picking surface and the adjacent picking surface; and determining the constraint relationship between the picking surface and its adjacent picking surface based on the first picking direction and the second picking direction, where the second picking direction represents a direction from a picking point on the picking surface to a picking point on the adjacent picking surface.
[0123] The first picking direction may be determined according to the following method for representing the stacking relationship between the picking faces in the group.
[0124] (1) If the ratio of the largest picking surface to the smallest picking surface among the picking surfaces belonging to the same group exceeds a ratio threshold, calculate a first vector as a first picking direction, wherein the first vector represents a direction from a picking point on the largest picking surface to a picking point on the smallest picking surface.
[0125] (2) If the ratio of the largest picking surface to the smallest picking surface among the picking surfaces belonging to the same group does not exceed a ratio threshold, calculating a second vector based on the inclination angle of the picking surfaces belonging to the same group, wherein the second vector represents that the entire inclination direction of the picking surfaces belongs to the same group; if the angle between the first vector and the second vector does not exceed a third threshold, determining the first vector as the first picking direction; if the angle between the first vector and the second vector exceeds the third threshold, determining the opposite direction of the first vector as the first picking direction.
[0126] In some embodiments, determining the constraint relationship between the picking plane and the adjacent picking plane based on the first picking direction and the second picking direction comprises determining that the picking plane has a constraint with respect to the adjacent picking plane if an angle between the first picking direction and the second picking direction does not exceed a fourth threshold.
[0127] FIG. 8 shows a schematic diagram of determining constraint relationships between proximity picking surfaces according to some embodiments of the present disclosure.
[0128] As shown in Figure 8, the first picking directions and second picking directions B2 to C2 of the group of picking faces A2, B2, and C2 are similar to those in Figure 4d. Assuming the second threshold is 90 degrees, the angle between the first picking direction and the second picking direction is greater than the second threshold. Therefore, picking face A2 has no constraint on picking face C2. Conversely, picking face C2 has a constraint on picking face A2.
[0129] Image recognition methods used in picking systems such as robotic arms (e.g., to recognize images and determine height, contact, etc.) employ supervised learning, which requires large amounts of annotated data. For 2D picking surface recognition methods, annotation techniques are mature and cost-effective, making manual annotation easier. According to some embodiments of the present disclosure, constraint relationships between picking surfaces are calculated for each picking surface to determine the picking order and thereby reduce costs.
[0130] In some embodiments, determining a picking order for the plurality of items based on their constraint relationships comprises generating a sorting result based on the picking information, the sorting result representing a difficulty of picking the items, and determining a picking order for the plurality of items based on the sorting result and the constraint relationships between the items.
[0131] For example, a sorting result indicating the difficulty of picking different items is generated based on influence factors that represent the difficulty of picking the items. For example, the sorting result is generated based on a weighted sum of the influence factors, or the influence factors are input into a machine learning model to obtain the sorting result. The influence factors include the area of each picking surface, the area of the largest inscribed circle of each picking surface centered at the picking point, the angle between the normal vector at the picking point and the normal vector of the bottom surface of the box, the distance from the picking point to each surface of the box, the distance from the picking point to the center of the box, and the probability that the item recognized by the machine learning model is the item to be picked.
[0132] In step S4, a picking order is determined for the plurality of items based on the constraint relationships.
[0133] For example, an item must be placed before other items that are constrained by this item in the picking order. For item i, if it is constrained by item j, then item j is before item i in the picking order, and we obtain a picking order for multiple items in this way.
[0134] In some embodiments, determining a picking order for the plurality of items based on the constraint relationship comprises generating a sorting result based on the picking information, the sorting result representing a difficulty of picking the items, and determining a picking order for the plurality of items based on the sorting result and the constraint relationship between the items.
[0135] For example, the picking order for multiple items must meet the following conditions:
[0136] (1) Constraint relationships between items must be satisfied, i.e., an item must be placed before other items that are constrained by this item in the picking sequence.
[0137] (2) Subject to the condition (1) being satisfied, the items whose sorting position cannot be determined shall be sorted according to the sorting result indicating the difficulty of picking the items. That is, in the case of a conflict between (1) and (2), it is necessary to ensure that the final item picking order satisfies the condition (1).
[0138] If the constraint relationships between items are directly obtained in step S3, multiple items are sorted when calculating the sorting results and the final picking order. If the constraint relationships between picking surfaces are calculated in step S3, multiple picking surfaces are sorted when calculating the sorting results and the final picking order.
[0139] The present disclosure first sorts items based on their difficulty in picking to obtain a sorted result. Then, based on the constraint relationship, the initial sorted result is rearranged to obtain a final picking order. In addition to satisfying the requirement of the constraint relationship between the items for the picking order, the difficulty of picking the items is also taken into consideration, which not only solves the problem that stacked items affect the gripping effect and cause item damage, but also achieves the goal of picking easier items first, thereby improving the success rate and efficiency of picking.
[0140] FIG. 9 illustrates a block diagram of an item-picking device according to some embodiments of the present disclosure.
[0141] As shown in FIG. 9, the item picking device 9 includes an acquisition module 91, a proximity relationship determination module 92, a constraint relationship determination module 93, and a picking order determination module 94.
[0142] The acquisition module 91 is configured to acquire picking information of a plurality of items in a box, the picking information including a picking surface and a picking point of the items and an image of the box in which the items are located. For example, the acquisition module 91 performs step S1 shown in FIG.
[0143] The proximity relationship determination module 92 is configured to determine proximity relationships between the items according to the images of the boxes the items are located in. For example, the proximity relationship determination module 92 performs step S2 shown in FIG.
[0144] The constraint relationship determination module 93 is configured to determine a constraint relationship between the items based on a proximity relationship between the items and at least one of an inclination angle or a height of the items, where the inclination angle of the item is determined based on a picking plane and a picking point of the item, and the height of the item is determined based on an image of a box in which the item is located. For example, the constraint relationship determination module 93 executes step S3 shown in FIG. 1.
[0145] The picking order determination module 94 is configured to determine a picking order for the plurality of items based on the constraint relationship. For example, the picking order determination module 94 executes step S4 shown in FIG.
[0146] FIG. 10 shows a block diagram of an item-picking device according to another embodiment of the present disclosure.
[0147] 10, the item picking apparatus 10 includes a memory 101 for storing an item picking method, and a processor 102 coupled to the memory 101. The processor 102 is configured to execute the item picking method according to some embodiments of the present disclosure based on instructions stored in the memory 101.
[0148] FIG. 11 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.
[0149] 11, computer system 110 may be represented in the form of a general-purpose computing device. Computer system 110 includes memory 1110, a processor 1120, and a bus 1100 connecting various system components.
[0150] The memory 1110 may comprise, for example, a system memory, a non-volatile storage medium, etc. The system memory stores, for example, an operating system, application programs, a boot loader, and other programs. The system memory may comprise a volatile storage medium, such as random access memory (RAM) and / or a cache memory. The non-volatile storage medium stores, for example, instructions for executing an item picking method according to any embodiment of the present disclosure. The non-volatile storage medium includes, but is not limited to, magnetic disk storage, optical storage, flash memory, etc.
[0151] The processor 1120 may be implemented by discrete hardware components, such as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gates, or transistors. Thus, each device, such as the determining device and decision device, may be implemented by a central processing unit (CPU) that runs instructions to perform the corresponding steps, or may be implemented by dedicated circuitry that performs the corresponding steps.
[0152] Bus 1100 may have any of a variety of bus structures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.
[0153] The computer system 110 may further include an input / output interface 1130, a network interface 1140, a storage interface 1150, etc. These interfaces 1130, 1140, 1150, the memory 1110, and the processor 1120 may be connected through a bus 1100. The input / output interface 1130 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 1140 provides a connection interface for various networked devices. The storage interface 1150 provides a connection interface for an external storage device such as a floppy disk, a flash disk, or an SD card.
[0154] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and any combination of blocks, can be implemented by computer-readable program instructions.
[0155] These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable device to create a machine, such that the instructions executed by the processor create means for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.
[0156] These computer-readable program instructions may also be stored in a computer-readable storage device and may cause a computer to operate in a particular manner to produce an article of manufacture, including instructions for implementing functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.
[0157] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements.
[0158] The item picking method and apparatus and computer-readable medium provided in the above embodiments can improve the success rate of picking items.
[0159] Up to now, the item picking method and apparatus and computer-readable storage medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art will not be described. Based on the above description, those skilled in the art can understand how to implement the technical solutions disclosed herein. [Explanation of symbols]
[0160] 9. Item picking device 10. Item picking device 91 Acquisition Module 92 Proximity Relation Determination Module 93 Constraint Relationship Decision Module 94 Picking sequence determination module 101 Memory 102 processors 110 Computer Systems 1100 Bus 1110 memory 1120 processor 1130 Input / Output Interface 1140 Network Interface 1150 Storage Interface
Claims
1. acquiring picking information for a plurality of items in a box, the picking information comprising picking surfaces and picking points for the plurality of items and an image of the box in which the plurality of items are located; determining proximity relationships between the plurality of items based on the image of the box in which the plurality of items are located; determining constraint relationships among the plurality of items based on the proximity relationships among the plurality of items and at least one of tilt angles or heights of the plurality of items, wherein the tilt angles of the plurality of items are determined based on the picking planes and the picking points of the plurality of items, and the heights of the plurality of items are determined based on the image of the boxes in which the plurality of items are located; determining a picking order for the plurality of items based on the constraint relationships; An item picking method comprising:
2. determining the proximity relationships between the plurality of items based on the images of the boxes in which the plurality of items are located, determining that two articles of the plurality of articles are proximate articles when the two articles are in contact with each other; determining that if one article of the plurality of articles is in contact with two other articles of the plurality of articles, the two other articles are the proximate articles to each other; The item picking method according to claim 1, comprising:
3. determining the constraint relationships between the plurality of items; 3. The method of claim 2, further comprising determining that an item of the plurality of items has a constraint on a neighboring item if the height of the item is greater than the height of the neighboring item.
4. determining the constraint relationships between the plurality of items; grouping the plurality of articles based on tilt angles of the plurality of articles; If the article and its neighboring articles belong to the same group, determining a constraint relationship between the article and its neighboring articles based on the tilt angles of the article and its neighboring articles; The item picking method according to claim 2, comprising:
5. If the item and the neighboring item of the item belong to the same group, determining the constraint relationship between the item and the neighboring item based on their tilt angles, determining a first picking direction based on the area of the picking surface of the items belonging to the same group, wherein the first picking direction indicates a first picking order of the items belonging to the same group; determining a second picking direction for the item and the adjacent item based on the picking points of the item and the adjacent item, wherein the second picking direction indicates a second picking order for the item and the adjacent item; determining the constraint relationship between the item and the neighboring item based on the first picking direction and the second picking direction; The item picking method according to claim 4, comprising:
6. determining the first picking direction based on the area of the picking surface of the items belonging to the same group, 6. The item picking method of claim 5, further comprising the step of: calculating a first vector as the first picking direction when an area ratio of the largest picking surface to the smallest picking surface among the picking surfaces of the items belonging to the same group exceeds a ratio threshold, wherein the first vector represents a direction from a picking point on the largest picking surface to a picking point on the smallest picking surface.
7. determining the first picking direction based on the area of the picking surface of the items belonging to the same group, If the area ratio of the largest picking surface to the smallest picking surface among the picking surfaces of the items belonging to the same group does not exceed the ratio threshold, calculating a second vector based on the tilt angles of the items belonging to the same group, wherein the second vector represents that the entire tilt directions of the items belong to the same group; determining the first vector as the first picking direction if the angle between the first vector and the second vector does not exceed a first threshold; if the angle between the first vector and the second vector exceeds the first threshold, determining an opposite direction of the first vector as the first picking direction; The item picking method according to claim 6, comprising:
8. 6. The method of claim 5, wherein the second picking direction is from a picking point on the picking surface of the item to a picking point on the picking surface of the adjacent item.
9. determining the constraint relationship between the item and the neighboring item based on the first picking direction and the second picking direction, 6. The method of claim 5, further comprising determining that the item has a constraint relative to the adjacent item if an angle between the first picking direction and the second picking direction does not exceed a second threshold.
10. determining the constraint relationships between the plurality of items; 5. The item picking method of claim 4, further comprising: determining that the item has a constraint on the adjacent item if the item and the adjacent item do not belong to the same group and the height of the item is greater than the height of the adjacent item.
11. the plurality of items having a plurality of picking surfaces, and the step of determining the proximity relationship between the plurality of items based on the image of the box in which the plurality of items are located, If two picking surfaces of the plurality of picking surfaces contact each other, determining the two picking surfaces as adjacent picking surfaces; determining that if one picking surface of the plurality of picking surfaces contacts two other picking surfaces of the plurality of picking surfaces, the other two picking surfaces are the adjacent picking surfaces to each other; The item picking method according to claim 1, comprising:
12. determining the constraint relationships between the plurality of items; determining that a picking surface has a constraint with respect to an adjacent picking surface if the height of the picking surface is greater than the height of the adjacent picking surface; The item picking method according to claim 11.
13. an inclination angle of an article comprising an inclination angle of the picking surface of the article, and the step of determining the constraint relationship between the plurality of articles comprises: grouping the pick surfaces of the plurality of items based on the tilt angle of the pick surfaces; determining the constraint relationship between a picking surface and its adjacent picking surface based on their inclination angles if the picking surface and its adjacent picking surface belong to the same group; The method of claim 11, comprising:
14. determining the constraint relationship between the picking surface and the adjacent picking surface based on their inclination angles if the picking surface and its adjacent picking surface belong to the same group; determining a first picking direction based on the area of the picking surfaces belonging to the same group, wherein the first picking direction indicates that a first picking order of the picking surfaces belongs to the same group; determining a second picking direction for the picking surface and the adjacent picking surface based on the picking points of the picking surface and the adjacent picking surface, wherein the second picking direction indicates a second picking order for the picking surface and the adjacent picking surface; determining the constraint relationship between the picking plane and the adjacent picking plane based on the first picking direction and the second picking direction; The method of claim 13, comprising:
15. determining the first picking direction based on the area of the picking surfaces belonging to the same group, 15. The item picking method of claim 14, further comprising the step of: calculating a first vector as a first picking direction when an area ratio of a largest picking surface to a smallest picking surface among the picking surfaces belonging to the same group exceeds a ratio threshold, wherein the first vector represents a direction from a picking point on the largest picking surface to a picking point on the smallest picking surface.
16. determining the first picking direction based on the area of the picking surface of the items belonging to the same group, If the ratio of the largest picking surface to the smallest picking surface among the picking surfaces belonging to the same group does not exceed the ratio threshold, calculating a second vector based on the inclination angles of the picking surfaces belonging to the same group, wherein the second vector represents that the entire inclination directions of the picking surfaces belong to the same group; determining the first vector as the first picking direction if the angle between the first vector and the second vector does not exceed a third threshold; if the angle between the first vector and the second vector exceeds the third threshold, determining an opposite direction of the first vector as the first picking direction; The method of claim 15, comprising:
17. 15. The method of claim 14, wherein the second picking direction represents a direction from a picking point on the picking surface to a picking point on the adjacent picking surface.
18. determining the constraint relationship between the picking plane and the adjacent picking plane based on the first picking direction and the second picking direction, 15. The method of claim 14, further comprising determining that the picking plane has a constraint relative to the adjacent picking plane if the angle between the first picking direction and the second picking direction does not exceed a fourth threshold.
19. determining the constraint relationships between the plurality of items; 14. The method of claim 13, further comprising determining that the picking surface has a constraint on the proximal picking surface if the picking surface and the proximal picking surface do not belong to the same group and if the height of the picking surface is greater than the height of the proximal picking surface.
20. determining the picking order for the plurality of items based on the constraint relationships; generating a sorting result based on the picking information, the sorting result representing a difficulty of picking the plurality of items; determining the picking order of the plurality of items based on the sorting results and the constraint relationships among the plurality of items; The item picking method according to claim 1, comprising:
21. an acquisition module for acquiring picking information of a plurality of items in a box, the picking information comprising an image of a picking surface and a picking point of the plurality of items and the box in which the plurality of items are located; a proximity relationship determination module 92 for determining proximity relationships between the plurality of items according to the image of the box in which the plurality of items are located; a constraint relationship determination module for determining constraint relationships between the plurality of items based on the proximity relationships between the plurality of items and at least one of tilt angles or heights of the plurality of items, wherein the tilt angles of the plurality of items are determined based on the picking planes and picking points of the plurality of items, and the heights of the plurality of items are determined based on the images of the boxes in which the plurality of items are located; a picking order determination module 94 for determining a picking order for the plurality of items based on the constraint relationship; An article picking device comprising:
22. Memory and a processor coupled to the memory; 21. An item-picking apparatus comprising: a processor configured to perform the item-picking method of any one of claims 1 to 20 based on instructions stored in the memory.
23. 21. A computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implements the item-picking method of any one of claims 1 to 20.
24. 21. A computer program comprising instructions that, when executed by a processor, cause the processor to perform the method for picking an item according to any one of claims 1 to 20.