Suction Cup Array Verification Method, Apparatus, Electronic Equipment and Storage Medium

Abstract

This application discloses a suction cup array verification method, apparatus, electronic device, and storage medium. The suction cup array verification method includes: grouping all suction cups in the array according to the array's shape; acquiring the state information of the suction cup array; determining the number of operable suction cups and the grouping information of the operable suction cups based on the state information of the suction cup array; determining whether the number of operable suction cups meets preset conditions and / or whether the grouping information of the operable suction cups meets preset conditions; and determining whether the suction cup array can perform grasping in this state based on the determination result. This invention is specifically designed for industrial scenarios involving glass grasping using suction cup arrays. It can pre-verify whether the grasping method used can correctly grasp the glass, avoiding problems such as glass damage or loss due to instability during the grasping process, and improving the accuracy and stability of glass grasping using suction cup arrays.

Description

Technical Field

[0001] This application relates to the field of robotic arm control technology, and more specifically, to a suction cup array verification method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, with the widespread adoption of intelligent programmed robots, more and more items can be grasped and transported using these robots. For example, logistics packaging can be grasped by intelligent programmed robots, significantly improving grasping efficiency. To improve grasping efficiency and flexibly adapt to various objects, intelligent programmed robots are typically equipped with gripper sets consisting of multiple clamps, allowing for flexible use of different grippers depending on the object being grasped. Different grippers can grasp different objects; for example, a suction cup array can pick up objects like glass, but will fail to grasp rubber or protrusions. Objects or structures that such grippers cannot grasp are called obstacles.

[0003] Conventional intelligent robots can only perform obstacle avoidance grasping in situations where the size, shape, and position of the grasped object and obstacle are fixed. Existing technology sets the gripper at the center of the grasped object, away from the obstacle, based on the object's model and the obstacle's location, thus avoiding the obstacle during grasping. However, this obstacle avoidance grasping method has the following drawbacks: First, this grasping method can only be used for items with fixed obstacles and models. That is, when the model of the item to be grasped is unknown, or when the model is known but the obstacle's position is not fixed, it cannot accurately avoid obstacles during grasping. Second, this grasping method does not determine whether multiple grippers can stably grasp the object. If the number of grippers is insufficient or the grippers are not properly arranged (e.g., multiple grippers arranged in a straight line), the center of gravity is unstable after grasping the object, and the object may swing during the grasping process, causing it to fall or collide with other unexpected objects and be damaged. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to overcome or at least partially solve the above problems. Specifically, according to the above embodiments, firstly, the present invention can first obtain the possible gripping methods of the clamp, select the best gripping method that will not grab obstacles, and perform the gripping of the item, so that even if there are obstacles on the item that cannot be gripped, the item can be accurately gripped by avoiding obstacles; secondly, for obstacles such as protrusions / indentations that may prevent the clamp from gripping correctly, the present invention proposes a method to determine whether there is a non-planar structure in a specific area by determining the depth value difference of each pixel in the group and the number of different depth values. Since this method uses numerical statistics rather than determining the specific location of the non-planar structure to determine whether the non-planar structure exists, This process is highly efficient and practical, and can also be applied to industrial scenarios beyond gripping where it may be necessary to determine whether an object's surface has a non-planar structure. Furthermore, this invention pre-verifies the gripping method before the fixture assembly performs the gripping action, improving gripping stability and preventing damage or loss of the gripped object due to instability during the gripping process. Finally, based on general obstacle avoidance gripping methods and fixture verification methods, this invention develops a dedicated obstacle avoidance gripping method and suction cup verification method for the industrial scenario of using suction cup arrays for glass gripping, improving the accuracy and stability of glass gripping using suction cup arrays. Therefore, this invention solves various problems encountered in industrial scenarios involving gripping objects using fixtures.

[0005] All solutions disclosed in the claims and specification of this application possess one or more of the aforementioned innovative features, and correspondingly, are able to solve one or more of the aforementioned technical problems. Specifically, this application provides a suction cup array verification method, apparatus, electronic device, and storage medium.

[0006] The suction cup array verification method of the embodiments of this application includes:

[0007] Group all the suction cups in the array according to their shape;

[0008] Obtain the status information of the suction cup array;

[0009] The number of operable suction cups and the group information of the operable suction cups are determined based on the status information of the suction cup array.

[0010] Determine whether the number of operable suction cups meets the preset conditions and / or whether the grouping information of operable suction cups meets the preset conditions;

[0011] Based on the judgment result, determine whether the suction cup array can perform grasping in this state.

[0012] In some embodiments, grouping all the suction cups in the array according to the shape of the suction cup array includes dividing the suction cup array into four regions: upper left, lower left, upper right, and lower right, and dividing all the suction cups into four groups according to the regions where all the suction cups are located.

[0013] In some implementations, the number of activatable suction cups that need to be satisfied is preset based on the weight of the object to be grasped.

[0014] In some implementations, the grouping information of the activatable suction cups is preset based on the center of gravity of the object to be grasped, and the conditions that need to be met.

[0015] In some implementations, the grouping information of the operable suction cups needs to meet the following conditions: the operable suction cups are distributed in at least three groups.

[0016] In some implementations, determining whether the suction cup array can perform grasping in this state based on the judgment result includes: determining that grasping can be performed when the number of suction cups meets a preset condition and the suction cup grouping information meets a preset condition.

[0017] The suction cup array verification device according to the embodiments of this application includes:

[0018] The grouping module is used to group all the suction cups in the array according to the shape of the suction cup array;

[0019] The status information acquisition module is used to acquire the status information of the suction cup array;

[0020] The information determination module is used to determine the number of operable suction cups and the group information of the operable suction cups based on the status information of the suction cup array.

[0021] The condition determination module is used to determine whether the number of operable suction cups meets the preset conditions and / or whether the grouping information of operable suction cups meets the preset conditions.

[0022] The grasping determination module is used to determine whether the suction cup array can perform grasping in this state based on the judgment result.

[0023] In some implementations, the grouping module is specifically used to divide the suction cup array into four regions: upper left, lower left, upper right, and lower right, and to divide all suction cups into four groups according to the regions where all suction cups are located.

[0024] In some implementations, the number of activatable suction cups that need to be satisfied is preset based on the weight of the object to be grasped.

[0025] In some implementations, the grouping information of the activatable suction cups is preset based on the center of gravity of the object to be grasped, and the conditions that need to be met.

[0026] In some implementations, the grouping information of the operable suction cups needs to meet the following conditions: the operable suction cups are distributed in at least three groups.

[0027] In some implementations, the grasping determination module is specifically used to: determine that grasping can be performed when the number of suction cups meets a preset condition and the suction cup grouping information meets a preset condition.

[0028] The electronic device of the embodiments of this application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the suction cup array verification method of any of the above embodiments.

[0029] The computer-readable storage medium of the embodiments of this application stores a computer program thereon, which, when executed by a processor, implements the suction cup array verification method of any of the above embodiments.

[0030] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0031] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0032] Figure 1 This is a flowchart illustrating the obstacle avoidance and gripping method of a clamp according to certain embodiments of this application;

[0033] Figure 2 This is a flowchart illustrating a method for determining non-planar structures on the surface of an article according to certain embodiments of this application;

[0034] Figure 3 This is a flowchart illustrating a fixture verification method according to certain embodiments of this application;

[0035] Figure 4 This is a flowchart illustrating a glass gripping method using a suction cup according to certain embodiments of this application.

[0036] Figure 5 This is a schematic diagram of the glass and obstacle point cloud and suction cup arrangement in some embodiments of this application;

[0037] Figure 6 This is a flowchart illustrating the obstacle determination process during glass grasping in some preferred embodiments of this application;

[0038] Figure 7 This is a schematic diagram of the adhesive application process for the strip to be applied according to certain embodiments of this application;

[0039] Figure 8This is a flowchart illustrating the suction cup array verification method according to certain embodiments of this application;

[0040] Figure 9 This is a schematic diagram of the structure of the clamp obstacle avoidance gripping device according to some embodiments of this application;

[0041] Figure 10 This is a schematic diagram of the structure of the non-planar structure determination device for the surface of an article according to certain embodiments of this application;

[0042] Figure 11 This is a schematic diagram of the fixture verification device according to certain embodiments of this application;

[0043] Figure 12 This is a schematic diagram of the structure of a glass gripping device using a suction cup according to certain embodiments of this application;

[0044] Figure 13 This is a schematic diagram of the structure of a suction cup array verification device according to certain embodiments of this application;

[0045] Figure 14 This is a schematic diagram of the structure of an electronic device according to certain embodiments of this application. Detailed Implementation

[0046] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0047] Figure 1 A flowchart illustrating a robot-based gripper obstacle avoidance and grasping method according to an embodiment of the present invention is shown, as follows: Figure 1 As shown, the method includes:

[0048] Step S100: Obtain the point cloud information of the group of items to be captured.

[0049] A group of objects to be grasped may include one or more objects to be grasped, and may also include one or more obstacles. Obstacles may be present on the objects to be grasped, such as protrusions on the objects, adhesive strips attached to the objects, etc.; obstacles may also be outside the objects to be grasped, for example, the objects to be grasped may be stacked in layers with rubber pads between the layers. Accordingly, the point cloud information of the group of objects to be grasped may include the point cloud information of one or more objects to be grasped and / or obstacles.

[0050] As an example, point cloud information can be acquired using a 3D industrial camera. A typical 3D industrial camera is equipped with two lenses, each capturing the object to be grasped from different angles. After processing, a 3D image of the object can be displayed. The object to be grasped is placed below a vision sensor, and both lenses capture images simultaneously. Based on the relative pose parameters of the two images, a general binocular stereo vision algorithm is used to calculate the X, Y, and Z coordinates and orientation of each point on the glass to be coated, thus converting it into point cloud data of the object to be grasped. In practice, point clouds can also be generated using laser detectors, visible light detectors such as LEDs, infrared detectors, and radar detectors. This invention does not limit the specific implementation method.

[0051] The point cloud data obtained through the above methods is three-dimensional data. In order to filter out the data corresponding to the dimensions that have little impact on grasping, reduce the amount of data processing, and thus accelerate the data processing speed and improve efficiency, the obtained three-dimensional point cloud data of the object to be grasped can be orthographically projected onto a two-dimensional plane.

[0052] As an example, a depth map corresponding to the orthographic projection can also be generated. A two-dimensional color image corresponding to the 3D object region and a depth map corresponding to the two-dimensional color image can be obtained along a direction perpendicular to the depth of the object. The two-dimensional color image corresponds to the image of a planar region perpendicular to the preset depth direction; each pixel in the depth map corresponding to the two-dimensional color image corresponds one-to-one with each pixel in the two-dimensional color image, and the value of each pixel is its depth value.

[0053] Step S110: Generate a search state based on the point cloud information of the group of items to be grabbed and the search boundary factor parameters of the fixture.

[0054] The search boundary factor parameters of the fixture include the control boundaries of the fixture's controllable parameters, the control granularity, and other boundary factor parameters. Depending on the specific circumstances, such as the type of fixture used, the type and size of the item to be gripped, the search boundary factor parameters and their values ​​may vary.

[0055] As an example, controllable parameters can include the movement distance in the X direction, the movement distance in the Y direction, and the rotation angle. The movement distance boundary (i.e., the controllable boundary) in the X direction can be set from -500mm to 500mm, with a control granularity of 100mm. This allows the fixture to move from -500mm to 500mm in 100mm increments in the X direction, resulting in 11 search states in the X direction. The movement distance boundary in the Y direction can be set from -100mm to 100mm, with a control granularity of 50mm. This allows the fixture to move from -100mm to 100mm in 50mm increments in the Y direction, resulting in 5 search states in the Y direction. The rotation angle can be set from -50 degrees to +50 degrees, with a control granularity of 10 degrees. This allows the fixture to rotate from -50 degrees to +50 degrees in 10-degree increments, resulting in 11 search states for rotation angle. Under the above settings for search boundary factors, there are a total of 11 × 5 × 11 = 605 search states. In addition, other boundary factor parameters can be set. For example, the movement distance of the clamp can be set to not exceed 100mm from a certain side, so that when the clamp moves towards that side, it can only move to a position 100mm away from that side.

[0056] To facilitate control of the gripper, its shape, size, and other information can be configured. This configuration information can be saved using a JSON file. Depending on the specific situation, such as the type of gripper used or the object to be grasped, the configuration information may differ. For example, when using a suction cup array for grasping, the configuration information may include the position of a single suction cup relative to the center of the entire array, the number of each suction cup, and its radius.

[0057] The solution of this invention can be used for various types of clamps, including general-purpose clamps. General-purpose clamps refer to clamps with standardized structures and a wide range of applications, such as three-jaw and four-jaw chucks for lathes, and flat-jaw vises and indexing heads for milling machines. Furthermore, based on the clamping power source used, clamps can be classified into manual clamping clamps, pneumatic clamping clamps, hydraulic clamping clamps, pneumatic-hydraulic linkage clamping clamps, electromagnetic clamps, vacuum clamps, etc. This invention does not limit the specific type of clamp, as long as it can achieve the operation of gripping objects.

[0058] Step S120: For each search state, perform obstacle determination and set the search state that passes the obstacle determination as the alternative search state.

[0059] Obstacle determination is used to assess the obstacle situation faced by the gripper in a certain search state, such as whether the obstacle exists or not, and whether it affects gripping. If there is no obstacle, or the obstacle does not affect gripping, the obstacle determination is passed; otherwise, it is failed. If multiple grippers or a gripper array composed of multiple grippers are used for gripping, obstacle determination can be performed on each gripper separately. One of the key points of this invention is to traverse all search states and perform obstacle determination, therefore, the obstacle determination method is not limited, and any obstacle determination method can be used in this invention. The obstacle situation of the object to be grasped can be determined using only one obstacle determination method, or multiple obstacle determination methods can be combined for determination. If multiple determination methods are combined, the multiple determination methods can be performed in a certain order or in parallel. When any one of the multiple determination methods fails, the obstacle determination for that search state is considered to have failed.

[0060] In one alternative implementation, the present invention exemplarily provides four obstacle determination methods: boundary obstacle avoidance, fixed obstacle avoidance, protrusion / depression obstacle avoidance, and custom obstacle avoidance.

[0061] Boundary obstacle determination

[0062] Some grippers cannot grasp items at their edges. When using these grippers, the item's boundary itself becomes an obstacle to grasping. Therefore, it is necessary to identify the item's boundary to avoid grasping the item at the boundary. To identify the boundary, point cloud data of the group of items to be grasped can be obtained. Specifically, the point cloud data can be obtained using the method in step 100. The portion of the point cloud data at the edge is then cropped to obtain the outline point cloud of the group of items to be grasped. Since the obtained outline point cloud of the group of items to be grasped is three-dimensional point cloud data, and three-dimensional point cloud data can be affected by various external or internal factors in determining the outline of the item's edge, in order to more accurately define the outline of the item's edge, the outline point cloud of the item's edge can be projected onto a two-dimensional plane to obtain the outline points of the edge of the group of items to be grasped. Since the outline points of the edge of the group of items to be grasped are two-dimensional data, the outline points of the edge of the group of items to be grasped can be more clearly defined based on this two-dimensional data.

[0063] To avoid incomplete point cloud acquisition, the outline points at the four corners of the objects to be grasped can be obtained from the 2D pattern of the object group. Based on the outline points at the four corners of the glass to be coated, the smallest bounding rectangle of the glass to be coated is obtained, and this smallest bounding rectangle is regarded as the outline quadrilateral of the object group. Alternatively, the outline points at the four corners can be connected in a clockwise order from left to right and from top to bottom to form the edge line of the object to be grasped, and the quadrilateral enclosed by the edge line is regarded as the outline quadrilateral of the object group.

[0064] For each search state, determine the relationship between the center of the clamp and the edge contour point of the item in that search state. If the center of the clamp is outside the obtained contour, it means that the clamp is located at the boundary of the item to be grabbed. In this case, it is determined that there is a boundary obstacle and the obstacle determination fails. If the center of the clamp is inside the contour, the obstacle determination passes.

[0065] Fixed obstacle determination

[0066] The object to be grasped may be placed together with certain fixed obstacles to form a group of objects to be grasped. For example, when the object to be grasped is a steel plate or glass, multiple steel plates or glass may be stacked, with each layer of steel plate or glass separated by rubber pads or foam. When the upper layer of steel plate or glass is grasped, the rubber pads or foam will be left on the lower layer of steel plate or glass, thus becoming an obstacle during grasping. In such cases, it is necessary to determine the obstacles that are fixed but whose positions are uncertain and affect the gripper's grasping of the object to be grasped.

[0067] To identify fixed obstacles, 3D point cloud data of the group of items to be grasped can be acquired and mapped onto a 2D plane. The point cloud data can be acquired and mapped using the method in step 100. Since fixed obstacles typically have uniform specifications, such as uniform size or shape, specification parameters for the fixed obstacles can be preset. For each search state, based on the 2D point cloud data, it can be determined whether there are any items below the gripper that meet the preset specification parameters in that search state. If they exist, a fixed obstacle is determined to exist, and the obstacle determination fails; otherwise, the obstacle determination passes.

[0068] Determination of protruding / depressed obstacles

[0069] Depending on the item being grasped, it may have protrusions or indentations. These protrusions / indentations may be inherent to the item or caused by collisions or other reasons. Some grippers may fail to grasp items correctly at protrusions or indentations. For example, if a suction cup gripper is used on a protruding edge, the suction cup may not adhere completely to the object, causing air leakage, and thus preventing the gripper from grasping the item at that location. Therefore, in some situations, protrusions / indentations need to be considered as obstacles in obstacle assessment.

[0070] During the research and development process, the applicant discovered that protrusions / depressions, as part of the object to be grasped, are difficult to distinguish using contour point cloud data. Furthermore, unlike fixed obstacles, protrusions / depressions usually do not have fixed specifications, and conventional methods cannot identify protrusions / depressions of objects.

[0071] To address this problem, this invention proposes a method for determining the non-planar structure of an object's surface. This method is also one of the key aspects of this invention. It can be used in some embodiments of this invention to determine protruding / depressed obstacles, and can also be used in other situations where it is necessary to determine whether an object's surface has a non-planar structure, rather than being limited to use in obstacle avoidance and grasping schemes. Figure 2 A flowchart illustrating a method for determining non-planar structures on an object surface according to the present invention is shown. Non-planar structures include protrusions, depressions, and other similar structures. Figure 2 As shown, the method includes:

[0072] Step 200: Obtain the two-dimensional planar depth map information of the area to be determined on the surface of the object.

[0073] After orthographically projecting the 3D point cloud data of the object onto a 2D plane, a depth map corresponding to this orthographic projection is generated as the 2D plane depth map information of the area to be determined. This depth map can be obtained in a manner similar to step 100. Alternatively, instead of obtaining a depth map of the entire object, a depth map of a portion of the object or the area to be determined can be obtained.

[0074] Step 210: Obtain the depth value of each pixel in the region to be determined based on the two-dimensional planar depth map information.

[0075] Each pixel in the depth map corresponds one-to-one with each pixel in the orthographically projected 2D map, and each pixel's value is its depth value. To determine whether there are non-planar structures such as protrusions or depressions within the region to be judged, the depth value of each pixel within that region can be obtained in this step.

[0076] Step 220: Group the obtained depth values.

[0077] In planar structures, the depth values ​​of all pixels are the same or have small differences, while in non-planar structures such as protrusions or depressions, the depth values ​​of all pixels will vary significantly. In this step, the depth values ​​of all obtained pixels are grouped. If there are multiple identical depth values, they can be merged into one depth value before grouping.

[0078] During grouping, all obtained depth values ​​can be grouped into sets of two or more. The grouping method can be random or any other arbitrary method; this invention does not limit this. As a preferred embodiment, all depth values ​​can first be sorted, then the highest and lowest values ​​can be grouped together, the second highest and second lowest values ​​can be grouped together, and so on, until all depth values ​​are grouped.

[0079] Step 230: Calculate the difference in depth values ​​within each group and compare it with a preset difference threshold; and / or, calculate the number of groups and compare it with a preset group number threshold.

[0080] The difference in depth values ​​reflects the distance between pixels. The larger the difference in depth values ​​between two pixels, the greater the height difference between the two points. Therefore, the difference in depth values ​​can reflect the degree of unevenness in the area to be determined. Furthermore, the number of groups reflects the number of pixels with different heights. The larger the number of groups in the area to be determined, the more pixels with varying heights. Therefore, the number of groups can reflect the degree of unevenness in the area to be determined from another perspective. The difference in depth values ​​or the number of groups can be used alone to determine whether an object has a non-planar structure. As a preferred embodiment, the difference in depth values ​​and the number of groups can also be used together to determine whether an object has a non-planar structure. The applicant has found that this method can significantly increase the accuracy of non-planar structure determination compared to using them alone. In another embodiment, the difference threshold and / or the number of groups threshold can be set according to actual needs. For example, when applying the method to obstacle avoidance, for some grippers, small protrusions do not affect gripping. Therefore, the threshold can be set relatively large so that small protrusions are not judged as ungripable obstacles.

[0081] Step 240: Determine the non-planar structure of the region based on the comparison results.

[0082] The non-planar structure condition can include whether the non-planar structure exists. A non-planar structure is determined to exist when the difference in depth values ​​and / or the number of groups exceeds a threshold. It can also include the degree of undulation of the non-planar structure, and multiple levels of undulation can be set. In other embodiments, multiple threshold levels can be set, and the existence of a non-planar structure or the degree of undulation can be determined by combining different thresholds. As a preferred embodiment, when using the difference in depth values ​​and the number of groups to jointly determine the non-planar structure condition, a non-planar structure can be determined to exist when the difference in depth values ​​exceeds a difference threshold and the number of groups exceeds a group count threshold.

[0083] Custom obstacle determination

[0084] In industrial settings, some obstacles cannot be identified using robot vision technology. For example, when the object to be grasped is glass, it may be coated with adhesive. This adhesive is usually quite transparent and its features are not obvious, making it difficult to identify correctly. Furthermore, the gripper cannot grasp the glass at the adhesive-coated location. To identify such obstacles, the size, shape, and location of the obstacle can be customized before grasping. Thus, for each search state, the presence of an obstacle under the gripper is determined based on the predefined obstacle information. In one implementation, the edge contour of the obstacle can be generated first based on the customized obstacle information. Then, for each search state, the relationship between the center of the gripper and the customized obstacle edge contour is determined. If the gripper center is outside the obtained contour, an obstacle is determined, and the obstacle determination fails; if the gripper center is inside the contour, the obstacle determination passes. In another embodiment, an inward shrinkage distance of the edge contour can be set, that is, the original edge contour of the obstacle shrinks inward by a certain distance before determining the relative position of the gripper center and the edge contour.

[0085] In step S120, the scheme can also be extended or improved as follows:

[0086] For a given search state, if the clamp passes the obstacle, the clamp is set to open in that search state; if it fails, the clamp is set to close in that search state.

[0087] If multiple grippers or a gripper array consisting of multiple grippers are used for gripping, then when performing obstacle determination, obstacle determination can be performed on each of the multiple grippers.

[0088] If at least one clamp passes the obstacle determination in a certain search state, then the search state can be set as an alternative search state.

[0089] When performing obstacle determination, the obstacle determination can be passed if there is an obstacle but it does not affect the grasping, and it can be failed only if there is an obstacle and it affects the grasping.

[0090] Step S130: Select the best search state from the alternative search states.

[0091] Depending on the actual situation, such as the different items to be grasped and the different clamps used, the method for selecting the optimal search state will also be different. This invention does not limit this, and any selection method can be used in the solution of this invention. As a preferred embodiment, the optimal search state can be selected based on the distance between the clamp and the center of the item to be grasped in each search state. Generally speaking, the closer the gripping position of the clamp is to the center of the item, the more stable the grip. Therefore, the search state with the closest distance between the clamp and the center of the item can be set as the optimal search state. To calculate the distance between the clamp and the center of the item, the center position P1 of the clamp can be determined first in a certain search state. Then, the bounding rectangle of the item can be calculated. Then, the center position P2 of the item can be calculated based on the bounding rectangle of the item. The distance between P1 and P2 is the distance between the clamp and the center of the item.

[0092] If multiple grippers or a gripper array composed of multiple grippers are used for grasping, the optimal search state can be selected based on the number of grippers that have passed the obstacle determination, or by comprehensively considering the distance of the gripper from the center of the item and the number of grippers that have passed the obstacle determination. In a preferred embodiment, selection can be prioritized based on quantity. If there are multiple search states with the highest number of passes, the search state with the closest gripper to the center of the item is selected as the optimal search state from among the multiple passed search states. In other embodiments, a quantity threshold can be preset. Search states where the number of grippers opened exceeds the threshold can be selected first, and then the search state with the closest gripper to the center can be selected as the optimal search state from among these search states.

[0093] Step S140: Grab the item to be grabbed based on the optimal search state.

[0094] The robot sets the gripper's position, angle, and quantity based on the configuration parameters of the gripper under the optimal search state. It then picks up the item and places it in the designated location. The designated location may include the ground, an item rack, etc. Some items have specific positional and orientation requirements, such as needing to be placed vertically or too high or too low. Therefore, the robot can also calculate the distance from the gripper center to the designated edge of the object in the final planning result, thus accurately placing the item in the specified location.

[0095] The inventors discovered that when using multiple grippers or a gripper array composed of multiple grippers for grasping, if the number of grippers used in the optimal search state is insufficient or the arrangement of multiple grippers is inappropriate (e.g., multiple grippers arranged in a straight line), it will cause instability in the center of gravity during grasping. As a result, the object may swing, fall, or collide with unexpected objects and be damaged. Therefore, this invention also proposes a gripper grasping verification method. This method can determine whether the grasping method to be used can stably grasp the object before grasping it, thereby avoiding unnecessary losses during grasping. This method can be used in the obstacle avoidance grasping method of this invention, as well as in other grasping scenarios. This invention does not limit the specific application; as long as multiple grippers or a gripper array composed of multiple grippers are used in a grasping scenario, this method can be applied for verification. For ease of explanation, multiple grippers or a gripper array composed of multiple grippers are collectively referred to as a gripper group in this invention.

[0096] Figure 3 A flowchart illustrating a method for verifying a fixture assembly according to an embodiment of the present invention is shown. Figure 3 As shown, the method includes:

[0097] Step S300: Obtain the status information of the fixture group.

[0098] The state of a fixture group refers to a combination of specific parameters for each fixture within the group. When a fixture group performs a gripping action in a particular state, the fixtures are configured and execute the gripping action based on these parameters. For example, suppose a fixture has three fixtures, designated 1-3. Fixture 1 is on the left edge of the object to be gripped, with a rotation angle of 30 degrees; fixture 2 is on the right edge of the object, with a rotation angle of 0 degrees; fixtures 1 and 2 are both set to open; and fixture 3 is outside the object and set to closed. This combination of specific parameters constitutes a state of the fixture group. State information can include the position information of the fixture group in that state, the position and angle information of each fixture within the group, and whether these fixtures can be opened in the current state, and which fixtures can be opened.

[0099] Step S310: Determine the number of openable fixtures and / or the position information of openable fixtures based on the status information of the fixture group.

[0100] When determining the position information of an openable clamp, the position of the clamp relative to the object to be gripped can be determined based on the outline information of the object to be gripped and the clamp boundary parameters; alternatively, the position of the clamp relative to other clamps or the position of the clamp relative to a clamp array can be used as the clamp position. In one embodiment, multiple clamps or clamp arrays can be divided into multiple regions, and the position information of the clamp can be represented by the region where the clamp is located.

[0101] Step S320: Determine whether the number of openable fixtures meets the preset conditions; and / or determine whether the position information of the openable fixtures meets the preset conditions.

[0102] Insufficient number of unclamped grippers or improper gripper placement can both lead to unstable gripping. If the number of unclamped grippers is not adjustable or does not require adjustment, or if the gripper placement is not adjustable or does not require adjustment, only one of these conditions needs to be considered. Those skilled in the art will understand that considering both conditions simultaneously significantly increases the probability of stable gripping compared to considering only one condition. Conditions that need to be met can be preset based on information about the item to be gripped, such as its weight. For example, a condition regarding the number of unclamped grippers must be set. For instance, when gripping heavier items, the number of grippers can be set to 5, so that the preset condition is met only when the number of unclamped grippers exceeds 5; for lighter items, 3 grippers can be used. When the density of the items to be gripped is uniform, the number of grippers required can also be set based on the size of the items. For example, for larger items, the preset condition is 5 grippers, and for smaller items, it is 3 grippers. The area of ​​the item can be determined based on its outline information. The conditions that the position information of the unclamped clamps needs to meet are generally to avoid multiple clamps being located in a straight line or approximately in a straight line. Depending on the actual needs, more specific position information can also be set. For example, the line connecting the positions of multiple unclamped clamps can be set to form a stable triangle. Alternatively, the position information that the unclamped clamps need to meet can be set according to the center of gravity of the object to be grasped to ensure a stable center of gravity during grasping. For example, when the center mass of the object to be grasped is large, the preset condition can be that there must be an unclamped clamp near the center of the object to be grasped or that there must be a specific number of unclamped clamps.

[0103] Step S330: Determine whether the fixture group can perform gripping in this state based on the judgment result.

[0104] In practice, gripping can be performed only when the number of unclamped fixtures meets the condition or when the position information of the unclamped fixtures meets the condition. Alternatively, gripping can be performed only when both conditions are met. If there are multiple position information conditions, gripping can be performed only when all conditions are met.

[0105] In one implementation, clamp verification can be performed after determining that an item should be grasped in a certain state, but before the grasping is actually performed. If used in conjunction with an obstacle avoidance grasping scheme, steps S300-S330 can be executed between steps S130 and S140 of the aforementioned embodiment. That is, after selecting the optimal search state, clamp verification is performed using the optimal search state as the state of the clamp array to determine whether the optimal search state can be used for grasping. In another implementation, clamp verification can also be used to determine that an item should be grasped in a certain state. If used in conjunction with an obstacle avoidance grasping scheme, steps S300-S330 can be executed in steps S120 or S130 of the aforementioned embodiment. For example, alternative search states can be selected through clamp verification, or after selecting alternative search states, these alternative search states are first used as the state of the clamp array, and search states that cannot be grasped are eliminated using clamp verification. Then, the optimal search state is selected based on the number of clamps and the distance from the center.

[0106] The obstacle avoidance and gripping method and the fixture gripping and verification method of the present invention are not limited to specific fixtures or application scenarios. In order to apply the above methods in industrial scenarios where suction cup arrays are used to grip glass, the inventors have put in a lot of effort to further refine the methods to adapt to this scenario, which is also one of the key points of the present invention.

[0107] like Figure 4 A flowchart of a glass obstacle avoidance and grasping method using a suction cup array according to a preferred embodiment of the present invention is shown, the method comprising:

[0108] Step S400: Obtain point cloud information of the glass and obstacles to be grasped.

[0109] In this embodiment, the glass can be stacked in layers, with rubber pads separating each layer. Point clouds of the rubber pads and the object itself can be acquired using a method similar to step S100, and the point clouds can be orthographically projected onto a 2D image to generate the corresponding depth map. Figure 5 The image shows a point cloud obtained in this way, where the black areas represent transparent glass and the white areas represent the point cloud of non-transparent areas.

[0110] Step S410: Generate the search state based on the point cloud information and the search boundary factor parameters of the suction cup array.

[0111] The search state of the suction cup includes its different positional states on the object and its rotational state. Specified search boundary parameters can include the X and Y axes, rotation, and the distance from the suction cup boundary to a specified edge. Multiple different search states are generated based on different search boundary parameters. For example... Figure 5 As shown, the suction cup array used consists of 10 suction cups, numbered 1-10. Figure 5 The diagram shows the positions of the 10 suction cups in a certain search state. In other search states, the suction cups may be in different positions or have different rotation angles. As an example, the movement distance boundary (i.e., the controllable boundary) in the X direction can be set to -500mm to 500mm, with a control granularity of 100mm. The clamp can then move from -500mm to 500mm in 100mm increments in the X direction, resulting in 11 search states in the X direction. The movement distance boundary in the Y direction can be set to -100mm to 100mm, with a control granularity of 50mm. The clamp can then move from -100mm to 100mm in 50mm increments in the Y direction, resulting in 5 search states in the Y direction. The rotation angle can be set to -50 degrees to +50 degrees, with a control granularity of 10 degrees. The clamp can then rotate from -50 degrees to +50 degrees in 10-degree increments, resulting in 11 search states in terms of rotation angle. With the above search boundary factor parameters set, there are a total of 11 × 5 × 11 = 605 search states. It can also be set that the clamp's movement distance cannot exceed 100mm from the upper boundary, meaning the clamp can only move 100mm from the upper boundary in the Y direction.

[0112] Step S420: For each search state, determine whether there is an obstacle under each suction cup. If there is, turn off the suction cup in that search state.

[0113] like Figure 6 As shown, determining whether there is an obstacle below the suction cup may include the following steps:

[0114] Step S421: Determine whether there is a glass boundary obstacle below the suction cup.

[0115] In the current search state, for each suction cup, check whether the center of this individual suction cup is inside the object's outline. This can be determined by judging the proportion of the point cloud within the area below the suction cup; the point cloud proportion refers to the ratio of the point cloud area below the suction cup to the overall area. When the suction cup is near the glass boundary, the point cloud will significantly increase. Figure 5 Suction cups 4-6 are located at the glass boundary. The white point cloud area below these suction cups occupies a significantly larger proportion. Therefore, a point cloud proportion threshold can be preset. When the point cloud proportion in the area below the suction cup exceeds this threshold, it is determined that there is a glass boundary obstacle below the suction cup. The proportion threshold can be set according to actual needs, such as suction cup strength and obstacle conditions. This invention does not limit the specific value. The inventors found that selecting a proportion threshold within the range of 5%-40% can improve the accuracy of boundary obstacle detection, with 10% being optimal.

[0116] Step S422: Determine if there is a rubber pad obstruction under the suction cup.

[0117] In this embodiment, glass layers are stacked one on top of another, separated by rubber pads. Except for the top layer, each lower layer of glass has a rubber pad on it to separate the glass layers. Thus, after the top layer of glass is picked up, the lower layer will have remaining rubber pads. Since the size of the rubber pads is fixed and known, their presence can be filtered out based on their size. In one implementation, the obstacle point cloud area can be pre-set. When the point cloud area in the region below the suction cup exceeds the obstacle point cloud area, it is determined that a rubber pad obstacle exists below the suction cup. The obstacle point cloud area can be arbitrarily set according to the size of the rubber pads used.

[0118] Step S423: Determine if there is an adhesive strip obstruction below the suction cup.

[0119] In industrial settings, adhesive may be applied near the edges of glass. The resulting adhesive strip is typically transparent, making its point cloud features indistinct and difficult to identify. However, this also prevents the suction cups from gripping the glass at the adhesive strip. To correctly identify this obstacle, the user can define the strip's position. The robot then uses this user-defined obstacle information to determine if an obstacle exists below the suction cup array. Specifically, an adhesive strip can be generated around the edge of the glass, allowing the user to set its position, indentation distance, and width. Two strips can also be generated on the same side. Figure 7 Four different adhesive application processes are shown. Process 1 requires no adhesive strips and no preset adhesive strip information. Process 2 involves two segments, inner and outer layers. Adhesive strips can be applied to the uncoated areas between the inner and outer layers, or the inner layer track can be generated separately with a specified edge without applying adhesive strips. Process 3 applies adhesive strips to the open segment (the segment without a track on the right side of the diagram). Process 4 applies adhesive strips to the open segment outside the inner layer track (segments similar in position to the open segment in Process 3). For processes 2-4, adhesive strip obstacle information is preset at the desired application locations. Thus, after application, the robot can determine whether there are adhesive strip obstacles in the area below the suction cup based on the user-set adhesive strip information.

[0120] Step S424: Determine if there is a protruding obstacle below the suction cup.

[0121] The glass surface may have protrusions. If the suction cup adheres to a protruding edge, it cannot fully adhere to the glass surface, resulting in air leakage and suction failure. The presence of protruding obstacles in the area can be determined by the depth difference between pixels in the point cloud of the region below the suction cup and the number of pixels. In one embodiment, the depth values ​​of all point clouds in the region below the suction cup can be obtained, all depth values ​​can be sorted, and then the highest depth value can be paired with the lowest, the second highest with the second lowest, the third highest with the third lowest, and so on. The difference between the paired depth values ​​is calculated and compared with a preset depth difference threshold, and the number of pairs is compared with a preset logarithmic threshold. If both the depth difference and the number of pairs exceed the thresholds, it is determined that a protruding obstacle exists in the region below the suction cup. The depth difference threshold and logarithmic threshold can be set according to actual needs, such as suction cup force and obstacle conditions; this invention does not limit the specific values. The inventors discovered that selecting the logarithmic threshold and depth difference threshold within the following ranges can improve the accuracy of obstacle detection: the logarithmic threshold can be selected within 10 to 50 pairs, with 20 pairs being optimal; the depth difference threshold can be selected within 0.500 mm to 0.005 mm, with 0.015 mm being optimal.

[0122] In a preferred embodiment, obstacle determination can be performed strictly in the order of steps S421-S424. After an obstacle is determined to exist and the suction cup is closed in the previous step, subsequent steps are not executed. For example, if a glass boundary obstacle is determined to exist below the suction cup in step S421, the suction cup is closed, and steps S422-S424 are not executed. Since the above four steps are relatively independent, the obstacle situation of the glass can also be determined by using only the obstacle determination method of any one of steps S421-S424, or any combination of steps can be combined for determination. For example, steps S421 and S422 can be combined instead of steps S423 and S424. If multiple steps are combined, the multiple steps can be performed in a certain order or in parallel. It should be noted that although obstacle judgment can be achieved by a single step or various combinations, if the obstacle judgment method of sequentially executing steps S421-S424 of the preferred embodiment of the present invention is used, compared with other judgment methods, it can not only improve the accuracy of obstacle judgment, but also improve the efficiency of obstacle judgment because it first judges the obstacles with a higher probability of occurrence and uses a simpler algorithm first, thereby improving the efficiency of robot grasping. Therefore, it has a special advantage in industrial scenarios.

[0123] Step S430: Select alternative search states based on the suction cup activation status of each search state.

[0124] The selection criteria can be set by the user, such as the number of activated suction cups or whether a specific suction cup is activated. In one implementation, if at least one suction cup is activated, the search state is selected as a candidate search state.

[0125] Step S440: Select the best search state from the alternative search states based on the number of suction cups opened and / or the distance of the suction cups from the center of the glass.

[0126] The optimal search state can be selected based on the distance from the center of the glass to be grasped in each search state. Generally, the closer to the center of the object, the more stable the grip. Therefore, the search state closest to the center of the object can be set as the optimal search state. For each search state, the center position P1 of the clamp in that search state is determined, and then the center position P2 of the object is obtained through the outer rectangle of the glass. The distance between P1 and P2 is the distance of the clamp from the center of the glass. Alternatively, the search state with the most successful obstacle clearances can be selected as the optimal search state. In a preferred embodiment, the optimal search state can be selected by combining the number of suction cups opened and the distance of the suction cups from the center of the glass. For example, selection can be prioritized based on the number of successful clearances. If there are multiple search states with the most successful clearances, the search state closest to the center of the object can be further selected as the optimal search state from these multiple selected search states. In another embodiment, a quantity threshold can be preset. First, all search states with more than the threshold number of suction cups opened are selected, and then the search state closest to the center is selected as the optimal search state from these search states.

[0127] Step S450: Grab the glass using the best search state.

[0128] In industrial settings, it is often necessary to place the gripped glass on the ground or on a shelf. If the glass is placed on a shelf, it cannot extend too far beyond the edge of the shelf. Therefore, the distance from the center of the suction cup array to the specified edge of the glass can be calculated in the final planning result. This allows the robot to control the placement of the glass more precisely and place it accurately in the designated location.

[0129] like Figure 8 A flowchart illustrating a glass gripping and verification method using a suction cup array according to a preferred embodiment of the present invention is shown. Figure 8 As shown, the method includes:

[0130] Step S500: Group all the suction cups in the array according to the shape of the suction cup array.

[0131] The suction cups can be grouped according to their distribution on the suction cup array. For example... Figure 5In the embodiment shown, the suction cup array includes 12 suction cups, numbered 1-12. The suction cups can be divided into four groups according to their relative position with respect to the suction cup array: upper left, upper right, lower left, and lower right. Specifically, suction cups 3 and 9 are in the first group, suction cups 4 and 8 are in the second group, suction cups 1, 2, and 10 are in the third group, and suction cups 5, 6, and 7 are in the fourth group.

[0132] Step S510: Obtain the status information of the suction cup array.

[0133] The state of a suction cup array refers to a combination of specific parameters for each suction cup in the array. When a suction cup array performs a grasping action in a particular state, it means that the array is configured and executes the grasping action based on these parameters. For example, suppose a suction cup array has three suction cups, numbered 1-3. Suction cup 1 is on the left edge of the object to be grasped, with a rotation angle of 30 degrees; suction cup 2 is on the right edge of the object, with a rotation angle of 0 degrees; suction cups 1 and 2 are both set to "on"; and suction cup 3 is outside the object and set to "off". This combination of specific parameters constitutes a state of the suction cup array. The state information of a suction cup array includes the position of the array itself, the position and angle of each suction cup, and whether each suction cup can be activated in the current state, and which suction cups can be activated.

[0134] Step S520: Determine the number of operable suction cups and the group information of the operable suction cups based on the status information of the suction cup array.

[0135] In this step, for each suction cup state, the number of suction cups that can be opened in that state and the group to which the suction cups that can be opened belong are determined.

[0136] Step S530: Determine whether the number of operable suction cups meets the preset conditions and / or whether the grouping information of operable suction cups meets the preset conditions.

[0137] Insufficient number of activated suction cups or improper placement of the activated suction cups can lead to unstable gripping. A single threshold can be set for the number of activated suction cups, such as requiring more than 5 suction cups to allow gripping. Alternatively, a threshold can be preset based on the size of the glass to be gripped. For example, a minimum threshold of 3 suction cups can be set, meaning at least 3 suction cups must be activated regardless of the glass area; 4 suction cups can be set for glass areas of 1-2 square meters, and 5 suction cups for glass areas larger than 2 square meters. This method determines whether gripping can be performed based on the glass area and the number of activated suction cups. Therefore, when determining the number of activating suction cups, the glass area must also be considered. For example, if the glass area is 2 square meters, the number of activating suction cups can be 4. When the correspondence between area and number is satisfied, the quantity condition is considered met.

[0138] Regarding the grouping information of the suction cups that can be opened, it is possible to determine only which groups contain open suction cups. For example, in the embodiment where the suction cups are divided into 4 groups, it is possible to determine which groups from the first to the fourth group contain openable suction cups based on the opening status of the suction cups. Preset suction cup distribution conditions can be established; for example, it can be set that at least three groups (in which case the suction cups can be arranged in a stable triangle or quadrilateral) contain openable suction cups to satisfy the suction cup distribution conditions.

[0139] Step S540: Determine whether the suction cup array can perform grasping in this state based on the judgment result.

[0140] In practice, grabbing can be performed only when the number of operable suction cups meets the condition or the distribution of operable suction cups meets the condition. Alternatively, grabbing can be performed only when both conditions are met. If there are multiple preset conditions for suction cup grouping, grabbing can be performed only when all conditions are met.

[0141] Step S500 can be executed at any time before S510, as long as the grouping information is already determined when steps S510-S540 are executed sequentially. In one embodiment, suction cup verification can be performed after determining that glass grasping should be performed in a certain state, but before grasping. If used in conjunction with an obstacle avoidance grasping scheme, steps S510-S540 can be executed between steps S430 and S440 in the aforementioned embodiment. That is, after selecting the optimal search state, this optimal search state is used as the state of the suction cup array, and suction cup verification is performed to determine whether the optimal search state can be used for grasping. In another embodiment, the suction cup verification method can also be used to determine that object grasping should be performed in a certain state. If used in conjunction with an obstacle avoidance and grasping scheme, steps S510-S540 can be executed in steps S420 or S430 of the aforementioned embodiment. For example, alternative search states can be selected by a suction cup verification method, or after selecting alternative search states, the alternative search states can be used as the state of the suction cup array. Search states that cannot be grasped can be removed from the array according to the suction cup verification method, and then the best search state can be selected according to the number of suction cups and the distance from the center.

[0142] Furthermore, it should be noted that although this invention describes a general robot grasping method and a grasping method specifically for glass grasping in multiple embodiments, and the technical details of these embodiments are not entirely the same, those skilled in the art will understand that technical details not described in the specific method but described in the general method can actually be used in the specific method, and vice versa. In other words, although each embodiment of this invention has a specific combination of features, further combinations and cross-combinations of these features between embodiments are also possible.

[0143] According to the above embodiments, firstly, the present invention can first obtain the possible gripping methods of the fixture, select the best gripping method that will not grab obstacles, and perform the gripping of the item, so that even if there are obstacles on the item that cannot be gripped, the item can be accurately gripped by avoiding obstacles. Secondly, for industrial scenarios that may require determining whether there are non-planar structures on the surface of an object, the present invention proposes a scheme that can identify non-planar areas on the surface of the object. Thirdly, the present invention can pre-verify that the gripping method used can correctly grip the item before the fixture group performs the gripping, improving the stability of the gripping and avoiding problems such as instability of the center of gravity that may occur during the gripping process. Fourthly, based on general obstacle avoidance gripping methods and fixture verification methods, the present invention has developed an obstacle avoidance gripping method and a suction cup verification method specifically for the industrial scenario of using suction cup arrays to grip glass, which can improve the accuracy and stability of using suction cup arrays to grip glass. It can be seen that the present invention solves all aspects of the problems that occur in the industrial scenario of using fixtures to grip items.

[0144] Figure 9 A clamp control device according to yet another embodiment of the present invention is shown, the device comprising:

[0145] The point cloud acquisition module 600 is used to acquire the point cloud information of the group of items to be captured, that is, to implement step S100.

[0146] The search state generation module 610 is used to generate a search state based on the point cloud information of the group of items to be grabbed and the search boundary factor parameters of the fixture, that is, to implement step S110.

[0147] The obstacle determination module 620 is used to perform obstacle determination for each search state and set the search state that passes the obstacle determination as the alternative search state, that is, to implement step S120.

[0148] The optimal search state determination module 630 is used to select the optimal search state from the alternative search states, that is, to implement step S130.

[0149] The grasping module 640 is used to grasp the item to be grasped based on the optimal search state, that is, to implement step S140.

[0150] Figure 10 A robot-based non-planar structure determination device according to yet another embodiment of the present invention is shown, the device comprising:

[0151] The depth map acquisition module 700 is used to acquire a two-dimensional planar depth map of the area to be determined on the surface of the object, that is, to implement step S200.

[0152] The depth value acquisition module 710 is used to acquire the depth value of each pixel in the region to be determined based on the two-dimensional planar depth map, that is, to implement step S210.

[0153] Grouping module 720 is used to group the obtained depth values, that is, to implement step S220;

[0154] The comparison module 730 is used to calculate the difference in depth values ​​within each group and compare it with a preset difference threshold; and / or, calculate the number of groups and compare it with a preset group number threshold, i.e., to implement step S230.

[0155] The determination module 740 is used to determine the non-planar structure of the region based on the comparison result, that is, to implement step S240.

[0156] Figure 11 A schematic diagram of a fixture group verification device according to another embodiment of the present invention is shown. The device includes:

[0157] The status information acquisition module 800 is used to acquire the status information of the fixture array, that is, to execute step S300.

[0158] The information determination module 810 is used to determine the number of openable clamps and / or the position information of the openable clamps based on the status information, that is, to execute step S310.

[0159] The condition determination module 820 is used to determine whether the number of openable fixtures meets the preset conditions and / or whether the position information of the openable fixtures meets the preset conditions, that is, to execute step S320.

[0160] The gripping determination module 830 is used to determine whether the fixture group can perform gripping in this state based on the judgment result, that is, to execute step S330.

[0161] Figure 12 A schematic diagram of a suction cup array control device according to another embodiment of the present invention is shown. The device includes:

[0162] The point cloud acquisition module 900 is used to acquire point cloud information of the glass and obstacles to be grasped, that is, to execute step S400.

[0163] The search state generation module 910 is used to generate a search state based on point cloud information and search boundary factor parameters of the suction cup array, that is, to execute step S410.

[0164] The obstacle determination module 920 is used to determine whether there is an obstacle under each suction cup for each search state. If there is an obstacle, the suction cup is turned off in the search state, which is used to execute step S420.

[0165] The alternative search state selection module 930 is used to select an alternative search state based on the suction cup opening status of each search state. That is, it is used to execute step S430.

[0166] The optimal search state selection module 940 is used to select the optimal search state from the candidate search states based on the number of suction cups opened and / or the distance of the suction cups from the center of the glass, that is, to execute step S440.

[0167] The grasping module 950 is used to grasp the glass based on the optimal search state, that is, to perform step S450.

[0168] Figure 13 A schematic diagram of a suction cup array verification device according to another embodiment of the present invention is shown. The device includes:

[0169] Grouping module 1000 is used to group all the suction cups in the array according to the shape of the suction cup array, that is, to execute step S500.

[0170] The status information acquisition module 1010 is used to acquire the status information of the suction cup array, that is, to execute step S510.

[0171] The information determination module 1020 is used to determine the number of operable suction cups and the group information of the operable suction cups based on the status information of the suction cup array, that is, to execute step S520.

[0172] The condition determination module 1030 is used to determine whether the number of openable suction cups meets the preset conditions and / or whether the grouping information of the openable suction cups meets the preset conditions, that is, to execute step S530.

[0173] The grasping determination module 1040 is used to determine whether the suction cup array can perform grasping in this state based on the judgment result, that is, to execute step S540.

[0174] The above Figures 9-13 In the illustrated device embodiments, only the main functions of the modules are described. All functions of each module correspond to the corresponding steps in the method embodiments, and the working principles of each module can also be referred to the descriptions of the corresponding steps in the method embodiments, which will not be repeated here. Furthermore, although the above embodiments define the correspondence between the functions of the functional modules and the methods, those skilled in the art will understand that the functions of the functional modules are not limited to the above correspondence; that is, a specific functional module can also implement other method steps or a portion of method steps. For example, the above embodiments describe the grasping and determining module 1040 as a method for implementing step S540. However, depending on the actual needs, the grasping and determining module 1040 can also be used to implement the method or a portion of the method for steps S500, S510, S520, or S530.

[0175] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method of any of the above embodiments. It should be noted that the computer program stored in the computer-readable storage medium of this application can be executed by a processor of an electronic device. Furthermore, the computer-readable storage medium can be a storage medium built into an electronic device or a storage medium that can be plugged into an electronic device. Therefore, the computer-readable storage medium of this application has high flexibility and reliability.

[0176] Figure 14 The diagram shows a structural schematic of an electronic device according to an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

[0177] like Figure 14 As shown, the electronic device may include: a processor 1102, a communications interface 1104, a memory 1106, and a communications bus 1108.

[0178] in:

[0179] The processor 1102, communication interface 1104, and memory 1106 communicate with each other via communication bus 1108.

[0180] Communication interface 1104 is used to communicate with other network elements such as clients or other servers.

[0181] The processor 1102 is used to execute program 1110, specifically the relevant steps in the above method embodiments.

[0182] Specifically, program 1110 may include program code that includes computer operation instructions.

[0183] Processor 1102 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The electronic device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0184] Memory 1106 is used to store program 1110. Memory 1106 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0185] Specifically, program 1110 can be used to cause processor 1102 to perform various operations in the above method embodiments.

[0186] In summary, the invention includes:

[0187] A fixture control method, comprising:

[0188] Obtain the point cloud information of the group of items to be grabbed;

[0189] The search status is generated based on the point cloud information of the group of items to be grabbed and the search boundary factor parameters of the fixture;

[0190] For each search state, perform obstacle determination and set the search state that passes the obstacle determination as the candidate search state.

[0191] Choose the best search state from the alternative search states;

[0192] The item to be grabbed is grabbed based on the optimal search state.

[0193] Optionally, the group of items to be grabbed may include one or more items to be grabbed and / or obstacles.

[0194] Optionally, the fixture's configuration parameters can be saved using a JSON file.

[0195] Optionally, the obstacle determination includes at least one of the following: boundary obstacle determination, fixed obstacle determination, raised / depressed obstacle determination, and custom obstacle determination.

[0196] Optionally, the boundary obstacle determination includes determining whether a boundary obstacle exists based on the relationship between the center of the clamp and the edge contour points of the object.

[0197] Optionally, the fixed obstacle determination includes determining whether a fixed obstacle exists based on the obstacle's size or shape.

[0198] Optionally, the determination of protrusions / depressions includes determining whether protrusions / depressions exist based on two-dimensional planar depth map information.

[0199] Optionally, the custom obstacle determination includes generating the edge contour of a custom obstacle and determining whether a custom obstacle exists based on the edge contour.

[0200] Optionally, it also includes: performing a clamp check on the optimal search state to determine whether the item can be correctly grabbed using that search state.

[0201] A clamp control device, comprising:

[0202] The point cloud acquisition module is used to acquire point cloud information of the group of items to be captured;

[0203] The search status generation module is used to generate the search status based on the point cloud information of the group of items to be grabbed and the search boundary factor parameters of the fixture.

[0204] The obstacle determination module is used to perform obstacle determination for each search state and set the search state that passes the obstacle determination as the candidate search state.

[0205] The optimal search state determination module is used to select the best search state from the alternative search states;

[0206] The grabbing module is used to grab the item to be grabbed based on the optimal search state.

[0207] Optionally, the group of items to be grabbed may include one or more items to be grabbed and / or obstacles.

[0208] Optionally, it also includes: using a JSON file to save the fixture's configuration parameters.

[0209] Optionally, the obstacle determination module is used to perform at least one of the following obstacle determinations: boundary obstacle determination, fixed obstacle determination, raised / depressed obstacle determination, and custom obstacle determination.

[0210] Optionally, when performing boundary obstacle determination, the obstacle determination module determines whether a boundary obstacle exists based on the relationship between the center of the clamp and the edge contour points of the object.

[0211] Optionally, when performing fixed obstacle determination, the obstacle determination module determines whether a fixed obstacle exists based on the obstacle's size or shape.

[0212] Optionally, when the obstacle determination module performs the protrusion / depression obstacle determination, it determines whether there is a protrusion / depression obstacle based on the two-dimensional planar depth map information.

[0213] Optionally, when performing custom obstacle determination, the obstacle determination module determines whether a custom obstacle exists based on the edge contour of the custom obstacle.

[0214] Optionally, it also includes: performing a clamp check on the optimal search state to determine whether the item can be correctly grabbed using that search state.

[0215] A method for determining non-planar structures based on robots includes:

[0216] Obtain the two-dimensional planar depth map information of the area to be determined on the surface of the object;

[0217] Obtain the depth value of each pixel within the region to be determined based on the two-dimensional planar depth map information;

[0218] Group the obtained depth values;

[0219] Calculate the difference in depth values ​​within each group and compare it with a preset difference threshold; and / or, calculate the number of groups and compare it with a preset group number threshold;

[0220] The non-planar structure of the region is determined based on the comparison results.

[0221] Optionally, after obtaining the depth value of each pixel, for multiple identical depth values, only one is retained.

[0222] Optionally, the grouping includes grouping two different depth values ​​together.

[0223] Optionally, the grouping includes grouping all depth values ​​in the following manner: sorting all acquired depth values ​​from high to low, and grouping the first depth value with the second to last depth value, the second depth value with the second to last depth value, ..., the Nth depth value with the Nth to last depth value, where N is a natural number greater than or equal to 1.

[0224] Optionally, the preset difference threshold and / or group number threshold may include a difference threshold and / or group number threshold preset based on the fixture's capabilities.

[0225] Optionally, the non-planar structure includes either the presence of a non-planar structure or the absence of a non-planar structure.

[0226] Optionally, the non-planar structure includes the degree of undulation of the non-planar structure.

[0227] A robot-based non-planar structure determination device includes:

[0228] The depth map acquisition module is used to acquire a two-dimensional planar depth map of the area to be determined on the surface of an object;

[0229] The depth value acquisition module is used to obtain the depth value of each pixel in the region to be determined based on the two-dimensional planar depth map.

[0230] The grouping module is used to group the obtained depth values;

[0231] The comparison module is used to calculate the difference in depth values ​​within each group and compare it with a preset difference threshold; and / or, calculate the number of groups and compare it with a preset group number threshold.

[0232] The determination module is used to determine the non-planar structure of the region based on the comparison results.

[0233] Optionally, after obtaining the depth value of each pixel, the depth value acquisition module retains only one of the multiple identical depth values.

[0234] Optionally, the grouping module groups two different depth values ​​together.

[0235] Optionally, the grouping module groups all depth values ​​in the following manner: sorts all acquired depth values ​​from high to low, and groups the first depth value with the second to last depth value, the second depth value with the second to last depth value, ..., the Nth depth value with the Nth to last depth value, where N is a natural number greater than or equal to 1.

[0236] Optionally, the preset difference threshold and / or group number threshold may include a difference threshold and / or group number threshold preset based on the fixture's capabilities.

[0237] Optionally, the determination module determines whether a non-planar structure exists or not.

[0238] Optionally, the determination module determines the degree of undulation of the non-planar structure.

[0239] A fixture assembly verification method, comprising:

[0240] Obtain the status information of the fixture group;

[0241] Based on the status information, determine the number of unclamped fixtures and / or the position information of the unclamped fixtures in that status.

[0242] Determine whether the number of openable clamps meets the preset conditions and / or determine whether the position information of the openable clamps meets the preset conditions;

[0243] Based on the judgment result, determine whether the fixture group can perform gripping in this state.

[0244] Optionally, determining the position information of the unclamped clamp includes determining the position information of the unclamped clamp based on the outline information of the item to be gripped.

[0245] Optionally, the number of clamps that can be opened can be preset based on the weight of the item to be grabbed, and the conditions that need to be met.

[0246] Optionally, the position of the clamp can be preset based on the center of gravity of the object to be grasped, and the conditions that need to be met.

[0247] Optionally, the position of the openable clamp must meet the following conditions: the lines connecting the positions of multiple clamps cannot form a straight line.

[0248] Optionally, determining whether the fixture group can perform gripping in this state based on the judgment result includes determining that gripping can be performed when the number of fixtures meets preset conditions and the position of the fixtures meets preset conditions.

[0249] A fixture assembly calibration device, comprising:

[0250] The status information acquisition module is used to acquire the status information of the fixture array;

[0251] The information determination module is used to determine the number of openable clamps and / or the position information of the openable clamps based on the status information.

[0252] The condition determination module is used to determine whether the number of openable clamps meets the preset conditions and / or whether the position information of the openable clamps meets the preset conditions.

[0253] The gripping determination module is used to determine whether the fixture group can perform gripping in this state based on the judgment result.

[0254] Optionally, the information determination module determines the position information of the unclamped clamp based on the outline information of the item to be grasped.

[0255] Optionally, the number of clamps that can be opened can be preset based on the weight of the item to be grabbed, and the conditions that need to be met.

[0256] Optionally, the position of the clamp can be preset based on the center of gravity of the object to be grasped, and the conditions that need to be met.

[0257] Optionally, the position of the openable clamp must meet the following condition: the lines connecting the positions of multiple clamps cannot form a straight line.

[0258] Optionally, the grasping determination module determines that grasping can be performed when the number of clamps meets preset conditions and the clamp positions meet preset conditions.

[0259] A suction cup array control method, comprising:

[0260] Obtain point cloud information of the glass and obstacles to be grasped;

[0261] The search state is generated based on point cloud information and search boundary factor parameters of the suction cup array;

[0262] For each search state, determine whether there is an obstacle under each suction cup; if so, turn off the suction cup in that search state.

[0263] Based on the suction cup activation status of each search state, select alternative search states;

[0264] Select the best search state from the alternative search states based on the number of suction cups opened and / or the distance of the suction cups from the center of the glass.

[0265] Grab the glass using the best search state.

[0266] Optionally, the obstacle determination includes at least one of the following: glass boundary obstacle determination, rubber pad obstacle determination, rubber strip obstacle determination, and protrusion obstacle determination.

[0267] Optionally, the glass boundary obstacle determination includes determining whether a glass boundary obstacle exists based on the proportion of point cloud in the area below the suction cup.

[0268] Optionally, the rubber pad obstacle determination includes: determining whether a rubber pad obstacle exists based on the obstacle point cloud area, wherein the obstacle point cloud area is preset according to the rubber pad area.

[0269] Optionally, the protruding obstacle determination includes: determining whether a protruding obstacle exists based on a pre-set depth difference threshold and a logarithmic threshold.

[0270] Optionally, the step of selecting a candidate search state based on the suction cup opening status of each search state includes: if at least one suction cup is open, then that search state is selected as a candidate search state.

[0271] Optionally, the step of selecting the best search state from the candidate search states based on the number of suction cups opened and / or the distance of the suction cups from the center of the glass includes: selecting the search state with the most suction cups opened; if there are multiple search states with the most suction cups opened, then further selecting the search state with the suction cups closest to the center of the glass.

[0272] Optionally, it also includes: performing a fixture check on the optimal search state to determine whether the glass can be correctly gripped using that search state.

[0273] A suction cup array control device, comprising:

[0274] The point cloud acquisition module is used to acquire point cloud information of the glass and obstacles to be grasped;

[0275] The search state generation module is used to generate the search state based on point cloud information and the search boundary factor parameters of the suction cup array.

[0276] The obstacle detection module is used to determine whether there is an obstacle under each suction cup for each search state. If there is an obstacle, the suction cup is turned off in that search state.

[0277] The alternative search state selection module is used to select alternative search states based on the suction cup opening status of each search state.

[0278] The optimal search state selection module is used to select the optimal search state from the alternative search states based on the number of suction cups opened and / or the distance of the suction cups from the center of the glass.

[0279] The grabbing module is used to grab glass based on the best search state.

[0280] Optionally, the obstacle determination module is used to perform at least one of the following obstacle determinations: glass boundary obstacle determination, rubber pad obstacle determination, rubber strip obstacle determination, and protrusion obstacle determination.

[0281] Optionally, when the obstacle determination module performs a glass boundary obstacle test, it determines whether a glass boundary obstacle exists based on the proportion of point cloud in the area below the suction cup.

[0282] Optionally, when the obstacle determination module performs rubber pad obstacle determination, it determines whether a rubber pad obstacle exists based on the obstacle point cloud area, wherein the obstacle point cloud area is preset according to the rubber pad area.

[0283] Optionally, when the obstacle determination module performs the protruding obstacle determination, it determines whether a protruding obstacle exists based on a pre-set depth difference threshold and a logarithmic threshold.

[0284] Optionally, the alternative search state selection module selects a search state as an alternative search state when at least one suction cup is open in a certain search state.

[0285] Optionally, the optimal search state selection module selects the search state with the most suction cups open as the optimal search state. If there are multiple search states with the most suction cups open, the search state with the suction cups closest to the center of the glass is further selected.

[0286] Optionally, it also includes: performing a fixture check on the optimal search state to determine whether the glass can be correctly gripped using that search state.

[0287] A suction cup array verification method includes:

[0288] Group all the suction cups in the array according to their shape;

[0289] Obtain the status information of the suction cup array;

[0290] The number of operable suction cups and the group information of the operable suction cups are determined based on the status information of the suction cup array.

[0291] Determine whether the number of operable suction cups meets the preset conditions and / or whether the grouping information of operable suction cups meets the preset conditions;

[0292] Based on the judgment result, determine whether the suction cup array can perform grasping in this state.

[0293] Optionally, grouping all suction cups in the array according to the shape of the suction cup array includes dividing the suction cup array into four regions: upper left, lower left, upper right, and lower right, and dividing all suction cups into four groups according to the region where all suction cups are located.

[0294] Optionally, the number of suction cups that can be activated can be preset based on the weight of the item to be grasped, and the conditions that need to be met.

[0295] Optionally, the grouping information that can be activated by the suction cup can be preset based on the center of gravity of the object to be grasped, and the conditions that need to be met.

[0296] Optionally, the grouping information for the operable suction cups must meet the following conditions: the operable suction cups must be distributed in at least three groups.

[0297] Optionally, determining whether the suction cup array can perform grasping in this state based on the judgment result includes: determining that grasping can be performed when the number of suction cups meets a preset condition and the suction cup grouping information meets a preset condition.

[0298] A suction cup array verification device, comprising:

[0299] The grouping module is used to group all the suction cups in the array according to the shape of the suction cup array;

[0300] The status information acquisition module is used to acquire the status information of the suction cup array;

[0301] The information determination module is used to determine the number of operable suction cups and the group information of the operable suction cups based on the status information of the suction cup array.

[0302] The condition determination module is used to determine whether the number of operable suction cups meets the preset conditions and / or whether the grouping information of operable suction cups meets the preset conditions.

[0303] The grasping determination module is used to determine whether the suction cup array can perform grasping in this state based on the judgment result.

[0304] Optionally, the grouping module is specifically used to divide the suction cup array into four regions: upper left, lower left, upper right, and lower right, and to divide all suction cups into four groups according to the regions where all suction cups are located.

[0305] Optionally, the number of suction cups that can be activated can be preset based on the weight of the item to be grasped, and the conditions that need to be met.

[0306] Optionally, the grouping information that can be activated by the suction cup can be preset based on the center of gravity of the object to be grasped, and the conditions that need to be met.

[0307] Optionally, the grouping information for the operable suction cups must meet the following conditions: the operable suction cups must be distributed in at least three groups.

[0308] Optionally, the grasp determination module is specifically used to determine whether grasping can be performed when the number of suction cups meets the preset conditions and the suction cup grouping information meets the preset conditions.

[0309] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0310] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0311] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0312] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0313] It should be understood that various parts of the embodiments of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0314] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0315] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0316] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0317] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A suction cup array verification method, characterized in that, include: Group all the suction cups in the array according to their shape; Obtain the status information of the suction cup array; The number of operable suction cups and the group information of the operable suction cups are determined based on the status information of the suction cup array. Determine whether the number of operable suction cups meets the preset conditions and whether the grouping information of the operable suction cups meets the preset conditions. Based on the judgment result, determine whether the suction cup array can perform grasping in this state; Specifically, after determining the grasping method to be used, and before grasping the item, the suction cup array verification method is used to verify the grasping method to be used; The conditions that need to be met for the grouping information of the suction cup to be activated based on the preset center of gravity of the item to be grasped.

2. The suction cup array verification method according to claim 1, characterized in that, The step of grouping all the suction cups in the array according to the shape of the suction cup array includes dividing the suction cup array into four regions: upper left, lower left, upper right, and lower right, and dividing all the suction cups into four groups according to the region where all the suction cups are located.

3. The suction cup array verification method according to claim 1, characterized in that: The number of suction cups that can be activated is preset based on the weight of the item to be grabbed, and the conditions that need to be met.

4. The suction cup array verification method according to claim 2, characterized in that, The grouping information for operable suction cups must meet the following conditions: operable suction cups must be distributed in at least three groups.

5. The suction cup array verification method according to claim 1, characterized in that, Determining whether the suction cup array can perform grasping in this state based on the judgment result includes: when the number of suction cups meets the preset conditions and the suction cup grouping information meets the preset conditions, it is determined that grasping can be performed.

6. A suction cup array verification device, characterized in that, include: The grouping module is used to group all the suction cups in the array according to the shape of the suction cup array; The status information acquisition module is used to acquire the status information of the suction cup array; The information determination module is used to determine the number of operable suction cups and the group information of the operable suction cups based on the status information of the suction cup array. The condition determination module is used to determine whether the number of operable suction cups meets the preset conditions and whether the grouping information of operable suction cups meets the preset conditions. The grasping determination module is used to determine whether the suction cup array can perform grasping in this state based on the judgment result. Specifically, after determining the gripping method to be used, and before gripping the item, the suction cup array verification device is used to verify the gripping method to be used; The conditions that need to be met for the grouping information of the suction cup to be activated based on the preset center of gravity of the item to be grasped.

7. The suction cup array verification device according to claim 6, characterized in that, The grouping module is specifically used to divide the suction cup array into four regions: upper left, lower left, upper right, and lower right, and to divide all suction cups into four groups according to the regions where all suction cups are located.

8. The suction cup array verification device according to claim 6, characterized in that: The number of suction cups that can be activated is preset based on the weight of the item to be grabbed, and the conditions that need to be met.

9. The suction cup array verification device according to claim 7, characterized in that, The grouping information for operable suction cups must meet the following conditions: operable suction cups must be distributed in at least three groups.

10. The suction cup array verification device according to claim 6, characterized in that, The grasp determination module is specifically used to determine whether grasping can be performed when the number of suction cups meets the preset conditions and the suction cup grouping information meets the preset conditions.

11. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the suction cup array verification method according to any one of claims 1 to 5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the suction cup array verification method according to any one of claims 1 to 5.

Images