Automatic grabbing method and automatic grabbing system

Abstract

An automatic grabbing method and an automatic grabbing system. The automatic grabbing method comprises: building an AI model for recognizing grabbable objects (2a) in a predetermined region (1); acquiring a two-dimensional image of the predetermined region (1); on the basis of the AI model, determining, in the two-dimensional image, a target area (1a) corresponding to one or more grabbable objects (2a); when the target area (1a) meets a first confidence condition, acquiring a three-dimensional image of the target area (1a) and generating a point cloud for the one or more grabbable objects (2a); when the point cloud for the one or more grabbable objects (2a) meets a second confidence condition, determining positioning information about the position and orientation of at least one target object among the one or more grabbable objects (2a) on the basis of the corresponding point cloud, and sending the positioning information to a robot; and the robot grabbing at least one target object from the preset region on the basis of the positioning information. The automatic grabbing method and the automatic grabbing system can improve the object recognition effect.

Description

Automatic grasping method and automatic grasping system Technical Field

[0001] The present invention relates to the field of robotics, and in particular to an automatic grasping method and an automatic grasping system for a robot. Background Art

[0002] Currently, robots are increasingly used in scenarios such as assembly line production and cargo sorting. Robots can identify items in a predetermined area and accurately grasp the identified items for further operations (such as placement, processing, etc.). Currently commonly used automatic grasping systems are usually composed of computers, three-dimensional image acquisition components, robot operating systems, machine vision software and algorithms, AI (artificial intelligence) models, and robot path planning software. This automatic grasping system is based on a three-dimensional vision solution, that is, it uses three-dimensional images to identify graspable items in a predetermined area. However, for items with a smaller thickness, it takes a longer time to generate a three-dimensional point cloud due to the larger field of view; and for smaller items, due to the relatively large area of ​​the point cloud scene, it is difficult to accurately identify graspable items from the predetermined area, resulting in a low recognition rate.

[0003] Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to provide an automatic grasping method and an automatic grasping system that can improve the object recognition effect.

[0005] The above technical problems are solved by an automatic grasping method according to the present invention. The automatic grasping method is used to control a robot to automatically grasp items in a predetermined area. The automatic grasping method includes: a model building step, in which an AI model is built to identify graspable items in the predetermined area; a two-dimensional image acquisition step, in which a two-dimensional image of the predetermined area is acquired; a target range determination step, in which a target range corresponding to one or more graspable items is determined in the two-dimensional image based on the AI ​​model; a three-dimensional point cloud generation step, in which, when the target range meets a first confidence condition, a three-dimensional image of the target range is acquired and a point cloud of the one or more graspable items is generated; a positioning information determination step, in which, when the point cloud of the one or more graspable items meets a second confidence condition, positioning information regarding the position and posture of at least one target item among the one or more graspable items is determined based on the corresponding point cloud; and a robot grasping step, in which the robot grasps the at least one target item from the predetermined area based on the positioning information. In this grasping method, graspable items are first identified in the two-dimensional image and a corresponding target range is determined. Then, only the graspable items within the target range are further identified in the three-dimensional image. In this way, the object of three-dimensional image processing is narrowed down from the entire predetermined area to the target range obtained by two-dimensional image processing, thereby improving the speed and accuracy of object recognition.

[0006] According to a preferred embodiment of the present invention, the model building step may include training the AI ​​model. Training the AI ​​model includes: providing a predetermined number of training two-dimensional images, where each training two-dimensional image includes multiple objects; manually labeling the graspable objects in each training two-dimensional image; and validating the AI ​​model using a validation set. Training can improve the recognition accuracy of the AI ​​model.

[0007] According to another preferred embodiment of the present invention, the automatic grasping method may further include an object positioning adjustment step, wherein, after the target range determination step, if the target range does not meet a first confidence condition, and / or after the 3D point cloud generation step, if the point cloud of one or more graspable objects does not meet a second confidence condition, the position and / or posture of the objects in the predetermined area are adjusted, and then the method returns to the 2D image acquisition step. This allows for the redistribution of objects and the restart of graspable object identification when no graspable objects that can be accurately identified exist in the predetermined area.

[0008] According to another preferred embodiment of the present invention, the first confidence condition may be that the confidence level of the target range is greater than 0.99; and / or the second confidence condition may be that the confidence level of the point cloud is greater than 0.99. Performing recognition at a sufficiently high confidence level is conducive to accurately grasping the object.

[0009] According to another preferred embodiment of the present invention, the step of generating the three-dimensional point cloud may further include optimizing the point cloud using an interpolation algorithm. The interpolation algorithm can improve the accuracy of the point cloud, thereby providing more accurate positioning information for the robot.

[0010] According to another preferred embodiment of the present invention, the positioning information determining step may further include selecting the at least one target object from the one or more graspable objects based on the confidence level of the point clouds of the one or more graspable objects when the point clouds of the one or more graspable objects meet a second confidence condition. In this way, when multiple graspable objects are identified, a predetermined number of objects can be selected for grasping based on the confidence level.

[0011] The above technical problem is also solved by an automatic grasping system according to the present invention. The automatic grasping system includes a robot for automatically grasping items in a predetermined area. The automatic grasping system also includes: a model building module, which is configured to build an AI model for identifying graspable items in a predetermined area; a two-dimensional camera, which is configured to capture a two-dimensional image of the predetermined area; a target range determination module, which is configured to determine a target range corresponding to one or more graspable items in the two-dimensional image based on the AI ​​model; a three-dimensional camera, which is configured to capture a three-dimensional image of the target range when the target range meets a first confidence condition; a point cloud generation module, which is configured to generate a point cloud of one or more graspable items based on the three-dimensional image captured by the three-dimensional camera; and a positioning information determination module, which is configured to determine positioning information about the position and posture of at least one target item among the one or more graspable items based on the corresponding point cloud when the point cloud of the one or more graspable items meets a second confidence condition; wherein the robot is configured to grasp the at least one target item from the predetermined area according to the positioning information. The automated grasping system first identifies graspable items in a 2D image and determines a corresponding target range. It then further identifies only those graspable items within the target range in a 3D image. This reduces the 3D image processing area from the entire predetermined area to the target range obtained through 2D image processing, improving both the speed and accuracy of object recognition.

[0012] According to a preferred embodiment of the present invention, the automated grasping system may further include a flexible feeding device configured to carry items within a predetermined area and adjust the position and / or posture of the items within the predetermined area when a target range does not satisfy a first confidence condition and / or the point cloud of the one or more graspable items does not satisfy a second confidence condition. The predetermined area may be provided on a carrying surface of the flexible feeding device, and the flexible feeding device may shake the carrying surface to cause the items within the predetermined area to change position and / or posture.

[0013] According to another preferred embodiment of the present invention, the automatic grasping system may further include a reverse light source and an enhanced light source for objects in a predetermined area. The reverse light source may be disposed in the flexible feeding device. The reverse light source and the enhanced light source are used to provide suitable lighting conditions when the 2D camera captures the 2D image.

[0014] According to another preferred embodiment of the present invention, the model building module, the target range determination module, the point cloud generation module, and the positioning information determination module can be integrated into a computer processor. The processor can be connected to other hardware devices via wired or wireless communication to send or receive relevant signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention is further described below with reference to the accompanying drawings. Elements with the same function are represented by the same reference numerals in the drawings.

[0016] FIG1 shows a flow chart of an automatic grasping method according to an exemplary embodiment of the present invention;

[0017] FIG2 shows a flow chart of model training according to an exemplary embodiment of the present invention;

[0018] FIG3 is a schematic diagram showing a target range of an automatic grasping method according to an exemplary embodiment of the present invention; and

[0019] FIG. 4 shows a schematic diagram of a point cloud of a self-automatic grasping method according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes the specific embodiments of the automatic grasping method and automatic grasping system according to the present invention in conjunction with the accompanying drawings. The following detailed description and accompanying drawings are used to illustrate the principles of the present invention. The present invention is not limited to the preferred embodiments described. The scope of protection of the present invention is defined by the claims.

[0021] According to an embodiment of the present invention, an automatic grasping method for a robot is provided. Specifically, the automatic grasping method is used to identify graspable objects from a predetermined area where one or more objects are placed, so as to control the robot to automatically grasp the objects from the predetermined area.

[0022] Figure 1 shows a flow chart of an automatic grasping method according to an exemplary embodiment of the present invention. The automatic grasping method will be described below in conjunction with Figure 1. As shown in Figure 1, the automatic grasping method mainly includes steps S1 to S6.

[0023] In this automatic grasping method, a model building step S1 is first performed. This step builds an AI (artificial intelligence) model for identifying graspable items in a predetermined area 1. The AI ​​model is the foundation for performing two-dimensional image recognition and can be built based on a deep learning neural network algorithm. The AI ​​model can be stored as a software program in a computer processor, particularly a graphics processing unit (GPU). The AI ​​model influences the accuracy of image recognition.

[0024] Preferably, in the model building step S1, a preliminary AI model can be initially established, and then the original AI model can be trained to obtain an improved AI model. Specifically, as shown in Figure 2, training the AI ​​model includes the following steps. First, in step S11, a predetermined number of training 2D images are provided, where each training 2D image includes multiple items. The training 2D images can be collected on-site or from a stored database. The number of training 2D images can be set as needed, for example, 600, or more or less. Each training 2D image includes one or more graspable items, i.e., items placed on the top layer and therefore directly graspable by the robot. Preferably, these training 2D images can include items of various sizes and / or shapes and / or distributed in different ways. Next, in step S12, an operator manually marks the graspable items in each training 2D image (for example, via a human-computer interaction tool such as a touchscreen). The AI ​​model thereby learns which items in each training 2D image are graspable. Finally, in step S13, the AI ​​model is validated using a validation set. The validation process includes determining the accuracy and loss rate of the AI ​​model. The verified AI model can be used as a complete model for subsequent image processing and object recognition.

[0025] As shown in Figure 1, after the required AI model is obtained, the two-dimensional image acquisition step S2 is performed. The content of the two-dimensional image acquisition step S2 is to acquire a two-dimensional image of the predetermined area 1. One or more items 2 are placed in the predetermined area 1. These items 2 may have the same or different shapes and / or sizes and may be distributed in the predetermined area 1 in any manner (for example, in a discrete manner, an overlapping manner, or a combination thereof). The acquired two-dimensional image of the predetermined area 1 contains image information of the items 2 therein. An imaging device such as a two-dimensional camera can be used to acquire a two-dimensional image of the predetermined area 1. The two-dimensional camera is arranged above the predetermined area 1 and the items 2 therein. When using a two-dimensional camera to acquire a two-dimensional image, a reverse light source arranged below the item 2 and an enhanced light source arranged above the item 2 can be used to provide lighting. The enhanced light source can preferably be integrated with the two-dimensional camera.

[0026] After obtaining the two-dimensional image, the target range determination step S3 is performed. The target range determination step S3 includes determining the target range corresponding to one or more graspable items 2a in the acquired two-dimensional image based on the AI ​​model. That is, the AI ​​model identifies one or more graspable items 2a and the corresponding target range in the acquired two-dimensional image. The target range here is not necessarily the exact graphic range that corresponds exactly to the identified graspable items 2a, but can be an area slightly larger than the actual coverage range of the identified graspable items 2a. The target range can be a continuous area or a set of discrete areas consisting of multiple discrete graspable items. For example, as shown in Figure 3, multiple discrete graspable items 2a and multiple discrete areas containing these graspable items 2a are identified in the two-dimensional image of the predetermined area 1, and these discrete areas together constitute the target range 1a. The target area for the subsequent three-dimensional image processing is the target range 1a determined here, rather than the entire predetermined area 1.

[0027] Next, the automatic grasping method will perform a confidence judgment on the target range 1a obtained in step S3, and execute the three-dimensional point cloud generation step S4 when the target range 1a meets the first confidence condition. The first confidence condition is usually that the confidence of the target range 1a reaches a predetermined numerical range, for example, greater than 0.99, which means that the recognition result of the two-dimensional image has a higher accuracy. The three-dimensional point cloud generation step S4 includes collecting a three-dimensional image of the target range 1a, and thereby generating a point cloud corresponding to the graspable item 2a determined in step S3. The acquisition of the three-dimensional image can be achieved, for example, by a three-dimensional camera, which can be arranged above the predetermined area 1 and the item 2 therein, and preferably can be integrated with a two-dimensional camera. When using a three-dimensional camera to collect three-dimensional images, lighting can be provided by a structured light source. The structured light source can preferably be turned on only when the three-dimensional camera collects three-dimensional images and turned off when the two-dimensional camera collects two-dimensional images.

[0028] Figure 4 shows a schematic diagram of point clouds of multiple graspable items 2a generated in step S4. As shown in Figure 4, the point cloud generated based on the 3D image represents a collection of surface points of the graspable item 2a. Because the resolution of the initially generated point cloud is limited, step S4 of generating the 3D point cloud may also include optimizing the point cloud initially generated based on the 3D image using an interpolation algorithm to improve the accuracy of the point cloud's contours and feature recognition. The optimized point cloud can be used in subsequent steps.

[0029] The automatic grasping method then performs a confidence assessment on the point cloud obtained in step S4. If the point cloud obtained in step S4 for the graspable items 2a satisfies a second confidence condition, the positioning information determination step S5 is executed. Similar to the first confidence condition, the second confidence condition typically requires that the confidence level of the point cloud for the graspable items 2a falls within a predetermined numerical range, for example, greater than 0.99, indicating a high degree of accuracy in the three-dimensional image recognition result. Positioning information determination step S5 includes determining the positioning information of at least one target object based on the point cloud. The target object is an object to be grasped selected from the graspable items 2a described above. Each time step S5 is executed, the number of target objects determined can be singular (grasping only one object at a time) or plural (grasping multiple objects at a time), and can be determined based on a predetermined number and not greater than the number of graspable items 2a determined. Preferably, the target object is selected based on the confidence level of the point cloud for the graspable items 2a. Specifically, when the number of graspable items 2a is greater than the predetermined single-grab number, the item with the highest confidence can be selected as the target item from these graspable items 2a in descending order of confidence according to the predetermined single-grab number; when the number of graspable items 2a is less than or equal to the predetermined single-grab number, all graspable items 2a can be directly selected as target items.

[0030] Based on the point cloud corresponding to the target object, the target object's location information is obtained. This location information includes the target object's position and posture within the predetermined area 1. The target object's position and posture within the predetermined area 1 can be expressed as three-dimensional coordinate data. This process can be performed based on the PCL (Point Cloud Library) point cloud processing and coordinate generation algorithm. The determined location information of the target object is transmitted to the robot.

[0031] After the robot receives the target object's location information, it executes the robot grasping step S6. In step S6, the robot grasps the target object from the predetermined area 1 based on the received location information. The robot can then perform subsequent operations on the grasped target object, such as placing it in a designated area or processing it.

[0032] As shown in FIG1 , the automatic grasping method may preferably further include an item positioning adjustment step S7 to modify the position and / or posture of the item 2 in the predetermined area 1 when a precisely identified graspable item 2a is missing. The item positioning adjustment step S7 includes adjusting the position and / or posture of the item in the predetermined area 1 after the target range determination step S3 when the target range 1a does not meet a first confidence condition, and / or after the 3D point cloud generation step S4 when the point clouds of one or more graspable items 2a obtained do not meet a second confidence condition, and then returning to the 2D image acquisition step S2.

[0033] Preferably, the position and / or posture of the items in the predetermined area 1 can be adjusted, for example, using a flexible feeding device. Specifically, the predetermined area 1 can be arranged on the flexible feeding device, and the items in the predetermined area 1 are thereby carried by the flexible feeding device. The flexible feeding device can be shaken to change the position and / or posture of the items in the predetermined area 1. After the item positioning adjustment step S7, the items in the predetermined area 1 are redistributed, and the automatic grasping method is restarted from the two-dimensional image acquisition step S2.

[0034] According to an embodiment of the present invention, an automatic grasping system is also provided, which is configured to execute the automatic grasping method according to the present invention. The automatic grasping system includes a model building module, a two-dimensional camera, a target range determination module, a three-dimensional camera, a point cloud generation module, a positioning information determination module, and a robot.

[0035] The model building module is configured to correspondingly execute the model building step S1 of the automatic grasping method described above. Specifically, the model building module is configured to build an AI model for identifying graspable items in predetermined area 1. During the process of building the AI ​​model, the model building module may train the AI ​​model. The process of building and training the AI ​​model by the model building module is as described in step S1 of the automatic grasping method and will not be further elaborated here.

[0036] The two-dimensional camera is used to correspondingly execute step S2 of the automatic grasping method described above for acquiring two-dimensional images. Specifically, the two-dimensional camera is configured to acquire a two-dimensional image of a predetermined area 1. The automatic grasping system may also preferably include a reverse light source for providing reverse light to the two-dimensional camera and an enhanced light source for providing enhanced light to the two-dimensional camera. The process of acquiring a two-dimensional image with the two-dimensional camera is as described in step S2 of the automatic grasping method and will not be further elaborated here.

[0037] The target range determination module is configured to correspondingly execute step S3 of the automatic grasping method described above. The target range determination module is configured to determine a target range 1a corresponding to one or more graspable items 2a in the two-dimensional image based on the AI ​​model. The process by which the target range determination module determines the target range 1a is as described in step S3 of the automatic grasping method and will not be further elaborated here.

[0038] The 3D camera and point cloud generation module are used to correspondingly execute step S4 of the automatic grasping method described above. The 3D camera is configured to capture a 3D image of the target range 1a when the target range 1a meets the first confidence condition, and the point cloud generation module is configured to generate a point cloud of the one or more graspable items 2a based on the 3D image captured by the 3D camera. Preferably, the 3D camera and the 2D camera can be integrated. The automatic grasping system may also preferably include a structured light source for providing structured light to the 3D camera. The specific operating procedures of the 3D camera and the point cloud generation module are as described in step S4 of the automatic grasping method and will not be repeated here.

[0039] The positioning information determination module is configured to correspondingly execute the positioning information determination step S5 of the aforementioned automatic grasping method. When the point clouds of one or more graspable items 2a satisfy the second confidence condition, the positioning information determination module is configured to determine positioning information regarding the position and posture of at least one target item among the graspable items 2a based on the corresponding point clouds. The process for determining positioning information by the positioning information determination module is as described in step S5 of the automatic grasping method and will not be further elaborated here.

[0040] The robot is used to correspondingly execute the robot grasping step S6 of the above automatic grasping method. The robot obtains the positioning information of the target object provided by the positioning information determination module and grasps the determined target object from the predetermined area 1 according to the positioning information.

[0041] Preferably, the automated grasping system may further include a flexible feeding device. As described in the embodiments of the automated grasping method, the flexible feeding device is configured to carry items in the predetermined area 1 and adjust the position and / or posture of the items in the predetermined area 1 when the target range does not meet a first confidence condition and / or the point cloud of one or more graspable items 2a does not meet a second confidence condition. Preferably, the reverse light source for the 2D camera may be disposed in the flexible feeding device.

[0042] In the above automatic grasping system, the model building module, the target range determination module, the point cloud generation module and the positioning information determination module can be integrated into a computer processor as software programs.

[0043] The automatic grasping method and system of the present invention combine two-dimensional and three-dimensional vision technologies, first using two-dimensional vision to determine the target area, and then generating a point cloud for the designated target area. Compared to existing technologies that directly generate point clouds for the entire scene, the automatic grasping method and system of the present invention significantly reduce the target area for three-dimensional image processing, thereby shortening point cloud generation time, reducing the difficulty of target recognition, and improving grasping accuracy. This automatic grasping method and system can particularly improve the recognition and grasping efficiency of small and thin objects.

[0044] While the foregoing descriptions illustrate possible embodiments, it should be understood that numerous variations exist through combinations of all known and other technical features and implementations readily conceivable to a skilled artisan. Furthermore, it should be understood that the exemplary embodiments serve merely as examples and in no way limit the scope, application, or configuration of the present invention. The foregoing descriptions are intended primarily to provide a skilled artisan with technical guidance for implementing at least one exemplary embodiment. Various modifications, particularly regarding the functionality and structure of the components described, may be made without departing from the scope of the claims.

[0045] Reference Signs List 1 Predetermined area 1a Target range 2 Object 2a Graspable object

Claims

1. An automatic grasping method for controlling a robot to automatically grasp an item in a predetermined area (1), characterized in that, The automatic grasping method includes: A model establishment step (S1), in which an AI model is established for identifying graspable items (2a) in the predetermined area (1); A two-dimensional image acquisition step (S2), in which a two-dimensional image of the predetermined area (1) is acquired; A target range determination step (S3), in which a target range (1a) corresponding to one or more graspable items (2a) is determined in the two-dimensional image based on the AI model; A three-dimensional point cloud generation step (S4), in which when the target range (1a) meets the first confidence condition, a three-dimensional image of the target range (1a) is acquired and a point cloud of the one or more graspable items (2a) is generated; A positioning information determination step (S5), in which when the point cloud of the one or more graspable items (2a) meets the second confidence condition, positioning information about the position and pose of at least one target item among the one or more graspable items (2a) is determined based on the corresponding point cloud; and A robot grasping step (S6), in which the robot grasps the at least one target item from the predetermined area (1) according to the positioning information.

2. The automatic grabbing method according to claim 1, wherein The model establishment step (S1) includes training the AI model, and training the AI model includes: Providing a predetermined number of training two-dimensional images, where each training two-dimensional image includes multiple items; Manually marking the graspable items (2a) in each training two-dimensional image; and Using a validation set to validate the AI model.

3. The automatic grabbing method according to claim 1, characterized in that, The automatic grasping method further includes an item positioning adjustment step (S7), in which after the target range determination step (S3) when the target range (1a) does not meet the first confidence condition, and / or after the three-dimensional point cloud generation step (S4) when the point cloud of the one or more graspable items (2a) does not meet the second confidence condition, the position and / or pose of the items in the predetermined area (1) is adjusted, and then the process returns to the two-dimensional image acquisition step (S2).

4. The automatic grasping method according to claim 1, wherein The first confidence condition is that the confidence of the target range (1a) is greater than 0.99; and / or The second confidence condition is that the confidence of the point cloud is greater than 0.

99.

5. The automatic grasping method according to claim 1, wherein The three-dimensional point cloud generation step (S4) further includes optimizing the point cloud using an interpolation algorithm.

6. According to any one of claims 1 to 5, characterized in that, The positioning information determination step (S5) further includes when the point cloud of the one or more graspable items (2a) meets the second confidence condition, selecting the at least one target item from the one or more graspable items (2a) based on the confidence of the point cloud of the one or more graspable items (2a).

7. An automatic grasping system, comprising a robot for automatically grasping an article in a predetermined area (1), characterized in that, The automatic grasping system further includes: A model establishment module configured to establish an AI model for identifying graspable items (2a) in the predetermined area (1); A two-dimensional camera configured to acquire a two-dimensional image of the predetermined area (1); A target range determination module configured to determine a target range (1a) corresponding to one or more graspable items (2a) in the two-dimensional image based on the AI model; A three-dimensional camera configured to acquire a three-dimensional image of the target range (1a) when the target range (1a) meets a first confidence condition; A point cloud generation module configured to generate a point cloud of the one or more graspable items (2a) based on the three-dimensional image acquired by the three-dimensional camera; and A positioning information determination module configured to determine positioning information about the position and pose of at least one target item among the one or more graspable items (2a) based on the corresponding point cloud when the point cloud of the one or more graspable items (2a) meets a second confidence condition; Wherein the robot is configured to grasp the at least one target item from the predetermined area (1) according to the positioning information.

8. The automatic grasping system according to claim 7, characterized in that, The automatic grasping system further includes a flexible feeding device configured to carry the items in the predetermined area (1) and adjust the position and / or pose of the items in the predetermined area (1) when the target range (1a) does not meet the first confidence condition and / or the point cloud of the one or more graspable items (2a) does not meet the second confidence condition.

9. The automatic grasping system according to claim 8, wherein The automatic grasping system further includes a backlight source and an enhancement light source for the items in the predetermined area (1), and the backlight source is arranged in the flexible feeding device.

10. The automatic grasping system according to any one of claims 7 to 9, characterized in that, The model establishment module, the target range determination module, the point cloud generation module, and the positioning information determination module are integrated in the processor of the computer.

Images