A robot grasping detection method and system for occlusion and cluttered scenes

Abstract

The present application belongs to the field of machine vision, robot application and artificial intelligence, in particular to a robot grasping detection method and system for occlusion and cluttered scenes, comprising the following steps: collecting color images and depth maps of target workpieces in different poses in the target scene; training the color images to obtain a detection training model; and preparing a point cloud template library; collecting color images and depth maps of each target workpiece, and inputting the color images into the detection training model to obtain the positions and categories of each target workpiece in the color images; and obtaining the surface point clouds of each target workpiece collected; simultaneously performing point cloud registration with the workpiece point cloud template library to obtain the current poses of each target workpiece; and using an unordered grasping strategy to grasp according to the workpiece point cloud template library, the depth camera external parameters and the current poses of each target workpiece obtained. In the case that the training set input to the neural network is sufficient, the deep learning network recognition algorithm has high accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of machine vision, robot application, artificial intelligence, and specifically to a robot grasping detection method and system for occlusion and cluttered scenes. BACKGROUND

[0002] At present, in the industrial field, especially in the logistics industry, there are many technical researches on the recognition, positioning and grasping of simple workpieces. In recent years, with the increasing complexity of operation scenes, the application requirements for the accuracy, operability and speed of workpiece grasping are increasing, and the existing grasping technology has problems such as inaccurate visual detection, complex operation, simple application scene, large amount of calculation, etc. Therefore, a real-time and efficient visual grasping algorithm is needed.

[0003] IntelliPICK 3D system (Rasolzadeh B, Kootselaar E, 2017) of ISRA VISION company in Germany M, Huebner K, et al. An Active Vision System for Detecting, Fixating and Manipulating Objects in the Real World. [J]. International Journal of Robotics Research, 2011, 29(2-3): 133-154.) can identify and grasp special-shaped parts among various stacked parts, but its focus is on excluding non-target objects and there is no grasping strategy for the target. Li Miao invented a method of finding the target center point in the image coordinate system of the detected target for grasping, but it cannot solve the problem of target occlusion (Li Miao, Yan Lin, Zhang Shaohua, Sun Bin, Zhu Xiangjiang, Pan Xuyang, Mingpeng, Zhanjun, Fu Zhongtao, Fang Xiaoli, Zhou Xinzhao. A robot unordered grasping device and method based on single camera [P]. Hubei: CN107009358A, 2017-08-04.); Lu Zhiliang proposed two algorithms, an operation relationship reasoning algorithm that integrates position factors and a first-order grasping detection algorithm that embeds a channel attention structure (Lu Zhiliang. Research on stacking object grasping method based on deep learning [D]. Guangdong University of Technology, 2020. DOI:10.27029 / d.cnki.ggdgu.2020.000350.). But it only has a certain accuracy in simple scenes containing a single object, and the deviation is more obvious when in complex scenes with multiple objects; Song Yanan proposed an improved and sound search algorithm to find the optimal detection result combination to solve the problem that the non-maximum suppression algorithm cannot effectively handle the occlusion environment problem, and proposed a point cloud coding method based on local region search for how to make the deep learning network efficiently use the local spatial information of the point cloud (Song Yanan. Object recognition and robot grasping detection method based on deep learning [D]. Huazhong University of Science and Technology, 2020. DOI:10.27157 / d.cnki.ghzku.2020.001429.), but it does not solve the problem of object self-occlusion or mutual occlusion, and the distortion of two-dimensional images and the disorder and unstructured nature of three-dimensional point clouds also need further research.

[0004] With the continuous development of industrial automation, robot grasping technology has gradually been applied to production practice. The current industrial environment often faces complex scenes, unordered placement of target workpieces, and stacking, so a visual detection and grasping algorithm with flexible and intelligent manufacturing performance is needed. Most existing detection and grasping methods are based on manual features combined with prior models, and the overall accuracy is low, which cannot achieve optimal grasping. SUMMARY

[0005] This invention proposes a robot grasping and detection method for occluded and cluttered scenes: based on visual detection 6D pose estimation and combined with a deep learning network, the method takes the color image, depth image and object point cloud template library of the detected object as input and the 6D pose estimation matrix as output, and proposes a highly applicable grasping strategy that can effectively grasp in such situations.

[0006] The technical solution adopted by the present invention to achieve the above objectives is: a robot grasping and detection method for occluded and cluttered scenes, comprising the following steps:

[0007] 1) For each target workpiece: Take pictures of the target workpiece using a depth camera, and collect color images of the target workpiece in different poses in the target scene, as well as the depth images corresponding to the color images;

[0008] 2) Train the color image to obtain the detection training model H; and create a point cloud template library for the target workpiece;

[0009] 3) Acquire color images of each target workpiece in the target scene and the corresponding depth images using a depth camera, and input the color images of the target workpieces into the detection training model H to obtain the position and category of each target workpiece in the color images;

[0010] 4) Based on the location of the target workpiece in the color image, map it onto the depth map, and use the camera intrinsic parameters to obtain the surface point cloud O of each target workpiece. i Where i represents i target workpieces;

[0011] 5) Collect the surface point cloud O of each target workpiece i Point cloud registration is performed with the workpiece point cloud template library to obtain the current pose S of each detection target workpiece. i ;

[0012] 6) Based on the workpiece point cloud template library, combined with the extrinsic parameters of the depth camera and the current pose S of each detected target workpiece, i The target workpiece is grasped by adopting an unordered grasping strategy.

[0013] Step 2) includes the following steps:

[0014] 2-1) Annotate the color image of each target workpiece, marking the outline information and workpiece category of the target workpiece, and generate a mask image;

[0015] 2-2) Send the workpiece category from the mask image to the workpiece point cloud template library;

[0016] 2-3) Based on the contour information in the mask image, obtain the pixel position occupied by the target workpiece in each color image and generate a dataset. Perform dataset augmentation processing on the dataset containing the pixel position occupied by the target workpiece in the color image to expand the dataset.

[0017] 2-4) Input the expanded dataset into the deep neural network for training to obtain the detection training model H;

[0018] 2-5) Based on the 3D structure of the workpiece in the depth map of each target workpiece, create a workpiece point cloud template library using 3D drawing software.

[0019] The dataset augmentation process includes:

[0020] The obtained dataset was then processed and expanded using rotation, noise addition, and simulated lighting methods in sequence.

[0021] The rotation refers to rotating the color image by a set angle;

[0022] The simulated lighting refers to changing the contrast and saturation of the color image.

[0023] The detection training model H is the Mask R-CNN deep learning model.

[0024] Step 5) specifically involves:

[0025] The surface point cloud O of each target workpiece was acquired using the iterative nearest point method. i Perform registration; based on the category of the target workpiece, select the complete point cloud template of the target workpiece from the workpiece point cloud template library, and make the point cloud O i Registration is performed with the workpiece point cloud template library to sequentially obtain the current pose S of each detection target workpiece. i .

[0026] Step 6), which involves using a disordered grasping strategy to grasp the target workpiece, includes the following steps:

[0027] 6-1) Calculate the pose S of each target workpiece at this position. i At that time, the surface point cloud under the camera and the object point cloud when there is no occlusion: combining the workpiece point cloud template library, camera extrinsic parameters and the current pose S of the actual workpiece. i Through camera simulation rendering, the position S of the target workpiece under this camera's viewpoint is obtained. i Unoccluded single-view point cloud T i ;

[0028] 6-2) Sequentially acquire the single-view point cloud T of each target workpiece under unobstructed conditions. i Surface area SA TiSimultaneously, the point cloud O of each target workpiece in the actual scene is acquired sequentially. i Surface SA Oi Where i represents the i-th target workpiece;

[0029] 2) Obtain the point cloud O of each target workpiece in the actual scene. i Surface area SA Oi And its unobstructed single-view point cloud T i Surface area SA Ti One-to-one correspondence, obtaining the non-occlusion rate of the target workpiece, that is:

[0030]

[0031] Where i represents the i-th target workpiece, SA IOUi It is the ratio of the point cloud surface area of ​​the workpiece in the actual scene to the complete point cloud surface area of ​​the workpiece in the template library, that is, the non-occlusion rate of the target workpiece.

[0032] 3) Compare the uncoverage rate (SA) for all target workpieces. IOUi Set a threshold σ to determine whether occlusion affects the captured metrics, where the value of σ is set according to the actual situation;

[0033] 4)SA IOUi If the value is greater than σ, the grasping requirement is met. σ ranges from 0 to 1. The closer it is to 1, the smaller the occluded surface is, and vice versa.

[0034] 5) In all cases where SA is satisfied IOUi Among the target workpieces with a value greater than σ, compare the centroids of each target workpiece and prioritize grabbing the target workpiece with the highest centroid position.

[0035] A detection system for robot grasping and detection in occluded and cluttered scenes includes:

[0036] The image acquisition module is used to acquire color images of the target workpiece in different poses of the target scene obtained by the depth camera scanning each target workpiece, as well as the depth images of the target workpiece to be detected containing each target workpiece and the depth images of the target workpieces to be detected, as well as the depth images of the target workpieces to be detected, as well as the depth images of the target workpieces to be detected, and send them to the model building module.

[0037] The image processing module is used to annotate the acquired color image of the target workpiece, annotate the outline information and workpiece category of the target workpiece, and generate a mask image.

[0038] The model training module is used to obtain the pixel positions occupied by the target workpiece in the color image based on the contour information in the mask image, and generate a dataset. The dataset containing the pixel positions occupied by the target workpiece is augmented to expand the dataset. The expanded dataset is then input into the deep neural network for training to obtain the detection training model H.

[0039] The workpiece point cloud template library is used to create a workpiece template for each target workpiece based on the workpiece category in the mask image and the three-dimensional structure of the depth map of each target workpiece.

[0040] The point cloud registration module is used to select the point cloud template of the target workpiece from the workpiece point cloud template library according to the category of the target workpiece, and register it with the workpiece point cloud template library by means of the iterative nearest point method, so as to obtain the current pose S of each target workpiece to be detected. i ;

[0041] The target workpiece grasping module is used to grasp the workpiece point cloud template library in combination with the extrinsic parameters of the depth camera and the current pose S of each detected target workpiece. i The target workpiece is grasped by adopting an unordered grasping strategy.

[0042] The present invention has the following beneficial effects and advantages:

[0043] 1. This invention selects a detection method that combines deep learning with 6D pose estimation, which improves the extraction of effective features. Moreover, most grasping algorithms cannot solve the problem of recognizing numerous targets and complex backgrounds. When objects are occluded, the grasping efficiency will decrease. This invention proposes a simple and accurate high-efficiency grasping strategy that can be widely applied in real-world scenarios.

[0044] 2. When the training set input to the neural network is large enough and the number of training iterations is large enough, the recognition algorithm of the deep learning network can achieve a high accuracy rate.

[0045] 3. When the target has a special shape and the workpiece is placed in a disorderly manner with obstructions, the present invention adopts the method of obtaining 6D pose estimation by ICP point cloud registration, which can output the point information of the target that is easy for the robotic arm to grasp.

[0046] 4. In terms of grasping planning, the unique grasping strategy of this invention can achieve fast and accurate grasping of targets. This invention solves the current problem of how to extract effective features and achieve disordered grasping in occluded and cluttered scenes. Attached Figure Description

[0047] Figure 1 This is a flowchart of the algorithm of the present invention;

[0048] Figure 2The color image shows the T-shaped pipe and building blocks arranged haphazardly and partially obscured in an embodiment of the present invention.

[0049] Figure 3 As described in the embodiments of the present invention Figure 2 The result of instance segmentation;

[0050] Figure 4 The image shows a complex background with obscured T-pipes and building blocks arranged haphazardly in a color illustration of an embodiment of the present invention.

[0051] Figure 5 This is an embodiment of the present invention. Figure 4 The result image after segmentation. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0053] To achieve disordered grasping by robots in cluttered scenes, this invention provides a robot vision detection and grasping algorithm that can solve the problem of occlusion of target workpieces. The grasping algorithm flowchart is as follows. Figure 1 As shown, the overall process is divided into two parts: training and inference, specifically including the following steps:

[0054] ① Training Section

[0055] Step (1) Data Acquisition:

[0056] For each target workpiece: use a depth camera to take pictures of the target workpiece to obtain color images and depth maps of the target workpiece at different positions and orientations in the target scene;

[0057] Step (2) Dataset creation:

[0058] Each acquired color image of a target workpiece is annotated to indicate its contour information and workpiece category, generating a mask image. The pixel positions occupied by the target workpiece in the color image can be obtained using the mask image, thus generating a dataset.

[0059] Step (3) Dataset augmentation:

[0060] The obtained dataset was processed sequentially using methods such as rotation, noise reduction, and simulated lighting to expand the dataset;

[0061] Step (4) Instance segmentation algorithm training:

[0062] The enhanced dataset is input into the Mask R-CNN deep network for training, and a Mask R-CNN deep learning model H with better detection performance is generated by parameter tuning;

[0063] Step (5) Create a workpiece point cloud template library using the target workpiece CAD:

[0064] Create a CAD template library for the target workpiece using 3D drawing software or 3D reconstruction.

[0065] ②Detection section

[0066] Step (1) Prepare the data to be tested:

[0067] Prepare data (color image) of the target workpiece to be inspected in the target scene;

[0068] Step (2) Instance segmentation algorithm reasoning:

[0069] The data to be detected is input and trained to obtain the algorithm model H, which can identify the position and category of each target workpiece; based on the position of the target workpiece's color image and mapped onto the depth map, the surface point cloud O of each target workpiece can be obtained using camera intrinsic parameters. i (i represents the i-th target workpiece);

[0070] Step (3) Point cloud registration:

[0071] The purpose of point cloud data registration is to obtain the pose and position information of the grasped object in space, i.e., 6D pose. This invention uses the Iterative Closest Point (ICP) method for registration. First, based on the category of the target workpiece, a complete point cloud template of the target workpiece is selected from the CAD template library, allowing the point cloud O... i By registering with the point cloud template, the current pose S of each detection target workpiece can be obtained sequentially. i (i represents the i-th target workpiece).

[0072] Step (4) Unordered fetching strategy

[0073] 1) Calculate the pose S of each template workpiece at this position. i At that time, the surface point cloud and its surface area under the camera (the object point cloud when it is not occluded): combined with the point cloud CAD template library, camera extrinsic parameters, and the current pose S of the actual workpiece. i The point cloud T of the workpiece under unobstructed conditions from the camera's perspective was obtained using simulation rendering. i (Workpiece in position S) i (Unobstructed point cloud under certain conditions), calculate the single-view point cloud T for each target workpiece under unobstructed conditions in turn. i Surface area SA Ti (i represents the i-th target workpiece); calculate the point cloud O of each target workpiece in the actual scene in sequence. i Surface SA Oi (i represents the i-th target workpiece);

[0074] 2) The point cloud O of each target workpiece in the actual calculation scenario will be calculated.i Surface area SA Oi And its unobstructed single-view point cloud T i Surface area SA Ti One-to-one correspondence, and calculate the unobstructed rate (SA) of the target workpiece in sequence. IOUi (i represents the i-th target workpiece) as shown in equation (4-1):

[0075]

[0076] 3) Compare the uncoverage rate (SA) for all target workpieces. IOUi Set a threshold σ (a metric indicating whether occlusion affects crawling), SA IOUi If the value is greater than σ, the grasping requirement is met. σ ranges from 0 to 1. The closer it is to 1, the smaller the occluded surface is. Conversely, if it is less than 1, it is considered occluded. The value of σ is set according to the actual situation.

[0077] 4) In all cases where SA is satisfied IOUi Among target workpieces with a value greater than σ, compare the centroids of each target workpiece and prioritize grabbing the target workpiece with the higher centroid position.

[0078] Example:

[0079] Take, for example, a tee pipe and colored wooden blocks arranged haphazardly in a target scene:

[0080] Step 1: Use a depth camera to take pictures, obtaining color images and depth maps of the T-shaped pipe and the colored wooden block at different positions and poses in the target scene; for example, color images of the T-shaped pipe and the colored wooden block at different positions and poses are shown below. Figure 2 and Figure 4 As shown;

[0081] Step 2: Label the obtained color images of the T-pipe and the wooden block to obtain the outline information and type of the T-pipe and the colored wooden block. Based on the mask image generated after labeling, the pixel positions occupied by the T-pipe and the colored wooden block in the color image can be obtained, and a dataset can be generated.

[0082] Step 3: Expand the dataset of the prepared T-pipe and colored wooden blocks by rotating, adding noise, and simulating lighting in sequence.

[0083] Step 4: Input the expanded dataset into the Mask R-CNN deep network for training, and generate an algorithm model with better detection performance by adjusting the parameters;

[0084] Step 5: Based on the detected T-pipes and colored wooden blocks and their category information, use 3D drawing software or 3D reconstruction to draw the point cloud CAD template library of the unobstructed T-pipes and colored wooden blocks;

[0085] Start testing:

[0086] Step 1): Input the data of the T-shaped pipe and the colored wooden block to be detected into the trained model. The model can then identify the contour information and category of the T-shaped pipe and the colored wooden block, such as... Figure 3 , Figure 5 ;

[0087] Step 2): Map the positions of the color images of the T-pipes and colored wooden blocks onto the depth map. The point cloud of each T-pipe and colored wooden block can be obtained using the camera intrinsic parameters.

[0088] Step 3): Select the complete point cloud template of the T-pipe and colored wooden block from the CAD template library according to their categories, and register the point cloud of the T-pipe and colored wooden block with the point cloud template to obtain the 6D pose of the T-pipe and colored wooden block in the current scene.

[0089] Step 4): Based on the pre-defined point cloud CAD template library of T-pipes and colored wooden blocks, combined with the camera extrinsic parameters and the 6D pose of the T-pipes and colored wooden blocks in the current scene, calculate the point cloud surface area of ​​each T-pipe and colored wooden block in the current pose.

[0090] Step 5): Using the disordered grasping strategy of the present invention, the point cloud surface area of ​​each T-pipe and colored wooden block in the current pose is calculated sequentially under the camera. The ratio of the surface area of ​​the T-pipe and colored wooden block to the surface area of ​​the corresponding unoccluded point cloud template in the point cloud CAD template library (i.e., the non-occlusion rate, obtained by formula (4-1)) is calculated. T-pipes and colored wooden blocks with non-occlusion rates greater than the threshold in the current scene are found. T-pipes and colored wooden blocks with higher centroids among the unoccluded T-pipes and colored wooden blocks are grasped first.

[0091] In summary, for situations where T-pipes and colored wooden blocks are scattered or tightly stacked with occlusion, deep learning networks are used for object recognition; for situations where T-pipes and colored wooden blocks are tightly connected or the T-pipes are hollow, 6D pose estimation needs to be performed based on object recognition, and then an unordered grasping strategy should be adopted for grasping.

[0092] This visual grasping algorithm can be applied to the following scenarios:

[0093] Application Scenario 1: such as Figure 2 As shown, the instances of the randomly stacked and obstructed T-pipes and building blocks are segmented as follows. Figure 3 As shown;

[0094] Application Scenario 2: such as Figure 2 As shown, in a complex background, there are randomly stacked and obstructed T-pipes and building blocks. The instance segmentation result is as follows. Figure 5 As shown.

[0095] In practical applications, due to the unique shapes, disordered placement, and occlusion of certain targets, a robot vision detection algorithm capable of accurate identification and orderly grasping is needed. Previous vision grasping algorithms primarily detect 2D or 3D features, resulting in limited effective feature extraction and often significant errors. This invention addresses this issue by employing a detection method incorporating deep learning-based 6D pose estimation, thereby improving the extraction of effective features.

Claims

1. A robot grasping and detection method for occluded and cluttered scenes, characterized in that, Includes the following steps: 1) For each target workpiece: Take pictures of the target workpiece using a depth camera, and collect color images of the target workpiece in different poses in the target scene, as well as the depth images corresponding to the color images; 2) Train the color image to obtain the detection training model H; and create a point cloud template library for the target workpiece; 2-1) Annotate the color image of each target workpiece, marking the outline information and workpiece category of the target workpiece, and generate a mask image; 2-2) Send the workpiece category from the mask image to the workpiece point cloud template library; 2-3) Based on the contour information in the mask image, obtain the pixel position occupied by the target workpiece in each color image and generate a dataset. Perform dataset augmentation processing on the dataset containing the pixel position occupied by the target workpiece in the color image to expand the dataset. 2-4) Input the expanded dataset into the deep neural network for training to obtain the detection training model H; 2-5) Based on the 3D structure of the workpiece in the depth map of each target workpiece, create a workpiece point cloud template library using 3D drawing software; 3) Acquire color images of each target workpiece in the target scene and the corresponding depth images using a depth camera, and input the color images of the target workpieces into the detection training model H to obtain the position and category of each target workpiece in the color images; 4) Based on the location of the target workpiece in the color image, map it onto the depth map, and use the camera intrinsic parameters to obtain the surface point cloud O of each target workpiece. i Where i represents i target workpieces; 5) Collect the surface point cloud O of each target workpiece i Point cloud registration is performed with the workpiece point cloud template library to obtain the current pose S of each detection target workpiece. i ; 6) Based on the workpiece point cloud template library, combined with the extrinsic parameters of the depth camera and the current pose S of each detected target workpiece, i The target workpiece is grasped by adopting an unordered grasping strategy; The method of using an unordered grasping strategy to grasp the target workpiece includes the following steps: 6-1) Calculate the pose S of each target workpiece at this position. i At that time, the surface point cloud under the camera and the object point cloud when there is no occlusion: combining the workpiece point cloud template library, camera extrinsic parameters and the current pose S of the actual workpiece. i Through camera simulation rendering, the position S of the target workpiece under this camera's viewpoint is obtained. i Unoccluded single-view point cloud T i ; 6-2) Sequentially acquire the single-view point cloud T of each target workpiece under unobstructed conditions. i Surface area SA Ti Simultaneously, the point cloud O of each target workpiece in the actual scene is acquired sequentially. i Surface SA Oi Where i represents the i-th target workpiece; 2) Obtain the point cloud O of each target workpiece in the actual scene. i Surface area SA Oi And its unobstructed single-view point cloud T i Surface area SA Ti One-to-one correspondence, obtaining the non-occlusion rate of the target workpiece, that is: Where i represents the i-th target workpiece, SA IOUi It is the ratio of the point cloud surface area of ​​the workpiece in the actual scene to the complete point cloud surface area of ​​the workpiece in the template library, that is, the non-occlusion rate of the target workpiece. 3) Compare the uncoverage rate (SA) for all target workpieces. IOUi Set a threshold σ to determine whether occlusion affects the captured metrics, where the value of σ is set according to the actual situation; 4)SA IOUi If the value is greater than σ, the grasping requirement is met. σ ranges from 0 to 1. The closer it is to 1, the smaller the occluded surface is, and vice versa. 5) In all cases where SA is satisfied IOUi Among the target workpieces with a value greater than σ, compare the centroids of each target workpiece and prioritize grabbing the target workpiece with the highest centroid position.

2. The robot grasping and detection method for occluded and cluttered scenes according to claim 1, characterized in that, In steps 2-3), the dataset augmentation process includes: The obtained dataset was then processed and expanded using rotation, noise addition, and simulated lighting methods in sequence. The rotation refers to rotating the color image by a set angle; The simulated lighting refers to changing the contrast and saturation of the color image.

3. The robot grasping and detection method for occluded and cluttered scenes according to claim 1, characterized in that, The detection training model H is the Mask R-CNN deep learning model.

4. The robot grasping and detection method for occluded and cluttered scenes according to claim 1, characterized in that, Step 5) specifically involves: The surface point cloud O of each target workpiece was acquired using the iterative nearest point method. i Perform registration; based on the category of the target workpiece, select the complete point cloud template of the target workpiece from the workpiece point cloud template library, and make the point cloud O i Registration is performed with the workpiece point cloud template library to sequentially obtain the current pose S of each detection target workpiece. i .

5. The detection system for robot grasping and detection in occluded and cluttered scenes according to claim 1, characterized in that, include: The image acquisition module is used to acquire color images of the target workpiece in different poses of the target scene obtained by the depth camera scanning each target workpiece, as well as the depth images of the target workpiece to be detected containing each target workpiece and the depth images of the target workpieces to be detected, as well as the depth images of the target workpieces to be detected, as well as the depth images of the target workpieces to be detected, and send them to the model building module. The image processing module is used to annotate the acquired color image of the target workpiece, annotate the outline information and workpiece category of the target workpiece, and generate a mask image. The model training module is used to obtain the pixel positions occupied by the target workpiece in the color image based on the contour information in the mask image, and generate a dataset. The dataset containing the pixel positions occupied by the target workpiece is augmented to expand the dataset. The expanded dataset is then input into the deep neural network for training to obtain the detection training model H. The workpiece point cloud template library is used to create a workpiece template for each target workpiece based on the workpiece category in the mask image and the three-dimensional structure of the depth map of each target workpiece. The point cloud registration module is used to select the point cloud template of the target workpiece from the workpiece point cloud template library according to the category of the target workpiece, and register it with the workpiece point cloud template library by means of the iterative nearest point method, so as to obtain the current pose S of each target workpiece to be detected. i ; The target workpiece grasping module is used to grasp the workpiece point cloud template library in combination with the extrinsic parameters of the depth camera and the current pose S of each detected target workpiece. i The target workpiece is grasped by adopting an unordered grasping strategy.

Images