Stacked workpiece disorderly grabbing method and system based on 3D vision system

Abstract

The application discloses a kind of based on 3D vision system's stacked workpiece disorderly grabbing method and system, it is related to computer vision technical field, method includes: the RGB image and depth image of stacked workpiece are acquired by depth camera, and point cloud data is generated;RGB and depth image are carried out instance segmentation using the improved Mask R-CNN model of integrated MSDA module, and workpiece pose estimation is realized by combining ICP algorithm and template point cloud registration;The coordinate relationship of camera and mechanical arm is established by hand-eye calibration, and the 6D pose of workpiece under the base coordinate system of mechanical arm is acquired;According to point cloud centroid distance and point number, calculate the priority of grabbing, select optimal target and carry out grabbing, and complete the disorderly grabbing of all workpieces in cycle.The application realizes accurate instance segmentation and pose estimation to stacked workpiece by improved Mask R-CNN model of integrated MSDA module and ICP algorithm combined with 3D vision system, to accurately complete disorderly grabbing task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a stacked workpiece disordered grabbing method and system based on a 3D vision system. BACKGROUND

[0002] In the field of robot operation, three-dimensional object picking in a scattered scene is a classic problem of great significance. By using a mechanical arm to realize the disordered grabbing of scattered stacked workpieces, the automation of the robot can be realized. This ability enables the robot to accurately perceive, locate and grab target objects in a complex and variable environment, thereby improving production efficiency and reducing labor costs. This has important significance for the development of industrial and service robots.

[0003] With the popularization of robot technology, three-dimensional object grabbing in a scattered scene has become a key problem in industrial automation. In existing pose estimation methods, the 2D feature point method relies on object texture features, and there is a bottleneck in feature extraction in low-texture workpieces. The method based on template matching realizes pose estimation through feature comparison, such as the LINEMOD algorithm combined with RGB-D information and ICP correction, but it is sensitive to occlusion; RANSAC and its improved algorithm SAC-IA improve accuracy but sacrifice efficiency, and are mainly used in the initial registration stage. In recent years, deep learning technology has gradually replaced traditional handcrafted features, and PointNet and its derivative networks realize point cloud feature extraction through end-to-end learning, but two-dimensional mapping methods are prone to loss of geometric information, and VoxelNet and other improved schemes still face challenges in three-dimensional feature representation optimization. For the task of 3D object detection and pose estimation, a structure similar to PointNet is designed to realize the 3D target detection task.

[0004] Currently, the methods for workpiece pose estimation mainly include 2D feature point method, template matching method, and deep learning-based method. Among them, the 2D feature point method is a common method that extracts multiple feature points of the workpiece and uses the corresponding relationship between these feature points and the reference coordinate system to infer the pose of the workpiece. This method performs well on objects with rich textures and can achieve high-precision positioning. The template matching method finds the template pose with a matching error less than a threshold as the initial pose of the scene target by generating and comparing object features and templates. In recent years, with the emergence of deep convolutional neural networks, traditional hand-designed point cloud feature descriptors have been gradually replaced by learning-based descriptors. Deep convolutional neural networks have strong feature extraction capabilities and can automatically learn effective feature representations suitable for point cloud data. Compared with hand-designed feature descriptors, learning-based descriptors can better capture important features of point cloud data and achieve better performance in point cloud processing and analysis tasks. However, there are still some challenges and limitations.

[0005] For example, in the process of pose estimation, there are several challenges and limitations. First, feature point extraction becomes particularly difficult for objects with single texture, which leads to low-texture workpieces being unable to be accurately processed. Therefore, when processing such workpieces, other more suitable methods need to be explored to achieve accurate positioning. Second, the integrity of point cloud data is very sensitive to the algorithm, especially in a production environment, the stacking of workpieces can make it complex and difficult to obtain the complete point cloud data of a single workpiece. In addition, the accuracy of pose estimation may be affected in the presence of complex occlusions. Furthermore, some pose estimation algorithms have high requirements for computing resources, which means they may require a large amount of computing resources to complete the pose estimation task, making them impractical in actual applications. Finally, when processing stacked workpieces, the problem of occlusion of upper workpieces to lower workpieces must be considered, and the optimal workpiece grasping sequence must be determined by the algorithm. SUMMARY

[0006] To solve the above problems, the present application proposes a stacked workpiece disordered grasping method and system based on a 3D vision system, which performs instance segmentation on RGB images and depth images by integrating an improved Mask R-CNN model with an MSDA module, registers point cloud data using an ICP algorithm, and realizes coordinate conversion using a hand-eye calibration technique. Based on the distance and number of point cloud centroids, the grasping priority is determined, thereby achieving accurate 6D pose estimation and efficient disordered grasping of stacked workpieces.

[0007] The specific scheme is as follows:

[0008] On the one hand, the stacked workpiece disordered grasping method based on the 3D vision system comprises:

[0009] S1, capturing the stacked workpieces by a depth camera and extracting the RGB image and the depth image, generating the stacked workpiece point cloud data based on the RGB image and the depth image, and performing instance segmentation on the RGB image and the depth image by an improved Mask R-CNN model integrated with an MSDA module to obtain segmented pixels;

[0010] S2, pose estimation of the stacked workpieces, using a two-stage algorithm, the first stage maps the segmented pixels to the stacked workpiece point cloud data to extract local point clouds of each target, and the second stage registers each target local point cloud with a preset template point cloud through an ICP algorithm to obtain a rigid transformation matrix of the target local point cloud relative to the template;

[0011] S3, using a hand-eye calibration method, calibrating the camera coordinates and the base coordinates of the robot arm to obtain a hand-eye calibration matrix, combining the rigid transformation matrix and the hand-eye calibration matrix to obtain 6D pose information of the stacked workpieces in the base coordinate system of the robot arm;

[0012] S4, calculates the grasping priority score based on the distance between the center of mass of the stacked workpiece point cloud data and the camera and the number of points in the stacked workpiece point cloud data, and selects the workpiece with the highest grasping priority score as the optimal target;

[0013] S5, based on the 6D pose information, guides the robotic arm to grasp the optimal target. After the grasping is completed, the next workpiece to be grasped is selected in a loop until all the workpieces to be grasped have completed the grasping task.

[0014] Furthermore, the improved Mask R-CNN model integrating the MSDA module adopts ResNet-101 as the feature extraction backbone network of the Mask R-CNN model, and connects the MSDA module to the feature map output by ResNet-101 to achieve attention enhancement on the output feature map.

[0015] Furthermore, the MSDA module includes a multi-scale convolution branch unit, a feature fusion unit, a hole convolution path unit, an attention map generation unit and an attention weighted output unit:

[0016] Multi-scale convolutional branch units are used to extract feature maps output by the backbone network Perform 1×1, 3×3, and 5×5 convolution operations to obtain multi-scale context feature maps. The calculation formula is as follows:

[0017] ;

[0018] ;

[0019] ;

[0020] in, Represents a 1×1 convolution scale feature map; Represents a 3×3 convolution scale feature map; Represents a 5×5 convolutional scale feature map; Represents a 1×1 convolution operation; Represents a 3×3 convolution operation; Represents a 5×5 convolution operation;

[0021] The feature fusion unit is used to splice the feature maps of the three scales and then perform a layer of convolution compression to generate a fused intermediate feature map. The calculation formula is as follows:

[0022] ;

[0023] in, Represented as a fused intermediate feature map; ) represents feature splicing;

[0024] The hollow convolution path unit is used for respectively passing the fusion intermediate feature map through a plurality of hollow convolution operations, and the hollow rates are respectively set to 1, 2 and 3, so as to obtain three hollow convolution results, and the formula is as follows:

[0025] ;

[0026] ;

[0027] ;

[0028] wherein, represents the first hollow convolution result; represents the second hollow convolution result; represents the third hollow convolution result; represents the hollow convolution with a convolution kernel of 3x3 and an expansion rate of 1; represents the hollow convolution with a convolution kernel of 3x3 and an expansion rate of 2; represents the hollow convolution with a convolution kernel of 3x3 and an expansion rate of 3;

[0029] The attention map generation unit is used for element-wise addition of the hollow convolution results, and after ReLU activation, 1x1 convolution and Sigmoid function are connected to output a spatial attention map, and the calculation formula is as follows:

[0030] ;

[0031] wherein, represents the spatial attention map;

[0032] The attention weighting output unit is used for applying the spatial attention map to the fusion intermediate feature map, and performing element-wise multiplication of each channel with the spatial weight, so as to obtain an enhanced feature map, and the formula is as follows:

[0033] ;

[0034] wherein, represents element-wise multiplication; represents the enhanced feature map.

[0035] Further, the first stage maps the segmented pixels to the stacked workpiece point cloud data to extract local point clouds of each target, specifically including:

[0036] The segmented pixel position (u, v) and the depth value Z(u, v) of the corresponding depth map are back-projected into a 3D coordinate point to obtain a local point cloud of each target , and the calculation formula is as follows:

[0037] ;

[0038] wherein K is an intrinsic matrix of the camera; denotes an abscissa; denotes an ordinate; denotes a fixed constant.

[0039] Further, the second stage registers the target local point cloud with the preset template point cloud through an ICP algorithm to obtain a rigid transformation matrix of the target relative to the template, and the specific process is as follows:

[0040] By solving the optimization task, a rigid transformation matrix is obtained so that the template point cloud is registered with the current target local point cloud after rotation and translation, and the objective function formula is as follows:

[0041] ;

[0042] wherein denotes a point in the template point cloud; denotes a point in the target local point cloud; denotes a rotation matrix; denotes a translation vector;

[0043] The process for solving the optimization task is as follows:

[0044] First, the centroid is removed, the point cloud is centralized, the center of gravity of the point cloud is 0, the points of the new point cloud are obtained, and then the points are brought into the objective function again, and the two parts are obtained after reorganizing:

[0045] ;

[0046] wherein the first term of the added term is a rotation part optimization objective, and the second term is a translation part optimization objective, , denote the centroid vectors of the template point cloud and the target local point cloud respectively;

[0047] Second, R of the rotation part is solved, and a covariance matrix H between the centralized template point cloud and the target point cloud is constructed:

[0048] ;

[0049] and singular value decomposition is performed thereon, specifically:

[0050] ;

[0051] wherein U, and V respectively denote the left singular matrix, the singular value diagonal matrix and the right singular matrix of the matrix H;

[0052] According to the singular value decomposition result, the optimal solution of the rotation matrix R is calculated as follows:

[0053] ;

[0054] In the third step, the translation vector t is calculated as follows:

[0055] ;

[0056] Finally, the rigid transformation is applied to the template point cloud, and the matching update is repeated until convergence, obtaining the rigid transformation matrix of the target relative to the template.

[0057] Further, the calculation formula of the grasping priority score is:

[0058] ;

[0059] wherein, represents the grasping priority score; is a weight coefficient of the point cloud centroid position; is a weight coefficient of the point cloud quantity; represents the distance from the point cloud centroid to the depth camera; represents the point cloud quantity; is a standard constant.

[0060] On the other hand, the stacking workpiece disordered grasping system based on the 3D vision system comprises:

[0061] The segmentation module captures the stacking workpiece by the depth camera and extracts the RGB image and the depth image, generates the stacking workpiece point cloud data based on the RGB image and the depth image, performs instance segmentation on the RGB image and the depth image by the improved Mask R-CNN model integrated with the MSDA module, and obtains the segmented pixels.

[0062] The rigid transformation matrix acquisition module estimates the pose of the stacking workpiece, adopts a two-stage algorithm, maps the segmented pixels to the stacking workpiece point cloud data in the first stage to extract local point clouds of each target, and performs registration on the local point clouds of each target and the preset template point cloud in the second stage by the ICP algorithm to obtain the rigid transformation matrix of the local point clouds of each target relative to the template.

[0063] The 6D pose information acquisition module adopts a hand-in-eye calibration method to calibrate the robot base and the camera coordinates to obtain a hand-eye calibration matrix, combines the rigid transformation matrix and the hand-eye calibration matrix, and obtains the 6D pose information of the stacking workpiece in the robot base coordinate system.

[0064] An optimal target confirmation module calculates a grabbing priority score based on the distance between the centroid of the stacked workpiece point cloud data and the camera and the number of stacked workpiece point cloud points, and regards the workpiece with the highest grabbing priority score as the optimal target;

[0065] A grabbing module guides the mechanical arm to grab the optimal target based on the 6D pose information, and after the grabbing is completed, the next workpiece to be grabbed is selected in a loop until all the workpieces to be grabbed complete the grabbing task.

[0066] The present application adopts the above technical solution and has beneficial effects:

[0067] (1) The improved Mask R-CNN model integrated with the MSDA module is used to perform instance segmentation on the RGB image and the depth image in the present application, so that each object in the stacked workpiece can be more accurately recognized and separated.

[0068] (2) The present application uses the ICP algorithm to accurately register the target local point cloud with the preset template point cloud to obtain a rigid transformation matrix, thereby improving the reliability of the workpiece pose estimation in the presence of occlusion, and enabling the mechanical arm to more accurately position and grab the target workpiece.

[0069] (3) The present application adopts the hand-out-of-eye calibration method to realize accurate conversion between the mechanical arm base coordinate system and the camera coordinate system, so that the 6D pose information obtained from the point cloud data can be directly applied to the mechanical arm control, and the grabbing path planning is optimized. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The flowchart of the stacked workpiece disordered grabbing method based on the 3D vision system of the embodiment of the present application;

[0071] Figure 2 The ResNet-101 feature extraction network diagram of the embodiment of the present application;

[0072] Figure 3 The improved feature extraction module diagram of the embodiment of the present application;

[0073] Figure 4 The workpiece stacking data set diagram of the embodiment of the present application;

[0074] Figure 5 The workpiece CAD model and point cloud diagram of the embodiment of the present application;

[0075] Figure 6 The hand-out-of-eye calibration diagram of the embodiment of the present application;

[0076] Figure 7 The stacked workpiece disordered grabbing system diagram based on the 3D vision system of the embodiment of the present application. DETAILED DESCRIPTION

[0077] The application will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the application are not limited thereto. As shown in the drawings, the application is based on a 3D vision system-based stacked workpiece disordered grabbing method, which comprises: Figure 1

[0078] S1, taking a stacked workpiece by a depth camera and extracting an RGB image and a depth image, generating stacked workpiece point cloud data based on the RGB image and the depth image, performing instance segmentation on the RGB image and the depth image by an improved Mask R-CNN model integrated with an MSDA module (Multi-Scale Dilated Attention), and obtaining segmented pixels.

[0079] Specifically, the improved Mask R-CNN model integrated with the MSDA module adopts ResNet-101 as the feature extraction backbone network of the Mask R-CNN model, and performs attention enhancement on the output feature map by connecting the MSDA module in the ResNet-101 backbone network.

[0080] Specifically, the MSDA module comprises a multi-scale convolution branch unit, a feature fusion unit, a dilated convolution path unit, an attention map generation unit and an attention weighted output unit.

[0081] The multi-scale convolution branch unit is used for performing 1×1, 3×3 and 5×5 convolution operations on the feature map output by the feature extraction backbone network to obtain multi-scale context feature maps, and the calculation formula is as follows:

[0082] ;

[0083] ;

[0084] ;

[0085] wherein, represents a 1×1 convolution scale feature map; represents a 3×3 convolution scale feature map; represents a 5×5 convolution scale feature map; represents a 1×1 convolution operation; represents a 3×3 convolution operation; represents a 5×5 convolution operation;

[0086] The feature fusion unit is used for splicing the feature maps of the three scales and then compressing them through a layer of convolution to generate a fusion intermediate feature map, and the calculation formula is as follows:

[0087] ;​

[0088] wherein, is represented into a fusion intermediate feature map; ) represents feature concatenation;

[0089] The hollow convolution path unit is used for respectively passing the fusion intermediate feature map through a plurality of hollow convolution operations, and the hollow rates are respectively set to 1, 2 and 3 to obtain three hollow convolution results, and the formula is as follows:

[0090] ;

[0091] ;

[0092] ;

[0093] wherein, is the first hollow convolution result; is the second hollow convolution result; is the third hollow convolution result; represents a hollow convolution with a convolution kernel of 3x3 and an expansion rate of 1; represents a hollow convolution with a convolution kernel of 3x3 and an expansion rate of 2; represents a hollow convolution with a convolution kernel of 3x3 and an expansion rate of 3;

[0094] The attention map generation unit is used for element-wise addition of the hollow convolution results, and after ReLU activation, a 1x1 convolution and a Sigmoid function are connected to output a spatial attention map, and the calculation formula is as follows:

[0095] ;

[0096] wherein, is the spatial attention map;

[0097] The attention weighted output unit is used for applying the spatial attention map to the fusion intermediate feature map, and each channel is multiplied by the spatial weight element by element to obtain the final output feature map, and the formula is as follows:

[0098] ;

[0099] wherein, is element-wise multiplication; is the final output feature map.

[0100] S2, pose estimation is performed on the stacked workpiece, a two-stage algorithm is adopted, in the first stage, the segmented pixels are mapped to the stacked workpiece point cloud data to extract local point clouds of each target, in the second stage, the ICP algorithm is used to register the local point clouds of the target with the preset template point clouds, and a rigid transformation matrix of the local point clouds of the target relative to the template is obtained.

[0101] Specifically, the first stage maps the segmented pixels to the stacked workpiece point cloud data to extract local point clouds of each target, specifically including:

[0102] The position (u, v) of the segmented pixels and the depth value Z(u, v) of the corresponding depth map are back-projected into 3D coordinate points to obtain local point clouds of each target , the calculation formula is as follows:

[0103] ;

[0104] Wherein, K is the camera intrinsic matrix; represents the horizontal coordinate; represents the vertical coordinate; represents a fixed constant.

[0105] Specifically, in the second stage, the ICP algorithm is used to register the local point clouds of the target with the preset template point clouds, and a rigid transformation matrix of the target relative to the template is obtained, and the specific process is as follows:

[0106] By solving the optimization task, a rigid transformation matrix is obtained, so that the template point cloud is registered with the current local point cloud of the target after rotation and translation, and the objective function formula is as follows:

[0107] ;

[0108] Wherein, represents a point in the template point cloud; represents a point in the local point cloud of the target; represents a rotation matrix; represents a translation vector;

[0109] The process of solving the optimization task is as follows:

[0110] First, the centroid is removed, the point cloud is centralized, the center of gravity of the point cloud is 0, the points of the new point cloud are obtained, and then the target function is re-inputted, and the two parts are obtained after re-arrangement:

[0111] ;

[0112] Wherein, the first term of the added term is the rotation part optimization target, and the second term is the translation part optimization target, , respectively represent the centroid vectors of the template point cloud and the target local point cloud.

[0113] Secondly, solve R of the rotation part, and construct the construction covariance matrix H between the decentralized template point cloud and the target point cloud:

[0114] ;

[0115] And singular value decomposition is performed, specifically:

[0116] ;

[0117] Wherein, U, , V are the left singular matrix, singular value diagonal matrix and right singular matrix of matrix H respectively.

[0118] According to the singular value decomposition result, the optimal solution of the rotation matrix R is:

[0119] ;

[0120] Thirdly, the translation vector t is calculated, and the formula is as follows:

[0121] ;

[0122] Finally, the rigid transformation is applied to the template point cloud, and the matching is repeatedly updated until convergence, so as to obtain the rigid transformation matrix of the target relative to the template.

[0123] Specifically, in order to solve the problems of too many point clouds, slow calculation speed, waste of resources and the like that may occur when the method based on template matching is used to process multi-target scenes, a two-stage algorithm is proposed in the embodiment:

[0124] In the first stage, the local point cloud region containing a single target is extracted from the image; in the second stage, the target pose is estimated in the local point cloud region. In this way, the problem is transformed into the case of single target pose estimation, which is easier to handle and solve. The two-stage method can effectively cope with the challenges brought by multi-target scenes and improve the accuracy and robustness of pose estimation.

[0125] In addition, in order to solve the problem that the effect of the traditional visual method is not ideal when sorting the workpieces stacked and placed in disorder, the Mask R-CNN model is improved by introducing an attention mechanism, and a ResNet-101 network is selected as the feature extraction network of the Mask R-CNN model, as shown in Figure 2The illustrated. In the feature extraction stage, first, the input picture is subjected to layer-by-layer convolution operation. Through this process, different levels of feature maps are gradually extracted from the original image. We denote these feature maps as 、 、 、 、 In the feature fusion stage, first, the is subjected to 1x convolution, and the output is obtained; second, the is subjected to 2x2 up-sampling, and the corresponding elements are summed with the 1x1 convolution to obtain , and so on, respectively, to obtain and ; then, the , , , are subjected to 3x3 convolution to eliminate the aliasing effect brought by the up-sampling operation; finally, the after 3x3 convolution is subjected to pooling to obtain , and the final output , , , , is taken as the input feature of the region proposal network (RPN) network for subsequent training, as shown in Figure 3 .

[0126] In the above process, the calculation formula (not including 3x3 convolution) of , , , is as follows:

[0127] ;

[0128] ;

[0129] In the formula, ResNet-101 network denotes the feature map extracted in the first stage; denotes the feature map extracted in the second stage; denotes the feature map extracted in the third stage; denotes the feature map output after feature fusion; denotes the 1x1 convolution layer; denotes the corresponding element addition operation; denotes the 2x2 up-sampling.

[0130] The CBAM attention mechanism is used to modify the feature fusion stage of the feature extraction network of the Mask R-CNN model, and the CBAM attention module is added after the 1*1 convolution layer of C2, C3, C4 and C5, so as to improve the channel and spatial focusing ability of the Mask R-CNN model on the stacked bow and arrow information, as shown in FIG. 5. Since the CBAM attention module is directly embedded into the Mask R-CNN model and participates in the network parameter iterative update of the whole model, with the continuous training of the model, the CBAM attention weight matrix will also be continuously optimized and gradually converged to obtain the optimal weight matrix.

[0131] The improved P5, P4, P3 and P2 calculation formulas are:

[0132] ;

[0133] ;

[0134] In the formula, represents the CBAM attention weight matrix corresponding to the i-th stage feature map; represents the corresponding element multiplication operation.

[0135] Since there is no suitable public workpiece stacking dataset available on the network, the dataset used by the present application is artificially collected, and a single or multiple workpiece stacks are selected on the workbench, and 600 workpiece stacking images in different poses are collected by a camera. Then, the labelme tool is used to manually annotate the stacked workpiece dataset. The annotated workpiece stacking dataset is small in size, and the network training performed only on this dataset is difficult to achieve full convergence of the network, and cannot achieve a satisfactory accuracy in image instance segmentation. Therefore, before performing stacked workpiece segmentation, the improved Mask R-CNN network is first pre-trained on the COCO dataset. COCO is a large target recognition dataset containing more than 100,000 images of 80 categories, and is a commonly used dataset in the target recognition field. After obtaining the pre-training weights, the workpiece stacking dataset made by oneself is used for training, which can greatly speed up the network convergence speed.

[0136] Specifically, Figure 4 The workpiece stacking dataset shown in the figure includes original images, corresponding mask files and annotation information for image segmentation tasks; after mapping the image segmentation result to the three-dimensional space to obtain the point cloud of the target workpiece, registration with the template point cloud is required to restore the spatial transformation matrix between the two point clouds, and then the target pose is obtained. In this embodiment, UG is used to model the grasped workpiece, and Cloudcompare is used to convert the modeled model into a pcd format point cloud as a template point cloud, and the modeling effect and the template point cloud are as shown in the figure. Figure 5 ​

[0137] S3, a hand-eye calibration matrix is obtained by calibrating the camera coordinate and the robot base coordinate in the eye-out-hand calibration mode, and a 6D pose information of the stacked workpiece in the robot base coordinate system is obtained by combining the rigid transformation matrix and the hand-eye calibration matrix.

[0138] Specifically, Figure 6 The eye-out-hand calibration mode adopted by the application fixes the robot base on the workbench, and the camera position is also fixed. The transformation matrix of the camera relative to the robot base coordinate is obtained by calibration. The camera coordinate system is , the calibration board coordinate system is , the robot end coordinate system is , the robot base coordinate system is , and the conversion of several coordinate systems is involved, that is, the conversion relationship of the calibration board coordinate system to the camera coordinate system , the conversion relationship of the camera coordinate system to the robot base coordinate system is X, and the conversion relationship of the robot base to the robot end coordinate system is , wherein the conversion relationship X of the camera coordinate system to the robot base coordinate system is the hand-eye calibration matrix to be solved. In the above case, the calibration board is fixed at the end of the robot arm, and in a certain pose, the points on the calibration board have coordinate values in the calibration board coordinate system , after the coordinate system conversion relationship of , , , the points on the calibration board can be converted to the coordinate values in the robot end coordinate system , and the conversion relationship is as follows: ; then the robot changes the pose, and another set of formulas in the same form as the above can be obtained, that is: ; the , in the above formula can be obtained by the pose output of the robot, , can be obtained by the camera calibration external parameter, which can be converted into the following form: further converted to: ; it can be understood as the form of A X = X B, wherein , , wherein A and B are known numbers. The hand-eye calibration equation AX=XB is solved by changing the pose of the robot end multiple times, and the value of the hand-eye conversion matrix X is obtained.

[0139] S4, the grabbing priority score is calculated based on the distance between the centroid of the stacked workpiece point cloud data and the camera and the number of stacked workpiece point cloud points, and the workpiece with the highest grabbing priority score is taken as the optimal target.

[0140] Specifically, the calculation formula of the grabbing priority score is:

[0141] ;

[0142] wherein, represents the grabbing priority score; is a weight coefficient of the point cloud centroid position; is a weight coefficient of the point cloud quantity; represents the distance from the point cloud centroid to the depth camera; represents the point cloud point number; is a standard constant.

[0143] S5, based on the 6D pose information, guiding the robot arm to grab the optimal target, after the grabbing is completed, the next to-be-grabbed workpiece is selected in a loop, until all the to-be-grabbed workpieces complete the grabbing task.

[0144] As Figure 7 shown, the embodiment also discloses a stacked workpiece disordered grabbing system based on a 3D vision system, comprising:

[0145] The segmentation module 71 captures the stacked workpiece through the depth camera and extracts the RGB image and the depth image, generates the stacked workpiece point cloud data based on the RGB image and the depth image, performs instance segmentation on the RGB image and the depth image through the improved Mask R-CNN model integrated with the MSDA module, and obtains the segmented pixels.

[0146] The rigid transformation matrix acquisition module 72 performs pose estimation on the stacked workpiece, adopts a two-stage algorithm, the first stage maps the segmented pixels to the stacked workpiece point cloud data to extract local point clouds of each target, and the second stage registers the target local point cloud with the preset template point cloud through the ICP algorithm to obtain the rigid transformation matrix of the target local point cloud relative to the template;

[0147] The 6D pose information acquisition module 73 adopts a hand-in-eye calibration method to calibrate the robot base and the camera coordinates to obtain a hand-eye calibration matrix, combines the rigid transformation matrix with the hand-eye calibration matrix, and obtains the 6D pose information of the stacked workpiece in the robot base coordinate system;

[0148] The optimal target confirmation module 74 calculates the grabbing priority score based on the distance between the centroid of the stacked workpiece point cloud data and the camera and the point cloud point number of the stacked workpiece, and takes the workpiece with the highest grabbing priority score as the optimal target;

[0149] The grabbing module 75, based on the 6D pose information, guides the robot arm to grab the optimal target, after the grabbing is completed, the next to-be-grabbed workpiece is selected in a loop, until all the to-be-grabbed workpieces complete the grabbing task.

[0150] Although the present application has been particularly shown and described with respect to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application as defined in the appended claims.

Claims

1. A method for grabbing stacked workpieces in disorder based on a 3D vision system, characterized in that: include: S1 uses a depth camera to shoot stacked workpieces and extract RGB images and depth images. Based on the RGB images and depth images, point cloud data of the stacked workpieces is generated. The improved Mask R-CNN model with the integrated MSDA module is used to perform instance segmentation on the RGB images and depth images to obtain segmented pixels. S2, pose estimation of stacked workpieces, adopts a two-stage algorithm. In the first stage, the segmented pixels are mapped to the stacked workpiece point cloud data to extract each target local point cloud. In the second stage, each target local point cloud is aligned with the preset template point cloud through the ICP algorithm to obtain the rigid transformation matrix of the target local point cloud relative to the template. S3, using the hand-outside-eye calibration method, calibrate the robot base and camera coordinates to obtain the hand-eye calibration matrix. The rigid transformation matrix is ​​combined with the hand-eye calibration matrix to obtain the 6D pose information of the stacked workpiece in the robot base coordinate system; S4, calculates the grasping priority score based on the distance between the center of mass of the stacked workpiece point cloud data and the camera and the number of points in the stacked workpiece point cloud data, and selects the workpiece with the highest grasping priority score as the optimal target; S5, based on the 6D pose information, guides the robotic arm to grasp the optimal target. After the grasping is completed, the next workpiece to be grasped is selected in a loop until all the workpieces to be grasped have completed the grasping task.

2. The method for grabbing stacked workpieces in disorder based on a 3D vision system according to claim 1, characterized in that: In S1, the improved Mask R-CNN model with integrated MSDA module adopts ResNet-101 as the feature extraction backbone network of the Mask R-CNN model, and connects the MSDA module to the feature map output by ResNet-101 to achieve attention enhancement on the output feature map.

3. The method for grabbing stacked workpieces in disorder based on a 3D vision system according to claim 2, characterized in that: The MSDA module includes a multi-scale convolution branch unit, a feature fusion unit, a hole convolution path unit, an attention map generation unit, and an attention weighted output unit: Multi-scale convolutional branch units are used to extract feature maps output by the backbone network Perform 1×1, 3×3, and 5×5 convolution operations to obtain multi-scale context feature maps. The calculation formula is as follows: ； ； ； in, Represents a 1×1 convolution scale feature map; Represents a 3×3 convolution scale feature map; Represents a 5×5 convolutional scale feature map; Represents a 1×1 convolution operation; Represents a 3×3 convolution operation; Represents a 5×5 convolution operation; The feature fusion unit is used to splice the feature maps of the three scales and then perform a layer of convolution compression to generate a fused intermediate feature map. The calculation formula is as follows: ； in, Represented as a fused intermediate feature map; ) represents feature splicing; The dilated convolution path unit is used to perform multiple dilated convolution operations on the fused intermediate feature maps, with the dilation rates set to 1, 2, and 3, respectively, to obtain the three-way dilated convolution results. The formula is as follows: ； ； ； in, Represents the result of the first hole convolution; Represents the result of the second-way dilated convolution; Represents the result of the third-way dilated convolution; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 1; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 2; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 3; The attention map generation unit is used to add the dilated convolution results element by element, activate them through ReLU, and then perform 1×1 convolution and Sigmoid function to output the spatial attention map. The calculation formula is as follows: ； in, represents the spatial attention map; The attention weighted output unit is used to apply the spatial attention map to the fused intermediate feature map, multiplying each channel bitwise with the spatial weight element-by-element to obtain the enhanced feature map. The formula is as follows: ； in, Represents element-wise multiplication; Represents the enhanced feature map.

4. The method for grabbing stacked workpieces in disorder based on a 3D vision system according to claim 1, characterized in that: In S2, the first stage maps the segmented pixels to the stacked workpiece point cloud data to extract the local point cloud of each target, specifically including: The segmented pixel position (u, v) and the depth value Z(u, v) of the corresponding depth map are back-projected into 3D coordinate points to obtain the local point cloud of each target , the calculation formula is as follows: ； Among them, K is the camera intrinsic parameter matrix; represents the horizontal axis; Indicates the vertical coordinate; Represents a fixed constant.

5. The method for grabbing stacked workpieces in disorder based on a 3D vision system according to claim 1, characterized in that: In S2, the second stage uses the ICP algorithm to align the target local point cloud with the preset template point cloud to obtain the rigid transformation matrix of the target relative to the template. The specific process is as follows: By solving the optimization task, we get a rigid transformation matrix , so that the template point cloud After rotation and translation, it is aligned with the current target local point cloud. The objective function formula is as follows: ； in, Represents a point in the template point cloud; Represents a point in the target local point cloud; shows the rotation matrix; represents the translation vector; The process of solving this optimization task is: The first step is to remove the centroid and decentralize the point cloud so that the point cloud center of gravity is 0, obtain the points of the new point cloud, and then reintroduce it into the objective function and rearrange it to obtain two decomposed parts: ； Among them, the first item of the addition is the optimization target of the rotation part, and the second item is the optimization target of the translation part. 、 Represent the centroid vectors of the template point cloud and the target local point cloud respectively; The second step is to solve the R of the rotation part and construct the structural covariance matrix H between the decentralized template point cloud and the target point cloud: ； And perform singular value decomposition on it, specifically: ； Among them, U, and V denote the left singular matrix, singular value diagonal matrix and right singular matrix of matrix H respectively; According to the singular value decomposition results, the optimal solution of the rotation matrix R is calculated as follows: ； The third step is to calculate the translation vector t. The formula is as follows: ； Finally, a rigid transformation is applied to the template point cloud and repeat the matching update until convergence to obtain the rigid transformation matrix of the target relative to the template.

6. The method for grabbing stacked workpieces in disorder based on a 3D vision system according to claim 1, characterized in that: In S4, the calculation formula of the crawling priority score is: ； in, Indicates the crawl priority score; is the weight coefficient of the point cloud centroid position; is the weight coefficient of the number of point clouds; Indicates the distance from the point cloud centroid to the depth camera; Indicates the number of points in the point cloud; is a standard constant.

7. A stacked workpiece disordered grasping system based on a 3D vision system, characterized in that: include: The segmentation module uses a depth camera to capture stacked workpieces and extract RGB and depth images. Based on these images, it generates point cloud data for the stacked workpieces. The improved Mask R-CNN model integrated with the MSDA module performs instance segmentation on the RGB and depth images to obtain segmented pixels. The rigid transformation matrix acquisition module estimates the pose of stacked workpieces using a two-stage algorithm. In the first stage, the segmented pixels are mapped to the stacked workpiece point cloud data to extract each target local point cloud. In the second stage, each target local point cloud is aligned with the preset template point cloud through the ICP algorithm to obtain the rigid transformation matrix of the target local point cloud relative to the template. The 6D pose information acquisition module uses a hand-outside-eye calibration method to calibrate the robot base and camera coordinates to obtain a hand-eye calibration matrix. The rigid transformation matrix is ​​combined with the hand-eye calibration matrix to obtain the 6D pose information of the stacked workpiece in the robot base coordinate system. The optimal target confirmation module calculates the grasping priority score based on the distance between the center of mass of the stacked workpiece point cloud data and the camera and the number of points in the stacked workpiece point cloud data, and selects the workpiece with the highest grasping priority score as the optimal target; The grasping module guides the robotic arm to grasp the optimal target based on 6D pose information. After the grasping is completed, the next workpiece to be grasped is selected in a loop until all workpieces to be grasped have completed the grasping task.

Images