Stacked workpiece disordered grabbing method and system based on 3D visual system

Abstract

The invention discloses a stacked workpiece disordered grabbing method and system based on a 3D visual system, and relates to the technical field of computer vision, and the method comprises the steps: obtaining an RGB image and a depth image of a stacked workpiece through a depth camera, and generating point cloud data; the improved Mask R-CNN model integrated with the MSDA module is used for performing instance segmentation on RGB and depth images, and workpiece pose estimation is realized by combining an ICP algorithm and template point cloud registration; establishing a coordinate relation between the camera and the mechanical arm through hand-eye calibration, and obtaining a 6D pose of the workpiece under a mechanical arm base coordinate system; and according to the point cloud centroid distance and the point number, the grabbing priority is calculated, the optimal target is selected for grabbing, and disordered grabbing of all the workpieces is completed circularly. According to the method, the improved Mask R-CNN model integrating the MSDA module and the ICP algorithm are combined with the 3D visual system, so that accurate instance segmentation and pose estimation of stacked workpieces are realized, and therefore, an unordered grabbing task is accurately completed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for randomly grasping stacked workpieces based on a 3D vision system. Background Art

[0002] In the field of robotic manipulation, picking up three-dimensional objects in cluttered scenes is a classic and significant challenge. By utilizing robotic arms to grasp randomly stacked workpieces, robotic automation is possible. This capability enables robots to accurately perceive, locate, and grasp target objects in complex and changing environments, thereby improving production efficiency and reducing labor costs. This is of great significance for the development of industrial and service robots.

[0003] With the widespread adoption of robotics, grasping three-dimensional objects in cluttered scenes has become a key challenge in industrial automation. Among existing pose estimation methods, 2D feature point methods rely on object texture features, which presents a bottleneck in feature extraction for low-texture workpieces. Template matching-based methods achieve pose estimation through feature comparison, typically using the LINEMOD algorithm, which combines RGB-D information with ICP correction, but suffers from occlusion sensitivity. RANSAC and its improved algorithm, SAC-IA, improve accuracy but sacrifice efficiency and are often used in the initial registration phase. In recent years, deep learning technology has gradually replaced traditional hand-crafted features. PointNet and its derivative networks achieve point cloud feature extraction through end-to-end learning, but two-dimensional mapping methods are prone to loss of geometric information. Improved solutions such as VoxelNet still face the challenge of optimizing three-dimensional feature representation. For 3D object detection and pose estimation tasks, a PointNet-like structure was designed to achieve 3D target detection.

[0004] Currently, the main methods for workpiece pose estimation include 2D feature point methods, template matching methods, and deep learning-based methods. The 2D feature point method is a common method that extracts multiple feature points from the workpiece and uses the correspondence between these feature points and a reference coordinate system to infer the workpiece's pose. This method performs well on objects with rich textures and can achieve high-precision positioning. Template matching methods generate and compare object features and templates to find a template pose with a matching error less than a threshold, which serves as the initial pose of the scene object. In recent years, with the emergence of deep convolutional neural networks, traditional hand-crafted point cloud feature descriptors have been gradually replaced by learned descriptors. Deep convolutional neural networks possess powerful feature extraction capabilities and can automatically learn effective feature representations for point cloud data. Compared to hand-crafted feature descriptors, learned descriptors can better capture the important features of point cloud data and achieve better performance in point cloud processing and analysis tasks. However, some challenges and limitations still exist.

[0005] For example, when performing pose estimation, there are several challenges and limitations. First, feature point extraction becomes particularly difficult for objects with a single texture, which makes it impossible to accurately process low-texture workpieces. Therefore, when processing such workpieces, it is necessary to explore other more suitable methods to achieve precise positioning. Second, the integrity of point cloud data is very sensitive to the algorithm. Especially in a production environment, the stacking of workpieces may make it complicated and difficult to obtain complete point cloud data for a single workpiece. In addition, in the presence of complex occlusions, the accuracy of pose estimation may be affected. Furthermore, some pose estimation algorithms have high requirements for computing resources, which means that they may require a large amount of computing resources to complete the pose estimation task, making it impractical in practical applications. Finally, when processing stacked workpieces, the occlusion of the upper workpiece on the lower workpiece must be considered, and the algorithm must determine the optimal workpiece grasping order. Summary of the Invention

[0006] To solve the above problems, the present invention proposes a method and system for disordered grasping of stacked workpieces based on a 3D vision system. Instance segmentation is performed by integrating an improved Mask R-CNN model with an MSDA module, and the ICP algorithm is used to align point cloud data. Hand-eye calibration technology is used to achieve coordinate conversion, and the grasping priority is determined based on the point cloud centroid distance and number, thereby achieving accurate 6D pose estimation and efficient disordered grasping of stacked workpieces.

[0007] The specific plan is as follows:

[0008] On the one hand, a method for randomly grasping stacked workpieces based on a 3D vision system includes:

[0009] S1 uses a depth camera to shoot stacked workpieces and extract RGB images and depth images. Based on the RGB images and depth images, point cloud data of the stacked workpieces is generated. The improved Mask R-CNN model with the integrated MSDA module is used to perform instance segmentation on the RGB images and depth images to obtain segmented pixels.

[0010] S2, pose estimation of stacked workpieces, adopts a two-stage algorithm. In the first stage, the segmented pixels are mapped to the stacked workpiece point cloud data to extract each target local point cloud. In the second stage, each target local point cloud is aligned with the preset template point cloud through the ICP algorithm to obtain the rigid transformation matrix of the target local point cloud relative to the template.

[0011] S3, using the hand-outside-eye calibration method, calibrate the robot base and camera coordinates to obtain the hand-eye calibration matrix. The rigid transformation matrix is ​​combined with the hand-eye calibration matrix to obtain the 6D pose information of the stacked workpiece in the robot base coordinate system;

[0012] S4, calculates the grasping priority score based on the distance between the center of mass of the stacked workpiece point cloud data and the camera and the number of points in the stacked workpiece point cloud data, and selects the workpiece with the highest grasping priority score as the optimal target;

[0013] S5, based on the 6D pose information, guides the robotic arm to grasp the optimal target. After the grasping is completed, the next workpiece to be grasped is selected in a loop until all the workpieces to be grasped have completed the grasping task.

[0014] Furthermore, the improved Mask R-CNN model integrating the MSDA module adopts ResNet-101 as the feature extraction backbone network of the Mask R-CNN model, and connects the MSDA module to the feature map output by ResNet-101 to achieve attention enhancement on the output feature map.

[0015] Furthermore, the MSDA module includes a multi-scale convolution branch unit, a feature fusion unit, a hole convolution path unit, an attention map generation unit and an attention weighted output unit:

[0016] Multi-scale convolutional branch units are used to extract feature maps output by the backbone network Perform 1×1, 3×3, and 5×5 convolution operations to obtain multi-scale context feature maps. The calculation formula is as follows:

[0017] ;

[0018] ;

[0019] ;

[0020] in, Represents a 1×1 convolution scale feature map; Represents a 3×3 convolution scale feature map; Represents a 5×5 convolutional scale feature map; Represents a 1×1 convolution operation; Represents a 3×3 convolution operation; Represents a 5×5 convolution operation;

[0021] The feature fusion unit is used to splice the feature maps of the three scales and then perform a layer of convolution compression to generate a fused intermediate feature map. The calculation formula is as follows:

[0022] ;

[0023] in, Represented as a fused intermediate feature map; ) represents feature splicing;

[0024] The dilated convolution path unit is used to perform multiple dilated convolution operations on the fused intermediate feature maps, with the dilation rates set to 1, 2, and 3, respectively, to obtain the three-way dilated convolution results. The formula is as follows:

[0025] ;

[0026] ;

[0027] ;

[0028] in, Represents the result of the first hole convolution; Represents the result of the second-way dilated convolution; Represents the result of the third-way dilated convolution; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 1; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 2; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 3;

[0029] The attention map generation unit is used to add the dilated convolution results element by element, activate them through ReLU, and then perform 1×1 convolution and Sigmoid function to output the spatial attention map. The calculation formula is as follows:

[0030] ;

[0031] in, Represents the spatial attention map;

[0032] The attention weighted output unit is used to apply the spatial attention map to the fused intermediate feature map, multiplying each channel bitwise with the spatial weight element-by-element to obtain the enhanced feature map. The formula is as follows:

[0033] ;

[0034] in, Represents element-wise multiplication; Represents the enhanced feature map.

[0035] Furthermore, the first stage maps the segmented pixels to the stacked workpiece point cloud data to extract the local point cloud of each target, specifically including:

[0036] The segmented pixel position (u, v) and the depth value Z(u, v) of the corresponding depth map are back-projected into 3D coordinate points to obtain the local point cloud of each target , the calculation formula is as follows:

[0037] ;

[0038] Among them, K is the camera intrinsic parameter matrix; represents the horizontal axis; Indicates the vertical coordinate; Represents a fixed constant.

[0039] Furthermore, in the second stage, the target local point cloud is registered with the preset template point cloud through the ICP algorithm to obtain the rigid transformation matrix of the target relative to the template. The specific process is as follows:

[0040] By solving the optimization task, we get a rigid transformation matrix , so that the template point cloud After rotation and translation, it is aligned with the current target local point cloud. The objective function formula is as follows:

[0041] ;

[0042] in, Represents a point in the template point cloud; Represents a point in the target local point cloud; shows the rotation matrix; represents the translation vector;

[0043] The process of solving this optimization task is:

[0044] The first step is to remove the centroid and decentralize the point cloud so that the point cloud center of gravity is 0, obtain the points of the new point cloud, and then reintroduce it into the objective function and rearrange it to obtain two decomposed parts:

[0045] ;

[0046] Among them, the first item of the addition is the optimization target of the rotation part, and the second item is the optimization target of the translation part. 、 Represent the centroid vectors of the template point cloud and the target local point cloud respectively;

[0047] The second step is to solve the R of the rotation part and construct the structural covariance matrix H between the decentralized template point cloud and the target point cloud:

[0048] ;

[0049] And perform singular value decomposition on it, specifically:

[0050] ;

[0051] Among them, U, and V denote the left singular matrix, singular value diagonal matrix and right singular matrix of matrix H respectively;

[0052] According to the singular value decomposition results, the optimal solution of the rotation matrix R is calculated as follows:

[0053] ;

[0054] The third step is to calculate the translation vector t. The formula is as follows:

[0055] ;

[0056] Finally, a rigid transformation is applied to the template point cloud and repeat the matching update until convergence to obtain the rigid transformation matrix of the target relative to the template.

[0057] Furthermore, the calculation formula of the crawl priority score is:

[0058] ;

[0059] in, Indicates the crawl priority score; is the weight coefficient of the point cloud centroid position; is the weight coefficient of the number of point clouds; Indicates the distance from the point cloud centroid to the depth camera; Indicates the number of points in the point cloud; is a standard constant.

[0060] On the other hand, the stacked workpiece disordered grasping system based on the 3D vision system includes:

[0061] The segmentation module uses a depth camera to capture stacked workpieces and extract RGB and depth images. Based on these images, it generates point cloud data for the stacked workpieces. The improved Mask R-CNN model integrated with the MSDA module performs instance segmentation on the RGB and depth images to obtain segmented pixels.

[0062] The rigid transformation matrix acquisition module estimates the pose of stacked workpieces using a two-stage algorithm. In the first stage, the segmented pixels are mapped to the stacked workpiece point cloud data to extract each target local point cloud. In the second stage, each target local point cloud is aligned with the preset template point cloud through the ICP algorithm to obtain the rigid transformation matrix of the target local point cloud relative to the template.

[0063] The 6D pose information acquisition module uses a hand-outside-eye calibration method to calibrate the robot base and camera coordinates to obtain a hand-eye calibration matrix. The rigid transformation matrix is ​​combined with the hand-eye calibration matrix to obtain the 6D pose information of the stacked workpiece in the robot base coordinate system.

[0064] The optimal target confirmation module calculates the grasping priority score based on the distance between the center of mass of the stacked workpiece point cloud data and the camera and the number of points in the stacked workpiece point cloud data, and selects the workpiece with the highest grasping priority score as the optimal target;

[0065] The grasping module guides the robotic arm to grasp the optimal target based on 6D pose information. After the grasping is completed, the next workpiece to be grasped is selected in a loop until all workpieces to be grasped have completed the grasping task.

[0066] The present invention adopts the above technical solution and has the following beneficial effects:

[0067] (1) The present invention performs instance segmentation on RGB images and depth images by integrating an improved Mask R-CNN model with an MSDA module, which can more accurately identify and separate individual objects in stacked workpieces.

[0068] (2) The present invention uses the ICP algorithm to accurately align the target local point cloud with the preset template point cloud to obtain a rigid transformation matrix, which improves the reliability of workpiece pose estimation in the presence of occlusion, enabling the robotic arm to more accurately locate and grasp the target workpiece.

[0069] (3) The present invention adopts the hand-outside-eye calibration method to achieve accurate conversion between the robot base coordinate system and the camera coordinate system, ensuring that the 6D pose information obtained from the point cloud data can be directly applied to the robot arm control and optimizing the grasping path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a flow chart of a method for randomly grasping stacked workpieces based on a 3D vision system according to an embodiment of the present invention;

[0071] Figure 2 Schematic diagram of the ResNet-101 feature extraction network according to an embodiment of the present invention;

[0072] Figure 3 Schematic diagram of an improved feature extraction module according to an embodiment of the present invention;

[0073] Figure 4 A schematic diagram of a workpiece stacking data set according to an embodiment of the present invention;

[0074] Figure 5 A schematic diagram of a workpiece CAD model and point cloud according to an embodiment of the present invention;

[0075] Figure 6 This is a schematic diagram of eye-outside-hand calibration according to an embodiment of the present invention;

[0076] Figure 7 This is a diagram of a system for randomly grasping stacked workpieces based on a 3D vision system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The present invention will be described in further detail below with reference to the examples and accompanying drawings, but the embodiments of the present invention are not limited thereto. Figure 1 As shown, the present invention is based on a method for grasping stacked workpieces in an unordered manner using a 3D vision system, comprising:

[0078] S1 uses a depth camera to capture stacked workpieces and extract RGB and depth images. Point cloud data of the stacked workpieces is generated based on the RGB and depth images. An improved Mask R-CNN model with an integrated MSDA (Multi-Scale Dilated Attention) module is used to perform instance segmentation on the RGB and depth images to obtain segmented pixels.

[0079] Specifically, the improved Mask R-CNN model integrating the MSDA module adopts ResNet-101 as the feature extraction backbone network of the Mask R-CNN model, and enhances the attention of the output feature map by connecting the MSDA module to the ResNet-101 backbone network.

[0080] Specifically, the MSDA module includes a multi-scale convolution branch unit, a feature fusion unit, a hole convolution path unit, an attention map generation unit, and an attention weighted output unit:

[0081] Multi-scale convolutional branch units are used to extract feature maps output by the backbone network Perform 1×1, 3×3, and 5×5 convolution operations to obtain multi-scale context feature maps. The calculation formula is as follows:

[0082] ;

[0083] ;

[0084] ;

[0085] in, Represents a 1×1 convolution scale feature map; Represents a 3×3 convolution scale feature map; Represents a 5×5 convolutional scale feature map; Represents a 1×1 convolution operation; Represents a 3×3 convolution operation; Represents a 5×5 convolution operation;

[0086] The feature fusion unit is used to splice the feature maps of the three scales and then perform a layer of convolution compression to generate a fused intermediate feature map. The calculation formula is as follows:

[0087] ;

[0088] in, Represented as a fused intermediate feature map; ) represents feature splicing;

[0089] The dilated convolution path unit is used to perform multiple dilated convolution operations on the fused intermediate feature maps, with the dilation rates set to 1, 2, and 3, respectively, to obtain the three-way dilated convolution results. The formula is as follows:

[0090] ;

[0091] ;

[0092] ;

[0093] in, Represents the result of the first hole convolution; Represents the result of the second-way dilated convolution; Represents the result of the third-way dilated convolution; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 1; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 2; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 3;

[0094] The attention map generation unit is used to add the dilated convolution results element by element, activate them through ReLU, and then perform 1×1 convolution and Sigmoid function to output the spatial attention map. The calculation formula is as follows:

[0095] ;

[0096] in, represents the spatial attention map;

[0097] The attention weighted output unit is used to apply the spatial attention map to the fused intermediate feature map, multiplying each channel bitwise with the spatial weight element-by-element to obtain the final output feature map. The formula is as follows:

[0098] ;

[0099] in, Represents element-wise multiplication; Represents the final output feature map.

[0100] S2, pose estimation of stacked workpieces, adopts a two-stage algorithm. In the first stage, the segmented pixels are mapped to the stacked workpiece point cloud data to extract each target local point cloud. In the second stage, the target local point cloud is aligned with the preset template point cloud through the ICP algorithm to obtain the rigid transformation matrix of the target local point cloud relative to the template.

[0101] Specifically, the first stage maps the segmented pixels to the stacked workpiece point cloud data to extract the local point cloud of each target, which includes:

[0102] The segmented pixel position (u, v) and the depth value Z(u, v) of the corresponding depth map are back-projected into 3D coordinate points to obtain the local point cloud of each target , the calculation formula is as follows:

[0103] ;

[0104] Among them, K is the camera intrinsic parameter matrix; represents the horizontal axis; Indicates the vertical coordinate; Represents a fixed constant.

[0105] Specifically, in the second stage, the target local point cloud is registered with the preset template point cloud through the ICP algorithm to obtain the rigid transformation matrix of the target relative to the template. The specific process is as follows:

[0106] By solving the optimization task, we get a rigid transformation matrix , so that the template point cloud After rotation and translation, it is aligned with the current target local point cloud. The objective function formula is as follows:

[0107] ;

[0108] in, Represents a point in the template point cloud; Represents a point in the target local point cloud; shows the rotation matrix; represents the translation vector;

[0109] The process of solving this optimization task is:

[0110] The first step is to remove the centroid and decentralize the point cloud so that the point cloud center of gravity is 0, obtain the points of the new point cloud, and then reintroduce it into the objective function and rearrange it to obtain two decomposed parts:

[0111] ;

[0112] Among them, the first item of the addition is the optimization target of the rotation part, and the second item is the optimization target of the translation part. 、 Represent the centroid vectors of the template point cloud and the target local point cloud respectively.

[0113] The second step is to solve the R of the rotation part and construct the structural covariance matrix H between the decentralized template point cloud and the target point cloud:

[0114] ;

[0115] And perform singular value decomposition on it, specifically:

[0116] ;

[0117] Among them, U, , V are the left singular matrix, singular value diagonal matrix and right singular matrix of matrix H respectively.

[0118] According to the singular value decomposition results, the optimal solution of the rotation matrix R is:

[0119] ;

[0120] The third step is to calculate the translation vector t. The formula is as follows:

[0121] ;

[0122] Finally, a rigid transformation is applied to the template point cloud and repeat the matching update until convergence to obtain the rigid transformation matrix of the target relative to the template.

[0123] Specifically, in this embodiment, in order to solve the problems that may arise when the template matching method processes a multi-target scene, such as a large number of point clouds, inability to effectively match in real time, slow calculation speed, and waste of resources, a two-stage algorithm is proposed:

[0124] In the first stage, a local point cloud region containing a single target is extracted from the image. In the second stage, the target pose is estimated within this local point cloud region. This transforms the problem into a single-target pose estimation scenario, making it easier to handle and solve. This two-stage approach effectively addresses the challenges posed by multi-target scenarios and improves the accuracy and robustness of pose estimation.

[0125] In addition, in order to solve the problem that the traditional visual method is not ideal when sorting disordered stacked workpieces, the present invention improves the Mask R-CNN model by introducing the attention mechanism, and selects the ResNet-101 network as the feature extraction network of the Mask R-CNN model. Figure 2As shown in the figure. In the feature extraction stage, the input image is first subjected to layer-by-layer convolution. Through this process, feature maps of different levels are gradually extracted from the original image. We will record these feature maps as 、 、 、 、 In the feature fusion stage, first, Perform 1× convolution and output Secondly, Perform 2×2 upsampling and sum the corresponding elements after 1×1 convolution to obtain , and so on, we can get and ; then 、 、 、 Perform 3×3 convolution respectively to eliminate the aliasing effect caused by the upsampling operation; finally, perform 3×3 convolution on the Pooling is performed to obtain , and the final output { 、 、 、 、 } is used as the input feature of the region proposal network (RPN) for subsequent training, such as Figure 3 shown.

[0126] In the above process, 、 、 、 The calculation formula (excluding 3×3 convolution) is:

[0127] ;

[0128] ;

[0129] Where, Represents the feature map extracted by the ResNet-101 network in the first stage; express Represents the feature map output after feature fusion; represents a 1×1 convolutional layer; Indicates the addition operation of corresponding elements; Indicates 2×2 upsampling.

[0130] The CBAM attention mechanism is used to transform the feature fusion stage of the Mask R-CNN model's feature extraction network. By adding CBAM attention modules after the 1×1 convolutional layers C2, C3, C4, and C5, the Mask R-CNN model's channel and spatial focusing capabilities on stacked arrow information are improved, as shown in Figure 5. Because the CBAM attention module is directly embedded in the Mask R-CNN model and participates in the iterative update of the entire model's network parameters, as the model continues to train, the CBAM attention weight matrix is ​​continuously optimized and gradually converges to the optimal weight matrix.

[0131] The improved calculation formulas for P5, P4, P3, and P2 are:

[0132] ;

[0133] ;

[0134] Where, Represents the CBAM attention weight matrix corresponding to the feature map of the i-th stage; Represents the corresponding element-wise multiplication operation.

[0135] Since there is no suitable public workpiece stacking dataset available online, the dataset used in the present invention is manually collected. A single or multiple workpieces are selected to prevent them from being stacked on the workbench, and 600 images of workpiece stacking in different postures are collected by camera. The labelme tool is then used to manually annotate the stacked workpiece dataset. The labeled workpiece stacking dataset is relatively small in scale, and network training performed solely on this basis is difficult to achieve sufficient convergence of the network, and the network cannot achieve a satisfactory accuracy in the instance segmentation of the image. Therefore, before performing stacked workpiece segmentation, the improved Mask R-CNN network is first pre-trained on the COCO dataset. COCO is a large-scale target recognition dataset containing more than 100,000 images of 80 categories. It is a commonly used dataset in the field of target recognition. After obtaining the pre-training weights, training with the self-made workpiece stacking dataset can greatly accelerate the network convergence speed.

[0136] Specifically, Figure 4 The workpiece stacking dataset shown contains the original images used for image segmentation tasks and their corresponding mask files and annotation information. After mapping the image segmentation results to three-dimensional space to obtain the point cloud of the target workpiece, it is necessary to align it with the template point cloud to restore the spatial transformation matrix between the two point clouds, and then obtain the target pose. In this embodiment, UG is used to model the grasped workpiece, and the modeled model is converted into a point cloud in pcd format using Cloudcompare as a template point cloud. The modeling effect is similar to the template point cloud. Figure 5 shown.

[0137] S3 uses the hand-outside-eye calibration method to calibrate the robot base and camera coordinates to obtain the hand-eye calibration matrix. The rigid transformation matrix is ​​combined with the hand-eye calibration matrix to obtain the 6D pose information of the stacked workpiece in the robot base coordinate system.

[0138] Specifically, Figure 6 The present invention adopts the "eye outside the hand" calibration method. The base of the robot arm is fixed on the workbench, and the position of the camera is also fixed. The transformation matrix of the camera relative to the robot arm base coordinate is obtained through calibration. Among them, the camera coordinate system is , the calibration plate coordinate system is , the coordinate system of the end of the robot arm is , the robot arm base coordinate system is , involving the conversion of several coordinate systems, that is, the conversion relationship from the calibration plate coordinate system to the camera coordinate system , the conversion relationship from the camera coordinate system to the robot base coordinate system is X, and the conversion relationship from the robot base to the robot end coordinate system is , where the transformation relationship X from the camera coordinate system to the robot arm base coordinate system is the hand-eye calibration matrix that needs to be solved. In the above case, the calibration plate is fixed to the end of the robot arm, and in a specific posture, the points on the calibration plate have coordinate values ​​in the calibration plate coordinate system ,go through , , After the coordinate system conversion relationship is converted, the points on the calibration plate can be converted to the coordinate values ​​of the robot end coordinate system. , the conversion relationship is as follows: ; Then the robot arm changes its posture, and another set of formulas with the same form as above can be obtained, namely: ; Replace the above formula with 、 It can be obtained through the robot's posture output, 、 It can be obtained through the external parameters of camera calibration and can be converted into the following form: It can be further converted into: ; Understand it as AX = XB, then 、 , where A and B are both known numbers. By repeatedly transforming the end-arm posture, the hand-eye calibration equation AX=XB is solved to obtain the value of the hand-eye conversion matrix X.

[0139] S4, calculates the grasping priority score based on the distance between the center of mass of the stacked workpiece point cloud data and the camera and the number of points in the stacked workpiece point cloud data, and selects the workpiece with the highest grasping priority score as the optimal target.

[0140] Specifically, the calculation formula for the crawl priority score is:

[0141] ;

[0142] in, Indicates the crawl priority score; is the weight coefficient of the point cloud centroid position; is the weight coefficient of the number of point clouds; Indicates the distance from the point cloud centroid to the depth camera; Indicates the number of points in the point cloud; is a standard constant.

[0143] S5, based on the 6D pose information, guides the robotic arm to grasp the optimal target. After the grasping is completed, the next workpiece to be grasped is selected in a loop until all the workpieces to be grasped have completed the grasping task.

[0144] like Figure 7 As shown, this embodiment also discloses a stacked workpiece disordered grasping system based on a 3D vision system, comprising:

[0145] A segmentation module 71 captures the stacked workpieces using a depth camera and extracts RGB images and depth images, generates point cloud data of the stacked workpieces based on the RGB images and depth images, and performs instance segmentation on the RGB images and depth images using an improved Mask R-CNN model integrated with the MSDA module to obtain segmented pixels.

[0146] The rigid transformation matrix acquisition module 72 estimates the pose of the stacked workpieces using a two-stage algorithm. In the first stage, the segmented pixels are mapped to the stacked workpiece point cloud data to extract each target local point cloud. In the second stage, the target local point cloud is registered with the preset template point cloud using the ICP algorithm to obtain the rigid transformation matrix of the target local point cloud relative to the template.

[0147] The 6D pose information acquisition module 73 uses a hand-outside-eye calibration method to calibrate the robot base and camera coordinates to obtain a hand-eye calibration matrix. The rigid transformation matrix is ​​combined with the hand-eye calibration matrix to obtain the 6D pose information of the stacked workpiece in the robot base coordinate system.

[0148] The optimal target confirmation module 74 calculates a grasping priority score based on the distance between the center of mass of the stacked workpiece point cloud data and the camera and the number of points in the stacked workpiece point cloud data, and selects the workpiece with the highest grasping priority score as the optimal target;

[0149] The grasping module 75 guides the robot arm to grasp the optimal target based on the 6D posture information. After the grasping is completed, the next workpiece to be grasped is selected in a loop until all the workpieces to be grasped have completed the grasping task.

[0150] Although the present invention has been particularly shown and described in conjunction with preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. A method for grabbing stacked workpieces in disorder based on a 3D vision system, characterized in that: include: S1 uses a depth camera to shoot stacked workpieces and extract RGB images and depth images. Based on the RGB images and depth images, point cloud data of the stacked workpieces is generated. The improved Mask R-CNN model with the integrated MSDA module is used to perform instance segmentation on the RGB images and depth images to obtain segmented pixels. S2, pose estimation of stacked workpieces, adopts a two-stage algorithm. In the first stage, the segmented pixels are mapped to the stacked workpiece point cloud data to extract each target local point cloud. In the second stage, each target local point cloud is aligned with the preset template point cloud through the ICP algorithm to obtain the rigid transformation matrix of the target local point cloud relative to the template. S3, using the hand-outside-eye calibration method, calibrate the robot base and camera coordinates to obtain the hand-eye calibration matrix. The rigid transformation matrix is ​​combined with the hand-eye calibration matrix to obtain the 6D pose information of the stacked workpiece in the robot base coordinate system; S4, calculates the grasping priority score based on the distance between the center of mass of the stacked workpiece point cloud data and the camera and the number of points in the stacked workpiece point cloud data, and selects the workpiece with the highest grasping priority score as the optimal target; S5, based on the 6D pose information, guides the robotic arm to grasp the optimal target. After the grasping is completed, the next workpiece to be grasped is selected in a loop until all the workpieces to be grasped have completed the grasping task.

2. The method for grabbing stacked workpieces in disorder based on a 3D vision system according to claim 1, characterized in that: In S1, the improved Mask R-CNN model with integrated MSDA module adopts ResNet-101 as the feature extraction backbone network of the Mask R-CNN model, and connects the MSDA module to the feature map output by ResNet-101 to achieve attention enhancement on the output feature map.

3. The method for grabbing stacked workpieces in disorder based on a 3D vision system according to claim 2, characterized in that: The MSDA module includes a multi-scale convolution branch unit, a feature fusion unit, a hole convolution path unit, an attention map generation unit, and an attention weighted output unit: Multi-scale convolutional branch units are used to extract feature maps output by the backbone network Perform 1×1, 3×3, and 5×5 convolution operations to obtain multi-scale context feature maps. The calculation formula is as follows: ； ； ； in, Represents a 1×1 convolution scale feature map; Represents a 3×3 convolution scale feature map; Represents a 5×5 convolutional scale feature map; Represents a 1×1 convolution operation; Represents a 3×3 convolution operation; Represents a 5×5 convolution operation; The feature fusion unit is used to splice the feature maps of the three scales and then perform a layer of convolution compression to generate a fused intermediate feature map. The calculation formula is as follows: ； in, Represented as a fused intermediate feature map; ) represents feature splicing; The dilated convolution path unit is used to perform multiple dilated convolution operations on the fused intermediate feature maps, with the dilation rates set to 1, 2, and 3, respectively, to obtain the three-way dilated convolution results. The formula is as follows: ； ； ； in, Represents the result of the first hole convolution; Represents the result of the second-way dilated convolution; Represents the result of the third-way dilated convolution; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 1; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 2; Indicates a dilated convolution with a kernel of 3x3 and a dilation rate of 3; The attention map generation unit is used to add the dilated convolution results element by element, activate them through ReLU, and then perform 1×1 convolution and Sigmoid function to output the spatial attention map. The calculation formula is as follows: ； in, represents the spatial attention map; The attention weighted output unit is used to apply the spatial attention map to the fused intermediate feature map, multiplying each channel bitwise with the spatial weight element-by-element to obtain the enhanced feature map. The formula is as follows: ； in, Represents element-wise multiplication; Represents the enhanced feature map.

4. The method for grabbing stacked workpieces in disorder based on a 3D vision system according to claim 1, characterized in that: In S2, the first stage maps the segmented pixels to the stacked workpiece point cloud data to extract the local point cloud of each target, specifically including: The segmented pixel position (u, v) and the depth value Z(u, v) of the corresponding depth map are back-projected into 3D coordinate points to obtain the local point cloud of each target , the calculation formula is as follows: ； Among them, K is the camera intrinsic parameter matrix; represents the horizontal axis; Indicates the vertical coordinate; Represents a fixed constant.

5. The method for grabbing stacked workpieces in disorder based on a 3D vision system according to claim 1, characterized in that: In S2, the second stage uses the ICP algorithm to align the target local point cloud with the preset template point cloud to obtain the rigid transformation matrix of the target relative to the template. The specific process is as follows: By solving the optimization task, we get a rigid transformation matrix , so that the template point cloud After rotation and translation, it is aligned with the current target local point cloud. The objective function formula is as follows: ； in, Represents a point in the template point cloud; Represents a point in the target local point cloud; shows the rotation matrix; represents the translation vector; The process of solving this optimization task is: The first step is to remove the centroid and decentralize the point cloud so that the point cloud center of gravity is 0, obtain the points of the new point cloud, and then reintroduce it into the objective function and rearrange it to obtain two decomposed parts: ； Among them, the first item of the addition is the optimization target of the rotation part, and the second item is the optimization target of the translation part. 、 Represent the centroid vectors of the template point cloud and the target local point cloud respectively; The second step is to solve the R of the rotation part and construct the structural covariance matrix H between the decentralized template point cloud and the target point cloud: ； And perform singular value decomposition on it, specifically: ； Among them, U, and V denote the left singular matrix, singular value diagonal matrix and right singular matrix of matrix H respectively; According to the singular value decomposition results, the optimal solution of the rotation matrix R is calculated as follows: ； The third step is to calculate the translation vector t. The formula is as follows: ； Finally, a rigid transformation is applied to the template point cloud and repeat the matching update until convergence to obtain the rigid transformation matrix of the target relative to the template.

6. The method for grabbing stacked workpieces in disorder based on a 3D vision system according to claim 1, characterized in that: In S4, the calculation formula of the crawling priority score is: ； in, Indicates the crawl priority score; is the weight coefficient of the point cloud centroid position; is the weight coefficient of the number of point clouds; Indicates the distance from the point cloud centroid to the depth camera; Indicates the number of points in the point cloud; is a standard constant.

7. A stacked workpiece disordered grasping system based on a 3D vision system, characterized in that: include: The segmentation module uses a depth camera to capture stacked workpieces and extract RGB and depth images. Based on these images, it generates point cloud data for the stacked workpieces. The improved Mask R-CNN model integrated with the MSDA module performs instance segmentation on the RGB and depth images to obtain segmented pixels. The rigid transformation matrix acquisition module estimates the pose of stacked workpieces using a two-stage algorithm. In the first stage, the segmented pixels are mapped to the stacked workpiece point cloud data to extract each target local point cloud. In the second stage, each target local point cloud is aligned with the preset template point cloud through the ICP algorithm to obtain the rigid transformation matrix of the target local point cloud relative to the template. The 6D pose information acquisition module uses a hand-outside-eye calibration method to calibrate the robot base and camera coordinates to obtain a hand-eye calibration matrix. The rigid transformation matrix is ​​combined with the hand-eye calibration matrix to obtain the 6D pose information of the stacked workpiece in the robot base coordinate system. The optimal target confirmation module calculates the grasping priority score based on the distance between the center of mass of the stacked workpiece point cloud data and the camera and the number of points in the stacked workpiece point cloud data, and selects the workpiece with the highest grasping priority score as the optimal target; The grasping module guides the robotic arm to grasp the optimal target based on 6D pose information. After the grasping is completed, the next workpiece to be grasped is selected in a loop until all workpieces to be grasped have completed the grasping task.

Images