A multi-task robotic grasping method and system for object pile-up scenes

Abstract

A multi-task robot grasping method and system for an object stacking scene, the multi-task robot grasping method comprising: constructing a data set; constructing a multi-task robot grasping network; inputting the data set into the multi-task robot grasping network, outputting a grasping detection result, a target detection result and a target operation relationship result through a grasping detection module, a target detection module and a visual operation relationship prediction module respectively, calculating a total loss according to the grasping detection result, the target detection result and the target operation relationship result, and adjusting the multi-task robot grasping network until the total loss is minimized; deploying the trained multi-task robot grasping network into a multi-task robot grasping system, and performing target detection, grasping detection and visual operation relationship prediction to obtain a final grasping result. The bidirectional feature fusion module with a built-in symmetric structure in the full-flow bidirectional feature fusion backbone network can effectively improve the representation ability of the model for details and global structures.

Description

Technical Field

[0001] This invention relates to the field of robot grasping technology, and in particular to a multi-task robot grasping method and system for object stacking scenarios. Background Technology

[0002] In unstructured environments, accurately grasping objects buried under heavy stacks is a crucial skill for robots. Beyond picking up objects in the home, in practical applications of warehouse robots or service robots, robots often work closely with humans. In these cases, robots not only need to accurately grasp objects but also determine their location and category. In recent years, computer vision technology has made significant progress in object classification, detection, and segmentation. However, robots still face many limitations in understanding complex relationships between objects and planning appropriate operational sequences. This lack of capability severely restricts the robot's interaction with its environment, especially in dynamic or unstructured scenarios. To achieve reliable grasping, detection, and operational relationship reasoning in complex environments, an efficient network structure integrating multimodal feature information is particularly critical.

[0003] With the development of deep learning, many deep learning-based grasping methods have been proposed. These methods mostly take color or depth images as input, then extract features using CNNs or Transformers, and use these features for classification or regression to predict the final grasping location. These methods have made significant progress in grasping tasks in some simple and even complex environments. However, for certain scenarios (such as grasping specific objects in complex stacked environments), robots also need a certain level of reasoning ability to grasp targets in an orderly manner.

[0004] To achieve this, we must first examine our own grasping abilities. For humans, when picking up specific objects in complex stacked environments, our vision system first processes the manipulation relationships between objects in the scene within 100 milliseconds. Then, we know to pre-grasp objects stacked on top of the target object during the grasping process, and finally complete the grasp of the target object. Inspired by this grasping strategy, researchers proposed a visual manipulation relationship network. This network can analyze the stacking relationships of objects in a scene based on extracted features, outputting reasonable object manipulation relationships to help robots perform grasping tasks in stacked scenes. However, it is worth noting that this network only uses RGB images as input and does not consider geometric information in the depth image, causing the method to degrade in the face of severe occlusion.

[0005] In summary, although deep learning-based robotic grasping has made significant progress, it still faces many challenges. For example, when grasping specific objects in complex stacked environments, the robot may fail to accurately capture the spatial relationships between objects, leading to grasping failure. Furthermore, there are substantial modal differences between RGB images and depth images; how to fully utilize these two complementary data sets is also an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a multi-task robot grasping method and system for object stacking scenarios to solve the technical problems mentioned in the background art.

[0007] To achieve the above objectives, the technical solution of the present invention is implemented as follows: This invention provides a multi-task robot grasping method for object stacking scenarios, comprising the following steps: S1. Collect multiple RGB images and multiple corresponding depth images of a scene containing multiple stacked objects; use the RGB images and corresponding depth images as image pairs to construct a dataset; S2. Construct a multi-task robot grasping network. The multi-task robot grasping network includes a full-flow bidirectional feature fusion backbone network, a target detection module, a grasping detection module, and a visual operation relationship prediction module. The output of the full-flow bidirectional feature fusion backbone network is connected to the grasping detection module and the target detection module, respectively. The output of the target detection module and the output of the full-flow bidirectional feature fusion backbone network are connected to the input of the visual operation relationship prediction module, respectively. S3. Input the dataset into the multi-task robot grasping network. The grasping detection module, target detection module, and visual operation relationship prediction module output the grasping detection result, target detection result, and target operation relationship result, respectively. Calculate the total loss based on the grasping detection result, target detection result, and target operation relationship result, and adjust the multi-task robot grasping network according to the total loss until the total loss is minimized to obtain the trained multi-task robot grasping network. S4. Deploy the trained multi-task robot grasping network into the multi-task robot grasping system, and use the trained multi-task robot grasping network to perform grasping detection in multi-object stacking scenarios to obtain detection results and operation relationship tree.

[0008] In another aspect, the present invention provides a multi-task robot grasping system, including a device end, which is configured to execute the above-described multi-task robot grasping method.

[0009] The beneficial effects of this invention are: This invention discloses a multi-task robot grasping method for object stacking scenarios, and internally discloses a multi-task robot grasping network to complete target detection, grasping detection and operation relationship inference in multi-object stacking scenarios; Furthermore, a full-flow bidirectional feature fusion backbone network (BFFB) with a symmetrical structure is integrated into the multi-task robot grasping network to enhance the feature fusion and interaction capabilities between modalities. Then, a bidirectional feature fusion module (BMF) with a symmetrical structure is embedded in the full-flow bidirectional feature fusion backbone network. By combining local and global contextual information, the bidirectional feature fusion module (BMF) can capture the complementarity of features at different scales, effectively improving the multi-task robot grasping network's ability to represent details and global structures. The bidirectional feature fusion module (BMF) integrates a multi-scale channel attention module (MS-CAM), which can make full use of the additional appearance and geometric information of different branches to enhance feature representation capabilities. In addition, the multi-task robot grasping network also incorporates a grasping detection module, a target detection module, and a visual operation relationship prediction module. By combining the final cross-modal fusion features and the predicted bounding boxes of objects, it achieves accurate prediction of the object category and operation sequence in complex scenes. Attached Figure Description

[0010] Figure 1 This is a block diagram of the operation of the multi-task robot grasping network in this invention; Figure 2 This is a structural block diagram of the bidirectional feature fusion module (BMF) in this invention; Figure 3 This is a partial structural diagram of the multi-task robot grasping system in this invention; Figure 4 This is a schematic diagram of the crawling and detection results of the present invention on the VMRD dataset; Figure 5 This is a schematic diagram of the crawling and detection results of this invention on the Cornell dataset; Figure 6 This is a schematic diagram of the crawling and detection results of this invention on the Jacquard dataset; Figure 7 This is a schematic diagram of the target detection, grasping detection, and visual relationship reasoning results of the present invention on the VMRD dataset, wherein Figures (a) to (b) are the operation relationship trees in simple stacking scenarios and complex stacking scenarios, respectively. Detailed Implementation

[0011] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0012] It should be noted that when a component is referred to as "fixed" or "set" on another component, it can be directly on or indirectly on the other component. When a component is referred to as "connected" to another component, it can be directly connected to or indirectly connected to the other component.

[0013] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0014] It should also be noted that in the embodiments of this application, the same reference numerals are used to represent the same component or part. For the same part in the embodiments of this application, the reference numerals may only be used to mark one part or component as an example in the figure. It should be understood that the reference numerals are also applicable to other identical parts or components.

[0015] Reference Figure 1 This application provides a multi-task robot grasping method for object stacking scenarios, including the following steps: S1. Collect multiple RGB images and multiple corresponding depth images of a scene containing multiple stacked objects; use the RGB images and corresponding depth images as image pairs to construct a dataset; S2. Construct a multi-task robot grasping network. The multi-task robot grasping network includes a full-flow bidirectional feature fusion backbone network, a target detection module, a grasping detection module, and a visual operation relationship prediction module. The output of the full-flow bidirectional feature fusion backbone network is connected to the grasping detection module and the target detection module, respectively. The output of the target detection module and the output of the full-flow bidirectional feature fusion backbone network are connected to the input of the visual operation relationship prediction module, respectively. S3. Input the dataset into the multi-task robot grasping network. The grasping detection module, target detection module, and visual operation relationship prediction module output the grasping detection result, target detection result, and target operation relationship result, respectively. Calculate the total loss based on the grasping detection result, target detection result, and target operation relationship result, and adjust the multi-task robot grasping network according to the total loss until the total loss is minimized to obtain the trained multi-task robot grasping network. S4. Deploy the trained multi-task robot grasping network into the multi-task robot grasping system, and use the trained multi-task robot grasping network to perform grasping detection in multi-object stacking scenarios to obtain detection results and operation relationship tree. Figure 7 Operation relationship trees are presented for four different stacking scenarios.

[0016] In some embodiments, the full-flow bidirectional feature fusion backbone network in S2 includes three layers of full-flow bidirectional feature fusion units connected in sequence. Each layer of full-flow bidirectional feature fusion unit includes two first convolutional layers and a bidirectional feature fusion module (BMF), wherein the outputs of the two first convolutional layers are respectively connected to the bidirectional feature fusion module (BMF). The grasping and detection module includes three second convolutional layers and three 1×1 convolutional layers. The output terminals of the three second convolutional layers are connected in sequence and then connected to the three 1×1 convolutional layers respectively. The target detection module includes two third convolutional layers, one first average pooling layer, and two first fully connected layers. The output of the two third convolutional layers is connected in sequence and then connected to the first average pooling layer. The output of the first average pooling layer is connected to the two first fully connected layers respectively.

[0017] In some embodiments, the bidirectional feature fusion module (BMF) adopts a symmetrical design, including two 1×1 convolutional layers and two multi-scale channel attention modules (MS-CAM). The input of one of the 1×1 convolutional layers is connected to two first convolutional layers of the corresponding full-flow bidirectional feature fusion unit via a concatenation operation. Its output is connected to the input of one of the multi-scale channel attention modules (MS-CAMs). The output of the MS-CAM is then connected to the two first convolutional layers via a concatenation operation, followed by a summation operation, and finally connected to the input of another MS-CAM. The output of the MS-CAM is then connected to the two first convolutional layers via a summation operation and finally connected to another 1×1 convolutional layer. The bidirectional feature fusion module (BMF) acts as a communication bridge between adjacent full-flow bidirectional feature fusion units, helping them utilize additional appearance and geometric information from the other full-flow bidirectional feature fusion unit to enhance their representation learning capabilities. In some embodiments, S3 specifically includes the following steps: S31. First, select RGB images and corresponding depth images from the dataset and input them into the full-flow bidirectional feature fusion backbone network to obtain the final cross-modal fusion features of each object in the multi-object stacking scene. S32. Input the final cross-modal fusion features obtained in S31 into the grasping and detection module to obtain the grasping and detection results; S33. Input the final cross-modal fusion feature obtained in S31 into the object detection module. After the convolution operation of the two third convolutional layers in the object detection module, the feature dimension is reduced by the first average pooling layer. Finally, the predicted bounding box and predicted category of each object in the multi-object stacking scene are output through the two first fully connected layers respectively. S34. Input the predicted bounding box of each object and the final cross-modal fusion features into the visual operation relationship prediction module to perform operation relationship prediction and obtain the operation relationship tree. S35. Construct a total loss function based on the grasping detection results, target detection results, operation relationship tree, and real data. Adjust the parameters of the multi-task robot grasping network based on the total loss calculated by the total loss function. S36. Determine whether the iteration stopping condition has been met. If so, output the trained multi-task robot grasping network. Otherwise, loop from S31 to S36 until the iteration stopping condition is met, and output the trained multi-task robot grasping network.

[0018] In some embodiments, S31 specifically includes the following steps: S311. First, the input features of RGB images and corresponding depth images are selected from the dataset and input into the first layer of the full-flow bidirectional feature fusion backbone network. After passing through two first convolutional layers, the feature vector is obtained. and , to feature vector and The input is fed into the bidirectional feature fusion module (BMF) to obtain the cross-modal fusion features of the first layer. This achieves cross-modal feature fusion, which preserves the features of the original modality while enabling feature interaction between modalities, effectively improving the representational ability of the target features. The above process is expressed by the following formula: ; in, This represents the first convolutional layer; , These represent RGB images and depth images, respectively. S312, Fuse the cross-modal features of the first layer The input is fed into the full-flow bidirectional feature fusion unit of the second layer, and after passing through two first convolutional layers, it is fed into the bidirectional feature fusion module (BMF) again to obtain the cross-modal fusion features of the second layer. S313. The cross-modal fusion features of the second layer are input into the full-flow bidirectional feature fusion unit of the third layer. After passing through two first convolutional layers, they are input into the bidirectional feature fusion module BMF again to obtain the final cross-modal fusion features of each object in the multi-object stacking scene.

[0019] In some embodiments, the feature vector in S311 and The input is fed into the bidirectional feature fusion module (BMF) to obtain the fused features of the first layer. Specifically, the steps include the following: S3111, The feature vector and The concatenation operation is performed along the channel dimension, and then the data is input into a 1×1 convolutional layer 2 for cross-channel feature combination, thereby obtaining preliminary fused features. The specific formula is expressed as follows: ; in, This represents a 1×1 convolutional layer 2; Indicates a splicing operation; S3112, Subsequently, the initial fusion features will be... The input is fed into the multi-scale channel attention module MS-CAM, and then compared with the feature vectors respectively. and Perform a stitching operation along the channel dimension to obtain the fused features. , The formula is as follows: ; S3113, Integrating Features , After performing element-wise addition, the data is input into the next multi-scale channel attention module (MS-CAM) to obtain the fused features. The formula is as follows: ; in, This indicates an element-wise addition operation; S3114, The feature vector and and fusion features The fused features are obtained after performing element-wise addition. The formula is as follows: ; S3115, and then merge the features The input is fed into a 1×1 convolutional layer 2 to obtain the cross-modal fusion features of the first layer; the formula is as follows: .

[0020] In some embodiments, S32 specifically includes the following steps: S321. Input the final cross-modal fusion features obtained in S31 into the grasping detection module. The feature dimension is restored to the same dimension as the input features of the RGB image and depth image through three second convolutional layers. Then, they are input into three 1×1 convolutional layers to obtain the grasping quality pixel map, grasping angle pixel map and grasping width pixel map respectively. S322. Combine the capture quality pixel map, capture angle pixel map, and capture width pixel map to obtain the capture detection result of the topmost object in the object stacking scene, expressed by the following formula: ; in, This indicates the capture and detection result of the topmost object in a scene of stacked objects; These represent the pixel image of the capture quality, the pixel image of the capture angle, and the pixel image of the capture width, respectively. Represents the set of real numbers; H , W These represent the height and width of the image, respectively. S323. Map the angle information in the capture angle pixel image to two trigonometric functions, with the angle output range extending to [-π / 2, π / 2]. Finally, integrate them to obtain the capture angle scalar value for each pixel in the capture angle pixel image, expressed by the following formula: ; in, Π represents the scalar value of the capture angle for each pixel in the capture angle pixel map; Π represents a natural number.

[0021] In some embodiments, S34 specifically includes the following steps: S341. The predicted bounding box of each object and the final cross-modal fusion features are input into two sets of convolutional layers, each set of convolutional layers includes two fourth convolutional layers connected in sequence, to extract the object features of each object. S342. The object features of each object are reduced in dimensionality using two second average pooling layers. Finally, the operation relationships are predicted using two second fully connected layers to obtain the operation relationship tree. The operation relationship tree and the predicted class of each object constitute the target detection result.

[0022] In some embodiments, the total loss function in S35 is calculated as follows: ; in, Indicates the total loss; This represents the associated loss for predicting the bounding box and the predicted category; Indicates the loss detected; Indicates the predicted loss based on operational relationships; In the formula, the relevant loss for predicting the bounding box and the predicted category is... The calculation formula is as follows: ; in, It is the scaling factor for this loss during training; It is a category of network prediction. It is the ground truth value of the category; It is the location of the bounding box predicted by the network; It is the ground truth value of the bounding box; and It is a normalized project; Represents the cross-entropy loss function; It is smooth loss; In the formula, the capture detection loss The calculation formula is as follows: ; in, n It is the total number of pixels in the image. , and These are the pixel images of capture quality, capture angle, and capture width, respectively, corresponding to the [missing information]. i The value of each pixel. , and These are the real labels pre-labeled in the dataset; , and It is the proportional factor for each loss during training; In the formula, the operational relationship predicts the loss. The calculation formula is as follows: ; in, This represents the total number of all possible object pairs; The object detection module predicts possible object pairs. It is the ground truth value.

[0023] In some embodiments, S4 specifically includes the following steps: S41. Deploy the trained multi-task robot grasping network into the multi-task robot grasping system; S42. Using the trained multi-task robot grasping network, perform visual operation relationship prediction, target detection, and grasping detection in a multi-object stacking scenario, and obtain the operation relationship tree, prediction category, and grasping detection results of the first object starting from the bottom of the operation relationship tree. S43. Start the dual-arm robot on the multi-task robot grasping system. The dual-arm robot uses electric grippers to grasp the first object at the bottom of the operation relationship tree in the object stacking scene and place it in other designated positions. S44. Then, the grab detection result of the first object at the bottom of the operation relationship tree is obtained again through the grab detection module. S45, repeat S43 to S44 until the multi-task robot grasping system completes the grasping of multiple objects in the object stacking scenario.

[0024] This invention integrates a symmetrically designed full-flow bidirectional feature fusion backbone network (BFFB) into a multi-task robot grasping network, enhancing the feature fusion and interaction capabilities between modalities. Then, a symmetrically designed bidirectional feature fusion module (BMF) is embedded within the full-flow bidirectional feature fusion backbone network. By combining local and global contextual information, the BMF can capture the complementarity of features at different scales, effectively improving the multi-task robot grasping network's ability to represent details and global structures. Furthermore, the BMF integrates a multi-scale channel attention module (MS-CAM), which fully utilizes the additional appearance and geometric information of different branches to enhance feature representation capabilities. In addition, the multi-task robot grasping network also incorporates a grasping detection module, a target detection module, and a visual operation relationship prediction module. By combining the final cross-modal fusion features and the predicted bounding boxes of objects, it achieves accurate prediction of the object category and operation sequence in complex scenes.

[0025] The following verification experiments demonstrate the effectiveness of the present invention; First, this application selects the following three datasets for validation experiments: VMRD dataset, Cornell dataset, and Jacquard dataset. (1) VMRD dataset: The VMRD dataset contains 31 object classes and 4683 grasping images, with over 100k grasping labels. In each image, there are 2-5 objects stacked and occluded together. Since the VMRD dataset not only provides grasping annotations but also includes object instance-level bounding boxes and uses tree diagrams to label manipulation relationships, it can be used simultaneously for training object detection, grasping detection, and visual manipulation relationship tasks; (2) Cornell Dataset: The Cornell dataset contains 885 RGBD images and 8019 labels, representing over 280 classes of grasping objects. They are labeled with multiple positive and negative grasping postures; (3) Jacquard dataset: The Jacquard dataset contains over 5.4K sets of RGBD images and 1.1 million crawling labels; Since the Cornell and Jacquard datasets only provide object grasping labels and not object bounding boxes, only the grasping detection module in the multi-task robot grasping network was trained on the Cornell and Jacquard datasets. Furthermore, to enhance the generalization ability of the multi-task robot grasping network and prevent overfitting, data augmentation methods were introduced during training, including random brightness and contrast adjustments, color space conversion, and horizontal flipping.

[0026] Experimental Setup: A total of five experiments were designed. The first experiment was an object detection experiment on the VMRD dataset, the second experiment was a grasping detection experiment on the Cornell, Jacquard, and VMRD datasets, and the third experiment was a visual manipulation relationship experiment on VMRD, aiming to verify the generality of the invention. The fourth experiment was a network ablation experiment, analyzing the contribution of each module to the performance of the multi-task robot grasping network. The fifth experiment was a grasping experiment on a real robot in a complex stacked environment, further verifying the robustness and feasibility of the invention.

[0027] Evaluation metrics: The evaluation metrics are divided into three parts: object detection, grasping detection, and visual manipulation relationships; In the object detection part, if the intersection-over-union (IOU) of the predicted bounding box and the ground truth value are greater than 0.5, the predicted bounding box can be considered a correct prediction.

[0028] In the grasping detection part, the same method as in Reference 1 (Morrison D, Corke P, Leitner J. Learning robust, real-time, reactive robotic grasping[J]. The Internationaljournal of robotics research, 2020, 39(2-3): 183-201) is used to evaluate the grasping detection results of the present invention on the image; if the predicted bounding box and the ground truth box simultaneously meet the following conditions, the predicted grasping is considered to be correct: (1) the angle difference between the predicted bounding box and the ground truth value is within 30°; (2) the intersection-union ratio (IOU) between the predicted bounding box and the ground truth value is greater than 25%; In the operation relation section, if the prediction of the operation relation corresponds to the correct operation relation in the image, then the prediction is considered correct.

[0029] Experiment 1: Object detection experiment of this invention on the VMRD dataset; To verify the performance of this invention on the object detection task, it was trained and tested on the VMRD dataset. The predicted bounding boxes, predicted categories, and confidence scores of the output objects were all obtained, and the object confidence scores all exceeded 90%. In the object detection task, this invention used the final cross-modal fusion features generated by the full-flow bidirectional feature fusion backbone network (BFFB) to output the predicted bounding boxes and predicted categories. The results show that the full-flow bidirectional feature fusion backbone network can effectively improve the detection performance of objects in stacked scenes.

[0030] Experiment 2: Crawl detection experiments of this invention on the Cornell, Jacquard, and VMRD datasets; The crawling and detection experiments of this invention on the VMRD dataset: Figure 4 This is the crawling detection result of the present invention on the VMRD dataset. The present invention first generates a series of crawling candidate objects, sorts the crawling candidates according to the predicted crawling quality, and then selects the five crawling detection boxes with the highest scores and displays them in different colors, where green represents the final crawling detection boxes. Table 1: Experimental data of this invention on the VMRD dataset;

[0031] Among them, the second document is: Zhou X, Lan The third document is: Zhang H, Lan X, Zhou Reference 4 is: Yu S, Zhai DH, Xia Y. EGNet: Efficient robotic graspdetection network[J]. IEEE Transactions on Industrial Electronics, 2022, 70(4): 4058-4067.

[0032] Table 1 shows a performance comparison of the crawling detection of the present invention and existing methods on the VMRD dataset. It can be seen that the average accuracy of the present invention is 92.3%, and the processing time per image is 24 ms, representing improvements of 5.2% and 4 ms respectively compared to the EGNet method in Reference 4. The results indicate that, compared to previous methods, the present invention uses the final cross-modal fusion features generated by the full-flow bidirectional feature fusion backbone network (BFFB) to generate the crawling detection box (i.e., the predicted bounding box). This invention effectively improves the model's ability to represent details and global structure, achieving higher detection accuracy and speed.

[0033] Crawl detection experiments of this invention on the Cornell dataset: Reference Figure 5The invention is evaluated using two commonly used data segmentation methods: (i) Image-level segmentation (IW): In IW mode, the dataset is randomly segmented, allowing different images of the same object to appear in the training and test sets; (ii) Object-level segmentation (OW): In OW mode, object instances are randomly segmented to ensure that the objects in the training and test sets are completely different. OW segmentation more accurately reflects the model's generalization ability when encountering new objects.

[0034] Figure 5 This is a partial result of the grabbing detection of the object using the Cornell dataset, where green represents the final grabbing detection bounding box. It can be seen that the present invention can effectively detect the grabbing region of the object.

[0035] Table 2: Experimental data of this invention on the Cornell dataset;

[0036] Among them, document five: Kumra S, Kanan C. Robotic grasp detection using deepconvolutional neural networks[C] / / 2017 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017: 769-776; Document 6: Kumra S, Joshi S, Sahin F. Antipodal robotic grasping using generative residual convolutional neural network[C] / / 2020 IEEE / RSJInternational Conference on Intelligent Robots and Systems (IROS). IEEE, 2020: 9626-9633; Reference 7: Q. Yu, W. Shang, Z. Zhao, S. Cong and Z. Li, ”Robotic Grasping of Unknown Objects Using Novel Multilevel Convolutional Neural Networks: From Parallel Gripper to Dexterous Hand,” IEEE Transactions on Automation Science and Engineering, vol. 18, no. 4, pp. 1730-1741, 2021; Reference 8: Wang S, Zhou Z, Kan Z. When transformer meets robotic grasping: Exploits context for efficient grasp detection[J]. IEEE robotics and automation letters, 2022, 7(3): 8170-8177; Reference 9: Y. Zhang, X. Qin, T. Dong, Y. Li, H. Song, Y. Liu, Z. Li, and Q. Liu” DSNet: Double Strand Robotic Grasp Detection Network Based on Cross Attention”, IEEE Robotics and Automation Letters, vol. 9, nol. 5, pp. 4702 - 4709, 2024; Reference 10: Gao H, Zhao J, Sun C. A Real-Time Grasping Detection Network Architecture for Various Grasping Scenarios[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024.

[0037] Table 2 shows a comparison of the crawling detection performance of the present invention and some existing methods on the Cornell dataset. It can be seen that the accuracy of the present invention in IW split and OW split is 99.4% and 98.8%, respectively, which is 0.6% and 1.9% higher than that of GARDSCN. In addition, the detection speed of the present invention is 31 ms per frame, which meets the requirements of real-time detection.

[0038] Crawl detection experiments of this invention on the Jacquard dataset: Reference Figure 6 This invention presents partial crawl detection results on the Jacquard dataset. Similarly, the green candidate crawl boxes represent the final crawls.

[0039] Table 3: Experimental data of this invention on the Jacquard dataset;

[0040] Among them, document 11: S. Yu, D. -H. Zhai, Y. Xia, H. Wu and J. Liao, "SE-ResUNet: A Novel Robotic Grasp Detection Method," IEEE Robotics andAutomation Letters, vol. 7, no. 2, pp. 5238-5245, 2022; Document 12: S. Yu, D. -H. Zhai and Y. Xia, "SKGNet: Robotic GraspDetection With Selective Kernel Convolution," IEEE Transactions on AutomationScience and Engineering, vol. 20, no. 4, pp. 2241-2252, 2023.

[0041] Table 3 shows a performance comparison of the crawling detection capabilities of the present invention and some existing methods on the Jacquard dataset. It can be seen that the accuracy of the present invention is 96.8%, with a single-frame image processing time of approximately 32 ms. The accuracy is improved by 1.6% and 0.9% compared to the GARDSCN and SKGNet methods, respectively.

[0042] The crawling and detection experiments on different datasets show that the present invention uses a bidirectional feature fusion module (BMF) to effectively fuse appearance information in RGB images and geometric information in depth images, which significantly improves accuracy compared to other RGB-D-based methods.

[0043] Experiment 3: Visual operation relationship experiment of the present invention on VMRD; This invention was trained and tested on the VMRD dataset to evaluate its performance in predicting visual operational relationships.

[0044] The visual manipulation relationship prediction experiment was divided into a simple stacking scenario with 2 or 3 objects and a complex stacking scenario with 5 or 6 objects. The experimental results are shown in Table 4. The results show that the present invention can provide good results for object detection, grasping detection and visual manipulation relationship detection.

[0045] Table 4: Experimental data of this invention on the VMRD dataset;

[0046] Among them, document 13: H. Zhang, Document 14: D. Park, Y. Seo, D. Shin, J. Choi and SY Chun, "A SingleMultiTask Deep Neural Network with Post-Processing for Object Detection with Reasoning and Robotic Grasp Detection," in IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2020, pp. 7300-7306.

[0047] Table 4 shows the comparison results of the present invention and some existing methods on the VMRD dataset. Similar to Reference 4, if object detection, grasping detection, and operation relationship detection all meet the evaluation metrics, then the detection of the image can be considered a correct prediction. It can be seen that the average accuracy of the present invention is 87.5%, which is 9.3% higher than EGNet. The processing time of the present invention for a single frame image is approximately 36ms, which meets the real-time requirements.

[0048] Experiment 4: Ablation Experiment; To further verify the effectiveness of the proposed full-flow bidirectional feature fusion backbone network and bidirectional feature fusion module BMF, ablation experiments were designed. Three new networks were reconstructed—ResNet-Grasp, Eff-Grasp, and the multi-task robot grasping network of this invention—by removing these two modules respectively. The original multi-task robot grasping network of this invention was then compared with these three new networks on the Cornell, Jacquard, and VMRD datasets. ResNet-Grasp removed the full-flow bidirectional feature fusion backbone network but introduced ResNet-101 as the backbone network. Eff-Grasp removed the full-flow bidirectional feature fusion backbone network but introduced EfficientNet V2 as the backbone network. This invention removes the bidirectional feature fusion module BMF, resulting in an improved multi-task robot grasping network. The comparison results are shown in Table 5. Table 5: Ablation Experiment Data;

[0049] A comparison of ResNet-Grasp, EfficientNet-Grasp, the improved version, and the original multi-task robot grasping network shows that the full-flow bidirectional feature fusion backbone network of this invention exhibits superior performance. This is attributed to the symmetrical structural design of the full-flow bidirectional feature fusion backbone network, which integrates the bidirectional feature fusion module (BMF) and the CloFormer module. This effectively enhances the feature fusion and interaction capabilities between modalities. A comparison of the improved version and the original multi-task robot grasping network demonstrates that the original multi-task robot grasping network, employing the bidirectional feature fusion module (BMF), shows a significant performance improvement. This proves that the bidirectional feature fusion module (BMF) can effectively fuse complementary feature information between different modalities and effectively improve the overall network performance.

[0050] Experiment 5: Grasping experiment on a real robot; To further evaluate the effectiveness of this invention, we conducted grasping experiments using a CR3 robot in cluttered and stacked scenarios. The invention was first tested in a cluttered environment where the target object was a pen. This invention was used to detect the pen's position, category, and predict its bounding box. Since there were no stacked objects, the robot could directly grasp the target object based on the grasping rectangle.

[0051] The invention was subsequently tested in a stacking environment. In this experiment, the robot performed grasping operations in both simple and complex stacking scenarios. Analyzing the simple stacking scenario, the robot's target was the screwdriver. To grasp the screwdriver, the robot first needed to remove other objects from it. When multiple objects were on the target object, the robot would remove the object with the higher index value first. However, when there was no stacking relationship between the objects, the robot would grasp them randomly. Finally, the servo, pen, and screwdriver were identified and successfully grasped by the robot in that order.

[0052] Table 6: Results of grasping experiments with real robots;

[0053] Furthermore, this invention conducted 120 grasping experiments in both cluttered and stacked scenarios, and calculated the success rate. A grasp was defined as successful if the robot could accurately grasp the target object according to the operation relationship tree. The average success rate of the grasping experiments is shown in Table 6. Compared with previous methods, this invention achieves a higher grasping success rate.

[0054] In another aspect, the present invention provides a multi-task robot grasping system, including a device end, which is configured to execute the above-described multi-task robot grasping method.

[0055] Reference Figure 3 The device includes a deep learning platform and a robot gripping platform electrically connected to the deep learning platform. The robot gripping platform includes a dual-arm robot, a WiFi module, a PC device, and a visual perception device. The dual-arm robot has robotic arms installed on both its left and right sides, and the ends of the robotic arms are equipped with jog grippers. The PC device is electrically connected to the dual-arm robot, the jog grippers, and the visual perception device via the WiFi module. The visual perception device includes a webcam, a RealSense camera, a Kinect camera, and a spectral camera.

[0056] The deep learning platform is an NVIDIA GeForce RTX 3090Ti; Ubuntu 18.04LTS is installed on the NVIDIA GeForce RTX 3090Ti; the deep learning platform is used to train the multi-task robot grasping network and store the trained multi-task robot grasping network.

[0057] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-task robot grasping method for object stacking scenarios, characterized in that, Includes the following steps: S1. Collect multiple RGB images and multiple corresponding depth images of a scene containing multiple stacked objects; use the RGB images and corresponding depth images as image pairs to construct a dataset; S2. Construct a multi-task robot grasping network. The multi-task robot grasping network includes a full-flow bidirectional feature fusion backbone network, a target detection module, a grasping detection module, and a visual operation relationship prediction module. The output of the full-flow bidirectional feature fusion backbone network is connected to the grasping detection module and the target detection module, respectively. The output of the target detection module and the output of the full-flow bidirectional feature fusion backbone network are connected to the input of the visual operation relationship prediction module, respectively. S3. Input the dataset into the multi-task robot grasping network. The grasping detection module, target detection module, and visual operation relationship prediction module output the grasping detection result, target detection result, and target operation relationship result, respectively. Calculate the total loss based on the grasping detection result, target detection result, and target operation relationship result, and adjust the multi-task robot grasping network according to the total loss until the total loss is minimized to obtain the trained multi-task robot grasping network. S4. Deploy the trained multi-task robot grasping network into the multi-task robot grasping system, and use the trained multi-task robot grasping network to perform grasping detection in a multi-object stacking scenario to obtain the detection results and operation relationship tree. The full-flow bidirectional feature fusion backbone network in S2 includes three layers of full-flow bidirectional feature fusion units connected in sequence. Each layer of full-flow bidirectional feature fusion unit includes two first convolutional layers and a bidirectional feature fusion module (BMF). The outputs of the two first convolutional layers are respectively connected to the bidirectional feature fusion module (BMF). The grasping and detection module includes three second convolutional layers and three 1×1 convolutional layers. The output terminals of the three second convolutional layers are connected in sequence and then connected to the three 1×1 convolutional layers respectively. The target detection module includes two third convolutional layers, one first average pooling layer, and two first fully connected layers. The output of the two third convolutional layers is connected in sequence and then connected to the first average pooling layer. The output of the first average pooling layer is connected to the two first fully connected layers respectively. The bidirectional feature fusion module (BMF) adopts a symmetrical design, including two 1×1 convolutional layers and two multi-scale channel attention modules (MS-CAM). The input of one of the 1×1 convolutional layers is connected to the two first convolutional layers on the corresponding full-flow bidirectional feature fusion unit through a concatenation operation. The output is connected to the input of one of the multi-scale channel attention modules MS-CAM. The output of the multi-scale channel attention module MS-CAM is connected to the two first convolutional layers through a concatenation operation, and then connected to the input of another multi-scale channel attention module MS-CAM through a summation operation. The output of the multi-scale channel attention module MS-CAM is connected to the two first convolutional layers through a summation operation, and then connected to another 1×1 convolutional layer.

2. The multi-task robot grasping method for object stacking scenarios according to claim 1, characterized in that, S3 specifically includes the following steps: S31. First, select RGB images and corresponding depth images from the dataset and input them into the full-flow bidirectional feature fusion backbone network to obtain the final cross-modal fusion features of each object in the multi-object stacking scene. S32. Input the final cross-modal fusion features obtained in S31 into the grasping and detection module to obtain the grasping and detection results; S33. Input the final cross-modal fusion feature obtained in S31 into the object detection module. After the convolution operation of the two third convolutional layers in the object detection module, the feature dimension is reduced by the first average pooling layer. Finally, the predicted bounding box and predicted category of each object in the multi-object stacking scene are output through the two first fully connected layers respectively. S34. Input the predicted bounding box of each object and the final cross-modal fusion features into the visual operation relationship prediction module to perform operation relationship prediction and obtain the operation relationship tree. S35. Construct a total loss function based on the grasping detection results, target detection results, operation relationship tree, and real data. Adjust the parameters of the multi-task robot grasping network based on the total loss calculated by the total loss function. S36. Determine whether the iteration stopping condition has been met. If so, output the trained multi-task robot grasping network. Otherwise, loop from S31 to S36 until the iteration stopping condition is met, and output the trained multi-task robot grasping network.

3. The multi-task robot grasping method for object stacking scenarios according to claim 2, characterized in that, S31 specifically includes the following steps: S311. First, the input features of RGB images and corresponding depth images are selected from the dataset and input into the first layer of the full-flow bidirectional feature fusion backbone network. After passing through two first convolutional layers, the feature vector is obtained. , , to feature vector , The input is fed into the bidirectional feature fusion module (BMF) to obtain the cross-modal fusion features of the first layer. The specific formula is as follows: ； in, This represents the first convolutional layer; , These represent RGB images and depth images, respectively. S312, Fuse the cross-modal features of the first layer The input is fed into the full-flow bidirectional feature fusion unit of the second layer, and after passing through two first convolutional layers, it is fed into the bidirectional feature fusion module (BMF) again to obtain the cross-modal fusion features of the second layer. S313. The cross-modal fusion features of the second layer are input into the full-flow bidirectional feature fusion unit of the third layer. After passing through two first convolutional layers, they are input into the bidirectional feature fusion module BMF again to obtain the final cross-modal fusion features of each object in the multi-object stacking scene.

4. A multi-task robot grasping method for object stacking scenarios according to claim 3, characterized in that, In S311, the feature vector , The input is fed into the bidirectional feature fusion module (BMF) to obtain the fused features of the first layer. Specifically, the steps include the following: S3111, The feature vector , The concatenation operation is performed along the channel dimension, and then the data is input into a 1×1 convolutional layer 2 for cross-channel feature combination, thereby obtaining preliminary fused features. The specific formula is expressed as follows: ； in, This represents a 1×1 convolutional layer 2; Indicates a splicing operation; S3112, Subsequently, the initial fusion features will be... The input is fed into the multi-scale channel attention module MS-CAM, and then compared with the feature vectors respectively. , Perform a stitching operation along the channel dimension to obtain the fused features. , The formula is as follows: ； S3113, Integrating Features , After performing element-wise addition, the data is input into the next multi-scale channel attention module (MS-CAM) to obtain the fused features. The formula is as follows: ； in, This indicates an element-wise addition operation; S3114, The feature vector , and fusion features The fused features are obtained after performing element-wise addition. The formula is as follows: ； S3115, and then merge the features The input is fed into a 1×1 convolutional layer 2 to obtain the cross-modal fusion features of the first layer. The formula is as follows: 。 5. A multi-task robot grasping method for object stacking scenarios according to claim 4, characterized in that, S32 specifically includes the following steps: S321. Input the final cross-modal fusion features obtained in S31 into the grasping detection module. The feature dimension is restored to the same dimension as the input features of the RGB image and depth image through three second convolutional layers. Then, they are input into three 1×1 convolutional layers to obtain the grasping quality pixel map, grasping angle pixel map and grasping width pixel map respectively. S322. Combine the capture quality pixel map, capture angle pixel map, and capture width pixel map to obtain the capture detection result of the topmost object in the object stacking scene, expressed by the following formula: ； in, This indicates the capture and detection result of the topmost object in a scene of stacked objects; These represent the pixel image of the capture quality, the pixel image of the capture angle, and the pixel image of the capture width, respectively. Represents the set of real numbers; H , W These represent the height and width of the image, respectively. S323. Map the angle information in the capture angle pixel image to two trigonometric functions, with the angle output range extending to [-π / 2, π / 2]. Finally, integrate them to obtain the capture angle scalar value for each pixel in the capture angle pixel image, expressed by the following formula: ； in, Π represents the scalar value of the capture angle for each pixel in the capture angle pixel map; Π represents a definite natural number.

6. A multi-task robot grasping method for object stacking scenarios according to claim 5, characterized in that, S34 specifically includes the following steps: S341. The predicted bounding box of each object and the final cross-modal fusion features are input into two sets of convolutional layers, each set of convolutional layers includes two fourth convolutional layers connected in sequence, to extract the object features of each object. S342. Reduce the feature dimension of each object's features using two second average pooling layers, and finally predict the operation relationship using two second fully connected layers to obtain the operation relationship tree.

7. A multi-task robot grasping method for object stacking scenarios according to claim 6, characterized in that, The formula for calculating the total loss function in S35 is as follows: ； in, Indicates the total loss; This represents the associated loss for predicting the bounding box and the predicted category; Indicates the loss detected; Indicates the predicted loss based on operational relationships; In the formula, the relevant loss for predicting the bounding box and the predicted category is... The calculation formula is as follows: ； in, It is the scaling factor for this loss during training; It is a category of network prediction. It is the ground truth value of the category; It is the location of the bounding box predicted by the network; It is the ground truth value of the bounding box; and It is a normalized project; Represents the cross-entropy loss function; It is smooth loss; In the formula, the capture detection loss The calculation formula is as follows: ； in, n It is the total number of pixels in the image. , and These are the pixel images of capture quality, capture angle, and capture width, respectively, corresponding to the [missing information]. i The value of each pixel. , and These are the real labels pre-labeled in the dataset; , and It is the proportional factor for each loss during training; In the formula, the operational relationship predicts the loss. The calculation formula is as follows: ； in, This represents the total number of all possible object pairs; The object detection module predicts possible object pairs. It is the ground truth value.

8. A multi-task robot grasping system, characterized in that, Includes a device that is configured to perform the multi-task robot grasping method according to any one of claims 1 to 7.

Images