A transparent object grasping method and system based on 6D pose estimation

Abstract

A method and system for grasping transparent objects based on 6D pose estimation, comprising: 1. inputting an original input image containing a transparent object into a target detection module to obtain a 2D bounding box and confidence score of the target object; 2. extracting target features from the preprocessed original input image and performing similarity matching with features in a reference image library to obtain a reference image with the highest similarity; 3. generating a preliminary 6D pose of the target feature, then refining and adjusting the preliminary 6D pose to obtain a final 6D pose of the target feature; 4. dividing the path between the robotic arm and the target feature into three segments, converting the segment into a polynomial interpolation function, and solving the polynomial interpolation function using a particle swarm optimization algorithm to obtain an optimized trajectory; 5. the robotic arm grasping the target feature along the optimized trajectory. The present invention improves the accuracy of 6D pose estimation for transparent objects, resulting in higher precision in grasping transparent objects.

Description

Technical Field

[0001] The present invention relates to the technical field of target grasping, and in particular to a transparent object grasping method and system based on 6D pose estimation. Background Art

[0002] In recent years, with the improvement of industrial automation, computer vision, as an important research field of artificial intelligence, has been widely used in various industries. Among them, vision-based robotic arm grasping has gradually become a current research hotspot.

[0003] Machine vision can also be unable to accurately identify and locate certain special objects, such as transparent ones. Sajjan et al. proposed a two-stage method for depth recovery of transparent objects. This method first estimates surface normals, occlusion boundaries, and segmentation from RGB images, and then calculates the refined depth through global optimization. However, optimization is very time-consuming and heavily dependent on previous network predictions. Transparent objects lack obvious color or texture features and cannot be accurately identified.

[0004] In addition, vision-based robotic grasping is also a current research hotspot, but it is not mature for grasping transparent objects. ChenWang et al. proposed a network to estimate the 6D pose of an object from an RGB-D image. The object pose is directly predicted based on the RGB-D image information. Its distinctive features include dense prediction, confidence setting and a new pose iterative optimization algorithm. However, there is a certain limitation, that is, the generalization ability is limited. It can only accurately predict the trained targets that have been seen, but it is difficult to guarantee the accuracy of the prediction results for new targets that have never been seen.

[0005] In summary, robotic grasping based on 6D pose estimation has made significant progress. However, 6D pose estimation and grasping still face great challenges when facing transparent and textureless target objects. Summary of the Invention

[0006] The present invention provides a transparent object grasping method and system based on 6D pose estimation to solve the technical problems mentioned in the background technology.

[0007] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0008] The present invention provides a transparent object grasping method based on 6D pose estimation, comprising:

[0009] S1. Input the original input image containing transparent objects into the TOG-Net target detection module to obtain the category, 2D bounding box and confidence score of the target object in the original input image;

[0010] S2, extracting target features from the preprocessed original input image based on the target object category, 2D bounding box, and confidence score, and performing similarity matching with the features in the reference image library to obtain the reference image with the highest similarity;

[0011] S3. Estimate the rotation and translation information of the plane where the target feature is located based on the reference features in the reference image with the highest similarity, generate a preliminary 6D pose of the target feature, and then refine and adjust the preliminary 6D pose of the target feature to obtain the final 6D pose of the target feature;

[0012] S4, dividing the path between the robot arm and the target feature into three segments, converting the three segments into polynomial interpolation functions, and solving the polynomial interpolation functions using the particle swarm optimization algorithm to obtain the optimized trajectory;

[0013] S5. The robotic arm grasps the target feature according to the optimized trajectory.

[0014] On the other hand, the present invention also provides a transparent object grasping system based on 6D pose estimation, including a robotic arm, a visual perception device, an electric gripper and a built-in PC unit. The visual perception device and the electric gripper are both installed at the end of the robotic arm, and the built-in PC unit is installed below the robotic arm. The robotic arm, the visual perception device and the electric gripper are all electrically connected to the built-in PC unit; the robotic arm grasps the transparent object according to the transparent object grasping method.

[0015] Beneficial effects of the present invention:

[0016] 1. The present invention discloses a transparent object grasping method based on 6D pose estimation, which uses a hybrid encoder. The hybrid encoder includes an intra-scale feature interaction module AIFI and a cross-scale fusion module CCFM connected in sequence. The intra-scale feature interaction module AIFI achieves more efficient feature representation capabilities by recombining different parts of the input feature map, and the cross-scale fusion module CCFM achieves more efficient feature representation capabilities by learning local and global contexts.

[0017] 2. The transparent object grasping method based on 6D pose estimation disclosed in the present invention uses a method combining a particle swarm optimization algorithm with a polynomial interpolation function to provide a smooth and continuous trajectory for the robotic arm, avoiding sudden changes, violent acceleration or jitter in joint movement. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a logic block diagram of the transparent object grasping method of the present invention;

[0019] Figure 2 This is a structural block diagram of the backbone network in the TOG-Net target detection module of the present invention;

[0020] Figure 3 It is a structural block diagram of the ConvNeXt network in the backbone network of the present invention;

[0021] Figure 4 This is a structural block diagram of the cross-scale fusion module CCFM in the present invention;

[0022] Figure 5 This is a physical diagram of the transparent object grasping system in an embodiment of the present invention;

[0023] Figure 6 1 is a comparative example diagram of different 6D pose estimation methods and the present method in an embodiment of the present invention;

[0024] Figure 7 Graphs showing the unoptimized position, velocity, and acceleration of each joint in the trajectory planning experiment according to an embodiment of the present invention;

[0025] Figure 8 Graphs showing the optimized position, velocity, and acceleration of each joint in the trajectory planning experiment according to an embodiment of the present invention;

[0026] Figure 9 A motion trajectory curve diagram of the end of the robotic arm in the trajectory planning experiment in an embodiment of the present invention;

[0027] Figure 10 This is a convergence curve diagram of the robot arm joint 1 in the trajectory planning experiment in an embodiment of the present invention;

[0028] Figure 11 This is a convergence curve diagram of the robot arm joint 2 in the trajectory planning experiment in an embodiment of the present invention;

[0029] Figure 12 This is a convergence curve diagram of the robot arm joint 3 in the trajectory planning experiment in an embodiment of the present invention;

[0030] Figure 13 This is a convergence curve diagram of the robot arm joint 4 in the trajectory planning experiment in an embodiment of the present invention;

[0031] Figure 14 These are example diagrams of target capture under different lighting environments in the embodiments of the present invention, where (a) is an example diagram of unobstructed capture under normal lighting; (b) is an example diagram of unobstructed capture under dark lighting; (c) is an example diagram of normal lighting obstacle capture; and (d) is an example diagram of dark lighting obstacle capture. DETAILED DESCRIPTION

[0032] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present invention. However, the present invention may be implemented in many other forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present disclosure.

[0033] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0034] Reference Figure 1 , an embodiment of the present application provides a transparent object grasping method based on 6D pose estimation, comprising:

[0035] S1. Input the original input image containing transparent objects into the TOG-Net object detection module to improve the diversity of features, thereby obtaining the category, 2D bounding box and confidence score of the target object in the original input image;

[0036] S2, extracting target features from the preprocessed original input image based on the target object category, 2D bounding box, and confidence score, and performing similarity matching with the features in the reference image library to obtain the reference image with the highest similarity;

[0037] S3. Estimate the rotation and translation information of the plane where the target feature is located based on the reference features in the reference image with the highest similarity, generate a preliminary 6D pose of the target feature, and then refine and adjust the preliminary 6D pose of the target feature to obtain the final 6D pose of the target feature;

[0038] S4, dividing the path between the robot arm and the target feature into three segments, converting the three segments into polynomial interpolation functions, and solving the polynomial interpolation functions using the particle swarm optimization algorithm to obtain the optimized trajectory;

[0039] S5. The robotic arm grasps the target feature according to the optimized trajectory.

[0040] In some embodiments, the TOG-Net target detection module includes a backbone network, a network layer, and a prediction layer connected in sequence;

[0041] Reference Figure 2The backbone network includes a convolutional sampling layer and four stage units connected in series. The four stage units are the first stage unit to the fourth stage unit. The first stage unit includes the LN normalization layer and the ConvNeXt network (a computer vision model) connected in series. The second stage unit to the fourth stage unit are all downsampling layers and ConvNeXt networks connected in series. The specific structure of the ConvNeXt network is referenced. Figure 3 As shown in the figure, it specifically includes a sequentially connected depthwise separable convolutional layer, two convolutional sampling layers, and a layer scaling layer. The ConvNeXt network significantly improves local feature extraction capabilities by improving traditional convolutional layers (CNNs). The depthwise separable convolutional layers in the ConvNeXt network can more effectively identify the contours and surface characteristics of transparent objects, capturing the reflections of glass or the refraction patterns of transparent materials through high-order feature layers.

[0042] The network layer is a hybrid encoder, which can achieve cross-channel and cross-scale fusion on the basis of retaining more information, reducing unnecessary redundant calculations; the hybrid encoder includes an intra-scale feature interaction module AIFI (intra-scale interaction) and a cross-scale fusion module CCFM (cross-scale fusion) connected in sequence; the input of the intra-scale feature interaction module AIFI is the output of the fourth stage unit, i.e., the feature map The output of the intra-scale feature interaction module AIFI is the feature map The input of the cross-scale fusion module CCFM is the output of the second stage unit to the fourth stage unit, which are feature maps , feature map , feature map , expressed by the formula, as follows:

[0043] ;

[0044] in, Represents the features after fusion by the cross-scale fusion module CCFM;

[0045] Reference Figure 4 As shown in the figure, the cross-scale fusion module CCFM inserts a fusion block composed of multiple convolutional layers into the fusion path and fuses the features of two adjacent scales into a new feature. Specifically, the cross-scale fusion module CCFM contains two 1×1 convolutional layers, two 1×1 convolutional layers to adjust the number of channels, and uses N RepBlocks composed of RepConv for feature fusion, and fuses the outputs of the two paths by element-wise addition.

[0046] The Cross-Scale Fusion Module (CCFM) improves the network's detection accuracy for transparent objects by adaptively fusing feature information across scales and channels. Specifically for transparent object detection, the CCFM utilizes multi-scale feature pyramid fusion to help the model better capture details of transparent objects at different scales. It also incorporates a channel attention mechanism to enhance the representation of important channel features, effectively suppressing background interference and improving the recognition of transparent objects.

[0047] In some embodiments, the step S1 specifically includes the following steps:

[0048] S11. First, input the original image of the transparent object Input into the convolution sampling layer in the backbone network for multi-level feature extraction to obtain the initial feature map , expressed by the formula, as follows:

[0049] ;

[0050] in, It is a 4×4 convolution operation; represents the real three-dimensional space, are the height and width of the image respectively; Indicates the number of channels of the convolution sampling layer;

[0051] S12, the initial feature map Input into four stage units, perform multi-stage multi-scale feature extraction, and obtain multiple feature maps, namely feature maps , feature map , feature map ;in, When, The multi-scale feature operation of the stage unit is expressed by the formula as follows:

[0052] ;

[0053] in, Indicates the Feature maps obtained by multi-scale feature operations of stage units; It is The number of channels of the stage unit; Represents the set of operations of the LN normalization layer and the ConvNeXt network, or the set of operations of the downsampling layer and the ConvNeXt network;

[0054] S13, feature map Flatten to a one-dimensional vector to generate the query matrix , key matrix Sum Matrix The input data is input into the scale feature interaction module AIFI, the input data is processed by the scale feature interaction module AIFI, and then the Reshape operation is used to restore the feature map. The same shape, get the feature map , expressed by the formula, as follows:

[0055] ;

[0056] in, Represents a flattening operation; represents the shape recovery operation, i.e. The inverse operation of

[0057] S14, feature map , feature map And feature maps All are input into the cross-scale fusion module CCFM to obtain multi-scale features;

[0058] S15. Input the multi-scale features into the prediction layer to obtain the category, 2D bounding box and confidence score of the target object in the input image.

[0059] In some embodiments, the step S2 specifically includes the following steps:

[0060] S21, preprocessing the original input image to obtain a preprocessed input image;

[0061] S22. Based on the category, 2D bounding box, and confidence score output by the TOG-Net target detection module, the corresponding target region is cropped from the preprocessed input image to obtain a query image. Then, multiple features are extracted within the corresponding target region (i.e., the query image) to obtain extracted target features.

[0062] S23, matching the extracted target features with the features in the reference image library, calculating the similarity scores between the target features and the reference features using a similarity metric, and selecting the top N reference images with the highest similarity scores as candidate viewpoints;

[0063] The similarity score calculation process is as follows: First, the initial similarity score is calculated using the following formula;

[0064] ;

[0065] in, is the initial similarity score; Represents the initial similarity score calculation operation; For two features, the initial similarity score needs to be calculated;

[0066] Then the final similarity score is calculated based on the initial similarity score. The calculation formula of the final similarity score is as follows:

[0067] ;

[0068] in, represents the final similarity score, represents the weight coefficient; Represents the refraction path consistency score based on ray tracing; represents the query image; Indicates the Reference images.

[0069] S24. Introduce refraction path consistency as an additional selection criterion, calculate the refraction path differences between the preprocessed input image and N reference images, and select the viewpoint with the smallest difference. The reference image corresponding to the viewpoint with the smallest difference is the reference image with the highest similarity.

[0070] In some embodiments, S3 specifically includes the following steps:

[0071] S31, based on the reference features in the reference image with the highest similarity, estimate the rotation and translation information of the plane where the target feature is located, combine the rotation and translation information, and generate a preliminary 6D pose of the target feature;

[0072] S32, training the posture refiner to obtain a trained posture refiner;

[0073] Specifically, to train the pose refiner, 323 voxel points are sampled in the unit cube of the target coordinate system. These voxel points are then transformed to the input camera coordinate system using the input pose. The loss is the difference between the distance of the sample point transformed by the ground truth similarity transformation and the distance of the sample point transformed by the predicted similarity transformation;

[0074] ;

[0075] in is the sampling point coordinate in the input camera coordinate system, represents the set of real numbers, and are the predicted scale and the true scale, and are the predicted rotation and the true rotation, respectively. and are the predicted 2D translation and the actual 2D translation, is the error term based on light refraction, Weight coefficient. This can be calculated by simulating the refraction path of light on the surface of a transparent object. Specifically, a ray tracing algorithm can be used to calculate the difference between the refraction path under the predicted pose and the actual pose, and this difference is used as an error term. This ensures that the final 6D pose estimation not only considers geometric transformations but also incorporates the optical properties of transparent objects, achieving more accurate and robust pose prediction.

[0076] S33. Use the trained pose refiner to adjust the preliminary 6D pose to obtain the output of the pose refiner, that is, the final 6D pose of the target feature.

[0077] In some embodiments, the output of the posture refiner in S33 is expressed by a formula as follows:

[0078] ;

[0079] in, represents the output of the pose refiner, i.e., the final 6D pose of the target feature; Represents feature extraction operation; represents the initial 6D pose; Represents the 2D bounding box of the target object in the original input image.

[0080] Specifically, the pose refiner adjusts the initial 6D pose as follows:

[0081] First, the bounding box detected from the original input image The features are extracted and expressed by the formula as follows:

[0082] ;

[0083] in, Represents feature extraction operation;

[0084] Then, based on the features extracted in the previous step, the initial 6D pose is optimized to obtain the pose correction value. , expressed by the formula, as follows:

[0085] ;

[0086] in, Indicates optimization operation;

[0087] Finally, according to the attitude correction The final 6D pose of the target feature is calculated and expressed using the formula as follows:

[0088] .

[0089] In some embodiments, the S4 specifically includes the following steps:

[0090] S41, taking the initial position of the manipulator as the starting point and the final 6D pose of the target feature as the intermediate end point; adding two path points between the starting point and the end point, thereby dividing the path between the manipulator and the target feature into three segments, namely the first segment to the third segment;

[0091] S42. Using a cubic polynomial interpolation function in the first and third segments, and using a quintic polynomial interpolation function in the second segment, thereby completing the construction of the polynomial interpolation function;

[0092] S43. A particle swarm optimization algorithm is used to solve the constructed polynomial interpolation function to obtain an optimized trajectory.

[0093] In some embodiments, the cubic polynomial interpolation of the first and third segments, and the quintic polynomial interpolation of the second segment in S42 are specifically as follows:

[0094] ;

[0095] ;

[0096] ;

[0097] in, 、 and The robot arm The first segment of cubic polynomial, the second segment of quintic polynomial and the third segment of cubic polynomial trajectory of each joint; 、 、 、 Respectively The 3rd, 2nd, 1st, and 0th coefficients of the interpolation function of the first trajectory of each joint; 、 、 、 、 、 Respectively The 5th, 4th, 3rd, 2nd, 1st, and 0th coefficients of the interpolation function of the second trajectory of each joint; 、 、 、 Respectively The 3rd, 2nd, 1st and 0th coefficients of the interpolation function of the third trajectory of each joint; 、 、 Respectively The time of the first segment of cubic polynomial, the second segment of quintic polynomial and the third segment of cubic polynomial trajectory corresponding to each joint.

[0098] In some embodiments, the S43 specifically includes the following steps:

[0099] S431, initialize the particle swarm, generate the position and velocity of each particle, and divide the sub-population, then set the maximum number of iterations D, initialize the individual optimal value and the global optimal value;

[0100] S432, update inertia weight , group learning factor and self-learning factors , construct particle trajectory equation; inertia weight The calculation formula is as follows:

[0101] ;

[0102] in, 、 Inertia weight The maximum and minimum values of ; k represents the number of iterations;

[0103] Group learning factor and self-learning factors The calculation formula is as follows:

[0104] ;

[0105] ;

[0106] S433. Deriving the particle trajectory equation to obtain the velocity and acceleration equations of each particle, and then calculating the velocity and acceleration of each particle using the velocity and acceleration equations of each particle;

[0107] S434, determine whether the velocity and acceleration of each particle meet the constraint conditions; if not, set the particle fitness value to infinity until the conditions are met; if all conditions are met, enter S435;

[0108] S435, based on self-learning factors Calculate the fitness and update the individual optimal value based on the fitness; then based on the inertia weight , individual optimal value and group learning factor Update the global optimal value;

[0109] S436. Determine whether the current number of iterations reaches the maximum number of iterations D. If so, output the global optimal value, which is the optimized trajectory. If not, return to S432.

[0110] The following is a comparison and verification of the grasping effect of a transparent object grasping method based on 6D pose estimation provided by the present invention by combining various different grasping methods:

[0111] First, the hardware used in the present invention is described in detail, including a 6D pose grasping system of a robotic arm. The 6D pose grasping system of the robotic arm mainly includes a deep learning platform and a transparent object grasping system including a robotic arm. The GPU of the deep learning platform is NVIDIA Geforce RTX2080ti, and the built-in software environment is Ubuntu18.04 LTS system. The transparent object grasping system is a DobotCR series CR3 intelligent robot of Yuejiang Technology. The main components include: Realsense D435i depth camera, EPG50-060 series electric parallel gripper, NUC onboard microcomputer, etc. The transparent object grasping system is as follows: Figure 5 shown.

[0112] Due to the light transmittance of transparent objects, the Trans10K dataset only has plane-level label information such as 2D bounding boxes or masks, so it is only suitable for target detection and instance segmentation; the ClearGrasp dataset contains deep label information for transparent objects, which is mainly used for depth estimation, but still lacks specific 6D pose labels, so it is still not suitable for 6D pose estimation of transparent objects. Existing generalizable pose estimators either require high-quality object models or require additional depth maps or object masks during testing, which greatly limits the scope of application. Therefore, the present invention adjusts the strategy for making datasets, and does not need to consider the complex reflection and refraction phenomena of transparent objects. Only some pose images of the target object are required, and the dataset GLASS8K is made for transparent objects. In addition, the images captured by the dataset GLASS8K are not cropped, and only a single transparent object target is retained in scenes with other interferences, which is closer to the real scene than other datasets.

[0113] In order to test the effectiveness of the present invention and the dataset produced under different material objects, a transparent object was placed among a variety of objects including three interference objects and without occlusion. A 6D pose estimation comparison experiment was conducted on the self-made dataset GLASS8K using the PVNet algorithm, Pi2Pose algorithm, 6-PACK algorithm and the method provided by the present invention. Figure 6 As shown:

[0114] Figure 6It can be seen that under normal lighting, the pose prediction effects of the 6-PACK algorithm and the Pi2Pose algorithm are poor, and large-scale pose offsets occur. Obviously, the depth problem of transparent objects affects the pose refinement of the ICP algorithm. Secondly, the PVNet algorithm predicts the target pose better in the 12th and 43rd frames, but there are large offsets in the 80th and 56th frames. The last row shows the results of the method provided by the present invention. It can be seen that the method provided by the present invention is better and more accurate than the first three methods under different viewing angles.

[0115] The following quantitative analysis experiments are conducted by combining multiple different data sets and multiple different crawling methods:

[0116] The quantitative analysis is divided into ablation experiments of target detection algorithms and comparative experiments with different algorithms on the GLASS8K dataset, Trans10K dataset, and ClearGrasp dataset.

[0117] TOG-Net target detection module ablation experiment:

[0118] To verify the effectiveness of adding the ConvNeXt network and the cross-scale fusion module CCFM, the method provided by the present invention is compared with four algorithms using the YOLOV11 algorithm as a prototype:

[0119] 1) Prototype YOLOV11 backbone network (C3K2);

[0120] 2) In the YOLOV11 framework, FasterNet is used to replace the backbone network;

[0121] 3) Add a lightweight cross-scale feature fusion module CCFM to the YOLOV11 framework;

[0122] 4) In the TOG-Net target detection module provided by the present invention, the backbone network is replaced by the ConvNeXt network, and a lightweight cross-scale feature fusion module CCFM is added.

[0123] In order to quantitatively analyze the performance of target detection, several common evaluation indicators are used, such as precision, recall, and detection accuracy. The results are shown in Table 1:

[0124] Table 1: TOG-Net target detection module ablation experiment

[0125]

[0126] As shown in Table 1, the performance of the prototype YOLOV11 backbone network and the YOLOV11 framework using the ConvNeXt network to replace the backbone network was first compared. With little change in the accuracy, recall and average precision indicators, the number of parameters of the model dropped significantly from 5.10M to 4.00M, indicating that the computational complexity was significantly reduced. Secondly, the effects of adding a lightweight cross-scale fusion module CCFM and a TOG-Net target detection module to the YOLOV11 framework were compared. The results show that with only an increase of 0.02M parameters, the accuracy of the TOG-Net target detection module is improved by about 1.2% compared with YOLOV11; among them, the detection accuracy of transparent objects is significantly improved. Therefore, under the condition that the GPU computing resources of the grasping platform are limited, the TOG-Net target detection module proposed in this invention is more suitable for application in robotic arm grasping tasks.

[0127] Pose estimation experiment:

[0128] In order to quantitatively analyze the method provided by the present invention, comparative analysis experiments were conducted on the PVNet algorithm, Pi2Pose algorithm, 6-PACK algorithm and the method provided by the present invention on the GLASS8K dataset, Trans10K dataset and ClearGrasp dataset, as shown in Table 2.

[0129] Table 2: Quantitative evaluation of 6D pose on the GLASS8K dataset

[0130]

[0131] Table 2 shows that the average object prediction accuracy of the proposed method reaches 95.6%, which is 6.4%, 6.9%, and 2.8% higher than the PVNet, Pi2Pose, and 6-PACK algorithms, respectively. The detection accuracy of object 3 is lower than that of the 6-PACK algorithm. This is mainly due to the fact that the surface texture of the water bottle is relatively high, while the other objects are relatively transparent.

[0132] Next, some objects were selected from the Trans10K dataset and the method provided by the present invention was compared with the PVNet algorithm, Pi2Pose algorithm and 6-PACK algorithm respectively. The comparison results are shown in Table 3.

[0133] Table 3: Quantitative evaluation of 6D pose on the Trans10K dataset

[0134]

[0135] As shown in Table 3, the method provided by the present invention has little improvement compared with other algorithms. The reason is that the research objects of the Trans10K dataset are mostly other items and do not contain many transparent objects. Therefore, they do not have the characteristics of transparent objects that will produce erroneous depth.

[0136] Table 4: Quantitative evaluation of 6D pose on the ClearGrasp dataset

[0137]

[0138] The method provided by the present invention was compared with the PVNet algorithm, the Pi2Pose algorithm, and the 6-PACK algorithm on the ClearGrasp dataset. The results are shown in Table 4. When the target object is transparent, the method provided by the present invention significantly improves the accuracy of PVNet by 13.5% and 6.2% compared to 6-PACK. For transparent object pose estimation, the test results show that the accuracy of the method provided by the present invention is significantly improved.

[0139] Robotic arm trajectory planning experiment:

[0140] To verify the robotic arm grasping method based on piecewise polynomial trajectory planning combined with particle swarm optimization and joint space planning, this embodiment is simulated and verified by MATLAB Robotics Toolbox and Optimtool toolbox.

[0141] Using the improved power function operator, add it to the inertia weight, and based on the total number of iterations, let each particle expand the search space during the search to increase the population diversity. The operating parameters of the basic particle swarm algorithm are set as follows: Group learning factor The value is 1.5, the self-learning factor The value is 1.7, the population size S pop The value is 20, the population dimension is 10, and the experiment is run 20 times, with 500 iterations each time. The optimal trajectory planning is as follows: Figures 7 to 13 As shown:

[0142] Reference Figure 7 and Figure 8As shown in the figure, during the manipulator's motion, each joint remains stable, with smooth and non-aggressive changes in angular velocity and angular acceleration, while also satisfying the constraints of each joint. After optimization using the improved particle swarm optimization algorithm, the position of joint 1 converged after 34 iterations, with the minimum times required to pass through the three polynomial trajectories being 0.6385s, 0.7162s, and 0.446s, respectively, which is significantly less than the total time required by the basic particle swarm optimization algorithm. The position of joint 2 converged after 30 iterations, while the positions of joints 3 and 4 converged after 15 and 6 iterations, respectively, requiring significantly fewer iterations than the basic PSO algorithm. These results demonstrate that this approach can effectively meet the manipulator's grasping needs in complex scenarios.

[0143] Robotic arm grasping experiment:

[0144] Considering the impact of transparent objects under different lighting conditions, this experiment mainly discusses the robotic arm grasping in complex scenes under the influence of lighting changes and obstacles. The grasped objects mainly use glass bottle reagents and other transparent objects in the GLASS8K dataset. In order to verify the performance of the method provided by the present invention in practical applications, this experiment conducted two groups of grasping: the first group: the robotic arm grasped in normal lighting and dark lighting environments without obstacles; the second group: the robotic arm grasped in normal lighting and dark lighting environments with obstacles. First, the depth camera detects the object and the robotic arm grasps the object, and then performs the next grasp.

[0145] Table 5: Robotic arm grasping success rate in different environments (%)

[0146]

[0147] As shown in Table 5, in the absence of obstacles, the average grasping success rate of the robotic arm under normal lighting conditions is 88.7%, while the average grasping success rate under dim lighting conditions is only 79%, 9.7% lower than under normal indoor lighting conditions. Specifically, the image clarity captured by the camera in dim lighting conditions is low, hindering feature extraction. Therefore, the grasping rate is higher under normal lighting conditions. Secondly, in the presence of obstacles, the average grasping success rate under normal lighting conditions is 85.7%, only 3% lower than in the absence of obstacles, demonstrating the effectiveness of the proposed 3-5-3 particle swarm optimization algorithm.

[0148] In addition, the results in the figure show that under normal lighting conditions, the success rate of grasping transparent glass is high. Specifically, the recognition of the pose is not affected by the reflection of light on the glass, but the detection of the depth camera is affected by the lighting, resulting in a low grasping rate in a dark environment. In a dark environment, grasping failures may occur. Figure 14As shown in the figure, combined with the obstruction of the object, the entire target object lacked clear texture and features, ultimately leading to grasp failure. Another factor contributing to grasp failure was the gripper's strength, which in some cases damaged the glass bottle during grasping. In this experiment, foam was added to both sides of the gripper to effectively control the force, reducing this problem. While the grasp success rate decreased in the presence of obstacles, the reduction was only 3%. Therefore, the method proposed in this article can successfully grasp objects with and without obstacles.

[0149] Reference Figure 5 On the other hand, the present invention also provides a transparent object grasping system based on 6D pose estimation, including a robotic arm, a visual perception device, an electric gripper (EPG50-060 series electric parallel gripper) and a built-in PC unit (NUC onboard microcomputer). The visual perception device and the electric gripper are both installed at the end of the robotic arm, and the built-in PC unit is installed below the robotic arm. The robotic arm, the visual perception device, and the electric gripper are all electrically connected to the built-in PC unit. The robotic arm grasps the transparent object according to the transparent object grasping method.

[0150] The visual perception equipment includes a webcam, a Realsence camera (Realsense D435i depth camera), a Kinect peripheral, and a light field camera. The electric gripper is wrapped with cotton, and the robotic arm is a six-axis robotic arm.

[0151] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A transparent object grasping method based on 6D pose estimation, characterized in that: include: S1. Input the original input image containing transparent objects into the TOG-Net target detection module to obtain the 2D bounding box and confidence score of the target object in the original input image; S2, extracting target features from the preprocessed original input image based on the 2D bounding box and confidence score of the target object, and performing similarity matching with the features in the reference image library to obtain the reference image with the highest similarity; S3. Estimate the rotation and translation information of the plane where the target feature is located based on the reference features in the reference image with the highest similarity, generate a preliminary 6D pose of the target feature, and then refine and adjust the preliminary 6D pose of the target feature to obtain the final 6D pose of the target feature; S4, dividing the path between the robot arm and the target feature into three segments, converting the three segments into polynomial interpolation functions, and solving the polynomial interpolation functions using the particle swarm optimization algorithm to obtain the optimized trajectory; S5, the robotic arm grasps the target feature according to the optimized trajectory; The TOG-Net target detection module includes a backbone network, a network layer, and a prediction layer connected in sequence; The backbone network includes a convolutional sampling layer and four stage units connected in series. The four stage units are the first stage unit to the fourth stage unit. The first stage unit includes the LN normalization layer and the ConvNeXt network connected in series. The second stage unit to the fourth stage unit are all downsampling layers and ConvNeXt networks connected in series. The network layer is a hybrid encoder, which includes an intra-scale feature interaction module AIFI and a cross-scale fusion module CCFM connected in sequence; the input of the intra-scale feature interaction module AIFI is the output of the fourth stage unit, that is, the feature map The output of the intra-scale feature interaction module AIFI is the feature map The input of the cross-scale fusion module CCFM is the output of the second stage unit to the fourth stage unit, which are feature maps , feature map , feature map .

2. The transparent object grasping method based on 6D pose estimation according to claim 1, characterized in that: The S1 specifically includes the following steps: S11. First, input the original image of the transparent object Input into the convolution sampling layer in the backbone network for multi-level feature extraction to obtain the initial feature map , expressed by the formula, as follows: ； in, It is a 4×4 convolution operation; represents the real three-dimensional space, are the height and width of the image respectively; Indicates the number of channels of the convolution sampling layer; S12, the initial feature map Input into four stage units, perform multi-stage multi-scale feature extraction, and obtain multiple feature maps, namely feature maps , feature map , feature map ;in, When, The multi-scale feature operation of the stage unit is expressed by the formula as follows: ； in, Indicates the Feature maps obtained by multi-scale feature operations of stage units; It is The number of channels of the stage unit; Represents the set of operations for downsampling layers and ConvNeXt networks; S13, feature map Flatten to a one-dimensional vector to generate the query matrix , key matrix Sum Matrix The input data is input into the scale feature interaction module AIFI, the input data is processed by the scale feature interaction module AIFI, and then the Reshape operation is used to restore the feature map. The same shape, get the feature map , expressed by the formula, as follows: ； in, Represents a flattening operation; represents the shape recovery operation, i.e. The inverse operation of S14, feature map , feature map And feature maps All are input into the cross-scale fusion module CCFM to obtain multi-scale features; S15. Input the multi-scale features into the prediction layer to obtain the category, 2D bounding box and confidence score of the target object in the input image.

3. The transparent object grasping method based on 6D pose estimation according to claim 2, characterized in that: The S2 specifically includes the following steps: S21, preprocessing the original input image to obtain a preprocessed input image; S22. Based on the category, 2D bounding box, and confidence score output by the TOG-Net target detection module, the corresponding target area is cropped from the preprocessed input image, and then multiple features are extracted in the corresponding target area to obtain the extracted target features; S23, matching the extracted target features with the features in the reference image library, calculating the similarity scores between the target features and the reference features using a similarity metric, and selecting the top N reference images with the highest similarity scores as candidate viewpoints; S24. Introduce refraction path consistency as an additional selection criterion, calculate the refraction path differences between the preprocessed input image and N reference images, and select the viewpoint with the smallest difference. The reference image corresponding to the viewpoint with the smallest difference is the reference image with the highest similarity.

4. The transparent object grasping method based on 6D pose estimation according to claim 3, characterized in that: The S3 specifically includes the following steps: S31, based on the reference features in the reference image with the highest similarity, estimate the rotation and translation information of the plane where the target feature is located, combine the rotation and translation information, and generate a preliminary 6D pose of the target feature; S32, training the posture refiner to obtain a trained posture refiner; S33. Use the trained pose refiner to adjust the preliminary 6D pose to obtain the output of the pose refiner, that is, the final 6D pose of the target feature.

5. The transparent object grasping method based on 6D pose estimation according to claim 4, characterized in that: The output of the posture refiner in S33 is expressed by the formula, which is as follows: ； in, represents the output of the pose refiner, i.e., the final 6D pose of the target feature; Represents feature extraction operation; represents the initial 6D pose; Represents the 2D bounding box of the target object in the original input image.

6. The transparent object grasping method based on 6D pose estimation according to claim 5, characterized in that: The S4 specifically includes the following steps: S41, taking the initial position of the manipulator as the starting point and the final 6D pose of the target feature as the intermediate end point; adding two path points between the starting point and the end point, thereby dividing the path between the manipulator and the target feature into three segments, namely the first segment to the third segment; S42. Using a cubic polynomial interpolation function in the first and third segments, and using a quintic polynomial interpolation function in the second segment, thereby completing the construction of the polynomial interpolation function; S43. A particle swarm optimization algorithm is used to solve the constructed polynomial interpolation function to obtain an optimized trajectory.

7. The transparent object grasping method based on 6D pose estimation according to claim 6, characterized in that: The cubic polynomial interpolation of the first and third segments and the quintic polynomial interpolation of the second segment in S42 are specifically as follows: ； ； ； in, 、 and The robot arm The first segment of cubic polynomial, the second segment of quintic polynomial and the third segment of cubic polynomial trajectory of each joint; 、 、 、 Respectively The 3rd, 2nd, 1st, and 0th coefficients of the interpolation function of the first trajectory of each joint; 、 、 、 、 、 Respectively The 5th, 4th, 3rd, 2nd, 1st, and 0th coefficients of the interpolation function of the second trajectory of each joint; 、 、 、 Respectively The 3rd, 2nd, 1st and 0th coefficients of the interpolation function of the third segment of the trajectory of each joint; 、 、 Respectively The time of the first segment of cubic polynomial, the second segment of quintic polynomial and the third segment of cubic polynomial trajectory corresponding to each joint.

8. The transparent object grasping method based on 6D pose estimation according to claim 7, characterized in that: The S43 specifically includes the following steps: S431, initialize the particle swarm, generate the position and velocity of each particle, and divide the sub-population, then set the maximum number of iterations D, initialize the individual optimal value and the global optimal value; S432, update inertia weight , group learning factor and self-learning factors , construct particle trajectory equation; inertia weight The calculation formula is as follows: ； in, 、 Inertia weight The maximum and minimum values of Indicates the number of iterations; Group learning factor and self-learning factors The calculation formula is as follows: ； ； S433. Deriving the particle trajectory equation to obtain the velocity and acceleration equations of each particle, and then calculating the velocity and acceleration of each particle using the velocity and acceleration equations of each particle; S434, determine whether the velocity and acceleration of each particle meet the constraint conditions; if not, set the particle fitness value to infinity until the conditions are met; if all conditions are met, enter S435; S435, based on self-learning factors Calculate the fitness and update the individual optimal value based on the fitness; then based on the inertia weight , individual optimal value and group learning factor Update the global optimal value; S436. Determine whether the current number of iterations reaches the maximum number of iterations D. If so, output the global optimal value, which is the optimized trajectory. If not, return to S432.

9. A transparent object grasping system based on 6D pose estimation, characterized in that: The present invention comprises a robotic arm, a visual perception device, an electric gripper and a built-in PC unit, wherein the visual perception device and the electric gripper are both installed at the end of the robotic arm, the built-in PC unit is installed below the robotic arm, and the robotic arm, the visual perception device and the electric gripper are all electrically connected to the built-in PC unit; the robotic arm grasps the transparent object according to the transparent object grasping method described in any one of claims 1 to 8.

Images