Bulb rod disordered grabbing method based on 3D vision

Abstract

The invention relates to a ball head rod disordered grabbing method based on 3D vision, belongs to the technical field of robot operation and automatic manufacturing, and particularly relates to a workpiece pose estimation method based on combination of YOLOv8 and Triplet Attention modules, which is used for disordered grabbing tasks in complex scenes. According to the method, the color and depth images of the workpiece are acquired through the RGB-D camera, and the efficient lightweight Triplet Attention module is introduced, so that the processing speed and precision of the model on the target detection and segmentation task on the mobile terminal equipment are remarkably improved. And the Triplet Attention module enhances the space attention mechanism of the model through three-way rotation convolution operation on the features, so that the detection and segmentation precision is remarkably improved. The segmented mask region is applied to the depth image, corresponding point cloud data are extracted, and six-degree-of-freedom pose estimation is carried out by using a normal consistency optimization (NDT) algorithm. The NDT algorithm is excellent in processing noise and partially shielded scenes. Experimental results show that compared with an original YOLOv8 model, the model added with the Triplet Attention module is improved in segmentation precision and recognition accuracy. The method has high accuracy and robustness in the aspects of workpiece recognition and pose estimation, and is suitable for real-time industrial application scenes.

Description

Technical Field

[0001] This invention relates to the fields of robot operation and automated manufacturing technology, specifically to a workpiece pose estimation method based on the combination of YOLOv8 and the Triplet Attention module, for unordered grasping tasks in complex scenarios. Based on the image and point cloud features of different types of workpieces, it achieves efficient identification and accurate pose estimation of target workpieces for unordered grasping. Technical Background

[0002] In the fields of robotics and automated manufacturing, disordered grasping tasks are a significant challenge in robotics technology, especially in complex industrial settings. Accurate identification and grasping of workpieces is crucial for improving production efficiency. Disordered grasping involves robots identifying, locating, and grasping arbitrarily placed workpieces in a chaotic environment. This technology is widely used in industrial automation, such as assembly lines, warehousing, and manufacturing. Its core challenge lies in achieving precise workpiece identification and location through a vision system.

[0003] In recent years, with the development of RGB-D cameras, point cloud processing technology, and deep learning algorithms, significant progress has been made in the application of six-DOF pose estimation for workpieces in complex scenes. However, traditional grasping systems still face many problems and challenges:

[0004] Workpiece recognition in complex scenarios: Industrial production involves a wide variety of workpieces, varying in size, shape, and material. In environments where workpieces are arranged in a disordered manner, quickly and accurately identifying graspable target workpieces is a significant technical challenge. Traditional algorithms often struggle to handle situations where workpieces are partially occluded or stacked.

[0005] Real-time requirements: Industrial production lines demand high real-time performance for grasping tasks. The system needs to complete workpiece identification, positioning, and grasping planning within a very short time. This places high demands on the processing speed of the algorithm, especially on mobile devices where computing resources are limited. How to improve computational efficiency while ensuring accuracy is an urgent problem to be solved.

[0006] Environmental interference: Factors such as changes in lighting and noise interference in complex industrial environments often affect the recognition accuracy of vision systems. For example, strong light, shadows, and dust can degrade image quality, thus affecting the accuracy of pose estimation.

[0007] Robustness of the algorithm: Existing pose estimation algorithms often lack robustness when dealing with noisy and partially occluded workpieces. Especially in industrial environments such as workpieces with highly reflective surfaces and complex stacking scenarios, the algorithm needs to be able to adapt to various complex situations and maintain high recognition accuracy and stability. Summary of the Invention

[0008] The technical problem solved by this invention is to provide a workpiece pose estimation method based on the combination of YOLOv8 and the Triplet Attention module, aiming to solve the problem of unordered grasping in complex scenes. Specifically, this invention solves the following technical problems:

[0009] 1. Efficient workpiece identification and segmentation: To address the challenge of workpiece identification in complex industrial environments, this invention introduces a Triplet Attention module to enhance the spatial attention mechanism of the YOLOv8 model, significantly improving the accuracy of workpiece detection and segmentation, especially in partially occluded and complex backgrounds, enabling more accurate identification of target workpieces.

[0010] 2. Real-time performance and lightweight design: This invention addresses the real-time requirements of industrial production lines by employing a lightweight TripletAttention module. This reduces the computational complexity of the model, improves processing speed, and enables image processing and segmentation tasks to be completed quickly on mobile devices, thereby meeting the real-time requirements of industrial production.

[0011] 3. Robust Point Cloud Matching and Pose Estimation: In the pose estimation stage, this invention employs the Normal Consistency Optimization (NDT) algorithm to effectively handle workpiece noise and partial occlusion issues. Compared to the traditional ICP algorithm, the NDT algorithm exhibits higher robustness and accuracy in workpiece pose estimation in complex scenes.

[0012] 4. Adaptable to grasping tasks in complex scenarios: This invention combines RGB-D cameras to acquire color and depth images and uses the NDT algorithm for point cloud matching, which can accurately estimate the six-degree-of-freedom pose of the workpiece. It is suitable for disordered grasping tasks in industrial scenarios such as highly reflective surfaces and complex stacks, which significantly improves the flexibility and adaptability of industrial automated production lines.

[0013] The technical solution adopted by this invention to achieve the above objectives is: a method for disordered grasping of a ball head based on 3D vision, comprising the following steps:

[0014] Step 1: Acquire RGB and depth images using an RGB-D camera;

[0015] Step 2: For the acquired RGB image, use the YOLOv8-TripletAttention algorithm to identify unoccluded and graspable workpieces, and segment them to form mask data;

[0016] Step 3: Apply the acquired mask data to the depth image to obtain segmented point cloud data;

[0017] Step 4: Perform NDT matching between the segmented point cloud data and the standard point cloud of the workpiece to obtain the pose information of the workpiece;

[0018] Step 5: Send the pose information of the unobstructed and graspable workpiece to the robotic arm to execute the grasping process.

[0019] The process of identifying unoccluded and graspable workpieces using the YOLOv8-TripletAttention algorithm on the acquired RGB image includes the following steps:

[0020] Based on the YOLOv8 network structure, three TripletAttention modules are added between the three segmentation heads and their respective adjacent C2f modules to form the network structure of the YOLOv8-TripletAttention algorithm.

[0021] Specifically, for the middle segmentation head, the adjacent TripletAttention module is located between the C2f module and the convolution module, so that the output of the C2f module is output to the convolution module and the middle segmentation head respectively after passing through the TripletAttention module.

[0022] The acquired RGB image is input into the network structure of the YOLOv8-TripletAttention algorithm to obtain mask data, which represents the unoccluded and graspable workpieces.

[0023] The segmented point cloud data is matched with the standard point cloud of the workpiece using NDT to obtain the workpiece's pose information, including the following steps:

[0024] For segmented point cloud data, the three-dimensional space where the point cloud is located is divided into several regular voxels, and each point in the point cloud data is assigned to the corresponding voxel; statistical analysis is performed on the points in each voxel, and the mean vector and covariance matrix of these points are calculated to describe the normal distribution characteristics of the point cloud within the voxel.

[0025] For the point cloud data within each voxel, calculate its mean vector and covariance matrix;

[0026] An objective function is constructed based on the mean vector and covariance matrix to characterize the difference between the segmented point cloud data and the standard point cloud of the workpiece in the normal distribution of voxels;

[0027] The objective function is optimized to reach its minimum value, resulting in the best-aligned segmented point cloud data and standard point cloud.

[0028] During the optimization process, each point of the segmented point cloud data to be registered is adjusted to a new position through a rigid body transformation matrix. The normal distribution of the point cloud at the new position is compared with that of the standard point cloud. The transformation increment is obtained according to the objective function value, and the rigid body transformation matrix is ​​updated.

[0029] A 3D vision-based unordered club gripping system includes:

[0030] The image acquisition module is used to acquire RGB images and depth images using an RGB-D camera;

[0031] The segmentation point cloud data acquisition module is used to identify unoccluded and graspable workpieces from the acquired RGB image using the YOLOv8-TripletAttention algorithm, segment them to form mask data, and apply the acquired mask data to the depth image to obtain segmentation point cloud data.

[0032] The point cloud matching module is used to perform NDT matching between segmented point cloud data and the standard point cloud of the workpiece to obtain the pose information of the workpiece.

[0033] The gripping control module is used to send the pose information of the unobstructed and grippable workpiece to the robotic arm to execute the gripping process.

[0034] A 3D vision-based unordered club gripping device includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the 3D vision-based unordered club gripping method when the computer program is executed.

[0035] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for disordered grasping of a ball head based on 3D vision.

[0036] The present invention has the following beneficial effects and advantages:

[0037] (1) Efficient recognition and segmentation, improving grasping accuracy: Compared with the traditional disordered grasping method, this invention enhances the spatial attention mechanism of the YOLOv8 model by introducing the TripletAttention module, which greatly improves the accuracy of workpiece recognition and segmentation, especially in partially occluded and complex backgrounds, and can achieve a higher grasping success rate.

[0038] (2) Real-time performance and lightweight design: The present invention adopts a lightweight TripletAttention module, which significantly reduces the consumption of computing resources while ensuring segmentation accuracy, enabling the system to achieve efficient real-time processing on mobile devices and meet the real-time requirements of industrial production lines for grasping tasks.

[0039] (3) Strong robustness and adaptability: This invention combines the Normal Consistency Optimization (NDT) algorithm to effectively handle noise and partial occlusion problems, significantly improving the robustness of pose estimation in complex industrial environments. At the same time, the system can adapt to workpieces of various shapes and materials, and has strong environmental adaptability and application flexibility.

[0040] (4) Accurate pose estimation: Utilizing the NDT algorithm, this invention can more accurately estimate the six-degree-of-freedom pose of the workpiece, especially performing well in handling complex stacked scenarios and workpieces with highly reflective surfaces. This provides a strong guarantee for achieving more efficient and accurate gripping operations and is applicable to various industrial automation scenarios. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method modules of the present invention;

[0042] Figure 2 This is a diagram of the YOLOv8-TripletAttention model structure.

[0043] Figure 3 An abstract representation of the triple attention mechanism;

[0044] Figure 4 YOLOv8 confusion matrix diagram;

[0045] Figure 5 The confusion matrix diagram for YOLOv8-TripletAttention;

[0046] Figure 6 For comparison of segmentation results;

[0047] Figure 7 To segment the point cloud map;

[0048] Figure 8 Matching the point cloud with the resulting image;

[0049] Figure 9 The original scene mapping is used to match the point cloud effect. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0051] This invention presents a workpiece pose estimation method based on YOLOv8 and a Triplet Attention module, applicable to unordered grasping tasks in complex scenes. The method acquires color and depth images of the workpiece using an RGB-D camera and introduces a highly efficient and lightweight Triplet Attention module, significantly improving the processing speed and accuracy of the model for object detection and segmentation on mobile devices. The Triplet Attention module enhances the model's spatial attention mechanism through three-way rotational convolution operations on features, thereby significantly improving detection and segmentation accuracy. The segmented masked region is applied to the depth image to extract the corresponding point cloud data, and the Normal Consistency Optimization (NDT) algorithm is used for six-DOF pose estimation. The NDT algorithm performs excellently in noisy and partially occluded scenes. Experimental results show that compared to the original YOLOv8 model, the model with the Triplet Attention module improves both segmentation accuracy and recognition accuracy. This method exhibits high accuracy and robustness in workpiece recognition and pose estimation, making it suitable for real-time industrial applications.

[0052] A method for disordered grasping of ball head sticks based on 3D vision, including YOLOv8-TripletAttention detection and segmentation model, NDT point cloud matching algorithm, and the hardware platform required for the experiment;

[0053] The hardware platform required for the experiment includes: Intel D435i RGB-D camera 1, and Topstar robotic arm 2.

[0054] This invention includes the following steps:

[0055] Step 1: Acquire RGB and depth images using an RGB-D camera;

[0056] Step 2: The RGB image captured by the camera is used to identify unoccluded and graspable objects using the YOLOv8-TripletAttention algorithm, and then the objects are segmented into instances.

[0057] Step 3: Apply the obtained mask information to the depth map to obtain the segmented point cloud;

[0058] Step 4: Perform NDT matching between the segmented point cloud and the standard point cloud to obtain the object's six-degree-of-freedom pose information;

[0059] Step 5: The robotic arm receives the pose data and executes the grasping process.

[0060] The specific workflow is as follows: Figure 1 As shown.

[0061] In step 1, a high-precision RGB-D camera with an error of less than 2% is required. Before starting the process, the camera and robotic arm must be calibrated to determine the correspondence between image coordinates and robotic arm coordinates. The calibration process for the intrinsic and extrinsic parameters of the RGB-D camera and the distortion correction process are as follows:

[0062] Step 1: Use an RGB-D camera to take multiple color and depth images of the calibration board from different angles and distances, ensuring that the images are clear and the depth data is accurate, and covering the entire field of view of the camera as much as possible in order to obtain more comprehensive calibration data.

[0063] Step 2: Use a corner detection algorithm to automatically identify checkerboard corners in the color image, and associate these corners with the corresponding depth map data to ensure the registration accuracy between the color image and the depth map.

[0064] Step 3: Based on the corner points of the chessboard and multiple views, establish the corresponding equations of the same points, and combine the depth information to establish the mapping relationship between the 3D point coordinates and the image pixel coordinates.

[0065] Step 4: Solve the equations using the nonlinear least squares method to preliminarily estimate the internal parameters of the RGB-D camera, including focal length, principal point coordinates, distortion coefficients, etc.

[0066] Step 5: Optimize internal parameters by minimizing reprojection error to ensure consistency between the camera's color and depth images from different viewpoints, thereby further improving calibration accuracy.

[0067] Step 6: Using the known world coordinate points (calibration plate corner points) and their mappings on the color image and depth map, combined with the internal parameters, use the PnP algorithm to solve for the rotation and translation of the camera relative to the world coordinate system in each view.

[0068] Step 7: Similar to the internal parameters, further optimize the camera's external parameters, including the camera's position and orientation at different viewpoints, by minimizing the reprojection error.

[0069] Step 8: Based on the internal parameters and distortion coefficients of the RGB-D camera, use the undistort function in OpenCV to correct the distortion of the color image, and at the same time perform corresponding distortion processing on the depth map to ensure the geometric consistency between the color image and the depth image.

[0070] Step 2, the YOLOv8-TripletAttention algorithm needs to be implemented as follows:

[0071] Accurate identification and instance segmentation of unobstructed, graspable ball heads in complex stacked ball head scenarios.

[0072] Step 4: Using the NDT point cloud matching algorithm, obtain the workpiece's six-degree-of-freedom pose information, including:

[0073] Three-dimensional translation: The position of an object in space is determined by the displacement along three coordinate axes, usually represented as x, y, and z. These correspond to the object's forward / backward, left / right, and up / down movement in space.

[0074] X-axis: Displacement along the X-axis (left or right movement)

[0075] Y-axis: Displacement (forward or backward movement) along the Y-axis

[0076] Z-axis: Displacement (vertical movement) along the Z-axis

[0077] Three-dimensional rotation: The orientation of an object in space is determined by three rotation angles, usually represented as rotations about the X, Y, and Z axes. These rotations can typically be represented using Euler angles (Roll, Pitch, Yaw) or quaternions.

[0078] Rotation about the X-axis (Roll): Rotation in the forward / backward direction

[0079] Rotation around the Y-axis (Pitch): Rotation in the left-right direction.

[0080] Rotation about the Z-axis (Yaw): Rotation in the up-down direction.

[0081] A workpiece pose estimation method based on YOLOv8 and the Triplet Attention module is proposed for unordered grasping tasks in complex scenes. The method includes the following steps:

[0082] 1. Workpiece Image and Point Cloud Data Acquisition: Use an RGB-D camera to acquire color and depth images of the workpiece to be captured. Input the color images into an improved YOLOv8 model for workpiece detection and segmentation, and generate the segmentation mask region of the target workpiece.

[0083] 2. Enhanced Workpiece Segmentation and Recognition: By introducing the Triplet Attention module, the YOLOv8 model is optimized to enhance its spatial attention mechanism. This module captures cross-dimensional features of the workpiece through three-way rotational convolution operations, improving the accuracy of detection and segmentation, especially for partially occluded workpieces in complex scenes.

[0084] 3. Depth Image Point Cloud Extraction: The segmented mask region is applied to the depth image to extract the point cloud data of the target workpiece and generate the corresponding 3D point cloud model.

[0085] 4. Normal Consistency Optimization (NDT) Matching: The extracted point cloud is matched with the standard workpiece point cloud using the Normal Consistency Optimization (NDT) algorithm to estimate the workpiece's six-degree-of-freedom pose. The NDT algorithm effectively handles noise and partial occlusion issues by constructing a point cloud normal distribution model, ensuring the robustness and accuracy of pose estimation.

[0086] 5. Grasping Planning and Execution: The estimated six-DOF pose information is transmitted to the robotic arm control system to generate a grasping path plan and execute the workpiece grasping task. The grasping path is automatically adjusted based on the pose information of different workpieces to ensure a high success rate in complex scenarios.

[0087] The image data acquisition specifically involves acquiring color and depth images of the workpiece using an RGB-D camera. The color image is used by the YOLOv8-TripletAttention model for target workpiece detection and instance segmentation, while the depth image is used to extract the segmented point cloud data for subsequent pose estimation.

[0088] The workpiece recognition and segmentation described herein specifically utilizes the YOLOv8 model combined with the Triplet Attention module. The Triplet Attention module enhances the model's spatial attention mechanism through three-way rotational convolution operations, improving the accuracy of workpiece recognition and segmentation, and exhibiting strong robustness, particularly in complex scenes. See the image description for a detailed description.

[0089] The pose estimation specifically involves matching the extracted point cloud with the standard workpiece point cloud using the Normal Consistency Optimization (NDT) algorithm to estimate the workpiece's six-degree-of-freedom pose. The NDT algorithm effectively handles noise and partial occlusion issues by establishing a normal distribution model of the point cloud, thereby improving the accuracy and stability of pose estimation. Specifically, it includes:

[0090] 1. Mesh Generation

[0091] The algorithm divides the 3D space containing the point cloud into several regular grid cells (voxels), which are typically cubic in shape. By setting the size of the voxels, the number of points contained in each voxel and the spatial resolution are controlled. The choice of voxel size needs to be determined based on the density of the point cloud and the actual application scenario.

[0092] Each point in the point cloud data is assigned to a corresponding voxel. The voxel number of a point is determined by its 3D coordinates. The voxel number of a point (x, y, z) can be calculated using the following formula:

[0093]

[0094] Among them, Vi is a voxel within the reference point cloud, x, y, z are the coordinates of a point in the reference point cloud, and s is the side length of the voxel.

[0095] Statistical analysis is performed on the points within each voxel, calculating their mean vector and covariance matrix to describe the normal distribution characteristics of the point cloud within the voxel. These statistical characteristics are used as the basis for matching in the subsequent registration process. Through this mesh generation method, the NDT algorithm transforms the complex 3D registration problem into a series of smaller local problems, each described and processed using normal distribution, thereby improving the robustness and stability of the algorithm.

[0096] 2. Normal distribution modeling

[0097] For the point cloud data within each voxel, calculate its mean vector μ. The mean vector represents the center position of all points within that voxel.

[0098]

[0099] Where, μ i It is the mean vector calculated from the points in the reference point cloud within the i-th voxel. N i It is the i-th voxel unit V i The number of points in the voxel, where x is the coordinate of a point within the voxel.

[0100] For each point within a voxel, calculate the covariance matrix Σ. The covariance matrix describes the distribution of the point cloud data in various directions.

[0101]

[0102] Where, Σ i It is the covariance matrix calculated from the points in the reference point cloud within the i-th voxel.

[0103] The calculated mean vector μ and covariance matrix Σ are used to describe the normal distribution model of the point cloud within each voxel. These models will be used to calculate the degree of matching between point clouds during the registration process. By performing the above statistical analysis on the points within each voxel, the NDT algorithm can generate a normal distribution model describing the point cloud data, providing necessary statistical information for the subsequent registration process.

[0104] 3. Definition of the objective function

[0105] Define the difference between the normal distributions of the point cloud to be registered and the reference point cloud in each grid cell. The goal is to minimize the negative log-likelihood of the point cloud to be registered under the normal distribution of the reference point cloud.

[0106]

[0107] Where y comes from the point cloud to be registered, which is the point that needs to be transformed and matched.

[0108] 4. Optimize the solution

[0109] The rigid body transformation parameters (translation and rotation) of the point cloud to be registered are iteratively adjusted using optimization algorithms (such as gradient descent) to minimize the objective function, thereby achieving the best alignment of the two sets of point clouds.

[0110] During the optimization process, each point y in the point cloud to be registered will undergo a rigid body transformation matrix T. The transformation formula is as follows:

[0111] y′=T·y=R·y+t

[0112] Here, T is the transformation matrix, which rotates and translates the point cloud. Through this transformation, the point y of the point cloud to be registered is adjusted to a new position y′, and then compared with the normal distribution of the reference point cloud at that position.

[0113] Transformation matrix T:

[0114]

[0115] Where R is the rotation matrix and t is the translation vector.

[0116] Update rules:

[0117] T k+1 =T k +ΔT

[0118] Here, ΔT is the transformation increment for each iteration, which is calculated by minimizing the objective function.

[0119] The aforementioned grasping path planning and execution specifically involves transmitting the estimated six-degree-of-freedom pose information to the robotic arm control system, and automatically adjusting the grasping path based on the pose information of different workpieces to ensure a high grasping success rate in complex industrial scenarios.

[0120] like Figure 1 The diagram shown is a flowchart illustrating the overall steps of the method of this invention. This flowchart demonstrates an unordered grasping task based on RGB-D camera data acquisition, YOLOv8-TripletAttention model segmentation processing, and six-degree-of-freedom pose estimation. RGB and depth data are acquired through camera calibration. After image segmentation and point cloud extraction, pose estimation is performed using the Normal Consistency Optimization (NDT) algorithm. Finally, the position information and rotation angle of the workpiece are output, completing the grasping task.

[0121] like Figure 2The image shows the YOLOv8-TripletAttention detection and segmentation model of this method. Based on YOLOv8, a TripletAttention module is added between the three segmentation heads and the C2f module.

[0122] Figure 3 Here is an abstract representation of the triple attention mechanism, where:

[0123] Branch (a): This branch directly processes the input tensor without rotation, and then extracts features through residual transformation.

[0124] Branch (b): This branch first rotates the input tensor along the dimensions of width (W) and channel (C), and then performs a residual transformation.

[0125] Branch (c): This branch rotates the input tensor along the dimensions of height (H) and channel (C), and then performs a residual transformation as well.

[0126] Figure 4 and Figure 5 The images show the confusion matrices of YOLOv8 and YOLOv8-TripletAttention on the golf club dataset. The YOLOv8-TripletAttention model identifies all whole-class samples as whole, achieving a 3% improvement over YOLOv8 and reaching 100% recognition accuracy. This ensures that all unoccluded golf clubs can be captured during the grasping task. However, both models sometimes identify the background as partial. This is because the dataset is relatively small, and whole-class features can be learned better, while partial-class features are more difficult to learn due to varying occlusion conditions. Increasing the size of the dataset will help improve the model's accuracy.

[0127] Figure 6 To compare the segmentation results, for the original... Figure 1 Both models accurately identified the whole class, with YOLOv8-TripletAttention showing better segmentation performance for whole class samples. For the original... Figure 2 YOLOv8 failed to accurately identify all whole samples, which directly led to the failure of crawling.

[0128] Figure 7 This is a segmentation point cloud map obtained by mapping the segmentation mask onto the depth map.

[0129] Figure 8 The matching result image obtained by performing NDT matching on the segmented point cloud is shown in the figure. The normal vector of the point cloud determines the pose information of the point cloud.

[0130] Figure 9 This is a mapping of the original scene to the point cloud matching effect. The matching results are mapped back to the original image to facilitate observation of the matching effect.

[0131] This invention introduces an RGB-D camera to acquire color and depth images of the workpiece, and combines this with YOLOv8 and a Triplet Attention module to effectively improve the accuracy of workpiece recognition and segmentation. This method can cope with lighting changes in complex industrial environments, ensuring the stability of the acquired images, and improves recognition accuracy and grasping success rate through a high-precision segmentation algorithm.

[0132] This invention employs a lightweight Triplet Attention module, reducing computational complexity and minimizing the system's reliance on computing resources while ensuring segmentation accuracy. This improvement enables the system to achieve efficient real-time processing on mobile devices, meeting the real-time requirements of industrial production lines for grasping tasks and significantly improving work efficiency and adaptability.

[0133] This invention enhances the robustness of six-DOF pose estimation for a workpiece by employing the Normal Consistency Optimization (NDT) algorithm. The NDT algorithm can accurately estimate the workpiece pose even in the presence of noise and partial occlusion, avoiding measurement errors caused by noise and partial occlusion in traditional algorithms, thereby further improving the accuracy and stability of the system.

[0134] The above description is only the best specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the protection scope of the present invention.

[0135] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A method for disordered grasping of a ball head based on 3D vision, characterized in that, Includes the following steps: Step 1: Acquire RGB and depth images using an RGB-D camera; Step 2: For the acquired RGB image, use the YOLOv8-TripletAttention algorithm to identify unoccluded and graspable workpieces, and segment them to form mask data; Step 3: Apply the acquired mask data to the depth image to obtain segmented point cloud data; Step 4: Perform NDT matching between the segmented point cloud data and the standard point cloud of the workpiece to obtain the pose information of the workpiece; Step 5: Send the pose information of the unobstructed and graspable workpiece to the robotic arm to execute the grasping process.

2. The method for disordered grasping of a ball head based on 3D vision according to claim 1, characterized in that, The process of identifying unoccluded and graspable workpieces using the YOLOv8-TripletAttention algorithm on the acquired RGB image includes the following steps: Based on the YOLOv8 network structure, three TripletAttention modules are added between the three segmentation heads and their respective adjacent C2f modules to form the network structure of the YOLOv8-TripletAttention algorithm. Specifically, for the middle segmentation head, the adjacent TripletAttention module is located between the C2f module and the convolution module, so that the output of the C2f module is output to the convolution module and the middle segmentation head respectively after passing through the TripletAttention module. The acquired RGB image is input into the network structure of the YOLOv8-TripletAttention algorithm to obtain mask data, which represents the unoccluded and graspable workpieces.

3. The method for disordered grasping of a ball head based on 3D vision according to claim 1, characterized in that, The segmented point cloud data is matched with the standard point cloud of the workpiece using NDT to obtain the workpiece's pose information, including the following steps: For segmented point cloud data, the three-dimensional space where the point cloud is located is divided into several regular voxels, and each point in the point cloud data is assigned to the corresponding voxel; statistical analysis is performed on the points in each voxel, and the mean vector and covariance matrix of these points are calculated to describe the normal distribution characteristics of the point cloud within the voxel. For the point cloud data within each voxel, calculate its mean vector and covariance matrix; An objective function is constructed based on the mean vector and covariance matrix to characterize the difference between the segmented point cloud data and the standard point cloud of the workpiece in the normal distribution of voxels; The objective function is optimized to reach its minimum value, resulting in the best-aligned segmented point cloud data and standard point cloud. During the optimization process, each point of the segmented point cloud data to be registered is adjusted to a new position through a rigid body transformation matrix. The normal distribution of the point cloud at the new position is compared with that of the standard point cloud. The transformation increment is obtained according to the objective function value, and the rigid body transformation matrix is ​​updated.

4. A ball-handling club unordered grasping system based on 3D vision, characterized in that, include: The image acquisition module is used to acquire RGB images and depth images using an RGB-D camera; The segmentation point cloud data acquisition module is used to identify unoccluded and graspable workpieces from the acquired RGB image using the YOLOv8-TripletAttention algorithm, segment them to form mask data, and apply the acquired mask data to the depth image to obtain segmentation point cloud data. The point cloud matching module is used to perform NDT matching between segmented point cloud data and the standard point cloud of the workpiece to obtain the pose information of the workpiece. The gripping control module is used to send the pose information of the unobstructed and grippable workpiece to the robotic arm to execute the gripping process.

5. A ball-head club disordered grasping device based on 3D vision, characterized in that, It includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement, when executing the computer program, a 3D vision-based unordered club gripping method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements a 3D vision-based disordered ball head gripping method as described in any one of claims 1-3.

Images