Robot visual identification, positioning and grabbing system based on RGBD point cloud

Abstract

The invention discloses a robot visual identification, positioning and grabbing method based on RGBD point cloud, and the method comprises the following steps: 1, carrying out the camera point cloud data processing, synthesizing an RGB image and a depth image into a point cloud image, and carrying out the filtering and denoising processing of the point cloud data; step 2, target detection and capture attitude generation: performing target detection and instance segmentation on the RGB image, identifying a target object and generating a corresponding mask frame, and applying a mask to the depth image to extract depth point cloud data corresponding to the target object; obtaining the depth point cloud data, generating a plurality of groups of grabbing postures, and screening out an optimal grabbing posture; and thirdly, the mechanical arm grabbing execution module receives the optimal grabbing posture information output by the target detection and grabbing posture generation module and controls the robot to complete the grabbing action.

Description

Technical Field

[0001] The present invention relates to an industrial robot, and in particular to a robot visual recognition, positioning and grasping system based on RGBD point cloud. Background Art

[0002] With the development of industrial automation and intelligence, robot grasping systems have been widely used in manufacturing, logistics, and service industries. Existing vision-based object detection and grasping systems have certain shortcomings when dealing with complex environments and application scenarios. For stacked, complex, uncalibrated objects, and relatively cluttered industrial or general scenes, there are problems such as inability to achieve reasonable recognition, low detection accuracy, insufficient real-time performance, and inaccurate grasping posture.

[0003] Therefore, the present invention proposes a visual target detection and grasping system that combines RGBD camera, fusion deep learning and point cloud processing technology to improve the accuracy, stability and adaptability of the humanoid robot system. Summary of the invention

[0004] This application proposes a visual target detection and grasping system based on RGBD camera, deep learning and point cloud processing, which achieves the technical effects of high detection and grasping accuracy and strong real-time performance.

[0005] The present application provides a robot visual recognition, positioning and grasping system based on RGBD point cloud, characterized in that it includes: a camera point cloud data processing module, a target detection and grasping posture generation module, and a robotic arm grasping execution module;

[0006] The camera point cloud data processing module synthesizes the RGB image and the depth image into a point cloud image and performs filtering and denoising on the point cloud data;

[0007] The target detection and grasping posture generation module includes a YOLOv8 neural network model and a GraspNet multi-stage deep learning model; the YOLOv8 neural network model performs target detection and instance segmentation on the RGB image, identifies the target object and generates a corresponding mask frame, and applies the mask to the depth image to extract the depth point cloud data corresponding to the target object; the GraspNet multi-stage deep learning model obtains the depth point cloud data, generates several groups of grasping postures, and selects the optimal grasping posture;

[0008] The robot arm grasping execution module receives the optimal grasping posture signal output by the target detection and grasping posture generation module and controls the robot to complete the grasping action.

[0009] Preferably, the robotic arm grasping execution module also includes a robotic arm hand-eye calibration module and a gravity compensation module; the robotic arm hand-eye calibration module is used to calculate the posture transformation from the end of the robotic arm to the base coordinate, and the gravity compensation module is used to slightly adjust the grasping posture in combination with the change in the force of the six-dimensional force sensor at the end of the robotic arm when grasping.

[0010] The present invention also discloses a robot visual recognition, positioning and grasping method based on RGBD point cloud, which is characterized by comprising the following steps:

[0011] The first step is to process the camera point cloud data, synthesize the RGB image and the depth image into a point cloud image, and perform filtering and denoising on the point cloud data;

[0012] The second step is target detection and grasping posture generation. Target detection and instance segmentation are performed on the RGB image, the target object is identified and a corresponding mask frame is generated. The mask is applied to the depth image to extract the depth point cloud data corresponding to the target object; the depth point cloud data is obtained, several groups of grasping postures are generated, and the optimal grasping posture is selected;

[0013] The third step is the robot grasping execution module, which receives the optimal grasping posture information output by the target detection and grasping posture generation module and controls the robot to complete the grasping action.

[0014] Preferably, in the first step, the Kdtree algorithm is combined with the K-means clustering denoising algorithm to remove outlier noise points, filter out isolated outlier points in the data, and then use the K-means algorithm to remove outlier noise points for filtering processing.

[0015] Preferably, the K-means clustering denoising algorithm is used for point cloud data C i Clustering is performed, and the Euclidean distance from the point to the cluster center is used to determine whether the point is a noise point. The algorithm steps include:

[0016] 1.1 Initialize k category centers μ1, μ2, ..., μ from the point cloud k ;

[0017] 1.2 For each point x in the point cloud i ∈D, marking it as the class closest to the center of the class, that is: label i =argmin||x i -μ j ||, 1<j<k;

[0018] 1.3 Update each category center to the mean of all samples belonging to that category

[0019] C jis the sample set belonging to the kth cluster, x i is a single point cloud in this sample;

[0020] 1.4 Repeat steps 1.2 and 1.3 until the set number of iterations is exceeded or the distance change of all category centers in two iterations is less than the initially set threshold λ=0.001.

[0021] Preferably, the first step further includes generating a 6D grasping posture, estimating a point cloud normal vector based on a PCA algorithm, and the steps include:

[0022] A. With the sampling point as the center, establish a local neighborhood point set P with a radius of r, P = {p i =(x i ,y i , z i )∈r 3 i=1, 2, ..., N}, N is the total number of point clouds;

[0023] B. Assume that the sampling point p i ∈P, r-ball neighborhood

[0024] Where: δ is the neighborhood radius, n is the number of neighborhood points;

[0025] C. Use the least squares principle to fit the local optimal tangent plane and construct the objective function J(v, d) to solve the target plane parameters. Where: v is the normal vector of the target plane, d is the spatial distance from the origin to the target plane;

[0026] D. Perform eigenvalue decomposition on the covariance matrix C composed of the local neighborhood point set.

[0027] Where: P is the center of gravity of the neighborhood point set, λ m is the eigenvalue of matrix C, e m is m , where m = 1, 2, 3, and λ1≥λ2≥λ3, then (λ3, e3) is the sampling point P i An estimate of the normal vector for .

[0028] Preferably, the method further includes smoothing the PCA normal after estimation, the steps comprising:

[0029] a. For a given point P i and the corresponding normal n i Measure the input normal n i The normal difference with other points in its neighborhood,

[0030]

[0031] In the formula, ||·| refers to the L2 norm, s i =(p i , n i ), δ p Represents a given neighborhood range;

[0032] b. Set the space and normal weight function to be defined as:

[0033]

[0034] In formula (3-2), the angle parameter σ n Used to measure the similarity of normals, the default value is 15°;

[0035] c. Minimize the objective function ∑ i∈I f(p i , n i ), that is, minimize the difference between the normal of the point to be estimated and the normal of the points in its neighborhood to obtain the smoothed normal;

[0036] d. Use bilateral filtering to iteratively update. The formula is as follows:

[0037]

[0038] Preferably, the second step of target detection and grasping posture generation comprises the following steps:

[0039] 2.1. Input the acquired RGB image into the YOLOv8 neural network model, perform target detection and instance segmentation on the image, identify the target object, and generate the corresponding mask box;

[0040] 2.2. Using the mask box generated by YOLOv8, apply the mask to the depth image to extract the point cloud data corresponding to the target object;

[0041] 2.3. 6D grasping posture generation based on GraspNet, including:

[0042] 2.3.1. Given an input image Where C, W, H represent the number of channels, width and height of the input, and its training loss function is designed as follows:

[0043]

[0044] In the formula, The normalized score map of the convolutional neural network CNN representing the sampled image i is calculated by applying SoftMax to the output of the last layer of the network, namely:

[0045]

[0046] In the formula, f(x i ) represents the sample x i The output of the convolutional neural network CNN network layer, yi, W, b represent the category label, weight and bias of the corresponding layer respectively; the generated 5-dimensional grasping box R = [R x ,R y ,R w ,R h ,R θ ], where R x ,R y represents the center of mass of the grasp, R w Indicates the grab width, R h Indicates the grasping height, R θ Indicates the grasping direction;

[0047] 2.3.2. Six-degree-of-freedom grasping posture generation: Use the DBSCAN clustering algorithm to screen out the most densely populated group of grasping postures, get the top 3 grasping postures based on the grasping scores, and combine the normal vector of the grasping plane in the area in the first step to select the optimal grasping posture as the final posture and output it to the robotic arm grasping execution module.

[0048] Preferably, the third step completes the robot grasping action according to the generated 6D grasping posture, combines the control module of the robotic arm, integrates graspnet to generate the grasping posture, calculates the joint angle in the joint space according to the posture transformation matrix, and sends motion control instructions to complete the grasping.

[0049] Preferably, the third step further includes a step of converting the generated grasping posture into a grasping posture in the joint space of the robot arm and a step of fine-tuning the grasping posture by six-dimensional force; the step of converting the generated grasping posture into a grasping posture in the joint space of the robot arm includes:

[0050] Assumptions Represents the pose transformation matrix of coordinate system a relative to coordinate system b, R is a 3×3 rotation matrix, t is a 3×1 translation vector, so:

[0051]

[0052] T net is the transformation matrix generated by graspnet, multiplied by graspnet on the right to define the correction transformation T adj , get the position transformation of the grasped object relative to the camera is the pose transformation matrix from the camera to the end of the robotic arm, which can be obtained by calibrating the eye on the hand. It is the pose transformation from the end of the robot to the base coordinates, which is calculated by the DH formula of the robot kinematics;

[0053] Gripping position when grasping That is

[0054] It is the pose matrix of the end of the robot arm during grasping, which is the input matrix needed for joint space solution.

[0055] In the six-dimensional force fine-tuning grasping posture step, the robot arm gripper first moves to a position 2-4 cm above the grasping prediction point, and moves forward along the gripper direction. When grasping, the gripping posture is slightly adjusted in accordance with the change of the force of the six-dimensional force sensor at the end; the six-dimensional force sensor monitors the grasping force in real time and provides six-dimensional force and torque information. x , ΔF x , ΔF z , ΔT x , ΔT y , ΔT z )’s force and torque characteristics are used to fine-tune the grasping posture.

[0056] The technical solution provided by this application has at least the following technical effects or advantages:

[0057] 1. The present invention uses yolov8 to perform masking and region segmentation on the grasped objects, obtains a more effective grasping posture set, and can also specify the grasping of such objects.

[0058] 2. Before the posture estimation, the point cloud data is processed and denoised, and the normal vector is smoothed, which further optimizes the generation effect of the grasping posture.

[0059] 3. In the grasping posture obtained by graspnet, the present invention obtains the best grasping posture through DBSCAN clustering algorithm and regional point cloud normal vector, combined with grasping estimation score.

[0060] 4. The present invention combines a six-dimensional force sensor to fine-tune the grasping posture in the final stage of grasping, quickly realizes secondary grasping, and improves the grasping safety and success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is an overall block diagram of the robot grasping system in an embodiment of the present application.

[0062] Figure 2 This is a flowchart for generating the robot grasping posture in an embodiment of the present application.

[0063] Figure 3 This is an implementation diagram of the robot grasping system in the embodiment of the present application.

[0064] Figure 4This is a logic diagram of the robot grasping process in an embodiment of the present application.

[0065] Figure 5 This is a scene effect diagram of the grabbing experiment in the embodiment of this application. DETAILED DESCRIPTION

[0066] Example 1

[0067] This embodiment provides a visual target detection and grasping system based on RGBD camera, deep learning and point cloud processing. The overall block diagram of the grasping system is as follows: Figure 1 As shown, it mainly includes:

[0068] The first step is to process the camera point cloud data, synthesize the RGB image and the depth image into a point cloud image, and perform filtering and denoising on the point cloud data;

[0069] The second step is target detection and grasping posture generation. Target detection and instance segmentation are performed on the RGB image, the target object is identified and a corresponding mask frame is generated. The mask is applied to the depth image to extract the depth point cloud data corresponding to the target object; the depth point cloud data is obtained, several groups of grasping postures are generated, and the optimal grasping posture is selected;

[0070] The third step is the robot grasping execution module, which receives the optimal grasping posture information output by the target detection and grasping posture generation module and controls the robot to complete the grasping action.

[0071] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0072] The first step comprises:

[0073] 1. Camera point cloud data processing:

[0074] Based on the RGBD camera, the RGB image and depth image of the target scene can be obtained in real time. Based on the camera API interface, the RGB image and depth image can be synthesized into a point cloud image;

[0075] When using a camera for grabbing tasks, filtering the point cloud data is a key step. The Kdtree algorithm is combined with the K-means clustering algorithm to remove outlier noise points and filter out isolated outliers in the data to reduce the number of classes and indirectly reduce the K value. The K-means algorithm is then used to remove outlier noise points, thereby enabling fast and effective filtering.

[0076] 2. K-means clustering denoising algorithm for point cloud data C i Clustering is performed, and the Euclidean distance from the point to the cluster center is used to determine whether the point is a noise point. The algorithm steps include:

[0077] 1.1 Initialize k category centers μ1, μ2, ..., μ from the point cloud k ;

[0078] 1.2 For each point x in the point cloud i ∈D, marking it as the class closest to the center of the class, that is: label i =argmin||x i -μ j ||, 1<j<k;

[0079] 1.3 Update each category center to the mean of all samples belonging to that category

[0080] C j is the sample set belonging to the kth cluster, x i is a single point cloud in this sample;

[0081] 1.4 Repeat steps 1.2 and 1.3 until the set number of iterations is exceeded or the distance change of all category centers in two iterations is less than the initially set threshold λ=0.001.

[0082] 3. Generate 6D grasping posture, estimate the surface normal vector of the object point cloud, and estimate the point cloud normal vector based on the PCA algorithm. The basic steps of the PCA algorithm are as follows:

[0083] A. With the sampling point as the center, establish a local neighborhood point set P with a radius of r.

[0084] P = {p i =(x i ,y i , z i )∈r 3 i=1, 2, ..., N}, N is the total number of point clouds,

[0085] B. Assume that the sampling point p i ∈P, r-ball neighborhood

[0086] Where: δ is the neighborhood radius, n is the number of neighborhood points;

[0087] C. Use the least squares principle to fit the local optimal tangent plane and construct the objective function J(v, d) to solve the target plane parameters.

[0088]

[0089] Where: v is the normal vector of the target plane, d is the spatial distance from the origin to the target plane;

[0090] D. Perform eigenvalue decomposition on the covariance matrix C composed of the local neighborhood point set.

[0091] Where: P is the center of gravity of the neighborhood point set, λ m is the eigenvalue of matrix C, e m is m , where m = 1, 2, 3, and λ1≥λ2≥λ3, then (λ3, e3) is the sampling point P i An estimate of the normal vector of ;

[0092] 4. After using PCA to calculate the point cloud, in order to maintain the edge features of the model, the normal is smoothed. The algorithm steps are as follows:

[0093] a. For a given point P i and the corresponding normal n i Measure the input normal n i The normal difference with other points in its neighborhood,

[0094] In the formula, ||·|| refers to the L2 norm, s i =(p i , n i ), δ p Represents a given neighborhood range;

[0095] b. Set the space and normal weight function to be defined as:

[0096]

[0097] In formula (3-2), the angle parameter σ n Used to measure the similarity of normals, the default value is 15°.

[0098] c. Minimize the objective function ∑ i∈l f(p i , n i ), that is, minimize the difference between the normal of the point to be estimated and the normal of the points in its neighborhood to obtain the smoothed normal.

[0099] d. Use bilateral filtering to iteratively update. The formula is as follows:

[0100]

[0101] The second step includes:

[0102] 1. Use YOLOv8 for target detection and segmentation: YOLOv8's Neck and FPN structures effectively extract and fuse multi-scale features through operations such as feature pyramid networks and feature fusion, thereby improving the performance and robustness of target detection. This enables the model to better adapt to targets of different scales and sizes and achieve more accurate detection results in complex scenarios. Specifically:

[0103] 2.1 Input the acquired RGB image into the YOLOv8 neural network model, perform target detection and instance segmentation on the image, identify the target object, and generate the corresponding mask box.

[0104] 2.2 Using the mask box generated by YOLOv8, apply the mask to the depth image to extract the point cloud data corresponding to the target object.

[0105] 2.3 6D grasping posture generation based on GraspNet. The GraspNet model is a multi-stage deep learning model that aims to solve the 6D grasping posture estimation problem in robot grasping tasks. The embodiment of the present invention introduces a CNN network structure, including an encoder branch and a decoder branch, using a Dilated Dense Fire (DDF) module, and squeeze and dilation convolutions with dense residual connections. Specifically including:

[0106] 2.3.1 Given an input image Where C, W, H represent the number of channels, width and height of the input, and its training loss function is designed as follows:

[0107] In the formula, Represents the CNN normalized score map of sampled image i, calculated by applying SoftMax to the output of the last layer of the network, i.e.:

[0108] In the formula, f(x i ) represents the output of the CNN network layer of sample xi, yi, W, b represent the category label, weight and bias of the corresponding layer respectively; the generated 5-dimensional grab box R = [R x ,R y ,R w ,R h ,R θ ], where R x ,R y represents the center of mass of the grasp, R w Indicates the grab width, R h Indicates the grasping height, R θ Indicates the grasping direction;

[0109] 2.4 Six-degree-of-freedom grasping posture generation. There are several 6D grasping postures generated by Graspnet, which can be distributed in different parts of the object to be grasped. In order to improve the grasping stability, the most dense group of grasping postures is selected according to the DBSCAN clustering algorithm. The top 3 grasping postures are obtained according to the grasping score. Combined with the normal vector of the grasping plane in the area mentioned above, the optimal grasping posture is selected as the final posture. The workflow is as follows: Figure 2 shown.

[0110] In the third step, the robot grasping system is specifically implemented as follows: Figure 3-Figure 5 As shown, the system should integrate visual perception, grasping detection, robot control and other functions; the specific steps are as follows:

[0111] 3.1 Crawl process

[0112] The robot grasping action is implemented according to the generated 6D grasping posture. Combining the robot arm kinematics, force control correlation, and graspnet to generate the grasping posture, the joint angle in the joint space is calculated according to the posture transformation matrix, and the motion control command is sent to complete the grasping. The process logic diagram is as follows Figure 4 As shown:

[0113] 3.2 Grasping posture transformation

[0114] The generated grasping posture is converted into the grasping posture of the robot arm joint space. The posture transformation includes:

[0115] Assumptions Represents the pose transformation matrix of coordinate system a relative to coordinate system b, R is a 3×3 rotation matrix, t is a 3×1 translation vector, so:

[0116] T net is the transformation matrix generated by graspnet, multiplied by graspnet on the right to define the correction transformation T adj , get the position transformation of the grasped object relative to the camera is the pose transformation matrix from the camera to the end of the robotic arm, which can be obtained by calibrating the eye on the hand. It is the pose transformation from the end of the robot to the base coordinates, which can be calculated by the DH formula of the robot kinematics;

[0117] Gripping position when grasping That is

[0118] It is the pose matrix of the end of the robot arm during grasping, which is the input matrix needed for joint space solution.

[0119] 3.3 Combined with the six-dimensional force to fine-tune the grasping posture, during the grasping process, the robot arm gripper will first move to a position 2-4cm above the grasping prediction point and move forward in the direction of the gripper. When grasping, it will cooperate with the changes in the force of the end six-dimensional force sensor to make slight adjustments to the grasping posture and quickly perform a second grasp, thereby improving the grasping success rate.

[0120] At the same time, the six-dimensional force sensor is used to monitor the force in the grasping process in real time, providing six-dimensional force and torque information. x , ΔF x , ΔF z , ΔT x , ΔT y , ΔT z )’s force and torque characteristics are used to fine-tune the grasping posture.

[0121] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention. These improvements should also be regarded as within the scope of protection of the present invention.

Claims

1. The robot visual recognition, positioning and grasping system based on RGBD point cloud is characterized by: include: Camera point cloud data processing module, target detection and grasping posture generation module, and robotic arm grasping execution module; The camera point cloud data processing module synthesizes the RGB image and the depth image into a point cloud image and performs filtering and denoising on the point cloud data; The target detection and grasping posture generation module includes a YOLOv8 neural network model and a GraspNet multi-stage deep learning model; the YOLOv8 neural network model performs target detection and instance segmentation on the RGB image, identifies the target object and generates a corresponding mask frame, and applies the mask to the depth image to extract the depth point cloud data corresponding to the target object; the GraspNet multi-stage deep learning model obtains the depth point cloud data, generates several groups of grasping postures, and selects the optimal grasping posture; The robot arm grasping execution module receives the optimal grasping posture signal output by the target detection and grasping posture generation module and controls the robot to complete the grasping action.

2. The robot visual recognition, positioning and grasping system based on RGBD point cloud according to claim 1, characterized in that: The robotic arm grasping execution module also includes a robotic arm hand-eye calibration module and a gravity compensation module; the robotic arm hand-eye calibration module is used to calculate the posture transformation from the end of the robotic arm to the base coordinate, and the gravity compensation module is used to slightly adjust the grasping posture in combination with the change in the force of the six-dimensional force sensor at the end of the robotic arm when grasping.

3. A robot visual recognition, positioning and grasping method based on RGBD point cloud, characterized in that: The steps include: The first step is to process the camera point cloud data, synthesize the RGB image and the depth image into a point cloud image, and perform filtering and denoising on the point cloud data; The second step is target detection and grasping posture generation. Target detection and instance segmentation are performed on the RGB image, the target object is identified and a corresponding mask frame is generated. The mask is applied to the depth image to extract the depth point cloud data corresponding to the target object; the depth point cloud data is obtained, several groups of grasping postures are generated, and the optimal grasping posture is selected; The third step is the robot grasping execution module, which receives the optimal grasping posture information output by the target detection and grasping posture generation module and controls the robot to complete the grasping action.

4. The robot visual recognition, positioning and grasping method based on RGBD point cloud according to claim 3 is characterized in that: In the first step, the Kdtree algorithm is combined with the K-means clustering denoising algorithm to remove outlier noise points, filter out isolated outlier points in the data, and then use the K-means algorithm to remove outlier noise points for filtering processing.

5. The robot visual recognition, positioning and grasping method based on RGBD point cloud according to claim 4 is characterized in that: The K-means clustering denoising algorithm is applied to point cloud data C i Clustering is performed, and the Euclidean distance from the point to the cluster center is used to determine whether the point is a noise point. The algorithm steps include: 1.1 Initialize k category centers μ1,μ2,...,μ from the point cloud k ; 1.2 For each point x in the point cloud i ∈D, marking it as the class closest to the center of the class, that is: label i =argmin||x i -μ j ||, 1 <j<k; 1.3 Update each category center to the mean of all samples belonging to that category C j is the sample set belonging to the kth cluster, χ i is a single point cloud in this sample; 1.4 Repeat steps 1.2 and 1.3 until the set number of iterations is exceeded or the distance change of all category centers in two iterations is less than the initially set threshold λ=0.

001.

6. The robot visual recognition, positioning and grasping method based on RGBD point cloud according to claim 4 is characterized in that: The first step also includes generating a 6D grasping posture and estimating a point cloud normal vector based on a PCA algorithm, the steps comprising: A. With the sampling point as the center, establish a local neighborhood point set P with a radius of r, P = {p i =(x i ,y i , z i )∈r 3 i=1, 2, ..., N}, N is the total number of point clouds; B. Assume that the sampling point p i ∈P r Ball Neighborhood Where: δ is the neighborhood radius, n is the number of neighborhood points; C. Use the least squares principle to fit the local optimal tangent plane and construct the objective function J(v, d) to solve the target plane parameters. Where: v is the normal vector of the target plane, d is the spatial distance from the origin to the target plane; D. Perform eigenvalue decomposition on the covariance matrix C composed of the local neighborhood point set. Where: P is the center of gravity of the neighborhood point set, λ m is the eigenvalue of matrix C, e m is m , where m = 1, 2, 3, and λ1≥λ2≥λ3, then (λ3, e3) is the sampling point P i An estimate of the normal vector of .

7. The robot visual recognition, positioning and grasping method based on RGBD point cloud according to claim 6 is characterized in that: It also includes smoothing after PCA normal estimation, the steps include: a. For a given point P i and the corresponding normal n i Measure the input normal n i The normal difference with other points in its neighborhood, In the formula, ||·|| refers to the L2 norm, s i =(p i , n i ) δ p Represents a given neighborhood range; b. Set the space and normal weight function to be defined as: In formula (3-2), the angle parameter σ n Used to measure the similarity of normals; c. Minimize the objective function ∑ i∈I f(p i , n i ), that is, minimize the difference between the normal of the point to be estimated and the normal of the points in its neighborhood to obtain the smoothed normal; d. Use bilateral filtering to iteratively update. The formula is as follows:

8. The robot visual recognition, positioning and grasping method based on RGBD point cloud according to claim 3 is characterized in that: The second step of target detection and grasping posture generation includes the following steps: 2.

1. Input the acquired RGB image into the YOLOv8 neural network model, perform target detection and instance segmentation on the image, identify the target object, and generate the corresponding mask box; 2.

2. Using the mask box generated by YOLOv8, apply the mask to the depth image to extract the point cloud data corresponding to the target object; 2.

3. 6D grasping posture generation based on GraspNet, including: 2.3.

1. Given an input image Where C, W, and H represent the number of channels, width, and height of the input, respectively. The training loss function is designed as follows: In the formula, Represents the CNN normalized score map of sampled image i, calculated by applying SoftMax to the output of the last layer of the network, i.e.: In the formula, f(x i ) represents the sample x i The output of the CNN network layer, yi, W, b represent the category label, weight and bias of the corresponding layer respectively; the generated 5-dimensional grasping box R = [R x ,R y ,R w ,R h ,R θ , where R x ,R y represents the center of mass of the grasp, R w Indicates the grab width, R h Indicates the grasping height, R θ Indicates the grasping direction; 2.3.

2. Six-degree-of-freedom grasping posture generation: Use the DBSCAN clustering algorithm to screen out the most densely populated group of grasping postures, get the top 3 grasping postures based on the grasping scores, and combine the normal vector of the grasping plane in the area in the first step to select the optimal grasping posture as the final posture and output it to the robotic arm grasping execution module.

9. The robot visual recognition, positioning and grasping method based on RGBD point cloud according to claim 3 is characterized in that: The third step completes the robot grasping action according to the generated 6D grasping posture, combines the control module of the robotic arm, integrates graspnet to generate the grasping posture, calculates the joint angle in the joint space according to the posture transformation matrix, and sends the motion control command to complete the grasping.

10. The robot visual recognition, positioning and grasping method based on RGBD point cloud according to claim 9, characterized in that: The third step also includes a step of converting the generated grasping posture into a grasping posture in the joint space of the robot arm and a step of fine-tuning the grasping posture by six-dimensional force; the step of converting the generated grasping posture into a grasping posture in the joint space of the robot arm includes: Assumptions Represents the pose transformation matrix of coordinate system a relative to coordinate system b, R is a 3×3 rotation matrix, t is a 3×1 translation vector, so: T net is the transformation matrix generated by graspnet, multiplied by graspnet on the right to define the correction transformation T adj , get the position transformation of the grasped object relative to the camera is the pose transformation matrix from the camera to the end of the robotic arm, which can be obtained by calibrating the eye on the hand. It is the pose transformation from the end of the robot to the base coordinates, which is calculated by the DH formula of the robot kinematics; Gripping position when grasping That is It is the pose matrix of the end of the robot arm during grasping, which is the input matrix needed for joint space solution. In the six-dimensional force fine-tuning grasping posture step, the robot arm gripper first moves to a position 2-4 cm above the grasping prediction point, and moves forward along the gripper direction. When grasping, the gripping posture is slightly adjusted in accordance with the change of the force of the six-dimensional force sensor at the end; the six-dimensional force sensor monitors the grasping force in real time and provides six-dimensional force and torque information. x , ΔF x ,ΔF z , ΔT x , ΔT y ,ΔT z )’s force and torque characteristics are used to fine-tune the grasping posture.

Images