Manipulator grabbing guide and control method based on three-dimensional vision

Abstract

The invention discloses a manipulator grabbing guide and control method based on three-dimensional vision, which comprises the following steps of: acquiring point cloud data of a target area on the surface of an object through three-dimensional vision equipment, and reconstructing a grid model of the target area; segmenting the grid model by adopting a region growing method, and extracting target region sub-blocks; flatness evaluation is carried out on all surface elements in the grid model, and the surface elements meeting a threshold value are screened to serve as grabbing guide surface elements; determining a grabbing guide reference based on the target area sub-blocks; a manipulator grabbing path is generated according to the grabbing guide reference, and grabbing operation is executed through a closed-loop control system; and the grabbing force is monitored in real time and dynamically adjusted. According to the method, surface element level analysis is conducted on the reconstructed three-dimensional grid model, the flatness quality index function is constructed, the surface elements meeting the threshold value are screened based on the function to serve as the grabbing guide surface elements, the grabbing guide benchmark is further determined, the area with the high flatness can be preferentially selected to serve as the contact face, and grabbing stability and reliability are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial automation control, and in particular to a three-dimensional vision-based manipulator grasping, guiding and controlling method. Background Art

[0002] With the development of society, robotic arms are increasingly being used to replace manual labor. Vision technology, a key robotic perception method, mimics human vision, analyzing information scanned by 3D vision modules to perceive the physical environment, enabling measurement and judgment, and enabling interaction with the physical environment. High-precision and flexible operations require intelligent machine recognition, helping robotic arms proactively reach non-fixed target locations in 3D space. Research is essential to adding 3D vision guidance capabilities to robotic arms.

[0003] Existing robotic grasping guidance and control methods are mostly based on predefined object models or two-dimensional image processing, which are difficult to adapt to the grasping needs of objects with random postures, stacks or irregular shapes, resulting in large segmentation errors and low grasping success rates. For example, for objects with large changes in surface curvature or local depressions, traditional methods are prone to uneven distribution of grasping force due to improper selection of contact surfaces, causing objects to slip or robotic arm overload. Therefore, a robotic grasping guidance and control method based on three-dimensional vision is urgently needed. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the above technical defects and provide a three-dimensional vision-based manipulator grasping guidance and control method.

[0005] In order to solve the above problems, the technical solution of the present invention comprises the following steps: S1. Acquire point cloud data of a target area on the surface of an object through a three-dimensional vision device, and reconstruct a grid model of the target area based on the point cloud data; S2. Segment the grid model using a region growing method to extract target region sub-blocks; S3. Flatness evaluation is performed on all facets in the mesh model. A flatness quality index function is constructed based on the facet centroid coordinates, minimum normal vector angle, angle between the normal vector and the Z axis, minimum principal curvature, and maximum and minimum principal curvature differences. Facets that meet the threshold are selected as guide facets for grasping. S4, determining a grasping guidance benchmark based on the target region sub-block where the grasping guidance surface element is located; S5, generating a manipulator grasping path according to the grasping guidance reference, and performing a grasping operation through a closed-loop control system; S6. During the grasping process, the contact force between the manipulator and the target object is monitored in real time, and the grasping force is dynamically adjusted to maintain a stable grasp; S7. When a collision risk is detected, a safety control mechanism is activated to ensure operational safety.

[0006] Furthermore, the specific process of the octree depth-first search in step S1.3 includes the following steps: S1.3.1. Set the regional point cloud data and select a point as the target point; S1.3.2. With the target point as the center, search for adjacent point clouds within the set radius; S1.3.3. Divide the search area into eight sub-cubes and recursively perform sub-cube search until a preset depth is reached; S1.3.4. Reconstruct the dense point cloud data of the target area through the index mapping relationship.

[0007] Furthermore, the specific formula of the bin flatness quality index function in step S3 is: Among them, C is the coordinate of the centroid of the surface element, is the minimum normal vector angle of the surface element, is the angle between the surface element normal vector and the Z axis, is the minimum principal curvature of the surface element, is the difference between the maximum and minimum principal curvatures of the surface element, to is the weight coefficient.

[0008] Furthermore, the method for determining the grasping guide reference in step S4 includes the following steps: S4.1. Based on the bin planarity sequence, select the first N bins with the largest planarity; S4.2. Calculate the areas of the triangular facets corresponding to the N facets, and select facets whose areas are greater than a set threshold; S4.3. Use the sub-block of the target area where the filtered surface element is located as a grasping guidance benchmark.

[0009] Furthermore, the dynamic adjustment of the gripping force in step S6 includes the following steps: S6.1. Real-time monitoring of the contact force between the manipulator end and the target object; S6.2, when the contact force exceeds a set threshold, reducing the grasping force through a PID controller or MPC algorithm; S6.3. When the contact force is lower than the set threshold, gradually increase the grasping force until a stable grasping state is reached.

[0010] Furthermore, the point cloud data acquisition and mesh model reconstruction method in step S1 includes the following steps: S1.1. Collect initial point cloud data using an RGB-D camera or laser scanner. S1.2. Setting a two-dimensional search criterion based on pre-stored object surface feature data to extract a point cloud subset of the target area; the object surface feature data includes the size, texture, material, and color of the geometric surface; S1.3. Construct a target search tree using an octree algorithm, perform a depth-first search on the point cloud subset, and generate a point cloud index; S1.4. Reconstruct a continuous 3D mesh model based on the index using Poisson reconstruction or Delaunay triangulation algorithm.

[0011] Furthermore, the implementation of the region growing method in step S2 includes the following steps: S2.1. Calculate the surface normal vector and curvature parameters of each facet in the 3D mesh model; S2.2. Generate contour lines based on the difference in normal vector orientation; S2.3. Split the mesh model according to the normal vector orientation, and gradually enlarge the model by setting the step size until the surface normal vector and curvature meet the boundary characteristics; S2.4. The target area sub-block that meets the boundary characteristics is taken as the segmentation result.

[0012] Furthermore, the bin selection rules in step S3 include: the bin centroid is located in the target area sub-block, the minimum normal vector angle is and the angle between the normal vector and the Z axis are all less than the set threshold, the minimum principal curvature Greater than the set threshold, and the maximum and minimum principal curvature differences Less than the set threshold.

[0013] Furthermore, the closed-loop control system in step S5 includes the following modules: A feedforward control module, wherein the feedforward control module is used to compensate for errors in the manipulator dynamics model; a feedback control module that adjusts gripping force based on force / torque sensor data; A redundant degree of freedom coordination module is provided, wherein the redundant degree of freedom coordination module optimizes the motion smoothness of the multi-joint manipulator.

[0014] Furthermore, the safety control mechanism in step S7 includes immediately initiating emergency braking and switching to a flexible grasping mode when a risk of collision between the manipulator and an obstacle or a human body is detected. The flexible grasping mode is achieved by reducing joint stiffness, decreasing grasping speed, and adjusting the contact force threshold.

[0015] The advantages of the present invention compared with the existing technology are: The present invention provides a grasping guidance and control method for a manipulator based on 3D vision. The method collects point cloud data of a target area on the surface of an object through a 3D vision device (such as an RGB-D camera or a laser scanner), and uses an octree algorithm to construct a target search tree. The point cloud data is indexed by performing a depth-first search on the point cloud data, and the point cloud data of the target area is then reconstructed. A surface reconstruction algorithm (such as Poisson reconstruction or Delaunay triangulation algorithm) is used to generate a continuous, high-precision 3D mesh model. Compared with traditional 2D vision or preset model methods, the 3D mesh model contains the depth, curvature and surface detail information of the object, enabling the manipulator to more accurately identify the object's position and morphological features. In scenarios such as industrial sorting and logistics warehousing, the manipulator can dynamically adjust its grasping strategy based on the 3D model even when faced with stacked, tilted or partially obscured objects, significantly improving the grasping success rate and operating efficiency. The present invention provides a robot grasping guidance and control method based on three-dimensional vision, which performs facet-level analysis on the reconstructed three-dimensional mesh model, constructs a flatness quality index function by calculating geometric features such as the facet centroid coordinates, the minimum normal vector angle, the angle between the normal vector and the Z-axis, the minimum principal curvature, and the maximum and minimum principal curvature difference. Based on the function, facets that meet the threshold are screened as grasping guidance facets, and the grasping guidance benchmark is further determined. The area with high flatness can be preferentially selected as the contact surface to avoid object slipping or robot overload due to local deformation, thereby significantly improving grasping stability and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a system diagram of a three-dimensional vision-based manipulator grasping guidance and control method of the present invention. DETAILED DESCRIPTION

[0017] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] like Figure 1As shown, this embodiment proposes a three-dimensional vision-based manipulator grasping guidance and control method, including the following steps: S1. Obtain point cloud data of the target area on the surface of the object through a three-dimensional vision device, and reconstruct a grid model of the target area based on the point cloud data; S2. Use the region growing method to segment the grid model and extract the target area sub-blocks; S3. Flatness evaluation is performed on all facets in the mesh model. A flatness quality index function is constructed based on the facet centroid coordinates, minimum normal vector angle, angle between the normal vector and the Z axis, minimum principal curvature, and maximum and minimum principal curvature differences. Facets that meet the threshold are selected as guide facets for grasping. S4, determining a grasping guidance benchmark based on the target region sub-block where the grasping guidance surface element is located; S5, generating a manipulator grasping path according to the grasping guidance benchmark, and executing the grasping operation through a closed-loop control system; S6. During the grasping process, the contact force between the manipulator and the target object is monitored in real time, and the grasping force is dynamically adjusted to maintain a stable grasp; S7. When a collision risk is detected, a safety control mechanism is activated to ensure operational safety.

[0020] Furthermore, the point cloud data acquisition and mesh model reconstruction method in step S1 includes the following steps: S1.1. Collect initial point cloud data using an RGB-D camera or laser scanner. S1.2. Setting a two-dimensional search criterion based on pre-stored object surface feature data to extract a point cloud subset of the target area; the object surface feature data includes the size, texture, material, and color of the geometric shape surface; S1.3. Use the octree algorithm to construct a target search tree, perform a depth-first search on the point cloud subset, and generate a point cloud index; S1.4. Reconstruct a continuous 3D mesh model based on the index using Poisson reconstruction or Delaunay triangulation algorithm.

[0021] Furthermore, the specific process of the octree depth-first search in step S1.3 includes the following steps: S1.3.1. Set the regional point cloud data and select a point as the target point; S1.3.2. With the target point as the center, search for adjacent point clouds within the set radius; S1.3.3. Divide the search area into eight sub-cubes and recursively perform sub-cube search until a preset depth is reached; S1.3.4. Reconstruct the dense point cloud data of the target area through the index mapping relationship.

[0022] Furthermore, the implementation of the region growing method in step S2 includes the following steps: S2.1. Calculate the surface normal vector and curvature parameters of each facet in the 3D mesh model; S2.2. Generate contour lines based on the difference in normal vector orientation; S2.3. Split the mesh model according to the normal vector orientation, and gradually enlarge the model by setting the step size until the surface normal vector and curvature meet the boundary characteristics; S2.4. The target area sub-block that meets the boundary characteristics is taken as the segmentation result.

[0023] Furthermore, the specific formula of the panel flatness quality index function in step S3 is:

[0024] Among them, C is the coordinate of the centroid of the surface element, is the minimum normal vector angle of the surface element, is the angle between the surface element normal vector and the Z axis, is the minimum principal curvature of the surface element, is the difference between the maximum and minimum principal curvatures of the surface element, to is the weight coefficient.

[0025] The specific formula for the coordinates of the surface element centroid is:

[0026] in, is the coordinate vector of the ith vertex of the face element, and n is the total number of face element vertices.

[0027] The specific formula for the minimum normal vector angle of the surface element is:

[0028] in, and is the normal vector of the surface element and its adjacent surface elements.

[0029] The specific formula for the angle between the surface element normal vector and the Z axis is:

[0030] Where n is the normal vector of the surface element and k is the unit vector of the Z axis (0,0,1).

[0031] The specific formula for the minimum principal curvature of a surface element is:

[0032] in, is the adjacent edge vector of the face element, and d is the edge length.

[0033] The specific formula for the maximum and minimum principal curvature difference of a surface element is:

[0034] to is the weight coefficient, satisfying the formula:

[0035] Furthermore, the bin selection rules in step S3 include: the bin centroid is located in the target area sub-block, the minimum normal vector angle and the angle between the normal vector and the Z axis are all less than the set threshold, the minimum principal curvature Greater than the set threshold, and the maximum and minimum principal curvature differences Less than the set threshold.

[0036] Furthermore, the method for determining the grabbing guide reference in step S4 includes the following steps: S4.1. Based on the bin planarity sequence, select the first N bins with the largest planarity; S4.2. Calculate the areas of the triangular facets corresponding to the N facets, and select facets whose areas are greater than a set threshold; S4.3. Use the sub-block of the target area where the filtered surface element is located as a grasping guidance benchmark.

[0037] Furthermore, the closed-loop control system in step S5 includes the following modules: Feedforward control module, the feedforward control module is used to compensate for the error of the manipulator dynamics model; A feedback control module adjusts the gripping force based on the force / torque sensor data; Redundant degree of freedom coordination module,The redundant degree of freedom coordination module optimizes the motion smoothness of the multi-joint manipulator.

[0038] Furthermore, the dynamic adjustment of the gripping force in step S6 includes the following steps: S6.1. Real-time monitoring of the contact force between the manipulator end and the target object; S6.2, when the contact force exceeds a set threshold, reducing the grasping force through a PID controller or MPC algorithm; S6.3. When the contact force is lower than the set threshold, gradually increase the grasping force until a stable grasping state is reached.

[0039] Furthermore, the safety control mechanism in step S7 includes immediately initiating emergency braking and switching to a flexible grasping mode when a collision risk between the manipulator and an obstacle or a human body is detected. The flexible grasping mode is achieved by reducing joint stiffness, decreasing grasping speed, and adjusting the contact force threshold.

[0040] Example 1: In specific use, an Intel RealSense D435 depth camera is used, the resolution is set to 1280×720, the frame rate is 30fps, the working distance is 0.5m to 1.5m, the camera is used to collect point cloud data of the object surface, and the original point cloud density is generated to be 50 points per cubic centimeter. The Poisson reconstruction algorithm is used, the reconstruction depth is set to 8 layers, and the average error of the generated three-dimensional mesh model is ±0.2mm.

[0041] The normal vector angle threshold is set to 15°, the curvature threshold is set to 0.1mm⁻¹, and the minimum volume of the target area sub-block is set to 5cm³. Calculation of the centroid of the face: take the arithmetic mean of the coordinates of all vertices in the face, and the number of vertices should be no less than 4.

[0042] Minimum normal vector angle: Set the threshold to 10° to filter out facets with high normal vector consistency.

[0043] Angle between the normal vector and the Z axis: Set the threshold to 30° to ensure that the surface element orientation is consistent with the grasping direction.

[0044] Minimum principal curvature: Set the threshold to 0.05mm⁻¹ to exclude non-planar areas with excessive curvature.

[0045] Maximum and minimum principal curvature difference: Set the threshold to 0.02 mm⁻¹ to filter out surface elements with uniform curvature.

[0046] Flatness quality index function weight:

[0047] The surface elements with flatness index Q ≥ 0.85 are selected as the grasping guide surface elements.

[0048] Based on the screened facets, the RRT* algorithm is used to plan a collision-free path with a path step size of 2 mm and 500 iterations. The target contact force is set to 10 N, the allowable error range is ±1 N, and a PID controller is used with the following parameters: proportional coefficient Kp=2.0, integral coefficient Ki=0.5, differential coefficient Kd=0.1. The grasping force is adjusted in real time with a response time of 50 ms.

[0049] The safety distance threshold is set to 10 mm. When the distance between the manipulator and the obstacle is less than this value, emergency braking is triggered, the joint stiffness is reduced to 30% of the original value, the grasping speed is reduced to 50 mm / s, and the contact force threshold is adjusted to 5 N.

[0050] Example 2: A Velodyne VLP-16 laser scanner is used with a scanning range of 0.1m to 100m and an angular resolution of 0.1°. The generated point cloud density is 80 points per cubic centimeter, covering the complete morphology of the stacked objects. The Delaunay triangulation algorithm is used with a maximum side length of 5mm. The average error of the generated mesh model is ±0.1mm.

[0051] The normal vector angle threshold is set to 10°, the curvature threshold is set to 0.05 mm⁻¹, and the minimum volume of the target area sub-block is set to 2 cm³. The above data are substituted into the flatness quality index function, and the surface elements with Q ≥ 0.90 are selected as the grasping guide surface elements.

[0052] The A* algorithm is used to plan the optimal path, with a path step size of 1mm, 1000 iterations, a target contact force of 15N, an allowable error range of ±1.5N, an MPC algorithm with a prediction time domain of 0.1s, a control time domain of 0.05s, and real-time adjustment of the grasping force with a response time of 30ms.

[0053] Set the safety distance threshold to 5mm, the delay time for triggering emergency braking to 10ms, reduce the joint stiffness to 20% of the original value, reduce the grasping speed to 30mm / s, and adjust the contact force threshold to 3N.

[0054] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0055] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0056] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. A three-dimensional vision-based manipulator grasping guidance and control method, characterized in that: The following steps are involved: S1. Acquire point cloud data of a target area on the surface of an object through a three-dimensional vision device, and reconstruct a grid model of the target area based on the point cloud data; S2. Segment the grid model using a region growing method to extract target region sub-blocks; S3. Flatness evaluation is performed on all facets in the mesh model. A flatness quality index function is constructed based on the facet centroid coordinates, minimum normal vector angle, angle between the normal vector and the Z axis, minimum principal curvature, and maximum and minimum principal curvature differences. Facets that meet the threshold are selected as guide facets for grasping. S4, determining a grasping guidance benchmark based on the target region sub-block where the grasping guidance surface element is located; S5, generating a manipulator grasping path according to the grasping guidance reference, and performing a grasping operation through a closed-loop control system; S6. During the grasping process, the contact force between the manipulator and the target object is monitored in real time, and the grasping force is dynamically adjusted to maintain a stable grasp; S7. When a collision risk is detected, a safety control mechanism is activated to ensure operational safety.

2. A three-dimensional vision-based manipulator grasping guidance and control method according to claim 1, characterized in that: The method for acquiring point cloud data and reconstructing a grid model in step S1 includes the following steps: S1.

1. Collect initial point cloud data using an RGB-D camera or laser scanner. S1.

2. Setting a two-dimensional search criterion based on pre-stored object surface feature data to extract a point cloud subset of the target area; the object surface feature data includes the size, texture, material, and color of the geometric surface; S1.

3. Construct a target search tree using an octree algorithm, perform a depth-first search on the point cloud subset, and generate a point cloud index; S1.

4. Reconstruct a continuous 3D mesh model based on the index using Poisson reconstruction or Delaunay triangulation algorithm.

3. The three-dimensional vision-based manipulator grasping guidance and control method according to claim 2, characterized in that: The specific process of the octree depth-first search in step S1.3 includes the following steps: S1.3.

1. Set the regional point cloud data and select a point as the target point; S1.3.

2. With the target point as the center, search for adjacent point clouds within the set radius; S1.3.

3. Divide the search area into eight sub-cubes and recursively perform sub-cube search until a preset depth is reached; S1.3.

4. Reconstruct the dense point cloud data of the target area through the index mapping relationship.

4. The three-dimensional vision-based manipulator grasping guidance and control method according to claim 1, characterized in that: The implementation of the region growing method in step S2 includes the following steps: S2.

1. Calculate the surface normal vector and curvature parameters of each facet in the 3D mesh model; S2.

2. Generate contour lines based on the difference in normal vector orientation; S2.

3. Split the mesh model according to the normal vector orientation, and gradually enlarge the model by setting the step size until the surface normal vector and curvature meet the boundary characteristics; S2.

4. The target area sub-block that meets the boundary characteristics is taken as the segmentation result.

5. The three-dimensional vision-based manipulator grasping guidance and control method according to claim 1, characterized in that: The specific formula of the panel flatness quality index function in step S3 is: ； Among them, C is the coordinate of the centroid of the surface element, is the minimum normal vector angle of the surface element, is the angle between the surface element normal vector and the Z axis, is the minimum principal curvature of the surface element, is the difference between the maximum and minimum principal curvatures of the surface element, to is the weight coefficient.

6. The three-dimensional vision-based manipulator grasping guidance and control method according to claim 1, characterized in that: The bin selection rules in step S3 include: the bin centroid is located in the target area sub-block, the minimum normal vector angle and the angle between the normal vector and the Z axis are all less than the set threshold, the minimum principal curvature Greater than the set threshold, and the maximum and minimum principal curvature differences Less than the set threshold.

7. The three-dimensional vision-based manipulator grasping guidance and control method according to claim 1, characterized in that: The method for determining the grasping guide reference in step S4 comprises the following steps: S4.

1. Based on the bin planarity sequence, select the first N bins with the largest planarity; S4.

2. Calculate the areas of the triangular facets corresponding to the N facets, and select facets whose areas are greater than a set threshold; S4.

3. Use the sub-block of the target area where the filtered surface element is located as a grasping guidance benchmark.

8. The three-dimensional vision-based manipulator grasping guidance and control method according to claim 1, characterized in that: The closed-loop control system in step S5 includes the following modules: A feedforward control module, wherein the feedforward control module is used to compensate for errors in the manipulator dynamics model; a feedback control module that adjusts gripping force based on force / torque sensor data; A redundant degree of freedom coordination module is provided, wherein the redundant degree of freedom coordination module optimizes the motion smoothness of the multi-joint manipulator.

9. The three-dimensional vision-based manipulator grasping guidance and control method according to claim 1, characterized in that: The dynamic adjustment of the gripping force in step S6 includes the following steps: S6.

1. Real-time monitoring of the contact force between the manipulator end and the target object; S6.2, when the contact force exceeds a set threshold, reducing the grasping force through a PID controller or MPC algorithm; S6.

3. When the contact force is lower than the set threshold, gradually increase the grasping force until a stable grasping state is reached.

10. The three-dimensional vision-based manipulator grasping guidance and control method according to claim 1, characterized in that: The safety control mechanism in step S7 includes immediately initiating emergency braking and switching to a flexible grasping mode when a collision risk between the manipulator and an obstacle or a human body is detected. The flexible grasping mode is achieved by reducing joint stiffness, decreasing grasping speed, and adjusting the contact force threshold.

Images