Obstacle-avoiding stable grabbing method for any object in disordered scene

Abstract

The invention discloses an obstacle avoidance stable grabbing method for any object in a disordered scene, and the method comprises the steps: carrying out the obstacle avoidance grabbing pose planning of any object in a disordered environment before grabbing; in the grabbing process, tactile force prediction and self-adaptive clamping control are carried out. Before grabbing, obstacle avoidance grabbing pose planning for any object in the disordered environment mainly can be divided into two sub-tasks of target object segmentation and grabbing pose determination. In the grabbing process, gradient variance, gravity center displacement and wavelet entropy features of a pressure matrix are extracted in real time by using a multi-dimensional tactile feature fusion method, and a grabbing state is divided into three modes including a stable mode, a sliding mode and a deformation mode in combination with a support vector machine (SVM) classification model. And a double-PID adaptive control strategy is adopted for different states. The grabbing efficiency and safety can be effectively balanced, and objects are prevented from being damaged or slipping off.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot operation and computer vision, and particularly relates to a cluttered object segmentation and obstacle avoidance grasping strategy on a plane. The scheme aims to solve the technical problem of efficient and accurate target object grasping by robots in complex environments, especially considering the diversity of objects, the complexity of the environment, and the mutual influence of target objects and surrounding obstacles. BACKGROUND

[0002] With the continuous development of artificial intelligence and robot technology, robots are increasingly widely used in various fields, especially in automated production, logistics distribution, intelligent manufacturing, and home service, and the tasks of robots are gradually transformed to more complex and variable scenarios. In these tasks, the grasping of target objects in cluttered scenes is one of the core problems of robot technology. SUMMARY

[0003] The present application provides a stable grasping scheme for any object in a cluttered environment on a plane, which is specifically implemented by the following technologies:

[0004] The present application mainly includes two parts: obstacle avoidance grasping pose planning for any object in a cluttered environment before grasping; and tactile force prediction and adaptive clamping control during grasping.

[0005] Before grasping, obstacle avoidance grasping pose planning for any object in a cluttered environment mainly includes two sub-tasks: target object segmentation and grasping pose determination. Target object determination mainly uses mouse clicks on the picture of the object to be grasped, and implements fast segmentation of the target object on the picture through a lightweight MobileSAM model, combined with a multi-level mask screening strategy verification mechanism to improve the segmentation accuracy in cluttered scenes. Then the contour line of the object is extracted, and the contour line is made smoother using B-spline-based curve fitting, and the tangent lines are calculated at equal distances on the contour line. By constructing an adjustable region of interest (ROI) and depth gradient analysis, the distribution of obstacles is perceived, the normal line is calculated using the tangent line, combined with bidirectional collision detection in the normal direction and length dynamic scaling strategy, the optimal grasping line is selected which is safe and reliable. The midpoint of the intersection line of the bidirectional extension of the normal line and the contour is taken as the grasping point. Finally, through a camera-robot joint calibration model, the grasping pose in the image coordinate system is mapped to the robot base coordinate system, and the precise pose planning of the end effector is realized.

[0006] During the grabbing process, a multi-dimensional tactile feature fusion method is used to extract the gradient variance, barycenter displacement and wavelet entropy features of the pressure matrix in real time, and a support vector machine (SVM) classification model is combined to divide the grabbing state into three modes: stable, sliding and deformation. A double-PID adaptive control strategy is adopted for different states: a high-response proportional coefficient is used to quickly increase the clamping force in the sliding state; in the deformation state, the low-gain parameter is gradually adjusted to the safety threshold; and in the stable state, the clamping force is maintained and continuously monitored. This strategy can effectively balance the grabbing efficiency and safety, and avoid object damage or slipping.

[0007] In a cluttered scene, stable grabbing of any object involves the problems of robust segmentation of the target object, obstacle avoidance grabbing pose planning for any object, and real-time prediction and control of appropriate grabbing force. The present invention aims to achieve efficient grabbing through the following main steps: Step 1: Target object segmentation. Click on the picture with the mouse to select the object to be grabbed, and use the lightweight MobileSAM model to quickly segment the target object in the picture. Combine a multi-level mask screening strategy verification mechanism to improve the segmentation accuracy in a cluttered scene. Step 2: Grabbing pose planning. Extract the contour line of the object and use B-spline-based curve fitting to make the contour line smoother. Calculate the tangent lines at equal distances on the contour line. By constructing an adjustable region of interest (ROI) and depth gradient analysis, perceive the obstacle distribution, calculate the normal line using the tangent line, and combine the normal direction bidirectional collision detection and length dynamic scaling strategy to select the optimal safe and reliable grabbing line. The midpoint of the intersection line of the extended normal line and the contour is taken as the grabbing point. Finally, through the camera-robot joint calibration model, the grabbing pose in the image coordinate system is mapped to the robot base coordinate system to achieve accurate pose planning of the end effector. Step 3: Grabbing state recognition. Use a multi-dimensional tactile feature fusion method to extract the gradient variance, barycenter displacement and wavelet entropy features of the pressure matrix in real time, and combine a support vector machine (SVM) classification model to divide the grabbing state into three modes: stable, sliding and deformation. Step 4: Adaptive clamping control. A double-PID adaptive control strategy is adopted for different states: a high-response proportional coefficient is used to quickly increase the clamping force in the sliding state; in the deformation state, the low-gain parameter is gradually adjusted to the safety threshold; and in the stable state, the clamping force is maintained and continuously monitored. This strategy can effectively balance the grabbing efficiency and safety, and avoid object damage or slipping.

[0008] The method designed in this invention has the following three key advantages: 1. By jointly optimizing the MobileSAM model and the depth map, the robustness of object segmentation in complex scenes is significantly improved; 2. By combining depth gradient analysis and morphological processing, an obstacle map is constructed, which enhances the safety of the gripping line generation; 3. By utilizing multi-dimensional tactile features and a state classification model, dynamic adaptive adjustment of the gripping force is achieved, breaking through the limitations of traditional static threshold control. The accuracy of state recognition is improved through the SVM classification model, and the dual PID control strategy achieves fine-grained gripping force adjustment.

[0009] Experiments show that the solution of this invention can efficiently generate safe grasping poses in cluttered planar scenes and achieve real-time, stable dynamic gripping control. Its lightweight design is suitable for embedded device deployment and has broad application prospects in fields such as industrial sorting and home service robots. Attached Figure Description

[0010] Figure 1 This is a flowchart of the overall workflow of the present invention.

[0011] Figure 2 This is a diagram showing the segmentation effect of an object.

[0012] Figure 3 This is a preview of the candidate capture line.

[0013] Figure 4 This is a diagram showing the effect of obstacle detection.

[0014] Figure 5 This is the final result image showing the determined gripping points and gripping lines.

[0015] Figure 6 This is a flowchart of adaptive clamping control.

[0016] Figure 7 This is a diagram illustrating the effect of clamping an object. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings.

[0018] Figure 1 This is the overall flowchart of the obstacle avoidance and grasping scheme for arbitrary objects in a chaotic environment under a plane proposed in this invention. Specifically, it can be divided into two parts: (1) determination of grasping points and grasping lines; (2) design of adaptive clamping control.

[0019] The determination of the gripping point and gripping line mainly includes two parts: the segmentation of the selected object and the determination of the gripping point and gripping line.

[0020] Object segmentation stage: First, the MobileSAM model was used to segment the clicked object and obtain the mask image of the object to be captured.

[0021] This invention further improves the accuracy of mask images through confidence thresholding and depth map verification methods. The confidence level of the mask image provided by the MobileSAM model can be used for initial screening of the obtained masks. This initial screening removes masks with obvious segmentation errors, leaving mostly masks that are basically accurate in segmentation or have some defects. From these remaining masks, a 3D screening algorithm is used in conjunction with the depth map to select the most accurate mask. The method for selecting the correct mask for the target object is as follows:

[0022] (1)

[0023] (2)

[0024] (3)

[0025] in, This is the set of valid candidate masks selected by formula (2); for A single candidate mask in; For mask The set of outline pixels (i.e., the set of boundary coordinate points); The depth gradient is calculated using formula (3); This is the threshold for deep mutation; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. The set of masks output by the MobileSAM model; For mask The confidence score (derived from MobileSAM's mask quality prediction); This indicates the criteria for determining the existence of high-quality masks in the set, which is used to qualify "suboptimal" masks to participate in the competition; For depth map in pixels The depth value; For selection from neighborhood set Horizontal and vertical offsets; This is the set of neighboring pixel offsets used for comparison. Figure 2 The left image is a cluttered scene described by a 2D image, where the green dot is the midpoint of the target object clicked by the mouse. The right image is a mask image of the target object segmented from the image.

[0026] In the stage of determining the capture points and capture lines: the mask contour is obtained using the selected target object mask image, and spline interpolation is performed on the contour to achieve a smooth contour effect. Based on this, several tangent lines are calculated at equal intervals along the contour, and normal lines are drawn for these tangent lines. For example... Figure 3 As shown, black represents the obtained tangent line, and purple represents the obtained normal line, i.e., the candidate grab line. This can be expressed by the formula:

[0027] (4)

[0028] in, The normalized "progress bar" along the curve is represented by 0, where 0 indicates the first point and 1 indicates the last point. For the curve in parameters Cartesian coordinates at the location; For the first One B-spline basis function; For the original contour point set; Let B be the order of the spline. This represents the upper limit of the control point sequence number.

[0029] (5)

[0030] (6)

[0031] The tangent vector indicates the direction of the curve. The unit normal vector is the tangent line. The perpendicular vector.

[0032] Since the gripper needs to avoid obstacles around the target object during the gripping process to achieve obstacle avoidance, it is necessary to accurately identify and determine the location of obstacles. In this invention, depth information within the Region of Interest (ROI) is used to determine obstacles, and the effect is as follows: Figure 4 As shown, red represents obstacles, and blue represents the target object. The specific judgment method is as follows:

[0033] (7)

[0034] (8)

[0035] (9)

[0036] (10)

[0037] in, These are the pixel coordinates in the depth map; , The centroid coordinates of the target object are calculated using profile moments. The size of the circular ROI is determined by the detection radius; For depth map in The depth value at that location; A set of pixels for a circular ROI; The average depth within the ROI; Depth difference threshold; For the set of candidate obstacle pixels; For morphological dilation operators; It has a square-shaped core. This represents the number of expansion iterations.

[0038] The gripper's gripping position can be simplified to the normal to the tangent on the contour. Converting the gripper width to the normal length allows us to approximate the gripper as such. In the gripping strategy, choosing a shorter gripping position improves the gripping action; therefore, it's necessary to calculate and select the two contact points between the gripper and the target object. Simultaneously, gripping closer to the centroid ensures the object remains stable during gripping, preventing tipping or falling. Therefore, finding a gripping position closer to the target object's centroid is crucial. Once the gripping line is determined, the midpoint of its intersection with the target object's contour is the gripping point. Using coordinate space transformation, the end effector position is calculated from the gripping point, and the end effector rotation angle is calculated from the gripping line slope, thus achieving gripping of the target object in a plane. The results of obtaining the gripping point and gripping line from the algorithm in this invention are as follows: Figure 5 As shown, the specific steps are as follows:

[0039] (11)

[0040] (12)

[0041] (13)

[0042] For the first A single normal feature, including the coordinates of the starting point. and normal direction ; The coordinates of the centroid of the target object; The point where the normal line intersects the target contour; The Euclidean distance between the left and right intersection points; Center of mass The perpendicular distance to this normal. , The maximum value from the current candidate set is used for normalization; Weights for the evaluation function; This is the set of candidate normals after "short width filtering"; for Value quantiles; The goal is to select the one with the smallest score from the remaining set. For collision detection between the extended normal and the obstacle.

[0043] Design of adaptive clamping control: such as Figure 6 As shown, the adaptive clamping control unit achieves accurate recognition of the grasping state and adaptive adjustment of the clamping force through multi-dimensional tactile feature fusion, support vector machine state classification, and dual PID dynamic control strategy.

[0044] Multi-dimensional feature extraction method: The pressure matrix is ​​acquired in real time using a 4×4 pressure sensor array. After Gaussian filtering and smoothing to eliminate noise interference, a preprocessed matrix is ​​obtained. The Sobel operator is used to calculate the gradient of the pressure matrix in the x and y directions. The gradient variance is calculated using the gradient direction angle to reflect the uniformity and directionality of the pressure distribution. Simultaneously, the centroid coordinates of the current pressure matrix and the previous time step are calculated to obtain the centroid displacement, representing the dynamic change of the object's contact position. The pressure matrix is ​​decomposed, and the fuzzy entropy of each layer is extracted to capture the multi-scale complexity features of the pressure signal. The gradient variance, centroid displacement, wavelet entropy, and features such as the mean, variance, standard deviation, and total pressure of the pressure matrix are fused into a 10-dimensional feature vector. The specific process is shown below:

[0045] (14)

[0046] (15)

[0047] (16)

[0048] The fuzzy entropy of the approximation coefficient is: ,in To capture the multi-scale complexity features of pressure signals by representing probability distributions under different embedding dimensions. The gradient direction angle, The gradient variance reflects the uniformity and directionality of the pressure distribution.

[0049] Support Vector Machine (SVM) State Classification: An SVM classification model is trained using labeled feature data, and the classifier is constructed using the RBF kernel function. The grasping states are divided into three categories: stable (0), sliding (1), and deformed (2). Real-time input feature vectors Compared with the current state This enables accurate identification of the capture status.

[0050] Dual PID dynamic control strategy: The corresponding PID controller is activated based on the SVM classification results. When in a sliding state (S = 1), a high proportional gain sliding controller is used to quickly increase the clamping force to the target value to cope with object slippage. When in a deformation state (S = 2), a low proportional gain deformation controller is used to gradually adjust the clamping force to the safety threshold to avoid excessive compression of the object. When in a stable state (S = 0), the current clamping force is maintained and state changes are continuously monitored to ensure stable object gripping. The gripping effect is as follows:Figure 7 As shown.

[0051] In actual experiments, this invention achieved stable obstacle avoidance and grasping of any object in a cluttered scene on a flat plane, verifying the effectiveness of the algorithm in this invention.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications to the technical solutions described in the foregoing embodiments, or equivalent substitutions for some or all of the technical features, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A stable obstacle avoidance and grasping method for any object in a cluttered scene, characterized in that, It consists of two main parts: before grasping, obstacle avoidance and grasping posture planning for any object in a cluttered environment; during grasping, tactile force prediction and adaptive gripping control. Before grasping, obstacle avoidance and grasping pose planning for any object in a cluttered environment is divided into two sub-tasks: target object segmentation and grasping pose determination. Target object determination involves clicking on the object to be grasped in the image using the mouse, and a lightweight MobileSAM model is used to quickly segment the target object in the image. A multi-level mask screening strategy verification mechanism is combined to improve the segmentation accuracy in cluttered scenes. Then, the contour lines of the object are extracted, and curve fitting based on B-splines is used to make the contour lines smoother. Tangents are calculated at equal intervals on the contour lines. By constructing an adjustable region of interest (ROI) and performing depth gradient analysis, the distribution of obstacles is perceived. Normals are calculated using tangents, and a safe and reliable optimal grasping line is selected by combining bidirectional collision detection in the normal direction and a dynamic length scaling strategy. The midpoint of the intersection of the bidirectionally extended normal and the contour is used as the grasping point. Finally, through a camera-robotic arm joint calibration model, the grasping pose in the image coordinate system is mapped to the robot base coordinate system to achieve accurate pose planning for the end effector. During the grasping process, a multi-dimensional tactile feature fusion method is used to extract the gradient variance of the pressure matrix, the center of gravity displacement, and the wavelet entropy features in real time. Combined with the support vector machine (SVM) classification model, the grasping state is divided into three modes: stable, sliding, and deformation. A dual-PID adaptive control strategy is adopted for different states: in the sliding state, a high response proportional coefficient is used to quickly increase the clamping force; in the deformation state, the low gain parameter is used to gradually adjust to the safety threshold; in the stable state, the clamping force is maintained and continuously monitored. This strategy can effectively balance grasping efficiency and safety, and avoid damage or slippage of the object.

2. The obstacle avoidance and stable grasping method for any object in a cluttered scene according to claim 1, characterized in that, The process includes the following steps: Step 1: Object segmentation; The object to be captured is selected by clicking on it in the image, and a lightweight MobileSAM model is used to quickly segment the target object. A multi-level mask filtering strategy is combined with a verification mechanism to improve the segmentation accuracy in cluttered scenes. Step 2: Capture pose planning; The object's contour is extracted, and B-spline-based curve fitting is used to smooth the contour. Tangents are calculated at equal intervals along the contour. An adjustable Region of Interest (ROI) is constructed, and depth gradient analysis is performed to perceive obstacle distribution. Normals are calculated using tangents, and a bidirectional collision detection and dynamic length scaling strategy is used to select the safe and reliable optimal capture line. The intersection of the bidirectionally extended normal with the contour is then calculated. The midpoint is used as the grasping point; finally, through the camera-robotic arm joint calibration model, the grasping pose in the image coordinate system is mapped to the robot base coordinate system to achieve accurate pose planning of the end effector; Step 3: Grasping state recognition; using a multi-dimensional tactile feature fusion method, the gradient variance of the pressure matrix, the center of gravity displacement, and the wavelet entropy features are extracted in real time. Combined with the support vector machine (SVM) classification model, the grasping state is divided into three modes: stable, sliding, and deformation; Step 4: Adaptive gripping control; a dual PID adaptive control strategy is adopted for different states: in the sliding state, a high response proportional coefficient is used to quickly increase the gripping force; in the deformation state, the low gain parameter is used to gradually adjust to the safe threshold; in the stable state, the gripping force is maintained and continuously monitored.

3. The obstacle avoidance and stable grasping method for any object in a cluttered scene according to claim 1, characterized in that, First, the MobileSAM model is used to segment the clicked object to obtain a mask image of the object to be captured. The obtained masks are initially screened using the confidence score of the mask image provided by the MobileSAM model. This initial screening removes masks with obvious segmentation errors, leaving mostly masks that are basically accurate or have minor segmentation issues. These remaining masks are then further refined using a 3D filtering algorithm combined with the depth map to select the most accurate mask. The method for selecting the correct mask for the target object is as follows: （1） （2） （3） in, This is the set of valid candidate masks selected by formula (2); for A single candidate mask in; For mask The set of outline pixels is the set of boundary coordinate points; The depth gradient is calculated using formula (3); This is the threshold for deep mutation; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. The set of masks output by the MobileSAM model; For mask Confidence score; This indicates the criteria for determining the existence of high-quality masks in the set, which is used to qualify "suboptimal" masks to participate in the competition. For depth map in pixels The depth value; For selection from neighborhood set Horizontal and vertical offsets; This is the set of neighboring pixel offsets used for comparison.

4. The obstacle avoidance and stable grasping method for any object in a cluttered scene according to claim 3, characterized in that, The mask contour is obtained using the selected target object mask image, and spline interpolation is performed on the contour to achieve a smooth contour effect; expressed by the formula: （4） in, The normalized "progress bar" along the curve is represented by 0, where 0 indicates the first point and 1 indicates the last point. For the curve in parameters Cartesian coordinates at the location; For the first One B-spline basis function; This is the original set of contour points; Let B be the order of the spline. This represents the upper limit of the control point sequence number; （5） （6） The tangent vector indicates the direction of the curve. The unit normal vector is the tangent line. The perpendicular vector; Obstacles are identified using depth information within the Region of Interest (ROI). The method is as follows: （7） （8） （9） （10） in, These are the pixel coordinates in the depth map; , The centroid coordinates of the target object are calculated using profile moments. The size of the circular ROI is determined by the detection radius; For depth map in The depth value at that location; A set of pixels for a circular region of interest (ROI); The average depth within the ROI; Depth difference threshold; For the set of candidate obstacle pixels; For morphological dilation operators; It has a square-shaped core. This represents the number of expansion iterations.

5. The obstacle avoidance and stable grasping method for any object in a cluttered scene according to claim 4, characterized in that, The gripper's position is simplified to the normal to the tangent on the contour. Converting the gripper width to the normal length allows us to approximate the gripper as such. Using coordinate space transformation, the end effector position is calculated from the gripping point, and the end effector rotation angle is calculated from the slope of the gripping line, thus achieving gripping of the target object in a plane. （11） （12） （13） For the first A single normal feature, including the coordinates of the starting point. and normal direction ; The coordinates of the centroid of the target object; The point where the normal line intersects the target contour; The Euclidean distance between the left and right intersection points; Center of mass The perpendicular distance to this normal; , The maximum value from the current candidate set is used for normalization; Weights for the evaluation function; This is the set of candidate normals after "short width filtering"; for Value quantiles; The goal is to select the one with the smallest score from the remaining set. For collision detection between the extended normal and the obstacle.

6. The obstacle avoidance and stable grasping method for any object in a cluttered scene according to claim 5, characterized in that, Multi-dimensional feature extraction method: The pressure matrix is ​​acquired in real time by a 4×4 pressure sensor array. After Gaussian filtering and smoothing to eliminate noise interference, a preprocessed matrix is ​​obtained. The gradient of the pressure matrix in the x and y directions is calculated using the Sobel operator. The gradient variance is calculated by the gradient direction angle to reflect the uniformity and directionality of the pressure distribution. The centroid coordinates of the current pressure matrix and the previous time step are calculated to obtain the centroid displacement, which characterizes the dynamic change of the object's contact position. The pressure matrix is ​​decomposed, and the fuzzy entropy of each layer is extracted to capture the multi-scale complexity features of the pressure signal. The gradient variance, centroid displacement, wavelet entropy, and the mean, variance, standard deviation, and total pressure features of the pressure matrix are fused into a 10-dimensional feature vector. The specific process is as follows: （14） （15） （16） The fuzzy entropy of the approximation coefficient is: ,in To capture the multi-scale complexity features of pressure signals by representing probability distributions under different embedding dimensions. The gradient direction angle, The gradient variance reflects the uniformity and directionality of the pressure distribution.

7. The obstacle avoidance and stable grasping method for any object in a cluttered scene according to claim 6, characterized in that, Support Vector Machine (SVM) State Classification: An SVM classification model is trained using labeled feature data, and an RBF kernel function is used to construct the classifier, classifying the grasping state into three categories: stable (0), sliding (1), and deformed (2). Feature vectors are input in real-time. Compared with the current state This enables accurate identification of the capture status.

8. The obstacle avoidance and stable grasping method for any object in a cluttered scene according to claim 7, characterized in that, Dual PID dynamic control strategy: Activate the corresponding PID controller according to the SVM classification result; when in a sliding state, use a high proportional coefficient sliding controller to quickly increase the clamping force to the target value to cope with the object sliding; When in a deformed state, a low-proportion coefficient deformation controller is used to gradually adjust the clamping force to a safe threshold to avoid excessive squeezing of the object; when in a stable state, the current clamping force is maintained and the state changes are continuously monitored to ensure stable gripping of the object.

Images