Robot dynamic grabbing control method based on visual sense and tactile sense depth fusion

Abstract

The invention relates to a robot dynamic grabbing control method based on visual sense and tactile sense depth fusion. The method comprises the following steps that firstly, based on visual information, surface geometric features of a target object are extracted, grabbing adaptability scores are calculated, an optimal grabbing point is selected, and a collision-free approaching track is planned; secondly, in a contact establishment stage, detecting a contact event through a touch sensor, fusing vision and touch data to unify a coordinate system, calculating a vision-touch consistency score, and applying an initial grabbing force; and thirdly, in the stable holding stage, slip detection is conducted through wavelet packet energy entropy and pressure gradient, and the grabbing force and impedance parameters are dynamically adjusted in combination with self-adaptive impedance control. According to the method, intelligent control over the whole process from approaching to contact to stable holding is achieved, and the adaptability, stability and safety of robot grabbing in the dynamic environment are improved.

Description

Technical Field

[0001] This invention belongs to the field of robotics, specifically relating to a dynamic grasping control method for robots based on deep fusion of vision and touch. Background Technology

[0002] Traditional robotic grasping methods often rely on either visual or tactile information alone, making them ill-suited for grasping tasks in complex and dynamic environments. While visual information provides object position and orientation, it fails to perceive force changes and slippage during contact; tactile information, while reflecting contact state, lacks an understanding of the object's global geometry. Furthermore, existing methods suffer from low registration accuracy and response lag in fusing visual and tactile data, leading to slippage, unstable grasping forces, and even object damage during grasping. Especially in unstructured environments, object shape, surface characteristics, and external interference can all affect grasping success rates. Therefore, there is an urgent need for a control method that can deeply integrate visual and tactile information, perceive contact state in real time, and adaptively adjust grasping strategies. Summary of the Invention

[0003] This invention discloses a robot dynamic grasping control method based on visual-tactile deep fusion, which includes the following steps: Step S1: Visual-guided gripping point selection and trajectory planning: Based on visual information, acquire point cloud data of the target object, extract surface geometric features and calculate gripping adaptability score, select the optimal gripping point and posture, calculate contact compensation amount, and plan a collision-free approach trajectory. Step S2: Multimodal fusion control during the contact establishment phase: Detect contact events using tactile sensors, establish a coordinate system of visual and tactile data, calculate a visual-tactile consistency score, and apply an initial grasping force based on the score; Step S3: Slip detection and adaptive impedance control during the holding phase: During the holding phase, the tactile pressure signal is analyzed in real time, and slip detection is performed using wavelet packet energy entropy and pressure gradient. Based on the slip detection results, the gripping force and impedance control parameters are adaptively adjusted.

[0004] Specifically, in step S1, the formula for calculating the grasping adaptability score is: , where K g R is the Gaussian curvature, H is the mean curvature, and R is the mean curvature. finger σ is the width of the gripper contact surface, ∇S is the surface gradient, and σ is the... g This is the gradient sensitivity parameter.

[0005] Specifically, in step S1, the formula for calculating the contact compensation amount is: , , where F N To predict the gripping force, R objectLet E be the local radius of curvature of the object at the grasping point. * It is the equivalent elastic modulus.

[0006] Specifically, in step S1, the approach trajectory is a Bézier curve trajectory.

[0007] Specifically, in step S2, the visual and tactile coordinate system one is achieved by solving the following optimization problem: Where T is the transformation to be determined, and D vis For visual observation of the deformable field, D tac Let x be the deformation field for tactile measurement, α be the regularization parameter, ▽ be the regularization term, and x be the deformation field for tactile measurement. i Let be the coordinates of the i-th sparse feature point in the visual coordinate system.

[0008] Specifically, in step S2, the formula for calculating the visual-tactile consistency score is: , of which M vis For visual contact area mask, P tac Let Ω be the pressure value of the tactile sensor at point (x,y), and Ω be the integration region.

[0009] Specifically, in step S3, the slip detection includes: performing N-level wavelet packet decomposition on the tactile pressure signal to obtain 2 N For each sub-band, calculate the energy distribution probability p of each sub-band. k Then calculate the slip energy entropy. Simultaneously, the gradient ∇P of the pressure distribution is calculated, and the fusion is used to obtain the slip detection value. , where λ and μ are weighting coefficients.

[0010] Specifically, in step S3, the adjustment amount of the gripping force based on the slip detection value is: K p and K i These are control parameters.

[0011] Specifically, in step S3, the adaptive adjustment of the impedance control parameters includes adjusting the stiffness K. d and damping B d .

[0012] Specifically, adjust the stiffness K d and damping B d The formula is: K d0 and damping B d0 Let κ and ξ be the initial impedance parameters, and κ and ξ be the adjustment coefficients.

[0013] Beneficial technical effects: This method effectively integrates visual geometric information and tactile force information through heterogeneous data registration and consistency scoring mechanisms, improving the perception accuracy and decision reliability during the grasping process; it introduces wavelet packet energy entropy and pressure gradient features to detect object slippage in real time, and combines adaptive impedance control to dynamically adjust grasping force, stiffness, and damping parameters, significantly improving the stability and safety of grasping; it forms a closed-loop control from grasping point selection, trajectory planning, contact establishment to holding and releasing, enhancing the robot's grasping adaptability and robustness in dynamic and unstructured environments; through pre-deformation compensation and multimodal feedback, it reduces the number of repeated attempts and adjustments, improving the success rate and efficiency of grasping, and is suitable for various application scenarios such as precision operation and logistics sorting. Attached Figure Description

[0014] Appendix Figure 1 The flowchart is a flowchart of the robot dynamic grasping control method based on visual-tactile depth fusion according to the present invention.

[0015] Appendix Figure 2 This is a schematic diagram of a grasping operation using the method of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] like Figure 1 As shown, this invention discloses a robot dynamic grasping control method based on visual-tactile deep fusion, which includes: Step 1: Visually guided gripping point selection and trajectory planning: Determine the optimal gripping position and posture based on visual information, and plan a safe approach trajectory.

[0018] Point cloud data of the target object is acquired using a depth camera. Surface geometric features of the target object, including surface curvature and normal vectors, are extracted from the point cloud data. Gaussian curvature Kg and average curvature H are calculated. A gripping adaptability score is then calculated based on the gripper's geometric parameters. K g R is the Gaussian curvature, calculated from point cloud data, reflecting the inherent curvature of the surface; H is the average curvature, calculated from point cloud data, reflecting the external curvature of the surface; and R is the mean curvature. finger σ is the width of the gripper contact surface, a known mechanical parameter; ∇S is the surface gradient, used to characterize the degree of surface inclination, calculated from point cloud data; σ is... gThis is a gradient sensitivity parameter used to control the weight of gradient influence. The score combines local curvature (affecting contact stability) and surface gradient (affecting the grasping approach direction).

[0019] Select the area with the highest grasping adaptability score as the grasping point, and calculate the grasping posture, that is, the direction of the gripper approaching is aligned with the surface normal vector of the grasping point.

[0020] Calculate the contact compensation amount: , , where F N To predict the gripping force, R is estimated based on the object's weight and the coefficient of friction. object E represents the local radius of curvature of the object at the grasping point, which can be obtained from point cloud fitting. * The equivalent elastic modulus is determined by the material of the object and the gripper. The contact compensation amount is used for fine-tuning the gripping trajectory. In subsequent gripping control, the pre-deformation is compensated by adjusting the gripping point position to ensure contact stability.

[0021] Generate a collision-free Bézier curve trajectory connecting the current position to the gripping point, and control the robot to accurately approach the target along the planned trajectory.

[0022] Step 2: Multimodal fusion control during the contact establishment phase: At the instant the robot gripper contacts the object, tactile signals are used to detect the contact, a visual-tactile consistency score is calculated, and an initial gripping force is established.

[0023] A tactile sensor monitors pressure distribution in real time. When a sudden increase in pressure is detected, a contact event is triggered. Heterogeneous data registration based on physical constraints is established to unify the visual and tactile coordinate systems, ensuring consistency between the visual point cloud data (including object surface geometry) and the contact pressure distribution data obtained from tactile measurements. Specifically, the surface geometry (normal vector, curvature) of the contact area is extracted from the visual point cloud, and the center and direction of action of the contact area are extracted from the tactile pressure distribution. An optimization problem is established to find the optimal transformation T that aligns the tactile contact area with the visual surface. The objective function for optimization is... Then calculate the visual-tactile consistency score. D vis Let D be the deformable field as observed visually (or zero if there is no deformation). tac For the deformation field of tactile measurement, M vis This is a visual contact area mask, with a value of 1 within the contact area and 0 otherwise. (P) tacLet Ω be the pressure value of the tactile sensor at point (x, y), α be the regularization parameter, ▽ be the regularization term to ensure that the transformation T is not excessively distorted, Ω be the integration region, i.e., the entire contact area, and (x, y) be the coordinates within the contact area. If the visual-tactile consistency score exceeds a predetermined threshold, a preset initial grasping force is applied; otherwise, the position is adjusted to retry the contact or an alarm is triggered. The grasping force is obtained through an impedance control law.

[0024] Step 3: Slip Detection and Adaptive Impedance Control during the Grasping and Holding Phase: During the grasping and holding phase, object slippage is detected in real time, and grasping parameters are adaptively adjusted to maintain stable grasping. Details are as follows: 1. Slip Detection: The tactile pressure time series, i.e., the tactile pressure signal P(t), is analyzed using the wavelet packet energy entropy method. The tactile pressure signal P(t) is decomposed into N layers of wavelet packets to obtain 2... N For each subband, calculate the energy and energy distribution probability p of each subband. k Then calculate the slip energy entropy: Simultaneously, the gradient ▽P of the pressure distribution is calculated as an auxiliary slip index, thus obtaining the multi-feature slip fusion value: , where λ and μ are weighting coefficients.

[0025] 2. Adaptive Force Control: Based on slip detection, the impedance parameters and gripping force are adjusted. Let the current gripping force be F. current The current gripping force is obtained through impedance control. The new gripping force command is: F new =F current +ΔF, adjustment amount is K p and K i To control parameters, impedance parameters are adjusted based on contact conditions to maintain compliance while ensuring the gripping force remains within a safe range, not exceeding the object damage threshold or the robot's maximum gripping force. Impedance parameters include stiffness K. d and damping B d The impedance parameter adjustment specifically involves reducing K when the contact force changes drastically. d To increase compliance; when the contact force is stable, increase K. d To improve gripping stiffness; when slippage is detected, increase B d Damped vibration. The adjustment formula can be designed as follows: K d0 and damping B d0 Let κ and ξ be the initial impedance parameters, and κ and ξ be the adjustment coefficients.

[0026] When the object being grasped is moved to the designated position and needs to be released, the robot can control the gripper to open and ensure a smooth release process.

[0027] like Figure 2As shown, the method of this invention exhibits superior gripping performance for workpieces that are difficult to grasp, such as spherical, ellipsoidal, and smooth surfaces. During workpiece gripping, the system accurately identifies geometric features through visual perception, automatically locates the optimal gripping point, and utilizes the compliant characteristics of impedance control to adaptively conform to the curved surface, ensuring uniform distribution of gripping force along the normal direction and effectively avoiding local stress concentration. Through the deep integration of high-frequency tactile monitoring and visual analysis, early identification and rapid compensation of minute slip signals are achieved, maintaining stable gripping even under low friction coefficient conditions. Compared with traditional methods, this method overcomes the limitations of pre-programming, achieving online adaptive gripping of arbitrary smooth surfaces. It exhibits extremely strong robustness and reliability under dynamic working conditions, effectively solving problems such as slippage and damage in gripping smooth curved workpieces, and providing an innovative technical solution for precision manufacturing and automated assembly.

[0028] In the several embodiments provided by this invention, it should be understood that the disclosed methods can be implemented in other ways. For example, the embodiments described above are merely illustrative. It will be apparent to those skilled in the art that this invention is not limited to the details of the above exemplary embodiments, and that it can be implemented in other specific forms without departing from the basic characteristics of the invention. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this invention and not to limit it. Although this invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this invention without departing from the spirit and scope of the technical solutions of this invention.

Claims

1. A robot dynamic grasping control method based on visual-haptic depth fusion, characterized in that, The method comprises the following steps: Step S1: visual-guided grasp point selection and trajectory planning: obtaining point cloud data of the target object based on visual information, extracting surface geometric features and calculating grasp adaptability score, selecting the optimal grasp point and pose, calculating contact compensation, and planning a collision-free approach trajectory; Step S2: multi-modal fusion control in the contact establishment stage: detecting the contact event using a tactile sensor, establishing the coordinate system of the visual and tactile data, calculating the visual-tactile consistency score, and applying the initial grasping force based on the score; Step S3: slip detection and adaptive impedance control in the grasp holding stage: in the holding stage, real-time analysis of the tactile pressure signal is performed, slip detection is performed through wavelet packet energy entropy and pressure gradient, and the grasping force and impedance control parameters are adaptively adjusted based on the slip detection result.

2. The method of claim 1, wherein, The formula for calculating the gripping adaptability score in step S1 is: where K g is the Gaussian curvature, H is the mean curvature, R finger is the gripper contact surface width, ∇S is the surface gradient, and σ g is the gradient sensitivity parameter.

3. The method of claim 1, wherein, The formula for calculating the contact compensation amount in step S1 is: , where F N is the predicted gripping force, R object is the local radius of curvature of the object at the gripping point, and E * is the equivalent elastic modulus.

4. The method of claim 1, wherein, In the step S1, the approach trajectory is a Bezier curve trajectory.

5. The method of claim 1, wherein, In the step S2, the visual and tactile coordinate systems are unified by solving the following optimization problem: where T is the transformation to be solved, D vis is the deformation field of visual observation, D tac is the deformation field of tactile measurement, and a is a regularization parameter, and ▽ is a regularization term.

6. The method of claim 5, wherein, The formula for calculating the visual-haptic consistency score in step S2 is: wherein M vis is the visual contact area mask, P tac is the pressure value of the haptic sensor at point (x, y), and Ω is the integration region.

7. The method of claim 1, wherein, The step S3 includes: performing N-layer wavelet packet decomposition on the tactile pressure signal to obtain two N subbands, calculating energy distribution probability p k of each subband, and then calculating slip energy entropy Meanwhile, the gradient ∇P of the pressure distribution is calculated, and the slip detection value is obtained by fusion , wherein λ and μ are weight coefficients.

8. The method according to claim 1 or 7, characterized in that, In the step S3, the slip detection value-based grip force adjustment amount is: where K p and K i are control parameters.

9. The method of claim 1, wherein, In the step S3, the adaptive adjustment of the impedance control parameters comprises adjusting the stiffness K d and the damping B d .

10. The method of claim 9, wherein, Adjusting stiffness K d and damping B d The formula is: where K d0 and damping B d0 are initial impedance parameters, and κ and ξ are adjustment coefficients.

Images