Mechanical arm autonomous grabbing system and method based on visual guidance and visual touch fusion

Abstract

The invention discloses a mechanical arm autonomous grabbing system and method based on visual guidance and visual touch fusion. The mechanical arm autonomous grabbing system at least comprises a visual semantic guidance and grabbing pose generation module, a smooth grabbing control module based on visual touch fusion and a dynamic planning module based on model prediction path integration. The visual module efficiently generates a six-degree-of-freedom grabbing pose from the point cloud data by using a deep learning network; the smooth grabbing control module monitors object slippage in real time and adaptively adjusts clamping force by fusing visual and tactile information, and stable grabbing of fragile or slippery objects is achieved. And the dynamic planning module adopts a model prediction path integral algorithm MPPI to realize collision-free path planning of the mechanical arm in a complex dynamic environment. The problems that an existing mechanical arm is insufficient in sensing capacity, poor in grabbing robustness and low in path planning efficiency in an unstructured scene are solved, and the autonomous operation capacity and safety of the mechanical arm in the scenes of accompanying, intelligent sorting and the like are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the technical field of artificial intelligence and automatic control, and mainly relates to an autonomous grasping system and method for robotic arms based on visual guidance and visual-touch fusion. Background Technology

[0002] With the increasing global trend of population aging and the growing demand for disability services, service robots, especially robotic arms with manipulative capabilities, are showing great application potential to assist humans in completing daily tasks. However, applying robotic arms to perform caregiving tasks in unstructured and dynamically changing environments such as homes and hospitals has always been a complex and challenging task. Traditional robotic grasping systems often rely on single visual perception, which has many limitations when facing complex scenarios.

[0003] Single-vision-based grasping methods suffer from insufficient recognition and positioning accuracy when handling transparent, reflective, or uniquely textured objects. Furthermore, relying solely on visual information makes it difficult to accurately determine the object's physical properties (such as fragility or slipperiness) and to perceive the real-time contact force and pressure distribution during grasping, easily leading to grasping failures. For example, excessive force may damage fragile items, or insufficient gripping force may cause the object to slip. In addition, in chaotic environments, quickly and accurately generating the optimal six-DOF grasping posture from 3D point cloud data is a major bottleneck for current technologies. Finally, existing robotic systems face difficulties in performing collision-free grasping and delivery in the presence of obstacles, directly impacting operational safety.

[0004] Therefore, existing robotics technologies have shortcomings in environmental perception, object interaction, and operational dexterity, making it difficult to meet the practical needs of refined and high-security scenarios such as daily services. Developing an autonomous system capable of integrating multiple sensory information and achieving compliant and stable object grasping is of great significance for improving the intelligence level and practicality of service robots. Summary of the Invention

[0005] This invention addresses the shortcomings of existing robotic arms, such as low perception accuracy in unstructured environments, poor adaptability to object characteristics, and low grasping success rate. It proposes an autonomous grasping system and method for robotic arms based on visual guidance and visual-touch fusion, comprising at least a visual semantic guidance and grasping pose generation module, a compliant grasping control module based on visual-touch fusion, and a dynamic path planning module based on model-predicted path integral. The visual semantic guidance and grasping pose generation module receives visual information from the environment, identifies and locates the target object, and calculates a six-degree-of-freedom grasping pose from the point cloud data of the target object. The compliant grasping control module based on visual-touch fusion drives the robotic arm to perform grasping according to the grasping pose, and during the grasping process, it integrates feedback information from tactile sensors to evaluate the grasping state in real time and adaptively adjust the gripping force. The dynamic path planning module based on model-predicted path integral plans a collision-free motion trajectory from the current position to the target position after the robotic arm grasps the object. The system achieves precise positioning and planning of the grasping target through the close integration of visual guidance and efficient pose generation. It also solves the problem of compliant grasping of complex objects such as fragile and slippery objects by using visual-touch fusion technology. Furthermore, it realizes trajectory planning by using model prediction path integral. Ultimately, it significantly improves the success rate, safety and intelligence level of the robotic arm's autonomous grasping in unstructured environments.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is: an autonomous grasping system for a robotic arm based on visual guidance and visual-touch fusion, comprising at least a visual semantic guidance and grasping pose generation module, a compliant grasping control module based on visual-touch fusion, and a dynamic path planning module based on model-predicted path integral.

[0007] The visual semantic guidance and grasping pose generation module is used to receive visual information from the environment, identify and locate the target object, and calculate and generate a specified number of six-degree-of-freedom grasping poses from the point cloud data of the target object.

[0008] The compliant grasping control module based on visual-touch fusion is used to drive the robotic arm to perform grasping according to the grasping posture, and during the grasping process, it integrates feedback information from the tactile sensor to evaluate the grasping state in real time and adaptively adjust the gripping force.

[0009] The dynamic path planning module based on model-predicted path integral is used to plan a collision-free motion trajectory from the current position to the target position after the robotic arm grasps an object.

[0010] The system achieves precise positioning and planning of the grasping target through the close integration of visual guidance and efficient pose generation. It also solves the problem of compliant grasping of complex objects such as fragile and slippery objects by using visual-touch fusion technology. Furthermore, it realizes trajectory planning by using model prediction path integral algorithm, which ultimately significantly improves the success rate, safety and intelligence level of the robotic arm's autonomous grasping in unstructured environments.

[0011] As an improvement to the present invention, in the visual semantic guidance and grasping pose generation module...

[0012] First, the YOLOv11 object detection network is used to identify target objects in the image and obtain their two-dimensional bounding boxes;

[0013] The local point cloud of the target object is segmented from the three-dimensional point cloud based on the two-dimensional bounding box.

[0014] The local point cloud is input into an end-to-end six-degree-of-freedom grasping and detection network, which uses a sparse convolutional network as the backbone network to extract the geometric features of the point cloud and directly output the grasping pose.

[0015] The grasping and detection network is an optimized sparse geometry-semantic co-enhancement network. Its core is to combine an adapted sparse convolutional backbone network with a self-designed sparse adaptive channel attention module to improve the model's ability to focus on features.

[0016] The sparse convolutional backbone network is often used to efficiently extract multi-scale geometric features of point clouds and can output point-by-point features for grasping detection.

[0017] The sparse adaptive channel attention module is integrated into the sparse convolutional backbone network to learn and enhance feature channels that are crucial for the grasping task on sparse data, thereby improving the discriminative power of features without significantly increasing the computational burden.

[0018] As an improvement of the present invention, the compliant grasping control module based on visual-touch fusion is used to execute and optimize the physical grasping process after visual guidance is completed, so as to achieve safe and stable grasping of objects with different characteristics, specifically including:

[0019] First, the robotic arm drives the gripper to close toward the target object based on the initial grasping pose provided by the vision module, and performs a pre-grabbing action.

[0020] When the built-in tactile sensor of the gripper detects the initial contact with the object, the system enters the gripping stability verification stage.

[0021] During the verification phase, the robotic arm is controlled to perform a slight lifting or disturbance motion. At the same time, the tactile sensor monitors the pressure distribution or force feedback data on the contact surface with the object in real time at a high sampling frequency to determine whether slippage exists.

[0022] If a signal indicating slippage is detected (such as a sudden drop in pressure or high-frequency vibration), the system will automatically enter a closed-loop feedback control process: control the gripper to gradually increase the clamping force in small steps, and repeat the slippage monitoring after each increase in force. This process is executed cyclically until the slippage signal disappears or decreases below the preset safety threshold.

[0023] At this point, the system locks the current clamping force to the minimum force required to ensure stable gripping and continues to perform subsequent handling actions, thereby achieving adaptive compliant gripping of fragile or smooth-surfaced objects.

[0024] As an improvement of the present invention, the dynamic path planning module based on model-predicted path integral is used to plan and execute an efficient, collision-free motion trajectory from the current position to the target position after the robotic arm successfully grasps the object, specifically including:

[0025] This module uses the Model Predictive Path Integral (MPPI) algorithm as its core to generate motion control commands for the robotic arm in real time.

[0026] Within each control cycle, the MPPI algorithm first generates a large number of candidate control sequences by randomly sampling in parallel within the control time domain, and then predicts multiple candidate motion trajectories driven by these control sequences based on the dynamic model of the robotic arm.

[0027] Subsequently, the system quantitatively evaluates each candidate trajectory according to a predefined cost function, which comprehensively considers various constraints, including but not limited to the risk of collision with dynamic or static obstacles in the environment and the distance to the final target point.

[0028] Next, the algorithm calculates a weight for each trajectory based on its cost value, assigning a higher weight to trajectories with lower cost values.

[0029] Finally, by performing a weighted average of all candidate control sequences, the optimal control command is calculated and applied to the robotic arm. Through parallel computation on the GPU, the entire "sampling-prediction-evaluation-weighting" process can be executed at a high frequency, enabling the robotic arm to respond to environmental changes in real time and achieve dynamic and efficient autonomous obstacle avoidance and path planning.

[0030] To achieve the above objectives, the present invention also adopts the following technical solution: an autonomous grasping method for a robotic arm based on visual guidance and visual-touch fusion, comprising the following steps:

[0031] S1: Visual perception and pose generation: Acquire environmental visual information, identify target objects, and use a grasping detection network to process the point cloud of the target objects to generate a six-degree-of-freedom grasping pose.

[0032] S2: Compliant gripping execution: Drives the robotic arm to perform gripping according to the generated pose, and adjusts the gripping force in real time through visual-touch fusion feedback control to achieve stable gripping;

[0033] S3: Dynamic Path Planning and Movement: After stable capture, the model predicts the path integral algorithm to plan and execute a collision-free motion trajectory to the target point.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] Technically, this system deeply integrates visual semantic guidance and visual-tactile perception technologies. Through an innovative six-degree-of-freedom grasping pose generation network, it achieves efficient and accurate inference of grasping posture from complex 3D scenes. Simultaneously, by introducing closed-loop feedback control based on tactile information, it effectively solves the challenge of compliant grasping of fragile, slippery, and other special objects, which is difficult to handle with traditional vision solutions. This significantly improves the safety and stability of physical interaction and ensures a high success rate for grasping operations.

[0036] In terms of applications, this invention provides service robots such as companion robots with powerful autonomous operation capabilities, enabling them to safely and reliably perform delicate companionship tasks such as assisting with object retrieval and delivery, significantly improving the quality and efficiency of companionship services and reducing the labor intensity of caregivers. Furthermore, this system and method are also applicable to industrial scenarios with high requirements for grasping accuracy and flexibility, such as intelligent manufacturing and logistics sorting, and can improve the automation level of production lines and their adaptability to non-standard objects.

[0037] Its unique technological advantages and wide range of applications will inject new impetus into the development of the service robot and intelligent equipment industry, and have important practical significance for addressing population aging and improving the level of social services. Attached Figure Description

[0038] Figure 1 This is the overall architecture diagram of the system of the present invention;

[0039] Figure 2 This is a flowchart of the visual semantic guidance and pose generation module;

[0040] Figure 3 This is a flowchart of a compliant grasping control module based on visual-touch fusion;

[0041] Figure 4 This is a schematic diagram illustrating the principle of the MPPI dynamic path planning module. Detailed Implementation

[0042] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0043] Example

[0044] An autonomous grasping system for robotic arms based on vision guidance and visual-touch fusion, such as Figure 1 As shown, it includes at least a visual semantic guidance and grasping pose generation module, a compliant grasping control module based on visual-touch fusion, and a dynamic path planning module based on model-predicted path integral.

[0045] Visual semantic guidance and grasping pose generation module: used to receive visual information from the environment, identify and locate target objects, and calculate and generate a specified number of six-degree-of-freedom grasping poses from the point cloud data of the target objects.

[0046] First, the YOLOv11 object detection network is used to identify target objects in the image and obtain their two-dimensional bounding boxes;

[0047] The local point cloud of the target object is segmented from the three-dimensional point cloud based on the two-dimensional bounding box.

[0048] The local point cloud is input into an end-to-end six-DOF grasping and detection network, which uses a sparse convolutional network as the backbone network to extract the geometric features of the point cloud and directly output the grasping pose.

[0049] The compliant gripping control module based on visual-touch fusion is used to drive the robotic arm to perform gripping based on the gripping pose. During the gripping process, it integrates feedback information from the tactile sensors, evaluates the gripping state in real time, and adaptively adjusts the clamping force.

[0050] First, the robotic arm drives the gripper to close toward the target object based on the initial grasping pose provided by the vision module, and performs a pre-grabbing action.

[0051] When the built-in tactile sensor of the gripper detects the initial contact with the object, the system enters the gripping stability verification stage.

[0052] During the verification phase, the robotic arm is controlled to perform a slight lifting or disturbance motion. At the same time, the tactile sensor monitors the pressure distribution or force feedback data on the contact surface with the object in real time at a high sampling frequency to determine whether slippage exists.

[0053] If a signal indicating slippage is detected (such as a sudden drop in pressure or high-frequency vibration), the system will automatically enter a closed-loop feedback control process: control the gripper to gradually increase the clamping force in small steps, and repeat the slippage monitoring after each increase in force. This process is executed cyclically until the slippage signal disappears or decreases below the preset safety threshold.

[0054] At this point, the system locks the current clamping force to the minimum force required to ensure stable gripping and continues to perform subsequent handling actions, thereby achieving adaptive compliant gripping of fragile or smooth-surfaced objects.

[0055] A dynamic path planning module based on model-predicted path integrals is used to plan and execute an efficient, collision-free motion trajectory from the current position to the target position after the robotic arm successfully grasps an object.

[0056] This module uses the Model Prediction Path Integral (MPPI) control algorithm as its core to generate motion control commands for the robotic arm in real time.

[0057] Within each control cycle, the MPPI algorithm first generates a large number of candidate control sequences by randomly sampling in parallel within the control time domain, and then predicts multiple candidate motion trajectories driven by these control sequences based on the dynamic model of the robotic arm.

[0058] Subsequently, the system performs a quantitative evaluation of each candidate trajectory based on a predefined cost function.

[0059] Next, the algorithm calculates a weight for each trajectory based on its cost value, assigning a higher weight to trajectories with lower cost values.

[0060] Finally, by performing a weighted average of all candidate control sequences, the optimal control command is calculated and applied to the robotic arm. Through parallel computation on the GPU, the entire "sampling-prediction-evaluation-weighting" process can be executed at a high frequency, enabling the robotic arm to respond to environmental changes in real time and achieve dynamic and efficient autonomous obstacle avoidance and path planning.

[0061] This invention deeply integrates visual semantic guidance and visual-tactile perception technologies. Through an innovative six-degree-of-freedom grasping pose generation network, it achieves efficient and accurate inference of grasping posture from complex 3D scenes. Simultaneously, by introducing closed-loop feedback control based on tactile information, it effectively solves the challenge of compliant grasping of fragile, slippery, and other special objects, which is difficult to handle with traditional vision solutions. This significantly improves the safety and stability of physical interaction and ensures a high success rate for grasping operations. It can inject new impetus into the development of service robots such as companion robots, as well as intelligent equipment robots in intelligent manufacturing and logistics sorting fields. It also has significant practical implications for addressing population aging, improving social service levels, and enhancing work efficiency.

Claims

1. A mechanical arm autonomous grasping system based on visual guidance and visual-tactile fusion, characterized in that: At least comprising a visual semantic guidance and grasp pose generation module, a compliant grasp control module based on visual-tactile fusion, and a dynamic path planning module based on model predictive path integral, The visual semantic guidance and grasp pose generation module is configured to receive visual information of an environment, identify and locate a target object, and calculate a six-degree-of-freedom grasp pose from point cloud data of the target object, The compliant grasp control module based on visual-tactile fusion is configured to drive a robot arm to perform grasping according to the grasp pose, and in the grasping process, fuse feedback information of a tactile sensor, assess the grasping state in real time, and adjust the clamping force adaptively, The dynamic path planning module based on model predictive path integral is configured to plan a collision-free motion trajectory from a current position to a target position after the robot arm grasps an object.

2. The autonomous robotic grasping system based on visual guidance and vision-haptics fusion of claim 1, wherein: The visual semantic guidance and grasp pose generation module specifically comprises: First, a target detection network YOLOv11 is used to identify the target object in the image and obtain its two-dimensional bounding box; According to the two-dimensional bounding box, a local point cloud of the target object is segmented from the three-dimensional point cloud; The local point cloud is input into an end-to-end six-degree-of-freedom grasp detection network, which uses a sparse convolutional network as a backbone network to extract geometric features of the point cloud and directly output the grasp pose.

3. The visual guidance and tactile fusion based autonomous robotic grasping system of claim 1, wherein: The compliant grasp control module based on visual-tactile fusion is configured to execute and optimize the physical grasping process after visual guidance to achieve safe and stable grasping of objects with different characteristics, specifically comprising: First, the robot arm drives the gripper to close to the target object according to the initial grasp pose provided by the visual module to perform a pre-grasping action; When the tactile sensor built-in the gripper detects initial contact with the object, the system enters a grasp stability verification stage; In the verification stage, the robot arm performs a slight lifting or disturbance action, and the tactile sensor monitors the pressure distribution or force feedback data on the contact surface with the object in real time at a high sampling frequency to determine whether there is slippage; If the slippage signal is monitored, the system automatically enters a closed-loop feedback control process: the gripper gradually increases the clamping force in a small step manner, and after each increase in force, the slippage monitoring is repeated, and the process is repeated until the slippage signal disappears or decreases to below a preset safety threshold; At this time, the system locks the current clamping force as the minimum force to ensure stable grasping, and continues to perform subsequent moving actions, thereby achieving adaptive compliant grasping of fragile or smooth-surface objects.

4. The visual guidance and tactile fusion based autonomous robotic grasping system of claim 1, wherein: The dynamic path planning module based on model predictive path integral is configured to plan and execute an efficient and collision-free motion trajectory from the current position to the target position after the robot arm successfully grasps the object, specifically comprising: The module uses a model predictive path integral control algorithm (MPPI) as the core to generate motion control instructions for the robot arm in real time; In each control cycle, the MPPI algorithm first generates a large number of candidate control sequences by random sampling in parallel within the control horizon, and then predicts a plurality of candidate motion trajectories driven by these control sequences according to the dynamics model of the robot arm; Subsequently, the system quantitatively evaluates each candidate trajectory according to a pre-defined cost function, which takes into account various constraints, including the risk of collision with dynamic or static obstacles in the environment and the distance to the final target point; Then, the algorithm calculates a weight for each trajectory according to its cost value, and the trajectory with a lower cost value is assigned a higher weight; Finally, the current optimal control command is calculated by weighted averaging of all candidate control sequences and applied to the robot arm. Through parallel computing on the GPU, the entire "sampling-prediction-evaluation-weighting" process is executed at a high frequency, enabling the robot arm to respond to environmental changes in real time and achieve dynamic and efficient autonomous obstacle avoidance and path planning.

5. A method for autonomous grasping of a robot arm based on visual guidance and vision-haptics fusion, characterized in that: The method comprises the following steps: S1: visual perception and pose generation: acquire environmental visual information, identify target objects, and process the point cloud of the target objects using a grasp detection network to generate a six-degree-of-freedom grasp pose; S2: compliant grasping execution: drive the robot arm to perform grasping according to the generated pose, and adjust the clamping force in real time through visual-tactile fusion feedback control to achieve stable grasping; S3: dynamic path planning and movement: after stable grasping, use the model predictive path integral algorithm to plan and execute a collision-free motion trajectory to the target point.

Images